Facebook is one of the largest social media platforms in the world, with over 2.8 billion monthly active users as of Q3 2021. With so many people using Facebook to connect, share information, and express themselves, content moderation is a significant challenge.
Facebook relies on a combination of automated systems and human reviewers to moderate content on its platforms. However, with billions of posts, photos, videos, and comments uploaded every day, it’s impossible for Facebook to review everything.
That’s why Facebook developed moderation assist tools to help prioritize content for human reviewers to assess. Moderation assist uses machine learning to detect potentially problematic content and send it for human review.
But what kind of comments should moderation assist hide or remove from Facebook? Here are some factors to consider:
Hate Speech
One of the most problematic types of content on social media is hate speech. This includes comments that attack or dehumanize others based on characteristics like race, religion, nationality, gender, sexual orientation, disability, or other traits.
For example, moderation assist should flag comments using racial slurs, calling for violence against minority groups, or accusing entire religions of terrorism. Allowing hate speech to spread can normalize prejudice, promote radicalization, and lead to real-world violence.
Examples of hate speech
- “All [racial slur] are criminals and should be kicked out of this country!”
- “I hope someone shoots up that [religious group] and their heathen temples.”
- “[Gender] are inferior and shouldn’t be allowed to vote or hold jobs.”
Violent Threats
Threatening violence against individuals or groups also has no place on Facebook. Moderation assist should prioritize comments that threaten physical harm, including explicit or implicit threats:
“I’m going to [violent act] you [target].”
“[Target] deserves to die a painful death!”
“Can’t wait to watch [tragic event] happen to [target].”
Allowing violent threats normalizes them and can lead to real-world harm. Facebook has a duty to protect users from threats on their platform.
Examples of violent threats
- “I’ll punch your face in if I ever see you on the street.”
- “Someone should bomb their headquarters for what they did.”
- “I hope you get hit by a truck and end up paralyzed.”
Bullying and Harassment
Bullying and harassment also make Facebook an unsafe space for many users. Moderation assist should flag:
- Comments mocking, insulting, or demeaning an individual or group
- Sexually harassing comments focused on gender, appearance, or sexuality
- Repeated negative comments about a user from multiple accounts
- Malicious rumors or lies about a user
This type of behavior directly attacks other users and violates Facebook’s bullying and harassment policies. Failing to address it facilitates toxic behavior.
Examples of bullying and harassment
- “You are so ugly and stupid.”
- “I hope you get raped.”
- “Everyone knows [rumor/lie] about you.”
Private Information
Posting someone’s private or confidential information without their consent, also known as doxxing, can put them at risk for further harassment or identity theft. Moderation assist should flag:
- Home addresses, phone numbers, email addresses
- Identity documents like social security numbers
- Bank account and payment details
- Private photos and videos
This gives the subject a chance to have the information removed before it leads to real-world harm.
Examples of private information
- “Here is [name]’s phone number – [xxx-xxx-xxxx].”
- “I have [name]’s nude photos that she sent me privately.”
- “[name] lives at [address] and works at [company].”
Scams
Scams are also common on Facebook, designed to profit from users through deception. Moderation assist should flag:
- Pyramid schemes or multi-level marketing pitches
- Loan offers with impossibly good terms
- Free gift card offers or competitions that require an upfront payment
- Crypto investment “opportunities” guaranteeing unrealistic returns
These prey upon vulnerable populations and have no place on the platform. Facebook has a duty to protect users from predatory scams.
Examples of scams
- “Join my essential oils team and earn thousands per month from home!”
- “Free $500 Walmart gift card – just pay $1 shipping fee.”
- “Turn $100 into $50,000 with this hot new crypto coin!”
Misinformation
Facebook also struggles with large amounts of misinformation, especially around politics, health, and science. Moderation assist should flag:
- Health claims about miracle cures or anti-vaccine propaganda
- Climate change denial arguments contradicting scientific consensus
- Election and voting misinformation or conspiracy theories
- Other conspiracy theories around events like mass shootings
This misinformation erodes public trust and causes real-world harm to individuals and institutions. Facebook must balance free speech with users’ safety.
Examples of misinformation
- “Cancer is easily cured with this special diet – doctors don’t want you to know!”
- “Climate change is a hoax created by the Chinese.”
- “Ballots were dumped in swing states to steal the election.”
Illegal Activities
Finally, moderation assist also needs to flag content facilitating illegal activities on Facebook, including:
- Drug and weapons sales
- Soliciting minors for sexual activity
- Sharing pirated copyrighted content
- Content violating Facebook’s Terms of Service
Facebook has to comply with laws regarding illegal content and activities on their platform. Proactively detecting this protects users and Facebook itself.
Examples of illegal activities
- “I have [illegal drug] for sale, high quality product!”
- “Who wants access to [pirated movie]? Just Venmo me.”
- “I’ll pay for nude photos of any underage teens.”
Balancing Free Speech with Safety
Moderating a platform used by billions is an enormous challenge. Facebook must balance free speech with keeping users safe from real harm.
While moderation assist should flag potentially violating content for review, human moderators ultimately make final decisions about what stays up or comes down based on careful review of context.
No automated system can perfectly detect the nuance of language and intent. But prioritizing the most dangerous types of content allows human reviewers to focus their efforts.
With a thoughtful, nuanced approach combining humans and AI, Facebook can work to protect its users while allowing free expression.
Conclusion
Facebook’s moderation assist tools have a vital purpose – detecting and flagging content like hate speech, threats, harassment, scams, misinformation, and illegal activities for human review.
This allows Facebook to balance its values of free speech and connecting people with keeping users safe from real harm.
While not perfect, prioritizing the most dangerous types of content helps Facebook moderate billions of posts with limited resources.
With a thoughtful approach and responsible policies, Facebook can meet its moral obligation to limit real-world damage through harmful content shared on its platforms.
No system can remove human judgment. But combining the power of machines and people offers the best path forward for social platforms.