Content moderation is the process of monitoring and removing harmful, illegal, or otherwise inappropriate user-generated content from online platforms. This can include graphic violence, hate speech, pornography, harassment, misinformation, and more. While vital for maintaining a safe online environment, constant exposure to such disturbing material can take a serious toll on content moderators’ mental health and wellbeing.
What do content moderators do?
Content moderators are tasked with reviewing posts, images, videos, and other content uploaded by users across social media, forums, and other platforms. This involves:
- Scanning through large volumes of content to identify policy violations
- Making quick decisions on whether to remove or restrict access to violating content
- Escalating complex cases to supervisors
- Providing feedback to users whose content is removed
Due to the sheer amount of content generated daily, moderators may review hundreds or even thousands of disturbing images, videos, and text every shift. They are exposed to the worst of humanity and the internet, including graphic depictions of violence, hate, terrorism, animal cruelty, child exploitation, and more.
What types of traumatic content do moderators view?
Here are some examples of the deeply disturbing content regularly encountered by moderators:
- Videos of murders, suicides, torture, rape
- Mass shooting footage
- Graphic war imagery
- Bestiality, necrophilia
- Hate speech and graphic threats against people
- Child sexual abuse material
- Extreme pornography and fetish content
Moderators may be exposed to thousands of such images in a single 8-hour shift. The content is often horrific, shocking, and sticks with moderators long after their work day ends.
What are the mental health effects on moderators?
Repeated exposure to such traumatic content can have profound consequences for moderators’ mental health, including:
- PTSD: Many moderators exhibit symptoms of post-traumatic stress disorder, including anxiety, insomnia, nightmares, and flashbacks.
- Depression: Viewing such negative imagery can lead to feelings of sadness, hopelessness, anger, and numbness.
- Substance abuse: Some moderators turn to drugs or alcohol to cope with the stress.
- Suicidal thoughts: Severe depression as a result of the work may lead to suicidal thinking.
- Cynicism: Constant exposure to the worst of humanity can lead to a bleak, cynical worldview.
In essence, absorbing hours of graphic and harmful content daily can be psychologically damaging. Without proper support, moderators’ mental health suffers dramatically.
What are other job impacts?
In addition to mental health consequences, content moderation can impact other aspects of moderators’ lives and performance:
- Physical strain: The sedentary nature of moderation and need for accuracy subjects workers to eyestrain, headaches, wrist pain, and more.
- Desensitization: With enough exposure, moderators may become desensitized to cruelty, violence, and harm.
- Empathy loss: Excessive exposure can reduce empathy and emotional reactivity over time.
- Personal relationships: Moderators often feel unable to discuss the abhorrent content with friends and family, leading to social isolation.
- Poor job performance: Trauma symptoms like lack of concentration, fatigue, and mood changes can impair work quality.
Constant immersion in the darkest corners of the internet desensitizes moderators and detaches them emotionally from their work. This spills over into their personal lives, harming relationships and overall wellbeing.
What workplace factors increase risk?
Certain workplace conditions and demands often exacerbate the impacts of content moderation:
- Production pressure: Quotas requiring reviewers to moderate hundreds of pieces of content an hour allow little time for self-care.
- Repetitive exposure: Viewing thousands of disturbing images daily without sufficient breaks creates a cumulative effect.
- Insufficient training: Moderators thrown into the role without guidance on how to handle the content are at greater risk.
- Understaffing: Too few moderators leads to extreme workloads and increased burnout risk.
- Low pay: Compensation is often low for the highly complex nature of content moderation work.
- Weak social support: A lack of team bonding and peer support exacerbates feelings of isolation.
High production quotas, short breaks between exposures, information overload, understaffing, low pay, and social isolation worsen the toll of content moderation on workers.
What are signs of moderator burnout?
Without interventions to protect their wellbeing, moderators often experience ‘compassion fatigue’ and burnout. This manifests through:
- Chronic stress and anxiety
- Persistent low mood and cynicism
- Physical and emotional exhaustion
- Feeling constantly traumatized and ‘numb’
- Detachment from friends, family, and others outside work
- Sleep disturbances and lack of concentration
- Using negative coping mechanisms like substance abuse
- Frequent sick days and absence from work
In essence, burnout reflects the cumulative toll of constant trauma exposure, inadequate support, and persistent stress moderators face daily.
What are some real-world examples of moderator trauma?
| Company | Examples of Moderator Trauma |
|---|---|
| -Moderator developed PTSD after months of viewing child exploitation, bestiality, torture, and suicide footage -Sueed for poor working conditions leading to PTSD |
|
| YouTube | -Moderators exposed to hate speech, animal abuse, pornography and other disturbing content -One moderator attempted suicide after viewing distressing images |
| Microsoft | -BBC investigation found Microsoft moderators in the US viewed extreme pornography, bestiality, and child abuse images -Moderators described crying, sleep loss, and having to take leaves due to trauma |
These examples demonstrate the profound mental health impacts moderation work can have without proper safeguards in place. The consequences of constant exposure to the worst of humanity should not be underestimated.
How can companies support moderator wellbeing?
Platforms have an ethical obligation to implement support systems and safeguards that protect moderators from trauma and burnout. This should include:
- Psychological support: On-site counseling, trauma therapy, and peer support groups.
- Improved workflow: Reasonable quotas, frequent breaks, control over exposure types, and flexibility.
- Robust wellness policies: Generous leave, mental health days, relaxation spaces, and workshops on coping mechanisms.
- Training: Resilience building, trauma management, suicide prevention, and mindfulness.
- Community: Team bonding opportunities and spaces that cultivate social support.
- Fair compensation: Wages commensurate with the role’s demands and hazards.
Holistic wellbeing support and humane working conditions are essential duty of care responsibilities platforms owe their moderators.
Conclusion
Content moderation plays a vital role in maintaining positive online communities. However, constant exposure to the most horrific, graphic, and abusive content takes an immense toll on moderators’ mental health and wellbeing. Without proper workplace support, moderators face immense risk of PTSD, depression, burnout, and even suicide ideation. Companies have an ethical obligation to implement comprehensive psychological care, humane workflows, resilience training, and other robust wellbeing policies. Only then can the safety of online spaces be upheld without endangering frontline moderators.