Editor’s Note: This story contains images and language that some readers may find disturbing.
Mark Zuckerberg — one of the most insightful, adept leaders in the business world — has a problem. It’s a problem he has been slow to acknowledge, even though it’s become more apparent by the day.
Several current and former Facebook employees tell NPR there is a lot of internal turmoil about how the platform does and doesn’t censor content that users find offensive. And outside Facebook, the public is regularly confounded by the company’s decisions — around controversial posts and around fake news.
Behind whatever the controversy of the moment happens to be, there’s a deep-seated problem. The problem is this: At age 19, the then-boy genius started a social network that was basically a tech-savvy way to check out classmates in school. Then, over the course of 12 years, he made some very strategic decisions that have morphed Facebook into the most powerful distributor on Earth — the new front page of the news for more than 1 billion people every day. But Zuckerberg didn’t sign up to head a media company — as in, one that has to make editorial judgments.
He and his team have made a very complex set of contradictory rules — a bias toward restricted speech for regular users, and toward free speech for “news” (real or fake). And the company relies on a sprawling army of subcontractors to enforce the rules. People involved in trying to make it work say they’re in way over their heads. As one employee put it, “We started out of a college dorm. I mean, c’mon, we’re Facebook. We never wanted to deal with this s***.”
Subcontractors running the show
NPR got the official version of how the company censors and leaves up content this summer when Facebook’s head of policy, Monika Bickert, agreed to a phone interview. We spoke with 10 current and former employees total, on the record and on background, for this investigation.
It’s hard to remember this sometimes, but Facebook has never claimed to be a free-speech platform. The company is trying to create a safe space where, unlike on Twitter, people can share without being trolled or shamed. Bickert is in charge of setting the content policies. The Community Standards, which are posted online, are the rules for everyday users.
She explained that when a user reports a piece of content that might be offensive, the company exercises its power to censor with precision.
“Context is so important. It’s critical when we are looking to determine whether or not something is hate speech, or a credible threat of violence,” she said. “We look at how a specific person shared a specific post or word or photo to Facebook. So we’re looking to see why did this particular share happen on Facebook? Why did this particular post happen?”
However, three of Bickert’s former colleagues tell a very different story of how Facebook deals with controversial content. They and others declined to be named for fear of job repercussions (at Facebook or at their current employers, also Internet companies), but their descriptions are consistent with each other.
When a user flags a post on Facebook — whether it’s a picture, video or text post — it goes to a little-known division called the “community operations team.”
In 2010, the sources say, the team had a couple hundred workers in five countries. Facebook found it needed more hands on deck. After trying crowdsourcing solutions like CrowdFlower, the company turned to the consulting firm Accenture to put together a dedicated team of subcontractors. Sources say the team is now several thousand people, with some of the largest offices in Manila, the Philippines, and Warsaw, Poland.
Current and former employees of Facebook say that they’ve observed these subcontractors in action; that they are told to go fast — very fast; that they’re evaluated on speed; and that on average, a worker makes a decision about a piece of flagged content once every 10 seconds.
Let’s do a back-of-the-envelope calculation. Say a worker is doing an eight-hour shift, at the rate of one post per 10 seconds. That means they’re clearing 2,880 posts a day per person. When NPR ran these numbers by current and former employees, they said that sounds reasonable.
A Facebook spokesperson says response times vary widely, depending on what is being reported; that the vast majority of flagged content is not removed; and that the numbers are off. Facebook did not provide alternative numbers.
If the sources — who have firsthand knowledge and spoke separately with NPR — are correct, then this may be the biggest editing — aka censorship — operation in the history of media. All the while, Facebook leaders insist they’re just running a “platform,” free of human judgment.
A person who worked on this area of “content management” for Facebook (as an employee, not a subcontractor) says most of the content you see falls neatly into categories that don’t need deep reflection: “That’s an erect penis. Check.” So it’s not like the workers are analyzing every single one in detail.
The problem is, simple and complex items all go into the same big pile. So, the source says, “you go on autopilot” and don’t realize when “you have to use judgment, in a system that doesn’t give you the time to make a real judgment.”
A classic case is something like “Becky looks pregnant.” It could be cyberbullying or a compliment. The subcontractor “pretty much tosses a coin,” the source says.
Here’s another huge barrier. Because of privacy laws and technical glitches (such as a post that is truncated to only show part of the conversation), the subcontractors typically don’t get to see the full context — to which Bickert referred so often.
That could be the cause of frequent errors.
NPR decided to stress-test the system by flagging nearly 200 posts that could be considered hate speech — specifically, attacks against blacks and against whites in the U.S. We found that Facebook subcontractors were not consistent and made numerous mistakes, including in instances where a user calls for violence.
We say they were mistakes because the company changed its position in dozens of instances, removing some and restoring others — either when we flagged it a second time through the automated system or brought it to the attention of Facebook headquarters in Menlo Park, Calif.
Consider this post: