Inside the Making of Facebook’s Supreme Court
The company has created a board that can overrule even Mark Zuckerberg. Soon, it will decide whether to allow Trump back on Facebook.
On a morning in May, 2019, forty-three lawyers, academics, and media experts gathered in the windowless basement of the NoMad New York hotel for a private meeting. The room was laid out a bit like a technologist’s wedding, with a nametag and an iPad at each seat, and large succulents as centerpieces. There were also party favors: Facebook-branded notebooks and pens. The company had convened the group to discuss the Oversight Board, a sort of private Supreme Court that it was creating to help govern speech on its platforms. The participants had all signed nondisclosure agreements. I sneaked in late and settled near the front. “Clap if you can hear me,” the moderator, a woman dressed in a black jumpsuit, said.
Since its founding, in 2004, Facebook had modelled itself as a haven of free expression on the Internet. But in the past few years, as conspiracy theories, hate speech, and disinformation have spread on the platform, critics have come to worry that the company poses a danger to democracy. Facebook promised to change that with the Oversight Board: it would assemble a council of sage advisers—the group eventually included humanitarian activists, a former prime minister, and a Nobel laureate—who would hear appeals over what kind of speech should be allowed on the site. Its decisions would be binding, overruling even those of Mark Zuckerberg, the company’s founder. Zuckerberg said he had come to believe that a C.E.O. shouldn’t have complete control over the limits of our political discourse. “Maybe there are some calls that just aren’t good for the company to make by itself,” he told me.
In 2019, Facebook agreed to let me report on the process, and I spent eighteen months following its development. Last month, the board ruled on its first slate of cases, which dealt with, among other topics, the glorification of Nazis and misinformation about the coronavirus pandemic. In the next few months, it will decide an even larger question: whether Donald Trump should be cut off indefinitely from his millions of followers for his role in inciting the insurrection at the Capitol, on January 6th. Nathaniel Persily, a law professor at Stanford, told me, “How the board considers the issues and acts in that case will have dramatic implications for the future of the board, and perhaps for online speech in general.”
In the beginning, Facebook had no idea how the board would work. To come up with ideas, the company held workshops with experts in Singapore, New Delhi, Nairobi, Mexico City, Berlin, and New York. “My job was to go all over the world and get as much feedback as possible,” Zoe Darmé, who oversaw the consultation process, told me. At the workshop in New York, in the hotel basement, participants sat in tables of eight or nine and ran simulations of cases. I sat between Jeff Jarvis, a journalism professor, and Ben Ginsberg, a Republican lawyer who represented George W. Bush in Bush v. Gore.
For our first case, the moderator projected a picture of a smiling girl in a yearbook photo, with a cartoon thought bubble that read “Kill All Men.” Facebook had removed the post for violating its hate-speech rules, which ban attacks based on “sex, gender identity.” To many, this seemed simplistic. “It’s a joke,” one woman said. “There has to be an exception for humor.” Facebook’s rules did include a humor exception, for instances in which the user’s intent was clear, but it was difficult to discern this person’s motivation, and attendees worried that a broad carve-out for jokes could easily provide cover for hate speech. Carmen Scurato, who works at Free Press, an Internet-advocacy organization, pointed out the historical disadvantage of women, and argued that hate-speech policies ought to take power dynamics into account. In the end, the group voted to restore the photo, though no one knew exactly how to write that into a rule.
This kind of muddy uncertainty seemed inevitable. The board has jurisdiction over every Facebook user in the world, but intuitions about freedom of speech vary dramatically across political and cultural divides. In Hong Kong, where the pro-democracy movement has used social media to organize protests, activists rely on Facebook’s free-expression principles for protection against the state. In Myanmar, where hate speech has contributed to a genocide against the Rohingya, advocates have begged for stricter enforcement. Facebook had hoped, through the workshops, to crowdsource beliefs about speech, but the results were more contradictory than anticipated. In New York, for example, sixty per cent of people voted to reinstate the “Kill All Men” post, but only forty per cent did so in Nairobi. Amid other theories, Darmé speculated, “Where countries are perhaps more concerned about safety, because they live in an area with less rule of law—and therefore there’s a chance of a group actually maybe trying to kill all men—there’s less concern about free speech.” The full explanation is likely more complex; regardless, the divergent results underscored the difficulty of creating a global court for the Internet.
Some of the workshops devolved into disarray. In Singapore, Nairobi, and New Delhi, a few participants refused to sign the nondisclosure agreements, protesting Facebook’s lack of transparency; in Germany, someone commandeered the microphone and berated the company for killing democracy. “We had to learn to put on our body armor,” Darmé said. In New York, the session remained civil, but just barely. Some participants thought that the board would be ineffectual. “The whole thing seemed destined for failure,” Sarah T. Roberts, a professor of information studies at U.C.L.A., told me. “Skeptics will think it’s captured by the corporate interest of Facebook. Others will think it doesn’t do enough, it’s a pseudo-institution.” Some predicted the board would come to have grand ambitions. Tim Wu, a law professor at Columbia, said, “If the board is anything like the people invited to New York, I wouldn’t be surprised if it got out of control and became its own little beast that tried to change the world one Facebook decision at a time.”
Participants had been instructed to use an app called Slido to submit questions for group discussion, which could be voted up or down the agenda. The results were projected on a screen at the front of the room. The app had worked well abroad, but in New York it became a meta-commentary on content moderation. Sophisticated questions about “automatic tools for takedowns” and the “equity principle with diverse communities” were soon overtaken by a joke about “Game of Thrones.” Posts were initially anonymous, but users quickly found a way around the system; “Harold” wrote, “I figured out how to identify self,” which provoked laughter. The moderator shouted to regain control of the room. In the midst of the chaos, someone posted, “Can we abandon Slido and talk?” which quickly accumulated likes.
The idea that Facebook, like a fledgling republic, would need to institute democratic reforms might have seemed silly a decade ago. In 2009, shortly after the company was criticized for quietly changing its terms of service to allow it to keep users’ data even after they deleted their accounts, it released a video of Zuckerberg, clad in an uncharacteristic button-up shirt and a tie, announcing a “new approach to site governance.” People would be able to vote on Facebook’s policies; the company called it “a bold step toward transparency.” In the first referendum, on whether to change the terms of service, only 0.32 per cent of users voted. “In its own eyes, Facebook has become more than merely a recreational website where users share photos and wish each other a happy birthday,” a columnist for the Los Angeles Times wrote. “It is now a global body of citizens that should be united and protected under a popularly ratified constitution. But it’s hard to have a democracy, a constitution or a government if nobody shows up.” In 2012, the project was quietly shuttered, and, as with Crystal Pepsi, Google Wave, and the Microsoft Zune, no one remembers that it existed.
This was still a hazily optimistic time for Facebook. The company promised to “give people the power to share and make the world more open and connected.” As more users joined tech platforms, companies instituted rules to sanitize content and keep the experience pleasant. Airbnb removed housing ads that displayed Nazi flags; Kickstarter disallowed crowdfunding for “energy food and drinks”; Etsy told users to be “helpful, constructive, and encouraging” when expressing criticism. Facebook hired content moderators to filter out pornography and terrorist propaganda, among other things. But, because it saw itself as a “neutral platform,” it tended not to censor political speech. The dangers of this approach soon became apparent. Facebook now has some three billion users—more than a third of humanity—many of whom get their news from the site. In 2016, Russian agents used the platform in an attempt to sway the U.S. Presidential election. Three years later, a white supremacist in New Zealand live-streamed a mass shooting. Millions of people joined groups and followed pages related to QAnon, a conspiracy theory holding that the world is controlled by a cabal of Satan-worshipping, pedophilic Democrats. The First Amendment has made it difficult for the U.S. government to stop toxic ideas from spreading online. Germany passed a law attempting to curb the dissemination of hate speech, but it is enforceable only within the country’s borders. As a result, Facebook has been left to make difficult decisions about speech largely on its own.