Federated platforms face considerable obstacles to robust and scalable governance, particularly with regard to persistent threats such as coordinated behavior and spam.
Yoel Roth is a nonresident scholar in the Technology and International Affairs Program. He is also a technology policy fellow at the University of California Berkeley Goldman School of Public Policy and the former head of trust and safety at Twitter. For more than seven years, he led the teams responsible for Twitter’s content moderation, integrity, and platform security efforts, including policy development, threat investigation, product, design, research, and operations. His current research and writing focus on the trust and safety industry and how technology companies manage the conflicting values and incentives built into content moderation at scale. Before joining Twitter, Yoel received his PhD from the Annenberg School for Communication at the University of Pennsylvania. His research and teaching examined the technical, policy, business, and cultural dynamics of social networking and dating apps at the dawn of the “App Store” age.
Federated platforms face considerable obstacles to robust and scalable governance, particularly with regard to persistent threats such as coordinated behavior and spam.
The broader challenge here — and perhaps, the inescapable one — is the essential humanness of online trust and safety efforts. It isn’t machine learning models and faceless algorithms behind key content moderation decisions: it’s people. And people can be pressured, intimidated, threatened and extorted.
Many discussions about social media governance and trust and safety are focused on a small number of centralized, corporate-owned platforms. The emergence and growth in popularity of federated social media services introduces new opportunities, but also significant new risks and complications.
Major social media and technology companies continue to make algorithmic, user interface, and policy changes to their products to address information integrity challenges on their platforms.