Researchers, civil society actors, and policymakers have long urged online platforms and search engines to standardize their operational reporting.1 The European Commission’s Strengthened Code of Practice on Disinformation (CoP), published in 2022, exemplifies this approach. The CoP will become part of a broader co-regulatory framework with the Digital Services Act and the legislation on Transparency and Targeting of Political Advertising.2 With thirty-four signatories, including most major online platforms based in North America, CoP represents a framework for companies to report periodically and systematically on their interventions to counter disinformation.
So far, companies have submitted two sets of reports, published in February 2023 and September 2023, covering the period of January to July 2023.3 The legislation’s efforts to bring diverse stakeholders to the table to standardize transparency reporting has been commendable, yet the initial reports have been criticized for incomplete data that renders cross-platform comparison incomparable and for answers with limited usefulness.4 Going forward, how can companies and policymakers work toward harmonized reporting? And how can reporting requirements be improved to provide better insight into a company’s efforts to counter disinformation and other types of information pollution?5
In Service of Understanding Impact
The existing framework of reporting for the CoP mainly asks platforms to detail the actions taken (for example, the number of accounts taken down), or number of impressions (for example, the number of people exposed to a specific fact-check label). However, this information has its limitations. Activities do not necessarily produce outcomes. Even if disinformation on a topic has been removed from one online platform, it does not prove that users have changed their beliefs on that topic. And even if an individual is exposed to a certain intervention such as an information hub or fact-check label, it does not mean that they are convinced by it.
What researchers and policymakers most want to know is whether platform interventions resulted in a change in how users engage with disinformation. That is why impact measurements are necessary in addition to measurements of actions and impressions. How was a given intervention intended to cause a change in cognition or behavior? Was the intervention successful in producing such a change? In an ideal world, impact metrics should help analysts evaluate whether interventions were effective in addressing the problem of disinformation. However, while measuring the effectiveness of interventions is important, it is also challenging in practice, and rarely reported by platforms.6
To understand how to include more impact measurements into the CoP, we analyzed the first two sets of reports published by online platforms that were signatories to the CoP. For the first reports submitted in January 2023, this included Adobe, Google, Microsoft, Meta, TikTok, Twitch, Twitter (which signed the CoP but subsequently withdrew in May 2023), and Vimeo.7 In July 2023, Google, Meta, Microsoft, and TikTok all submitted updated reports. We created a database accounting for the frequency of interventions, whether platforms reported the actions, impressions, and impacts associated with each intervention, and suggested additional impact metrics platforms could report on.
Several platforms disclosed concrete impact measurements that could be used to understand how effective different interventions are in addressing disinformation. Google and Microsoft used internal data on click-through rates and estimated impression costs to calculate the potential revenue demonetized pages and domains lost. This reporting sheds light on the degree of financial impact demonetization had on users that violated site policies. TikTok reported users’ share cancel rates after seeing an “unverified content” label, while Meta reported the percentage of users who did not click through to the original content after seeing fact-check labels on a post. Both were measurements of impact, as they calculated changes in user behavior after users were exposed to an intervention. These are positive and welcome developments. Unfortunately, this made up the minority of reporting. For the first round of reporting, while six of eight platform reports contained measurements of impact, these measurements only constituted 13 percent of all coded measures. These rates remained proportionately similar for the second set of reports.
In addition to companies’ reports, the CoP also publishes semiannual reports on structural indicators intended to assess the impact of the CoP, in terms of its “effectiveness . . . in reducing the spread of online disinformation.”8 The European Digital Media Observatory developed an initial proposal for structural indicators that could be used to assess the effectiveness of the CoP in May 2023.9 Following that, the first report by TrustLab was released in September 2023. The report examines two of the six proposed structural indicators, namely the prevalence and sources of disinformation, in Poland, Spain, and Slovakia.10
When researchers and policymakers communicate findings from the structural indicators report, they should note that the report draws from a strategic sample of posts and accounts based on searches for popular disinformation keywords and is not representative of disinformation across the platform. In future reports, stakeholders involved in writing the report should also explore ways to share the posts, keywords, and claims analyzed, so that their results could be replicated by third-party researchers.
The structural indicators report has the potential to present cumulative metrics on the presence of disinformation on platforms, which allows researchers and policymakers to evaluate the effectiveness of the CoP in the long run. However, this does not mitigate the need for impact measurement on an intervention-by-intervention level, which provides insight on the effectiveness of individual interventions, and contextualizes the role they play in different platforms’ Trust & Safety architecture.
Quick Fixes: Increasing Impact Measurements
There are several ways platforms and policymakers can experiment with increasing impact measurements in platform reporting. Commitment 18 of the CoP calls for signatories to “commit to minimize the risk of viral propagation of Disinformation by adopting safe design practices as they develop their systems, policies and features.” Going beyond qualitative reporting, evaluating the impact of algorithmic changes can include examining whether people engage more frequently with elevated content and less frequently with harmful content. It is standard practice within the tech industry to test algorithm and user interface changes experimentally with A/B testing, for instance to observe whether those changes increase or decrease usage of a platform. Under Commitment 18, platforms could share aggregate results of the A/B testing conducted prior to implementing algorithmic changes intended to decrease the circulation of disinformation.11 In their reporting, platforms could report on testing methodology and the impacts of the intervention in the testing process, including changes in user views and engagement with elevated or deplatformed content.
Under the reporting section on “platforms’ pledge to empower users,” which ranges from Commitments 17 to 25, platforms reported on a range of deployed content labels—either for fact-checking or information panels for more context—and special search features. To demonstrate the effect of these interventions, the CoP can also ask platforms to report on click-through rates to third-party links, canceled re-share rates, and reshare rates of fact-checks. Companies can report on the effectiveness of different user experience (UX) treatments, such as the types of labels, pop-ups, and language users see when interacting with interventions. This can help researchers and tech companies understand the types of labels and panels that are the most effective.
In the reporting categories for “empowering the research community” and “empowering the fact-checking community,” companies reported on their media literacy and research partnerships with civil society organizations and academic researchers. Though they often discussed who they funded and how much money was distributed, they stopped short of reporting the outcomes of such partnerships. TikTok and Microsoft reported how their partnerships with civil society organizations impacted either content removal on their platforms or the effectiveness of their existing tools. In future iterations of their reports, companies could include descriptions of the projects they funded, how research outcomes informed interventions made by the companies, and the effectiveness of the intervention. For example, when companies report on a media literacy campaign, they can go beyond reporting the funding they provided, and report on the effectiveness of the campaign in changing user behavior.
In reporting on platform responses to the COVID-19 pandemic and the war of aggression by Russia on Ukraine, both Microsoft and Google reported on the number of accounts, ads, or content taken down and impressions prior to removal. To extend this analysis to impact measurements, other innovative metrics can help. For instance, TikTok reported the number of fake likes and followers prevented from their takedown of content, and the buckets of topics that removed content fell into (for example, COVID-19, Holocaust denial). Additional metrics can include the average time that posts or accounts were up before removal, which Microsoft included in its reporting, and the recidivism rate of accounts or content that has been taken down, which refers to the number of users that rejoin the platform after their accounts have been banned. Measurements of fake likes that were prevented and recidivism rates will likely be methodologically complex. In their reporting, platforms should include adequate detail to help researchers contextualize how these numbers were calculated.
At this early stage, both signatories and policymakers will need time to hone reporting requirements and the information they have ready to report out. Platforms have demonstrably improved their reporting from their first to the second round. Instead of reporting on the total number of videos removed as a single number, Google reported on the number of removed videos based on different ranges of views. TikTok provided reporting on the percentage of videos removed as a result of a fact-checking assessment relative to the number of videos removed for violating misinformation policy, to demonstrate the significance of the measure relative to the rest of the company’s content removal infrastructure. These are welcome improvements that can be built on. While reporting could be expanded in a breadth of different ways, placing an emphasis on impact reporting creates the potential for research and evaluation that is difficult to conduct with existing data.
Addressing Structural Concerns
Structural differences across platforms can make standardized operational reporting a lofty goal. Online platforms differ in functionality—whether they are search engines or social media platforms, even whether the content they provide is text-based or video-based. The existing CoP accounts for this by letting platforms sign on to different measures. However, the difference in functionality and structure changes the kinds of data they can collect and report on, contradicting the goal of standardized reporting. Moreover, companies differ in their operational capacity. Reports from Adobe and Twitch were sparse, possibly due to capacity limitations compared to other online platform signatories, but also because most measures were asking them for data they did not or could not collect. For the first round of reporting submitted in January 2023, Google did not fill in the framework provided by the CoP for Commitment 14 on impermissible manipulative behavior, linking instead to its Threat Analysis Group Bulletin, as their internal record-keeping infrastructure did not map onto the framework provided by the CoP.12 They did, however, provide reporting for that measure in the second set of reports.
Beyond differences internal to platforms, regulatory measures restrict certain types of operational reporting that may help observers understand whether measures to combat disinformation and other information pollution work. Data privacy laws, such as the General Data Protection Regulation (GDPR), put restrictions on processing and maintaining records of personal data.13 In some cases, this means either platforms cannot log certain kinds of data and in other cases, they may lose the data entirely.
Given the structural issues internal and external to companies, what can policymakers and the companies do to help improve operational reporting in the CoP? First, in compliance with data privacy laws, companies can assess the kinds of data they are not currently recording and logging that would result in meaningful impact measurements. Changing data reporting structures takes time and validation by data scientists. As future iterations of the reports roll out, companies’ policy teams should adapt their data logging and reporting structures. Second, as part of the DSA, platforms will be expected to undergo audits examining their compliance with the DSA and the CoP. Part of the process can include auditors with data science expertise that can work with company representatives to understand the data companies have available and develop ways to adapt data collection frameworks for standardization across companies.
A Path Forward for Researchers
Platforms reporting on their operations is only one piece of understanding the role online platforms play in the information environment. Longitudinal analysis, cross-platform research, and processing of data released are responsibilities of researchers. Assessments of measures included in the CoP and repositories of academic studies on platform interventions create reference points for policymakers on actions helpful in combating disinformation.14
One of the motivations for developing the CoP was to provide researchers with data that could be used for cross-platform research. However, at present, data released through the CoP have received limited researcher engagement beyond a report from the European Digital Media Observatory’s Ireland Hub and the German-Austrian Digital Media Observatory.15 This report assessed how well signatories fulfilled the obligations they signed onto in their baseline reports, and how the CoP could be edited to enable more effective reporting.
There are several reasons why engagement with the CoP has been limited so far. The format and quality of data released makes it difficult for researchers to conduct quantitative analysis. Company reports are released as PDFs instead of CSVs, making it difficult for researchers to extract the data they need. Another possible reason is that the policy gaps the CoP seeks to address do not align with research agendas or make for the right research questions. For example, while policymakers behind the CoP might be more focused on measuring the number of influence operations campaigns taken down across platforms, researchers may be more interested in identifying recurring actors and specific patterns of behavior from these campaigns. Going forward, policymakers are exploring ways for researchers to conduct long-term structured monitoring of the CoP, to find ways to integrate researchers’ insights on how the CoP could be contextualized in a way that is beneficial to research. One way to do this is to create funding structures for researchers to develop evaluations such as scorecards of how well different companies perform, or databases of companies’ reported interventions and their effectiveness.
Researchers’ current lack of engagement and capacity is indicative of a larger structural problem. The existing research model is broken. Most grants are issued on a project-by-project basis, incentivizing researchers to reproduce costly engineering infrastructure and limiting them from conducting work that builds on existing efforts. The costs of storing, securing, and processing data can rise into the millions of dollars, leaving it inaccessible to most researchers, especially from the Majority World. A complete reconfiguration of the current research model through the creation of a large-scale multistakeholder body can help centralize resources and close gaps in funding, tooling, and knowledge-building.16 A multistakeholder research body can help cross-reference existing research, conduct and maintain meta-studies, and connect funders with those working on creating a safe information environment.
The Code of Practice on Disinformation is working toward the meaningful goal of harmonized operational reporting across countries. This is an important step in the right direction for democracies and online platforms in addressing the myriad of problems online. In conjunction with increased researcher data access for longitudinal research, enabled by Article 40 of the Digital Services Act (DSA), which mandates that Very Large Online Platforms (VLOPs) share data with vetted researchers, there is hope that researchers, policymakers, and platforms can take collaborative effort in understanding the real-life effects of online activity.17
Notes
1 Samantha Lai, Naomi Shiffman and Alicia Wanless, “Operational Reporting By Online Services: A Proposed Framework,” Carnegie Endowment for International Peace, May 18, 2023, https://carnegieendowment.org/2023/05/18/operational-reporting-by-online-services-proposed-framework-pub-89776; Aleksandra Urman and Mykola Makhortykh, "How transparent are transparency reports? Comparative analysis of transparency reporting across online platforms," Telecommunications Policy 47, no. 3 (2023): 102477.
2 European Commission, “2022 Strengthened Code of Practice on Disinformation,” June 16, 2022, https://digital-strategy.ec.europa.eu/en/library/2022-strengthened-code-practice-disinformation.
3 The Code of Practice on Disinformation Transparency Centre, “Reports Archive,” accessed October 23, 2023, https://disinfocode.eu/reports-archive/?years=2023.
4 Kirsty Park and Stephan Mündges, “CoP Monitor Baseline Reports: Assessment of VLOP and VLOSE Signatory Reports for the Strengthened Code of Practice on Disinformation,” EDMO Ireland and the German-Austrian Digital Media Observatory, September 7, 2023, https://edmohub.ie/index.php/major-platforms-disinformation-fight-under-scrutiny-in-new-cop-monitor-report/.
5 Alicia Wanless, “The More Things Change: Understanding Conflict in the Information Environment Through Information Ecology,” Doctoral Thesis, King’s College, London (April 1, 2023) https://kclpure.kcl.ac.uk/portal/en/studentTheses/the-more-things-change.
6 Kamya Yadav, “Platform Interventions: How Social Media Counters Influence Operations,” Carnegie Endowment for International Peace, January 25, 2021, https://carnegieendowment.org/2021/01/25/platform-interventions-how-social-media-counters-influence-operations-pub-83698.
7 Ewa Krukowska, “Twitter Withdraws From EU Disinformation Code, Commissioner Says,” Time, May 27, 2023, https://time.com/6283183/twitter-withdraws-from-eu-disinformation-code-commissioner-says/.
8 Code of Practice on Disinformation Transparency Centre, “Structural Indicators, accessed October 24, 2023, https://disinfocode.eu/structural-indicators/; TrustLab, “Code of Practice on Disinformation: A Comparative Analysis of the Prevalence and Sources of Disinformation across Major Social Media Platforms in Poland, Slovakia, and Spain, Semi-Annual Report,” September 2023, https://disinfocode.eu/wp-content/uploads/2023/09/code-of-practice-on-disinformation-september-22-2023.pdf.
9 Iva Nenadic, Elda Brogi and Konrad Bleyer-Simon, Structural indicators to assess effectiveness of the EU’s Code of Practice on Disinformation, European University Institute, RSC, Working Paper, 2023/34, Centre for Media Pluralism and Media Freedom, European Digital Media Observatory (EDMO), https://cadmus.eui.eu/handle/1814/75558.
10 TrustLab, “Code of Practice on Disinformation: A Comparative Analysis of the Prevalence and Sources of Disinformation across Major Social Media Platforms in Poland, Slovakia, and Spain, Semi-Annual Report,” September 2023, https://disinfocode.eu/wp-content/uploads/2023/09/code-of-practice-on-disinformation-september-22-2023.pdf.
11 Yasmin Green, Andrew Gully, Yoel Roth, Abhishek Roy, Joshua A. Tucker, and Alicia Wanless, “Evidence-Based Misinformation Interventions: Challenges and Opportunities for Measurement and Collaboration,” Carnegie Endowment for International Peace, January 9, 2023, https://carnegieendowment.org/2023/01/09/evidence-based-misinformation-interventions-challenges-and-opportunities-for-measurement-and-collaboration-pub-88661; Anna-Katharina Meßmer and Martin Degeling, “Auditing Recommender Systems: Putting the DSA into practice with a risk-scenario-based approach,” Stiftung Neue Verantwortung, February 7, 2023, https://www.stiftung-nv.de/de/publication/auditing-recommender-systems#risk-scenario-based-process.
12 Shane Huntley, “Threat Analysis Group Bulletin: Q3. 2022,” Google, October 26, 2022, https://blog.google/threat-analysis-group/tag-bulletin-q3-2022/.
13 European Parliament, “Article 9 GDPR. Processing of special categories of personal data,” accessed October 23, 2023, https://gdpr-text.com/read/article-9/#:~:text=Article%209%20 prescribes%20that%20a,by%20article%204%20(11); European Parliament, “Article 17 GDPR. Right to erasure (‘right to be forgotten’)”, accessed October 23, 2023, https://gdpr-info.eu/art-17-gdpr/.
14 Anastasia Kozyreva, Laura Smillie and Stephan Lewandowsky (2023), Incorporating psychological science into policy making: The case of misinformation. European Psychologist, 28(3), 206-224, https://doi.org/10.1027/1016-9040/a000493; Kamya Yadav, “Platform Interventions: How Social Media Counters Influence Operations,” Carnegie Endowment for International Peace, January 25, 2021, https://carnegieendowment.org/2021/01/25/platform-interventions-how-social-media-counters-influence-operations-pub-83698.
15 Kirsty Park and Stephan Mündges, “CoP Monitor Baseline Reports: Assessment of VLOP and VLOSE Signatory Reports for the Strengthened Code of Practice on Disinformation,” EDMO Ireland and the German-Austrian Digital Media Observatory, September 7, 2023, https://edmohub.ie/index.php/major-platforms-disinformation-fight-under-scrutiny-in-new-cop-monitor-report/.
16 Alicia Wanless and Jacob N. Shapiro, “A CERN Model for Studying the Information Environment,” Carnegie Endowment for International Peace, November 17, 2022, https://carnegieendowment.org/2022/11/16/cern-model-for-studying-information-environment-pub-88408.
17 John Albert, “A guide to the EU’s new rules for researcher access to platform data,” Algorithm Watch, December 7, 2022, https://algorithmwatch.org/en/dsa-data-access-explained/.
