In the lead-up to the 2016 U.S. presidential election, disinformation on social media platforms threatened the integrity of the electoral process. Social media platforms have assured the public that maintaining election integrity is one of their top priorities ahead of the 2020 U.S. presidential election. However, synthetic media (for example, deepfakes) presents new and critical challenges for platforms in maintaining this integrity. On November 16, 2018, the Carnegie Endowment for International Peace (CEIP) assembled a group of experts and platform representatives who have insights into addressing the problems that synthetic media presents for platforms specifically and society more broadly. The end goal was a set of potential safeguards and a road map for implementing those safeguards.

Two presentations set the stage for discussion by surveying, respectively, the state of synthetic media and potential harms to society.

The State of Synthetic Media

Historically, people have considered cameras to be truth-capturing devices even while the perspective and context in images and videos could be manipulated by photo- and videographers to shape perceptions of what actually happened (for example, to tell stories). Until recently, the ability to significantly manipulate the actual subjects of an image or video without the average person detecting such changes was the purview of visual effects experts. And the visual effects community still outperforms everyone else in this regard. However, the capacity to create hyperrealistic fake subjects (deepfakes) has expanded thanks to AI-enabled technical improvements. As a result, non-experts can create fakes that an average person deems real. In this way, deepfakes have passed through the “uncanny valley.”

But even though an average person may deem a deepfake authentic, media-forensics experts can still identify them as fake—at least for the time being; there is an arms race between automated techniques that create synthetic media and automated forensics that detect them. Meanwhile, various actors are emerging in the synthetic media market, with some providing software and others providing a complete service. One thing is for sure: there is no silver bullet to the synthetic media problem. The best solutions may be multidisciplinary, engaging policymakers, psychologists, behavioral scientists, and other social scientists alongside technologists.

A Topology of Harms

Unlike earlier forms of manipulated media, synthetic media is highly credible and more readily available. Potential harms caused by synthetic media may be organized into three categories: individual harms, organizational harms, and strategic harms. Potential individual harms include emotional abuse, sabotage, extortion, and physical harm. These can occur in public or private settings and may be motivated by revenge, money, or simple amusement. As evidenced by the massive problem of sextortion, individual harms can be severe and life-altering.

Organizational harms include organizational sabotage. For instance, a leader of an organization may be shown doing something that negatively impacts their organization. Strategic harms include a range of national- and international-level actions by and against nonstate and nation-state actors. They include covert action, incitement of violence (including projecting inflammatory language), diplomatic disruption (especially by nonstate actors), and electoral impacts (including individual-level political sabotage). Different harms require different levels of virality to be effective. For example, political sabotage must be very viral to reach a broad audience; covert action and diplomatic sabotage may not.

Complicating remediation are the “liar’s dividend” and evidence authentication. The liar’s dividend refers to the ability of someone depicted in an authentic photo, video, recording, or other medium to sow doubt about the authenticity of the depiction by “crying deepfake.” Defendants have challenged the authenticity of evidence in courts for a long time, and courts have experience with evidence authentication. But a concern with synthetic media is that jurors will be less likely to believe video evidence, regardless of standards for admissibility. Experts in this area are rare and expensive.


Threat actors can use synthetic media to expand or alter existing threats and introduce new threats, which are not discontinuous from threats that exist at the broader level of disinformation (which are far more complex in many ways). Within the realm of disinformation tactics, synthetic media may encompass a tangible and approachable threat set.

A significant threat platforms face is scaling decisionmaking processes for verifying information that keep pace with the velocity of their platforms. This challenge is exacerbated in the synthetic media context by difficulties involved in verifying whether an image or video is authentic or manipulated, benign or malicious. Journalists face similar challenges. As arbiters and amplifiers of truth, they must verify information amid intense pressure to be the first to disseminate it.

Platforms also have a role in developing ethical guidelines for applications from third-party developers and assessing what types of content are permissible. These ethical guidelines and assessment frameworks will need to evolve as synthetic media proliferates. Perhaps there are applications, software development kits, and libraries that should not be allowed in the marketplace.

Functional differences between search indexing and content moderation can lead to value judgments that may result in a deepfake being removed from a platform but still available to those intent on finding it somewhere on the internet. Some platforms have tools in place, such as national security toolsets and disinformation strategies, but mitigating threats to user safety and off-platform amplification remains hard.

Threats from synthetic media may be organized into three categories:

Easiest Threats (because companies already have relatively advanced tools to address them)

  • Insider attacks (for example, an insider posts a deepfake in a way that subverts rules/controls) 
  • External intrusions (for example, phishing attacks that use deepfakes to gain trust)

Hardest and Most Important Challenges

  • Tension exists between the velocity of content movement and the time needed to verify its veracity and take action.
  • Actions taken by platforms are not guaranteed to be effective because of off-platform amplification channels.
  • Identifying a user’s intent when posting synthetic media is difficult, as is identifying context that might indicate malicious synthetic.
  • Privacy and copyright concerns conflict with anticipated benefits of cross-platform information sharing.
  • Societal standards and definitions that distinguish between benign and malicious synthetic content are lacking, further complicated by value differences across countries.

Possible Areas for Collaboration Among Stakeholders

  • Norms, standards, or definitions of what constitutes malicious synthetic media and an appropriate response.
  • A collaborative mechanism for sharing synthetic media information between platforms, including provenance of synthetic media (for example, whether certain manipulators have a style).
  • Public and private sector coordination that counters actors using misinformation, for instance between platforms and newsrooms, that supports lower-level analysis of media uploaded to platforms.


Standards, definitions, and principles: Setting standards for what is benign, acceptable, and trustworthy versus what is malicious, false, and deceptive can be a first order of business in areas where technology and security issues intersect (like counterterrorism and cybersecurity). While norms or standards for synthetic media may be valuable, there is concern about the difficulty in establishing standards and low expectations that success can be achieved quickly.

Collaborating on shared definitions and broader principles may be more fruitful, particularly if such definitions and principles are grounded in existing U.S. common law (for example, torts like defamation, intentionally inflicted emotional distress, and privacy invasion) that have 150 years of precedent.

Detection of manipulated content and detection of malicious intent: Detection of manipulated content and detection of malicious intent are distinct problems with different technical characteristics and likely responses.

Viable technical approaches either exist or are being developed to detect synthesized content, yet further improvements are needed. Advances in media forensics have been helpful, and some platforms have identified behaviors common to actors spreading disinformation. Another technical improvement within reach is to embed a synthetic-media indicator into the generalized adversarial networks (GANs) that create synthesized content. Such indicators do not yet exist but services developing GANs may be willing to create them. Detecting malicious intent, while important, presents a harder problem with a number of exceptions and edge cases.

Information sharing: Both technical and legal architectures for sharing information about deepfakes between platforms and between the platforms and the public sector are needed.

Technical architectures for information sharing must work at scale, which is hard to do because of underlying data structures. In theory, elements that could be shared are images and videos (and their hashes), assessments of synthetic manipulations, assessments of intent, provenance, attribution, and context. Privacy laws protect sharing of some user data, including some metadata, that might help to identify a malicious deepfake. A Deepfakes Information Sharing Act akin to the Cybersecurity Information Sharing Act (CISA) of 2015, which increased information horizontally (between companies) and vertically (increasing liability protections and the declassification of information so more is shared between the public and private sectors), may be helpful. Like CISA, information related to deepfakes could be categorized into threat actors, tactics, and technical information and could be shared, without heavy government involvement (similar to the way banks share information about fraud). Even while information could be shared across platforms, they could still make independent decisions.

Responses: Options for platforms to respond to deepfakes include removing them entirely, demoting them in search results, blocking applications that disseminate them, and marking synthetic content as such in the software that creates them, among others. Additionally, there are a suite of responses that involve communicating directly with users and journalists via in-line context that a) indicates that a piece of content is synthesized; b) provides educational information on what synthesized content is; and c) shows authentic information alongside the synthesized content, which some are already doing. Further discussion on the pros and cons of each response is needed along with a way for platforms to measure their progress. Future discussions could account for the range of legal and cultural situations around the world, and could include diverse functional perspectives from, for example, the victims and creators of synthetic media.

In sum, there was not enormous appetite among workshop attendees to collaborate on formal standards, in part due to differences in understanding of and expectations about what standards would look like. By contrast, there was interest among some of the participants in establishing definitions and principles and information sharing horizontally (between platforms) and vertically (with government and civil society). Privacy laws may prevent sharing certain information, but it is worth exploring the extent to which sharing raw media, metadata, or assessments related to provenance, attribution, and intent is technically feasible and legally permissible. Cross-sector collaboration is critical to success. But notwithstanding potential collaboration around definitions/principles and information sharing, platforms will make individual determinations independently based on their unique legal, commercial, and ethical environments. Meanwhile, there are technical responses that could be deployed now or in the near future that, while not complete solutions, can help to mitigate threats from synthetic media.