The U.S. president, frustrated by the costs of involvement in the Middle East that had been a huge burden on America for years, ordered his intelligence services into action. The course they chose was regime change. Operatives took the initiative, secretly raising an army to depose the offending ruler.

The president was Thomas Jefferson. The date was 1805. Since its early days, the United States, like European nations, had been forced by the Barbary pirates into paying tribute to avoid the capture and harassment of sailing vessels. Jefferson had previously hoped to raise an international coalition to depose Tripoli’s pasha, but due to European hesitation, he acted unilaterally. The result was what the CIA now describes on its website as “the United States’ first covert attempt to overthrow a foreign government.”

David Rothkopf
David Rothkopf was a visiting scholar at the Carnegie Endowment as well as the former CEO and editor in chief of the FP Group.
More >

Initially, the U.S. effort was a seeming success; as soon as it was clear that the Americans were posed to remove the pasha and replace him with his brother, a treaty was struck. American hostilities in the region, however, were not fully resolved for another decade.

Much in this story is chilling in its familiarity. The United States doesn’t seem to have come too far in the intervening two centuries. The country is again embroiled in interminable hostilities in the Middle East. And once more at the center of the action are U.S. intelligence services.

During the nation’s first years of existence, President George Washington sought funding for secret services that he felt were essential to U.S. security. The allocations for those spying activities consumed roughly 10 percent of the U.S. budget—a grand sum measuring roughly $1 million. In 2014, the reported budget for U.S. intelligence activities was approximately $68 billion, down from its 2010 high of $80 billion.

Despite its deep roots and the resources that have been poured into it, today’s intelligence community stands at a watershed. It has been battered by criticism for its role in torture programs and other abuses, by a long string of costly intelligence failures, and by shock at the overreaching surveillance activities of the National Security Agency. It has gone through several major overhauls, including the post-9/11 creation of the Office of the Director of National Intelligence (ODNI) and a more recent reshuffling at the CIA overseen by Director John Brennan. The formation of the ODNI, intended to enhance coordination among spy agencies, actually exacerbated tensions. Rivalries among special operators in the military, military intelligence, and the CIA over whose teams should take the lead in combating terrorism threats worldwide have compounded internal strains. The failure of top policymakers to effectively utilize intelligence or understand its inherent limitations has only worsened these problems.

All this turmoil has come at a time when the profound transformational consequences of the advent of the information age have raised serious questions about the future of intelligence. Those questions extend to the role of the intelligence community in the national security apparatus, the means of achieving intelligence goals, and the appropriate limitations that ought to be placed on intelligence activities in a free society. Changes have been coming so rapidly that reflection about the best way to address such issues has been forced to take a back seat to the operational concerns that have bedeviled the intelligence community during the past decade and a half.

Yet there is an ever more urgent need for a rethink of how, why, when, where, and by what means intelligence is gathered, analyzed, and utilized. It is still the early days of the information revolution. The pace of breakthroughs will only accelerate, and the consequences will shift from incremental improvements in productivity and connectivity to fundamental changes in the nature of society, power, war, and peace.

By 2020, it is estimated, 50 billion devices will be connected to the Internet—most of them embedded microprocessors that will offer real-time insights into every aspect of life on the planet. Furthermore, effectively every human being, every organization, and every government on Earth will be connected in a man-made system for the first time in history. Each of those billions of microprocessors and each connection on the web will be a potential entry point for surveillance and spying. What’s more, thanks to drones and nanodevices that can be hidden and embedded on targets by the millions, humans stand at the dawn of an era of potentially ubiquitous sensing. (This is not to speak of the gradual impact artificial intelligence will have on how people direct, conduct, and analyze what is gathered.)

If the world does not set limits, preferably by international treaty, as to what is fair game in this system—in terms of both surveillance and cyberconflict—humanity runs the risk of entering a period that will make Big Brother dystopian fantasies pale by comparison. Central to this process of setting limits will be having a public debate about the philosophical building blocks of the system: what is privacy, who owns the data each sensor produces, how should people divvy up the rights of individuals, corporations, and states. Furthermore, intelligence agencies will have to be reorganized to deal with these new realities, and so too will entire national security systems. Increasingly, the Internet will be the terrain on which most future battles will be fought, won, or lost; information warriors, many of them from the IC, will be the principal combatants.

It will be essential that the world reconsider views on the classification of information. Vastly more information is publicly available than could likely ever be gathered covertly. Such open-source information is easier to verify, easier to share, of greater use to policymakers, and essential to the kind of public-private collaboration that will be required in the new security environment. Conversely, estimates from career intelligence consumers suggest the vast majority of what is available via classified channels is also available or discoverable via open sources. Not classifying it would save billions of dollars.

The most modern information systems are designed to serve consumers of information. The intelligence apparatus is perversely oriented to the needs and concerns of information producers. It is centralized when most systems are distributed, and it is hard to search when most focus on making that easy. This makes sense in some (though very few) cases in order to protect sources and methods. But overall, the apparatus should be reformed to enhance officials’ ability to make informed decisions using blended open- and closed-source data.

Moving analysts closer to the consumers also makes sense. The United States should be restoring its withering mechanisms of foresight, such as the National Intelligence Council, by relieving them of the burden of being glorified memo writers for the National Security Council. The most important and elusive objective of good intelligence systems is not providing the right answers—it is coming up with the right questions. That requires special processes, independence, and a creative freedom that today’s news-fed, politically driven system is becoming worse and worse at executing.

Finally, it is time to acknowledge that the reorganization—namely, the formation of the ODNI—that took place in the wake of the 9/11 terrorist attacks was a mess. It was in fact just the latest in a series of unsatisfactory efforts that began with the creation of the Office of Strategic Services during World War II and then included the formation of the CIA. All were meant to knit together the proliferating number of spy agencies in the United States (now up to 17). But in each case, the result was not coordination, but the creation of new bureaucracies and weakened communication.

The intelligence community needs fewer agencies, fewer bosses, less redundancy, lower costs, and more orientation toward focusing on the right missions in the right way. All the forces of the information age—the explosion of new sources and means of gathering intelligence—will create new temptations in the opposite direction. But if America gives in to them, it is likely to get results that will amplify the failures of recent decades, while failing to build on the tradition of serving a vital role that has marked America’s intelligence services since the infancy of the country.

This article originally appeared in Foreign Policy.