Euphoria swept across legions of observers of Arab politics two years ago. A series of unusual scenes on the streets of the Middle East nurtured an inspiring story line of an emerging “Arab spring” that mimicked the earlier triumph of democracy from the Philippines to Prague: mass demonstrations in Lebanon; joint rallies of Egyptian Islamists and liberals against the Mubarak regime; and elections in Iraq, the Palestinian territories, Lebanon, Egypt and even Saudi Arabia. Many of the most deeply entrenched Arab regimes appeared to be on the verge of losing their authoritarian grip. Symbolized by purple fingers in Iraq, orange t-shirts in Lebanon and the single word kifaya (“enough”) in Egypt, the fall of the Arab equivalent of the Berlin Wall seemed at hand.

Today, little of that euphoria remains. New political realities have either silenced the optimists or caused them to rue the consequences of the changes they had earlier hailed. Iraqi elections were followed by a dramatic upsurge in sectarian violence that has become a bloody civil war. Political stagnation in Lebanon and Egypt has not led to any democratic transformation—Egyptian semi-authoritarianism and Lebanese confessionalism have revealed themselves to be deeply entrenched and occasionally ferocious political forces. Throughout the region, the strong showing of Islamists in parliamentary elections has created doubts about Arab democratization and highlighted the risks it bears for American strategic interests. The tragic developments in Iraq, Lebanon and Palestine—all with weak or failed state institutions—have enabled Arab democracy pessimists to dismiss easily any talk about positive political reforms as the fantasy of Western well-intentioned humanists—or misguided ideologues—who do not understand the real Arab world and its unsuitability for democracy.

The problem with the manic debate in Washington—irrational exuberance followed by despair—is that it misses gradual but real changes occurring in the region. There are many deep political problems in the Arab world. But that should not mask a variety of political openings in the region—many of which are only visible when one takes a longer-range view. Despite rising disenchantment outside the Arab world regarding Arab democratization, regional political dynamics have been driven to a great extent by an indigenous freedom agenda. In the level of intellectual debate, the battle for democracy has been fought—and won. Some have even fashioned rhetoric oddly similar to that of the Bush Administration. Asked in 2005 what Egypt needs most, Supreme Guide of the Egyptian Muslim Brotherhood Muhammad Mahdi Akif responded, “Freedom. Freedom.” And among the broader public, there is growing support for political openness. The problem is that democracies are not built in salons and on satellite channels.

Dreams of democratic openings, competitive elections, the rule of law and wider political freedoms have captured the imagination of clear majorities in the Arab world. The dominance of the idea of democracy in the public space has even forced authoritarian ruling establishments to cast about for new pro-reform language in order to communicate their policies to the populace. Even Islamist and leftist opposition movements have, at least rhetorically, dropped most of their skepticism about political rights, freedoms and pluralist mechanisms, developing a strategic commitment to gradual democratic reform.

The rise of democracy is not confined to rhetoric; limited but real changes are taking place. Some of these changes have occurred so slowly and unevenly that they are often missed. But over the past two decades in much of the Arab world, ruling establishments have substantially eased the restrictions imposed on freedom of expression. Media outlets, intellectual forums and academic institutions have become venues of pluralist argumentation. The era of state monopolies over information and ideas has ended. Ordinary Arab citizens have gained access to multiple sources of information and become systematically exposed to competing perspectives of domestic and international events. Of course, significant differences continue among countries: The press in Morocco and Egypt has grown highly diverse, but pluralism is more limited in Syria and some of the Gulf countries. Furthermore, repressive practices against opinion-makers and state ownership of media outlets have hardly ceased—in places like Tunisia, Egypt and Yemen, the state reins in dissent less predictably than in the past, but occasionally quite severely.

Is it analytically legitimate or sound to describe these changes as democratization? Are they merely cosmetic changes, or have they increased the probability of successful democratic reform in the Arab world? While the situation differs from country to country, there can be no doubt that ruling establishments’ views of reforms are extremely constrained. On occasion they even seem Orwellian—such as when the Egyptian regime discovers a new interest in “citizenship”, not to empower Egyptians, but to discredit its main opposition (the Muslim Brotherhood, which is accused of dividing Egyptians along religious lines).

But we must move beyond understanding all political outcomes solely in terms of the intentions of leaders. Real change has occurred, but will it continue? And if so, will it be change for the better?

Arab regimes can be divided into three basic political categories: weak or failing states, strict authoritarian states and semi-authoritarian states. Much of the current despair comes from the odd focus on the first two categories—and these are precisely the least likely places for democratizing change to occur. It is in the third category, the semi-authoritarian states, that we find the best chances for reform. In all three cases, there is at least the thread of possibility for reform, but we need to be realistic about what kind of change is feasible.

Much of the renewed cynicism about democracy is based on mistaking the region’s most problematic states for the norm—weak, failing or (in one case) incomplete: Lebanon, Iraq and Palestine. Indeed, democratic mechanisms have hardly solved their problems. In many ways they have made matters worse by bringing deep divisions to the forefront without offering the prospect of resolution. In all three cases, state weakness has encouraged international actors to favor specific parties, fomenting internecine conflict.

In Iraq, the mechanics of democratic politics actually hastened the country’s descent into civil war and sectarian bloodletting. The dismantling of the country’s fearsome authoritarian institutions, the bloodthirsty and ruthless insurgency, and the eagerness of the country’s Shi‘a population to use its majority status have left little semblance of civil, much less reformed, politics.

Palestinian democracy was far more robust, but the international community recoiled at an embryonic democratic transition because of the victorious Hamas movement’s extreme words and deeds regarding Israel. With Hamas’s victory, the era of international support for Palestinian political reform ended, and the United States embarked on the task of bringing down any government—by illegal means if necessary—that did not meet international conditions, including acceptance of past agreements and recognition of Israel. Of course, American concerns about Hamas were quite soundly based, but what was remarkable about American policy was how quickly democracy shifted in American eyes (if not in official rhetoric) from being part of the solution to being the core of the problem.

Lebanon seems stable only by comparison to Iraq and Palestine, but even the Lebanese state is torn between two roughly balanced political coalitions—each with strong democratic claims but far more limited democratic credentials. Each also has powerful international backers more interested in the triumph of their side than in mediation and political compromise.

Such states present the most difficult challenge for would-be democratizers. Fostering democracy while simultaneously strengthening state institutions is a difficult task in any setting. In the midst of unresolved international conflicts, it may be nearly impossible.

Strict Authoritarian systems—a minority in the Arab world—have been largely impervious to any liberalization. But there are some opportunities for limited change. In Libya and Syria, the regimes have not left any space for the formation of organized opposition groups, and real domestic pressures to ease this repression have remained largely absent. Yet even among such closed polities, variation is emerging. In Saudi Arabia, Oman and the United Arab Emirates (UAE), monarchies have allowed greater media pluralism and experimented with minor reform measures while keeping their traditional mix of patrimonial policies and repressive security apparatuses. Between 2002 and 2006, Saudi Arabia held its first (partial) municipal elections since the 1960s and allowed various small openings in civil society, including granting women the right to vote and to run as candidates for newly legalized professional associations and chambers of commerce. In Oman, Sultan Qaboos bin Said turned the appointed national consultative council, the Shura Council, into an elected body in 2003 and granted the right to vote to all Omani citizens over the age of 21, male and female. Although the elected Shura Council continues to lack any real oversight powers, the elections exposed the population to the imagery of pluralist politics. The government of the uae adopted a national reform plan and held the country’s first partial elections in 2006.

In such countries—where politics has been void of any competition between rulers and oppositions for a long time—it makes little sense to speculate about fundamental democratic reforms in the near future. But it is not unrealistic to imagine that a combination of internal demands and external pressure might nudge some into gradually allowing a greater degree of pluralism in the political sphere and granting citizens some basic political rights, such as freer association or expression—these forces already induced lower-end reforms. Past generations of Arab leaders have allowed a measure of liberalization when they became convinced that it was beneficial in currying international support, balancing or taming would-be rivals and managing conflicts within the political elite. An international environment supporting political reform can make a difference in such cases, especially in the Gulf countries, where the leadership has tied national security to Western powers. But we must be aware that such changes would not bring democracy but only a more liberalized authoritarianism, which has been quite stable regionally. And it is to such semi-authoritarian regimes that we now turn.

Weak states and closed authoritarian systems, while distressing examples of Arab politics gone wrong, are not the norm. In the third set of Arab countries, authoritarianism is well-entrenched but not unlimited. Existing semi-authoritarian regimes in Morocco, Algeria, Egypt, Jordan, Yemen, Kuwait and Bahrain have opposed any true democratization. Democratic institutions and practices often exist on paper but are very weak and easily outflanked by the institutions of authoritarianism.

Yet these semi-authoritarian regimes allow some space for popular participation, and in many countries that space has increased significantly over the past two decades. Opposition groups have been allowed to operate and participate in legal politics. And even if opposition groups are often divided between weak armchair secular movements and more broadly based but less clearly democratic Islamist movements, they have still been able to gradually expand their representation in the political process.

To give some examples: In Morocco, the government introduced a series of reforms at the national and municipal levels aimed at enhancing government accountability, popular participation and human-rights records. It allowed the participation of the Islamist Party for Justice and Development in the parliamentary elections of 2002, enabling the party to win 13 percent of the seats. And civil-society organizations have grown ideologically more diverse and more active in shaping public policies. In January 2004, acting upon the recommendation of the official Advisory Council on Human Rights, King Muhammad VI inaugurated, for the first time in the Arab world, a national reconciliation committee to investigate human-rights violations by state security agencies in past decades.

In Algeria, the regime of President Abdelaziz Bouteflika partially moved beyond the militant legacy of the 1990s and tolerated the integration of non-violent Islamist parties into the political process. One of them, the Party for a Society of Peace, has participated continuously in ruling coalitions as a junior partner since 1996. In 2005, Bouteflika announced his initiative for national reconciliation and committed the regime to reform measures that are aimed to increase political competition and institutionalize checks and balances between government branches.

The ruling National Democratic Party in Egypt in 2002 announced initiatives to open up the political space and improve human-rights standards. In spite of the reluctance of different elite factions, an advisory National Council on Human Rights was established in 2002 and mandated to supervise the dealings of state institutions—including the Ministry of Interior—with regard to human rights and report any violations to the government. In 2005, the constitution was amended to allow for the country’s first multi-candidate presidential elections. Parliamentary elections also took place later the same year. Despite serious violations of the norms for free and fair elections, the opposition Muslim Brotherhood seized the opportunity and lurched in a democratic direction. Its decision paid off, as the movement performed well and won a fifth of the seats. And although Islamist electoral gains have resulted in a new wave of regime repression and motivated the government in 2007 to introduce constitutional amendments that seek to limit the Muslim Brotherhood’s participation, Egyptian politics still reflects growing pluralism.

The Yemeni political space grew more diverse in the last few years leading up to the pluralist presidential elections of 2006. The government of President Ali Abdullah Saleh accepted full judicial supervision of the electoral process and let in international observers. Opposition parties, most significantly the Socialist and the Islamist Islah Parties, formed a broad, cross-ideological alliance—the Yemeni Joint Meeting Parties—and fielded a joint candidate against Saleh. In spite of its defeat in the elections, the Yemeni opposition alliance remains alive, and preparations for a joint opposition list in the 2007 parliamentary elections are under way. Laws governing the establishment of political parties and NGOs were liberalized in accordance with international norms.

In Kuwait, outbursts of conflict between the elected parliament and the ruling family—which had resulted in the suspension of parliament in the 1970s and 1980s—have been met in recent years by government attempts to buy off some parliamentarians and divide the remainder. Such tactics often work, but they do so imperfectly. In 2006, the opposition managed to paste together a coalition supporting electoral reform (combining the country’s tiny 25 electoral districts into five), not only forcing the ruling family to accept reform but probably undermining its ability in the future to co-opt deputies. The larger districts will introduce a more ideological flavor to Kuwaiti politics.

In all these cases, the reforms were significant but were not breakthroughs. They have been limited and difficult to build upon. Not a single dramatic democratic breakthrough has taken place.

But the dynamics unleashed by regime-managed reforms have renewed and revitalized the interest of opposition movements in the political process. The outcome is an Arab political scene that looks far more lively and much less predictable when compared to the political stasis and stagnation of the 1990s.

THE PROBLEM is not the absence of political change in the Arab world; the stagnation that prevailed for a generation has been shaken. It is whether fundamental political reform—democratization, rather than mere tactical uses of liberalization—is possible.

In such semi-authoritarian regimes, three paths to further political reform are possible. The most likely may simply be more of the same: continued incremental change within the boundaries of existing political arrangements. While semi-authoritarian regimes are often quite stable, they are characterized by weak legislative and judicial institutions as well as by red lines that limit the boundaries of tolerated political freedom and activity. While the term “red lines”—widely used in the region—implies a clear demarcation between permissible and impermissible politics, the lines are almost always vague, shifting and contested. The main residue of the Arab spring is not a new kind of democratic politics but a handful of slightly more pluralistic or ambitious parliaments in Egypt and Kuwait, and even blurrier red lines—especially regarding freedom of expression and organization—in other countries.

A second possibility for reform promises fuller democratization not through carefully managed change, but through surprise. This is a risky route, particularly because diversion to less positive outcomes is very easy. Democratic change in other parts of the world has often resulted from a shock to the political system—economic crisis, defeat in war, leadership succession. The problem, of course, is that such shocks do not necessarily bring democratic change: They are just as likely to bring about instability, deepened autocracy or even civil war. But when political opposition is well-organized and undividedly committed to democratic change, it is often successful in obtaining favorable outcomes. In short, such shocks can provoke democratic change—if opposition leaders prepare for it. In southern Europe, Latin America, South Africa, and Central and Eastern Europe, democracy has fared far better when opposition movements had the time and inclination to develop democratic ideologies and practices before they were faced with the prospect of gaining power.

The semi-authoritarian regimes in the Arab world have shown impressive abilities to weather economic and political crises in the past: They lost wars, presided over economic hardship and passed through succession crises. In none of these cases, however, did they face competent and deep-rooted democratic oppositions ready to seize the opportunities presented by regime shocks. At least in some Arab countries, this has been changing. Oppositions in Morocco, Egypt, Yemen, Kuwait and Bahrain have grown better organized and more popular. Just as critically, secular and Islamist movements in some countries seem to be preparing—and sometimes experimenting with cooperation—for a democratic transition if the currently deeply entrenched regimes falter.

By their very nature, shocks are unpredictable, but there is one kind that is inevitable even if the timing is unknown: the incapacitation or death of an autocrat. The highly centralized political systems characterizing most Arab regimes (especially the republican ones, where the president has no royal family to report to) means that a ruler’s demise can unleash a disorienting crisis, activating latent conflicts within the regime and forcing new would-be rulers to obtain popular support that the late ruler learned to dispense with. Today’s Egypt seems to be in the beginning stages of such a succession crisis. While the ruling party increasingly maneuvers to secure the succession of President Hosni Mubarak’s son, Gamal, the effort has revealed splits in the leadership, very lively and explicit public debate, and signs even that parts of the military leadership are balking or dubious about the proposed succession. Egypt’s past two successions (in 1970 and 1981) were followed by political liberalization, but on both occasions the opposition was cowed and disorganized. The political scene in 2007 may lead to a less easily managed process that opens pressures for deeper democratization. Such a route is hardly inevitable; indeed, the existing regime seems to be working to preclude it by a brutal crackdown on its main opposition, the Muslim Brotherhood. But regime missteps at such uncertain times are not only more likely than normal but also more extensive in their repercussions. Future generations of Egyptians may pay dearly if the Egyptian government succeeds in its attempts to crush democratic dissent and block the Brotherhood’s move in a democratic direction.

A final avenue for political reform is negotiated power-sharing between a pressured government and a rising but frustrated opposition that realize they cannot vanquish each other. Up to this point, no Arab society has come close to such a standoff. (Algeria showed some potential in 1992 but moved to a very different and far more violent resolution.) Realistically, it is only mainstream Islamist movements who are likely to pose a strong challenge, and regimes have generally dealt with them as security issues (to be quashed or contained) rather than political rivals (who might be bargained with). But because such movements are broadly based—hardly limited to the political arena and encompassing religious, educational, social, cultural and charitable activities—they are not easily suppressed. Some regimes have experimented with easing the security obsession, opting to manage Islamist movements as serious political actors—and therefore as candidates not simply for arrests and detention but as partners and rivals for inclusion, domestication and co-optation. Jordan experimented with this path in the past but now seems to be turning away from it; Morocco has been experimenting with it since the second half of the 1990s; Kuwait has also brought Islamists into parliament and even the cabinet; the Sunni ruling establishment of Bahrain has come to tolerate significant participation by Shi‘a Islamists in parliament and municipal councils. Such negotiations become easier when the Islamist movement is able to present itself as reformist, rather than revolutionary, and to disperse doubts about its loyalty to the nation-state framework. Several of the leading Islamist movements in the region have therefore worked over the past few years to cultivate precisely that image. What is unclear is whether such an approach can evolve—on both sides—from a short-term tactic to a long-term strategy with real power-sharing.

What should be noted about all three of these scenarios is that none necessarily brings democratic nirvana overnight. The second and the third paths depend very much on an incorporation of Islamist movements as normal political actors. There simply is no way to democratize by crushing the region’s most popular groups—in almost all of these countries, Islamists. If Islamists are to be confronted as security challenges and suppressed, chances for democratization are bound to remain minimal. And there will be little liberalization as well; the repressive tools built by some states to use against Islamists can be turned against opposition forces of all stripes.

WE NEED to avoid choosing between unrealistic idealism and brutal cynicism. The wider regional scene does not look as bleak as the democratization pessimists in the United States tend to depict it. In contrast to the pessimists, we offer a sober but more hopeful view: Change has been occurring and further reform is possible. But it is neither inevitable nor bound to be purely democratic in nature. Moreover, while the American push for Arab democracy between 2003 and 2005 was not misguided in its essence, it was pursued in such a clumsy and manic manner that the current disillusion was inevitable.

From the beginning, the United States focused on the most difficult cases: failed or weak states, and unbridled autocracies. And while American leaders have spoken of a long-term struggle, they showed stunning impatience in practice. The United States quickly recoiled when initial efforts led down a worrying path. Lebanese demonstrations against Syrian hegemony served both our interests and our values, but when the Jordanian government cracked down on professional associations, the Bahraini government lashed out at demonstrators calling for constitutional reform, and the Egyptian government used some of its harshest tools (long detentions and military courts) to suppress the Muslim Brotherhood, short-term security interests forced American leaders into a shameless silence.

Finally, while the American thrust was accompanied by inspiring, almost soaring, rhetoric, it was never married to an effective set of policy tools. Indeed, it might even be said that there was no real policy—only a mentality and a rhetorical commitment that supported democracy and freedom in very general ways. This was coupled with a fairly familiar set of training, technical assistance and capacity-building programs that had been borrowed from very different experiences in Eastern Europe and Latin America, where the United States assisted a pre-existing democratization process bolstered by the strong support of significant local actors and generally good bilateral relations.

A course correction could help the United States promote positive change consistent with its own interests. First, the United States should concentrate on the most likely candidates for reform, the semi-authoritarian states. Democracy promotion should not be a tool solely to depose despots who do not cooperate with the United States or to conjure capable states into being. We will need to adopt a long-term view. There will be no sudden collapse of an “Arab Berlin Wall”, and we need to disabuse ourselves of the idea that all societies are democracies waiting to emerge. It would help if U.S. policymakers would be franker—and more sophisticated—about short-term tensions between security and democratization. When the effort has adopted an attitude of realistic, rather than messianic, democracy promotion—such as in Egypt in 2004 and 2005—it has produced tensions but also results, and the bilateral relationship remained largely intact.

Ultimately, we have to be able to devise an appropriate set of tools, especially in the diplomatic realm. The standard democracy-promotion tool kit—involving activities like parliamentary strengthening, support for the judiciary, civil-society training and electoral assistance—does little harm and in some Arab states actually has some accomplishments to claim. But it is unlikely to lead to systemic change.

This mix of approaches and policies will not transform the Arab world overnight. But it will bring a greater dose of realism, promote sustainable efforts rather than sudden lurches, lessen the tension between American interests and values, and put the United States more firmly on the side of positive political change in the region.

Nathan J. Brown is a professor of political science and international affairs at The George Washington University and a senior associate at the Carnegie Endowment for International Peace. Amr Hamzawy is a senior associate at the Carnegie Endowment for International Peace.