The prime ministership is not terminal – but it needs the right person for the times

Malcolm Turnbull was interviewed late last month on the ABC’s flagship news current affairs program, 7.30. It wasn’t pretty viewing. Turnbull responded to host Leigh Sales’ interrogation through gritted teeth. He tetchily accused her of being “negative”: of only wanting “to talk about politics”. The performance was the antithesis of the Turnbull of old — he of the leather jacket, who revelled in appearing on the national broadcaster, exuding charm and confidently expansive. In his place was a brittle and defensive prime minister.

We cannot know for sure what lies ahead for Turnbull — his boosters still wait expectantly for the green shoots of political recovery. Yet the monotonously negative opinion polls invite the suspicion that the public has given up on the Turnbull government. The Coalition’s divisions over key issues (now the clean energy target) and serial misadventures (such as the dual citizenship imbroglio) do little to instil confidence in Turnbull’s future. Like Tony Abbott, Julia Gillard and Kevin Rudd before him, he has become another in a line of beleaguered prime ministers.

As we have shown in our two-volume history of the office, being prime minister has never been easy. For many of its occupants, the office’s frustrations have been at least equal to its opportunities. Even those who have prospered in the role — since the second world war, think Robert Menzies, Bob Hawke and John Howard — endured fluctuations in personal popularity, electoral performance and ability to get things done.

Yet the disturbing trend of the past decade is hard to ignore. After Howard, the credit lines necessary to exercise effective leadership — keeping the nation’s ear even when delivering unpopular messages, articulating a coherent policy program, having reforms passed and securely embedded, and leading from the front (staring down one’s own colleagues and constituencies when necessary) — have eluded successive incumbents. They have struggled to resolve major policy problems and have suffered serious erosion of their personal popularity and political authority rapidly after achieving office.

Individual fallibility has played a part in this troubling story. If one mapped Australia’s 29 prime ministers along a spectrum of temperamental aptitude for the office, then Rudd and Abbott belong well towards the unsuitable end. Both lacked that essential quality German sociologist Max Weber called “the firm taming of soul”. All four of our most recent prime ministers have also made grave errors of judgment, but so have many of their predecessors.

Nor can lack of institutional resources provide a plausible answer for the contemporary malaise. To the contrary, since the second world war there has been a steady accretion of resources, both bureaucratic and personal staffing, at the heart of government. Prime ministers of the 21st century command a formidable institutional machine. They are supported by a responsive and professional Department of Prime Minister and Cabinet, and a large and powerful Prime Minister’s Office staffed by political loyalists.

Recent history suggests that, poorly directed, this “prime-ministerial machine” can go seriously awry, thereby exacerbating difficulties. But this factor is again insufficient to account for why national leadership has become so confounding.

A better explanation lies in the destabilising contextual changes of our times. First, post-2007 leaders are, like Whitlam and Fraser in the 1970s and ’80s, struggling with policy regime exhaustion: in their case, the expiring of the neoliberal experiment.

This condition is common to most advanced economies. The global financial crisis starting in 2008 was heralded at the time as a turning point. Much was said then and since about the necessity of a new social contract; regulation that would manage market failure; measures that would address inequality in resource distribution; and the need to stem the pervasive and increasing conviction among many that the economy was not working for them.

Yet, a decade on, we are still grasping uncertainly for those new ideas and a refreshed policy regime.

A second factor is a transforming party landscape. The central role that the established parties have played in our national political life can hardly be overstated: ours has been a “party democracy”.

Anchored in distinct social bases that have now broken up, the major parties have been the primary instruments through which voters organise their thinking about politics and express their choices. They’ve been the system’s ballast.

There have been significant periods in the past when one or other of the major parties was effectively moribund. But what we confront now is more fundamental, as major party membership dwindles and the public’s affiliation to them dissolves. The result is greater voter volatility.

For leaders, party change is having a paradoxical effect. As the parties have become less representative of society and as their philosophical moorings have weakened, they have leaned more heavily on leaders as the point of brand differentiation and to be spokesperson for all that they stand for. While potentially strengthening the authority and autonomy of leaders, this has equally meant they become chief target of blame and retribution if party fortunes suffer.

Moreover, the decline of the traditional parties seems to be manifesting in a pattern of intensifying fragmentation and factionalisation. Splinter parties on the left and right have emerged, with a growing divergence in the attitudes of the residual party membership and majority public opinion. In these circumstances, “broad church” arguments, such as those championed by Howard, have become difficult to sustain.

The plight of Turnbull since 2015 has illustrated this dilemma. To hew closely to party views — and especially to the vocal advocacy of ideological purists — risks extinguishing his public popularity. But to do what the public wanted has been to court internal revolt and possible loss of leadership.

Third, the ability of contemporary prime ministers to persuade has been eroded by disruptions to the mass media “broadcast” model of public communication that served their predecessors so well. That model conferred on prime ministers the ability to build relations with media proprietors and the press gallery, to use radio and television to speak directly to a broad audience, to lay out arguments, and to calculate the timing of releases.

It depended on opinion-leading broadsheets with a trickle-down influence on tabloids and radio and television news, and predictable news cycles. Leaders knew this game and structured their communication to achieve a match between their own skills and the options then available.

In the 21st century, however, the business models that sustained such practices were susceptible to changing perceptions of what would trigger audience “choice” – “infotainment” and celebrity undermining serious journalism. Above all, the new technology of dissemination, the internet, undermined the monopolies on which the model had depended.

The anarchic, real-time, “post-fact” logic of social media eroded the press gallery’s quasi-monopoly on meaning-making about politics for the public. New social media platforms became significant in opinion formation. In effect, a new form of “narrowcasting” eviscerated the “mass” media. The result is that media traffic has never been so intense, but public discourse has become fractured and fractious.

Has the prime minister’s role become impossible? With neither the parties nor the media serving as an effective means for explaining and justifying policy responses and promoting opinion aggregation, there is no question that the incumbency advantage in national debate enjoyed by prime ministers from the middle of last century has dissipated. Governing has become far more complicated, not just in Australia but globally.

The incumbents of the past decade have tried to cope with these challenges in different ways. For instance, in the area of public communication, Rudd became Australia’s first 24/7 leader, only for the media “logic” of his government to overwhelm its political and policy logic. The confections of Rudd’s obsessive media performing undermined his sense of authenticity.

Adjusting to the new realities of leading will demand further improvisation and adaption. At the same time, what we have learned from studying the history of the prime ministership gives grounds for optimism that there are ways out of the current fix.

First, policy cycles come and go. If the limitations of the market-liberalising ideas of the 1980s and ’90s have become apparent to many people, provoking disillusion, we have been there before. The innovative Deakinite “Australian settlement” was in large part a reaction to the Australian experience of the 1890s depression: it took the first Commonwealth decade and more to achieve.

The calamitous Great Depression period occasioned frustration for prime ministers James Scullin, Joseph Lyons and Menzies (mark 1) before the catalyst of war and post-war reconstruction fostered the Keynesian breakthrough, led by John Curtin, Ben Chifley and Menzies (mark 2). This met the problems of that time but, inevitably, circumstances changed again and the assumptions of the 1940s no longer served in the 1970s and ’80s.

Treasury began to question those assumptions and to revisit classical economics as early as 1971. But it was not until 1983 that Hawke, Paul Keating and later Howard drove the reforms that have since been evocatively dubbed by George Megalogenis as “the Australian moment”. But those changes have produced their own problems. The lags and transitions as policy regimes wane and reinventions occur are never short term — they may take ten or 15 years — but history suggests that they are usually achieved.

Second, let us talk of leadership. At times of crisis or deep disillusion, there are often calls for strong leadership. Our study has shown, however, that the complex challenges of social change have been best addressed by prime ministers who have fostered talented ensembles, capable of applying diverse skills to the task of government. The standout instances have been Alfred Deakin (with a rare capacity for attracting disciples and crafting alliances), Curtin and Chifley (united in co-operative endeavour by robust common sense allied with personal humility), and Hawke (who improbably matched overweening egotism with an inherent gift for orchestrating distributed leadership).

In each case, they operated at the turning point of a political cycle. Deakin inaugurated “the Australian settlement”. Curtin and Chifley initiated what Stuart Macintyre has described as “Australia’s boldest experiment”. Hawke (with Keating) started the reform cycle that assured prosperity and resilience as globalisation unsettled Australian expectations.

It perhaps seems a vain hope to trust in the wisdom or capacity of the right individuals to emerge again, but in the past difficult circumstances have conspired to produce them.