Explainer: how indicators have the power to shape our world

With the turn to evidence-based governance, the reliance on statistical data along with its synthesis into the kinds of scales, ranks, and composite indexes we refer to as indicators has become essential for policy formation and political decision-making.

The use of indicators in governance has expanded from economic and sector-specific quantitative data to the measurement of almost every phenomenon. Our recent book focuses on indicators of governance itself, specifically governance through law. These are indicators purporting to measure practices or perceptions of good governance, the rule of law, corruption, regulatory quality, and related matters.

The underlying theoretical framework of this book is the linkage between knowledge and power. Indicators are both a form of knowledge and a technology for governance.

Like other forms of knowledge, indicators influence governance. They do this when they form the basis for political decision-making, public awareness, and the terms in which problems are conceptualised and solutions imagined.

Conversely, the kinds of information embodied in indicators, the forms in which they are produced and disseminated, and how they function as knowledge, are all influenced by governance practices.

Indicators helps us make sense of the messy social world, manage and govern it.Cambridge

The production of indicators is itself a political process. It is shaped by the power to categorise, count, analyse and promote a system of knowledge which has effects beyond the producers.

In these respects, indicators are comparable to law. Law as a technology of governance can have very substantial effects on knowledge. For instance, the legal processes and legal forms of trials, investigations, inquests, legislative hearings, statutes and treaties can all be important sources of information that shape wider understandings of the world.

Like law, indicators order the buzzing array of actual behaviour into categories that can be understood universally. They not only make sense of the messy social world but also help to manage and govern it.

Use of indicators over time

We distinguish four phases of the development and use of indicators over time.

The first phase is the conceptualisation of the indicator. The indicator is named and its underlying theory of social change established. This requires a theoretical position, the development of categories for measurement, and modes of analysing the data.

When an indicator is formulated and labelled as measuring, for example, the rule of law or corruption, it builds on a theory of what constitutes a good society. It builds also on what constitutes a problem or pathology to overcome in the course of improving the national society and polity.

Influences on the conceptualisation of indicators can be analysed under four headings. First, the actors and institutions who develop an indicator determine which concepts are being deployed, for what purposes, and with which theoretical orientation. It matters who the actors and institutions are that create an indicator.

A second feature of indicator conceptualisation is the relevant expertise. For example, Freedom House began with the work of an academic with political science expertise. Many of the development indicators were developed by economists.

A third feature is the temporality of the process. Many of the successful indicators have been many years in the making. They have also acquired visibility based on their past reputations as well as their ability to adapt as situations changed. For example, Freedom House was born in the era of anti-Nazi activism of the 1930s, shifted to a Cold War orientation, and emphasised civil and political rights.

A fourth feature is resources. Conceptualising indicators and linking them to data, even if there is no additional data collection process, is an expensive and time consuming process. It requires significant input from governments, international organisations, wealthy advocacy groups, businesses hoping to turn a profit, or academics using existing data sets.

Which indicators are formed and what they measure depends in large part on where the financial support for the initiative comes from.

The Freedom in the World Index uses the standard technique of interactive maps with countries coded in different colours depending on their performance.Freedom House

The dependence on data

The second phase of an indicator’s trajectory is production, in which the conceptualisation of the indicator is married to available or created data. During this phase, those who know about available data search for appropriate or useful databases that either measure the desired phenomenon or serve as a proxy.

Indicators depend on data. Where the supply of data depends on widely dispersed individuals or entities to collect and supply it, indicator production depends on a degree of buy-in from these participants. Many of the participants may have incentives to under- or over-report or to reconstruct categories.

Resources matter, since data collection is very expensive. Some forms are less expensive, such as expert opinion surveys, while indicators that can use administrative data already collected by state agencies such as the police or the courts are clearly less expensive.

The interaction between conceptualisation and data collection is critically important, since each shapes the other. The order matters. If the indicator must rely on existing data, its definition is limited to what has already been measured or what can be interpreted from it. To move into new territory, it may be necessary to develop new data, which requires funding to collect it and even more funding to collect it over time.

A key element of production of an indicator is its presentation, packaging and dissemination. If an indicator is intended for a wide audience, there is likely to be a premium on formats that are clear and easy to use, conceptually simple and visually attractive. One such standard technique is the use of interactive maps with countries coded in different colours depending on their performance.

A basis for decision and action

The third phase is the one in which the indicator is treated as a source of knowledge. This means that it is used to form background beliefs or understandings, develop or test scientific hypotheses. It can also be used to form conclusions that provide a basis for decision and action.

The influence of a focal point on behaviour is heightened if use of the indicator becomes sufficiently embedded that they result in network effects.

The system of ratings sovereign bonds is so embedded in legal rules and market practices, including requirements that some entities invest only in bonds rated as investment grade, that network effects result. This makes major changes to the system more costly and less likely.

Effects and impact

The fourth phase is about assessment. Assessing the causal effects of indicators is almost impossible to do with any precision, as their influence can seldom be isolated from a myriad of other confounding causes or distortionary factors. The effects of indicators within individual organisations are the easiest to track, especially when those organisations are committed to transparency in decision-making.

Macro knowledge effects are more difficult to isolate, but there have been some notable efforts. For example, case studies show that there is significant indicator-influenced acceptance, fostered by the news media, that the concept of “development” is well articulated by the United Nations Development Programme’s Human Development Index.

Indicators vary in their influence. Some become well-established and remain influential while others receive little uptake or decline in importance.

World Bank President Jim Yong Kim. Indicators that prevail are often sponsored by powerful organisations such as the World Bank.Reuters/Mariana Bazo

Powerful backers make a big difference

Indicators exist in a complicated ecology characterised by relations of symbiosis, co-existence or competition and in environments that may change rapidly. Different indicators that express divergent theories compete in the same social field.

Those that prevail are often sponsored by powerful organisations such as the World Bank. However, some acquire visibility through their simplicity and the appeal of their underlying theory, such as Freedom House and Transparency International, an effect proponents of other ideas may seek to emulate through new indicators.

Indicators smuggle theories of corruption, rule of law, and development into apparently neutral systems of measurement. Some achieve such hegemonic worldwide acceptance that they shape legal consciousness while others are ignored.

Indicators and theories work together. An indicator is more acceptable if its implicit theory is accepted, while a theory may receive wider support if it is expressed in a clear and widely used indicator.

Indicators that become dominant persuade decision makers to follow their models. They also affect governance when they specify a standard such that decision making becomes an assessment of performance with relation to the metrics of that standard.

This article is an excerpt from The Quiet Power of Indicators: Measuring Governance, Corruption, and Rule of Law, which is edited by the authors.

Sally Engle Merry receives funding from the US National Science Foundation..

Benedict Kingsbury receives funding from the US National Science Foundation

Kevin E. Davis receives funding from US National Science Foundation