Questioning the Numbers: Sally Merry challenges the shorthand truth of global indicators

During the 1990s scholars and organizations developed global “indicators”—numerical assessments of countries’ performance in areas such as poverty level, immunization coverage, transparency, and rule of law. The rankings were a response to the growing desire for accountability, and enabled governments, donors, policymakers and international bodies to compare and rank performance against global standards.

But in a new book, Sally Merry, who teaches at the Law School as an affiliated professor of anthropology and is faculty director at the Center for Human Rights and Global Justice, calls for caution in relying on these numbers.  Merry conducted six years of ethnographic field research around the globe and, in The Seductions of Quantification: Measuring Human Rights, Gender Violence, and Sex Trafficking, she concludes that, despite their “aura of objective truth,” indicators produce information that is partial and misleading.

“I get frustrated about the extent to which this kind of quantitative knowledge is being used as a way to understand the world,” says Merry. “Detailed, qualitative studies are often dismissed as anecdotal. That really misses a lot of knowledge that is produced by deep, contextualized studies that give you a lot more insight into the way things work.”

Merry is part of a larger group of faculty members—and a network of research scholars around the globe—who have participated in a project to examine indicators. In 2012, the Law School’s Institute for International Law & Justice (IILJ) published Governance by Indicators: Global Power through Quantification and Rankings, a book that was edited (and substantially written) by Kevin Davis, vice dean and Beller Family Professor of Business Law; Angelina Fisher (LL.M. ’04), adjunct professor and IILJ program director; Benedict Kingsbury, Murry and Ida Becker Professor of Law, and Merry. And Merry, Davis, and Kingsbury were co-editors of The Quiet Power of Indicators: Measuring Governance, Corruption, and Rule of Law, published in 2015.

In research for her current book, which was sponsored by the National Science Foundation, Merry traced the way indicators were developed for use in three areas: human rights, violence against women, and trafficking. For each, she looked at the type of expertise involved in developing the indicator, what is measured, who funds the data collection, and which organization promotes the indicator. Looking behind the numbers in this way is necessary, Merry writes, because indicators “camouflage the political considerations that shape the collection and presentation of data.”

The ability to construct indicators, Merry notes, rests in the hands of experts who are typically from the Global North, and the process often excludes the input of those being measured. Furthermore, theories of social change are embedded within indicators, but Merry points to research by Professor of Clinical Law Margaret Satterthwaite (who authored two chapters in the 2012 IILJ book) showing that initiatives based on these theories may not always produce the anticipated outcomes. For instance, infant-nutrition training programs may be expected to result in fewer underweight infants, but social change will not happen if the problem is not ignorance but simply not enough food available. 

One case that exemplifies the politics of indicator construction is the United Nations Statistical Commission’s set of indicators for violence against women, which was developed in the 2010s. The indicators, Merry found, are based on a narrow definition of violence, measuring, for instance, physical and sexual violence in intimate and non-intimate relationships, but excluding state violence by police and the military, government sterilization, honor killings, and dowry murders.

As a solution to the seductive nature of indicators, Merry proposes greater “indicator literacy”—more skepticism about what these numbers mean and don’t mean. She argues for more local participation in the construction of the data behind global indicators, and for pairing quantitative indicators with qualitative analysis. “The current rush to quantification,” writes Merry, “risks sacrificing the insights of rich, ethnographic accounts.”

Posted August 29, 2016.