Friday, 10 December 2010

Measure for Measure, or Measure for Purpose

by Neil McCulloch

I am on my way back from a fascinating conference run by the EU Development Network and the French Development Agency (AFD). The title of the conference was “Measure for Measure” – alas no Shakespeare was involved – the conference was about how we measure development and wellbeing.

One paper in particular stood out to me, ‘Measuring Development: Different Data, Different Conclusions? by Angus Deaton, Princeton University.

Just how many poor people are there in the world?
Angus Deaton’s paper is fascinating – and depressing in almost equal measure. Deaton is the nearest thing to God in the field of measuring development. He is perhaps the single most respected economist working in this field having built a reputation over decades for his meticulous unpicking of all manner of development data – when Deaton says something, you listen.

His paper covers three ways of measuring global progress in poverty reduction – global poverty numbers; hunger and malnutrition; and self-reported wellbeing, and shows the difficulties of making sense of the data we have.

Take the global poverty numbers. In order to be able to calculate the number of poor people there are in the world, or even to compare the GDP of one country with another, it is necessary to convert data from each country into some common currency – typically the US dollar. But this is problematic. It is well known that you can’t just use market exchange rates to do this, since these don’t necessarily reflect differences in prices across different countries. What you need is a way of measuring Purchasing Power Parity exchange rates i.e. the exchange rate that would enable make a standard bundle of goods cost the same (in the common currency) in every country.

This challenge has been addressed by the ambitious International Comparison Program (ICP) which collects prices of a huge number of goods in a large number of countries around the world, so that the price levels (and therefore the GDPs, poverty etc) in different countries can be compared. As you can imagine, this is a costly undertaking (the last round in 2005 cost $45 million) so it isn’t done too often.

Once you have a set of PPP exchange rates for each pair of currencies in the world, then you can convert GDP figures (or consumption/income figures) in local currency into US dollars (or any other currency) and, once everything is in a common currency you can compare them. This is how we know that China’s economy is bigger than Japan’s, but smaller than the US; it is how we know that there is more poverty in India than in Africa. This means that it is important that the ICP PPP exchange rates are right. The problem is, as Deaton shows in great detail – they aren’t. The consumption pattern of a herder in Mali is rather different from an urban Japanese investment banker – indeed it would be hard to find any elements in common!

The ICP do a heroic job of trying to ensure that the goods are very carefully designed to ensure that they are comparable across countries – but the only way that they can deal with different bundles is to assume that the consumption baskets across two countries are, in effect, an average of the consumption baskets of the two countries – in other words that they don’t really represent either country. Now, it is easy to criticise (and hard to think of a better way of doing things) – the PPP rates from the latest revision are the best guess that we have of the relative price levels across countries – but they come with a Health Warning. As Deaton puts it “When people in different countries have different patterns of consumption, there is no non-arbitrary way of calculating cost-of-living index numbers with which to compare them.”

The 2005 round was more carefully done than the 1993 round – but its results are dramatically different. In particular, it found that, when you compare like with like, goods in developing countries are actually considerably more expensive relative to rich country goods than had previously been thought.

This change really matters. Take the $1/day poverty line (which actually is $1.25/day, but that’s another story). If prices in poor countries are higher relative to those in the US, then the $1/day translates into more money in local currency. Which of course means that more people in that country are below that poverty line. Now, combine this with the fact that the World Bank changed the group of poor reference countries that they use to calculate the international poverty line and you can a significant revision of the number of people in poverty in the world. How big a revision? 10 million, 50 million – go on, take a guess. The answer is more than half a billion!

Now it is very important to realise that nobody actually got any poorer as a result of this calculation. It is not that poverty has gone up – it is simply that, if you change the poverty line and the method used to compare across countries, you get very very different results.

As a final twist in the tale, the revisions also mean that if you were to calculate global poverty figures in Rupees instead of US dollars, you would find virtually no change in global poverty! Why? Because, the corresponding decrease in the income of US citizens measured in Rupees still doesn’t pull that many of them below the poverty line, because they are relatively rich already.

As Deaton says “there is much more uncertainty than is commonly recognized”.

Hungry for sensible numbers
Angus Deaton then goes on to look at our numbers for world hunger in his paper Measuring Development: Different Data, Different Conclusions? Given the huge uncertainty about income poverty numbers, one might be forgiving for hoping that figures of undernutrition and malnutrition might be more robust, since they don’t require complex calculations of the relative price levels of different countries. But sadly, Deaton shows that the situation is not a whole lot better.

Take undernutrition – i.e whether or not people get enough calories. The main figures for this come from the Food and Agriculture Organisation (FAO) – released in September 2010. How exactly does the FAO know how many calories people eat in 2010, when they release their figures in September i.e. three months before the end of 2010?! The answer is that they are based on forecasts from the US Department of Agriculture – in other words the latest data are never actual data, they are forecasts of what people might have been able to consume.

The problems don’t stop there. The way in which the FAO calculate these figures is based on constructing national “Food Balance Sheets”. This basically takes an estimate of how much food was grown in the country, adds imports, subtracts exports and a fudge factor for wastage and that is how much food there is in the country. They then convert the food into calories, and make an assumption (log normal) about how the calories are distributed across the population and that is the number of calories that are “available” per person.

As a rough and ready measure it has some merit. But estimates (almost always from Ministries of Agriculture) of how big harvests are well known to be subject to considerable errors and bias (as are the trade data). The calorie conversions and wastage factors are also approximations. And finally, their distribution assumption takes no account of how access to food is actually distributed in any particular country.

More fundamentally, it’s not clear whether calorie availability/consumption is necessarily a good guide to welfare. Take India (on which Lawrence Haddad, Director of IDS, has famously written and blogged). India has grown extremely rapidly in recent years – and its calorie consumption has fallen. There are several possible explanations for this. One is that the growth has totally bypassed the poor. This may well be true, and if it is it casts doubt on a development model focused so strongly on rapid urban industrial and service sector growth.

But it may not be this simple. There has been a massive shift in recent years away from manual labor towards office and factory based employment. The latter kind of employment simply requires less calories. Moreover, as people get richer their tastes change. In particular they tend to purchase fewer more expensive (and higher quality) calories. So it is not at all clear what the decline in calorie consumption in India means from a welfare and poverty reduction point of view. As Deaton puts it starkly “we cannot use calorie based measures to estimate the prevalence of hunger, either over space or over time.”

What does it all mean?
Reading papers by Deaton and listening to him talk is enriching and bewildering in almost equal measure. So what are the bottom line messages that I take away? I suggest two:

1. Understand how numbers are createdProducing figures is important; I think we simply need to be more honest and transparent about their weaknesses. We should continue to improve the ways in which we do calculations, but also recognise what we do not know.

2. Measure for measure or measure for purposeIn the discussion at the end of the conference, Joachim von Braun, from ZEF said “what we need is not measure for measures sake, but measures for purpose”. Moreover, as Deaton pointed out, the fact that revising the poverty figures upwards by more than half a billion people gave rise to “hardly any major reaction from the international community suggests that global measures play little or no role in international policymaking”. So the client is the North and even the North doesn’t care that much!

We need to shift the domain and purpose of development measures. We need development measures which are designed by, owned by, understood by and monitored by the people actually responsible and accountable for developmental actions. There is a clear place for international statistics as a global public good. But at the same time we need better national and sub-national statistics driven by local demand and linked to a real purpose.