The survival error in social science
Matt Berkley
Draft released as “People, words and numbers”, 5 February 2006
Retitled 13 February 2007
Summary
This document takes as a starting point an error in the theory of social science: failure to take account of survival rates when aggregating outcomes.
Several Millennium Goal indicators, other things being equal, show “better” progress if people die earlier.
This logical truth may indicate a need for reform in the education of social scientists in several disciplines.
The document goes on to examine the use of some words in large-scale social science.
Part 1: A suggestion on the aims of international development
Email to Professor Jeffrey Sachs,
chair of the World Health Organisation Commission on Macroeconomics and Health
The text of the email of 11 April 2001 begins below at “Subj: Economics of Survival”.
Quotations:
“Here are some theoretical points about mortality rates and the International Development Goals....
To me, no outcome measure is humane unless it takes into account what happened to people who started the period but didn't make it to the end. If the poorest die, the average income of those alive at the end of the period will be higher....
My suggestion is this: For any outcome measure....account needs to be taken of those within the relevant group who did not achieve the target, whether through death or any other path. ”
Introductory comments
In mid-2000 I became aware of a common error in economics: to claim to have aggregated outcomes for individuals without considering survival rates. Strictly speaking, the tradition in macroeconomics has been to confuse “the average rise” with “the rise in the average”, and “poverty alleviation” with “poverty reduction”. If some people die, the figures look better. I raised this problem with several senior academics and in 2001 with officials in the UK Government, the OECD and the World Bank.
In April 2001 my reading of material from the World Health Organisation Commission on Macroeconomics and Health had not revealed any attention to this problem. I wrote to Professor Sachs explaining the problem.
In December 2001 the Commission’s Report and at least one of its background papers did make the distinction in relation to national economic statistics. As an approach to aggregating benefits to people, the idea of looking at trends for people is clearly right, while the idea of looking at trends for statistics about populations in the abstract is clearly wrong. The Commission chose the former.
In relation to survival rates, other questions arise relating to:
a) how in future governments might agree clearer goals;
b) the need or otherwise for revisions of existing internationally agreed goals, targets and indicators;
c) how scientists might accurately describe their findings where data and/or demonstrably reliable inferences on relevant survival rates are not available;
d) how various social science disciplines should be reformed in this respect;
e) the implications of the existence of the error for the education of scientists, policy-makers or the public.
The Commission sensibly did not set an absolute value on years of life.
However, another general problem in the Commission’s approach was that its ascriptions of benefit, like many claims from economists, were strictly speaking based on a confusion between income and profit. Neither GDP per capita nor income adjusted by life length measures prosperity. One reason is that neither of those measures need for expenditure.
These two areas - the cost of dying and the cost of living - form the basis of some remarks about Millennium Goal indicators in part 2 of this document. Other remarks relate to aspects of demography other than life length.
Some thoughts have recently occurred to me in relation to the Commission’s assigning of monetary values to years of life. These thoughts may turn out to be fundamentally mistaken.
The Commission estimated that if certain inputs were made, there would be at least $186 billion of savings in income and increased life for people.
One thought runs along these lines:
“Other things being equal - if the Commission’s idea is accepted that financial values can be given in a meaningful sense to years of life, which is not necessarily uncontroversial - the figure of $186 billion of savings seems far too low, for a reason not connected to the value of years, but to the value of currency.
Suppose
A. Two people die early, in the
B. The US citizen was living on $1000 a year, and the Indian citizen on $333 a year.
C. Because of exchange rates, they could both afford the same (and everything else is equal).
D. They each lose 30 years of life.
It seems to me that the logic of the Commission would be that the
The value of wages lost by a person who dies early might perhaps reasonably be given in real money (though on its own it is perhaps not of great interest).
The question of assigning a value to the lost years is a different question. Without prejudging whether this is a sensible question or not, we might think about what the approach would be if we decided that it was a sensible question. At the minimum, it would seem sensible to use units which relate to comparable purchasing power in different countries. Otherwise, the value of life would depend on exchange rates.
In practice, international purchasing power estimates for people living on such amounts are not generally available.
If we ignore that problem, and for the sake of argument assume (perhaps unreasonably) that “purchasing power parity” estimates for whole countries are roughly right and appropriate for people’s basic needs, then other things being equal, if the rest of the Commission’s logic is accepted, the minimum estimate for savings from inputs into health care should have been over $500 billion rather than $186 billion”.
I do not mean here to imply that life can be given a monetary value. I do mean to imply that it is sensible to value people’s lives in a similar way to one’s own life in the absence of a convincing argument otherwise.
The general area of exchange rates spurs me on to another thought. Where in economic theory is the following aspect of real life modelled?
Person A has an excess of the money equivalent of 1 bowl of rice beyond their needs.
Persons B and C lack 1 bowl of rice each.
Person A lives in a country where both wages and prices are double those where B and C live.
So if A gives up 1 extra bowl, B and C each gain an extra bowl.
A second question along these lines might be, how would a theory of utility cope with such a situation?
The point of that part was to highlight the fact that money of people in rich countries is worth more to people in poor countries not only because of
a) diminishing returns (an extra bowl of rice is worth more to a starving person than to an obese person)
but also because of
b) the fact that the money buys the person in the poor country more than the person in the rich country can buy with it. (a bowl of rice is worth more than the half bowl which the same money would buy in the rich country).
I think this may have some bearing on discussions about the morality of - to the extent that moral discussions make sense - both international aid policies and policy decisions which affect populations of countries where money is worth widely differing amounts.
Subj: Economics of survival
11 April 2001
Dear Professor Sachs
I wonder if this of interest to the WHO Commission.
Here are some theoretical points about mortality rates and the International Development Goals, and then some practical points which are less simplistic.
Among the International Development Goals, progress has been faster on reducing the proportion of people in extreme poverty, and slower on child mortality.
The question no-one seems to be asking is this: Is the proportion of poor people getting smaller partly because child mortality is worse than we hoped?
Most of the goals [later correction: “several indicators” - 21 indicators for the Millennium Goals] are susceptible to the problem that if the worst-off die, we are closer to the target.
There are good grounds for thinking that the child mortality goal being on track provides a statistical safeguard among the goals - if this goal goes according to plan, it ensures that we do not get a false impression of progress towards the other goals simply through high death rates among the poorest.
Grounds for believing this include the following. Firstly, child mortality is concentrated among the poorest, so an improvement in the total may well reflect improvement among the target groups. Secondly, the child mortality rate is believed to give an indication of the rate of early deaths among adults Policies which reduce child mortality are likely to also reduce early mortality among adults.
If all this is true, then the closer we are to the required rate of progress on child mortality, the more poor people there are. Slow progress on child mortality (as now) makes for fewer poor people, so the poverty goal looks closer - simply because fewer poor people are alive, not because more of the survivors have raised their living standards.
My hunch is that slower progress on total child mortality means much slower progress on child mortality among the poorest. If this is so, then the effect is stronger.
To me, no outcome measure is humane unless it takes into account what happened to people who started the period but didn't make it to the end. If the poorest die, the average income of those alive at the end of the period will be higher than the average when the group included the poorest, even if none of the survivors' income has gone up. It even looks higher, if enough of the poorest die, when the average among the survivors goes down somewhat - simply because the poorest are no longer there to pull the average down.
If we measure the income of those alive in 1995 and then the income of those alive in 2000, we will not notice the decline in income of someone who died in 1998. The average income of those alive will be exactly the same as if he had survived and raised his income to the average of the group. In fact, since most people in poor countries work on the land, vulnerability is seasonal, and therefore the people who die may have a declining income for a few weeks or months before they die. This is too fast for measurements taken every five years.
My suggestion is this: For any outcome measure - reducing poverty, achieving 100% schooling - account needs to be taken of those within the relevant group who did not achieve the target, whether through death or any other path.
[Later note: “Reducing poverty” is strictly speaking not an outcome measure but an expression of an opinion. If poverty is a lack of prosperity, and prosperity is subjective, then by definition poverty cannot be measured.].
In practice, the relationship between child mortality and statistical progress on the goals would appear to need careful research (see below).
Practical considerations
In real life, there may not be such a clear division between the poorest and the less-poor. However, DHS data seem to point to assets as important determinants of child mortality - the lowest 10% can be far more vulnerable than the next 10% (Bonilla-Chacin and Hammer, "Life and Death among the Poorest", 1999, revised version 2001 forthcoming, World Bank). There may be a clear division, for example, in some geographical areas, between the landed and the landless.
In real life, policies which reduce the proportion of people living under $1 a day may also save the most vulnerable from death. This cannot be assumed, and may depend on the relative vulnerability of the poorest (see previous point).
In real life, the poorest may produce more children to replace those who have died. The total number of poor children could conceivably be the same in 2015 whatever the child mortality rate. But if adults as well as children die in hard times this is unlikely.
The statistical relationship between mortality and outcome measures can only be determined by careful research, together with an intimate knowledge of household behaviour.
Statistical progress on the goals needs to be translated into human terms. If there is any suspicion that apparent progress on any of the goals is helped by lack of progress on any of the others, then this is an argument for tackling the goals that are furthest behind, not the ones that are furthest ahead.
---------------------
Later notes:
1. A better title for the email would perhaps have been “statistics and survival”. My suggestion above to Professor Sachs related to any outcome measure, which implies “in any social science”. Also, the scope of theoretical points made in the email concerning economic data was wider than internationally agreed goals.
2. Perhaps a better hypothesis than
“the proportion of people may be falling due to child mortality”
might have been something like this:
“The puzzle that
fast reported progress on economic indicators which are traditionally thought to indicate aggregate outcomes for people
has co-existed with
slow progress on child mortality
may be partially explicable by
a) the logical truth that the economic indicators do not in reality aggregate outcomes for people, but for populations,
combined with
b) a hypothesis that current bad progress on child survival may be indicating low adult survival as well, among the target population for the economic indicators (the “poor”).
Part 2: Notes on Millennium Goal indicators
The Millennium Goal indicators listed below do not take survival rates into account. They are not in themselves guides to the aggregate progress of individuals. They risk being mistaken for such.
The points below relate to the interpretation of country-level trends as well as global trends.
a) Common sense is needed to consider which indicators may be problematic for which countries and time periods. For example, indicator 22 on malaria may not be problematic at all in context: another indicator deals with malaria deaths.
b) The application of common sense does not necessarily mean applying traditional inferences in social science. The guiding principle might reasonably be one of caution in inferring benefits to people without evidence on survival rates.
c) Indicator 18 on HIV would seem to be more problematic than that on malaria, given the absence of any indicator relating to HIV deaths.
d) The text of the letter to Professor Sachs, and notably the suggestion, deal with a fundamental principle of social science. A reasonable line of argument might be this: it is important for social scientists to understand the principles, so that methods can deal with any reasonably foreseeable circumstances.
e) The letter to Professor Sachs and this document do not make specific claims that particular indicators have shown the wrong trend as a result of mortality differences. Nor do they claim that any will do so in the future.
f) However, the context is of falls in life expectancy in some countries during the 1990s. The effects of AIDS on demographic projections in some African countries have been large. Reasonably foreseeable circumstances which would affect the statistics may already have come to pass.
g) It is perhaps worth noting here a point I made to Professor Frances Stewart of
h) The mortality flaw is a flaw in the logic of arguments used by social scientists.
i) Two aspects to the mortality flaw are mentioned in the letter to Professor Sachs.
One is that aggregate outcomes are not measurable without considering survival rates. Again, common sense should override this rule where necessary. Exceptions to this rule are where variations in survival rates have previously been excluded on reasonable grounds as insignificant or irrelevant. The point here is that some people’s outcomes are left out.
Another aspect is that differences between statistics across time, place or circumstance may be caused by differences in survival rates between people at different levels of the variable being studied.
A third aspect of the mortality flaw is that the effects of deaths on people’s happiness may not be limited to those who die. Parents generally prefer their children to stay alive. Where children die, the work put into being pregnant and raising them is wasted. In
j) Note that indicators may also be affected by birth rates - not just by the effects of birth rates on age structure, but also by the direct effects of birth rates on population numbers. It is important to understand the difference between “there are x million fewer people in a category now” and “x million people went from one category to another”. More detail is given below.
This list comprises 21 of the 48 indicators. The descriptions are those provided by the United Nations Statistics Division.
1. Proportion of population below $1 (1993 PPP) per day (World Bank)
2. Poverty gap ratio [incidence x depth of poverty] (World Bank) [* see note by MB below]
3. Share of poorest quintile in national consumption (World Bank) [** see note below]
4. Prevalence of underweight children under five years of age (UNICEF-WHO)
5. Proportion of population below minimum level of dietary energy consumption (FAO)
6. Net enrolment ratio in primary education (UNESCO)
8. Literacy rate of 15-24 year-olds (UNESCO)
15. Proportion of 1 year-old children immunized against measles (UNICEF-WHO)
17. Proportion of births attended by skilled health personnel (UNICEF-WHO)
18. HIV prevalence among pregnant women aged 15-24 years (UNAIDS-WHO-UNICEF)
19. Condom use rate of the contraceptive prevalence rate (UN Population Division)
19a. Condom use at last high-risk sex (UNICEF-WHO)
19b. Percentage of population aged 15-24 years with comprehensive correct knowledge of HIV/AIDS (UNICEF-WHO)
19c. Contraceptive prevalence rate (UN Population Division)
20. Ratio of school attendance of orphans to school attendance of non-orphans aged 10-14 years (UNICEF-UNAIDS-WHO)
22. Proportion of population in malaria-risk areas using effective malaria prevention and treatment measures (UNICEF-WHO)
29. Proportion of population using solid fuels (WHO)
30. Proportion of population with sustainable access to an improved water source, urban and rural (UNICEF-WHO)
31. Proportion of population with access to improved sanitation, urban and rural (UNICEF-WHO)
32. Proportion of households with access to secure tenure (UN-HABITAT)
45. Unemployment rate of young people aged 15-24 years, each sex and total (ILO)
46. Proportion of population with access to affordable essential drugs on a sustainable basis (WHO)
47. Telephone lines and cellular subscribers per 100 population (ITU)
48. Personal computers in use per 100 population and Internet users per 100 population (ITU)
Source: http://millenniumindicators.un.org/unsd/mi/mi_goals.asp .
* Note on the word “poverty”:
Indicator 2 is described as, “Poverty gap ratio [incidence x depth of poverty]”.
It is not clear why the United Nations Statistics Division used the word “poverty” here - or why indicator 1 might be thought of as indicating poverty, even in the absence of problems relating to life length. The data are mostly on what people said they spent. They are not data on economic shortfall relative to need.
Most people might think poverty is excess of needs over resources. Let us explore some thoughts on the assumption that this is a sensible definition of poverty.
Suppose a researcher has information about resources. How might they come to a judgement about poverty without looking at needs? Needs include needs for food, rent, water and so on. It is not clear how an argument might be advanced in favour of the idea that lowness or highness of resources might measure prosperity or poverty without reference to needs. Surely, if you earn X-plus-1 units but need to spend 2 more units on health or rent or food or water, you are worse off than your friend who earns X units.
One type of argument in favour of looking at resources without looking at needs might stem from hypotheses about general trends in real life. A person might say “in the real world, income differentials, across time, place and level, are so big that they overwhelm the effects of things like people moving to cities and needing to pay rent”. Intuitively, I do not find this line of reasoning very appealing. If landlessness is a big problem in some countries, then its most immediately obvious effects on command of material resources - lack of assets - may be compounded by the fact that people in cities may have to pay rent, and/or live in worse conditions than others who stay in villages. Quantifying the value to people of living in a slum versus a village seems to me an enterprise which would have a significant element of subjectivity. But even ignoring this problem, the idea that the costs of rent can safely be left out, in an era of fast urbanisation, on the grounds that incomes will be rising fast enough to render this kind of thing insignificant, seems to me to be implausible, or at least in need of empirical examples. In any case, the burden of proof is not on the sceptic who points out the scientist’s assumptions. The sceptic in such a case points out what the scientist must show. In this kind of situation, the burden of proof is on the scientist to show that their claim is reasonable.
Additionally, in this case, mostly the data are not on resources, but on spending. What kind of argument might be advanced in support of the idea that “if you spent 1% more, this shows you got 1% richer”? One argument could be of the form “on average people spend more when they have more”. One problem is that this type of argument would not tell us about any particular period of history, or any particular place. In this particular period of history, for instance, in some places AIDS might reasonably be thought to raise need for expenditure.
A question arises as to how social scientists might describe the existing statistics more accurately. “Poverty” seems wrong, since the data give no specific information about either prices or needs for the relevant people. Even if there had been data on prices, and judgements about relevant prices and relevant needs which were indisputable, income, spending and the money value of consumption could not measure economic poverty as a whole, even in a narrow sense related to legally-defined personal command over resources, since they leave out assets and debts. Statistics on income, spending or the money value of consumption could not by themselves measure income poverty or consumption poverty, since they do not measure consumption need or relevant prices. They do not measure necessary expenses.
A more accurate description of this indicator would be “spending gap ratio [prevalence x depth of spending gap]”. However, this would still not be entirely accurate even ignoring the demographic problems. This is so for at least three reasons.
The first reason why “spending gap” would not be accurate is this: the data are not on what people spent, but on what they said they spent.
The second reason is that people who are destitute may not be reachable. In order to understand the potential importance of this factor, it is necessary to understand how the mathematics works.
The whole point of these types of measures - traditionally claimed to show the “depth of poverty” - is to place more mathematical emphasis on people who are further from the official poverty line (“poorer”). So, for instance, measures like this look at how far each person’s number is from the official line, and add the numbers up in a way that gives more weight to people further from it. This makes sense in theory, at least from the philosophical standpoint implied by the traditional language of development macroeconomics, since the stated aim is to find out not only “how many poor people there are” but also “how poor they are”.
I say the stated aim, because from a philosophical standpoint where needs are not objectively quantifiable, any claim to have measured prosperity or poverty is logically false.
Part of the argument for this standpoint could be this: The concept of a “need” has no meaning except by reference to a particular purpose. You and I do not “need” anything in an absolute sense. We only need things if we want other things to happen. I don’t “need” to live. But I do need certain conditions to be met in order to live.
Beyond bare survival, the relative values of purposes reduce to value judgements about what activities are important.
A related argument could be this: In practice, assessing consumption needs of different people is too complex to be feasible at any reasonable cost. For instance, to come to a judgement about even one person’s food consumption needs you would need to know at least their size, age, gender, workload, the nutritional value of each component of their diet and how the components worked more or less successfully together to make a balanced diet for their body and their activities. For practical reasons, in one sense it is not surprising that official statements about poverty at the national level tend to be based on the cost of calories rather than costs of a balanced diet. Calories do not tell me about how well nourished you are. They are a measure of energy consumption. In another sense, it is surprising that anyone might think calories measured physical well-being.
For mathematical procedures such as those used for what are traditionally called “measures of the depth of poverty”, the absence of destitute people may be important. Without knowing about homeless people, even in the absence of other philosophical, theoretical or data problems, it is not clear how social scientists might justifiably claim to know what has happened to the poorest or to have measured the “poverty gap”.
The third reason why “spending gap” would not be accurate is that the data are not all on what people said they spent. Some data are on what people said they earned. Some are on money values which researchers assigned to people’s answers about what they grew for themselves to eat.
In the official description of indicator 2, the word “incidence” is strictly speaking, according to the most common usage, incorrect.
The tradition in economics is to use the word “incidence” for what people usually call “prevalence”.
Usually:
Incidence is the number of incidents over a period. It is “how often something happens to real people over time”.
Prevalence is “the prevailing rate in the population at one time”.
If incidence rises and so does mortality, prevalence can fall.
Incidence is about real people.
Prevalence is about populations, not what happens to real people.
The discrepancy between the terminology in economics and other social sciences can be seen in the list above.
** Note on indicator 3, “share of poorest quintile in national consumption”:
A “quintile” is a fifth.
It is not clear why the UN Statistics Division referred to “consumption” rather than “consumption expenditure”..
The data compiled by the World Bank from national statistical agencies were on financial amounts, not consumption amounts. The data are mostly on what people said they spent. It is important to distinguish “consumption” from “consumption expenditure”. If I have data on people’s spending, then my figures will relate to what might be called the “share” of spending, not the share of consumption. Economists sometimes shorten “consumption expenditure” to “consumption” rather than “spending”, and the reason is not clear. The word “consumption” in the phrase is not a noun, but an adjective.
Again, this is in addition to the problem about mortality, which makes the use of the word “share” problematic in any case. Because of the problem about mortality, the indicator could not show how much people in the quintile (fifth of the population) consumed even if the statistics had been on food, water, fuel and shelter. This is strictly speaking a confusion between a fifth of the population and a fifth of people. If the “poorest” consume less and some die, the “poorest quintile” now contains people from the next quintile up, who were consuming more in the first place. In such a case, the “poorest” people have had a lower share but the traditional macroeconomist would in the absence of other data say they have had a higher share.
Another aspect of demographic change is birth rates.
Two relevant aspects of falling birth rates are 1) while reducing total food need they increase average food need, and 2) they lower the count of people anyway.
One line of argument might be this: none of the financial indicators for the Millennium Goals measures poverty, because they do not consider food and shelter needs. Globally, the proportion of children is falling. Among countries, one example is this: other things being equal macroeconomists have underestimated current poverty in
A counterargument might be that life lengths have risen, so counteracting some of the effects of falling birth rates on age structure and efficiency. In the case of
In addition, neither the World Bank nor FAO methods consider the difference between “fewer babies being born” and “people eating better”.
This would still be the case even if they did actually measure consumption adequacy (which is not really possible even in principle, as it is partially a subjective concept) in different years.
The severity of hunger is less if fewer babies are born to hungry families. But that is different from saying “people ate better”, which would be the usual inference drawn, if not by social scientists or politicians, then by the public. The use by a couple of family planning, so that a baby is not born, is not the same thing as someone “rising out of poverty”.
Such a change, even in the absence of other methodological, theoretical and philosophical problems, would not show how much people’s consumption increased. Nor would it show how much their consumption adequacy increased.
Proposals for clearer language from social scientists appear at:
Social science and government aims .
....