How will we know if the SDGs are having any impact?
As long time readers of the blog will know, I’ve been a Sustainable Development Goal (SDG) sceptic since long before they were even
My problem with the SDGs is not with the subject matter – all very creditable – or the number of targets, which bugs some people, but not me. It’s the design, and in particular, the lack of analysis on how they could have political impact. For me, the key question for any international instrument ought to be traction – will this or that convention, undertaking etc (of which there are hundreds, if not thousands) influence the day-to-day behaviour of governments, sub national bodies, private sector or others and how? And yet weirdly, this was never raised during the design of the SDGs, which instead were dominated by adding more topics to the Christmas Tree of SDG issues, and long discussions about indicators, data and metrics.
Even odder, this question has barely been asked of their successor, the Millennium Development Goals. In a classic fudge of causation
Causation v Correlation 101
and correlation, the MDGistas said ‘look, extreme poverty has halved, the MDGs are a success!’ Yet much of that reduction was down to China, and no-one can credibly claim that it was the MDGs that got the Chinese Communist Party out of bed every morning to transform the Chinese economy.
There are a few exceptions I’ve managed to collect in response to repeated rants: Alice Evans showed how MDG5 on Maternal Health prompted the Zambian government to take action for fear of reputational damage; May Miller Dawkins wrote a nice paper on what the SDGs could learn from international environmental agreements in terms of design/traction. Moizza Binat Sarwar studied MDG implementation in five low and middle income countries (Indonesia, Liberia, Mexico, Nigeria and Turkey), largely by interviewing staff of relevant ministries. When Columbia University’s Elham Seyedsayamdost surveyed 50 countries’ implemention of the MDGs, she found that whether the goals were reflected in plans or not, they did not have any apparent influence on how governments spent their money. All great stuff, but hardly a body of research that matches the importance of the issue – a very odd lacuna indeed.
Now the same thing looks to be happening again. Lots of talk of monitoring and reporting against the SDG indicators, and as far as I can tell, no attempt to establish whether the SDGs (rather than other factors) are responsible for changes to those indicators.
So what would an ‘SDG politics watch’ look like?
First the data: we need to know which bodies are using the SDGs in their planning, budgeting, reporting etc. Maybe we could crowdsource this – governments, NGOs, academics etc can all upload evidence to some kind of WikiSDG so we can start to build up a picture of formal commitments
But that is nowhere near enough – how to distinguish between lip service and genuine traction? And why do the SDGs get traction in
Traction or Lip Service?
some places and not others? That’s where the academics could get stuck in. Alice Evans at Cambridge is issuing an open invitation to supervise a good PhD student on this topic. In her words ‘It’s not just about whether governments use the SDGs in their policy documents, but their attitudes towards these goals: do they dwell on issues they previously didn’t because they are concerned about regional benchmarking? Do civil servants in meetings spend time looking at SDG 9.ii (whatever that is), and put pressure on provinces to improve this indicator, or just whizz through it. Do parliamentarians raise these issues?’
Why does this matter? Because understanding when/how an international instrument has traction ought to be the starting point for tweaking SDGs, and designing future instruments that work better. If for example, we were to identify regional rivalry as a key driver in state action, we would put much more emphasis on reporting in regional league tables. If the key to impact is civil society picking up and using the instruments, then (as in the SDGs) it becomes even more pressing to involve CSOs in monitoring and reporting, as well as initial design. And so on.
Thoughts?
Previous rants on this topic include one paper and lots of blogs:
Paper: How Can a Post-2015 Agreement Drive Real Change? The political economy of global commitments
Posts:
Hello SDGs, What’s Your Theory of Change?
How will the SDGs differ from the MDGs?
The SDGs are just getting interesting – what needs to happen next to make them have impact?
17 Comments
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Translate
What might be interesting is to see the extent to which the SDG agenda can effectively be pushed in developing nations by bilateral donor agencies, if at all…!
Thanks, Duncan.
I would love to see a piece on data quality and the MDGs/SDGs. How reliable/ accurate are measurements of MDG achievements? Which SDG indicators can realistically be measured with any meaningful degree of accuracy? And how does reliability vary geographically?
Any takers?
Lots I would expect – there’s a big constituency for gathering data on health, water, education etc, just as there was for the MDGs. What is missing is any systematic research on the politics of SDG traction – that’s the real gap, IMO
You raise good questions. Have you followed the collective impact movement in the US? Here’s a link from one of the lead consulting firms on evaluating these efforts. http://fsg.org/publications/guide-evaluating-collective-impact I think there might be many of the same challenges. Do you see any cross-learning from the international SDG community and this collective impact movement?
Thanks for keeping the SDG discussion alive. Indeed, it’s rather weird in the first place that we now have “Sustainable” Development Goals, which suggest that the first MDG’s where not supposed to be sustainable? Or is this the first lesson to be learned from the MDG lobby failure?
So we make some progress at last in the way we think about development; it should be sustainable? Which brings us to the key question:”What exactly do we mean by “Sustainable”? For instance, is a community handpump in Africa that is maintained with the donations of good willing people in the USA sustainable, because people will always give to Water Charity NGOs? So it does not matter if the community gets a durable pump that they could maintain themselves, or a fragile one that is more expensive to maintain, because the USA donors will provide anyway. One could argue both sides.
Next question is “What is development?”. It’s not just about “Poverty Reduction”, because poverty is rather relative. For instance, wages in China are lower than in the USA, but does that mean that China is less developed? We tend to look at these issues from our own perspective, and we judge things based on our own perception.
Fact is, that the MDGs and now the SDGs are used as an “excuse” to start projects and to spend a lot of money for “Poverty Reduction”, because it is said that we need to go “From Poverty to Power”.
But i think it remains to be seen if this “Fighting Poverty” attitude is the end goal. It seems indeed more an excuse to justify all kind of initiatives to spend (free…)donor money in a black box, without any realistic monitoring of the impact or “Value” that is created for these poor. Most of the time, it’s the implementing organisation and it’s employees that profit the most.
So one could argue, that the money is not well spend, however, to me the key question (or problem) is not so much what they actually do with the money, but where the money comes from. So it’s not so much about FTM (Follow the Money), but more WITS (Where is the Source) and what the Source wants to achieve.
The sources, the public donates in good faith, and (Democratic) Governments that represent tax payers also donate because their voters want them to do so. By the way, do you know of any non-democratic government that has a donates as well?? Companies donate as well, but most of them mainly to boost their “public profile” to sell more and make more profit, and not so much from an empathic attitude to lift people out of poverty.
But at the end of the day, if their donations (and the MDGs and SDGs are basically all about donations…) comes in the hands of people that love to spend the money of others without taking any personal responsibility for it… , you end up with a weird situation, because indeed, how can we know if these donations make any difference, and how sustainable is this?
Just a simple example, how this works in the water sector MDGs, for instance in Benin, in the framework of the MDGs, the Dutch Government donated some 7 years ago 1.200 handpumps to the public with a 10 year functionality warrantee, but most of them don’t work anymore, because they break down all the time, so after a few years, maintenance became very expensive and spare parts are not available. Now they blame it to the people, that they don’t want to pay for repairs. In fact, it is seen as a national disaster, but nobody takes responsibility for it, all stakeholders point at each other, but the people are left without water again.
This is a typical example of thousands similar cases. What it indicates is even more worrying: It means that even in case monitoring of the impact is rather simple ( a pump works or does not work…) this failure to make an impact seems not important to change the policy or to give less funding. On the contrary, in the recent High Summit WASH meeting in New York, the NGOs just stated that “More funding is needed”, instead of asking themselves questions about their impact and efficiency.
Since we have Donald Trump as president of the USA, it seems to be normal to simply deny the truth and get away with it?
Conclusion: To have an impact, it is more efficient to work with your own money and not with a “free gift” that you can spend on whatever, just because of your “Good intentions”… And secondly, if it’s your money, like an investor, you want to follow up, before you give for the second time.
Albert Einstein said: “The definition of insanity is to do the same thing over and over again, and expect different results” …. I’m afraid this also counts for the MDGs and SDGs projects; as long as they spend other’s donation, there will be no “other” (sustainable) results…
For anyone interested in this there is a network called EVALSDGs, which aims to “inform, support, measure and assess development efforts around the SDGs”. It was set up in 2015 by the UN Evaluation Group and the International Organization for Cooperation in Evaluation. More info here: https://evalpartners.org/evalsdgs/about.
I absolutely agree with this. There’s a massive gap when it comes to thinking about the issue of additionality: what is happening because of the SDGs that wouldn’t have happened anyway?
An additional complication is that even if a government changes its priorities due to a particular goal or target, how do we know its not taking attention and funding away from another goal? You don’t have to think about that so much when you’re pushing a single-issue campaign.
I think it would be more fruitful, at least initially, to look at changes in language and norms. For instance, can we see if the shift in education from thinking about enrollment to learning been accelerated by the language used in the SDGs?
Kudos to Alice Evans for paying attention to issues of on-the-ground implementation–it’s generally overlooked by those interested in development, but it’s crucial if systemic change is to be successful.
Hi Duncan,
In 2011, CAFOD commissioned a study amongst civil society leaders in the global south to ask whether they found MDGs useful. (100 Voices http://cafod.org.uk/content/download/553/5570/version/2/file/Policy_Beyond-MDGs_100-voices-report_English-full-report.pdf ). Despite their limitations, the overall answer was that the MDGs provided a language, a hook, an entry point for the civil society to engage with government and coordinate at different levels, and most people felt that it helped achieve important objectives. This does not proof causation but at that time, CAFOD felt it was enough to mobilise and ask for another global development framework – this time designed in a more participatory way. The UN was not very proactive and thus a civil society network BEYOND2015 was founded to push for a new agenda. From 2012, in the process of getting to the SDGs I witnessed many important conversations on issues of development between different actors that weren’t happening before. These conversations are in itself an achievement. As countries defines national implementation plans, hopefully more interesting conversations will take place at different scales.
Recently, Felipe Castro, Director of Monitoring and Evaluation of Public Policies at the National Planning Department of Colombia, gave an interesting presentation (The SDGs: a tool for peacebuilding in Colombia) at UCL explaining how Colombia is aligning national and regional strategies with SDGs in a very interesting way, focusing on different SDGs according to local priorities.
(the talk is available here: https://www.mixcloud.com/dpuucl/the-sdgs-a-tool-for-peacebuilding-in-colombia they used good visual which are not available).
We can’t easily measure the impact/added value of SDGs as a framework, attribution is difficult. Even without the framework countries would have some sort of national development strategies.
SDGs in themselves will not be responsible for progress, national and local policies and their implementation deliver change. However, SDGs are an important common language which frames the path towards change and enables a number of actors to participate and converge in the process of defining local priorities.
Thanks Andrea, I remember the study. What might have been more interesting is to ask civil society leaders ‘what international instruments do you find most useful?’ and then seeing whether they mentioned the UN, the ILO, regional treaties or whatever. Asking ‘do you find the MDGs useful?’ is a very leading question!Then a follow up question could have been, and what is it about those instruments that make them relevant – then we could have tried to feed that into the design of the SDGs. But no-one ever asked…..
The Colombia case is interesting – definitely a candidate for WikiSDG!
Hi Duncan,
Agree the 100 voices was leading.
With lots of limitations what we tried to do with the Participate initiative and the COMPASS studies ( http://cafod.org.uk/content/download/11319/89078/version/10/file/CF-Compass_report.pdf ) was very much along the lines of what you suggest. The participatory research with women and men living in poverty was asking how change happened in their lives in the past 15 years, roughly the MDG period but the questions was open. These studies contributed to the design of the SDGs.
I know this is slightly different but Participate also worked on a review of how other policy framework embedded the views and perspectives of those living in poverty and what worked. (http://participate2015.org/publications/what-do-we-know-about-how-to-bring-the-perspectives-of-people-living-in-poverty-into-global-policy-making/ )
This is to agree completely with you on the need for more research and data on this, but wanted to point out that some initial steps in that direction were made.
Spot on!
Now, how are we going to make decision makers and the EU read (and care) about this post??
any suggestions Maria?!
Hi Duncan
I’m not an M&E specialist and I haven’t seen any systematic examinations of the impact of the MDGs (and I would love to do so!).
What I can draw on is my experience as an auditor in developing countries (especially Vietnam and Laos) for the last 20 years. In the last 5-6 years, discussions about development have tended to use the MDGs as a framework: as a result, all ideas about development (whether from international aid agencies, governments or local CSOs) were created and assessed in relationship to the MDGs. This was particularly noticeable in Laos, where the reduction of the impact of unexploded ordnance (UXOs) was phrased as a 9th MDG, specific to the Lao PDR. Part of this was driven by the UN and other aid agencies, but I have also been to meetings where the Lao Minister of Education was using the results against MDG targets to identify the specific priorities of his department.
I realise that the plural of anecdote is not data, and would therefore be very interested in seeing a formal study of the impact of MDGs.
Best wishes
Mark
On 28/11/2016, Authority Enquiries <authority.enquiries@statistics.gov.uk wrote:
Dear Mr Berkley,
…You cited two instances where the Millennium Development Goals were stated
as being agreed in 2000; once in Assessment Report 315 Statistics on
International Development[1] and once in a blog posting[2] on the GSS
website. I concur with you that these statements are factually incorrect.
…we have corrected and republished Assessment Report 315. …
Furthermore we have posted a correction note to the GSS blog posting.
It would seem the inaccurate reporting of what was agreed at the Millennium
Summit in 2000, where world leaders adopted the UN Millennium Declaration
not the Millennium Development Goals, has been common place. In that regard
we will raise this issue with DfID and also with the team working on
international development in the Office for National Statistics.
…
[1]
https://www.statisticsauthority.gov.uk/publication/statistics-on-international-development-2/
2
https://gss.civilservice.gov.uk/blog/2015/09/a-sustainable-approach-to-statistics-and-geography/
…
Victoria Leadbetter | UK Statistics Authority
I suggest that questions about assessing SDG success cannot reasonably be considered without looking at the truthulness of official statements about progress, pledges and research methods.
The concern is not just about past claims but claims being made now.
The widespread claim by people working in international development that generally easier, backdated 1990-2015 targets were agreed at the Millennium Summit has no basis in UN resolutions.
I posted about this on this blog in July and September 2015.
http://oxfamblogs.org/fp2p/how-should-the-aid-business-think-and-act-about-corruption/#comment-23594
http://oxfamblogs.org/fp2p/some-cautionary-thoughts-on-this-weeks-sdgs-summit/#comment-63868
Thomas Pogge of Yale has been saying that for many years. I made the mistake before reading his work.
Similarly, claims that the 25-year targets were agreed between UN member states in 2001 or 2002 are not supported by the texts of resolutions.
In reality, the 2002 Johannesburg Summit explicitly reaffirmed mortality pledges from “their prevailing rates in 2000”.
Oxfam, Oxford Reference books, at least two Nobel Prize winners and many others have been encouraging people to hold governments to account for the wrong pledges. The error appears on page 145 of “How Change Happens” and Oxfam educational material.
In wrongly claiming that the 1990-baseline “Millennium Development Goals” were agreed “almost fifteen years ago”, world leaders at the 2015 Summit continued misleading citizens, in the process breaching Goal 16 before it had started.
Not only does the Millennium Declaration not have a baseline of 1990, but its goals are wider, including on climate change.
The problem is worse, since the actual agenda for 2015 constantly reaffirmed by UN member states was not just the Millennium Declaration but the combined outcomes of major UN summits and conferences.
The concentration on MDGs therefore appears unduly influenced by governments’ and their agents’, propaganda.
Comparisons between SDGs and MDGs can easily mislead, since the environmental and other goals agreed at the Millennium Summit, Johannesburg and other conferences were in fact reaffirmed in later years even though they were not called “MDGs” or “MDG targets”.
The problem is not so much that the MDGs were defective, which is the preferred narrative of the powerful, but that people are now failing to describe properly or act on what governments were already committed to in August 2015. Some of these commitments have clearly not expired.
Since no-one else seems to have compiled the relevant UN resolutions making pledges and setting targets, and the relevant documents by civil servants, I have done this myself:
globalfactcheck.org
ungoals.org
millenniumdeclaration.org
poornews.org
poorscience.org
The widespread abuse of statistics shown by the documents goes far beyond issues of “reliability” or “accuracy”. It seems to render the idea of a “data revolution” in which official claims are still made which misrepresent official research, comically misleading.
Examples of three recurring problems:
1. Statements misrepresenting primary sources such as UN resolutions or World Bank methodology papers.
Example of the latter: BBC More or Less, broadcast on the World Service to poor countries. There are marked differences between what the programme’s claims about “poverty” research methodology and what the relevant official MDG/SDG methodology papers actually say.
http://oxfamblogs.org/fp2p/ricardo-fuentes-wants-you-to-apply-for-his-job-oxfams-head-of-research/#comment-23690
The net effect has been to misrepresent economists’ knowledge about extreme poverty. This is of wider importance than just for the MDG claims, since economists’ claims about “poverty” in different countries have been influential in deciding, or arguing for, policy.
2. Statements about poverty based on methods which people would never apply to themselves or people dear to them.
These include claiming people are out of poverty just by asking them about spending. The assumption is that their needs in a time of demographic, economic and environmental change are the same in every year. This is no more sane than saying “I am richer now because I spend more” without looking at necessary expenditure.
I have proposed an ethics test, or humanity test, pointing out what should be obvious: social science claims which you would not apply to yourself in relevantly similar circumstances cannot be valid. It is a necessary step before making the claims that the person making them has related them, or can relate them, to a reasonable degree to real life, including putting themselves in the position of subjects of the research. If you can’t justify your claims as humane, don’t make them.
https://oxfamblogs.org/fp2p/is-inequality-the-root-cause-of-global-crisis-the-world-banks-lead-research-economist-thinks-so/#comment-2647
https://oxfamblogs.org/fp2p/milanovic-on-inequality-continued-implications-for-politics-alliances-and-migration/#comment-4555
https://oxfamblogs.org/fp2p/we-measure-relative-poverty-in-rich-countries-absolute-poverty-in-poor-ones-what-if-we-combine-them/#comment-2672
https://oxfamblogs.org/fp2p/august-wonkwar-3-martin-ravallion-v-ricardo-fuentes-on-inequality/#comment-4108
https://oxfamblogs.org/fp2p/horizon-2025-the-future-of-aid-and-a-potentially-epic-nerdwar-on-poverty-numbers/#comment-3881
3. Unquestioning trust in official statistics supplied by national governments, national statistics offices or intergovernmental organisations.
Am I sure that none of these have exaggerated progress even on “accurate” and “reliable” statistics on, say, child mortality in the period leading up to 2015? No, especially after seeing some of the odd claims made by UN agencies and others.
The above are in addition to other instances of politicians, civil servants or academics making significant errors about pledges, targets and research on human progress.
Is it possible that the great, big contradiction of the SDGs – the hugely detailed content vs the almost total absence of attention to ‘traction’ – really is a benign conjuring trick?
In other words, the real power of the SDGs is symbolic. They provide a story, a myth, a starting point for collective imagination and action. The huge amount of detail on what the SDGs target is necessary to create and sustain ‘faith’, but is not actually critical in terms of what is done as a result.
In which case, perhaps name-checking the SDGs is not such a risk if we seek to keep and spread the faith. But, focusing on the detail of the actual content (rather than on other, more appropriate and specific standards/rules) in order to hold people and institutions accountable for their investments and actions may be a grave error – a misplaced kind of SDG fundamentalism, or a mistaking of the map for the territory.
Obviously, this would go against the grain of many Civil Society responses which articulate reasons for being concerned about name-checking and a strong call for strong monitoring and accountability mechanisms. But it does seem plausible to me.