Would you recommend this Ebola response to a friend?

Nick van Praag

Nick van Praag

By next Monday decision-makers in Sierra Leone’s National Ebola Response Centre and aid agencies fighting the virus will have Ground Truth’s first round of data on how the general public and front line workers perceive efforts to halt the spread of the disease. Gathering their views is an ambitious exercise but a bigger challenge is getting crisis responders to use it.

Using a mix of SMS surveys of the general public and phone calls to front line staff, the goal is to provide a regular flow of data from a representative sample of citizens as to whether, for example, they trust the health authorities enough to do what they are told, if women and girls have the same access to medical treatment as men, and whether burial teams respond to calls quickly. The survey also asks frontline workers if they feel safe and accepted by at-risk communities and, critically, whether they believe they are making progress in fighting the disease.

To minimize the burden, our methodology asks only a few questions around a set of key perception indicators that we track over time. In Sierra Leone we are collecting and analyzing data from a representative sample of the general public each week and from front line workers every fortnight.

As my colleague Kai Hopkins said in his blog last week, getting the right kind of data is key. In Sierra Leone we are working closely with DFID, which has funded the Ground Truth program there, and agencies like PLAN, Save the Children and Childfund International. Together, we have come up with questions that, in the parlance of the Sierra Leone and British military personnel who are leading the response, are ‘operational’. In other words, questions that yield answers that guide action.

More difficult than drafting good questions and providing robust data is encouraging responders to use the information. There are a range of explanations about why this is hard but they boil down to the need for the right incentives if we are to get busy people dealing with a huge threat to focus on a form of intelligence that, despite the supportive rhetoric from humanitarian high-ups, is not yet fully accepted as central to humanitarian action.

In Sierra Leone a lot will depend on the relevance of the data and the speed with which we analyze it (our goal is 24 hours). Ultimately, however, the utility of our contribution depends on whether the crisis managers take ownership of the evidence and use it to make course corrections when feedback indicates these are necessary.

20140710-Ebola-blog-ERU-Austrian-RC-Main-1

At risk communities: seen and soon heard

By sharing the data with multiple organisations involved in the response as well as the broader public, we hope that transparency and mutual accountability will promote uptake and action. Getting this kind of buy-in and follow-through means rewarding managers and their teams for taking feedback into account rather than expecting them to know it all – or accepting they manage their programs as if they do.

Donors have a major role in pushing this kind of accountability. Agencies do too as they build a cadre of humanitarians focused on measuring and managing their performance against the way people perceive the relevance and outcomes of their work.

In the grip of the Ebola crisis, Sierra Leone is a tough place to test this out. But the gravity of the crisis and DFID’s support in making the data available over the next critical months may provide the traction for a more responsive approach to managing humanitarian programs. I will let you know how it goes.

 

Nick van Praag directs the Ground Truth programme and leads Keystone’s work in the humanitarian space with the support of the Swiss Agency for Development and Cooperation, the IKEA Foundation and the Conrad Hilton Foundation.

Posted in Uncategorized | Tagged , , , , | Leave a comment

Putting citizens at the heart of the fight against Ebola

Kai Hopkins

Kai Hopkins

Freetown, November 2014

I arrived in Freetown last week and saw my first Ebola sufferer yesterday. While Zombies may not spread Ebola, like some initial rumors suggested, sufferers are still referred to as Zombies; they stumble forward slowly, expressionless, in a confused daze.

I was outside Freetown’s Connaught hospital, where the poor man was being led through the bustling streets. Outside the hospital gate crowds gather waiting for news, some cry, some pray, some shout to try and talk to loved ones inside. Round the corner at the morgue, it’s a similar scene with crowds gathered, many in tears. But here no one who is shouting expects a response. Elsewhere on the street, life goes on; people sell their wares and walk on by and ignore the commotion.

In Freetown the signs of the crisis are rather subtle. There are plenty of posters proclaiming that “Ebola is real”, there are buckets of chlorine outside supermarkets and offices, and people take your temperature as you come and go – a reassuringly consistent 36.6° C – but not quite what I expected to see when I left the UK. Freetown, it must be said, is somewhat unique and Ebola has, and continues to have, a massive impact on Sierra Leone and it’s people, especially in rural communities.

20141107_111302_resized

Connaught Hospital, Freetown

The indiscriminate virus is not, however, affecting lives as one might expect. There is growing concern about a ‘second crisis’ as major public health issues like malaria are sidelined and the suspension of free maternal and child health care leaves many vulnerable.

All hands on the Ebola deck produces other health risks in a country where the health system is already poorly resourced and staffed. Even if there are health services for other diseases, either the fear of catching Ebola or being stigmatized as an Ebola sufferer, makes people think twice about using them. There is also a concern as to whether the fear and stigmatization of Ebola is preventing the reporting of new suspected cases. As one NGO manager told me: “we are all interested in the apparent drop in reported cases in Liberia. Where once there were no beds, now centres sit empty. Why?”

Another aspect of the second crisis is the fear of an impending food shortage. Quarantines of Ebola houses, as well as broader restrictions on movement have hit all economic activity, especially around food, with the UN warning that “in some areas, hunger has become an even greater concern than the virus”. This invisible and deadly virus is leaving some very visible scars in a wide variety of places.

Such a far-reaching and devastating disease needs an equally far-reaching and powerful response, and despite being far too late, this is what we are beginning to see. The Sierra Leone government, with support from the international community, is stepping up its game.

There is still a lot to do but we now have management systems and humanitarian agencies in place coordinating the response. Given the complexity of this crisis, there is no room for complacency. The priority now is a regular and reliable source of data on the effectiveness of various aspects of the response. There are some streams of data; standard knowledge, attitudes and practices surveys (KAP), a long list of KPIs and output data, but what Keystone’s Ground Truth programme hopes to add to the data mix is weekly light touch feedback gathered on key perception indicators from both the broader population and direct from frontline staff, who are in a unique vantage point.

20141107_111424_resized

Frontline staff offer valuable insights into the fight against Ebola

The data, presented in a simple dashboard, will be fed direct into the management systems of both the central response committee and all implementing agencies, empowering them to track the response from the perspectives of those living at the centre of the crisis.

Perhaps the most noticeable indication of the crisis to a European visiting Freetown is that no one shakes hands. I walk into meetings and have bizarre, awkward interactions. We all smile and nod politely, firmly holding our arms down trying to resist the natural urge to extend a hand. I never realized how much we rely on a handshake to introduce ourselves and as a sign of working together. But don’t let it fool you; there is real collaboration going on, a real commitment to working together which extends beyond the symbolic gesture of touching hands. The only thing to ensure now is that we extend a hand to those who we are all here to help, and involve them and their views in the way this response is managed.

Kai Hopkins is a Senior Consultant at Keystone Accountability

Posted in Uncategorized | Tagged , , | 1 Comment

Ukraine’s Doestoevsky test

Nick van Praag

Nick van Praag

Does Doestoevsky’s maxim that you can judge a country by how it treats its prisoners hold true for internally displaced people? If the answer is yes, the new Ukrainian government has some catching up to do.

There’s a lot of coverge of the conflict in Ukraine’s eastern oblasts but the media pays relatively little attention to the experiences of the hundreds of thousands of people displaced by the fighting and even less to the government’s efforts to help them.

Frequent violations of the ceasefire continue to drive people from a conflict that has already claimed some 4,000 lives. In Ukraine, the number of IDPs stands at 417,000 but an accurate figure could be twice as high because a central registration system was only set up in mid-October and no single agency has a comprehensive overview of numbers. Russia, meanwhile, says it has taken in an additional half a million refugees. The vast majority of the displaced in both Ukraine and Russia are from the eastern regions of Donetsk and Luhansk.

The scale of the tragedy is modest by historical standards in a part of the world that Timothy Snyder, in his epic history of the region caught between Hitler and Stalin, describes as the blood lands. After suffering on such a scale, it is hard for today’s displaced people to rise above what media analysts call the ‘awareness threshold’.

The lack of government strategy adds to a sense of abandonment as IDPs grapple with challenges ranging from housing, heating, child care and pensions, to longer-term issues like compensation for what they’ve left behind and, for the young, enrolment in universities as regular students. The visitor status they’ve been accorded precludes them from graduating.

Frustrations are compounded for some IDPs who feel their hosts in other parts of Ukraine blame them for failing to stand up to pro-Russians who are fighting for greater autonomy or outright secession of Ukraine’s old industrial heartland.

When you see families with young children living in overcrowded dormitories in what was once a warehouse on the eastern side of the Dnieper River, it is hard to imagine what role they might have played in slowing the country’s disintegration.

Testing times for Ukraine’s displaced

What struck me most in a focus group of IDPs in Kyiv was their desire to have their views taken into account by the authorities as they try to piece their lives back together. This mirrors the aspirations that drove a million people into the streets of Kyiv last winter and ended up with the ouster of Viktor Yanukovych and his government.

With a new president and freshly elected government now in office, the focus is on regularizing the status of the IDPs and providing the most vulnerable with warm clothes, blankets and weatherproof tarps for their houses. In a place where savings and jobs are equally scarce, providing cash to people whose pensions and welfare payments are frozen is another challenge.

Reconciliation, too, is important. Major efforts are needed to overcome tensions between communities who are now forced to live together and share underfunded schools, overcrowded health facilities and, of course, to compete for employment in a country that according to the World Bank is now shrinking at 8% annually.

An initiative that may pay both a practical and a psychic dividend is the government’s plan, using Ground Truth’s methodology, to listen systematically to displaced people across the country, to find out whether they feel accepted where they now live, if they have access to information they need, if they are willing to play their part in improving their lot, and if reforms are making things better.

This kind of frontline intelligence from IDPs is very different from once-in-a-while listening – or heeding the loudest voices. It offers a month by month take on the perceptions of a representative cross-section of the displaced population whose feedback is rigorously analyzed and made sense of. If it is used conscientiously, it will enable the new government to treat the IDPs better. Ultimately, it is their judgement that counts.

 

Nick van Praag directs the Ground Truth programme and leads Keystone’s work in the humanitarian space with the support of the Swiss Agency for Development and Cooperation, the IKEA Foundation and the Conrad Hilton Foundation.

Posted in Uncategorized | Tagged , , , | Leave a comment

Closing the Citizen Feedback Loop

This article was originally posted by the Center for Effective Philanthropy on October 27, 2014.

David Bonbright

David Bonbright

Dennis Whittle

Dennis Whittle

What do people actually want?
Are they getting it?
If not, then what should we do differently?

These are the three questions that should be at the center of any funder’s strategy and operations. Yet they are far from the radar screen of many funders and implementing organizations, both domestic and international. Feedback Labs, with founding members Keystone AccountabilityGlobalGivingDevelopment GatewayAshokaFrontline SMSTwaweza,Ushahidi, and GroundTruth, are committed to helping funders find out and act on the answers to these questions.

Working closely with new members and key actors in the philanthropic ecosystem such as Charity NavigatorLIFTGlobal IntegrityIntegrity ActionCDA, and How Matters, Feedback Labs is trying to create a new culture that recognizes that collecting and responding to feedback from the people we are trying to help is both the right thing to do and the smart thing to do. It is the right thing to do because, just like in a democracy, the people themselves should be sovereign. It is the smart thing to do because numerous cases show that properly designed and executed feedback loops improve outcomes.

The new CEP report out on beneficiary voice provides ground for optimism, and also points to clear challenges and opportunities for the feedback community. The optimism derives from the fact that 95% of nonprofits responding in the survey assert that they collect feedback during service delivery. To collect is one thing, but to do something about what you hear is another. Here the signal is also encouraging since 61% of non-profit organizations say they use feedback to a “great” or “extreme” extent.

But the CEP report is based on self-reporting from the nonprofits, so to get a more complete picture of the extent and quality of feedback practices we looked for corroborative evidence. It is hard to find. The evidence that we did find points more to the challenges in current practice – and also to the opportunity to make rapid progress.

Charity Navigator’s newest rating criteria – which assess charities on their results reporting – provide the first large-scale external review of nonprofit feedback practices. After reviewing 1,250 charities, Charity Navigator has found that less than 7 percent publish beneficiary feedback of any kind, and only a fraction of this 7 percent provide evidence to suggest how representative that feedback may be.

So, given this mixed picture, how far along are we on the road to effective feedback loops for social change? The CEP report confirms that we are making good progress on at least one of the three ingredients required to spur adoption of a new performance management norm like effective feedback loops – awareness. Clearly nonprofits are aware that this is something they would benefit from doing.

The CEP report also tells us something important and actionable about the other two necessary ingredients – incentives and tools – namely, most funders are not yet playing their part.

Only rarely do funders ask, “What do the people you are trying to help actually think about what you are doing?” Participants in the CEP study say that funders rarely provide the resources to find the answer. Nor do funders seem to care whether or not grantees are changing behavior and programs in response to how the ultimate beneficiaries respond.

We do see some grounds for hope that incentives are about to change. Some major actors shaping funding decisions have already thrown down the feedback gauntlet. As noted earlier, Feedback Labs member Charity Navigator is now applying its new “Results Reporting” rating criteria, which include six data points regarding charities feedback practices. The new ratings will be factored into Charity Navigator star ratings from 2016. The World Bank president has decreed that the bank will require robust feedback from beneficiaries on all projects for which there is an identifiable beneficiary. The Hewlett, Ford, Packard, Rita Allen, Kellogg, JPB and LiquidNet for Good foundations have recently come together to create the Fund for Shared Insight to try to catalyze a new feedback culture within the philanthropy sector.

As incentives from rating agencies and funders get phased in, the binding constraint will increasingly be know-how and tools. Even if organizations are aware of the need to use feedback loops and have the incentives to do so, the actual value of feedback efforts will turn on their quality. Advances in technology, combined with tried and true low-tech solutions such as community meetings, are increasingly enabling us to listen and respond effectively to community views at low cost. The US anti-poverty organization, LIFT, for example has developed a rigorous feedback practice that involves asking two questions after every interaction it has with its service recipients. The information collected and subsequent follow through investigations is yielding transformative insight that is unleashing a new wave of creativity in an already path-breaking organization.

Over the past few years, Keystone Accountability has developed the Constituent Voice methodology for social change organizations like LIFT to manage their performance through systematically collecting feedback from those intended to benefit – their primary constituents – and using it to foster open learning dialogue leading to mutual understanding and agreed action for improvement. Until know Keystone has provided advice and written resources to support Constituent Voice practice, but in January 2015 it is launching an online feedback data sharing platform called the Feedback Commons that will enable any organization that wishes to create high quality feedback systems to do so easily and inexpensively. Among other things, the Commons will enable any organization to compare the feedback that it collects with the feedback of other like organizations.

GlobalGiving is combining both incentives and tools on its funding platform, with the introduction of its new Effectiveness Dashboard, which gives extra points to organizations that use information and data to improve their performance. Those points, in turn, determine the organization’s access to funding channels on the GlobalGiving platform – thus creating a virtuous cycle between funding and effectiveness. GlobalGiving makes available a variety of tools to partner organizations including the DIY toolkit and the Feedback Store (developed in collaboration with the World Bank). More tools and integrations are in the offing, including with the Feedback Commons (being developed by Keystone). GlobalGiving’s goal is to amplify the impact of good organizations as it grows rapidly from its current level of $150 million in funding to 10,000 projects in 160 countries.

Challenges remain, but much progress is being made. Given the recent increase in awareness, incentives and tools for effective feedback, we could be on the verge of a democratic revolution in aid and philanthropy. The success of that revolution will depend on the degree to key actors in the philanthropy and aid fields work together to bring about a new culture that always asks: What do people want? Are they getting it? If not, then what?

Dennis Whittle is co-founder of Feedback Labs and GlobalGiving. David Bonbright is co-founder and chief executive of Keystone Accountability.

Posted in Uncategorized | Tagged , , , , | Leave a comment

Confession of a Reluctant Counter

David Bonbright

David Bonbright

I am a reluctant counter. For the first 20 years of my working life I never saw an evaluation that I didn’t spurn. Like most of us consecrated in the struggle for social justice, the energy was in planning and, of course, in direct action. We saw injustice, made a study of our enemies, formed a plan, and then bled for it. Then we did that again and again. We learned lessons along the way, but not once in those years did I come across a logic model, let alone a randomized control trial study. My first real job, back in 1983, was to make grants to support South Africa’s anti-apartheid leaders. In 1994, I cast my vote for Nelson Mandela. Mission accomplished. Right?

blog1

 

A decade later I became convinced that our inability to measure and communicate our results was a major problem. Sophisticated funders were calling, with increasing insistence, for better results reporting. Unsophisticated funders were flocking to inappropriate proxy measures like overhead ratios.

Practitioners did not have the capacity or inclination to measure well. But they are creative and resilient, so they did what was required of them even if it meant – as Laura Quinn argued recently – misrepresenting or even making up the data.

 

 

blog2

 

 

Another decade later, I have to say that the overwhelming majority of social change leaders remain unmoved by “data-driven development”. They may say otherwise, but look to what they do. And don’t make the mistake of looking at techie circles, where data is the lifeblood. Look among the millions of service provider organizations that use the tools – and are admonished to collect and use data. What I see there is measuring as a chore. Measurement sets limits; it doesn’t bust barricades. It is about bureaucracy, not passion. It is stop, not go. At best it is to prove something that they already know to be true from day-to-day experience. It is not something that helps them do things better.

And they are right not to be excited – even as they are right to pretend outwardly that they are with it.

They are right because we have failed them. By we I mean me, and all the other measurement pedlars, pushing our particular flavours of measurement rigor. I mean the randomistas, the quasi-experimentalists, and the mixed methods mavens – the whole lot of evaluators clad in social science costumes. I mean the monetizers with their SROI gizmos. I mean the snake oil salespersons, who offer easy solutions that won’t stand up to scrutiny. And I most definitely mean the funders who ask for evidence, don’t want to pay for it, and rarely use it well when they get it.

We have failed them because we have not produced what they need in order to become hot-blooded measurers. We have not provided the tools and other resources that would allow them to transform their organizational cultures to be nimble, adaptive managers that learn and improve with those they aim to help. We have played a dirge, not a dance.

In my next blog post I will explain where I think we can find the right music. And to tip my cards, I think that until now we have been looking in the wrong place for the solution to the measurement problem. We have been looking in the dry mechanics of measuring and the rules of evidence as written by economists and statisticians. But as the limits of that search have become clear, a growing number of us are looking elsewhere – at the dynamics of relationships and the rules of evidence as written by psychologists, jurists and quantum physicists.

Posted in Uncategorized | Tagged , , | 2 Comments

Accountability to affected people made simple

Nick van Praag

Nick van Praag

Valerie Amos and Jasmine Whitbread are not easily fobbed off and it will be interesting to see how their peers in the humanitarian system respond to their challenge to get real about accountability to affected people (AAP).

In a refreshingly frank message to Inter-Agency Standing Committee principals – the heads of the biggest UN and NGO humanitarian agencies – the U.N.’s Emergency Relief Coordinator Valerie Amos and Save the Children International CEO Jasmine Whitbread voice disappointment with follow-up on the committee’s commitments on AAP. They call on their colleagues to explain the challenges they face in integrating and operationalizing these precepts, and ask how they plan to do better in future. Responses are due by September 30.

Sadly, this clarion call is unlikely to prompt much more than self-reported lists of piecemeal things the agencies say they are already doing or plan to do more of … a participatory design process here, a grievance mechanism there, a questionnaire somewhere else.

Part of the problem is that there are so many plausible avenues to accountability. You can claim you are doing one or more of a range of activities and effectively shut up calls from people like Amos and Whitbread. But there’s more to the disappointing performance of the humanitarian agencies. A crucial factor is that the system is long on accountability standards and commitments – things the agencies should do – and woefully short on means to verify they are doing them.

Despite promising good practice examples and hard evidence that AAP makes a difference, the last few years have shown that exhortations and voluntary compliance are not enough. How then to move things forward?  I am a believer in overcoming complexity with simplicity. In this instance a two-part approach. First, encourage the donors, who alone have the necessary clout, to set the accountability bar high enough to rule out business-as-usual. Second, place the onus on the agencies to respond in a way that truly makes a difference.

It implies moving away from prescriptive standards and instead letting a thousand accountability flowers bloom. The necessary counterpart to this approach is to simply and systematically ask the end-users of aid how things are panning out for them on the ground. They are well placed to know accountability when they experience it and should get a bigger say. The rest of us must step back and enable their voices to play an optimizing role. If the pressures from donors are sufficient and the feedback process transparent enough, the agencies will soon be performing a lot better than they are now.

Valerie Amos and Jasmine Whitbread: disappointed challengers.

One thing is to get the donors on side. The other is for the aid agencies to realize that accountability to affected people not only helps them do their jobs better but frees them of a lot of bureaucratic hassle. At the moment aid staff are weighed down by time-consuming and mostly ineffective reporting requirements. If light touch accountability practices could lessen that burden, which they can, the people at the front lines will become accountability’s best advocates.

Something like this has already happened in the consumer services world. As customer satisfaction surveys became a popular tool with companies, the numbers of questions asked exploded and companies got bogged down in the resulting data. In the late 1990s Fred Reichheld began to search for a single question that would yield all a company needed to know. Today the Net Promoter Score movement is the fastest growing way of surveying consumers, with its signature question: “How likely would you be to recommend this company or product…”

In the consumer services space, market pressures drove adoption of the simple Net Promoter Approach. Companies knew that they needed a better tool. Aid agencies, on the other hand, won’t become complicit simplifiers unless donors insist that the half-hearted compliance we see today is no longer an option. They should set the ball rolling by agreeing to cut back on their appetite for quantitative activity reports if, in return, the agencies can show they are consistently seeking and, this is the key, systematically using credible feedback from affected populations.

Deferring to people best placed to verify performance in the humanitarian space would likely trigger a more useful response than the reports from the IASC agencies that will soon hit the inboxes of Valerie Amos and Jasmine Whitbread.

 

Nick van Praag directs the Ground Truth programme and leads Keystone’s work in the humanitarian space with the support of the Swiss Agency for Development and Cooperation, the IKEA Foundation and the Conrad Hilton Foundation.

Posted in Uncategorized | Tagged , , , , | Leave a comment