Evaluation of a multi-country human security project

Sensemapping evaluation in a multi-purpose, multi-stakeholder and multi-country (continent) programme.

In 2015 we conducted an evaluation of a 7-country peacebuilding cum WASH / Food security programme, funded by the Dutch Ministry of Foreign Affairs. The joint denominator (in ‘elevator pitch’ mode) for this programme was: ‘Aiming for peace building through sector interventions’.

Sensemapping approach for the Evaluation

Through the Sensemapping approach integration of quantitative (e.g. findings from monitoring by various implementers) and qualitative data (a series of interviews by the evaluators) can easily be established. Sensemapping methodology is the only hybrid method that allows us bringing so many data sources together in an elegant way. Basically, in this approach a series of (topic) interviews is made with (carefully selected, mapped) beneficiaries, based on topic questions (formulated by the evaluation team); at the same time, predefined survey questions are put forward to the same respondents, allowing them to give significance to their statements in the interview, and allowing the researchers to quantify. Input for prompting questions and data sources are a combination of results from the desk study followed by a thorough participative discussion, as well as repeated discussions in ‘forum groups’ during the evaluation process.

In this case the desk study showed that there were huge differences between the partial projects (even in-country), so the challenge was to evaluating each project of this huge programme in its own right, and at the same time trying to answer several over-arching questions, necessary for the peace building department of the implementing organization. All in all, we worked with a team of 15 international and in-country evaluators, supported by another 15 enumerators, collecting stories and surveys.

The strength of Sensemapping in this regard is the possibility to give:
a. A good overview of mapped stakeholders per project and selected in such a way that they would be able to shed light on both local and overarching questions
b. A good overview of important joint answers on overarching questions
c. The participative way in which results are interpreted

The thorough stakeholders’ analysis (mapping) yielded a list of interviewees (tailor-made per project), with the following categories:
• ‘General’ target group: Implementing organisation(s), International NGOs active in the field, local implementing office and partner organisations, State actors involved
• ‘Local’ target group: Local authorities, Local informal leaders and/or traditional leaders, Implementers on the level of projects in the field, Beneficiaries and non- beneficiaries, Different sexes, Youth, etc.

A warning is in place here; although we use the results of the interviews (all in all over 150 stories + surveys) in diagrams, the methodology is first and foremost qualitative: interpretation of the different evaluators is the main way of drawing conclusions in this evaluation. In this case, given the size of the operation (7 different countries with each different – local – goals), the number of interviewees is not high enough, the sample is not representative and too diverse to draw quantitative conclusions. Meanwhile the diagrams do show tendencies in the interviews, that are not easily visible otherwise: that’s the way local evaluators used them, and also the way they were used by us, lead consultants, to show overarching tendencies.

Evaluation: Prompting Question, Survey
The ‘prompting’ question (evoking ‘stories’ from respondents was formulated as follows (for all countries): ‘Can you tell us your story about how the local peace building, livelihood- (like seed distribution, barn building, etc.) or education, water and sanitation interventions contributed to more security in your village / environment or with your (near or e.g. cross-border) neighbours?’
The survey questions reflect the research questions brought back to questions on implementation / beneficiary and non-beneficiary level and also contained ‘local’ questions.

All in all, 30 evaluators and enumerators ‘swarmed out’ over 7 countries and 8 projects answering 10 main evaluation questions, as established in the participative ‘kick-off’ meeting: ‘Aiming for peace building through sector interventions’: Is a combination of sector interventions (e.g. Livelihood, Wash, Education) with Peace Building more effective than the separate interventions alone?

The results of projects in Afghanistan and Ethiopia, as well as South-Sudan are encouraging. Although in Afghanistan no mutual reinforcing of the Peace Building and Water component could be detected (‘the real glue between the two was missing’). In Ethiopia and South Sudan, the livelihood projects indeed contributed to peace, be it by juxtaposition (both projects at the same time in the same place, but with no direct links): the multiplier effect turned out to work when people were involved in different aspects of the project. In Uganda, the water pond, education and adult literacy activities were relevant and helpful, but the human security outcomes are very limited: The Theory of Change in the project proposal (which was geared towards human security) was not brought into practice. To the contrary the outcomes in the Burundi and DRC are quite good: The Theory of Change was well thought trough and brought in practice; changes in context were (several times) taken into account; the sector interventions (mainly peace building and livelihood) reinforced each other in many aspects.

As an example of how Sensemapping can support the drawing of overall conclusions (of very different fruits in the same fruitbowl), here are two diagrams showing answers on survey questions (that were the same in all countries):

grafiek1

Diagram 1: restoration of confidence and trust

General conclusions on main research question
• In all countries, the overwhelming majority of beneficiaries answered the question in the mini-survey ‘Did the intervention (any of the activities that you mentioned above) help to build / restore confidence?’ positively. When asked for in which aspects, the answers are not significantly different between the various countries (also given the limited number of interviews) (Diagram 1).
• This shows in a certain sense also the limitations of the used ‘local peace committee (community) approach’: the projects were not reaching overarching ‘national’ conflict level, as the power conflicts between traditional and formal authorities (partly ethnic inspired - leading to warlike situations e.g. in the DRC) and the national power struggle between political parties.
• On the other hand, stories - interviews reveal that in some cases, specific sector interventions (e.g. water, livelihood, land demarcation) contributed to the prevention of more conflicts, while peace building activities (e.g. mediation skills applied) at the same time were mitigating or solving conflicts.
• In other cases, peace building activities were not even possible without starting with concrete (service) interventions.
• Diagram 2 below shows that in most countries also the contribution of service delivery is recognised by beneficiaries. The question was preceded by the question (yes/no) whether security had improved: all of the respondents answered positively except for one who answered “don’t know”. Except for Afghanistan the differences between countries are minimal (‘too close to call’). The zero response in Afghanistan for local government and livelihood are most probably due to the fact that the project here did not intervene in these sectors, the mediation shows the appreciation for the used ‘shuras’-approach.
• The tendency of restoring confidence on a community level (highest overall score) but with little effect on the relations with traditional and local formal authorities, suggests that the beneficiary communities use other ways of organisation (e.g. the Peace Committees).

grafiek2

 Diagram 2. (Question 16) ‘Has in your perception the security situation improved in your community over the last three years, and if so in which sense?’

Participative forums
The effects of discussions in focus groups, participative forums (where the interviews were discussed locally, per country and internationally) are not integrated in these graphs, for the simple reason that these discussions took place only after the stories and surveys (condensed in these diagrams) were made: of course they do influence the final results of the evaluation to a large extent: here again the Sensemapping approach shows its advantages: not just the evaluators do interpret the results from the different sources (desk study, monitoring data, stories, surveys) but this becomes more and more a collective effort of evaluators together with the different layers of stakeholders: learning takes place during the evaluation.