The purpose of this activity is to help you make sense of data collected by transforming it into findings that can help you assess whether your solution is leading to the outcomes and impact intended.

Analysis and reporting is a continuous process of interrogation and discovery. As such, this guidance should be viewed alongside the module on feedback and review mechanisms. You need to be able to understand what the data collected is telling you, compare findings and draw conclusions so that you can answer your research questions and meet your learning objectives. Analysis and reporting generally requires several steps:

• Organisation and interrogation of data: To establish strong foundations for analysis.
• Initial analysis: To detect patterns, themes, and relationships in the information.
• Attribution of cause and effect: To test hypotheses.
• Translation of findings into insights: To help draw conclusions and build recommendations.
• Report writing and dissemination: To share findings and encourage uptake.

Whether or not your Research Design makes use of quantitative or qualitative methods, or if it is carried out for the purposes of showing comparative improvement or demonstrated impact, or assessing performance and functionality, all analysis and reporting activities should consider these steps. The following guidance is intended to help you navigate these steps.

Typically, raw data is not very useful. Before you can begin any type of analysis, you will need to organise your data into a logical format that can be easily understood. All analysis should begin with this step, whether or not you are using qualitative or quantitative techniques, or whether you are working with primary or secondary data.

Systematically documenting (eg, transcribing interviews, documenting field notes), archiving (eg, labelling, storing), cleaning (eg, checking for errors), and organising (eg, labelling, sorting) research data are all key to making it as usable and valid as possible, and making it accessible for analysis.

It is at this point that further information from interviews and field notes can be coded (ie adding category labels and references to subject matter) and numerical data can be sorted, transformed (eg clustered, paired, re-ordered) and visualised in order to identify trends, patterns, themes and relationships that would otherwise be unclear.

Once your project data is organised and transformed into useful, usable formats, you will also want to interrogate the data to ensure that it is of the highest quality (determined by your learning objectives and evidentiary requirements). To ensure that your data is valid, reliable and accurate, you can use a variety of techniques, including:

• Substantiation: Asking knowledgeable, trusted and objective informants as to whether the data seems reliable.
• Triangulation: Checking your data against other research results and data available, or assessing the data against other data that has been collected using different approaches and methods from your own.
• Spot checking: Taking random selections of your data and checking to ensure it is accurate.
• Logic checks: Checking whether your data makes sense from a logical standpoint.

Take care to acknowledge your level of confidence in the data and check for possible outliers (data that is surprising or unexpected or seems contradictory).

Discover patterns, trends and themes

Your next job is to transform your data into useful, meaningful information by going through the raw data to determine what is significant in relation to your research questions and learning objectives. This process begins with identifying any interesting and significant features, themes, patterns, relationships and issues that are emerging. Common questions include:

• What patterns or common themes emerge around specific items in the data?
• How do these patterns (or lack of) help to shed light on the broader study questions?
• Are there any deviations from these patterns? If yes, what factors could explain these atypical responses?
• What interesting stories emerge from the data set, as a whole?
• Do any of the patterns or emergent themes suggest that additional data needs to be collected?

After identifying themes or patterns, try reorganising the data into graphic, table, matrix or textual display to help draw conclusions. Through this process you should be able to identify patterns and relationships observed within groups and across groups, and this will also help to point out outliers and unanticipated results. Regardless of what format you chose, it should be able to help you arrange and think about the data in new ways and assist you in identifying patterns and relationships across themes and/or content.

Attribute Cause and Effect

In the Pilot stage, you are introducing your solution to real-world contexts through a humanitarian intervention. At the very least, therefore, you should aim to understand if your solution has had any impact on the target group (demonstrable impact).

We consider impact to mean the positive or negative changes produced by an intervention – directly or indirectly, intended or unintended. You may also want to try and find out whether or not your innovation offers improvement over current interventions and ways of working (comparative improvement).

Both cases warrant an understanding of cause-and-effect relationships. You are not gathering evidence just to see whether a change of conditions has occurred, but also to understand the role of your intervention in producing this change.

In many cases, however, outcomes and impacts will be caused by a combination of factors. So rather than ‘casual attribution’, it can be more helpful to talk about ‘partial attribution’ and ‘causal contribution’, asking the question: Did the intervention contribute plausibly to the outcomes and impacts that have been observed? (Better Evaluation). The bottom line is that you need to explain the extent to which observed results (outcomes or impacts) have been produced by your intervention. Causality is a function of three things:

• Time order: Does the cause happen before the effect?
• Co-variation: Does the introduction (or increase) of one factor lead to an observable change in the result?
• Refutation of alternative explanations: Is the relationship based on cause, rather than correlation?

We can therefore investigate partial attribution or causal contribution in three ways:

• Check the results are consistent with causal contribution: Is there is a cause-effect relationship (co-variation) between the intervention and the observed impacts?
• Compare results to the counterfactual: Are the results from your intervention different to those that would’ve occurred if you hadn’t run your intervention? This is extremely difficult to do in humanitarian contexts, even when using Randomised Control Trials.
• Investigate possible alternative explanations for the impacts that have been observed: Is it possible the results observed have been caused by other means? There are a number of ways to do this, some need to occur during your data gathering exercise, and some can happen through how you interrogate your data.