How can third sector organisations collect and use impact data?

You are special

Third sector organisations are special. Unlike businesses, they do not exist to make a profit and unlike the public sector, they do not exist because the law requires it. Third sector organisations exist to address problems in society, and to effect positive, meaningful and enduring change in people's lives, which is what impact is.

This is why it is especially important for third sector organisations to be able to measure their impact.

In this article, we’ll explain how you can decide what impact data to collect, how to collect it and how to use it to improve your services. We’ll use a fictional organisation to illustrate this process. 

Theory of Change

When thinking about Impact data it is useful to think about a Theory of Change. This article isn’t about how to develop a Theory of Change. You can read more about that in this article by NPC. 

This is how we, at Data Orchard, think about Theory of Change.

Activities are the steps leading towards the enduring change or impact an organisation wants to see. Outcomes are changes in attitudes, behaviours, skills or knowledge, some of which an organisation can attribute to their interventions. Impact is the long term change that an organisation accepts to contribute to.

Dividing Activities from Outcomes and Impact is the 'line of attribution'.

To demonstrate, we’re going to use a fictional example of a community group to illustrate how to collect and use impact data, using a theory of change.

Introducing Penybryn

Penybryn Community Group runs a programme based at its community centre. The programme works with people from the local area who have been unemployed long term. They come to the community centre for workshops, they undertake work placements with local employers and community groups and they benefit from tailored coaching and support from volunteers.

Outputs and activities

Penbryn Community Group is good at measuring its activities.

Their records indicate how many people start the programme each month and how many of those complete the programme.

The number of people starting each month fluctuates between 10 and 23.

The percentage of those who complete the course fluctuates with an overall average of 83%.

They also track how many workshops, work placements and coaching sessions each participant attends.

The manager of the programme can use this data to monitor the programme and to make sure that they spot problems early and keep participants on track.

None of these measures the impact that the programme is having but measures the activities.

To measure the change experienced by people, or short-term outcomes, the project asks everyone to rate their confidence and mental wellbeing at the start of the programme and at the end. That means they can evidence that people, on average, leave the programme more self-confident and with higher levels of mental wellbeing.

They observe that average confidence increases from 3.7 at the start of the programme to 6.1 at the end of the programme and that average wellbeing increases from 4.1 at the start of the programme to 7.3 at the end of the programme.

Impact

Penybryn Community Group knows what sort of outcomes and impact they are hoping to have through this programme. They want people to move into stable work, earn more money and become more financially resilient.

It is harder to measure the medium and long term outcomes and impact but Penybryn does measure some of them. Six months and twelve months after someone has completed a programme a volunteer contacts them and asks them if they are in employment, how long they have been employed for, and for how many hours per week they are employed.

Based on this the community group can see that around 45% of people who complete the programme sustain at least 16 hours of employment over six months in the first 12 months after they have completed the programme. That’s quite good compared to the DWP's main programmes.

This isn’t completely down to Penybryn’s programme. It’s down to the circumstances of the individual, they might have other support, they might have found a job that suits them better than other jobs in the past, and more jobs might have become available in the local area.

However, it is a good indicator that Penybryn’s programme is helping and that it is doing what they intended.

One of the more powerful things about impact data is the power it gives to nonprofits to learn what is working and what isn’t. When Penybryn starts to dig into this impact data they see something interesting. Though overall 45% of people who complete the programme go on to sustained employment the impact is different for male and female participants.

59% of male participants who complete the programme go on to sustained employment whereas only 26% of female participants who complete the programme go on to sustained employment. Even though similar proportions of men and women complete the programme.

From this, they infer the programme is not having the impact on women the group hoped for.

Penybryn runs some focus groups with previous female participants to understand why it’s had less of an impact on them. They learn that many participants found that the work placement opportunities weren’t in areas they wanted to work in and that women in the programme are much more likely than men to cite caring responsibilities as a significant barrier to accessing employment.

As a result, they develop some theories that they decide to test in future versions of the program including:

• Providing more work on techniques for balancing caring responsibilities and employment within the programme
• Providing access to childcare and other care support after the programme has finished
• Widening the scope of work placements to include types of work identified in their focus groups.

They will be able to see whether these changes improve their impact but they will have to wait until a number of people have gone through the programme and then once 12 months have passed.

Longer term impacts

Penybryn Community Group expects that the programme will have longer term benefits.

In particular, they expect that as more people stay in employment they will gain greater financial independence. This will take longer until they’re able to observe this change because people will need to be in employment for quite a while and possibly progress in their jobs before they gain greater financial independence.

If that happens Penybryn Community Group expects that participants and their families will become more financially resilient. They may be able to avoid going into debt and even set some savings aside.

Measuring these impacts is hard because they occur after much more time has elapsed. Even if Penybryn can measure the impacts it will be impossible to prove that their programme made the difference.

Penybryn Community Group can and should measure these impacts though. They work with the local credit union, the council’s benefits team and housing association to understand how financial resilience in the local area is changing.

They use this data to understand how their existing programme might need to evolve. They can also use it to design new programmes that meet the needs of local people.

What this means for your organisation

Measuring impact is harder to do than measuring inputs and outputs but it is the only way to tell whether your service is having the intended impact.

Even if your service is providing the benefits you expected overall it may not be providing these benefits equally to all members of your community. Considering how different people are experiencing the impacts of your work can be a powerful way to drive improvements in your service.

In order to measure your impact a Theory of Change is a useful and worthwhile process which lays out what impacts you expect to have as a result of your activity, and to articulate the change you're aiming to make in people's lives.

Recap

Measuring impact helps voluntary organisations understand the positive effect their services have. It also helps to highlight where things are not working as expected. It is an important tool in improving services for your service users. 

In order to measure your impact you must have a theory of change and you will need to allocate some resources in gathering impact data. This investment will not only benefit your organisation but also your service users.