Measuring Impact

Photo by James Toose on Unsplash

We all want to help people and improve the world around us. We come up with new ideas to make things better and bravely launch them. But how do we know if we’ve succeeded? We must measure the impact of our projects to assess whether we are achieving the desired outcomes. But what exactly does that mean, and how do you do it?

In this blog post, we will explore some tips on how to create and carry out effective impact measurements. We’ll talk about what project impact indicators are, and how impacts can be measured using different types of data.

When we talk about project impact, we’re referring to both positive and negative outcomes. It is as important to know where we are failing as where we are succeeding, as failures bring us the opportunity to adapt and improve. Project outcomes can be measured in terms of changes in knowledge, attitudes, behaviors, skills, or any other desired result.

Project impact assessment is not an easy task. Many of these questions do not have simple “Yes” or “No” answers. Several factors can interfere with the way you will get your results.

This post will cover: Obstacles to Measurement, Benefits, Measurement Planning, Data Types, Data Collection, Data Analysis, and Data Sharing

Obstacles to Project Impact Measurement

Measuring the impact of a project is important to different stakeholders for different reasons: management, volunteers, investors and foundations want to know the effects of the projects they are funding, so that they can give support to those that best further their goals. Project impact information is important to the group carrying out the project, so that they can incorporate feedback and improve the results, be it a program that better serves individuals in need or a tool that better serves its users. Further, information on project impact is important to those who would engage with a given project as users or participants, to help them gauge if engaging with the project will be worthwhile.

One reason so much is written on the topic is that thorough project impact measurement can face a wide variety of obstacles:

Which outcomes are connected directly to the project rather than to other factors?
Is data on the project available immediately?
Is the available data of high or low quality?
Does the project team have the necessary resources available, including time available to gather and evaluate the data?
Does the project team have the expertise to evaluate the data available?

Despite these obstacles, project impact measurement remains crucial both to better understand whether a project is furthering its desired goals and to improve future iterations of the project. These reasons highlight the importance of investing time and resources in project impact measurement.

Benefits of Project Impact Measurement

There are numerous reasons why you might want to measure project impact.

Making changes to the project guided by the impact data can allow the team to reinforce desired outcomes and improve the project’s effectiveness, as well as improve project design and implementation. In addition, project impact measurement helps your team to make informed decisions about resource allocation (funds, team members, their time, computer space and processing time are all examples of resources that a project may need). If a project is shown to be effective, more resources can be added to expand the effects. On the other hand, if a project is not effective, more productive uses can be found for available resources.

Impact data can be incredibly effective in building support for the project among key stakeholders. Projects with stronger or broader impacts are more likely to receive continued and additional support and funding. And being aware of the success of a project can strengthen your team through positive feedback, while noting difficulties in the project can allow for refinements to improve future outcomes.

Showing accountability and transparency allows your team to demonstrate to key stakeholders that the project is achieving its intended results or note difficulties and proposed solutions.

How Will You Measure Project Impact?

We’ve talked about the why, so let’s take a look at one of the most challenging parts of measuring project impact: how.

Preparation is necessary.

Impact measurement can’t begin just any time. You need to outline what you are trying to change in your project plan to know how to measure that change.

You will need to select appropriate indicators to measure the impact of your project. Impact indicators might include such things as knowledge, attitudes, and behaviors. Your team will also need to identify your target population and gather baseline data. The target population should include the group or groups of people that you believe will be affected by the project. Baseline data should include both quantitative and qualitative data collected prior to the beginning of the project. This data will be compared to data collected during and at the close of the project to measure the impact via the chosen indicators.

The last crucial component in your project plan will be establishing a system for collecting impact data.

This should include deciding who will collect data, when data will be collected, what methodologies and resources will be used to collect data, and how data will be stored, protected and maintained.

Make sure your team considers:

What are the project’s goals and objectives?
What data will you collect to measure project outcomes?
How will you collect this data?
Will data be evaluated at multiple points in the development of the project to allow for course correction and/or periodic evaluation?
What qualitative and/or quantitative methods will you use to collect data on project outcomes?
Will the team be attempting to measure both direct and indirect effects of the project?

What Kind of Data Do You Need?

Two main types of data can be used to measure project impact:

Quantitative data: Typically collected through surveys, focus groups or interviews this data can be used to measure changes in project outcomes (e.g., knowledge, attitudes, and behaviors) over time.
Qualitative data: Usually collected through interviews or focus groups, qualitative data can be used to understand the project’s impact on participants’ lives (e.g., how the project has affected their relationships, work, and health).

Both quantitative and qualitative data are important for understanding project impact.

Quantitative data provides a more objective picture of project outcomes, while qualitative data provides a more in-depth understanding of the effects of the project on the lives of participants or users.

Planning to gather and analyze both types of data will allow your team to get a more complete, accurate and insightful picture of the project’s impact.

Collecting Data

Let’s do it!

The next step is to collect data on project outcomes. Here is a brief listing of possible methods for collection.

Surveys

If you wish to collect quantitative data, surveys are often your best choice. Administration of surveys is flexible, with online, mail, and in-person options. They can be used to measure outcomes by noting changes in data relative to your established baseline.

If you are using surveys to collect data, there are a few things to keep in mind:

Make sure the survey is short, easy to understand and simple to complete. If the survey is too long or confusing, people will be less likely to complete it.
Make sure the survey questions are clear and concise. Avoid using jargon or technical terms that people may not be familiar with.
Be sure to pilot the survey before administering it to your target population. This will help you identify any problems with the survey (e.g., unclear questions, confusion about instructions). A diverse group of testers in this pilot survey will allow you to recognize more ways to improve its content.
Make sure clear guidance is provided to participants on how the information they are sharing will be used and how any privacy concerns are being addressed.
Use of a reputable and well-recognized provider can be crucial to participant trust. Carefully consider the tool you are using and its reputation within the group or groups of people you wish to complete the survey.

Interviews

Interviews are typically the best method to collect qualitative data, as they provide an opportunity for participants to share their stories and experiences in their own words. They offer flexibility to the interviewee, as they can be conducted in person, by phone, or via video chat.

As you use interviews to collect data, there are a few things to keep in mind:

Plan ahead prior to conducting interviews. This includes having a list of questions prepared in advance. Sticking to this list of questions prevents unintentional discrimination.
A guide for the interview can also be helpful, outlining the topics you want to cover and the order in which you want to cover them.
Choose interviewees who are representative of your target population. Be careful in considering who your target population is and seek feedback from other team community members.
Ask open-ended questions that encourage participants to share their experiences and feelings.

Focus Groups

Focus groups are another type of qualitative data collection that involves bringing a group of people together to discuss a particular topic. Focus groups can be used to collect data on project outcomes and understand the project’s impact on participants’ lives.

When conducting focus groups, it is important to:

Choose participants who are representative of your target population.
Have a moderator who is skilled in leading discussions and keeping the conversation on track. Make sure this person is also trained to enable the quieter members of the group so that all opinions are represented in the group discussion.
Create a set of discussion guidelines for the focus group, which includes appropriate interaction guidelines, along with an outline of the topics you want to cover and the order in which you want to cover them.
If possible, record the session for later review. This can allow the analysts to note subtle cues that may affect the results of qualitative analysis.
Encourage participants to share their experiences and feelings about the project.
Create a report regarding the session.

Once you have collected data on project outcomes, you will need to analyze that data to understand the project’s impact and plan further actions. This can be done through quantitative or qualitative analysis or a combination of both.

Analysis of Data

Data analysis can seem a bit daunting, but there are a few things you can do to simplify the process:

Look at the big picture. What are the overall trends in the data?
Seek patterns and relationships in the data. Are there themes that appear? Are there competing themes? Do either of the competing themes present in a repeatable fashion?
Know that data analysis will have different steps depending on whether you are using quantitative or qualitative data.

When using quantitative data, you will likely use statistical methods to analyze your data. This analysis can help you to understand the project’s impact by looking at changes in project outcomes over time.

The following three factors must be taken into consideration when choosing an acceptable statistical method:

the goals and purposes of both the study of the project and the project itself
the type and distribution of the data used
whether the observations are paired or independent

All statistical techniques used to compare means are known as parametric techniques. Nonparametric techniques are used to compare measures other than means, such as median, mean ranks, and proportions.

Often project impact measurement benefits from the use of analysis software to evaluate the quantitative data gathered. Remember to take care in choosing this tool. Some points that should be checked include:

Is the software regularly maintained?
When was it last updated?
What are the noted reviews of the software?
Does the software have transparent hosting?
Who is providing the software?
Does the software respect user privacy?
- Does it protect personally identifiable information?
- Does it protect from information being gathered by artificial intelligence?
- Is any such data shared with a third party?
What are the limitations of the software?
- For instance, does it rely on one browser type, such as Chromium, for any automated data pulls?
- Are any other other information gaps are present in the software?
Are there any apparent cultural biases with the software?
What data outputs from the software need to be verified by a human?

When using qualitative data, you will likely use thematic analysis to analyze your data. This process involves looking for patterns and themes in the participants’ stories. This can help you to understand how the project has affected their lives. Thematic analysis focuses on identifying patterns, or themes, within the data.

Steps involved in thematic analysis are:

Familiarization
Coding the data
Highlighting key themes
Re-evaluating the themes
Defining and naming the themes
Creating a report to share your observations.

Don’t forget:

Quantitative analysis looks at changes in project outcomes over time and can be used to measure project impact.
Qualitative analysis looks for patterns and themes in participants’ stories and can be used to understand how the project has affected their lives.

Combining both quantitative and qualitative data helps to create a more complete picture of project impact.

Communicate Your Findings

Your project impact data gathering and analysis lack purpose if they are not shared with others. There are many options for reporting your conclusions. Whatever method you choose, make sure you communicate your findings in a way that is clear and concise. Your audience should be able to understand the findings and see the project’s impact.

When communicating your findings, it is important to:

Present the data in a visually appealing way.
Use charts, graphs, and tables to present quantitative data.
Use quotes and stories to present qualitative data.
Make sure your writing clearly conveys the data.
Focus on the project’s impacts rather than project outcomes.

When running a long-term project, it is important to plan for cycles of evaluation. Your project will change over time as your objectives and goals change. Reevaluating your project impact goals should happen each time your team updates its business, strategic or community goals.

In summary, project impact measurement is the process of understanding how your project has affected the lives of those it was intended to help. It is important to collect data on project outcomes and use both quantitative and qualitative analysis to understand the project’s impact. And remember to share your results with project stakeholders and beneficiaries!

I want to thank Sarah Womer for sharing her deep knowledge of data gathering and analysis.