Measuring Impact with Communities
Beneficiaries are the experts on their own neighborhoods and their own lives. It makes sense to include them in evaluation.
Different Needs…Same Metrics?
We measure impact because we want to account for benefits and prevent harms. But different stakeholders in the impact value chain have different needs:
❶ INVESTORS and DONORS deciding between opportunities need to compare and evaluate impact.
❷ SOCIAL ENTERPRISES and NONPROFITS trying to create benefits and avoid harms need to manage impact.
❸ BENEFICIARY communities directly affected by these impacts need decision-making power to choose projects that have value in their lives.
While it may seem impossible to select metrics that serve all three stakeholder groups, we at Do Big Good believe that’s not the case.
In our last post, we talked about how leading and lagging indicators can serve the evaluation needs of investors and donors, as well as the management needs of social enterprises and nonprofits. In this post, we’ll explain how to integrate beneficiary communities into the measurement process.
Why Include Beneficiaries?
When beneficiary communities aren’t included in evaluating the projects that affect them directly, it’s not only an ethical problem and a diversity, equity and inclusion (DEI) problem. It’s a information and validity problem as well.
Beneficiaries are Community Experts
First, the information problem. In a recent article in Stanford Social Innovation Review, public health and disability researchers Ebele Mọgọ, Jonathan Lai, and Keiko Shikako-Thomas remind readers that “researchers need to see communities as dynamic, expert citizens, instead of static end-users” by “working closely with them from the beginning, rather than waiting to translate knowledge as a last stage (or forgoing translation altogether).”
Excluding beneficiaries… means cutting out the people who have the deepest experiential knowledge of the systems that investors and philanthropists are trying to ameliorate
In the context of social impact, “working closely from the beginning” and recognizing expertise means engaging beneficiaries as experts on their own communities — the locations where social impact projects will play out and benefits and harms will be felt. It also means recognizing beneficiaries as experts in their own lives, with the ability to choose, prioritize, and amend solutions to problems they are experiencing.
Excluding beneficiaries from the selection and evaluation of projects means leaving information and agency on the table. It means cutting out the people who have the deepest experiential knowledge of the systems that investors and philanthropists are trying to ameliorate.
Beneficiaries are Community Owners
Without community input, impact investments and nonprofit projects lack validity. As the article notes, there’s a “real risk” of generating projects that don’t “address the community’s needs.” If a project isn’t serving the priorities of its beneficiaries, should it even exist? Probably not.
If a project isn’t serving the priorities of its beneficiaries, should it even exist?
It’s reasonable for those directly affected by a project to have some decision-making power over it. Imagine if your child’s school decided to change the curriculum without consulting parents. Imagine if your utility company decided to increase prices to dis-incentivize energy waste and you didn’t know until you saw your inflated bill. You’d be rightly angry in both cases.
Beneficiary communities… have less power to realize their preferences, but investors and donors can change that.
It’s no different for beneficiary communities. They have less power to realize their preferences, but investors and donors can change that. They can add beneficiaries to the decision-making process.
Beneficiaries are Customers
Finally, in the case of a social enterprise beneficiaries are customers. They will purchase the good or serve generated by the investment.
Asking for customer feedback, including in the design process, is a no-brainer. Whole fields, such as customer experience (CX), user experience (UX), and user-centered design are based on that premise.
How can you positively impact the lives of people you have never spoken to, let alone sell them something?
As the following quote from customer experience advisor Annette Franz succinctly puts it, “You can’t transform something you don’t understand.” How can you positively impact the lives of people you have never spoken to, let alone sell them something?
How to Include Beneficiaries?
So, you’re convinced that beneficiary input is valuable. But how, when, and where can you engege these beneficiaries? And how can you do so efficiently?
Fortunately, there’s a pre-existing framework for consulting with beneficiaries. It’s called community engagement.
In his new book, The Case for Everyday Democracy, community facilitator Milenko Matanovič writes, “Unless we ask for input, we should not call it engagement…. If there are no questions, there is no engagement.” At Do Big Good, we define community engagement in much the same way:
► COMMUNITY ENGAGEMENT: Giving intended beneficiaries decision-making power over where investments are made and how they are measured.
…use community engagement to seek beneficiary input in a way that is both meaningful and practical…
So, how does one implement authentic community engagement that gives decision-making power to communities directly affected by an investment or grant?
Example: Selecting Impact Metrics for a Clinic
To illustrate the principles above, we’ve developed the following process for engaging beneficiaries in designing impact measurement. Take what’s useful and leave what isn’t. This isn’t the process. It’s a process.
Preparing to Talk to Beneficiaries
For this example, let’s assume you’re in charge of measurement for an impact investment fund. Your fund wants to invest in a medical facility in a low-income neighborhood that neither you nor anyone in your firm lives in or knows well.
You’ve identified some possible health metrics using the Core Metric Sets on IRIS+.
Let’s assume you’ve already clarified the impact you want the investment or grant to achieve. You’ve identified some possible impact metrics (see examples) using the Core Metric Sets on IRIS+. You’ve run them by your investee, who selected the metric that would serve as useful impact management KPIs (key performance indicators).
Meeting Beneficiaries Where They Are… Literally
Now it’s time to get community input from the future patients of the clinic through a community consultation workshop. According to Matanovič, the best community meetings never last more than two hours. You want to gain information and grant power while respecting the time of your beneficiaries.
Work with your investee to identify a pre-existing community event.
While you don’t know the community well, hopefully the social enterprise you’re investing in does. This is where you begin. Work with your investee to identify a pre-existing community event where potential patients will already be gathered and that is open to the public.
You make it as easy and pleasant to be there as possible, and you empower the community at every phase.
Maybe it’s a block party where a large indoor area is available. Maybe it’s a school event that the public can attend. Maybe it’s at the local library. With the permission of the organizers, you tack your community consultation workshop onto this pre-existing event — either before, after, or during.
You pay a local designer to create a poster promoting the event. you pay a local health nonprofit to post them around the neighborhood a couple of weeks in advance. You have food available, catered by a local restaurant. You have a childcare room to make it easy for parents to attend. You hire a local healthcare activist to emcee. In short, you make it as easy and pleasant to be there as possible, and you empower the community at every phase.
Giving Beneficiaries Context
Once in the room, you briefly explain the details of the clinic project with a slide presentation or images printed out and placed on easels. It’s common to overdo the initial presentation, notes Matanovič, by providing unnecessary details and technical jargon until people’s “eyes glaze over.”
It’s common to overdo this initial presentation by providing unnecessary details… until people’s “eyes glaze over.”
The presentation should last no more than 15 minutes. It should include basic details that the beneficiaries need to understand what is happening, when, and what is possible.
The presentation should discuss the funds allocated and their use as well as the project timeline. It should introduce those running the project and describe their roles. It should include what is open to change and what is fixed, what is possible and what is not.
Questions for Beneficiaries
With this context, the beneficiaries can give their input. The facilitator uses participatory activities like spectrograms, small group brainstorms, post-it documentation, affinity diagrams, and plenary discussion to elicit the following information from attendees:
- To what extent does the project described meet the health challenges in your community? (to assess basic validity)
- What do you want the clinic to achieve in 5 years? (to identify the lagging impact indicator)
- What does progress toward that goal look like to you? (to identify leading impact indicators)
- What concerns do you have about this project? (to identify risks and prevent harms)
- How do you want to hear about the results of this meeting? (to create a mechanism for accountability)
Ensuring Balance of Benefit
In their new book Design Justice, Massachusetts Institute of Technology Associate Professor Sasha Costanza-Chock notes that “Many design approaches that are supposedly more inclusive, participatory, and democratic actually serve an extractive function.”
While most often this extraction is unintentional, sometimes “the goal is explicitly to generate ideas that will be turned into products and sold back to consumers” or, in this case, beneficiaries. This should never be the case with an impact investment or grant.
“Many design approaches that are supposedly more inclusive, participatory, and democratic actually serve an extractive function.”
To guard against this, Costanza-Chock suggests convenors take into account all the benefits emerging from a participatory design session. These include “products, patents, processes, credit, visibility, fame,” and, in this case, the impact of the ultimate project.
Consider how these benefits can be distributed back to the community. Paying local businesses to create the event and centering local community-based organizations to emcee is part of this work. Are there also students in the community who could be paid to assist and then use the experience to get further professional opportunities? What other ways are there to distribute the benefits of participatory design?
Translating Metrics into Accessible Language
During the course of the meeting, we also ask whether the metrics we’ve already identified actually measure what beneficiaries care about. This means using accessible language, i.e., expressing “% change in median client spending on healthcare” as “money spent going to the doctor,” or “% referred patients” as “whether a friend told them about us.”
This means using accessible language, expressing “% change in median client spending on healthcare” as “money spent going to the doctor.”
For example, the facilitator might say, “I heard Norma say she cares about doctors being cheaper. We’re thinking of asking our patients if they spend less money going to our doctors than they did before the clinic opened. How does that sound?”
The evening ends with a clear commitment regarding next steps for accessible information sharing that matches community requests expressed during the meeting.
Self-Check for Authentic Engagement
One self-check for authentic community engagement is whether any metrics actually changed. There should be at least one change in one metric, otherwise input wasn’t meaningfully collected.
There should be at least one change in one metric or input wasn’t meaningfully collected.
There may even be one entirely new metric emerging from the meeting based on community priorities. This is like a ⭐ community engagement gold star ⭐. It’s proof that the community’s decisions about what mattered to them will shape how the investment will be measured.
…one entirely new metric based on community priorities.
Though that additional metric most likely won’t be created at the meeting, you’ll have enough information to do so once you get back to the office. Then, you’ll publish the metrics on your firm and the clinic’s website to lock in accountability.
Impact Measurement for — and With — Everybody
The same metrics can serve all three groups of impact measurement stakeholders. By consulting along the impact value chain, from investor to beneficiary, everybody wins:
❶ INVESTORS and DONORS have confidence that the project has real value to beneficiaries.
❷ SOCIAL ENTERPRISES and NONPROFITS know the metrics they’ll be asked to report are measuring what their customers really want.
❸ BENEFICIARIES have power over investments in their neighborhoods. Investments are no longer only for, but also designed with.
In a recent presentation for Harvard Business Review’s Idea Lab, Catherine Dun Rappaport of BlueHub Capital shared the paradigm, “Nothing about us without us is for us.” This means that without the inclusion of communities in investment decisions, their realization of benefit is in doubt. This process works toward that goal of benefit through inclusion.
These investments are no longer only for, but also designed with.
At Do Big Good, we are proud to help people from different points on the impact value chain come together. We help diverse stakeholders solve wicked problems by centering the beneficiaries and communities directly affected by impact investments and grants. This is the path to social transformation and impact measurement that serves everybody.
by Mary Joyce, founder and principal of Do Big Good.
Thanks to Kathryn Shroyer, Do Big Good’s Sr. Design Advisor, for her help designing the beneficiary consultation process used in the clinic example and to Jara Dean-Coffey and Milenko Matanovič for their feedback.
