Central to any impact measurement cycle is the data collection and analysis phase, which in the case of the social economy will typically follow a multi-stakeholder, participatory approach. The structuring step often starts during the design phase, while data collection may partially overlap with the ongoing analysis. Social economy entities can choose among a wide range of solutions, from mainstream ones that are shared with other private sector actors to tailored ones that have explicitly been developed for them. When selecting a specific solution for data collection and/or analysis, each social economy entity should carefully consider the data and skill needs required, besides the impact areas being covered.
Measure, Manage and Maximise Your Impact
A Guide for the Social Economy
Local Economic and Employment Development (LEED)
2. Collect and analyse data
Abstract
Data collection and analysis can be broadly divided into three steps: structuring the data approach, which often starts during the design phase; data collection, which may partially overlap with the third step; and ongoing data analysis. Concretely, this means selecting indicators and identifying the data sources, then gathering and analysing quantitative or qualitative data (e.g. about changes in individual or group conditions, such as behaviours, skills, knowledge or health) to assess whether the intended impact outlined in the objectives has occurred (and the targets have been met). The complexity of the steps required to measure results will depend on which tool is selected. As an example, conducting a randomised control trial1 requires many more steps and protocols than writing a case study. At the same time, the method selected should depend on the types of impact results or dimensions needing to be measured.
While most available solutions were designed to match the needs of investors and conventional businesses, solutions are now increasingly addressing the situation of social economy entities. Examples range from identifying indicators that express the unique value creation by social economy entities (UNRISD, 2022[1]); integrating qualitative approaches preferred by social economy entities (Beer, Micheli and Besharov, 2022[2]); and building tools specific to the social economy’s needs and audiences like the Outcome StarTM and Mutual Value Measurement (MVM) Framework ©.
Structure the data approach
Selecting indicators based on the needs of the social economy entity
Based on the impact objectives, targets and available data sources, social economy entities can select those indicators that are most relevant to their learning needs. Indicators are the building blocks for measuring and analysing outcomes: they identify the points of change. When selecting indicators, entities should consider how well they meet each of the following criteria: relevance, usability, clarity, feasibility and comparability (Sinha, 2017[3]). Especially for small entities at the beginning of their impact-measurement journey, it is advisable to focus on a small number of indicators.
Social economy entities mostly draw their indicators from international standards, which are strongly influenced by the financial or business sector and poorly adapted to their capacities and needs (OECD, 2021[4]). The most popular standard is the IRIS+ Catalog of Metrics, developed by the Global Impact Investing Network to help impact investors and their investees measure social and environmental performance in a consistent and comparable manner.2 Some of the thematic areas covered by the IRIS+ Catalog – especially financial inclusion, diversity and inclusion, and quality jobs – are relevant to social economy entities. The Institute for Economic Research, for example, used IRIS+ metrics in the model it developed to measure the impact of social enterprises in Slovenia.3 Each indicator comes with guidance for its calculation and different available options, facilitating its implementation by entities with limited access to primary and secondary data sources. Access to the IRIS+ Catalog is free and requires only creating an account; the website provides training on using the guide.
Especially for social economy entities that are new to social impact measurement, using off-the-shelf indicators may be easier overall: harmonised indicators aim to ensure a minimum level of quality, to enable data aggregation and comparison within and across interventions. Still, many social economy entities claim these fail to capture some aspects of their results and are therefore difficult to exploit for learning purposes (Molecke and Pinkse, 2017[5]). This is especially true for impact areas where self-reporting by beneficiaries is necessary to understand whether, and to what degree, change has happened (e.g. in terms of well-being and social inclusion). Moreover, the same indicators may not be appropriate for all types of members, beneficiaries, customers or clients (Sinha, 2017[3]).
Several ongoing efforts within social economy systems are working to identify and develop indicators that can convey the unique value of social economy entities. In Canada, the “Common Foundations” framework emphasises that encouraging a mixture of four minimum flexible, community-driven standards, which are aligned with other standards and approaches, and tailored indicators can help social economy entities get more meaningful information from their impact-measurement efforts (Common Approach, n.d.[6]). Other notable examples at the international level include the United Nations Research Institute for Social Development (UNRISD) indicators for sustainable development (see Box 2.1), OECD work on well-being and inequalities,4 and indicators on the social and environmental impact of culture and sport.5 Importantly, since social economy entities often operate under severe resource constraints, the measures selected should not only correspond to the investors’ perspective, but also support the entity in understanding and improving on its unique attributes. The emerging consensus is that all indicators need to be co-defined with social economy actors; indeed, efforts to involve them are producing greater clarity about the specific indicators that can be used. See Box 2.2 for a framework developed to enable measuring the value created by co-operatives and mutuals. Infographic 2.2 summarises main considerations in choosing the right approach to selecting indicators.
Box 2.1. UNRISD Sustainable Development Performance Indicators for the social economy
UNRISD’s Platform on Sustainable Development Performance Indicators (SDPI) responds to the growing demand for measures that capture genuine progress towards the Sustainable Development Goals (SDGs). Having identified metrics to assess thresholds and targets indicative of sustainable development, the project provides an online platform where both for-profit and social economy entities can assess their impact for free.
Among a broader list of 55 indicators, UNRISD has identified six indicators which are specific to social economy entities and that will help them express their unique transformational value:
Attendance at annual general meetings: this indicator calls for disclosing attendance at the annual general meeting (or an equivalent mechanism for member participation in decision-making) to inform on the effectiveness of social economy entities' participatory governance mechanisms.
Democratic elections (Y/N answer): this indicator assesses social economy entities’ participatory decision-making practices and use of democratic processes to elect persons in managerial, executive and organisational governance roles.
Participatory management (Y/N answer): this indicator calls for the organisation to determine and disclose: the proportion of managers who are selected by their own staff and the specific ways in which staff have actually participated in the selection process (through a formal consultation, participation in the selection committee, etc.).
Stakeholder participation (non-employee): this indicator requires social economy entities to disclose whether formal mechanisms are in place to allow non-employee stakeholders (members, consumers, communities, etc.) to participate in strategic decision-making, underscoring their inclusive decision-making or multistakeholder deliberation practices.
Training of vulnerable groups: this indicator assesses the extent to which social economy entities engage in skill training and employment of vulnerable groups, showcasing their inclusivity.
Work integration: this indicator calls for disclosing the percentage of workers who received skill training through the social economy entity’s work integration programme(s) and subsequently went on to find employment or pursue education in the last two years.
The six indicators were developed to capture the importance of participatory governance and vulnerable groups as primary areas of value creation for social economy entities. As of November 2023, over half of UNRISD’s approximately 600 subscribers were affiliated with social economy entities. Of these, 32 cooperatives and 34 social enterprises have already used the beta version of the platform. For example, the German ethical bank GLS has used the SDPI indicators to help position its sustainability reporting in the global context, aligning its performance objectives for wage range, gender pay gap and water consumption with the context-based approach and specific targets proposed by the SDPI platform.
Source: https://sdpi.unrisd.org; (GLS Bank, 2020[7]).
Box 2.2. Measuring the total value of mutuals and co-operatives with the Mutual Value Measurement Framework (MVM) Framework©
The MVM Framework was developed by the Australian Business Council of Co-operatives and Mutuals to enable measuring the unique and total value created by this type of social economy entity. The framework focuses on six dimensions: commerciality, shaping markets, member relationships, community relationships, ecosystem and reciprocity, and mutual mindset. It uses a shared language to measure value across these dimensions and can in principle be used in any industry, or for any size of organisation.
The “community relationships” dimension refers to building and maintaining strong and sustainable relationships with the broader community, and is measured by indicators such as “engagement with community organisations” and “charitable relationships and support”. All indicators require “proof points” – either quantitative data that can be tracked or qualitative narratives that can be repeated. By incorporating the MVM into the impact measurement cycle, mutuals and co-operatives can identify the positive impact they have on members, customers, the community and the economy over time.
The Geraldton Fisherman’s Co-operative in Australia adopted the MVM to frame and improve its strategy for embedding itself as a positive force in local communities. It identified opportunities to improve the local fishing supply chain and engage with local citizens, such as through a ”blessing of the fleet” and offering “Christmas crays”. The co-operative now regularly measures its performance against the indicators.
Note: There is a fee to become MVM-accredited and another fee to use the tool itself.
Source: (Brolos, n.d.[8]).
Infographic 2.1. Choosing the right approach to selecting indicators
Source: OECD.
Identifying data sources
The data collected and analysed by a social economy entity to measure its impact may originate from a primary or secondary source, and be quantitative or qualitative. Primary data are any information collected that provides a first-hand account of the topic of interest to the entity, for instance through registration forms, questionnaires, surveys, written or oral feedback, employee notes, meeting minutes, books, diaries, statistics or field observations. Secondary data have not been directly collected by the social economy entity, but are relevant to understanding and assessing its impact. They are sourced from existing analysis on the topic, based on datasets and studies produced by public administrations,6 think tanks, academic bodies, published research and reports from other social economy entities. Not all secondary data sources are available free of charge; social economy entities should therefore earmark a budget to access them for whatever time will be necessary.
Accommodations can be made to make it easier for different vulnerable groups to participate in data collection and analysis. Social economy entities working in the area of migrant reception and integration can ask translators to attend individual or collective interviews to facilitate members’ active participation. Some flexibility can be factored in to meet specific beneficiary characteristics. For example, in situations where literacy levels may be low, oral interviews may replace written questionnaires or surveys. Where the survey method may be unsuitable, children may be asked to select images, such as smileys or sad faces, to indicate a positive or negative response, rather than having to answer along numbered scales. To accommodate participants’ various learning, neurodiversity or accessibility needs, staff-reported observations of behavioural changes may be used in lieu of self-reported changes. Finally, cost-reducing incentives could be offered to individuals with economic vulnerabilities (e.g. by allowing them to use online instead of in-person tools, or offering travel vouchers or child-minding services). However, changes in the way data are collected or analysed need to be approached with caution, as they may threaten the comparability of results, lead to conflicts of interest or promote internal bias. To avoid such risks, all decisions related to accommodations should be taken in a transparent manner, with stakeholder consultation.
Collect data
There exist many data-collection tools, stemming from different research areas (social sciences, psychology, public health…), and they are constantly evolving. Some of those presented below (stakeholder interviews, focus groups, case studies, surveys) are more mainstream, in that they can be used in all impact measurement approaches. Others (outcome journals, Outcome Stars) pertain more to social economy characteristics: they were developed by social economy entities, using language that is familiar to social economy workers and methods adapted to the needs of different stakeholders, especially beneficiaries.
Different data-collection tools often require specific capabilities to engage stakeholders successfully. Facilitating interviews and focus groups entails different skills than forecasting, deadweight and financial proxy calculations when performing cost-benefit analysis, or conducting literature reviews to understand the effects of different interventions on particular groups. Although much progress has been made to reduce the expense and accessibility of these measurement tools, employees will still need time and training to familiarise themselves with them, and learn how best to deploy them for particular stakeholder needs. The diverse range of professionals working within social economy entities, along with their associated working habits and background training, can represent an additional challenge to securing buy-in and motivation in the measurement process.
Stakeholder interviews
Interviews engage stakeholders in a conversation about their experiences with and perceptions of a service, product, intervention, activity, situation or organisation. They are primarily a way of asking them open-ended questions that delve into how they experience, feel or think about certain topics (Boyce and Neale, 2006[9]). Semi-structured interviews allow some flexibility in how questions are put to different individuals, which is more conducive to working with vulnerable groups (e.g. those with low literacy levels or neurodiversity). When several individuals are interviewed at the same time, the tool is called “group interview”, “workshop” or “focus group”.7 Social economy entities, for example, may conduct a focus group to collect data on how a group of people with similar demographics or profiles feel or think about a topic.
In the design phase, interviews can help understand what different groups expect of the social economy entity in terms of impact measurement. For example, the entities might want to ask stakeholders whether they are measuring the right outcomes (“Are the objectives identified the right ones for this intervention? Are the outcome measures representative of the type of change we expect to see? What others could we use instead?”). Interviews can also help determine the most appropriate role for individual stakeholders in the measurement cycle (“How much would they like to be involved? At what point in time? What skills can they contribute?”).
Conducting and analysing interviews requires specific skills. The interviewer will help the person being interviewed feel comfortable, usually by paying attention and responding to social cues and reactions. The interviewer must also preserve respondents’ anonymity. In cases where vulnerable groups are being interviewed or vulnerable subjects discussed, specialist training may be necessary.8 Transcribing interviews can also be time-consuming. Finally, analytical techniques for identifying important themes within the responses may require training in qualitative research techniques, to ensure rigour and transparency (Gioia, Corley and Hamilton, 2013[10]).
Surveys
Surveys involve asking respondents a series of questions on a topic of interest. They are conducted in written form (using paper and pen, online forms or mobile applications) or administered by research assistants (in person or over the telephone).9 Like interviews, surveys are a rather flexible tool which, depending on the objectives, may require advanced skills. Box 2.3 outlines the basic considerations when approaching survey design.
Box 2.3. Five considerations for survey design
When selecting the survey approach, social economy entities should consider five elements to ensure that the data collected will be relevant to impact objectives and create enough information for a rigorous analysis, with valid results:
The target audience: Are the total numbers of the stakeholder group populations known? Are they literate, and in what languages? Are those individuals accessible?
Sampling considerations: Are there contact details for the stakeholder group? Can all stakeholders be reached? Are enough stakeholders likely to respond? How can stakeholders be supported to answer the survey?
The questions: Is the language appropriate for the stakeholder group? How long will it take to answer the survey? What sequence of questions is best? Will stakeholders be expected to know the answers, or will answering require learning research?
Bias: How can “social desirability” (the eagerness to “look good” to others) be avoided? How can false answers be avoided and/or detected?
Administrative considerations: How much the survey will cost, what equipment will be needed to run the survey (on paper or online), and how much time and personnel with relevant skills will be needed for analysis.
Source: (Conjointly, n.d.[11]).
Surveys can be a useful way to collect data in cases where a wide range or group of stakeholders must be included. At times, creating several versions of a survey may be necessary to ensure that different stakeholders, for example from different age ranges, can respond. The “UCL Museum Wellbeing Measures Toolkit”, designed to capture evidence about the psychological well-being effects of museum initiatives, developed six different surveys, varying in length and the topics covered, to accommodate different age groups.10 Collecting all the answers can take considerable time, but newer online and mobile application technologies allow collecting and aggregating real-time survey feedback, providing social economy entities with timely and actionable insights. Nevertheless, deciding on the most appropriate survey technique entails considering its accessibility for stakeholders with certain disadvantages, with some groups preferring to respond with paper and pen.
Surveys have the power of translating qualitative changes into quantitative data through statistical treatment. In the absence of a baseline, they allow cross-sectional analysis, comparing different beneficiaries across time, locations or programmes. However, when the whole population cannot be targeted, sampling and stratification may become sensitive, depending on the factors of disadvantage.
Case studies
Case studies can combine quantitative data and qualitative observations to provide a narrative about a unit of interest (e.g. a product, service, process or outcome). They are a more complex data-collection tool which can rely on several individual or group interviews, as well as field observations and secondary data. Box 2.4 gives an example of a social enterprise using a mixture of data from survey responses and stakeholder interviews to create a case study depicting the organisation’s core activities and outcomes. Case studies are often told from the perspectives of different stakeholder groups. They can be structured in different ways, depending on the contributor or audience: they can, for example, illustrate someone’s character (e.g. a beneficiary, employee, partner); explore the reasons behind a change or outcome (i.e. the type of experience, the way the activity unfolded, the nature of relationships); consider a critical incident (i.e. an accident or a strong success moment); or help understand operations or effects (i.e. implementation protocols, standards, and why processes unfolded). They may be written for internal audiences, as a way of learning about ways to improve employee behaviours or processes, or for external audiences, as a form of accountability for the activities undertaken and the outcomes achieved.
Box 2.4. DrDoctor: Using case studies to demonstrate the core social change activities and outcomes
DrDoctor is a social enterprise based in the United Kingdom that works with doctors, caregivers and patients to improve healthcare service delivery through technological advancements.
In one project, DrDoctor helped a trust in Bradford improve patient experience and engagement by upgrading its technology. Working collaboratively with the Bradford Trust stakeholders, DrDoctor helped them identify patient letters as a critical activity that could be improved with a digital upgrade to save on the costs of physical letters and improve stakeholder reactions. DrDoctor helped the trust design and implement an online system for patient letters, with options for patients to receive a physical copy.
The case study about this project presented on the DrDoctor website uses a mix of project data, survey responses and stakeholder interviews to demonstrate its service quality and range of outcomes. For example, the digital offering was first explored through a pilot and then went live after 12 weeks, with 60% of patients signing up for the digital service. Testimonials from staff at the trust talk about the quality of the service exceeding expectations and the number of digital appointment letters sent out in the first year. Overall, the case study allowed DrDoctor to assemble a range of information to demonstrate its social change-related activities and outcomes.
Source: (DrDoctor, n.d.[12]).
Observations
Observation involves paying attention to important factors of stakeholder opinions, habits, behaviours and actions to determine whether change is happening, and what is contributing to that change. Observation can be made by an employee who is actively participating in the activities (participant observation) or by someone who is not involved (non-participant observation). It may be planned and structured, to capture specific details about a person, process or activity (i.e. what is working well and what is not working as expected), or unstructured, to capture general details about a project and its stakeholders. It may be especially useful with groups of stakeholders who have disadvantages that prevent them from engaging with other data-collection tools. Observation is a rather inexpensive approach to data collection that can be conducted with relatively little training or experience, beyond familiarity with the important factors to be observed. Infographic 2.2. provides a summary of these mainstream data collection tools.
Infographic 2.2. Relevance of mainstream data-collection tools for the social economy
Source: OECD.
Outcome Stars
The Outcome Star is a tool that focuses on measuring and managing transformational change within vulnerable groups. It was developed by a social enterprise consultancy to offer a measurement framework tailored to the needs of frontline organisations and their beneficiaries. It achieves this by using language that is familiar and accessible to entities working in specific social areas, such as homelessness, mental health, financial insecurity, empowerment, disaster recovery and community-building. Its design is accessible to people with different learning needs (the “stars” are easy to read and allow visualising the desired change). Finally, the tool is easy to use, as all stars have outcomes tailored to the social problem which staff and beneficiaries discuss and decide upon together. See Box 2.5 for an example of its implementation.
Box 2.5. STŘEP: Implementing the Outcome Star to improve family service outcomes
STŘEP is a non-governmental organisation (NGO) in the Czech Republic that works with families and children at high risk of neglect or abuse. The organisation adopted the “Family Star Plus” version of the Outcome Star alongside other existing risk assessment tools to provide insight into the families’ views about their situation and attitudes towards solutions. Using the information collected with the Outcome Star, STŘEP was able to monitor and act upon its key outcomes for family services, including the number of goals created by clients, goal achievement, number of children removed by court order, length of case management and number of services accessed.
The two-year period during which support workers implemented the Family Star Plus with families produced several outcomes related to service improvement. Individuals’ families created more goals, the goals they created were more precise, and families achieved on average 9.3 more goals than before the introduction of the Star; there were seven fewer court-ordered removals of children and three more children temporarily and voluntarily placed in residential centres at the parents’ request, thanks to their improved insight into their situation and needs; and the length of case management was extended, enabling more personalised goals to be set and supported.
The implementation of the measurement tool was also linked to several benefits for stakeholders as well. The organisation witnessed improved collaboration between families and the NGO, as well as other social service entities. The families were more motivated to engage in a wider set of services and had an overall better chance of not needing future support, as they had greater awareness of their situation and how to improve it.
Note: The improvements reported in this example are based on a two-year quasi-experimental study conducted at STŘEP, using a pre-post intervention design before and after the introduction of the Family Plus Star.
Source: www.outcomesstar.org.uk.
Outcome journals
Outcome journals are a means to collect information on an individual’s experience. Three different types of journals help understand in detail what progress is being made towards the intended changes in the map or where progress is being stalled, and why. Outcome journals focus on gathering qualitative observations from the beneficiaries of the intervention or the other partners (such as groups or organisations) that are also involved in delivering the change. Strategy journals record the activities conducted as part of the delivery and are compiled by the project managers. Finally, internal stakeholders at the social economy entity can create a learning journal, writing down their reflections and observations about what is working well overall, and what is challenging or creating obstacles to progress in an operational sense. See Table 2.1 for an example of an outcome journal template.
Table 2.1. Sample template for an outcome journal
Source: Adapted from www.outcomemapping.ca/resource/example-outcome-journal-template.
Taken together, these tools offer a wide range of options that can be tailored to the skill level and data needs of social economy entities. Yet when it comes to data collection with beneficiaries, who often have factors of disadvantage, there are additional considerations and precautions to be taken when implementing impact measurement, to safeguard and enhance their ability to participate in the data collection and analysis.11 Table 2.2 provides a list of typical factors of disadvantage, how they can be considered in data collection, and potential adaptations.
Table 2.2. Challenges encountered by disadvantaged stakeholders and potential adaptations for data collection
Source : OECD.
Analyse data
Triangulating different sets of data is important to strengthen their validity. Triangulation (or cross-analysis) may take place when the impact-measurement lead uses multiple sources of data collected from one project to determine whether there is evidence of impact, when multiple individuals analyse the evidence independently and come up with the same result, or when multiple methods are used to measure impact and converge on results.12 This process helps consolidate evidence on whether, how, and possibly why impact (including potentially negative or undesired consequences) has occurred; it also helps answer other learning questions, such as relevance and coherence.
A major focus of data analysis in the context of social impact measurement is establishing a causal relation between the activities implemented and the social change achieved. This allows the social economy entity to determine as clearly as possible whether its activities directly affected the targeted population in the intended manner. The entity can establish a causal relation by using data to prove that its activities can be directly attributed to a change or contributed to a change. Analysis that enables attribution claims involves counterfactual impact evaluations. Randomised control trials, for example, have been used on a large scale to understand global poverty mechanisms (e.g. productivity, educational outcomes and vaccine rates),13 but are often beyond the reach of social economy entities. Contribution analysis approaches are more accessible, since they use a combination of qualitative and quantitative data to demonstrate how a specific aspect of the programme contributed to the resulting changes (OECD, 2021[4]).
Contribution analysis is an evaluation approach that explores causal mechanisms and enables causal inference (Mayne, 2012[13]). Rather than attempt to prove that an intervention “caused” an outcome, it focuses on whether certain activities within the intervention could have contributed to the results, while also considering plausible alternative interventions or activities (e.g. by other social economy entities or government programmes) that could also have played a role. Often, this depends on the social economy entity having crafted a theory of change that outlines the assumptions about how an intervention will create change. The evaluation then attempts to infer whether the intervention’s activities can be reliably linked to any noticeable changes among beneficiaries. Several variations of contribution analysis are relevant to social economy entities:
Realist evaluation uses existing theory and empirical evidence, in the form of literature reviews and secondary data, to infer whether an intervention will be effective for people with specific characteristics.14 Rather than attempt to determine how an intervention “causes” a uniform effect on a population, it theorises what internal processes or behaviours at an individual level may be influenced by an intervention, and how this links to an observable change. Thus, realist evaluators pay attention to the context in which the intervention is conducted, the various mechanisms that drive internal change and the outcomes that can be observed. A realist evaluation often relies on capturing data before and after an intervention, after which it focuses on analysing variations in outcomes across the individuals involved in the intervention.
Developmental evaluation is an adaptive and flexible approach to analysis that uses real-time feedback and quick learning to determine the effectiveness of social innovations.15 This approach is best used in complex and emergent contexts, where unique solutions are not available and the effectiveness of different interventions is largely unknown (Patton, 2016[14]; Patton, 2010[15]). To minimise the uncertainty of decisions and actions, developmental evaluation encourages the use of available data, flexible measurement approaches to capture new data, and pattern recognition and relationship-building to respond to emerging understandings of how the initiative is working (or not) to address an issue and meet stakeholder needs.
Qualitative comparative analysis analyses the contextual aspects that are contributing to the effects and desired outcomes of an intervention. This requires theoretical identification of all the contextual features which, through their presence or absence, may contribute to the desired outcome, and analysing the configurations that are leading to an effect on the desired conditions. This type of analysis is especially useful when there is more than one factor contributing to an outcome or there are multiple possible routes to reach an outcome. It can also help reveal the necessary or sufficient conditions for change to occur.
Consider impact valuation
Social economy entities increasingly wish to understand the value created by their activity in monetary terms. This is a potential step that may (or may not) occur towards the end of the data collection and analysis phase (ESSEC Business School, 2021[16]; OECD, 2021[4]). A social economy entity may have several reasons for undertaking a monetisation exercise:
From a management perspective, monetisation makes it possible to overcome a common difficulty, namely, comparing resources and results. Although resources are often expressed as a monetary value, social impacts in general are not. When both variables are expressed in monetary equivalents, decision makers at the social economy entity can better understand whether the operating model is efficient and compare different courses of action, choosing those with a higher potential for social impact. This may ultimately promote a more efficient use of resources and bring the social economy entity closer to achieving its mission.
From a reporting perspective, monetising impacts can enrich the social economy entity’s accounting, offering an integrated view of its financial and non-financial performance.
From a communication perspective, monetisation can offer an argument for fundraising and advocacy towards public or private stakeholders.
As a general rule, social economy entities first need to choose the valuation frameworks they will use for the efficiency analysis, and then the techniques they will apply to monetise the impacts.
Valuation frameworks
Social return on investment (SROI) is one of the most common frameworks for analysing the efficiency of social economy entities.16 Originating in the field of social entrepreneurship and impact investment, SROI calculates the net present value of a monetary unit invested in the organisation (Nicholls, 2017[17]) (Social Value International, 2012[18]). The valuation work is carried out comprehensively, in the sense that all the organisation’s impacts are monetised. In a typical situation, every significantly impacted stakeholder (direct beneficiaries, indirect beneficiaries, employees and volunteers, public stakeholders) is taken into account.
The SROI ratio measures social value by monetising the value of the change created per every monetary unit spent on a project or intervention. Gathering, analysing and communicating social value using the SROI ratio involves adhering to eight principles: 1) involve stakeholders; 2) understand the positive and negative change; 3) value what matters based on stakeholder preferences; 4) include only what is material (relevant; 5) do not overclaim; 6) be transparent about the process and results; 7) verify results, and 8) be responsive to stakeholders. The data gathered with and from stakeholders are used to answer several questions: who is involved (the number of people and the effects on them), at what cost (how much time and money was invested, and what financial value this produced), and with what outputs (number of activities) and outcomes (change experienced by stakeholders).
Calculating a social value ratio requires using proxies to determine aspects of value (e.g. present value, net present value, discounting and sensitivity analysis). Measuring the results of an intervention (i.e. determining how much the change is worth), therefore, requires a social economy entity to determine how much change has happened, and the value of that change. It can do this by using primary data (direct consultations with stakeholders) or secondary data (leveraging previous studies or evaluations to determine the likely change and a proxy for its value, such as using the daily cost to the public purse of housing a homeless person in a shelter as an indication of the daily value of preventing homelessness for an individual). Given the need to undergo training to become a validated practitioner, apply formulas and have access to databases, SROI remains a complex option for social economy entities to measure results. Infographic 2.3. provides an overview of SROI. See Box 2.6 for an example of an SROI calculation.
Infographic 2.3. Understanding SROI
Note: 1When conducting SROI, general assumptions need to be made around impact: deadweight and displacement, attribution and drop off. These variables provide a way of estimating the degree to which the outcome would have happened anyway, and what proportion of the outcome can be isolated as being added by the implemented activities.
Source: OECD.
Box 2.6. SROI ratio calculation at the Coventry UK City of Culture 2021
SROI was one of the tools adopted by the non-profit trust set up to deliver and measure the social impact of Coventry UK City of Culture 2021 on marginalised groups. It was conducted by external evaluators with specific SROI training and applied to four events run by social economy entities, using a combination of qualitative and quantitative data sources and involving more than 10 000 participants. Among the social objectives of the events were increasing civic pride and improving subjective well-being scores.
When primary data were used, the evaluators consulted with stakeholders before the event to identify and rank outcomes, and then afterwards, to identify the change that had occurred and its value. For example, one of the projects commissioned by the trust, in partnership with local organisations (Pirates in the Canal Basin), aimed to improve participation by disabled artists. The evaluators consulted this group through interviews and workshops before and after the event, first to identify the outcomes they wanted to achieve – e.g. in terms of employment skills, access to jobs in the cultural sector and confidence to work – and second, to rank those outcomes and determine their monetary value (i.e. GBP 12 500 [pounds sterling] for employment skills, GBP 10 000 for cultural-sector jobs and GBP 8 000 for work confidence). Those values were then used when calculating the ratio (1 to 4.4).
When primary data were not available due to issues with timing or accessing stakeholders, secondary data were used to determine the baseline. For example, although the “ANIMALS” project involved local school children in creating a community performance, access to these pupils was not possible before the project. Instead, the evaluators referred to the local household survey that contained information about the number of times stakeholder groups engaged in a cultural activity before Coventry hosted the City of Culture, and afterwards. They could then allocate a value for the change, based on what others had previously said they would pay to receive a similar service (e.g. to attend a community performance) and on previous research that had investigated community life.
The use of the primary data sources in stakeholder-oriented SROI allowed the cultural events’ producers to identify the beneficiaries’ motives for participating in the event, which were sometimes different from the targeted outcomes. This helped them discuss the wider value created by their event and understand their beneficiary group more deeply.
Cost-benefit analysis is another useful analytical framework. Originating in the field of public policy evaluation and regularly applied by social economy entities, it calculates a cost-benefit ratio for one or several stakeholders by understanding the indirect economic benefits induced by social and environmental impacts (OECD, 2018[20]). Unlike SROI, the valuation work is more targeted, in that it only monetises some of the organisation’s impacts. Typically, the social economy entity will only consider the stakeholder it is trying to convince (for fundraising, commercial or advocacy purposes) and estimate the tangible costs, avoided costs and revenue induced by its activities in relation to this stakeholder’s economic situation. The recent study measuring the social economy’s contribution to social and territorial cohesion in Spain is a concrete example of cost-benefit analysis (OECD, 2023[21]).
Monetisation techniques
To apply either of these efficiency analysis frameworks, social economy entities can use one or more monetisation techniques based on avoided costs, perceived value, and restoration or renewal cost.
The “avoided cost” approach estimates the monetary value created by a social economy entity by measuring the indirect economic benefits produced for one or more stakeholders. Promoted by social entrepreneurship networks (Ashoka, 2012[22]), this technique is mainly used in the context of interactions between a social economy entity and public stakeholders around activities with the potential to prevent occurrence, perpetuation or aggravation of a social need. Despite its name, it generally takes into account both the avoidance of (public) expenses and the additional revenue generated for the relevant stakeholders. Hence, the economic value created for the public authorities by a person's return to employment will be estimated both in terms of the avoided expenses (e.g. social and unemployment benefits) and the additional tax revenue (taxes on wages and consumption) induced by the evolution of the person’s trajectory. Box 2.7 provides a concrete example.
The “perceived value” approach estimates the economic value created by a social economy entity by relying on the beneficiaries’ perception of its support and impacts. The umbrella term of “perceived value” is particularly prevalent in methodological publications on SROI and designates a variety of techniques, some of which are based on micro-economic utility functions.17 These include contingent valuation (the beneficiary puts a price on the service received and its impacts),18 monetisation based on proxies, transport costs, or even revealed preference19 (UK Government, 2011[23]); (Social Value International, 2012[18]). A common concern when using this approach is finding financial proxies that will provide an acceptable approximate of the monetary value of the good or service provided to beneficiaries. See Box 2.6 on SROI calculation at the Coventry UK City of Culture for a concrete example of perceived value. Section 2 describes available digital tools that list “peer-reviewed” proxies to facilitate and strengthen the valuation work.20
Finally, the “restoration” or “renewal cost” approach is used more specifically in the context of alternative or “triple capital” accounting, implemented by some social economy entities and a growing number of conventional companies. Unlike the two previous approaches, it was mainly developed to integrate an organisation’s negative social and environmental externalities in the financial assessment of its performance. It attributes a monetary value to these negative externalities, corresponding to the costs of renewing the human or environmental capitals degraded by the activity. Methodological resources to support the implementation of this approach, such as the Natural Capital Protocol (Capitals Coalition, n.d.[24]), the CARE-TDL method21 and impact-weighted accounts (Harvard Business School, n.d.[25]), are freely available.
Box 2.7. Avoided cost analysis for a sexual health centre in France
To increase government funding and gain recognition as an efficient partner in HIV prevention, the French NGO AIDES decided to estimate as precisely as possible the indirect economic impacts of its actions on the national health insurance system. It chose to apply the avoided social cost analysis to its sexual health centre, whose mission is to prevent the spread of sexually transmitted diseases.
Building on the available clinical research, impact data collected by the organisation among its beneficiaries, and public data describing the cost of care for people living with HIV, the AIDES study highlighted the centre’s indirect economic impact on France’s health-care system.
Building on the cost-benefit ratio highlighted by this analysis, the NGO could strengthen its advocacy, communication and fundraising efforts.
Notes: MSM = men who have sex with men; HCV: Hepatitis C virus.
Source: (Aides, n.d.[26]), (Avise, 2023[27]).
These different monetisation techniques require a significant set of data and technical skills. In the business setting, monetisation methodologies are still nascent and have been criticised for relying on strong assumptions and ad-hoc parameters that are not supported by economic theory or rooted in scientific best practice (Murtin and Siegerink, 2023[28]). They require 1) quantified data on the activity’s social impact (beneficiaries’ medium and/or long-term trajectories) and attribution (credible counterfactual data); 2) credible data on the public costs associated with the social needs addressed; and 3) above all, access to consensual calculation conventions (duration of the impacts considered, discount rate, assumptions about the costs of renewing human and environmental capital). Infographic 2.4. outlines the pros and cons as well as different needs related to these monetisation techniques.
Infographic 2.4. A comparative look at monetisation techniques
Source: OECD.
The need for data and technical skills increases significantly when several monetisation techniques are used simultaneously to offer a more comprehensive valuation of impacts, for instance in the areas of wage inequality and employee well-being (see Box 2.8).
Box 2.8. Valuing business impacts in the areas of wage inequality and employee well-being
Employment and working conditions at companies have a large impact on employee well-being. The European Quality of Life Survey helps confirm the important externalities exerted by firms on workers’ well-being:
Employed workers have higher well-being relative to inactive people.
Poor working conditions due to (for instance) excessive working hours, tensions with management and high job insecurity have a highly negative impact on workers’ well-being.
Long working hours, job insecurity, full employment and absence of tension with management have a welfare impact equal to 1.5%, 4.5%, 7.4% and 13.9% of household income, respectively.
Consolidating theoretical and empirical frameworks drawn from welfare economics, the OECD has been piloting a new method of monetising employee well-being, which currently covers only five dimensions: wage inequality, being employed, excessive working hours, relationships with management and job security.
The preliminary results from the OECD analysis show a large loss of welfare arising from within-firm wage inequality, as well as a strong impact of working conditions on workers’ well-being. On the aggregate, suppressing the negative externalities of the firm linked to excessive working hours, tensions with management and job insecurity would yield an increase in social welfare equivalent to a 25% increase in household income, representing many years of economic growth. Greater transparency on company wage distributions and working conditions is necessary to apply this valuation technique to real firms.
Source: (Murtin and Siegerink, 2023[28]).
Given these difficulties, social economy entities may need to mobilise external support to perform the monetisation exercise. Infographic 2.5. provides an overview of the data collection and analysis phase of the impact measurement cycle.
Infographic 2.5. Success factors and pitfalls to avoid in data collection and analysis
Source: OECD.
References
[26] Aides (n.d.), Aides, https://www.aides.org/en.
[22] Ashoka (2012), Impact economique de l’Entrepreneuriat Social, https://www.avise.org/ressources/etude-dimpact-de-lentrepreneuriat-social.
[27] Avise (2023), Neuf études de cas sur l’impact de la générosité, https://www.avise.org/ressources/neuf-etudes-de-cas-sur-limpact-de-la-generosite.
[2] Beer, H., P. Micheli and M. Besharov (2022), “Meaning, Mission, and Measurement: How Organizational Performance Measurement Shapes Perceptions of Work as Worthy”, Academy of Management Journal, Vol. 65/6, pp. 1923-1953, https://doi.org/10.5465/amj.2019.0916.
[9] Boyce, C. and P. Neale (2006), Conducting In-Depth Interview: A Guide for Designing and Conducting In-Depth Interviews for Evaluation Input, Pathfinder International.
[8] Brolos (n.d.), Mutual Value, https://brolos.com.au/corporate/mvm/.
[24] Capitals Coalition (n.d.), , https://capitalscoalition.org/capitals-approach/natural-capital-protocol.
[6] Common Approach (n.d.), The Common Foundations of Impact Measurement Version 2.0, https://www.commonapproach.org/wp-content/uploads/2021/10/Common-Foundations_Version-2_EN_031121.pdf.
[11] Conjointly (n.d.), Selecting the Survey Method, https://conjointly.com/kb/selecting-survey-method/#sampling-issues.
[19] Coventry UK City of Culture 2021 (n.d.), Social Value Assessment, https://coventry21evaluation.info/strategy-reports/social-value-assessment/.
[12] DrDoctor (n.d.), Unlocking digital-first healthcare at Bradford Teaching Hospitals, https://www.drdoctor.co.uk/unlocking-digital-bradford?hsCtaTracking=d3c9c1cb-8cd2-412c-90d2-e452e245d6db%7C55624e9e-c960-42d1-b198-6c4a1b2f480d.
[16] ESSEC Business School (2021), Panorama évaluation d’impact social, https://impactinitiative.essec.edu/wp-content/uploads/2021/12/Panorama-2021-1.pdf.
[10] Gioia, D., K. Corley and A. Hamilton (2013), “Seeking Qualitative Rigor in Inductive Research”, Organizational Research Methods, Vol. 16/1, pp. 15-31, https://doi.org/10.1177/1094428112452151.
[7] GLS Bank (2020), Integrated Sustainability Report 2020, https://www.gls.de/media/PDF/Broschueren/GLS_Bank/Nachhaltigkeit/About_NHB_2020_211209.pdf.
[25] Harvard Business School (n.d.), , https://www.hbs.edu/impact-weighted-accounts/Pages/default.aspx.
[13] Mayne, J. (2012), “Contribution analysis: Coming of age?”, Evaluation, Vol. 18/3, pp. 270-280, https://doi.org/10.1177/1356389012451663.
[5] Molecke, G. and J. Pinkse (2017), “Accountability for social impact: A bricolage perspective on impact measurement in social enterprises”, Journal of Business Venturing, Vol. 32/5, pp. 550-568, https://doi.org/10.1016/j.jbusvent.2017.05.003.
[28] Murtin, F. and V. Siegerink (2023), Valuing business impacts in the areas of wage inequality and employee well-being, OECD Papers on Well-being and Inequalities, No. 15, OECD Publishing, https://doi.org/10.1787/740deb2f-en.
[17] Nicholls, J. (2017), “Social return on investment—Development and convergence”, Evaluation and program planning, pp. 127-135.
[33] OECD (2023), “Impact indicators for culture, sports and business events: A guide - Part II”, OECD Local Economic and Employment Development (LEED) Papers, No. 2023/11, OECD Publishing, Paris, https://doi.org/10.1787/e2062a5b-en.
[21] OECD (2023), Policy Guide on Social Impact Measurement for the Social and Solidarity Economy, Local Economic and Employment Development (LEED), OECD Publishing, Paris, https://doi.org/10.1787/270c7194-en.
[29] OECD (2021), Boosting Social Entrepreneurship and Social Enterprise Development in Brandenburg, Germany, https://www.oecd-ilibrary.org/industry-and-services/boosting-social-entrepreneurship-and-social-enterprise-development-in-brandenburg-germany_b2e6825f-en (accessed on 6 October 2021).
[4] OECD (2021), “Social impact measurement for the Social and Solidarity Economy”, OECD Local Economic and Employment Development (LEED) Papers, No. 2021/05, OECD Publishing, Paris, https://doi.org/10.1787/d20a57ac-en.
[20] OECD (2018), Cost-Benefit Analysis and the Environment: Further Developments and Policy Use, OECD Publishing, Paris, https://doi.org/10.1787/9789264085169-en.
[14] Patton, M. (2016), Chapter 1: State of the art and practice of developmental evaluation, https://www.prpassworkshop.org/sites/prpassworkshop.org/files/patton-chapter1.pdf.
[15] Patton, M. (2010), Developmental evaluation: Applying complexity concepts to enhance innovation and use., Guilford press.
[30] Richard, J. (2020), Towards a new ecological and human type of national accounting for developing economies (The CARE/TDL model), BRICS Journal of Economics. 43-59, https://doi.org/10.38050/2712-7508-2020-4.
[3] Sinha, F. (2017), Guidelines on outcomes management for financial service providers, https://sptf.info/images/Guidelines-on-Outcomes-Management-for-FSPs.pdf.
[18] Social Value International (2012), The guide to SROI, https://www.socialvalueint.org/guide-to-sroi.
[23] UK Government (2011), Valuation Techniques for Social CBA, https://assets.publishing.service.gov.uk/media/5a7b9e3be5274a7318b8fd4c/greenbook_valuationtechniques.pdf.
[1] UNRISD (2022), Authentic sustainability assessment a user manual for the sustainable development performance indicators, https://sdpi.unrisd.org/platform/how-to-use-this-tool/.
[31] White, H., S. Sabarwal and T. de Hoop (2014), Randomized Controlled Trials (RCTs): Methodological Briefs - Impact Evaluation No. 7, UNICEF, https://www.unicef-irc.org/KM/IE/impact_7.php.
Notes
← 1. A randomised control trial is an experimental form of impact evaluation in which the population receiving the programme is chosen at random from the eligible population, and a control group is also chosen at random from the same eligible population (White, Sabarwal and de Hoop, 2014[31]). Often regarded as the most rigorous form of impact evaluation, randomised control trials require significant resources and expertise, and are thus not commonly implemented by individual social economy entities (OECD, 2021[4]).
← 4. www.oecd-ilibrary.org/social-issues-migration-health/oecd-papers-on-well-being-and-inequalities_4ca48f7c-en.
← 5. The OECD guide to measuring the impact of culture, sports and business events (OECD, 2023[33]) suggests nine indicators on the social impact of cultural and sports interventions: percentage of target groups reporting increased frequency of participation (in culture, sports, business, etc.); participation in sport; increase in physical activity; active and passive participation in culture; percentage of target groups reporting change in health and well-being; percentage of event participants from underrepresented groups; change in percentage of community residents reporting a sense of local pride; change in percentage of public reporting positive perception of underrepresented groups; percentage of volunteers motivated to volunteer more.
← 6. This is especially interesting for those counties where national satellite accounts have been created to track the activity of social economy entities, or where public observatories and regular monitoring surveys are in place (OECD, 2023[21]).
← 8. https://euaa.europa.eu/training-catalogue/interviewing-vulnerable-persons-0 or https://courses.epigeum.com/online-module/290?course_id=404
← 9. When it is not possible to write or type a response, a survey can be conducted using the interview approach.
← 11. See also NPC’s Centring Lived Experience guide.
← 16. See, for instance, the nationwide study conducted in Germany by the Federal Working Group on Workshops for Disabled in 2015 (OECD, 2021[29]).
← 17. Scientific monetisation frameworks rely on a model of individual preferences, called a “utility function” in economic jargon. A utility function allows deriving the welfare weights of non-monetary dimensions of well-being in order to assess their equivalent income or people’s willingness to pay for them. These weights reflect people’s actual preferences with respect to non-monetary dimensions, relative to income. Any weight attributed to non-monetary outcomes or income inequality therefore reflects the individual and social welfare that is created or destroyed by a change in those outcomes (Murtin and Siegerink, 2023[28]).
← 18. Stated preferences can be collected in terms of willingness to pay (to receive or avoid an outcome) or to accept (as compensation for a loss).
← 19. This technique involves inferring the implicit price consumers place on a good by examining their behaviour in a similar or related market.
← 20. See, for instance, the Social Value Engine platform: https://socialvalueengine.com/platform-features-and-benefits/.
← 21. The Comprehensive Accounting in Respect of Ecology/Triple Depreciation Line (CARE-TDL) model explores the operational modalities of capital conservation by applying a principle of depreciation across all three sources of capital. In so doing, the model is designed to fully integrate the costs (or expense) of maintaining human, financial and environmental capitals in corporate accounting (Richard, 2020[30]; Avise, 2020[32]).