This chapter evaluates what is different when policy is approached through a well-being lens. It describes the different ways in which well-being indicators could be used in the different stages of the policy cycle, from identifying priorities for action, to assessing the pros and cons of different strategies to achieve policy goals, to allocate the resources (budgetary, human, political) needed to implement the selected strategy, to monitor interventions in real time as they are implemented, and to assess the results achieved and take decisions on how to change policies in the future. The chapter argues that a broad framework encompassing the most important dimensions of people’s lives, paying attention not just to average outcomes but to how policies affect each of the segments of society, and giving a balanced consideration to well-being today, tomorrow and in other parts of the world, holds the promise of delivering better results and bridging the divide that separates policy-makers and ordinary people today.
Beyond GDP
Chapter 4. Country-experiences with using well-being indicators to steer policies
Abstract
4.1. Introduction
As shown in the Annex of this book, the development of better indicators of economic performance and social progress has played an important role in developing a shared evidence-based understanding of what makes for better lives.1 However, while broadening and improving metrics is needed to better assess countries’ performance and the problems that may lay ahead, they do not, in themselves, lead to a change in the setting and framing of policy In all countries, there is always the risk that the newly developed indicators become “just another report”, rather than leading to a new approach to policy making. Many of the well-being indicators included in the dashboards adopted by countries are already well-established, and they do play a role in orienting policies (e.g. unemployment). However, beyond the fact that, as argued in previous chapters, many such indicators still require improvement to help avoid some of policy mistakes of the past, what is also important is that institutional mechanisms are in place to foster the use of these indicators by governments when taking decisions.
This chapter asks, what is different when policy is approached through a well-being lens?
4.2. Using well-being indicators in policy-making
Box 4.1 describes the different ways in which well-being indicators could be used in the different stages in the policy cycle, from identifying priorities for action, to assessing the pros and cons of different strategies to achieve policy goals, to allocate the resources (budgetary, human, political) needed to implement the selected strategy, to monitor interventions in real time as they are implemented and, finally, to assess the results achieved and take decisions on how to change policies in the future (Figure 4.1). The argument made in this section is that a broad framework – encompassing the most important dimensions of people’s lives, paying attention not just to average outcomes but to how policies affect each of the segments of society, and giving a balanced consideration to well-being today, tomorrow and in other parts of the world – holds the promise of delivering better results and filling the chasm that separates policy-makers and ordinary people today.
Using well-being metrics in a policy context can deliver many advantages. These include:
Providing a more complete picture of people’s conditions in any given jurisdiction (e.g. a country, region or city) and drawing attention to outcomes that matter to people’s lives but are not routinely considered in policy analysis simply because of a lack of suitable metrics. In a sense, this was where the “Beyond GDP” agenda began; as we noted, GDP simply did not capture key aspects of people’s conditions.
Supporting the strategic alignment of outcomes across government departments. Throughout the policy cycle, co-operation and cohesion across government is essential. Government agencies often operate in silos, focusing on the resources and outputs for which they are directly responsible (housing, health, education, employment, etc.) and without reference to the wider impacts of their actions. For example, crime and justice agencies will focus on the direct impacts of their actions in reducing crime and enforcing safety measures, despite the fact that spill-overs from other policy areas and society-wide patterns (such as poverty, housing, or lack of public health and education services) may be large. Policy spill-overs also operate in the other direction, with personal safety as a major determinant of outcomes in other policy areas, e.g. as a driver of education outcomes and social connections. Similar spill-overs occur in all policy areas. By identifying the range of outcomes to be considered by all policies, frameworks for measuring people’s well-being can promote consistency across government and provide a common language for agencies to discuss these impacts. Additionally, a multi-dimensional well-being framework (i.e. a framework that encompasses the range of aspects that matter to people’s lives) can generate positive interactions between government agencies, pushing them to co-ordinate their actions and programmes to achieve some higher level objective and helping them to do so. These frameworks can also assist in clarifying responsibilities across and within different levels of government and different groups of stakeholders, increasing co-ordination among policies.
Highlighting the diversity of people’s experiences through more granular data. In contrast to many aggregate measures which focus on the performance of the economic system as a whole, metrics informing on outcomes at the individual and household level enable us to focus on inequalities, pockets of deprivation and vulnerability, and/or on groups whose outcomes are failing to keep pace with country-wide developments. The notions of “inclusive growth” at the OECD or of “shared prosperity” at the World Bank are attempts to put more focus on assessing how the benefits of economic growth are distributed.
Considering both well-being outcomes today and resources for tomorrow. As we highlighted in the previous chapter, a key limit of GDP is that it doesn’t take sustainability into account, either in terms of whether economic growth is itself sustainable over time or whether growth is being achieved with environmental and social costs that offset part (or most) of its economic benefits. The broad coverage (i.e. economic, environmental and social) of well-being measures is here a key advantage. In addition, most approaches to measuring well-being include forward-looking elements, such as indicators of natural, human, social and economic capital stocks that will support well-being in the future (although, as seen above, much progress still remains in measuring these capitals). This balances out the short-term focus of most policy decisions (on the “here and now”) and enables governments to examine whether progress on well-being today is being achieved at the expense of depleting stocks of resources for future generations (“later”) or in other countries (“elsewhere”).2
Promoting more comprehensive evaluations of the impact of specific policies on people’s lives. Encouraging different government departments to consider the wide range of well-being outcomes and impacts of their programmes has the potential to help policy-makers identify impacts of programmes, and articulate trade-offs and spill-overs more explicitly and transparently. Accountability for results is fundamental to efficient and effective governance. It is the ultimate rationale for evaluating policy interventions ex post, and is an important input into strategic priority setting. Well-being frameworks can form the basis for the accountability procedures of government agencies. In defining the set of desired outcomes expected from policy interventions through a range of indicators relating to people’s well-being, ex post policy evaluation can increase accountability for a wider range of outcomes than previously considered. Agreement on the dimensions and indicators of people’s well-being can also streamline external accountability measures, such those exercised through parliamentary oversight, audit agencies and by civil society, creating a common language and a consensus on the types of benefits expected from various policies and programmes.
Fostering public debate. Regular reporting, monitoring and evaluation using well-being metrics allows a country-wide discussion, based on a common concept of what makes for a good life, between all stakeholders, from politicians to civil society, businesses and ordinary citizens.
4.2.1. Mechanisms for integrating well-being indicators in policy decision-making
Routine reporting of well-being statistics can, in itself, help to highlight issues and inform policy decisions in the agenda-setting phase of the policy cycle. Simply making available data on levels, inequalities and trends in well-being to a wide range of stakeholders (e.g. civil society, politicians, business and the media) can shift opinion, inform debate, and influence priority-setting. For example, availability of credible data on students’ skills, based on standardised assessment of their competencies through the OECD PISA programme, has had a huge impact in shifting the focus of educational policies beyond attendance and graduation to what is actually learned by students in schools.
Nevertheless, to unleash some of the potential benefits of well-being metrics outlined above, it is necessary to go beyond simply making indicators available to wide audiences. For example, integrating well-being metrics in policy formulation and evaluation requires a conscious decision on the part of those performing or commissioning this research and analysis, as well as a demand from decision-makers for a more comprehensive evidence base on which to draw. It also requires an established set of tools, models and techniques recognised across the analyst profession within governments (such as the methods set out in the United Kingdom Treasury’s Green Book: Appraisal and Evaluation in Central Government (HM Treasury, 2018), to evaluate the costs and benefits, monetary and non-monetary, of all government programmes).
Several OECD countries have developed formal and/or structured mechanisms to ensure that well-being or “Beyond GDP” indicators are integrated into their policy processes. These mechanisms can target a specific stage of the policy cycle depicted in Figure 4.1 or encompass several. These specific mechanisms or procedures provide a good point of entry for a “Beyond GDP” analysis, for they allow to ascertain whether they give adequate weight or attention to the various other dimensions of well-being beyond aggregate economic output. For instance, many countries require a cost-benefit analysis, say for energy programmes, with the impacts in the future discounted. Current procedures used in many countries effectively imply that negligible weight is given to the impacts of climate change 40 years from now; in short, the procedures do not ensure sustainability.
Table 4.1 provides an overview of such experiences in 10 countries that have adopted well-being policy frameworks, identifying the leading agency and the stage of the policy cycle addressed by the mechanism put in place (see Exton and Shinwell (2018) which describes the settings in which the policy mechanisms and frameworks were developed in these countries). From these case studies, it is possible to identify some common themes, differences and challenges that arise when implementing well-being frameworks into policy settings. Common themes relate to the measurement framework itself, the process used to reach consensus on its features, and the political context of each country.
Box 4.1. Stages in the policy cycle at which well-being indicators and evidence can be used
One way to characterise the various opportunities for metrics to influence policy decisions is to consider the different stages of the policy cycle (Figure 4.1). Building on the various approaches described in the literature (e.g. Jann et al., 2006; Cairney, 2013), it is possible to distinguish among the following stages:
1. Priority/Agenda setting. Based on a strategic analysis of the current situation, including trends over time, and inequalities, a strategic review of policy goals may lead to identifying areas which require government intervention, followed by prioritisation and agenda setting. This stage typically involves national governments, national planning agencies and/or parliaments, as well as citizens, often acting through civil society, bringing to the attention of policy-makers perceived deficiencies in outcomes and processes.
2. Policy formulation (ex ante). This stage includes the investigation of policy options, the evaluation of their costs, benefits and feasibility, and finally the selection of relevant policy instruments and levers. This stage usually involves the national planning agency, government agencies responsible for designing and delivering the policy interventions, allocation of financial resources among government agencies, and government or parliament decisions on budgeting.
3. Implementation. This phase involves executing programmes and policy interventions by the government agencies responsible for implementation, and providing them with the necessary resources, in accordance with prioritisation and policy formulation.
4. Monitoring. A prerequisite for evaluating policy impacts is that the policy interventions are monitored, both during and after implementation. Monitoring involves taking stock of the inputs used for the policy intervention, the outputs generated and the outcomes observed; a comprehensive evaluation can also benefit from monitoring the counterfactual of the policy intervention. At this stage, both the national planning agency, the government agency implementing the policy, as well as external stakeholders, may be involved to monitor the impacts of policy interventions.
5. Evaluation (ex post). This stage requires assessing the results of the policy intervention in view of their goals, and deciding on either termination or continuation. This stage of policy-making can involve the national planning agency, the various potential stakeholders as well as central auditors’ offices.
Figure 4.1. The policy cycle
Source: Adapted from Exton C. and M. Shinwell (2018), “Policy use of well-being metrics: Describing countries’ experiences”, OECD Statistics and Data Working Paper, forthcoming, OECD Publishing, Paris.
4.3. Selected countries’ experiences with implementing well-being indicators in the policy cycle
As noted earlier, in some countries well-being indicators were not specifically developed with policy-use in mind. In others, however, the process started with a paradigm shift at the policy level, aiming to expand what is considered as important to improve people’s lives. In Ecuador, for example, the notion of buen vivir was embedded in the constitution. Somewhat similarly, in Scotland, well-being indicators were given a central role through their integration into a national performance framework and monitoring procedures. The stages of the policy cycle in which indicators are used differ as well. Well-being indicators are mostly used at the policy formulation stage (e.g. in New Zealand and Ecuador) or at the evaluation stage. In France, Italy and Sweden, indicators are more commonly used at the agenda setting stage, with parliamentary reporting based on these indicators at the start of the budget process (see Table 4.1). These are just selected examples. Other countries, such as Bhutan, Mexico, Colombia, Slovenia or Costa Rica have also developed well-being frameworks and indicators to guide their policies.
Table 4.1. Mechanisms and frameworks for integrating well-being metrics into policy-making in selected countries
Source: Exton C. and M. Shinwell (2018), “Policy use of well-being metrics: Describing countries’ experiences”, OECD Statistics and Data Working Paper, forthcoming, OECD Publishing, Paris.
The number and type of indicators used in policy settings varies significantly across countries. In Sweden, Italy and France, where policy use is mainly through reporting to parliament, the number of indicators is very limited (from six/ten in France, to 12 in Italy and 15 in Sweden) in order to facilitate parliamentary discussion. At the other extreme, in New Zealand and the United Kingdom a very wide variety of indicators are provided by the National Statistical Office and used for cost-benefit analysis or ex post evaluation of a range of policy interventions (in the United Kingdom, through the What Works for Well-being Centre)33 or for ex ante policy design (in New Zealand). Ecuador and Scotland lie between these two extremes, with a defined number of indicators and well-defined monitoring procedures (through the National Development Plan in Ecuador, and a National Performance Framework in Scotland).
The type of high-level outcome indicators also varies across countries depending on their specific use in the policy process. For example, in Italy one of the criteria for selecting indicators for the budget law was the ability to forecast trends in these indicators in the near future (the three years ahead). Because of this criterion, subjective well-being indicators were excluded from the list on account of their limited data collection, concerns about data quality, and limited evidence on how budgetary decisions could impact on their evolution. By contrast, the What Works for Well-Being Centre in the United Kingdom puts a strong emphasis on subjective well-being measures, and has identified a number of policies that should be changed when such a framework (rather than conventional cost benefit analysis) is used.
In some countries, well-being policy frameworks have engaged the Parliament, whether or not they were first initiated by parliament or government. In France, Italy and the Netherlands (as well as, to a lesser extent, in Sweden), the goal is that Parliament should have the evidence base needed to hold government accountable for its decisions, through annual reporting of the impacts of their policies on (selected) well-being indicators. In other cases, a central government agency has taken the lead, as in New Zealand, where the Treasury is in charge of the Living Standard Framework, and Ecuador, where the Buen Vivir Secretariat and the Planning Ministry (SENPLADES) were in the lead.
More generally, some of the initiatives presented in Table 4.1 have benefited from strong leadership, often involving a prominent political figure promoting the concept of well-being. A notable example is Ecuador, where former president Rafael Correa initiated a revision of the constitution to incorporate the concept of Buen Vivir. This was also the case in France (with the introduction of a law by a member of the Senate, Eva Sas), in the United Kingdom (with the launch of the Measuring National Well-Being Programme by former Prime Minister David Cameron) and in Scotland (following the accession to power of the Scottish National Party in 2007).
Most of these initiatives are quite recent, however, implying that adjustments and modifications are to be expected, and that it is too early to draw lessons. A crucial challenge is whether initiatives taken by one government can survive the election of the next government, especially when the latter comes from a different party or coalition. For this reason, ensuring continuity in political engagement with respect to well-being concepts is essential. This in turn may require using extensive forms of public consultations, so as to build a consensus and coalition in support of the framework that could outlive the incumbent government, or embed the framework in a multi-year development plan. Difficulties can also be compounded when the initiatives are strongly associated with a political figure – i.e. can the actions to implement well-being frameworks outlast their instigator? This most likely depends on whether a “Beyond GDP” approach is strongly supported by the public and mainstreamed within the civil service.
The development of well-being evidence-based policies will also require a continued iterative progress in data collection, dissemination, analysis and policy experimentation. Sustaining the use of “Beyond GDP” metrics over time is, therefore, a challenge. What steps are necessary in order to mainstream the use of metrics in policy and assure their longevity? What tools are most effective in widening the interest across stakeholders? Here we highlight one challenge, that of attributing changes in one or more well-being variables to a given set of policies.
Demonstrating causality in a public policy context is always difficult. Prime experimental conditions for establishing cause and effect are extremely rare when trying to improve people’s lives in a fair and balanced way through policy decisions. Collection of the right kinds of data, at the right time, and use of the most appropriate models and tools can help in building the necessary evidence base. Policies targeted at specific outcomes may inadvertently affect several other outcomes, or generate unintended effects. In the case of well-being metrics, additional challenges for policy are the multi-dimensional nature of well-being and, as discussed in the previous chapter, the difficulty in identifying inter-linkages between different well-being metrics. For many well-being indicators that have only recently been introduced in large-scale and high-quality data collections in countries’ national statistics, it will take time to build the time series needed to investigate policy questions of interest, to enable analytical work and forecasting. Routine inclusion of well-being metrics in studies to evaluate policy impacts is needed to build up the evidence base required.
An additional challenge in the use of a broad well-being framework in developing countries is the concern that this agenda may divert attention away from what is often viewed as fundamental, namely economic growth. A government failing to produce economic growth, an increase in GDP, may be tempted to defend its failure by arguing that it was pursuing a broader agenda. This critique needs to be addressed head on. GDP growth may be necessary to provide the resources needed for strengthening well-being, but it is not sufficient; growth that does not benefit most of the population and that is not sustainable, is not “good growth”, and it is only within our broader framework that one can ascertain whether growth is, in fact, of the kind that is leading to enhanced societal well-being. As we noted above, short-term growth that is achieved in ways which lead to more inequality, greater environmental degradation and lower trust, is not sustainable.
4.4. Use of subjective well-being data in policy analysis
Most of the well-being frameworks described in Table 4.1 include measures of people’s subjective well-being as one among other types of indicators covered. In other terms, how people evaluate and experience their life is considered as an important element for describing their own well-being, but not as the single variable encompassing the influence of all others.
Subjective well-being, however, has features that lend it to specific use in policy analysis. In particular, it has proved responsive to a wide range of objective circumstances, such as income, health, social relations and political voice. Because of this feature, analysis of the drivers of people’s subjective well-being (when based on large samples of respondents and covering many aspects of people’s circumstances) can improve policy design and evaluation by highlighting those impacts that are not captured (or only partially so) by standard income or other more objective measures.
Subjective well-being measures are also relevant to determine the non-monetary costs and benefits of projects and policies, making subjective well-being an important part of the design and evaluation of policy impacts.
Some public programmes have collected subjective well-being data among participants to highlight the impact of the policy intervention on programme participants. Such evaluations have often highlighted impacts that would have otherwise been missed (Ludwig et al., 2013). Subjective well-being data are also being used in cost-benefit analysis undertaken by public agencies to value non-monetary costs and benefits (Fujiwara and Campbell, 2011). The approach of the UK Treasury provides an example.
Subjective well-being can also be relevant for policies directed at more-specific outcomes, like health. In their Chapter in the accompanying volume, Krueger and Stone provide evidence, for instance, that subjective well-being is an important “mediator” for improving outcomes in other fields: it predicts people’s morbidity and mortality. Policies that enhance people’s subjective well-being indirectly achieve a multiplicity of other outcomes. It even affects childbearing and fertility decisions, and the self-esteem of cancer patients. As knowledge on the determinants of subjective well-being progresses, this research will provide policy-makers with additional tools for affecting both subjective well-being and those outcomes that are mediated by it. Even without a full understanding of the determinants of trust and of the multiple and complex relationships that determine outcomes in our complex social/economic/environmental system, there are things that can be done to enhance the likelihood of better societal outcomes.
4.5. Conclusions
From the very beginning in our research programme, we argued that better and different data and information inevitably shape the political debate. The politics today is different from what it was when we began our work more than a decade ago. It demands information about aspects of our society that were at the time little studied and understood. Our statistical systems need to reflect the concerns of our society – and to the extent possible anticipate future concerns. At the time we began our work, there was too little focus either on sustainability or on inequality. Some politicians may have hoped that if we just didn’t measure inequality, no one would know the extent to which it was growing. That was naïve and foolish. But now that we know that so little of the fruits of growth over the past decades have gone to the bottom 90%, we can’t ignore it. It is part of the political reality that has to be dealt with.
The very progress that has been achieved on measuring well-being following the release in 2009 of the Commission report has contributed to a change in the question that we face, from “how to develop credible metrics of people’s lives?” to “how to use these metrics in the policy process once you have developed them?”. This is a new and more difficult question, but also one that many governments are starting to confront in their day-to-day operations. It is our hope that these applications be pursued and extended to other countries in the future, to allow us to identify best practices of what works best in this field.
Notes
← 1. As we complete this report, the discussion in several countries on “fake news” presents a challenge to the concept of evidence-based research. Years of statistical research (both methodological and applied) have provided a sound foundation for judging the quality of empirical research, including ascertaining the extent to which something more than correlation has been established. All research, particularly in social sciences, is tentative and potentially refutable by subsequent studies, and good science provides indicators of the confidence by which a given result is held. Thus, we know few things with absolute certainty, while we know some things with a high degree of confidence. One of the contributors to the current lack of trust in the elite is that many made policy pronouncements that were well beyond those that could be made with a reasonable degree of confidence, e.g. that globalization would lead most, if not all, citizens to be better off in a relatively short period of time. This is a case where economic theory had provided a strong argument to the contrary, and where insufficient efforts were made to reconcile empirical studies based on limited data with these broader theoretical perspectives.
← 2. “Here and now” (current well-being), “later” (capital) and “elsewhere” (transboundary impacts) are the three dimensions drawn from the seminal Brundtland report (United Nations, 1987) underpinning the conceptual categorisation of sustainable development indicators put forward by the Conference of European Statisticians (UNECE, 2014).