This chapter underlines the fact that use of evidence, depends on capability, motivation and opportunity. It presents a core skillset for evidence-informed policy-making (EIPM) at individual level, including the capacity for understanding; obtaining; assessing; using; engaging with stakeholders; and applying evidence. The chapter also outlines the need to build capacity for EIPM at the organisational level, where capacity for evidence use can be supported or limited by resources or organisational culture. It also underlines the role of broader environmental capacity where use of evidence use can be affected by the relationship with external organisations and societal attitudes towards evidence use.
Building Capacity for Evidence-Informed Policy-Making
Lessons from Country Experiences
OECD Public Governance Reviews
3. What capacities and skills are needed for EIPM?
Abstract
Increasing the use of evidence depends on capability, motivation and opportunity
Increasing the use of evidence by policy-makers depends on behaviour change, such as using evidence and evaluation to influence policy debates, the resulting policy choices and the practical implementation of those choices (Langer, Tripney and Gough, 2016[1]). This can be conceptualised as components in an interacting system. The ‘COM-B’ model developed by Michie, van Stralen and West (2011[2]) posits that capability, opportunity and motivation interact in order to generate behaviour:
Capability is defined as the individual’s psychological and physical capacity to engage in the activity concerned. It includes having the necessary knowledge and skills.
Motivation is defined as all the processes that energize and direct behaviour, not just goals and conscious decision-making. It includes habitual processes, emotional responding, as well as analytical decision-making.
Opportunity is defined as all the factors that lie outside the individual that make the behaviour possible or prompt it.
Figure 3.1. The Capability Opportunity Motivation -Behaviour model
This framework has been used to characterise interventions designed to improve the design and implementation of interventions to increase the use of evidence by policy-makers (Langer, Tripney and Gough, 2016[1]). By changing capability, motivation or opportunity, separately or in combination, it can lead to creating the desired behavioural change. For example, in some cases the only barrier might be capability, in another a lack of opportunity, while in another, changes to all three might be necessary (Michie, van Stralen and West, 2011[2]). Capability to engage in EIPM includes an individual civil servant’s knowledge of different types of research methods, as well as fundamental skills of statistical and data literacy and the capacity to read and understand analytical products, often in English language. Motivation to engage in EIPM can include factors such as a civil servant’s belief that they have a mandate to use evidence, that the use of evidence will be rewarded and an understanding of how the use of evidence will improve the quality of policy-making and will ultimately make policies more trustworthy. The opportunity to engage in EIPM includes the strength of the connections between the policy-making and the research community and civil servant’s institutional access to evidence.
Addressing capacity at multiple levels
Building capacity for EIPM necessities consideration of the changes to capability, motivation and opportunity that will lead to the desired behaviour change. Critically, building capacity for EIPM also requires consideration of multiple levels, including individuals, teams, organisations or institutions and the wider environment. This is because ‘capacity’ is a multidimensional concept spanning different levels from the individual, interpersonal, organisational and environmental, with each of these levels shaping behaviour (Newman, Fisher and Shaxson, 2012[3]; OECD, 2017[4]). Each of these levels is likely to require different forms of capacity building initiatives (Haynes et al., 2018[5]). Furthermore, no level operates in isolation, they interact with each other, reinforcing or weakening each other. A visual presentation of the various levels is presented in Figure 3.2, which illustrates how the capacities for the use of evidence can exist within complex and multi-layered systems.
Figure 3.2. A model for increasing the use of evidence by policy-makers at multiple levels
Source: Stewart, Langer and Erasmus (2018[6]).
Towards a relevant set of skills, knowledge and attitudes for individuals
Individual capacity is the combination of capability, motivation and opportunity which together affect behaviour (Newman, Fisher and Shaxson, 2012[3]). Therefore, the first major step towards building capacity for evidence-informed policy-making is to specify the skills and competencies required for effective use of evidence and evaluation by the civil service.
The mapping of the relevant skills and competencies benefitted from a close cooperation between the OECD and the Joint Research Centre of the European Commission. This included an expert workshop held at the OECD in April 2018, in collaboration with country experts. The outcome of the workshop includes the six skills presented below for Understanding, Obtaining, Interrogating and Assessing, Using and Applying, Engaging with Stakeholders and Evaluating (see Figure 3.3). Although grouped in six skills clusters for the purposes of clarity, in practice the use of these competences is interconnected and not bound to a specific policy process or action. A number of these skills are of a crosscutting character and are applied in multiple occasions, such as critical thinking, systems thinking, engaging with stakeholders. These competences also need to be viewed as a collective skill-set for the public service of tomorrow rather than a full list of skills that each public servant needs to master.
Figure 3.3. The Skillset for Evidence-informed Policy-making
Understanding evidence-informed policy-making
Policy-makers with this skill will understand the role of evidence and its place in the policy-making cycle, as well as the challenges and opportunities, which come with the use of evidence. This will be underpinned by knowledge of different research methods and their purpose, as well as the fundamentals of statistical and data literacy (including big data, machine learning and artificial intelligence).
Obtaining evidence
Policy-makers1 with this skill will be able to gather existing evidence in their own policy area and know who to turn to as sources of evidence synthesis. They will be able to recognise where there are evidence gaps and commission high quality evidence to fill these gaps using a range of project management techniques.
Interrogating and assessing evidence
Policy-makers with this skill will make use of a set of holistic, systemic and critical thinking tools. They will be able to assess the provenance, reliability and appropriateness of evidence. They will have an ability to interrogate evidence by critically assessing its quality and context, using a range of techniques to challenge assumptions and biases.
Using and applying evidence in the policy-making
Policy-makers with this skill will understand their own policy context and recognise possible uses of evidence in the policy cycle. They will be proficient in knowledge management and understand the role of innovation, with an ability to assess and manage risks and challenges. They will be familiar with and know when to use innovative techniques like behavioural insights, design thinking, policy labs and foresight to support policy design and implementation.
Engaging with stakeholders in evidence-informed policy-making
Policy-makers with this skill will have strong engagement and communication skills, including the ability to create effective evidence-based messages for different types of audiences and to engage and inspire a variety of stakeholders. They will be able to manage and facilitate evidence-informed debate with policy-makers and citizens and maintain collaboration with the evidence community. They will have a good grasp of co-creation, co-production and participatory methodologies.
Evaluating the success of evidence-informed policy-making
Policy-makers with this skill will understand different evaluation approaches and tools, and know-how to use comparative examples to inform EIPM. They will understand that evaluation should be built in the policy cycle and should serve to inform and improve EIPM. They will know and use qualitative and quantitative indicators of successful evidence use.
Building capacity for EIPM uptake at the organisational level
Organisational capacity encompasses factors, which can either support or impede the use of evidence within organisations. This can include tangible factors such as well-maintained computer facilities, adequately resourced libraries and robust knowledge management processes. The dissemination and translation of the evidence cannot take place if such resources are not available or cannot be accessed in time.
Developing an evidence strategy can be an important way of ensuring institutional memory, which prevents organizational knowledge from vanishing altogether with the churn of staff. This requires a proper strategy for knowledge management within the civil service. Evidence strategies should set the strategic direction for how evidence will be generated and used, the learning and development needed, and consideration of what capacity building might need the recruitment or contracting of certain specialist skills (such as data science) The long-term institutionalisation of EIPM can be facilitated by the machinery of government, such as a strategy unit or another entity at the centre of government with a clear responsibility and mandate over EIPM.
Organisational capacity also includes less tangible factors such as the political context and organisational culture, which can also impact the demand for evidence (Newman, Fisher and Shaxson, 2012[3]). ‘Culture’ refers to the norms, values, and basic assumptions of a given organization (Damschroder et al., 2009[7]). One explanation for why so many change initiatives fail is that they fail to tackle these less tangible elements of an organisation’s capacity. Organisational capacity may also entail the capacity to access a full range of evidence and evaluation, overcoming some structural biases. For example, the contribution of behavioural sciences and social sciences can also be very important in very specific or technical fields, while ministries competent in those areas may tend to rely primarily on very technical or engineering capacity, overlooking the need for broader and more holistic approaches.
An organisation’s technical capacity, climate and culture collectively affect employee performance, including the adoption of innovative and evidence-based practices (Makkar et al., 2015[8]; Oliver et al., 2014[9]). Furthermore, policy-makers’ use of evidence can be improved if their organisations have a receptive attitude and culture towards evidence and use and invest in resources that support research use. These organisational factors can be understood with reference to the COM-B framework, in that organisational factors provide the incentives, which motivate the individual to use evidence (or not). For example, in a civil service, which explicitly includes the use of evidence in its competency framework, this may provide an incentive, which motivates civil servants to use evidence in their policy development. Organisational factors also enhance or constrain opportunities for individuals to use evidence. For example, a civil service, which has established systems for civil servants to be able to access research (such of research portals and journal access) and to come into contact with members of the research community, has increased opportunities for civil servants to use evidence compared to a civil service in which this activities are absent or difficult to access.
The role of wider environmental capacity
The wider environment beyond organisational boundaries also affects the demand for evidence. In the context of national governments, the wider capacity can refer to the extent to which ministries and departments are networked with other external organisations who can support evidence use, as organisations that support and promote external boundary-spanning roles of their employees are more likely to implement innovative practices quickly (Damschroder et al., 2009[7]) (Greenhalgh et al., 2004[10]).
Wider capacity is also related to government strategies to spread interventions, including policies and regulations, and also recommendations and guidelines. Decisions about how to gather, analyse and interpret evidence will also be shaped by the internal dynamics of individual government departments, as well as the wider bureaucratic and political pressures (Shaxson, 2019[11]). This includes civil service reform programmes, organisational cultures and internal structures and processes that impact upon how individuals and teams work with each other.
Cultural and attitudinal factors in the wider society also affect the extent to which evidence gets used in policy-making. Societal attitudes towards policy-making, and what and who should contribute to it, can also impact the use of evidence in policy-making (Newman, Fisher and Shaxson, 2012[3]). This is also connected to ideas about political accountability, the extent to which elected officials are held accountable by state or civil society organisations, including the media, for the quality of their policy-making (Newman, Fisher and Shaxson, 2012[3]). The extent to which there is a culture of inquiry and how this is developed through institutions such as higher education also determines the extent to which evidence is seen to be an important input to the policy process (Newman, Fisher and Shaxson, 2012[3]).
References
[7] Damschroder, L. et al. (2009), “Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science”, Implementation Science, Vol. 4/1, p. 50, http://dx.doi.org/10.1186/1748-5908-4-50.
[10] Greenhalgh, T. et al. (2004), “Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations”, The Milbank Quarterly, Vol. 82/4, pp. 581-629, http://dx.doi.org/10.1111/j.0887-378X.2004.00325.x.
[5] Haynes, A. et al. (2018), “What can we learn from interventions that aim to increase policy-makers’ capacity to use research? A realist scoping review”, Health Research Policy and Systems, Vol. 16/1, p. 31, http://dx.doi.org/10.1186/s12961-018-0277-1.
[1] Langer, L., J. Tripney and D. Gough (2016), The science of using science: researching the use of Research evidence in decision-making..
[8] Makkar, S. et al. (2015), “The development of ORACLe: a measure of an organisation’s capacity to engage in evidence-informed health policy”, Health Research Policy and Systems, Vol. 14/1, p. 4, http://dx.doi.org/10.1186/s12961-015-0069-9.
[2] Michie, S., M. van Stralen and R. West (2011), “The behaviour change wheel: A new method for characterising and designing behaviour change interventions”, Implementation Science, Vol. 6/1, p. 42, http://dx.doi.org/10.1186/1748-5908-6-42.
[3] Newman, K., C. Fisher and L. Shaxson (2012), “Stimulating Demand for Research Evidence: What Role for Capacity-building?”, IDS Bulletin, Vol. 43/5, pp. 17-24, http://dx.doi.org/10.1111/j.1759-5436.2012.00358.x.
[4] OECD (2017), Skills for a High Performing Civil Service, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264280724-en.
[9] Oliver, K. et al. (2014), “A systematic review of barriers to and facilitators of the use of evidence by policymakers”, BMC Health Services Research, Vol. 14/1, http://dx.doi.org/10.1186/1472-6963-14-2.
[11] Shaxson, L. (2019), “Uncovering the practices of evidence-informed policy-making”, Public Money & Management, Vol. 39/1, pp. 46-55, http://dx.doi.org/10.1080/09540962.2019.1537705.
[6] Stewart, R., L. Langer and Y. Erasmus (2018), “An integrated model for increasing the use of evidence by decision-makers for improved development”, Development Southern Africa, pp. 1-16, http://dx.doi.org/10.1080/0376835X.2018.1543579.
Note
← 1. Although this report recognises the importance of elected officials as consumers of evidence, ‘Policy- makers’ refers primarily to senior members of the civil service responsible for the substantive tasks of policy design and implementation.