This chapter analyses the processes that support evidence informed policy making in Ireland and the contribution of IGEES resources, including people to these processes. The chapter addresses recruitment and progression as well as learning and development. It also discusses the tools that are supporting EIPM, including the role of frameworks and guidelines and highlights the need for broadening the analytical framework beyond the spending code. It also underlines the role of the departments’ programme of work, and discusses the balance of work conducted internally versus externally.
The Irish Government Economic and Evaluation Service
Using Evidence-Informed Policy Making to Improve Performance
OECD Public Governance Reviews
2. Processes, tools and people
Abstract
This section will analyse the processes that support Evidence Informed Policy Making Ireland and the contribution of IGEES to these processes, as well as the tools that are used to support EIPM and the core IGEES that contribution makes through investing in people.
People
Recruitment and progression
One of the main functions of the IGEES unit in DPER is to coordinate the recruitment process for IGEES so that graduates and experienced economists/evaluators/policy analysts can join analytical resources in Departments. As IGEES is an established brand in Ireland among economics graduates, this has ensured a continuous inflow of quality trained professional staff in economics across government. The recruitment process involves on average 20 graduates per year, with an increase in intake in recent years, for a total number of 160 IGEES staff working across the departments. This is for the AO level, in addition, an open competition is organised for APs, with 18 positions filled in 2018. The successful candidates are comprised of a mix of existing IGEES policy analysts, who are being promoted, and policy analysts recruited externally. In addition, in 2018 IGEES introduced a pilot 3-month internship programme. Following on success of the pilot, the internship programme was rolled out in 2019, with the placement of thirteen interns across nine Departments.
Changes to the recruitment processes have been positive, as they have helped to broaden the expertise of the new recruits. IGEES is now recruiting social scientists as well as economists and despite a tightening labour marked, IGEES continues to attract skilled workers.
Opening up the recruitment to social scientists and others increased the diversity of the skill set of the new recruits. One senior manager noted that bringing a diversity of perspectives on a particular policy issues led to higher quality policy analysis. The Public Appointments Service oversees the recruitment processes for IGEES. A panel of qualified candidates results from the competitions and all those who are successful have the necessary skills for the position of IGEES AOs or APs but from time to time Departments may express a desire for more targeted skills. For example, getting health economists through IGEES recruitment processes is challenging, given the competition with the private sector with the pharmaceutical industry. This is also due to the way that current general IGEES recruitment processes are structured, and the fact that there is no guarantee that as a result of these processes, a specialised economist would work in their field. Another example is the Better Regulation Unit in DBEI, which requires more qualitative than quantitative expertise, and where there is a concern to assign jobs that correspond to people’s expertise and interests. In some countries such as France, the pool is managed centrally, with a common market place, which provides an opportunity to identify opportunities and facilitate lateral moves, with some central steering.
IGEES had also made further positive changes to the model of recruitment. Originally, IGEES recruited staff directly into DPER and subsequently they moved to the Departments on secondment. While this was an efficient model for recruitment in the early stages of IGEES, it had led to several issues. First, it led to a widespread sentiment, that IGEES staff were primarily identified and affiliated with DPER, limiting the ability of IGEES staff to immerse themselves and contribute positively to departments’ activities. The new model, where IGEES staff are placed directly in Departments has partly resolved this issue, even if there are still some IGEES members on secondment. While all staff at AO level are on the same pay scale, regardless of what Department they are working in, there is a perception that those working in the DPER are at an advantage when it comes to promotion opportunities and thus have access to higher pay scales at AP and PO levels. A contributory factor to this perception is the fact that in central Departments (PER, Finance and Taoiseach) the grades of AP and PO attract a higher pay scale. This is a function of the grading system in the Irish Civil Service and is not in any way confined or specific to IGEES. There was also an assumption that previous experience in a line department would help for working on DPER votes. While the current structure of the system allows such moves, as it is relatively loose, it does not systematically create expectations that such moves are seen as part of the system.
Overall, these initial recruitment processes tend to form the core of IGEES as a system. Still, as the system is currently maturing, attention also needs to be given to learning and development as well as to managing career opportunities in a broader sense. The extent to which job descriptions are being shared and needs being assessed from a common perspective across the various departments to allocate the IGEES cohorts remains unclear.
Building skills and knowledge: learning and development
IGEES supports capacity building and skills and knowledge development through a range of approaches, including slightly more flexible opportunities for mobility, a learning and development framework. This is complemented with specific learning and development opportunities as well as platform for discussion on analytical outputs and its relevance for policy such as conferences, seminars and policy discussions sessions.
Opportunities for mobility
Incentives towards mobility are encouraged within IGEES with a provision for AOs to move position after a period of two years, either within the Department or between Departments. The purpose of the mobility is to increase experiences and share skills. However, the system has had to evolve, as mentioned above to create increased shared ownership around IGEES, with direct placement of staff within Ministries at the beginning of the career. However, this direct placement of IGEES staff creates a challenge in terms of mobility, because Departments themselves face little incentive to release a member of staff after having trained them for a number of years. In addition, there is a perception of some form of an implicit career premium tends to exist for staff initially starting at DPER. There also appeared to be a mixed understanding of what the current arrangements for the movement of staff was, with differing views about how long IGEES staff should be in Departments before they were moved on to a different Department.
Overall IGEES has offered a wide network for professional growth and mobility opportunities within the Irish civil service. There is also an AP network (AP Forum) meeting 4 times per year.
Given that IGEES open recruitment is currently only focussed on AOs and APs, there are no promotion opportunities at PO level within the IGEES system, which creates incentives for staff to find such promotion opportunities within the system in a broader sense. Such consideration would also need to be contextualised to reflect that a significant number of Principal Officers in the Civil Service hold economics or related training already, which implies that the existing PO cohort might already draw on significant economic expertise, including from IGEES. Furthermore, a number of Departments already have the position of Chief Economist or Head of Research.
The question also remains as to whether or not to structure the analysis or design function within dedicated units within the various departments, which could create issues of legitimacy among the “policy people”, who work at the political and administrative interface with implications in terms of credibility and seniority.
Issues of mobility are to be addressed and discussed as part of the internal advisory group. Achieving mobility within a civil service is generally a significant challenge in many countries. Country size may matter, particularly for the smaller countries where the need for specific skills is acutely felt in some of the sectoral ministries. While Ireland seems to be more flexible than some of the Nordic countries, which lack structured mobility streams, it appears that these are not yet fully built as part of the career paths within the civil service. While Ireland is currently implementing a “One Civil Service Scheme”, broadening the scope for mobility in structural terms, building and expanding on the IGEES system might offer interesting opportunities. Considerable work has been done on ensuring that the recruitment processes are not gender biased and the gender diversity in the IGEES network reflects the diversity in the graduate pool. IGEES has also been successful in attracting international European talent, which makes it more diverse generally than other parts of the civil service in Ireland. Still, other remaining dimensions of diversity, such as in terms of ethnicity seem to have been less explored compared to other European countries.
Learning and Development Framework
The IGEES Learning and Development (L&D) Framework took effect from the beginning of 2018. Learning and development needs are agreed according to each individual’s development needs as well as the business needs of the Department according to the Business Planning and Strategic priorities. The IGEES L&D offer was intended to supplement the L&D already available through existing Departmental and One Learning L&D Frameworks. Following a consultation process, a cluster of skills and competencies specific to IGEES roles was developed (see Figure 2.1).
Figure 2.1. The IGEES Clusters of Skill Needs [pillars of learning]
Source: IGEES (2017) Learning and Development Framework 2017-2019. (IGEES, 2017[1]).
The current learning and development offer has contributed positively to the development of specialised skills in the civil service. The induction process for new starters was a good introduction to the civil service. In terms of the learning and development offer available once recruits were in post, the breadth and depth of the offering are positive. There is in general an important effort to coordinate the offer of courses to reduce the scope for duplication. Nevertheless, the offering could be improved. Some popular courses are typically oversubscribed and so finding ways to increase the number of such courses would be desirable. There is scope for further rationalisation of the offering of courses offered within the departments and those offered centrally by IGEES. In order to mainstream such analytic skills across the Irish civil service, the IGEES Unit in DPER delivers modules under the general Civil Service graduate training programme on evidence-based policymaking.
Building capacity for evidence informed policy making goes beyond investing in the skills of IGEES recruits and other skilled policy analysts. It is important for capacity to generate and use evidence for policy making to become a mainstream part of the civil service, which starts but should not stop with IGEES. For example, the Department of Taoiseach has commenced work under the Action 22 of the Civil Service Renewal Plan to strengthen policy making approaches in Ireland and has also produced a Handbook for policy making.1 Other OECD countries have experience of these twin challenges – trying to mainstream evidence informed policy making across the whole of the civil service whilst simultaneously developing specialist analytical capacity. For example, in New Zealand, the Policy Project sought to build the capacity for evidence informed policy making across the entire civil service Box 2.1.
Box 2.1. Building civil service capacity for EIPM in New Zealand – the Policy Project
The New Zealand Policy Project was launched in 2014 to improve the quality of policy advice being produced across government agencies. It deployed policy analytic tools to investigate current practices in policy design and developed a Policy Skills Framework that makes transparent the evidence, analysis and evaluation skills that are important in a civil servant’s skillset. (Acquah, Lisek and Jacobzone, 2019[2]).
A key aim was to ensure that policy advice was developed based on the best available evidence and insights, including an understanding of ‘what works’. The government recognised that there was a need to improve the evidence-base of their policy advice and to better design policies and programmes around the needs of users.
This included developing a ‘Policy Methods Toolbox’ which is a repository of policy development methods that helps policy practitioners identify and select the right approach for their policy initiative (Washington and Mintrom, 2018[3]):
The Toolbox includes a variety of resources including tools, guides and case studies. It is divided into four major themes: Start Right, Behavioural Insights, Design Thinking and Public Participation.
The Toolbox also provides concrete steps and actions that policy makers can take to improve the policy making process through making better use of research and science, using meta-data, feedback loops and input from frontline operational staff and various forms of evaluation.
Finally, while the scheme has currently led to a number of strong professionals being hired and dispersed through the departments, it seems that some of the divide between academia and government remains. An IGEES research fund was established in 2017 with the aim to promote Cross-Departmental cooperation on complex policy issues through research. Still, while this initiative is aimed as cross-sectoral issues, the possibilities to develop research within government through PhD in applied economics still appear limited, with very few partnerships with universities and opportunities for IGEES staff to develop at the PhD level2. Although not a PhD in applied economics, there is a possibility for a Doctorate in Governance at the Institute of Public Opportunity to synergise structured Doctoral studies with IGEES analytic outputs3.
Platforms for discussions
Several events are organised under IGEES auspices, including conferences, seminars and policy discussions. These are aimed at high-level policy issues. IGEES organised an annual conference, and thematic seminars, for example on the challenges faced by small advanced economies, on behavioural or tax issues.
These are very important to establish the quality of the brand and create a sense of community and sharing among participating IGEES economists. In addition, IGEES ensures active presence at the Dublin Economic Workshop. The question remains as to whether these efforts could be expanded at the international level, for example through participation in the European Economic Association or the American Economic Association conferences.
Gaps also exist for the junior isolated economists working in the line ministries, who have mentioned the lack of seminars to discuss on going work as a way to receive support from peers on technical issues, which they cannot necessarily receive in their professional environment. Active sharing and collaboration could be encouraged by organising team reviews (“show and tells”) where staff members get the opportunity to demonstrate and discuss their work. In the United States, the National Bureau of economic research organises several waves of thematic seminars where PhD Students as well as government economists can present and share on going work. In France, there are several series of annual seminars held on a monthly or biweekly basis (Fourgeaud Economics, INSEE seminars), where government economists can share on going applied analytical work for comment and discussion.
Tools
The role of frameworks and guidelines
Various guidelines have been developed in Ireland for carrying out different aspects of policy analysis. The Irish Government has established the Public Spending Code (PSC), Value for Money and Policy Reviews, Focussed Policy Assessments, a Spending Review, Performance Reporting, as well as Regulatory Impact Assessments, Tax Expenditure Evaluation Guidelines and a Code of Practice for the Governance of State Bodies. The Tax Expenditure Evaluation Guidelines introduced by the Department of Finance in 2014 for instance set out best practices and methodologies for ex ante and ex post evaluation.
The Public Spending Code guidelines has been developed by DEPR and was the main instrument used across the Departments included in the case study. Value for Money Guidelines also sit alongside the Public Spending Code. A Social Impact Framework also sets a methodology that the DPER uses. The IGEES skills are well suited and closely related to the Public Spending Code guidelines.
Strong focus on value for money, cost benefit analysis and spending review
IGEES, alongside the wider systems for policy analysis has made huge strides in increasing capacity to carry out and make use of economic analysis techniques in Ireland, and in particular to support the Spending Review process.
Spending Reviews are designed to inform Government spending allocation and maintenance and have played an important role to make expenditure policy more sustainable. The current rolling three-year Spending Review was announced on Budget day 2016 by the Minister for Finance and Public Expenditure and Reform, and is articulated between IGEES units in all Departments, Votes and the DPER (see Figure 1.1 on the Machinery of government involved in EIPM). During the crisis period, the approach to produce spending reviews consisted in each Department preparing submissions using guidance from the DPER. These central Departments provided a challenge role and sometimes conduct internal reviews, including reviews of horizontal or crosscutting issues. Since 2017, this process is less centralised and concerns more the work of other Departments. Although not a set requirement, this process may identify savings options for the Government and reinforce principles of expenditure efficiency and effectiveness into the wider budget process. The previous Comprehensive Reviews of Expenditure process, on the other hand, necessarily had to identify savings options.
Overall, the process creates a large stock of relevant analysis and evaluations across all Departments and Offices, identifying areas of expenditure requiring ongoing analysis and ensuring that such analysis takes part of the Estimates process. It allows for the systematic examination of existing spending programmes in terms of their effectiveness in meeting policy objectives and allows identifying the need to re-allocate funding to meet expenditure priorities.
Under the Spending Reviews, about 80 papers have been published, covering a wide range of important topics such as, health, education, justice, pay bill expenditure, workforce planning and digitalisation. In addition to improving the deficit position, reviews intend to ensure all expenditure is considered when Government is making budget decisions (Kennedy and Howlin, 2017[4]). While the Spending Reviews provide a useful platform to proliferate IGEES work, it might be useful to consider how to broaden the policy cycle beyond spending reviews and go beyond cost-efficiency and ex ante evaluations for instance.
By nature, the spending review process requires a coordination of expertise from the sectoral spending departments with a review and challenge function by the central departments. The role of IGEES is to ensure that both central and sectoral departments are equipped with appropriate capacity for economic and policy analysis, which is designed to make the process more impactful and more effective. Some recent developments have led to the Spending Reviews becoming a more collaborative process, whilst maintaining DPER’s critical challenge function as a finance ministry. Some departments and the Votes within DPER had collaborated on the selection of Spending Review topics. This had helped to make the engagement more productive and useful for both sides. On the other hand, in the Department of Children and Youth Affairs, only the Vote Section chose topics for both 2018 and 2019.
There were also examples of joint analysis both within and outside of the context of the Spending Review, which are a useful way of pooling expertise and bringing together different perspectives on the same problem, which was greatly helped by the distribution and sharing of IGEES staffing resources.
Broadening the analytical framework beyond the spending code and strict VFM approaches
Spending code guidelines may not capture all the dimensions that are relevant for policy evaluation, due to their strict focus on expenditure and value for money. Some Departments have also developed frameworks to support their own activity in particular areas. The Department of Business, Enterprise and Innovation (DBEI) has developed a robust framework for evaluation specifically in relation to enterprise support programmes. DBEI developed the framework based on a review of international best practice in enterprise evaluation. International experience confirms that guidelines for carrying out different elements of policy analysis can be a key component of improving the quality of policy analysis and evaluation (OECD, 2020[5]).
Most OECD countries tend to use some form of a guide for policy evaluation within government. Given the increasing focus on outcomes as well as on wellbeing in supporting policy priorities across a range of OECD countries, it is also important that such instruments can also support a focus on result, and be flexible enough to capture a broad range of policy outcomes. For example in Ireland, while the department of health mobilises some tools for economic analysis, it also develops broader frameworks for understanding the impact of health policy and resources on people, which requires attention to the many determinants of health status that are beyond the realm of the health care system itself. Canada has developed a range of frameworks and guidelines that cover several aspects of evaluation (see Box 2.2).
Box 2.2. The role of frameworks and guidelines for the promotion of EIPM in Canada
Evaluation in the Government of Canada
The Results Division of the Canadian Secretariat, successor of the Centre of Excellence (CEE) for Evaluation, is responsible for evaluation activities within the Government of Canada. It offers useful resources, information and tools for Government professionals and anyone interested in evaluation at the federal level. Moreover, requirements for evaluation practices are outlined in the Policy on Result, which took effect in 2016. Overall, the Secretariat has functional leadership regarding the implementation, use and development of evaluation practices across government. To support quality EIPM, the Treasury Board Secretariat offers a number of useful guidelines:
Guide to Rapid Impact Evaluation (RIE) (2017): this practical guide gives a range of methods for conducting RIE and advice on when and how these can be used in government. More precisely, it defines RIE, the time and resources needed to conduct one, its key benefits and challenges, and support for planning, analysis and reporting the results.
Assessing Program Resource Utilization When Evaluating Federal Programs (2013): this document is made for evaluators of federal government programmes, programme and financial managers, and corporate planners. It helps them understand, plan and undertake evaluations that include the assessment of resource utilization. It provides them with methodological support to ensure that they have the knowledge and competencies to conduct quality and credible programme resource utilization assessments.
Theory-Based Approaches to Evaluation: Concepts and Practices (2012): this document introduces key concepts of theory-based approaches to evaluation and their application to federal programmes. It should be complemented by additional readings and advice for step-by-step guidance on conducting evaluations.
Supporting Effective Evaluations: A Guide to Developing Performance Measurement Strategies (2010): this guide supports departments, programme managers and heads of evaluation in developing performance measurement to support evaluation activities. It provides recommendations, tools and frameworks for conducting clear and concise performance measurement strategies as well as guidance regarding the roles of those in charge of developing such strategies.
Sources: Based on Treasury Board of Canada, 2019[3], 2017[4], 2013[5], 2012[6],2010[7].
Processes
The department work programmes for policy analysis
Departments varied in the extent to which they decided and formalised their work programme for policy analysis. Some departments had a structured process for determining the work programme, which could include consulting with relevant stakeholders, consideration of the programme for government and considerations of existing and expected future resources. For example, in one department, the specialist unit for policy analysis led the development of the work programme. The unit conducted a range of consultation meetings with divisions across the Department as well as with other stakeholders. Assistant Secretaries then discussed and prioritised a range of options to take forward. The unit then developed these options into a work programme proposed to the Department’s management board, who make the final decisions on the work programme. The analysis work specified in the work programme was typically organised into two cycles. The first half of the year focused on planning the analysis and building the evidence base, while the second half of the year was devoted to analysing the evidence base and making policy focused recommendations. These sound arrangements ensure that the scarce IGEES and other human resources are being applied in a way that is well prioritised and avoids fragmentation and lack of structural attention due to constantly shifting short-term political demands.
In many other Departments, a less structured process for deciding policy analysis was the norm. Similarly not all Departments developed a formal work programme for analysis. This was typically the case in Departments with more limited analytical capacity – in terms of numbers of analysts – and in Departments where analysts worked in policy units, in which case the allocation of policy analysis was incorporated into the planning process for the policy unit. An absence of a formal work programme specifically devoted to policy analysis may result in the fact that IGEES staff would not necessarily be able to fully apply their analytical skills. Especially at the AO grade we were told that it could be challenging to protect time spent on the analysis in face of the pressures to manage the day to day of core civil service tasks such as ministerial briefings and responding to parliamentary questions. The challenge is to find a balance between responsiveness to political demands through ensuring that IGEES capacity is immersed in the policy clusters in the ministries, while ensuring the possibility to develop sound analytical products that can take advantage to the proximity of data to improve policy effectiveness.
The balance of work conducted internally versus externally
There was also wide variation between Departments in terms of how analytical resources were allocated and the balance of work done in house and work contracted out externally. Some larger Departments were able to draw on sufficient policy analysis resources internally to be able to complete the majority of the departmental work programme analysis in house. The far more common model was to have a mixture of policy analysis projects conducted in house and contracted externally.
The decision about whether to do policy analysis in house or externally involved a range of considerations including the skills and expertise available in house, the scale of the project, the timeframe for completion and any budgetary constraints. There was no one size fits all solution. There is a range of risks and issues with conducting work in house and contracting it out. When time is a critical factor, there is a tendency to commission out the work to an organisation who would be able to delivery quickly. Commissioning work externally also has the advantage of being somewhat protected from shifting priorities and a change in the availability of policy analysis available in house.
Research commissioning is a complex and specialist exercise that is subject to many risks. There can be challenges when an evaluation is externally commissioned, but where the in house analysts hold the relevant data sets and knowledge of its strengths and limitations for policy analysis. This situation requires sufficient time and resource so that the in house team can hand over and explain the data to the contractor. This situation can present resource challenges if the in house analysts are working on activities that are more urgent. Another issue that came up relatively frequently was ensuring that sufficient internal policy analysis capacity could be devoted to the commissioning process itself. In some Departments, every policy analysis project that is commissioned by the Department must go through the specialised unit for policy analysis. This has several advantages. First, it means that the in house analysts can take a decision about whether an existing policy analysis project fulfils the requirement. If not a decision can then be made, about whether there is the competency and capacity to do the work internally or whether it is necessary for it to be commissioned externally. Additionally, this ensures that findings from commissioned work can be interpreted, communicated and absorbed internally. Another advantage is that the analyst can act as a broker between the policy teams and the external contractor. This ensures that the invitation to tender clearly and accurately specifies the nature of the policy analysis. It also ultimately increases the likelihood of an appropriate and high quality project given the policy concern.
References
[2] Acquah, D., K. Lisek and S. Jacobzone (2019), “The Role of Evidence Informed Policy Making in Delivering on Performance: Social Investment in New Zealand”, https://www.oecd-ilibrary.org/docserver/74fa8447-en.pdf?expires=1568195730&id=id&accname=ocid84004878&checksum=44215ED1C60EC9362961EFE7B4B79984 (accessed on 11 September 2019).
[1] IGEES (2017), IGEES Learning & Development Framework 2017-2019, https://igees.gov.ie/wp-content/uploads/2017/12/IGEES-Learning-and-Development-Framework-2017-19-For-Launch.pdf (accessed on 12 August 2019).
[4] Kennedy, F. and J. Howlin (2017), “Spending reviews in Ireland-Learning from experience”, OECD Journal on Budgeting, Vol. 2016/2, pp. 93-109, https://www.oecd-ilibrary.org/docserver/budget-16-5jg30cchf0g0.pdf?expires=1565605875&id=id&accname=ocid84004878&checksum=53F7E58BBFC1BEF3E24DE76EFFB045D9 (accessed on 12 August 2019).
[5] OECD (2020), Institutionalisation, Quality and Use of Policy Evaluation, Governance Lessons from countries experience, forthcoming.
[3] Washington, S. and M. Mintrom (2018), “Strengthening policy capability: New Zealand’s Policy Project”, http://dx.doi.org/10.1080/25741292.2018.1425086 (accessed on 11 September 2019).
Notes
← 1. The Action 22 is about Strengthen policy-making skills and develop more open approaches to policy-making. See https://www.gov.ie/en/policy-information/fd9c03-civil-service-renewal/?referrer=/en/civil-service-renewal/.
← 2. IGEES staff do have the opportunity to apply for funding to support academic study through the Refund of Fees Scheme within Departments.