Monitoring and Evaluating the Strategic Plan of Nuevo León 2015-2030

Assessment and Recommendations

Toward a sound institutional system for planning, monitoring and evaluation

Formally established in 2014, the Nuevo León Council for Strategic Planning is an advisory body of the state executive for strategic planning and its evaluation. It aims to promote the sustainable development of Nuevo León through the implementation of the Strategic Plan for the State of Nuevo León 2015-2030. According to the Strategic Planning Law, the council is responsible for elaborating the Plan as well as for monitoring and evaluating the Plan’s implementation. The council also seeks to foster stakeholder engagement, as its members are representatives of the public and private sectors, academia and civil society.

The council is part of a broader planning system in Nuevo León. It includes the centre of government of Nuevo León’s state public administration, composed of the Head of the Executive, the Executive Office of the Governor and the Secretariat of Finance and General Treasury. The CoG plays a central role in the whole-of-government planning system. In particular, the Executive Office of the Governor (EOG) is responsible for conducting strategic planning activities and ensuring their execution and compliance. According to the Strategic Planning Law (Article 18) and its Guidelines (Chapter 2 and Chapter 3 Section 5), the council and the CoG’s responsibilities are complementary. However, in practice, the council has progressively expanded its actions to policy formulation, design and implementation. On the other hand, there are insufficient resources and mechanisms within the CoG dedicated to coordinating and monitoring the Plan’s objectives and projects.

This situation has resulted in political tensions between the state public administration and the council, and the latter losing focus of its main role of long-term planning. To strengthen the monitoring and evaluation system in Nuevo León, the Government could:

• Clarify the respective responsibilities on strategic planning of the council and of the centre of government. In particular:

  • The CoG should have a clear mandate to coordinate the implementation and monitoring activities of the Strategic Plan and of the State Development Plan. The CoG should be given the responsibilities and resources to ensure effective whole-of-government coordination, to carry out objective setting and prioritisation exercises, monitor policy priorities and conduct strategic internal and external communication. This can be done, in the short term, through revising the Internal Guidelines of the Executive Office of the Governor.

  • The council should strengthen its role as an advisory body and strategic knowledge broker, moving it away from implementation and monitoring. This requires building technical capacities in the council to provide objective and timely policy advice to the government in key priority areas and the selection of a limited number of policy priorities that require the concerted action of the state government, civil society and the private sector, for which the council would prepare focused evaluations with clear recommendations.

• The council should pursue efforts to implement a coherent strategy for strategic planning and evaluation, drawing on the respective strengths of the council, the Executive Office of the Governor and of the Ministry of Finance. Overall, the new apparatus requires a conceptual framework on monitoring and evaluation for the government, the inclusion of specific capacities and provisions to monitor and evaluate the Plans and a clarification of the function of the council in terms of planning, policy advice and evaluation, including a modification of the currently existing regulatory framework.

Identifying clear policy priorities and setting pathways for implementation

The main planning instrument of Nuevo León’s council is the Strategic Plan for the State of Nuevo León 2015-2030. It is the first integrated long-term planning instrument that has been jointly designed by government and civil society in a Mexican subnational government. It is perceived as an important initiative for the state, reflecting on its long-term development goals and offering the possibility to plan beyond the electoral cycle.

Yet, the methodology used to design the Plan in 2015 was not sufficiently robust. The diagnoses established for each commission did not systematically include a thorough assessment of previous reform initiatives and the methodology has omitted how different opportunity areas link to each other. Moreover, there is no clear methodological framework to articulate the elements of the Strategic Plan with the objectives of other planning instruments, such as the Sustainable Development Goals and the State Development Plan. In practice, the alignment between the Plan’s objectives and the state budget is limited and the implementation of performance-based budgeting remains an area for improvement.

Moreover, the structure of the document is complex and not fully coherent, challenging its use as a planning and implementation tool. Indeed, the Plan has too many layers and does not rest on a clear logical framework, which makes it difficult to link opportunity areas to outcome level objectives. Therefore, the following recommendations are made to increase the coherence of planning and aligning policy priorities:

• Clarify the ways in which the State Development Plan contributes to pursuing the goals of the longer term Strategic Plan. There is a possibility for greater alignment of some of the opportunity areas (Strategic Plan) with the objectives (State Development Plan). In this scenario, the SDP would focus on medium-term outcome level objectives and below (objectives), while the Strategic Plan focuses on longer-term outcome level objectives (prioritised opportunity areas). In particular, the manual for the elaboration of the future SDP should contain a methodology to relate the medium term objectives of the SDP with the long-term objectives of the Strategic Plan. Nuevo León could consider using a theory of change to make this link explicit.

• Clarify and communicate the coherence between the Strategic Plan and the SDGs. It would be possible to include a table showing the alignment of the Plan’s opportunity areas with the SDGs and to ensure that the prioritised opportunity areas refer to SDGs at least once.

• Clarify the responsibilities of the commissions in regard to overseeing the implementation of the SDGs. The council could consider systematically including a section on how the opportunity areas have contributed to the achievement of the SDGs at a local level, in the evaluation reports for each thematic area.

Moreover, to improve the overall coherence and robustness of the plan, Nuevo León could:

• Simplify the Strategic Plan in order to refocus it around impact and outcome level objectives. Only priority opportunity areas, their targets and indicators should be kept in the Plan while strategic lines, initiatives and projects should be included in the action plan, if relevant.

• Strengthen the problem analysis phase by integrating in each commission’s diagnosis the conclusions of a systematic review of the evaluation of previous plans in the relevant thematic, as well as considering the strategic use of reliable data for objective/target setting.

• Reduce the number of priority opportunity areas in the Plan to ensure greater harmonisation of the number and nature of prioritised opportunity areas assigned to each commission. To this end, Nuevo León could consider complementing the existing prioritisation methodology with an analysis and comparison of the opportunity areas under a logic model and/or theory of change or by prioritising only the areas that can be measured by an indicator.

• Clearly communicate, in the Plan itself, the reasons behind the selection of the priority opportunity areas in order to ensure support within and outside the administration.

Monitoring the Strategic Plan as a tool for results

Sound monitoring can facilitate planning and operational decision-making by providing evidence to measure performance and help raise specific questions to identify implementation delays and bottlenecks.

Yet, the monitoring set-up in Nuevo León, whether for the Strategic Plan 2015-2030 or the State Development Plan lacks clarity for its actors and legibility for citizens. Indeed, while a whole-of-government legal framework exists for monitoring the Strategic Plan 2015-2030, the definition of monitoring contained in the Strategic Planning Law is not comprehensive and may lead to confusion about its purpose. Furthermore, the monitoring methodology and tools for the Strategic Plan are not well defined. In order to establish a sound monitoring set-up for the Strategic Plan, the state public administration will need to increase its competences and resources for monitoring, and establish mechanisms that promote the use of performance results in decision-making.

In order to do so, Nuevo León could consider the following recommendations:

• Clarify the monitoring set-up to better support the delivery of the Plan by adopting a comprehensive definition of monitoring to establish a shared understanding of its objectives and modalities within the public sector and to clarify the roles of key actors. This requires updating the strategic planning regulation, and giving an explicit legal mandate to each of the actors in order to redefine their respective responsibilities. In particular, it will require setting-up a performance dialogue within the state public administration in order to improve operational decision-making at the level of line ministries and of the centre of government (CoG), regarding both the SP and the SDP simultaneously. Moreover, Nuevo Léon may wish to conduct an annual joint review of the SDP and SP’ objectives between the council’s thematic commissions and the secretariats. Finally, the state public administration could produce a communication leaflet on the Strategic Plan. This communication leaflet, which could be updated every three years by the state public administration, would replace the current monitoring/evaluation report conducted by the council and be designed first and foremost as a communication tool.

• Promote the quality of monitoring by strengthening the robustness of the Plan’s indicators, for instance by clarifying the coherence between the indicators and the Strategic Plan’s layers, as well as by providing fit-for-purpose and user-friendly analysis in its monitoring dashboards. Improving the quality of monitoring will also require the state public administration to invest in increasing its competences in this regard, and to allocate specific resources to this function. Finally, it will require the development of assurance mechanisms in order to ensure quality throughout the monitoring process, as well as strengthening control mechanisms currently in place to ensure the quality of the end-product meets certain pre-determined criteria.

• Strengthen communication and the use of the monitoring results. For this to be achieved, the state public administration could consider producing a communications leaflet –as mentioned above-, regularly updating the Avanza Nuevo León platform with indicators from the Plan, as well as feeding monitoring evidence into the budget-cycle.

Creating capacities for the evaluation of the Plan

Evaluation is critical to understand if policies are improving and sustaining the wellbeing of citizens and the prosperity of their state. Providing an understanding of what policies work, why, for whom, and under what circumstances, can contribute to generating feedback loops in the policy-making process. Particularly in the case of Nuevo León, policy evaluation and its strategic use throughout the policy cycle can support strategic planning by improving the links between policy interventions and their outcomes and impact. Indeed, the state shows a clear interest in ensuring government effectiveness and efficiency, as well as public accountability, through evaluation, in particular of the Strategic Plan.

Nevertheless, the State of Nuevo León currently fails to display a sound and robust evaluation system as is commonly understood among OECD countries, both from a whole-of-government perspective and for its Strategic Plan. Similarly to a majority of OECD countries, Nuevo León has embedded the evaluation of its Strategic Plan in a legal framework. Still, macro-level guidance on who carries out the evaluation of the Strategic Plan 2015-2030 and when is lacking. Quality assurance and quality control mechanisms remain embryonic. In particular, the council does not have the capacities to conduct in-house evaluations at this stage. Finally, use of evaluations remains a challenge in Nuevo León and can be addressed both internally by the council and by promoting use in the policy-cycle overall.

Therefore, in order to build a sound evaluation system for the Strategic Plan, Nuevo León may wish to consider the following recommendations:

• Establish a sound policy framework for evaluating the Strategic Plan, which should specify what programmes and policies are going to be evaluated, who will be the evaluator, and with what means.

• Promote the credibility of evaluations by developing explicit and systematic mechanisms within the council to ensure the quality of the evaluation process, as well as quality control mechanisms. This can entail developing guidelines or standards for evaluations that can build on the existing guidelines for the Consolidation of a Results-Based Budget and the Performance Evaluation System. It may also require developing appropriate evaluation competencies within the council. In any instance, quality assurance and control mechanisms should go hand in hand to ensure that the evaluation design, planning, delivery and report are properly conducted to meet pre-determined quality criteria. These can take the form of peer reviews by experts, meta-evaluations or self-evaluation checklists. They can also entail greater collaboration with actors outside the executive to strengthen internal quality assurance and control mechanisms, such as the audit institution for assessing the evaluation function and the evaluation system.

• Promote the use of evaluations by strengthening the role of internal stakeholders (within the commissions) and external stakeholders throughout the whole evaluation process and elaborating a communication strategy to adapt the way in which research findings are presented to their potential users. More importantly, promoting the use of evaluations may entail incorporating evaluation results in the policy-cycle in a more systematic way, for instance in the budgetary cycle or by discussing evaluation results at the highest political level by systematically holding discussions within the state public administration.

Promoting evidence-informed policy-making in Nuevo León

Increasing governments’ capacity for an evidence informed approach to policy making is a critical part of fostering good public governance to achieve broad societal goals, such as those set out in the Strategic Plan 2015-2030. The goal of evidence-informed policy-making is to enable agile and responsive governments, which are well equipped to address complex and at times “wicked” policy challenges. Despite the potential for policies to be based on evidence, an effective connection between the supply and the demand for evidence in the policy making process often remains elusive, insofar as many governments lack the necessary infrastructure, organisations and skills to build such effective connections.

Moreover, in a context in which justifying the use of public resources with accurate evidence is becoming increasingly important, the consultation of multiple sources of information and actors before the implementation of a public policy, programme or public service has become essential. As a result, many OECD countries have set up policy advisory systems to support evidence informed policy making with the best possible evidence. In many ways, the functions of the Nuevo Léon council can be analysed within this framework of policy advisory bodies.

However, the activity of the council across all stages of the policy cycle has exacerbated political tensions between the council and the state public administration, moving the council from its main mandate of long-term planning and decreasing the relevance and impact of its advice. Moreover, some stakeholders believe that the private sector is overrepresented in the composition of the council, while citizens and experts appear to have a limited voice. As a result, part of the challenge for the council in the future will be to expand its technical capacities and skills, as well as to ensure that its membership is neutral and represents the socioeconomic diversity of Nuevo León’s community, in order to continue providing relevant and credible advice.

In order to achieve a more responsive policy advisory system, as well as to reinforce council’s independence and technical legitimacy, in view of promoting a government-wide evidence-informed policy-making system, Nuevo León could consider:

• Strengthening its policy advisory system. This requires mapping advisory bodies in Nuevo León to determine technical gaps in the system, ensuring representativeness of different disciplines, and clarifying the role and functions of each actor in the process.

• Focusing the council’s mandate on evaluation and evidence provision. In particular, the council could leverage its status as a relatively autonomous body that represents the views of a wide range of stakeholders, at arm’s length from the government, to provide credible evidence in Nuevo León. On the other hand, bodies that are closer to the government and therefore more closely linked to the administrative mandate would be in a better position to contribute to policy implementation and monitoring; while the council could focus solely on evidence and evaluation and be supported by research institutes or think thanks in the field. This may alleviate political tensions and enable the council to focus on its main role of long-term planning.

• Fostering inclusiveness and expertise in the council’s decision-making process to provide relevant advice. This requires the council to consider a multitude of interdependent factors to provide advice, such as: making a calendar with the timing of its publications available; ensuring board members are neutral, that they provide expertise, and that they represent the sociodemographic and economic diversity of the state; and requiring professional qualifications and training from the members of the council.

• Implementing knowledge brokering methods to promote the impact and use of the council’s advice. The council should consider adopting knowledge-brokering approaches, to provide critical, independent and reliable evidence in a timely and attractive manner. This could imply using evidence synthesis methods to appraise the available evidence base and identify knowledge gaps. Lastly, the council has a role to play in translating evidence, to build an organisational culture for the effective adoption of evidence and to strengthen networks between knowledge producers and knowledge users.

• Developing capacities for evidence-informed policy-making in the state public administration, by expanding the skills for evidence, for instance by having the council offer trainings to civil servants, and more generally by establishing the appropriate infrastructure to generate and use evidence.