SME Policy Index: Western Balkans and Turkey 2019

Policy framework and assessment process

Introduction

The SME Policy Index is a benchmarking tool for emerging economies to monitor and evaluate progress in policies that support small and medium-sized enterprises (SMEs). It is structured around the ten principles of the Small Business Act for Europe (SBA), providing a wide range of pro-enterprise measures to guide the design and implementation of SME policies in the European Union (EU). The index was developed in 2006 by the OECD in partnership with the European Commission, the European Bank for Reconstruction and Development (EBRD), and the European Training Foundation (ETF). It has been applied 12 times, covering 33 economies in 4 regions: the Western Balkans and Turkey, Eastern partnership countries, North African and Middle East regions, and the Association of Southeast Asian Nations (ASEAN) countries.

While a number of other indices and benchmarking reports have been used to assess the business environment in the Western Balkans and Turkey (WBT), the SME Policy Index follows a holistic approach that seeks to provide policy makers with a single window through which to assess and, if necessary, readjust policies favouring the development of SMEs. This report identifies the nature and extent of their strengths and weaknesses in SME-related policy settings and their implementation, allowing comparisons to be made across economies and measuring convergence towards OECD and EU good practice.

Overview of the 2019 assessment framework

The SME Policy Index: Western Balkans and Turkey 2019 analyses and benchmarks SME policies across 12 policy areas (“dimensions”). These are further broken down into sub-dimensions and thematic blocks. Each of the 12 policy dimensions comprises up to 5 sub-dimensions that capture the critical elements of policy development in the policy dimension of which they are part. The sub-dimensions are, in turn, made up of thematic blocks that consist of concrete indicators, both quantitative and qualitative (Figure 9).

Figure 9. Overall assessment framework of the SME Policy Index 2019

The SME Policy Index assessment framework was developed to respond to the priorities identified in the ten principles of the SBA. Table 9 shows how the policy dimensions and sub-dimensions relate to each SBA principle.

Table 9. The Small Business Act assessment framework and its links to the SBA principles

SBA principle	Related policy dimension	Related sub-dimension
1. Create an environment in which entrepreneurs and family businesses can thrive and entrepreneurship is rewarded	1. Entrepreneurial learning and women’s entrepreneurship	Entrepreneurial learning
		Women’s entrepreneurship
2. Ensure that honest entrepreneurs who have faced bankruptcy quickly get a second chance	2. Bankruptcy and second chance	Preventive measures
		Bankruptcy procedures
		Promoting second chance
3. Design rules according to the “think small first” principle	3. Institutional and regulatory framework for SME policy making	Institutional framework
		Legislative simplification and regulatory impact analysis
		Public-private consultations
4. Make public administration responsive to SMEs	4. Operational environment for SMEs	Digital government services for enterprises
		Company registration
		Business licensing
		Tax compliance procedures for SMEs
5. Adapt public policy tools to SME needs	5a. Support services for SMEs	Business support services provided by the government
		Government initiatives to stimulate private business support services
	5b. Public procurement	Public procurement
6. Facilitate SME access to finance and develop a legal framework and business environment supportive of timely payments in commercial transactions	6. Access to finance for SMEs	Legal and regulatory framework
		Sources of external finance – bank financing
		Sources of external finance – non-banking financing
		Venture capital ecosystem
		Financial literacy
7. Help SMEs to benefit more from the opportunities offered by the Single Market	7. Standards and technical regulations	Overall co-ordination and general measures
		Harmonisation with the EU acquis
		SME access to standardisation
8. Promote the upgrading of skills and all forms of innovation	8a. Enterprise skills	Enterprise skills
	8b. Innovation policy for SMEs	Policy framework for innovation
		Government institutional support services for innovative SMEs
		Government financial support services for innovative SMEs
		SME and research institution collaboration and technology transfer
9. Enable SMEs to turn environmental challenges into opportunities	9. SMEs in a green economy	Framework for environmental policies targeting SMEs
		Incentives and instruments for SME greening
10. Encourage and support SMEs to benefit from growth markets	10. Internationalisation of SMEs	Export promotion
		Integration of SMEs into global value chains
		Promoting the use of e-commerce

The SME Policy Index 2019 assessment framework has been adjusted since the 2016 assessment to provide a more in-depth analysis and to better respond to the needs of the economies being assessed. Over the last decade, while most of the economies have demonstrated improvements in building up their institutional frameworks and designing comprehensive policies, the implementation of SME policies has largely remained below par, with limited impact on their local SME populations. In light of this, the assessment framework has been adjusted to account for policy implementation (Box 2).

Box 2. Refinements to the assessment framework

Compared to the 2016 SME Policy Index, the 2019 assessment includes new features introduced to strengthen the analysis and increase its impact. These refinements include:

• five new sub-dimensions to include policy aspects that were not covered in the previous cycle

• extending and amending other sub-dimensions to collect more in-depth information

• increased focus on the economy’s performance and implementation of SME policies

• private sector feedback to gauge the outcome of policies

• greater involvement of national statistical offices and more extensive collection of relevant statistical data.

A detailed description of how the assessment framework has been adjusted is presented in Annex A.

Methodological approach

Policy areas are given a score from 1 to 5, with Level 1 being the weakest and Level 5 the strongest (Table 10). Reaching Levels 4 and 5 is more difficult, requiring solid policy implementation and monitoring and evaluation. Scores are determined by assigning a numerical value to each qualitative indicator. The results are then aggregated with weightings applied at the sub-dimensional and thematic block levels, taking into consideration the importance of each indicator in policy formulation or implementation. A detailed description of the allocation of scores is presented in Annex A.

By assigning scores, the index transforms qualitative information into numerical information, facilitating cross-economy comparison and allowing for systematic monitoring of policy developments.

Table 10. Description of score levels

Level 5	Level 4 plus results of monitoring and evaluation inform policy framework design and implementation
Level 4	Level 3 plus evidence of a concrete record of effective policy implementation
Level 3	A solid framework, addressing the policy area concerned, is in place and officially adopted
Level 2	A draft or pilot framework exists, with some signs of government activity to address the policy area concerned
Level 1	No framework (e.g. law, institution) exists to address the policy topic concerned

In order to make it easier to compare performance across economies and over time, quantitative indicators are used to provide additional evidence on the performance of policy settings, processes and programmes. These include horizontal statistical data, to better understand the nature of the SME sector; and data specific to individual dimensions to assess policy outputs and outcomes relevant for implementation. Although the quantitative data are not included in the scoring, they have been taken into account in the narrative text, including the analysis of the overall SME policy setting and formulation of policy recommendations. The quantitative data were primarily collected by the OECD from national statistical offices.

The assessment process

The SME Policy Index is based on the results of two parallel assessments. The economies’ governments conducted a self-assessment, which involved completing a questionnaire and assigning a score for each policy indicator using the assessment grid, and providing an accompanying justification. The OECD and its partner organisations also conducted an independent assessment, with input from a team of local experts.

The final scores are the result of a highly collaborative and consultative consolidation of these two assessments, enhanced by further desk research by the OECD and the partner organisations, as well as consultations with government representatives. To achieve this, a series of reconciliation meetings were held in each of the assessed economies to discuss and compare the two parallel assessments, which helped to reduce bias and misjudgements (see below for more details).

Based on the information gathered in these reconciliation meetings, the OECD and its partner organisations then decided the final results. These were presented to the SBA Co-ordinators in a regional meeting at the end of the assessment process. Minor modifications of the scores were subsequently made.

Assessment methodology for Dimensions 1 and 8a

For this assessment, the methodology for the two human capital dimensions, Dimensions 1 and 8a, has been harmonised with the remaining dimensions. Previously, Dimensions 1 and 8a were assessed using 5-level qualitative indicators. Moving to the questionnaire, with its binary (yes/no), multiple-choice and open questions, has harmonised the assessment methodology across all the SBA dimensions (the other dimensions were first assessed in this way in 2016). However, comparison of the scores between the 2016 and 2019 assessments need to be made with caution because of this change.

For Dimensions 1 and 8a, the self-assessments took place with the support of an ETF local expert. Focus group meetings on each of the three sub-dimensions – entrepreneurial learning, women’s entrepreneurship and SME skills – were held in each economy, attracting between 10 and 75 stakeholders from all relevant sectors. The aim was to validate and interpret the initial assessment findings, and identify additional evidence to strengthen the analysis. The final results of the assessment have been agreed between the ETF and the national counterparts.

Assessment phases and timing of the 2019 assessment

The assessment was carried out between January 2017 and January 2019 in four main phases: 1) design; 2) data collection and evaluation; 3) consolidation; and 4) review and publication (Figure 10).

Figure 10. Overview of the assessment process phases

• Design phase (January 2017-August 2017). The OECD and the partner institutions revised the assessment framework in consultation with the SBA Co-ordinators, partner organisations and the European Commission. The assessment methodology for Dimensions 1 and 8a was harmonised with the remaining dimensions.

• Data collection and evaluation phase (September 2017-February 2018). Launch meetings were held in the seven capitals of the assessed economies to present the new assessment framework to the line ministries and public institutions which were expected to contribute to the information collection process. The two documents making up the assessment framework – the questionnaire and statistical data sheet – were explained in depth to the participants, directing particular attention to the newly added questions and indicators.

  Following the launch events, the SBA Co-ordinators distributed the questionnaires to the appropriate counterparts in the ministries and government agencies, as well as the statistical sheet to the national statistical offices. The SBA Co‑ordinators then compiled the data and documentation, and completed the questionnaire. Scores were assigned for each policy dimension, with an accompanying justification. The completed questionnaires and statistical data sheets were sent to the OECD team by February 2018. Meanwhile, the OECD and its partner organisations also conducted an independent assessment. This was based on input from a team of local experts who collected data and information, and conducted interviews with SME representatives as well as key public and civil society stakeholders. A detailed description of the interviews is presented in Annex C.

• Consolidation phase (February 2018-August 2018). Reviews of the inputs by the OECD and the partner institutions revealed the need for additional information on certain elements. These were requested and the revised questionnaires were sent back during February-April 2018.

  Once the data collection was complete, the OECD and EBRD organised reconciliation meetings in each of the assessed economies. These meetings were typically attended by 60-80 SME policy stakeholders, including ministries and government agencies, international donors, and representatives from civil society, academia, non-government organisations, the private sector and the EU delegations. These meetings aimed to close any remaining information gaps in the questionnaire, discussing any discrepancies between the two parallel assessments and filling in missing information. They also served as an opportunity to seek the views of a broad range of policy stakeholders on how SMEs are affected by current policies, and what else can be done across various policy areas to improve SMEs’ performance and competitiveness in the Western Balkans and Turkey.

  For the two policy dimensions (entrepreneurial learning and women’s entrepreneurship, and enterprise skills) whose assessment was led by the ETF, similar meetings took place during February-March 2018, as described above.

  After the reconciliation meetings, the assessment findings were consolidated. The OECD and partner organisations decided on the final scores under each policy dimension presented in this report. The preliminary findings and scores were subsequently presented to SBA Co-ordinators at a meeting in Paris on 12 July 2018.

• Review and publication phase (September 2018-May 2019). The draft SME Policy Index chapters were made available to the WBT governments, the European Commission and the EU delegations for review during September-December 2018. Meanwhile, the draft report was also peer reviewed internally by the OECD, the EBRD and the ETF. Following the review, final comments were integrated into the report. The publication was launched on 7 May 2019 in Sarajevo.

Strengths and limitations

The SME Policy Index: Western Balkans and Turkey 2019 has a number of strengths which make it a uniquely valuable report for policy makers, citizens, researchers and international organisations. However, it also has some limitations which need to be borne in mind (Table 11).

Table 11. Strengths and limitations of the SME Policy Index 2019

Strengths	Limitations
The indicators have been structured to be fully compatible with the EU Small Business Act.	Limited comparability of performance over time due to the inclusion of additional policy areas in the assessment framework and changes in scoring weights.
Independent and rigorous assessment enables it to benchmark the performance of WBT economies against OECD/EU policy standards.	Statistical coverage is limited in the seven economies, and comparison of policy performance can be challenging. Problems such as the lack of a uniform SME definition mean data cannot always be directly compared.
Good practice examples and policy recommendations, both at regional and economy level, offer concrete guidance to governments for policy reform.	The sub-dimension weightings are based on expert opinion, and therefore involve subjectivity
The report facilitates discussion about regional remedies for similar problems and the transferability of experience between economies	The application of the same set of indicators and weights can mask certain economy-specific characteristics.
Engagement with a wide range of stakeholders, including SME representatives, allows for better understanding of needs on the ground, and how effective government policies have been.	The assessment framework does not exhaustively cover all the policy areas affecting SMEs.
The tripartite participatory approach to evaluation and measurement brings together governments, the private sector and the OECD.	In economies where SME policies are mainly implemented at sub-national level, the implementation of government policy can be hard to measure.
Combines original data collected by the OECD, European Commission, ETF and EBRD with existing data from sources such as the World Bank.