Monitoring and evaluation is needed to assess the economic efficiency of SME and entrepreneurship policy actions. They should also inform the design and mix of SME and entrepreneurship policies by identifying those features which lead to desirable outcomes. Evaluation is fundamental to public accountability.
Reliable methods for the evaluation of SME and entrepreneurship policies using appropriate counterfactuals have been established and demonstrated. However, such methods, which can address the heterogeneous impacts of policies on different types of SMEs, are not widely used.
Key challenges include increasing the application of rigorous evaluation techniques; better specifying policy objectives, targets and indicators; making better use of data, including existing national administrative data sets for purposes such as tax and social security; and seizing the potential of Big Data.
It is important to make better use of evaluation in the policy cycle; evaluate systematically across the portfolio of SME and entrepreneurship interventions; and assess the impacts on SMEs and entrepreneurship of policies in areas where business development is not the primary objective.
Strengthening SMEs and Entrepreneurship for Productivity and Inclusive Growth
10. Monitoring and evaluation of SME and entrepreneurship programmes
Summary
Why is it important?
A core justification for SME and entrepreneurship policy is the presence of coordination failures and information asymmetries, which may limit SMEs’ ability to contribute to economic and industrial development, innovation, job creation and social cohesion. SME and entrepreneurship support can come in various forms, including advice, training, and enhanced access to finance and can help both, the individual SME owner as well as the rest of society through positive spill-over benefits in terms of job and wealth creation, as well as economic growth.
There is consequently substantial direct public expenditure on SME and entrepreneurship programmes and many other policy measures, which target SMEs, have important indirect public finance implications through foregone tax revenue. It is the responsibility of policy makers to use monitoring and evaluation1 to provide accountability, and to ensure that expenditure is in line with programme objectives and has the intended effects. Monitoring and evaluation is also needed to refine and redirect programme interventions, hence improving performance and “value for money”. Applied systematically across different types of policy interventions, it can help to ensure that policy, in aggregate, is coherent and that the policy mix is appropriate.
What are current trends and challenges?
There have been a number of recent advances in policy evaluation techniques, many of which are likely to be particularly valuable in the evaluation of SME and entrepreneurship programmes and policies. There have also been some important advances in data collection for SME and entrepreneurship policy development. SME and entrepreneurship programme monitoring is now widely established internationally, and monitoring frameworks for SME and entrepreneurship strategies are largely in place. For example, through the yearly SME Performance Review, the European Commission monitors and assesses countries’ progress in implementing the European Small Business Act (SBA). SBA country fact sheets focus on key performance indicators and national policy developments related to the SBA’s 10 policy dimensions (European Commission, 2017). Estonia has developed an SME policy monitoring and evaluation system for its SME strategy 2014-2020, which includes a full quantitative evaluation every two years with the support of foreign experts under the responsibility of the Ministry of Economic Affairs and Communication. However, progress has been less significant in governments’ use of these advances to make the most rigorous assessments of policy effectiveness, and to use the results for continuous policy improvement. In short, the creation of an evaluation culture has yet to be widely established and significant challenges remain.
SME and entrepreneurship policies are frequently implemented without clear objectives. The objectives of the intervention are best framed in terms of the market or institutional failure the intervention seeks to address or the social benefit sought. Targets or key performance indicators can then be established against which the outcomes (intended and unintended) of the policy action can be monitored. In particular, more attention is needed to better understand the mechanisms through which policy will lead to benefits and to consider the potential unintended consequences that the policy may have (positive or negative). Appropriate data need to be collected and analysed, reflecting this understanding of potential consequences.
There is also room to improve the data collection systems and national statistical information available for SME and entrepreneurship policy monitoring and evaluation. Data should be available at appropriate time intervals and levels of disaggregation, and refer to an outcome indicator that is relevant for the foreseeable future. In some cases dedicated data collection exercises may be required, but in most cases the evaluator can rely upon existing data sources.
Rich and relevant data often exist within different parts of the administration, but remain unexploited for SME and entrepreneurship policy evaluation, e.g. in tax records or the unemployment registry (OECD, 2017b). Other sources of data outside public administration can be helpful. For example, in the SME space, the use of bank client data for evaluation is another promising area, e.g. as exploited in Coad et al (2013).
Legal barriers, a lack of incentives to make the data available or a lack of incentives to utilise the data for assessments can prevent their use. To address this challenge, some OECD countries have made major steps in recent years to broaden access to confidential data and to link data from different sources, such as Denmark, Norway and Sweden. France is making administrative data available through remote access to authorised researchers (see also OECD, 2017).
Looking ahead, “big data” collected with digital technologies holds promise for improving evaluation. Recently-developed methodological tools to analyse big data could become an important resource in the area of SME and entrepreneurship policies.
A further challenge is to ensure that account is taken of the interactions between the outcomes of different SME and entrepreneurship policies and programmes. Only in this way can informed judgements be made about potential adjustments to the policy mix; i.e. identifying programmes that merit expansion and programmes that merit contraction or abrogation. However, SME and entrepreneurship programmes are highly diverse. Some are expected to have an impact in the very short term (e.g. export facilitation), while others are unlikely to have an observable impact in less than a decade (e.g. innovation).
The impacts on SMEs and entrepreneurship of policies targeted at other areas need to be evaluated, too. Ministries of economy and industry commonly have the formal responsibility for leading and co-ordinating SME and entrepreneurship policies across government. However, expenditures in other ministries such as those responsible for finance, education, employment and infrastructure, strongly influence entrepreneurship and SME activity. These include policies in the areas of taxation, social security, business regulation, immigration, competition etc.
The impact of their policies on SME and entrepreneurship activity needs to be assessed, for example through using monitoring and evaluation evidence to support Regulatory Impact Assessments and the SME Test, and by creating cross-cutting groups within government to undertake evaluation and reflect on evidence from evaluations on the impact of these policies on entrepreneurship and SME development.
There has been an increase in the use of the most reliable and rigorous evaluation techniques, including for SME and entrepreneurship policy. New econometric techniques can correct for selection bias which can plague the evaluation of many of the types of support measures, e.g. through propensity score matching2. There has also been increased use of Randomised Control Trials (RCT), whereby a group of eligible recipients and their performance is compared over time with those eligible recipients who were randomly excluded in order to establish a counterfactual. A number of recent exemplar RCT evaluations have been undertaken in the area of SME and entrepreneurship policy, for example on management and workforce training in SMEs in the United Kingdom (Georgiadis and Pitelis, 2016); the subsidised entry of the unemployed into new business creation in Germany (Caliendo, M., Künn, S., & Weißenberger, M. 2016); and entrepreneurship training in the United States (Fairlie et al, 2015), etc.
However, high-quality evaluations remain relatively rare in the field of SME and entrepreneurship policy. For example, the US Government Accountability Office report for 2012 reviewed 53 entrepreneurship programmes across four different agencies with a budget of USD 2.6 billion. It reported that for 39 of the 53 programmes, the four agencies had either never conducted a performance evaluation or had conducted only one in the past decade (GAO, 2012). In addition, the UK National Audit Office concluded that none of the UK government evaluations in the field of business support provided convincing evidence of policy impacts (NAO, 2006).
Effective monitoring and evaluation requires a commitment to evaluation as an integral part of the policy-making process. Often evaluations are undertaken as individual exercises and not embedded in the policy cycle. A monitoring and evaluation culture should permeate all stages of policy design, implementation, and reform. This could be built for example through targeted training and partnership with independent evaluation agencies and academic institutions. The use of monitoring and evaluation evidence also requires space for policy experimentation and acceptance of failure.
What are the key areas for policy to consider?
The methodologies and data available for SME and entrepreneurship policy evaluation have improved dramatically over the last decade. However, widespread and systematic evaluation continues to be lacking. There are several examples of best practice evaluation, but little evidence of a comprehensive evaluation culture in this policy space. The OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes has established a six-step approach to monitoring and evaluation, where Step I (analysis of take-up) is the simplest methodology and step VI (methods which take into account selection bias) is the most complex methodology (Table 10.1) (OECD, 2007).
Table 10.1. Six Steps: Methods for assessing the impact of SME policy
|
Monitoring |
|
|---|---|
|
STEP I |
Take up of schemes |
|
STEP II |
Recipients Opinions |
|
STEP III |
Recipients’ views of the difference made by the Assistance |
|
Impact Assessment and Evaluation (note that these are not necessarily sequential) |
|
|
STEP IV |
Comparison of the Performance of ‘Assisted’ with ‘Typical’ firms |
|
STEP V |
Comparison with ‘Match’ firms |
|
STEP VI |
Taking account of selection bias |
Source: Based on OECD, 2007, OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes.
In addition, the following elements are important:3
Clear policy objectives: in practice many policies have only vague objectives, which makes evaluation difficult, particularly in cases where there are multiple objectives.
A complete overview of the full policy mix: it is important to have a clear understanding of the policy levers implemented and the potential interactions of the potential outcomes of different policies, as some instruments may be complementary on the one hand or reciprocally offsetting on the other hand.
Good data: poor data quality is sometimes the main reason why studies fail to find any statistically significant effect of evaluated policies. More and better measures can not only widen the scope of the evaluation, but also improve its precision.
Widening the focus beyond outcome: There are several other variables for policy makers to consider that could play an important role in explaining its effectiveness. These include the eligibility criteria, the targeted sample, the spatial unit of reference (e.g., regions or municipalities), how agents are informed about the policy, etc.
A commitment to evaluation as an integral part of the policy-making process: A monitoring and evaluation culture should permeate all stages of policy design, implementation, and reform.
The OECD Framework for the Evaluation of SME and Entrepreneurship Programmes and Policies (OECD, 2007) provides a guiding tool for monitoring and evaluation of SME and entrepreneurship policies and programmes.
Further Reading
Autio, E. and Ranniko, H. (2015), "Retaining winners: Can policy boost high growth entrepreneurship", Research Policy, 45(1): 42-55.
Banerjee, A., Karlan, D. and Zinman, J. (2015), "Six Rangomized Evaluations of Microcredit: Introduction and Further Steps", American Economic Journal: Applied Economics, 7(1): 1-21.
Caliendo, M. and Kritikos, A. (2010), "Start-ups by the unemployed: Characteristics, survival and direct employment effects", Small Business Economics, 35(1): 71-92.
Caliendo, M. Künn, S. and Weissenberger,M. (2016), "Personaility traits and the evaluation of start-up subsidies", European Economic Review, 86: 87-108.
Coad, A, Frankish, J.S., Roberts, R.G. and Storey, D.J. (2013), "Growth Paths and Survival Chances: An Application of Gamblers Ruin Theory", Journal of Business Venturing, 26(6): 615-632.
European Commission (2017), Annual Report on European SMEs 2016/2017, Bruxelles, http://ec.europa.eu/growth/smes/business-friendly-environment/performance-review
Fairlie, R. W., Karlan, D. and Zinman, J. (2015), "Behind the GATE experiment: Evidence on Effects of and Rationales for Subsidized Entrepreneurship Training", American Economic Journal: Economic Policy, 7(2): 125-161.
GAO (2012), Annual Report: Opportunities to Reduce Duplication, Overlap and Fragmentation, Achieve Savings, and Enhance Revenue, Report to Congressional Addressees, Washington DC.
Georgiadis, A. and Pitelis, C. N. (2016), "The Impact of Employees’ and Managers’ Training on the Performance of Small- and Medium-Sized Enterprises: Evidence from a Randomized Natural Experiment in the UK Service Sector", British Journal of Industrial Relations, 54(2): 409–421.
Loersch, C. (2014), "Business Start-Ups and the Effect of Coaching Programs", University of Potsdam
NAO (2006), Supporting Small Business, HC 962 Session 2005-2006 | 24 May 2006, London
OECD (2017), "Making policy evaluation work: The case of regional development policy", OECD Science, Technology and Industry Policy Papers, No. 38, OECD Publishing, Paris, https://doi.org/10.1787/c9bb055f-en.
OECD (2007), OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes, OECD Publishing, Paris, https://doi.org/10.1787/9789264040090-en.
Pons Rotger G., Gørtz, M., & Storey, D. J. (2012), "Assessing the effectiveness of guided preparation for new venture creation and performance: Theory and practice", Journal of Business Venturing, 27(4): 506-521.
Yusuf, J. E. (2012), "Meeting entrepreneurs’ support needs: are assistance programs effective?", Journal of Small Business and Enterprise Development, 17(2) (2010): 294-307.
Notes
← 1. It is important to distinguish between monitoring and evaluation. Monitoring programmes involves the direct collection of data from policy-makers and/or the recipients of the policy to give a qualitative view of the programme outcomes. It may also involve the collection of data from third-party sources, including business registries and administrative data. In contrast, evaluation is based on linking such outcomes with the specific characteristics of the policy or programme, taking into account the role of other factors which may influence monitored outcomes.
← 2. Examples can be found across the fields of pre-start business advice (Pons Rotger et al, 2012), business assistance programmes (Yusuf, 2012), high-growth firm support (Autio and Ranniko, 2015), subsidised entry of the unemployed into new business creation (Caliendo and Kritikos, 2010; Caliendo, Künn and Weissenberger, 2016), business coaching (Loersch, 2014), etc. Machine Learning techniques can also be useful to predict what policy interventions may be needed, for example in assessing which SMEs suffer credit constraints.
← 3. See OECD (2017) for a discussion in the context of regional policy evaluation.