The OECD report Innovative Citizen Participation and New Democratic Institutions: Catching the Deliberative Wave (2020) demonstrated that public authorities from all levels of government increasingly turn to Citizens' Assemblies, Citizens’ Juries, and other representative deliberative processes to tackle complex policy problems. As the use of such processes increases, so does the need to determine and ultimately improve their quality. The purpose of this document is to help public authorities initiate and develop better representative deliberative processes by establishing a minimum standard for their evaluation. These guidelines provide policy makers and practitioners with an evaluation framework and methodology, as well as evaluation questionnaires.
Evaluation Guidelines for Representative Deliberative Processes
Abstract
Executive Summary
These are guidelines for policy makers, evaluators, and practitioners who want to evaluate the representative deliberative processes they initiate, commission, and implement. The guidelines establish a minimum standard for evaluation by providing rationales, a framework, measurement methods, and evaluation questionnaires. The evaluation of deliberative processes is a key element of their success. Timely evaluation strengthens the trust of policy makers, the public, and stakeholders in recommendations developed by a deliberative body, as it can demonstrate the quality and the rigour involved in generating them. By making a process subject to evaluation, the authorities commissioning it demonstrate a commitment to transparency and quality, earning them greater legitimacy. Evaluation also creates opportunities for learning by providing evidence and lessons for public authorities and practitioners about what went well and what did not.
Independent evaluations are the most comprehensive and reliable way of evaluating a deliberative process. For smaller and shorter deliberative processes, evaluation in the form of self-reporting by members and/or organisers of a deliberative process can also contribute to learning. With the Advisory Group on Evaluating Representative Deliberative Processes, the OECD has developed principles that can help guide an evaluation and ensure its quality and integrity, summarised as follows:
Maximum degree of independence of evaluation should be ensured, appropriate to the scale and length of a deliberative process.
The selection of the evaluators and the evaluation process itself should be clear and transparent.
Evaluations should be based on valid and reliable data, collected through a variety of methods, such as surveys, interviews, observation, and a review of materials used.
Evaluators should have access to sufficient financial resources and all necessary information required to assess a deliberative process.
The evaluation should be constructive and focus on quality and impact.
A comprehensive evaluation comprises three essential steps: evaluation of the process design integrity; the deliberative experience; and the pathways to impact of a deliberative process. Evaluation criteria has been identified for each step. Possible approaches and measurement methods to assess how a process meets the criteria include: member survey; public survey; organiser or expert witness survey; document review; deliberation observation; open-ended interviews; media coverage review, and policy analysis. Some evaluations consider wider impacts and long-term effects. The wider range of potential impacts includes changes to public attitudes and behaviour, long-run changes in the attitudes and behaviour of the deliberative process members, shifts in how public officials think and act, space created for civil society organisations, improved policy making, and changes in the logic of strategic actors in the political process.
It is highly recommended to evaluate the increasingly prevalent institutionalised structures and processes for public deliberation, as they are more durable, larger in scale, and potentially have a greater impact on decision making, the public, and policy makers. Additional evaluation criteria can be added to capture the ongoing nature of these processes.
