Evaluation Guidelines for Representative Deliberative Processes

Reader’s guide

Key terms

• Representative deliberative process: A process in which a broadly representative body of people weighs evidence, deliberates to find common ground, and develops detailed recommendations on policy issues for public authorities. For shorthand, representative deliberative processes are often referred to as deliberative processes in this document. Common examples of one-off processes are Citizens’ Assemblies, Juries, and Panels. There are also examples of institutionalised deliberative bodies, such as Agenda-setting Councils.

• Members of a deliberative process: The people selected via civic lottery to form a broadly representative group and take part in a representative deliberative process. Together, they form what is referred to as the deliberative body.

• Deliberation: Weighing evidence and considering a wide range of perspectives in pursuit of finding common ground. It is distinct from:

  • Debate, where the aim is to persuade others of one’s own position and to ‘win’, 

  • Bargaining, where people make concessions in exchange for something else, 

  • Dialogue, which seeks mutual understanding rather than a decision, 

  • and “Opinion giving,” usually witnessed in online platforms or at town hall meetings, where individuals state their opinions in a context that does not first involve learning, or the necessity to listen to others.

• “Rough consensus”: The aim is to find (as much as possible) a proposal or options that a large proportion of members can at least live with. When voting is used, it is either an intermediate step on the way to rough consensus, or a “fall back” mechanism when consensus cannot be reached.

• Civic lottery: A process used by public authorities to convene a broadly representative group of people to tackle a policy challenge. It is based on the ancient practice of sortition, which has a history ranging from Ancient Athens to the Doge of Venice. Today, it is used to select the members in Citizens’ Assemblies and other deliberative processes. The principle behind a civic lottery is that everyone has a more or less equal chance of being selected by lot. There are two stages to a civic lottery. First, a very large number of people, chosen by lot, receives an invitation to be part of the process from the convening public authority. These randomly selected recipients can volunteer by opting in to the lottery. Then, amongst the volunteers, members are chosen by lot to be broadly representative of the public. Civic lotteries aim to overcome the shortcomings and distortions of “open” and “closed” calls for participants, which result in non-representative groups of people who do not mirror the wider population and attract those with the most interest or stake in the issue. (For greater detail, see Chapter 4 in OECD, 2020a.)

Evaluation of a representative deliberative process: The structured and objective assessment of the design, implementation, and results of a one-off or institutionalised representative deliberative process. More specifically, in this document evaluation refers to ex-post evaluation in a broader sense (throughout or after the deliberative process) as opposed to ex-ante evaluation (assessing the opportunity to initiate a deliberative process in a specific situation).

How to use these guidelines

When conducting an evaluation using these guidelines, the aim is to assess a particular process and to use that evaluation to then improve the implementation of future deliberative processes. Evaluations use standard metrics (such as the member and organiser questionnaires) to gather reliable evidence and provide constructive feedback to commissioners and organisers, whether the overall evaluation is critical or positive.

These guidelines consist of three chapters. Chapter 1 outlines five principles of evaluation and discusses key elements of planning and designing for evaluation. Chapter 2 introduces the three-step evaluation cycle, framework, criteria, and measurement methods. Chapter 3 opens the discussion on the wider impact and long-term effects, as well as evaluating institutionalised structures and processes for public deliberation.

These guidelines should be used by implementing the following four steps.

1. Decide on the principles and logistics (timing, resources, staffing) of the evaluation based on guidance provided in Chapter 1 of the guidelines.

2. Set an evaluation framework, key criteria, and assessment methods based on guidance provided in Chapter 2 of the guidelines.

2a. Building on the minimum evaluation guidelines

These guidelines provide a minimum set of evaluation criteria. They should be used as a foundation rather than as a comprehensive framework, and they should be adapted to the specific context in which the deliberative process is taking place. Policy makers are encouraged to use these evaluation criteria in their entirety and to add additional criteria based on their specific context and needs. This can be done by identifying specific evaluation needs via conversations with the public authorities commissioning the process and the practitioners implementing it.

2b. Taking into account the full participation strategy

When a deliberative process is part of a broader participation strategy on an issue, it is important to take into account that larger context. Evaluators can examine how the deliberative process fits into that strategy. For example, does the deliberation complement other participation efforts that aim to capture broader public opinion?

2c. Adapting the questionnaire included in these guidelines for your evaluation needs

These guidelines include two questionnaires. One is for members of a representative deliberative process, as part of the evaluation process. The other is for the process organisers.

3. Consider if it is feasible to evaluate long-term impacts, occurring over a relatively long period of time after the completion of a representative deliberative process. If applicable, adapt the evaluation framework to institutionalised structures and deliberative processes based on guidance provided in Chapter 3 of the guidelines.

4. Find helpful resources and examples of good practice for evaluating representative deliberative processes in Annex A: Further resources.