Evaluation Guidelines for Representative Deliberative Processes

3.  Going beyond: Building on the minimum criteria

3.1 Wider impact and long-term effects

In addition to the criteria outlined in the previous section to evaluate the full cycle of a deliberative process, some evaluations consider potential wider impacts and long-term effects. Such evaluations take into account long-term changes in the attitudes and behaviours of members of deliberative processes, public officials, and/or the broader public over a longer period of time. As the focus of these guidelines is the minimum evaluation standards, detailed in Chapter 2 of this document, this section merely opens the discussion on evaluating wider impact and long term-effects, which is still an emerging practice.

To avoid delays in publishing the evaluation of the full cycle of a deliberative process, the wider impact and long-term evaluation report should be published at a later date as a separate document.

Evaluating broader impacts and long-term effects of deliberative processes gives a clearer estimate of the long-term value of public involvement in policy making. However, it requires additional time and financial resources, and proving causal effects presents additional challenges for evaluators.

Evaluating wider impact and long-term effects can be particularly useful when several deliberative processes have a cumulative impact, when the deliberative processes are large-scale and high-profile, and when structures and processes for public deliberation are ongoing.

The wider range of potential impacts includes changes to public attitudes and behaviour, long-run changes in the attitudes and behaviour of the deliberative process members, shifts in how public officials think and act, space created for civil society organisations, improved policy making, and changes in the logic of strategic actors in the political process.

Effects on the public

• Stable or rising levels of public trust in government

• Potential increase of voter turnout in elections

• Increased sense of political self-confidence

Empowering the members of deliberative process

• Increased democratic capacities (for example, self-expression, empathy, deliberation skills)

• Strengthened political knowledge and interest

• Increased open-mindedness and ability to appreciate complexity of perspectives

Effects on public officials and public bodies

• Serious consideration of and engagement with the recommendations given by members of deliberative processes

• Recognising the value of inviting everyday people to deliberate before reaching policy judgements

• Seeing a role for public deliberation in the policy-making process

• Increasing the uptake of deliberative processes

Creating space for civil society organisations

• Civil society organisations granted opportunities of access to a deliberative process to provide feedback on the design of the process, contribute with evidence and expertise

• Levelling the playing field and enhancing transparency and accountability of public, private, and civil society stakeholder influence on public decisions as undue influence on a public decisions is mitigated via deliberative processes

Improving policy making (policy formulation, implementation, outcomes)

• Policy outputs become more aligned with informed public judgement

• Wider range of voices heard in policy making

• Fewer policy failures due to more accurate policy responses

• More polarising issues addressed in respectful ways

Altering political strategy

• Issues raised by deliberative panels influence future legislative priorities and issue campaigns

• Stakeholder groups and public officials develop or test potential policy initiatives with deliberative processes to improve the quality of such proposals from the outset

Evaluation of these criteria can be done by employing the methods of interviews with public officials and members of a deliberative process, as well as longitudinal population surveys.

Annex A: Further resources provides more information on impact and long-term effect evaluation of deliberative processes.

Box 1. Examples: Assessing wider impact and long-term effects

Strengthened political knowledge and interest

Two case studies of representative deliberative processes, the Australian Citizens’ Parliament and Italy’s Iniziativa di Revisione Civica, were analysed to uncover the relationship between the process and the wider system and its deliberative capacity. The results show that these case studies have encouraged wider use of deliberative processes over the long term. Former members of a deliberative process have shown increased interest in political life and engagement in it. Public authorities were also inclined to promote the use of deliberative processes in the future by establishing a fund to further finance subsequent citizens’ participation councils.

Source: Felicetti, A., Niemeyer, S., & Curato, N. (2016). Improving deliberative participation: Connecting mini-publics to deliberative systems. European Political Science Review, 8(3), 427–448. https://doi.org/10.1017/S1755773915000119

Increased sense of political self-confidence

A two-wave panel survey of registered Oregon voters in 2010 and a second 2012 survey of registered Oregon voters were employed to investigate whether a Citizens’ Initiative Review (CIR) process in Oregon had an impact on the wider public. Results demonstrate that deliberative processes can support positive political attitudes among the members of the process, such as increased political efficacy, confidence in themselves and in democratic institutions. Similar effects were observed amongst the wider public, especially those who most utilise the published findings of the process. Knowing about the CIR taking place increased respondents’ external efficacy (belief in responsiveness of the government to citizens’ needs). Whereas those respondents who have read or even used the findings of the CIR felt an increase in internal efficacy (belief in their capacity to effectively participate in political life).

Source: Knobloch, K. R., Barthel, M., & Gastil, J. (2019). Emanating effects: The impact of the Oregon Citizens’ Initiative Review on voters’ political efficacy. Political Studies, 68, 426-445.  https://doi.org/10.1177/0032321719852254

3.2 Evaluating institutionalised structures and processes for public deliberation

The OECD Catching the Deliberative Wave report (2020) found that public authorities are increasingly looking for ways to make representative deliberative processes a more permanent and ongoing part of governance. The OECD guide Eight Ways to Institutionalise Deliberative Democracy (2021) has identified eight models for embedding public deliberation and civic lotteries in existing democratic institutions.

It is highly recommended to evaluate institutionalised structures and processes for public deliberation, as they are longer-term, larger-scale, and potentially have a bigger impact on decision-making, the public, and policymakers. Learning what works well and what requires improvement also allows for refinement and adaptation.

Overall, the criteria outlined in these guidelines are equally applicable to the evaluation of ongoing structures and processes for public deliberation. Some additional criteria can be added to capture the ongoing nature of these processes. Broadly, institutionalised structures and processes for public deliberation can be split in two groups:

1. Permanent or ongoing structures for representative citizen deliberation, such as within or adjoining the parliament or government. Such structures often have an agenda-setting power.

2. Ad hoc structures required in certain conditions, such as after a demand from the public, if a public decision is connected to a certain policy area, or requiring a specific threshold of spending. These processes are more demand-driven and dispersed in time and location, but are anchored in legislation or regulation that typically sets out the parameters and criteria for their establishment and implementation.

Keeping these differences in mind, there are additional criteria to consider when evaluating ongoing structures and processes for public deliberation.

Agenda-setting power

• All members of a particular group (the public/politicians/policy makers) have equal access to the possibility to initiate a deliberative process.

• Members of the public/policy makers/politicians have taken the opportunity to initiate a deliberative process.

Regularity and coherence

• Deliberative processes conform to the same standards across different instances/places/circumstances.

• Deliberative processes are initiated regularly.

Learning and capacity

• Learning mechanisms are in place to allow the commissioners/organisers of a deliberative process to pass on their lessons and experience to others organising such a process in the future.

• Guidance/training/support on the use of deliberative processes is provided when there are requirements to conduct deliberative processes under certain circumstances.

Rotation and retention

• In cases of a permanent or ongoing structure for representative citizen deliberation, the attrition rate of the members of the permanent body is low.

• When appropriate, an adequate and effective mechanism of rotation is provided for the members of the permanent body.