Evaluation Guidelines for Representative Deliberative Processes

Annex F. Table of comparison of existing frameworks of evaluating representative deliberative processes

Table F.1. Comparison of evaluation criteria of existing frameworks of evaluation for deliberative processes

Democracy in Motion: Evaluating the Practice and Impact of Deliberative Civic Engagement (2012)		OECD Good Practice Principles (2020)	Deliberative Processes in Practice (2016)	Deliberating Competence: Theoretical and Practitioner Perspectives on Effective Participatory Appraisal Practice (2008)	A Three-Stage Evaluation of a Deliberative Event on Climate Change and Transforming Energy (2008)	The Co-creation radar: a comprehensive public participation evaluation model (2019)	Involve (2021)	Jefferson Centre: Citizens Jury Handbook (2004)	Democracy Matters: Lessons from the 2015 Citizens' Assemblies on English Devolution (2016)	Evaluation of the Scottish Parliament’s Citizens’ Panels on Primary Care (2019)	Innovation in Democracy Programme Evaluation (2020)
Criteria of evaluation		Comparison
Context evaluation (cultural/political contextual analysis in which deliberative civic engagement takes place)						X
1) Design integrity
Unbiased framing	Transparent process by which issues are framed for deliberation	X	X	X		X
Procedural design involvement	Deliberative procedures developed in consultation with interested parties				X
	Resulting process in line with best practices of deliberation (learning, deliberation, decision-making)	X		X	X	X		X
Representativeness	Equal opportunity to participate	X	X		X	X
	Final group - representative of general population	X		X	X	X			X	X	X
	Inclusivity: presence of permanent minorities & identity groups	X		X		X			X
2) Democratic deliberation and judgements
Deliberative analytic process (evaluation of the talk that takes place)	Create a solid information base	X	X	X	X	X	X		X	X	X
	Prioritize the key values at stake				X
	Identify a broad range of solutions			X	X		X
	Weighing Pros and Cons, trade-offs	X		X	X				X	X
Democratic social process (evaluation of the social component of deliberation)	Equality of opportunity to speak/ adequately distributed speaking opportunities	X	X	X	X		X	X	X	X
	Ensuring mutual comprehension	X		X	X
	Consideration of other ideas and experiences	X	X	X	X		X		X		X
	Respect of other participants				X		X		X	X	X
Sound judgement (evaluation of the quality of the final decision or judgment)	Final decision making is uncohercive, uses one of any possible democratic voting systems and decision rules (consensus, majority rule, proportional outcomes etc.)
	Citizens' judgments become more enlightened as deliberation progresses				X		X		X	X
	After deliberation participants demonstrate more informed and coherent views, provide reasoning and explain arguments underlying alternative points of view					X	X		X	X
3) Influential conclusions and/or actions
Influential recommendations	When a clear majority of panellists favours a policy initiative, its chances of prevailing amongst policy makers should in crease	X			X	X					X
Effective, coordinated action (for those deliberative processes that attempt direct action)	Deliberative body is able to coordinate their post deliberative efforts				X	X
4) Long-term effects on public life
Transforming public attitudes and habits	Stable or rising levels of public trust, signs of reduced civic neglect						X		X
Potential increase of voter turnout in elections
Increased sense of efficacy/political self-confidence						X		X	X	X
Changing public officials’ attitudes/behaviour	Developing more favourable views of the judgments citizens make during deliberative events
	Awareness of the importance of citizen deliberation									X
	Changed attitudes toward the prospects of deliberation
Altering strategic political choices	Initiative and policy campaigns focus more on addressing issues raised by deliberative panels
	Routine pilot-testing of potential policy initiatives with deliberative processes
Additional criteria		Process design and all other materials made public; response to recommendations and monitoring of their implementation; respect for participants’ privacy; evaluation plan.	Publicity of outcomes and their rationales; existence of process for appeal, review/iteration.	Efficiency - process is cost-effective and timely.	In addition: group ownership of the agenda of deliberative process; the process was documented thoroughly	Appropriate and accessible digital tools used; what type of evaluation was planned; balance of quality of implementation and resources used; effects on the organisation commissioning/organising participation: increased skills & capacities.	Participant support for the use of deliberative process increased as a result.		Broader community engagement (opportunities for structured public engagement-public hearings, discussions etc). Truthfulness - participants were speaking what was truly on their mind. Unbiased facilitation - facilitations did not put forward their views. Influence of one or more participants over others - especially in deliberative processes where politicians take part. Process is fun.		Framing of the question - sufficiently challenging, carrying enough viable options to foster debate and deliberation. Transparency and communication about the purpose, activities and outcomes to a wider public.

Source: OECD Good Practice Principles (2020), Deliberative Processes in Practice (2016), Deliberating Competence: Theoretical and Practitioner Perspectives on Effective Participatory Appraisal Practice (2008), A Three-Stage Evaluation of a Deliberative Event on Climate Change and Transforming Energy (2008), The Co-creation radar: a comprehensive public participation evaluation model (2019), The Co-creation radar: a comprehensive public participation evaluation model (2019), Involve participant survey (2021), Jefferson Centre: Citizens’ Jury Handbook (2004), Democracy Matters: Lessons from the 2015 Citizens' Assemblies on English Devolution (2016), Evaluation of the Scottish Parliament’s Citizens’ Panels on Primary Care (2019), Innovation in Democracy Programme Evaluation (2020)