Evaluation Guidelines for Representative Deliberative Processes

Annex C. Member questionnaire

The following questionnaire is for the members of the deliberative body to respond to at the end of the process. Some of the questions could be asked both at the beginning and at the end of the process. These are marked with a symbol X2.

For longer, larger-scale deliberative processes it is recommended that all questions are included as a minimum evaluation. For smaller, shorter processes where administering the full questionnaire is not feasible, some questions could be omitted if less relevant. It is highly encouraged to keep the exact wording of the questions, as they have been designed following survey-writing standards to be non-leading and fair. Keeping the exact question wording also allows for comparability of evaluation results across the field.

To ensure they are non-leading, these questions should be included in a survey without headlines. Response options should be provided in a randomised order, where indicated.

It can be helpful to introduce a few demographic questions at the outset of the survey, such as gender, age, and some form of socioeconomic criteria. Answers to the survey should be confidential and members should have the right not to answer any questions. Survey results should be public, but analysed in aggregate, protecting individual responses.

This questionnaire was designed in consultation with the Expert Advisory group on Evaluating Representative Deliberative Processes, the Innovative Citizen Participation Network and the Democracy R&D Network. It was also informed by a comparative analysis of existing questionnaires designed for a similar purpose. As this is a new questionnaire yet to be tested in different contexts, we are interested in receiving feedback from those using it. Based on this feedback, an updated version of this questionnaire will be published.

1) Process design integrity

Clear and unbiased framing

1. Please describe the task you and other members were given in your own words. X2

2. Does the outcome of the deliberative process (such as the quality of the recommendations) match the expectations you had about this process when you received the initial invitation? Please explain.

3. To what extent, if at all, do you think that the task you were given allowed you to consider a narrow or a wide range of options for your recommendations? Please answer on a scale of 0 to 10, where 0 is “extremely narrow”, 5 is “just right” and 10 is “extremely wide”.

Suitable design

4. Do you think the length of the process was appropriate?

a) Yes

b) No, I think the amount of time was just right

c) No, I thought the process was too long

d) I’m not sure

5. If you consider the process needed more time, how much extra time do you think would have been useful?

a) Just a little bit more – a half day or less

b) At least one full day of deliberation

c) At least two-three more days of deliberation

d) A lot more time would have been useful – four days or more of deliberation

6. If you consider the process needed more time, how would you use the extra time? Please choose all relevant options. (randomised response order)

a) Hearing from more experts

b) Hearing from more stakeholders

c) Deliberating and weighing the different arguments before developing our recommendations

d) Developing our recommendations

e) Agreeing on the final wording of our recommendations

f) Having more/longer breaks

g) Other – please explain

7. To what extent, if at all, do you think that the time you had was well used to arrive at the final recommendations? Please answer on a scale of 0 to 10, where 0 means “not at all” and 10 means “extremely well used”.

8. Would it have been possible for you to have given more time to this process?

a) Yes

b) No

Transparency and governance

9. What is your understanding of what will happen next with the recommendations you will work on/were working on? X2

Representativeness and inclusiveness

10. How many of the other members did you feel had different views compared to your own?

a) None

b) A few of them

c) About half of them

d) Most of them

e) I don’t know

11. Did you feel there were any groups or parts of society that were not represented on this panel?

a) Yes

b) No

12. If you feel any groups or parts of society were not represented, which group or groups did you feel was/were missing?

13. Were there any obstacles that made it difficult for you to attend the sessions?

a) Yes

b) No

14. If there were obstacles that made it difficult for you to attend, what were they? Tick all that apply. (randomised response order for options a-e)

a) Barriers related to my personal life (for example, family commitments)

b) Barriers related to my work (for example, irregular working hours, busy schedule)

c) Financial barriers (for example, travelling costs)

d) The time this process demands

e) Yes, other barriers (please specify)

f) I don’t know

15. Do you have any suggestions of what could be done to improve the ability for anyone to attend such a process?

2) Deliberative experience

Neutrality and inclusivity of facilitation

16. How did you experience the balance between time spent in small group discussions and in plenary (whole group discussions) throughout the process? (randomised response order)

a) Too much time spent in small groups, not enough in plenary

b) Too much time spent in plenary, not enough in small groups

c) The balance between small groups and plenary was just right

17. To what extent did you feel that the facilitators were neutral or biased (favouring certain opinions or offering theirs)? Please answer on a scale of 0 to 10, where 0 means “completely neutral” and 10 means “very biased”.

Accessible, neutral, and transparent use of online tools

18. [If online tools used] To what extent did you find the online tools, such as [indicate the tool used] easy or difficult to use? Please answer on a scale of 0 to 10, where 0 is “very difficult” and 10 is “very easy”.

19. Did you receive sufficient technical support and equipment, if needed?

a) Yes, I received all the technical support and equipment I needed

b) Yes, I received some technical support and equipment but did not feel entirely supported

c) No, I felt I did not receive the technical support or equipment that I needed

d) I did not need technical support or equipment

20. Did you find that the online tools used did, or did not, were helpful to the process?

a) Yes

b) No

Breadth, diversity, clarity, and relevance of the evidence and stakeholders

21. To what extent do you feel that the information resources provided to you to help discussions were narrow or broad? Please answer on a scale of 0 to 10, where 0 means “the information provided was too narrow”, 5 means “the breadth of information provided was just right” and 10 means “the information provided was too broad”.

22. To what extent do you feel that the information resources provided, as a whole, neutral, with fair and diverse viewpoints represented? Please answer on a scale of 0 to 10, where 0 means “the information base felt very biased” and 10 means “the information base felt neutral with a large diversity of sources”.

23. Would you have liked to request presentations from additional experts or stakeholders beyond those lined up by the steering committee?

a) Yes

b) No

c) I did request presentations from additional experts

24. If you did request presentations from additional experts or stakeholders, were they called?

a) Yes, all additional experts requested were called

b) Yes, but only some of the additional experts requested were called

c) No, they were not called

25. If requested expert(s) could not attend, were you satisfied with the alternative expert(s)?

a) Yes

b) No

26. Were you able to request and obtain additional information beyond that which was initially provided to you by the organisers?

a) Yes, requested additional information, and it was provided

b) Yes, requested additional information, but it was not provided

c) Yes, we could request additional information, but we did not feel we needed any

d) No, we were not able to request additional information

e) I don’t know if we were able to request additional information

27. Did you find the evidence that was presented by the speakers easy or hard to understand?

a) I understood it easily from the beginning

b) Initially it was hard to understand, but by the end of the process I understood a lot of it much better

c) I found all of it hard to understand throughout

d) I don’t know

Quality of judgement

28. To what extent, if at all, do you feel that the final recommendations reflected the different views and judgements of the members? Please answer on a scale of 0 to 10, where 0 means “the diversity was not at all reflected” and 10 means “ultimately, our recommendations broadly satisfied the concerns of all members”.

29. To what extent, if at all, do you feel that the issue was discussed from a variety of perspectives (for example, considering underlying issues, existing structures, trade-offs values etc.)? Please answer on a scale of 0 to 10, where 0 means “from very limited number of perspectives" and 10 means “the issue was discussed from a wide variety of perspectives”.

30. To what extent do you feel that most members were providing justifications and explanations for their opinions? Please answer on a scale of 0 to 10, where 0 means “most members never provided justifications and explanations” and 10 means “most members always provided justifications and explanations”.

Perceived knowledge gains by members

Note: In addition to the questions below, knowledge gains can also be tested by asking a few factual questions about the policy issue that members tackled. Factual questions should be asked in the beginning and at the end of a deliberative process, in order to observe changes in the number of right answers.

Please answer all of the following questions on a scale of 0 to 10, where 0 means “not at all” and 10 means “to a great extent”. To what extent, if at all, do you feel that:

31. your understanding of the issue became clearer throughout the process?

32. you gained more arguments and perspectives to support your own opinion about the issue?

33. you understood the arguments, perspectives, and concerns of others?

34. your understanding of others’ opinions of the issue became clearer through this process?

35. On a scale of 0 to 10, where 0 means “not at all informed” and 10 means “very well informed”, to what extent, if at all, do you feel that you are informed at the moment on the following issues: (include a list of issues relevant to the key policy issue addressed) X2 (or more)

Accessibility and equality of opportunity to speak

Please answer all of the following questions on a scale of 0 to 10, where 0 means “not at all” and 10 means “to a great extent”. To what extent, if at all, do you feel that:

36. you had a fair number of opportunities to speak?

37. other members had a fair number of opportunities to speak?

38. all members were heard equally?

39. no members dominated the discussions?

40. you and your views were heard?

Respect and mutual comprehension

41. fellow members respected what you had to say, even when they didn't agree with you?

Free decision making and response

42. organisers, experts, or steering group members expressed their own views during members’ deliberation?

43. Imagine you are the decision-maker that convened this process. Would you implement the recommendations the deliberative process produced?

a) Yes, all of them

b) Yes, the vast majority (over 75%)

c) Yes, about half

d) Yes, some (between 25-50%)

e) Only a few (between 1-25%)

f) No, none of them

Please answer all of the following questions on a scale of 0 to 10, where 0 means “not at all” and 10 means “to a great extent”. To what extent, if at all, did you feel:

44. pressured to agree with ideas or arguments of others?

45. that your contributions made it into the recommendations?

Respect for members’ privacy

46. To what extent, if at all, do you feel your privacy (from undesired attention) was protected in this process? Please answer all of the following questions on a scale of 0 to 10, where 0 means “not at all” and 10 means “to a great extent”.

47. Did you have a choice or not about your identity being revealed at the end of the process?

a) Yes

b) No

48. Have you been approached by someone and offered more information or invited to exchange privately?

a) Yes

b) No

49. If you were offered more information or invited to exchange privately, who approached you?

a) Someone working for the media

b) Someone working for a private company or agency

c) Someone working for a non-governmental organisation

d) Other (please specify)

e) I don’t know

50. Do you know if any other members of the group received approaches from media/interest groups?

a) Yes, they did

b) I suspect they did

c) No, as far as I am aware

d) I don’t know