In 2008 and 2011, the OECD carried out cross-country household surveys designed to shed light on environmental behaviours in the domains of energy, waste, transport and food, as well as on how government policies affect these behaviours (OECD, 2013[1]; OECD, 2011[2]). The third round of the survey on Environmental Policy and Individual Behaviour Change (EPIC) was implemented in 2022 in nine countries: Belgium, Canada, France, Israel, the Netherlands, Sweden, Switzerland, the United Kingdom and the United States. Details on the project timeline are provided in Table A B.1.
How Green is Household Behaviour?
Sustainable Choices in a Time of Interlocking Crises
OECD Studies on Environmental Policy and Household Behaviour
Annex B. Survey methodology and sample statistics
Survey implementation
Timeline and selection of survey provider
Table A B.1. Project timeline
In June and July 2022, 1 800 households were surveyed in each of the nine participating countries using an internet-based questionnaire.1 Due to length constraints, respondents were randomly assigned to complete two of the four thematic sections in addition to the sections on household characteristics and environmental attitudes, which were completed by every household. This resulted in 900 observations per thematic domain per country (1 635 in the United States) and a total targeted sample of 8 835 observations in each of the four thematic areas: residential energy use, waste generation and recycling, personal transport choices and food consumption (Table A B.2).
Table A B.2. Targeted sample sizes
Selection of survey provider
The OECD released a Call for Tender to select a survey provider specialised in the implementation of large international web-surveys using online consumer panels in different countries. As in previous rounds of the survey, an internet-based survey was preferred due to lower cost per respondent and ease of implementation across multiple OECD countries. Survey providers were assessed based on their demonstrated ability to obtain a representative sample with the assessment criteria focusing on panel size, panel quotas, panel recruitment and management, experience with discrete choice experiments, online survey design abilities, responsiveness and value for money. Based on these criteria, the OECD selected Ipsos N.V. to implement the survey. The tasks performed by the survey provider included hosting and programming the online questionnaire, recruiting a nationally-representative sample of respondents from established online panel members, and collecting and cleaning the data.
Questionnaire design and pilot testing
The 2022 OECD EPIC survey questionnaire is composed of six sections. Two sections elicit socio-demographic and attitudinal characteristics, and four sections elicit information regarding household behaviour in the environmental areas of interest: waste generation and recycling, personal transport choices, residential energy use and food consumption. The questionnaire is composed of approximately 90 closed questions with a combination of binary, ordinal and Likert scale questions. Each of the four thematic sections also includes a discrete choice experiment.
The questionnaire was developed based on previous questionnaires and with the input from a Steering Group, a Scientific Advisory Committee, and an Internal Coordination Group. The Steering Group was composed of government representatives from the countries involved in the survey, the Scientific Advisory Committee included subject matter and methodological experts from academia, and the Internal Coordination Group involved other relevant OECD parties, specifically the Trade and Agriculture Directorate, the International Energy Agency and the International Transport Forum.
Although some continuity exists in survey questions with respect to previous rounds of the survey,2 changes were made to the 2022 version of the survey in order to account for evolution in environmental, economic, and political context over the past ten years. Questions about satisfaction with the local environment as well as importance of green space were included in the section on attitudes. More questions in the energy section focused on consumption reducing behaviour while the discrete choice experiment (DCE) asked about renewable energy which was the focus of previous survey rounds. The transport section, for example, added questions on electric vehicles as well as other transport options such as bikes and scooters. The waste section now includes questions on composting, electronic equipment and food waste as well as payment schemes for waste collection. The section on food systems was broadened in scope to include a variety of food consumption habits beyond organic food. In order to reduce cognitive burden, questions were simplified and shortened when possible.
The inclusion of discrete choice experiments is an important methodological novelty in the 2022 round of the survey. In DCEs, subjects are asked to make hypothetical choices by selecting a preferred alternative among a menu of options (Bateman et al., 2002[3]; OECD, 2018[4]). Stated preference data generated by discrete choice experiments allows for an estimation of how much respondents value the various characteristics of the alternatives presented. These characteristics could pertain to products (e.g. the price or GHG emissions of energy supply), actions (e.g. convenience of using disposable vs. refillable containers) or elements directly impacted by environmental policies (e.g. the cost and convenience of owning an electric vehicle).
The DCEs were also designed with input from the Steering Group, Scientific Advisory Committee, Internal Coordination Group, as well as leading researchers in choice modelling. Details regarding each experiment are provided in each thematic section. The general methodology followed in the design of the DCEs was the same across thematic sections. The first step in designing the DCEs was to characterize clearly the decision problem including the scope of potential changes in attributes and the types of values that are associated with these changes. Next, attributes and attribute levels were selected based on their credibility, ease of comprehension, and estimation properties. Where applicable, attribute levels are country- or respondent-specific to ensure realistic scenarios.
Pilot studies were implemented prior to fielding the survey in order to test the appropriateness of the selected attributes and their levels, as well as to ensure that the variation of attribute levels was adequate for identifying preference parameters in the choice model. Once the attributes, levels and number of alternatives were determined, statistical design theory was used to combine levels of the attributes into choice sets. An orthogonal fractional factorial design was used to determine the combination of attribute levels that appeared in the choice sets for the first pilot study. The preference parameters estimated from the pilot data were then used as prior estimates to inform a statistically efficient (D-efficient) design based on the predicted standard errors of parameter estimates. If any information on priors is known, these types of designs always outperform orthogonal designs (ChoiceMetrics, 2018[5]). To increase the robustness of the design to the misspecification of priors, a Bayesian efficient design specifying parameter distributions, rather than point estimates, was also employed. In addition to D-efficiency, attribute balance was an important criterion that was maintained in the design of the choice sets. When more than six choice scenarios were required in order to identify the econometric model, questions were blocked into groups so and randomly assigned to respondents so that each respondent saw six scenarios per thematic area.
A first pilot survey of 540 respondents run in April 2022 was used to refine the questionnaire. The pilot study was implemented in all participating countries and allowed for the identification of difficult questions based on the average completion time required per item. The responses from discrete choice experiments were used to optimise the choice sets for the second pilot study. The second pilot was implemented in May 2022. Additional changes made to the questionnaire following the second pilot study included further refinements to the choice sets based on updated preference parameter estimates as well as revisions to further improve readability of complex .
Survey technology and translation
The online questionnaires were programmed such that they could be completed standard web browsers on a variety of devices, including cell phones. The platform permitted a variety of response formats for different questions. For example, respondents could use a sliding bar to indicate how likely they thought specific events were, or how much they would support a given policy. The target median completion time for the online survey was approximately 20 minutes. Once the online questionnaire was programmed in British English, it was translated into Dutch, French, Swedish, Spanish, Italian, German, Hebrew and Arabic, with 15 distinct versions created once country-specific vocabulary (e.g., American and Canadian English versions), currencies and systems of measurement were taken into account. Members of the Steering Group were invited to conduct final checks on the translated questionnaires.
Respondent targeting, recruitment and quota sampling
The target sample consists of the general public aged 18 or over who had either full or partial responsibility for household expenses (such as utility bills or car and appliance purchases). Respondents were recruited from Ipsos’ in-country non-probability/volunteer online access panels based on their profile data (age, gender and region) and pre-defined sub-sample size. To ensure representativeness in the sample, quotas were set for age, gender, geographic region and income (see section below for how quota targets were set). When quotas were filled, respondents with these characteristics were stopped from completing the questionnaire. Panellists who were selected on the basis of these characteristics received e-mails inviting them to respond to the survey. No mention was made of the topics addressed in the questionnaire. To promote participation in the survey, panel members receive reward points for participating in a survey. The number of points allocated to each survey varies according to a number of factors, notably the length and complexity of the questionnaire. Potential respondents who started the questionnaire were asked whether they met the screening criteria (influential in household financial decisions). If they did not meet the criteria, they were thanked for their time and screened out of the sample. Despite rigorous efforts at stratification and quota sampling, it is important to acknowledge that there may be some respondent characteristics that were not observed and which correlate with internet use. This correlation of unobserved characteristics could introduce a selection bias in the sample. It is therefore recommended that researchers drawing conclusions from these data carefully consider how this selection bias based on internet use could affect their results.
Quality Control
Interview quality was closely monitored during the fieldwork. Completed interviews were checked for speeders, straight liners, and item non-response. Speeders were defined as respondents who completed a given section in less than half of the median survey length of that section. If respondents were flagged for at least two of the three quality checks, they were removed from the final dataset. Table A B.3 describes the screening process carried out by the survey provider.
Table A B.3. Number of interviews classified as bad quality during fieldwork
The OECD carried out a further quality control of the data based on a speeding criteria only, excluding an additional 506 respondents, or 2.8% of the remaining sample. These respondents completed the survey sections in less than one third of section- and country-specific median completion times. The total final sample size is 17 216 respondents.
Response times and drop-out rates
Table A B.4 shows the drop-out rates by country. The drop-out rate is calculated as the fraction of respondents who started the questionnaire but then did not complete it. Potential respondents who were removed because of quotas or were screened out of the questionnaire after the screening question are not included in the calculation.3 The overall drop-out rate was 26%.
Table A B.4. Screened, drop-outs and completed surveys by country
Note: The drop-out rate is calculated as: (Abandoned the survey)/(Abandoned the survey + Completed).
Quota targets and weighted sample statistics
To ensure a representative sample and avoid sample bias, the sample was stratified by income, age, gender and region in each of the nine countries. Sample quotas were established based on population data from official statistic sources.4 Income was stratified by after-tax income quintiles. Age was stratified using the following groups: 18-24, 25-34, 35-44, 45-54 and 55 years and over. Gender was approximately half male and half female, with slightly more females in some countries. Region was stratified by as few as four regions in the Netherlands and Belgium and over 12 for the UK and France. The survey allowed for the collection of a unique dataset of more than 17 670 households in nine countries.
To correct the imbalances between the targets on age, gender, region and income, weighting factors were calculated to ensure representativeness at the population level. Post-stratification weights were calculated on a country-by-country basis based on age, gender, region and income variables. For respondents who did not report income (approximately 11% of the sample), income was imputed using a multinomial logit model. Table A B.5 compares weighted sample proportions to target sample proportions.
Table A B.5. Quota targets relative to weighted sample
1. The weighted sample proportions do not sum to 100 because respondents also had the option to respond “other” and “prefer not to say.”
Country-level sociodemographic characteristics not used for quota sampling
Figure A B.1. Residential location
Source: OECD (2022), Environmental Policies and Individual Behaviour Change Survey.
Figure A B.2. Residential status
Source: OECD (2022), Environmental Policies and Individual Behaviour Change Survey.
Table A B.6. Education
Proportion of the population reporting having achieved higher education
Figure A B.3. Employment
Source: OECD (2022), Environmental Policies and Individual Behaviour Change Survey.
References
[3] Bateman, I. et al. (2002), “Economic valuation with stated preference techniques: a manual.”, Economic valuation with stated preference techniques: a manual..
[5] ChoiceMetrics (2018), “Ngene 1.2 User Manual and Reference Guide”, http://www.choice-metrics.com (accessed on 5 August 2022).
[4] OECD (2018), Cost-Benefit Analysis and the Environment Further Developments and Policy Use: PART I Discrete choice experiments, https://www.oecd-ilibrary.org/environment/cost-benefit-analysis-and-the-environment/discrete-choice-experiments_9789264085169-8-en (accessed on 17 February 2020).
[1] OECD (2013), Greening Household Behaviour: Overview from the 2011 Survey, OECD Studies on Environmental Policy and Household Behaviour, OECD Publishing, Paris, https://doi.org/10.1787/9789264181373-en.
[2] OECD (2011), Greening Household Behaviour: The Role of Public Policy, OECD Studies on Environmental Policy and Household Behaviour, OECD Publishing, Paris, https://doi.org/10.1787/9789264096875-en.
Notes
← 1. 3 270 households were surveyed in the United States.
← 2. Note that samples differ across survey rounds.
← 3. The drop-out rate is calculated as the proportion of those who passed the screening question and began the survey, but left before it was completed.
← 4. Eurostat 2020, American Community Survey (ACS), Statistics Canada Population estimates, 20-07421901 NATO Public Opinion MENA.