Behavioural Insights and Organisations

4. Findings from experiments on fostering a culture of safety in Canada, Ireland, Mexico and Oman

Abstract

This chapter presents results from a two-phased online experiment on applying behavioural insights to safety culture with regulators and regulated entities from Canada, Ireland, Mexico and Oman. It will discuss the key findings of these results for each country.

The preceding chapters introduced the theory behind applying behavioural insights (BI) to changing the behaviour of organisations, as well as explained the origins and initial scoping exercise conducted to understand how this theory could be applied to fostering elements of safety culture in the energy sector. These findings were complemented with case studies with five regulators to demonstrate how regulators currently address safety culture in their respective countries. Each of these pieces were discussed in the OECD Network of Economic Regulators (NER), which is a body of over 70 front-line regulators from around the world and has been central in developing research on applying BI to regulatory policy since 2013 (Lunn, 2014[1]); (OECD, 2016[2]); (OECD, 2017[3]); (OECD, 2017[4]).

The results of this collaboration with the NER community, academics and behavioural practitioners were a strong understanding of the enablers and barriers in safety in the energy sector, as well as insights into how behavioural science could support better outcomes in this field. Under the auspices of the NER and with the support of the Government of Canada (Natural Resources Canada and Canadian Energy Regulator, CER ); Ireland’s Commission for Regulation of Utilities (CRU); Mexico’s Agency for Safety, Energy and Environment (ASEA); and Oman’s Authority for Electricity Regulation (AER), a set of experiments were developed and implemented to test the application of BI to safety in the energy sector. This was accomplished in two phases: first, a survey-based experiment was conducted with the four project partners that tested perceptions of safety and effectiveness of behavioural vignettes and, second, a follow up survey-based experiment with Ireland’s CRU that tested how to improve conformity with safety regulations amongst small-scale gas and electricity installers.

These experiments constitute one of the first applications of BI to the study of safety, elements of safety culture, and the improvement of safety outcomes in the energy sector. The findings presented below enhance the practical understanding of the application of BI to organisational behaviour and pushed the frontier towards a more frequent integration of the field of BI and safety. As this project was in itself an innovative and experimental undertaking, results point to the need for further research and greater understanding in many areas, including a better understanding of contextual, sub-national, and group drivers. Furthermore, relatively small sample sizes for Canada and Ireland in phase one lead to caution interpreting results and the call for further research, especially in Canada who did not receive a follow up experiment compared to Ireland in Phase 2.

This chapter presented an overview of the methodology, as well as the results and key findings from these two phases. For phase one, the methodology and initial results were presented in (OECD, 2019[5]), which focused on the high-level horizontal findings for regulatory policy in the energy sector. This chapter will provide a brief summary of (OECD, 2019[5]) and complement these findings with additional country-based analysis and results to understand more clear what worked, and what did not, for each project partner. As phase two was still underway when (OECD, 2019[5]) was published, the second section of this chapter will present the full methods and results from this second experiment.

Context: Countries involved in the BI experiments on safety

As noted above, the overall project was conducted in partnership with energy regulators from Canada, Ireland, Mexico and Oman. As each country and sector possess particular characteristics with regards to how they regulate their sectors, it is worth describing the context and basic features of each regulator included in the analysis. This discussion was originally contained within (OECD, 2019[5]) and is replicated here as it is relevant for the proceeding discussions on methods, results and key findings. These countries were selected based on convenient sampling.

Canada

The Canadian Energy Regulator (CER), formerly the National Energy Board (NEB), is Canada’s energy and safety regulator. It makes regulatory decisions and recommendations that represent the interests and concerns of Canadians. In doing so, the CER factors in economic, environmental and social considerations. The CER oversees safety and environmental protection for the full life-cycle of a project – from approval to construction, operation, abandonment and works with communities, sharing the goal of making energy infrastructure as safe as it can be. The CER also monitors aspects of energy supply, demand, production, development and trade which the federal government controls. The CER reports to Parliament through the Minister of Natural Resources.

Ireland

The Commission for Regulation of Utilities (CRU) has responsibility for safety in the energy sector in three broad sectors:

• Regulating the activities of natural gas and liquid petroleum gas (LPG) undertakings with respect to safety under the Energy (Miscellaneous Provisions) Acts (2006[6]) and (2012[7]). This is carried out under the Gas Safety Framework, which covers shipping, supply, storage, transmission, distribution and use of natural gas, as well as certain specified LPG undertakings.

• Regulating upstream petroleum safety, including offshore safety under the Petroleum (Exploration and Extraction) Safety Acts, 2010 and 2015. This is carried out under the Petroleum Safety Framework (PSF) Requirement of the Petroleum Safety Framework (CER, 2017[8]).

• Designation and oversight of the safety supervisory bodies charged with monitoring natural and liquid petroleum gas installers and electrical contractors doing domestic gas and electrical works respectively, with respect to safety under the Energy (Miscellaneous Provisions) Acts (2006[6]) and (2012[7]) .

Mexico

Created in 2015, the Agency for Safety, Energy and Environment (Agencia de Seguridad, Energía y Ambiente, ASEA) is a technical regulator responsible for industrial and operational safety and environmental protection in Mexico’s hydrocarbons sector (OECD, 2017[3]). It oversees activities throughout the hydrocarbons value chain, from exploration and extraction to midstream and downstream transformation, production and storage as well as distribution and retail at the petrol station level. ASEA’s aims are mapped under five dimensions (clients; industry; process; organisation and learning; and financial resources) and within each of these dimensions, there are medium- to long-term visions.

Oman

The Authority for Electricity Regulation (AER) is responsible for regulating the electricity sector and some aspects of the water sector. It was established by Article 19 of the Law for the Regulation and Privatisation of the Electricity and Related Water Sector promulgated by Royal Decree 78/2004 on 1 August 2004 and Amended by Royal Decree 59/2009 and 47/2013 (“the Sector Law”). The authority is a financially and administratively independent organisation and reports directly to the Council of Ministers. The authority’s duties under the Sector Law are to protect the interests of its three main stakeholders: electricity customers, electricity sector companies, and the Government.

Phase 1: Cross-national experiment with Canada, Ireland, Mexico and Oman

As discussed in previous chapters, as well as in (OECD, 2019[5]), the departure point for these experiments were two-fold. First, while individual-level errors, such as inattention, forgetfulness and procedural violations, have long been regarded as principle factors behind safety incidents and disasters (Reason, 1990), there is also an understanding safety can often be linked to the organisational conditions within which the individuals work. The perception of these conditions are a fundamental driver in safety. Second, “safety culture,” defined as the set of “shared values, beliefs, attitudes, norms and practices related to safety within an organisation” (TRB, 2016[9]); (Cooper, 2000[10]) is an important aspect of larger organisational culture and crucial in preventing organisational accidents. These include high-consequence accidents, such as the nuclear safety systems failure at the Fukushima Daiichi plant in Japan in 2011 (OECD/NEA, 2013[11]) and the organisational and cultural lapses that contributed to the BP Deepwater Horizon oil spill in 2010 (Reader and O’Connor, 2014[12]); (Corkindale, 2010[13]).

Together, these departure points lend strong support to further research on safety culture and actions that regulators could take to use this better understanding to improve outcomes for society, the environment and the sector. While there has been some academic and government-led research on safety and regulatory policy previously conducted (see Chapter 3), this field is relatively under studied and the understanding by regulators and knowledge of how BI could help was seen as relatively low. For these reasons, the survey-based experiment was designed with representatives from both the regulators and regulated entities in the energy sector to test the application of BI to different dimensions of safety culture. The experiment was designed to capture participants’ perception of (OECD, 2019[5]):

1. The perceptions of workers from regulators and regulated entities regarding safety culture in their respective areas (safety culture).

2. The extent to which different actors would respond to the potential application of behavioural insights to common safety problems (scenarios/vignettes).

Behavioural insights tested in phase 1

As discussed further in Chapter 3 of this publication and in (OECD, 2019[5]), there are four key BI principles identified in the safety culture literature that were first identified as potentially viable for this experiment:

1. Messenger: The way we process the same information differs depending on who we receive it from (Clark et al., 2013[14]); (Eckel and Gintis, 2010[15]). For example, individuals are more likely to believe a message when it comes from an authority figure or expert, conform to the behavioural aspects of a message resulting in decreased violations overall, and appreciate information more when it comes from people they have positive feeling about or who are a bit like themselves.

2. Social influences: Humans are social creatures and look to others for information on how they themselves should behave (Bicchieri, 2005[16]); (Goldstein et al., 2008[17]). This is relevant to many psychological mechanisms, but two are important here:

a. Social benchmarking / feedback: People pay attention to feedback in almost everything they do and often cannot adjust their behaviour without it. Providing a benchmark can reduce mistakes and make the consequences of decisions more salient. For example, providing pre- and post-shift hearing test results to workers can increase the use of hearing protection in subsequent shifts, overcoming what is known as the “present bias” (Zohar, 1980[18]). However, we do not always get personal feedback on what we do and often look to the behaviour of others and the feedback they receive.

b. Social norms: Evidence also suggests that people tend to survey their social and physical environment for attitudinal and behavioural cues and they care deeply about what their neighbours do. This is especially true when their neighbours belong to their same social in-group. Social norms act as a standard, informing individuals of what others think and do. There are a number of behaviourally-informed initiatives that make use of social norms, which are detailed in (OECD, 2019[5]). Speaking up about unsafe practices in the workplace is particularly interesting for the purposes of this experiment.

3. Reciprocity: The power of “reciprocity” for inducing co-operation is also a well-replicated effect in the behavioural literature (Fehr, Fischbacher and Gächter, 2002[19]); (Rand, Yoeli and Hoffman, 2014[20]). As social beings, people like to keep promises and reciprocate. Therefore, when people observe that others are taking the time to do things for them, they are more likely to continue that engagement. (OECD, 2019[5]) has more details on this effect.

Through discussions with project partners and country contact points, it was decided to focus on messenger, feedback and social norms in the survey-based experiment.

Methodology

(OECD, 2019[5]) provides a detailed overview of the methodology, which is summarised below. The experiments conducted were the result of extensive scoping and engagement with project partners and the NER community. This was supported by discussions regarding the project and themes with representatives from each participating country to gain an understanding of each specific context. Further discussions were had with academics at the London School of Economics (LSE) and explored in a scoping literature review. Chapters 3 and 4, above, provide this scoping review that mapped the ways BI could be applied to safety and present case studies from five NER countries, respectively.

Informal focus groups composed of representatives from regulators and regulated entities were also engaged to gain detailed feedback on safety culture and the behavioural principles to be tested, and used to iterate versions of the questionnaire. Finally, a number of academic experts and practitioners from the wider safety culture and BI communities were engaged for additional feedback on the experimental design and application of BI principles. Ethical approval was obtained from LSE and researchers followed OECD principles of confidentiality and ethics (OECD, 2016[21]). The study was also pre-registered after data collection, but before data analysis, on the Open Sciences Framework (Tear and MacLennan, 2018[22]).

Experimental design

The experiment was conducted via a computer-based questionnaire that was distributed to respondents, who completed it in their own time. Distribution was via email sent to respondents within regulators, as well as contact points within regulated entities who passed the survey onwards to frontline staff, managers and senior management, including contractors. Exclusions and additional details are noted in (OECD, 2019[5]).

For Canada, Ireland, and Oman, the email and experiments were in English. For Mexico, they were both translated into Spanish. However, respondents could respond in either English or Spanish. Contact points in ASEA read through the translated documents and agreed on translation.

The questionnaire was sent in August 2018 and responses were collected for six weeks, with a reminder message in early September. This was particularly important as it was highlighted that some staff work on five-week rotations and likely would not receive the initial email.

The questionnaire was divided into four main sections (further details can be found in (OECD, 2019[5]), including the complete questionnaire in Annex 5.A of that publication):

1. Demographics: This section collected basic, unidentifiable data which could inform the analysis in the following section. Questions were purposely minimal so that individuals could not be identified and their privacy was protected, with the goal of especially promoting honesty in their responses to the following sections.

2. Safety culture/climate questions: This section was included based on discussions in the NER and scoping discussions had with academics and contact points. Emphasis was placed on understanding the extent to which views on safety culture different between individuals in regulated entities and in regulators. Respondents answered questions based on a 7-step Likert scale ranging from Strongly Agree to Strongly Disagree. Questions were administered in the same order for everyone.

3. Behavioural vignettes/scenarios: These asked respondents from regulators and regulated entities questions testing the application of the three BI principles in three scenarios: 1) the introduction of a new guideline regarding Personal Protective Equipment (PPE); 2, Reports of a bad lost-time injury rate; and 3. A situation where a supervisor asks a worker to carry out a task in an unsafe manner. This methodology is common in the BI literature, but not often used in safety culture literature. Thus, this experiment is innovative both in terms of subject (dimensions of strong safety culture) but also methods employed. (OECD, 2019[5]) acknowledges that these examples are not reflective of safety culture as a whole and have drawbacks. More details on these and vignette design can be found in (OECD, 2019[5]). While respondents were not randomised into treatment and control, the presentation of the vignettes were presented in randomised order and similar vignettes were not asked together as a group.

4. Qualitative: Respondents were provided a space at the end of the questionnaire to provide reasons why they selected as they did for each experiment. While people are often poor predictors of why they make certain decisions, this does provide useful additional information to interpret results.

Cross-national results

The following section describes the results of the study. The first part will be a summary of the cross‑national results presented in (OECD, 2019[5]), followed by a country-by-country analysis of the quantitative and qualitative data. For both the safety culture question as well as the BI experiments, three main hypotheses were tested:

1. Regulator vs. regulated entity: Are there differences between the regulator and regulated entities in terms of the effectiveness of the scenarios? This is an exploratory hypothesis.

2. Country differences: Are differences between regulator and entities driven by national context? Can national culture account for the differences?

3. Frontline vs. manager differences: In nations where power distance is high (Oman, Mexico to a lesser extent), are there differences in responses between frontline staff and management staff? Is this related to the safety culture of the organisation?

Overall response rate was N = 1 366. Those who responded to less than 50% of section 2 (safety culture items) or section 3 (vignettes) were removed. This left N = 1 033 for the safety culture sample and N = 855 for the behavioural vignettes. These are not independent samples – it is possible, and likely, that participants would appear in both samples.

Response by nationality is 28 from Canada, 92 from Ireland, 409 from Mexico and 504 from Oman. Given the analysis of the possible sample for Canada in (OECD, 2019[5]), it can be concluded that the sample size is very low and potentially susceptible to selection bias. Caution is encouraged inferring conclusions with the Canadian sample.

A full explanation of the cross-national results by each of the behavioural insights tested can be found in (OECD, 2019[5]). They can be summarised as:

• Messenger: Overall, the messenger of safety instructions seems to mostly only matter in Ireland and Oman. In those countries, the peer messenger was perceived to be the least effective messenger vehicle. From a Hofstedian perspective, Ireland and Oman do not share many cultural similarities. Where they do share similarity is in the rate of regulator worker to entity worker responses (Ireland 1:3; Oman 1:4). Perhaps this majority of entity worker responses explains why the peer messenger was perceived least favourably. Where there is a messenger effect, its direction is such that messages from managers and regulators are deemed more effective than messages from peers.

• Feedback: While feedback was overall the most impactful behavioural principle among the ones tested, A deeper investigation into the responses to the feedback vignettes failed to reveal any differences at the country, occupational or organisational level.

• Social norms: In general, norms were found to be the least effective behavioural principle overall. However, cross-national comparisons revealed some interesting trends. In particular, the Mexican sample was the only nationality for which there were statistically significant results with regards to norm type. Mexican respondents were found to react more strongly to descriptive norms than control or injunctive messages. From a Hofstedian perspective, Mexican samples score high on uncertainty avoidance, meaning that the clear signal from descriptive norms may be preferred over the motivationally unclear nature of injunctive norms. Mexican samples also score low on individualism, meaning that they may be more susceptible to group norms in general.

However, there are multiple limitations discussed in (OECD, 2019[5]). These include the varying degrees of English abilities, particularly in Oman, and that we relied on contact points within the regulated entities to pass along the information. Attempts were made to counteract these limitations, where possible. There is also no objective benchmark to compare workers’ safety cultural perceptions. We also cannot test regulator’s perceptions of the state of safety culture in the sector, nor does the data support such a conclusion. There is also likely a degree of overlap between the three BI principles tested.

Policy lessons from the cross-national results

These lessons were presented in (OECD, 2019[5]), but bear repeating again since they are relevant to the discussion going forward in this chapter.

Overall, the project constitutes one of the first applications of behavioural insights through online experiments to the study of safety improvement and elements of safety culture. It is intended to serve as a stepping stone towards a more frequent integration of the field of BI and safety. A number of key policy lessons emerge from the research.

• To avoid unintended negative consequences, it is important for regulators to take into consideration differences in perception within and between actors when designing new safety regulations or policies. The study found that the closer one is to the front line, the lower one’s perception of safety culture. From a system perspective, the study showed that regulators have a more negative perception of safety culture in the regulated entities than the entities themselves, perhaps due to their position overseeing the sector. Moreover, results show that senior managers reacted most favourably to the behavioural principles (i.e. feedback, messenger effects and social norming) than other occupations, indicating that there are differences of perceptions within entities (not only between entities and the regulator) regarding safety culture. When developing policy, it is important to take these differences in perception into consideration to ensure policies are targeted for different audiences.

• The study found that some feedback is better than no feedback but the results are inconclusive as to which type of feedback or benchmarking is best. Results show that respondents reacted most favourably to feedback vignettes, compared to messenger and norm vignettes, generally speaking. From a policy perspective, this highlights the importance of considering providing workers with some form of feedback. However, which form of feedback is most effective and whether feedback is beneficial in every context needs to be studied in further detail.

• The source of safety messages (messenger) still matters, which highlights the need to ensure regulators and senior managers in regulated entities are working together to encourage a culture of safety. Results showed that respondents reacted similarly to messages on safety coming from a regulator as well as senior management of the regulated entity. However, messages from peers were considered the least effective, which runs counter to conventional thinking about the use of norms in nudging.

• Social norming was perceived as the least effective across the sample, which requires more research to determine the benefit for policymakers of using social norms to encourage a culture of safety. Results for all social norm vignettes were statistically indistinguishable from the control. However, some differences were noted for respondents rating the descriptive norm vignette more positively than the control vignette, giving some possible avenues for future research.

• Differences between countries highlight the need for policymakers to take a location-based approach to strengthening elements of safety in their own contexts. While the above notes the trends for each behavioural insight tested, between-country differences were notable. For feedback, there were no statistically significant differences at the country level; however, results did show respondents from Oman reacted most favourably, followed by Irish respondents, and then Canadian and Mexican respondents alternating for least favourable responses. Caution should, however, be taken inferring results from the Canadian results due to small sample size. For the messenger effect, it seems that this really mattered most in Ireland and Oman, perhaps due to a similar regulator worker to entity worker response rate. For social norming, the Mexican sample was somewhat responsive to descriptive norms, though Irish and Omani responses were also favourable. Canadian responses were least favourable.

Phase 1: Country-level results

(OECD, 2019[5]) presented preliminary, cross-national results that were summarised above and presented in full in the aforementioned publication. This initial data analysis revealed a number of avenues for further exploration where the data were sufficient. Particularly, the analyses in (OECD, 2019[5]) explored differences between nations – there is a good opportunity to look at where differences exist within nations, that is, what works best within a nation rather than what works best across nations. We report on some of these additional analyses below.

The role of safety culture

Safety culture in these data refers to the perceptions that individuals have regarding the norms and behaviours related to safety in regulated entities. Safety culture is usually considered the outcome of many individual and group behaviours, perceptions, and norms interacting. Indeed, complex social systems can be constructed as micro- and macro-level social properties (Conte et al., 2007[23]). Under this conceptualisation, behaviours and norms would be considered micro-level social properties, which give rise to the macro-level social property of culture. An important notion here is that micro-social entities generate effects at the macro-social level that they do not perceive nor, therefore, aim at. Part of the generative process is external to their minds.

Yet, culture itself can generate effects at the lower level too, via comparatively simple or complex loops. In the context of safety culture, an organisation’s emergent safety culture is determined by the interaction of individuals’ behaviours, perceptions, and norms, but also determines the behaviours, perceptions and norms of individuals. This relationship between safety culture and perceptions of safety culture could be described as a feedback loop, where perceptions of safety culture also shape the culture itself.

Thus, it is important to consider the safety culture ecosystem that our surveys took place in. Practically, this means examining how individuals’ perceptions give rise to an organisational safety culture, but also how the safety culture of an organisation shapes individuals’ perceptions. To do this, we segment analyses by whether the participant views entity safety culture as average, below-average, or above-average.

What does a safety culture score mean?

Participants in our survey filled out a number of items targeting various different factors of safety culture: 1) management commitment to safety; 2) regulator commitment to safety; 3) colleague commitment to safety; 4) collaboration for safety; 5) reporting incidents; 6) communication for safety; 7) safety support; 8) relationship between the entity and the regulator. Participant perspectives on these factors are summarised into a single safety culture score, which is taken to reflect how positively or negatively they view the overall safety culture of regulated entities:

Safety culture scores represent how much safety culture the individual perceives in regulated entities (a measure of an organisation’s safety culture).

Safety culture scores might also reflect an individual bias within the participants themselves. That is, individuals might be predisposed to see safety culture more positively or negatively than it actually exists. This bias would be similar to an optimism or pessimism bias.

Safety culture scores represent just how likely each individual is to perceive safety culture (their baseline safety culture perception).

Our data are not able to determine which of these correct and it is likely that there are elements of both in an individual’s safety culture score.

Behavioural insights approaches to improving safety culture

Overall, the more positive your safety culture score, the more optimistic you are about the vignettes. That is, if you say your organisation/the regulated entity has a strong and positive approach to safety, then you are more likely to think that the BI principles would be effective for improving safety behaviour. Conversely, the people who thought the BI principles would be relatively less effective were also those who said that their organisation had relatively weaker and/or negative approach to safety, on average.

Based on the previously discussed two constructs that safety culture scores might reflect, there are at least two possible interpretations for this result. The first is that behavioural insights approaches to safety culture might be best suited to organisations with pre-existing positive safety cultures.

The second interpretation is that individuals with a safety culture optimism bias are more likely to endorse novel approaches to improving safety culture. Indeed, there is work highlighting that organisational safety attitudes (e.g. injunctive safety norms, the balance between safety and productivity) are significantly related to agreeableness, conscientiousness, prevention regulatory focus, and fatalism (Henning et al., 2009[24]). When considered in the context of the personality literature, there are strong links connecting optimism with agreeableness and conscientiousness (Sharpe, Martin and Roth, 2011[25]). Tying it back to the result – the positive relationship between safety culture and the endorsement of the BI principles might be explained by a third factor: individuals' optimism.

Thus, we must always consider that an individual’s safety culture score is influenced by at least two factors simultaneously: 1) the safety culture environment they exist in; and 2) their own propensity for optimism or pessimism. It is likely that these two factors are highly related, for example, it might be that organisations with positive safety cultures are filled with people who might be described as safety culture optimists – that is, strong organisational safety culture creates safety culture optimists.

Segmenting according to safety culture

Where we have sufficient data, we can group participants according to their safety culture score – below-average, average, or above-average safety culture. We can then use these groupings as lenses through which to view other data. While we are perhaps most interested in what works best in entities with average safety culture (i.e. the majority of entities in the case of a normal distribution), it is also useful to see what individuals think would be effective at the extremes of safety culture: comparatively weak or strong safety culture.

Below-average and above-average refers to individuals whose safety culture scores were ±1SD from the mean. Due to the nature of normal distributions around the mean, the below- and above-average groups have considerably smaller sample sizes than the average group, which means there’s a larger degree of error in the below- and above-average segments. This means that, while a difference may be significant in the average group, a larger difference may not be significant in the below- or above-average groups due to larger degrees of error.

Looking at the whole sample, below-average safety culture perceivers favour the norm vignettes, whereas average and above-average safety culture perceivers favour feedback (see Figure 4.1).

Figure 4.1. Vignette effectiveness as a function of safety culture

Note: Below average: norm significantly greater than messenger (p=.029). Average: feedback significantly greater than both messenger (p=.002) and norms (p<.001). Above average: trends for feedback to be significantly greater than messenger (p=.058) and norms (p=.067).

From this we can infer that participants believe that organisations with below-average safety culture might benefit most from norm interventions, whereas they believe that average or above-average safety culture organisations might benefit most from feedback interventions.

It is difficult to extrapolate the meaning of these results without further data for a number of reasons. First, a low organisational safety culture score could literally be taken to mean the organisation has weak safety culture. Conversely, if workers feel like they can voice criticism without penalty, then this is also indicative of a strong safety culture score. Second, ‘below-average’ in this context is not equivalent to “poor performance” All safety culture averages are above the scale midpoint and, thus, still reflect positive safety cultures. Third, these are hypothetical safety culture scores and are not reflective of any particular organisation. That is, these safety scores do not literally reflect the safety culture an organisation, rather they represent the imagined safety culture of a hypothetical regulated entity. Most simply, participants imagined the safety culture of typical entities in their sector, so to extrapolate beyond these data would be ill-advised.

To better understand why participants believed that norm interventions would work best in below-average safety culture organisations and feedback would work best in other organisations, researchers would need to speak further with participants to unpack their reasoning.

National contexts

Next we can segment according to national context to see how below-average, average, and above-average safety culture perceivers respond to the vignettes in each country. There were insufficient data in the Canadian sample for analyses of this complexity.

Hofstedian national cultural dimensions have often been used for interpreting organisational culture (Hofstede, 2001[26]); (Reader et al., 2015[27]). Figure 4.2 presents the Hofstedian cultural dimension values for the nations participating in this research. These scores are provided for potential context when interpreting the data following.

Note that the Hofstede model does not provide scores for Oman, instead grouping all Arabic-speaking countries together into one regional score. While there were insufficient quantitative data from the Canadian sample, it may be useful to reflect on the data from the Irish sample, given their cultural similarity to Canada.

Figure 4.2. Hofstedian cultural dimension values for participant countries

Source: (Hofstede, n.d.[28])

Ireland

These data are presented with the caveat that they are underpowered and should be interpreted with caution. Below-average safety culture perceivers saw the norm vignettes as more effective, whereas average safety culture perceivers saw norms as less effective. Above-average safety culture perceivers thought messenger interventions would be most effective (see Figure 4.3).

Qualitative data described later in this report provide greater detail for the successful implementation of messenger interventions.

Figure 4.3. Vignette effectiveness as a function of safety culture in Ireland

Note: Below average: no significant differences. Average: messenger significantly greater than norms (p=.001). Above average: no significant differences.

Mexico

Average and above-average safety culture perceivers were more likely to see the feedback vignettes as less effective (with a tendency towards endorsing norm vignettes), whereas below-average safety culture perceivers thought all principles would be equally effective (difference between principles not significant; see Figure 4.4).

Figure 4.4. Vignette effectiveness as a function of safety culture in Mexico

Note: Below average: no significant differences. Average: messenger significantly worse than feedback (p=.001) and norms (p<.001). Above average: no significant differences.

Oman

Below-average safety culture perceivers felt that norms were the most effective vignette, average safety culture perceivers felt that the feedback and norm vignettes would be more effective than norms, and above-average safety culture perceivers felt that feedback initiatives would be most effective. See Figure 4.5 for these data graphically represented.

Figure 4.5. Vignette effectiveness as a function of safety culture in Oman

Note: Below average: no significant differences. Average: feedback significantly greater than messenger (p=.038) and norms (p<.001); messenger significantly greater than norms (p=.008). Above average: feedback significantly greater than norms (p=.016).

Summary of safety culture segmentation

We can draw a number of conclusions from the above national analyses, which are summarised in Table 4.1. A plus indicates the vignette was statistically the best performing, a minus means statistically the worst performing, and a question mark indicates no statistically significant difference.

Norm interventions were predicted to be the least successful in Ireland, except in the cases of relatively weaker safety culture organisations. Rather, a messenger approach was often reported as being the most likely to have success.

For Mexico, norm interventions were consistently the favoured approach, with feedback sometimes showing promise.

The Omani sample favoured feedback, though messenger approached sometimes approached significance.

Table 4.1. Summary of vignette effectiveness across national contexts

		Messenger	Feedback	Norm
Ireland	Below average	?	?	?
	Average	+	?	–
	Above average	?	?	?
Mexico	Below average	?	?	?
	Average	–	+	+
	Above average	?	?	?
Oman	Below average	?	?	?
	Average	–	+	+
	Above average	?	+	–

Note: A plus indicates the vignette was statistically the best performing, a minus means statistically the worst performing, and a question mark indicates no statistically significant difference

Segmenting according to occupational role

The above data provide important implications for the roll-out of behavioural insights approaches to safety culture by demonstrating that the success of different approaches might depend on the level of safety culture in the entity. Below we continue these analyses by checking whether this relationship between entity safety culture and predicted impact of the behavioural insights approaches differs according to participants’ occupational role.

According to the literature, perceptions of safety culture are known to vary according to occupational role (Parand et al., 2010[29]); (Tear et al., 2020[30]), perhaps largely due to differences in organisational perspective associated with different roles (e.g. frontline staff have the most specific but least wide view of the organisation, senior managers have most wide but least specific view of the organisation). By taking this lens to the data we can determine if individuals in different occupational roles will be more affected by the safety culture of the entity when they report how effective the BI approaches might be.

It is useful at this point to remind the reader exactly what we have measured. The survey was distributed to workers in regulators and regulated entities. Those workers were asked to imagine a typical regulated entity in their sector and evaluate how effective various different BI approaches would be for improving safety culture and/or safety behaviours in that imagined entity. We also asked them to respond to questions about that imagined entity’s safety culture.

In the previous section, we demonstrated that when workers imagined the safety culture of a typical entity in their sector was more positive than average, then that worker was also more likely to think that the BI approaches would be successful. Conversely, when they imagined the safety culture of a typical entity to be less positive than average, then that worker was also less likely to think that the BI approaches would be successful.

Now we are checking to see how workers’ occupation relates to how they think about the typical entity and its safety culture. Does being a frontline worker change the way they see the safety culture and, thus, change how effective they think the BI approaches would be? If a senior manager and a worker from the frontline imagine typical entities with the same safety culture, do they think the BI approach will be similarly effective?

Because there are known links between national culture and organisational safety culture (e.g. national power distance accounts for the difference in perspective between frontline staff and management), we breakdown the following results according to national context.

Ireland

Looking at the overall Ireland data for how safety culture affects ratings of the vignettes, we can see that managers appear not to be affected. On the other hand, senior managers and other staff appear much more affected. Frontline staff are also affected but not as much as senior managers and other staff. A full breakdown of the vignette effectiveness data according to each specific item are provided in Table 4.2 below, and more in depth results are provided in Appendix 4.A. The Ireland data are highly volatile due to the small sample size.

Table 4.2. Breakdown of how safety culture affects perceived effectiveness of vignettes according to occupational role in Ireland

	Safety culture impacts perceived effectiveness of vignettes most for...	Safety culture impacts perceived effectiveness of vignettes least for...	Notes
(1) This safety information would attract <the attention of workers/my attention>			Safety culture affects all occupational roles equivalently
(2) This safety information should affect how <workers in entities do their job/I do my job>	Senior managers and other staff	Frontline staff and managers	Trend for negative effect of safety culture for frontline staff and managers
(3) This safety information would affect how <workers in entities do their job/I do my job>	Senior managers and other staff	Frontline staff and managers	Managers are unaffected by safety culture (non-significant)
(4) This safety information should affect how <the entity overall/how managers do their job>	Senior managers and other staff	Frontline staff and managers	Senior managers are the only group significantly affected by safety culture
(5) This safety information would affect how <the entity overall/how managers do their job>	Managers, senior managers, and other staff	Frontline staff	No significant difference between managers, senior managers, and other staff

Mexico

The effect of safety culture on the effectiveness of the vignettes is varied in the Mexican context according to occupational role. Senior managers are the most affected by safety culture (item 1) but not in other cases (items 2 and 4). Similarly frontline staffs’ ratings of the behavioural insights principles are most affected on some items (items 1 and 4) but least on others (items 2, 3, and 5). While these differences exist, it is important to note that they are small and that safety culture appears to have a roughly equivalent effect for all occupational groups. Full breakdown provided in Table 4.3 and specific graphs are presented in Annex 4.A.

Table 4.3. Breakdown of how safety culture affects perceived effectiveness of vignettes according to occupational role in Mexico

	Safety culture impacts perceived effectiveness of vignettes most for...	Safety culture impacts perceived effectiveness of vignettes least for...	Notes
(1) This safety information would attract <the attention of workers/my attention>	Frontline staff, senior managers, and other staff	Managers	No significant difference between frontline staff and senior managers
(2) This safety information should affect how <workers in entities do their job/I do my job>	Managers and other staff	Frontline staff and senior managers	Senior managers are unaffected by safety culture (non-significant)
(3) This safety information would affect how <workers in entities do their job/I do my job>	Other staff	Frontline staff, managers, and senior managers	No significant difference between frontline staff, managers, and senior managers
(4) This safety information should affect how <the entity overall/how managers do their job>	Frontline staff, managers	Senior managers, other staff	Senior managers are unaffected by safety culture (non-significant)
(5) This safety information would affect how <the entity overall/how managers do their job>	Other staff	Frontline staff, managers, and senior managers	No significant difference between frontline staff, managers, and senior managers

Oman

When we look at the effect of safety culture on the overall effectiveness of the vignettes in Oman, we can see that there are some differences between the between the occupations. Interestingly, safety culture appears to affect frontline staff and managers roughly similarly, though managers are sometimes more positive about the vignettes than frontline staff. Senior managers often have the most positive view of the vignettes and that view has less to do with the safety culture of the entity. A full breakdown of the results are provided in Table 4.4 and specific graphs are presented in Annex 4.A.

Table 4.4. Breakdown of how safety culture affects perceived effectiveness of vignettes according to occupational role in Oman

	Safety culture impacts perceived effectiveness of vignettes most for...	Safety culture impacts perceived effectiveness of vignettes least for...	Notes
(1) This safety information would attract <the attention of workers/my attention>	Frontline staff, managers	Senior managers, other staff	No significant difference between frontline staff and managers
(2) This safety information should affect how <workers in entities do their job/I do my job>	Frontline staff	Managers, senior managers, other staff	Senior managers are unaffected by safety culture (non-significant)
(3) This safety information would affect how <workers in entities do their job/I do my job>			Safety culture affects all occupational roles equivalently
(4) This safety information should affect how <the entity overall/how managers do their job>	Frontline staff, senior managers	Managers, other staff	Other staff are unaffected by safety culture (non-significant)
(5) This safety information would affect how <the entity overall/how managers do their job>	Senior managers	Frontline staff, managers, other staff	No significant difference between frontline staff, managers, and other staff.

Qualitative analysis

While there was insufficient quantitative data from the Canadian sample and less than ideal data for the Ireland sample, we were able to extract a number of themes from the qualitative items in the survey. These are described below. We have not analysed the qualitative data from the Mexican and Omani samples because of possible translation errors.

Reminder of Hofstede cultural dimensions

This section serves as a primer for interpreting the following qualitative sections. Canadian and Irish samples are quite similar on each of the dimensions. They believe that inequalities between individuals should be minimised (low power distance) and that individuals should look after themselves and be self-starters at work (high individualism). New ideas, creativity, and a willingness to try new things (low uncertainty avoidance) are characteristic of both nations. Both nations are also considered normative and a focus on quick results (low long term orientation). Finally, they are both considered indulgent (high indulgence) nations, with a willingness to enjoy life and be optimistic.

Where there is difference between the nations is on the masculinity scale, with Ireland being considered a more masculine society than Canada, meaning a greater focus on success and competition.

As with the initial introduction of the Hofstede cultural dimensions earlier, it is important to note that more proximal influences such as perceived management commitment to safety and efficacy of safety measures exert more impact on workforce behaviour and subsequent accident rates than fundamental national values (Mearns and Yule, 2009[31]).

Canada

The qualitative data from the Canadian sample was useful for extracting some specific information regarding the perceived effectiveness of different BI principles for improving safety culture. Regarding messenger effects, Canadian participants mentioned that the most effective message is one that is communicated consistently across all types of messengers (e.g. regulators, managers, colleagues). When the message is consistent across different messengers, then participants suggested that regulators would have the greatest impact — colleague messengers would be impactful to the extent they reinforce messaging from management or regulators. This result should be interpreted with caution, remembering that the Canadian sample was overrepresented by regulator workers by a factor of 6 to 1. Canadian respondents also felt that the size of an organisation would impact the effectiveness of different messengers. Extrapolating this idea, one could imagine that colleague messengers might have a greater impact in smaller organisations, for example.

The Canadian participants also felt that feedback had value, with opinions varied on how much value. There was some question about exactly how useful benchmarking data would be. Some felt that benchmarking was only important for management staff, who sometimes have bonuses tied to benchmarking outcomes, and would have little bearing on frontline staff who are less likely to have such bonuses. There were some who felt benchmarking might have an impact on frontline staff but only to the extent that benchmarking data was provided to them clearly and consistently — if they don’t have access to the data, then it can’t be useful. Participants generally believed that public information about performance relative to others would be effective. Feedback from the regulator might be useful but benchmarked data would be more useful.

Regarding norms, Canadian participants felt that safety culture is more strongly enhanced by visible action than by proclaimed values. That is, when asked about whether descriptive norms (norms about what people actually do) or injunctive norms (norms about what people ought to do) would have a greater effect on changing safety culture, some respondents said that descriptive norms would have the greater effect. Thus, people visibly implementing safety culture best practice will have a larger impact than organisational values for safety culture.

Ireland

Irish participants believed that regulator or manager messengers would have a stronger impact on safety culture than peer messengers for a number of suggested reasons, which is consistent with the quantitative findings. First, messages delivered by colleagues are susceptible to incorrect, difficult, and possibly reluctant delivery. For example, extolling the benefits of PPE to peers may lead to the peer messenger being ridiculed. Second, safety messages from peers would likely be elevated to senior management if they were important enough. So, a peer messenger might be effective but only via senior management. Management and regulators also tend to have more oversight and context on safety (but important detail can be lost from this perspective). On the other hand, some participants suggested that peer messengers may be impactful in other contexts because peers have shared/similar experiences. Some participants felt that documented or signed off directions will be carried out in the vast majority of situations, irrespective of whether the direction comes from a regulator, manager, or colleague.

The Irish participants noted that while benchmarking was often desired, its implementation was a tricky prospect in Ireland, where there are so few entities to benchmark against. Participants provided a number of factors to consider with benchmarking. First, benchmarking could be considered across industries. While the variation between contexts might be considerable, some participants believed that variation provided vital learning opportunities. Second, benchmarking should clearly distinguish occupational health and safety from process safety. Third, benchmarking is only effective when endorsed by entity management. Fourth, there should be consideration of how often to provide benchmarked data and guides on how to interpret the data. Finally, benchmarking data alone could be counter-productive — those data tell you that a difference exists between entities, but not why. Follow-up activities would be vital in understanding why differences exist in order to develop change strategies.

Irish participants believe that norms are important but described how conflict between injunctive (what ought to be done) and descriptive (what is done) norms can be especially damaging (e.g. we ought to do this but no-one does). It’s easy to express a value but at the end of the day, if people aren’t doing something, then that’s the most important factor.

Suggested policy responses and focus areas

Canada

The qualitative data revealed some further depth to the results presented in the OECD chapter. First, some respondents suggested we should focus less on choosing an effective messenger and more on making sure there is consistent messaging across all messengers. Second, benchmarking would need to be made relevant for all staff, not just management (who may have bonuses tied to benchmarking data). Finally, descriptive normative approaches were thought to have the most impact because they represent what people actually do.

Ireland

These additional analyses unpacked important nuance for the Ireland context. While messengers were established as the most likely to succeed in Ireland, the messenger data from the OECD chapter suggested that peer messengers would be the least effective messenger type. This was corroborated in the qualitative data, where some respondents indicated that it was tricky for individuals to espouse the value of safety to colleagues, potentially making them a target for ridicule. This also relates to the norm vignettes, which were often seen as the least likely to succeed, because Irish respondents would presumably look to management and regulators and be less focused on what their peers are doing. Thus, Ireland should focus on messengers, particularly regulator and management messengers (though important to consider norm approaches in below-average safety culture entities). Senior managers’ views on the BI approaches appear highly affected by the safety culture of the entity, which is highly important for pitching BI approaches.

Mexico

The additional analyses revealed that normative approaches were seen as the most effective BI principle, across all safety cultures. Perhaps owning the high uncertainty avoidance often associated with Mexican samples, there was a preference for descriptive norms over injunctive norms, with the notion that seeing what people are actually doing (less uncertainty) would have more impact than hearing about what they value (more uncertainty). Normative approaches might be paired with clear messages from the regulator for maximum uncertainty reduction. Senior managers’ views on the vignettes were relatively unaffected by an entity’s safety culture, so this should not be leaned upon when pitching to senior management.

Oman

From these additional analyses we can see that a primary focus for Oman should be on investigating the use of feedback with regulated entities. Communication is seen as an important guiding principle, especially from the regulator. It is also important to ensure there is consistency in the messaging, especially from regulators and entity management.

If targeting frontline staff, then it is important to consider the safety culture of the entity, as higher perceived safety culture is associated with stronger beliefs in the vignettes’ success. Management views on effectiveness of vignettes seem less affected by safety culture. This suggests that understanding the entity’s safety culture will be less useful in pitching BI approaches to management.

Reflections from the literature

Using norms to counteract negative effects from feedback

There were several instances where a norm approach was deemed to be most effective so it is important to consider lessons learnt from previous BI interventions regarding rollout of norm approaches.

A feedback approach was used by the Behavioural Insights Team (2011[32]) for tackling reduced energy consumption behaviours. Providing consumers with feedback on how their energy use compared with similar households in their neighbourhood was shown to reduce energy consumption in higher-than-average users (non-compliers).

There was an unintended negative effect of this feedback, however, for lower-than-average users (compliers). When they received their energy bill, the feedback indicated that they were using less than their neighbours, which conveyed a descriptive norm of greater energy use. This is known as a “boomerang” effect.

To address this unintended side-effect, the descriptive norm was then paired with an injunctive norm (in this case, a smiley face for lower-than-average users). The pairing of the descriptive and injunctive norms with feedback was effective at reducing energy consumption behaviours.

Another approach would be to make sure that feedback only targeted the higher-than-average users, who are not at risk of boomerang effects. This approach has been successfully used in the health context with success. General practitioners who over-prescribe antibiotics contribute to the spread of antimicrobial resistance worldwide. Interventions to reduce the amount of prescriptions within this specific cohort have taken a feedback approach with success. Letters sent to over-prescribing GPs, telling them that other practices were recommending the use of antibiotics in fewer cases, reduced the number of antibiotic prescriptions by 126 352 over a six month period – a reduction of between 9.3-14.6% (BETA, 2018[33]).

Norms approaches were thought to have most impact in two main contexts: 1) Irish entities with below‑average imagined safety culture; and 2) Mexican entities with average imagined safety culture. Based on the lessons learnt from previous norm-based approaches, these two contexts are not a risk for boomerang effects, whereas entities with above-average safety culture could suffer boomerang effects. Remember also that descriptive norms are perceived to have a stronger effect than injunctive norms, so there may be difficulty in finding an injunctive norm potent enough to counteract the boomerang.

Presenting benchmarked data

Benchmarking is wanted by most who participated in our study but there are serious questions about how best to present benchmarked data, particularly regarding safety culture. For example, there are aspects of national/organisational culture (e.g. high power distance, high uncertainty avoidance) that are linked to weak safety culture. Thus, some nations/organisations will always rank poorly in comparative benchmarking exercises simply because of their national culture. This limits the opportunities for identifying and sharing best practice.

Some researchers have developed methods for statistically adjusting for the effects of national/organisational culture to allow for a more nuanced safety culture benchmarking approach (Noort et al., 2016[34]). The safety culture against international group norms (SIGN) approach transforms safety culture scores into z-scores and presents the relative position of an organisation with a cultural cluster. SIGN scores highlight variations against a group norm on a normal distribution, and signal a relative position of safety culture strength rather than a direct comparison of raw scores. This means safety culture data are re-scaled to fit a given cultural context, with the assessment of safety culture being directed towards learning between organisations and regions.

For regulator purposes, it would be important to determine a set of parameters for segmenting regulated entities into groups of similar organisations for benchmarking purposes. For example, it might not make sense to compare large with small entities as smaller entities often struggle to devote resources for developing mature safety culture protocol.

Organisational position

A much greater understanding of the lived experience of various different workers should be sought before rolling out a BI intervention. Past research has shown that the work design of a particular occupational role may protect the worker from the effects of national/organisational culture. For example, in a recent published study (Tear et al., 2020[30]), researchers found that air traffic controllers experienced more negative safety culture the stronger their country’s national norms for power distance. Yet there was no comparable effect for engineers, whose safety culture perceptions were equivalent irrespective of the national power distance norms.

Qualitative follow-up revealed that, while controllers work quite closely with supervisors who have intimate knowledge of the airspace, engineers are usually subject matter experts and work more autonomously than controllers. If engineers are autonomous from their supervisors, then there is less opportunity for national power distance norms to shape engineers' experience.

This finding suggests that it is important to understand the work design on regulated entity workers before designing a BI intervention. If frontline staff work autonomously, then they may be less receptive to messenger interventions. If they work independently, then they might not pay attention to norm interventions.

We have only consulted a small subsection of regulator and regulated entity workers when trying to determine the effectiveness of behavioural insights approaches to safety culture. This provided a wide range of views on some very simple questions (e.g. “what would work?”) but now what is needed is a deep dive into the specifics of entities in various sectors to understand their specific safety culture context. Workshops with subject-matter experts will help to understand whether there are particular idiosyncrasies that would prevent particular behavioural approaches from working (e.g. norm-based approaches might not work for engineers if they are already quite autonomous).

Phase 2: Experiments with registered electrical contractors and gas installers in Ireland

Phase I studied how regulatory policy makers can foster elements of strong safety culture in Canada, Ireland, Mexico, and Oman. The respondents included the staff of the regulators and gas and energy regulated entities—primarily large firms. This second phase study moves from the global perspective to the national perspective thus enabling a more nuanced look into subgroups. This phase also continues to explore the messenger effects of the first study in addition to behavioural insights deemed relevant to the unique context of small scale regulated entities in Ireland under the auspices of the Commission for the Regulation of Utilities (CRU).

The CRU oversees the Safety Supervisory Bodies (SSBs), which administer the safety schemes for Registered Electrical Contractors (RECs) and Registered Gas Installers (RGIs). These are, respectively, the Safe Electric Register of Electric Contractors (hereafter Safe Electric) and the Register of Gas Installers of Ireland (RGII). To be registered, RECs and RGIs must hold relevant qualifications, have completed the necessary training, fulfil other requirements, and remain in good standing with the relevant SSB to carry out electrical and gas works. Each year every REC and RGI is inspected at least once, and any identified non-conformances must be resolved within certain pre-defined periods of time according to the level of severity of the non-conformance.

In 2017, there were over 4 100 RECs and over 2 900 RGIs registered with Safe Electric and RGII respectively. During that same year, more than 4 700 inspections of RECs and more than 3 200 inspections of RGIs were completed. Those inspections identified over 10 000 non‑conformances among RECs and 120 non-conformances among RGIs. In 2017, all immediate hazards (“code red”) were resolved; however, not all high risk (“code amber”) or low risk (“code yellow”) non-conformances were resolved within the specified periods of time.1

Case studies conducted previously among regulators in Ireland and other countries (see Chapter 3), highlighted the dilution or lack of consistency of safety messages and the lack of committed safety leadership together with inconsistent or contradictory messages as possible barriers to the emergence of a safety culture.

Small scale regulated entities in Ireland

RECs and RGIs work either as individuals or in small firms. Within the safety scheme, RECs may represent more than one electrician who work under an individual REC who is responsible for registering certifying electrical works; however, each individual RGI must be registered.2 Despite the fact that many RECs and RGIs work individually, they may share a sense of community through shared practices and their shared registration with the SSBs. The industry provides a unique vantage point from which to investigate the intersection of behavioural insights and safety culture. First, there is an indirect relationship between the regulator—the CRU, which is an independent statutory body—and the entities that is mediated by the SSBs with whom the RECs and RGIs register. Second, the regulated entities, RECs and RGIs, are often individuals or small-scale organisations, yet there is reason to suspect that they have elements of a shared culture due to their shared identity (as RECs and RGIs), mode of work, requirements, and sanctions.

These unique aspects broaden the findings from the Phase 1 study with larger regulated entities in the following ways:

• It investigates how the awareness and salience of safety within the scheme differs at three distinct levels: the regulator, the SSB, and the entities. On the one hand, it may be that safety culture is perceived as more positive among SSBs compared to RECs/RGIs, similar to the differences between the ‘management’ and ‘front line’ staff in the first phase study. On the other hand, the perception may be worse among SSBs compared to regulated entities in line with the differences between regulators and regulated entities uncovered in the first phase research. This will help reveal differences in awareness at the varied levels of this unique regulatory structure.

• It extends the first study’s investigation of the effect of receiving safety information from different messengers. While the first study found the lowest perceived efficacy of word-of-mouth messages from a peer (within the same organisation), a written message from a respected peer REC or RGI who previously conducted training may be perceived differently because they are usually not at the same organisation but still share a group identity which may be a powerful driver of behaviour (Terry, Hogg and White, 1999[35]); (Terry and Hogg, 2000[36]); (Goldstein, Cialdini and Griskevicius, 2008[37]). Likewise, this phase two study will be able to distinguish between different levels of the regulatory structure to determine if the intermediary SSBs are more effective messengers than the regulator due to their heightened social proximity combined with their regulatory authority.

• The larger number of REC and RGI entities enable us to more precisely investigate the degree to which there is a safety culture that goes beyond the level of a single organisation to the level of the industry, and the degree of heterogeneity across a large number of entities within the industry

In addition to these more direct extensions of the first study, this second study addresses new questions that are relevant to the context of Registered Electrical Contractors and Registered Gas Installers in Ireland and can offer insights into similar regulatory schemes where some inspection and supervision responsibilities have been delegated to an industry body.

Methodology

The goal of this study is to examine two aspects of behavioural insights applied to small scale regulated entities in Ireland. First, this study will provides a descriptive overview of the safety culture and behaviours in the two schemes. As a relatively new regulatory regime, there is limited information on the perceptions of the safety schemes among the CRU staff, the SSB staff, and the RECs and RGIs; as such, this study documents the safety attitudes and safety behaviour of the respondents to better understand the two industries.

Second, the study also uses a survey experiment to test the potential for behavioural insights to improve the effectiveness of inspections (see Table 4.5). More specifically, the experiment investigates the salience of safety as a concept that is associated with the SSBs and the CRU, changing the default ordering of inspection forms, prompting implementation intentions, and enhancing salience through the personalisation of communication. These behavioural insights map onto the top nine behavioural insights identified as having more of an evidence base in the process and safety literature by Lindhout and colleagues (Lindhout and Reniers, 2017[38]). The results from the survey experiment are the primary focus of this section.

Table 4.5. Descriptions of behavioural insights used in study

Insight	Description
Implementation Intention	Implementation intentions are detailed planning prompts that specify in detail when and under what circumstances an action should be taken. Through the creation of a specific plan they enhance commitment to action and the associated details serve as reminders to take action.
Messenger Effects	The type of messenger that is used for a call to action can affect the likelihood that individuals engage in the desired behaviour. Important dimensions to consider include how similar the messenger is to the person receiving the message, how trustworthy their expertise is on the topic of the message, and how unbiased the messenger is perceived to be.
Primacy	Primacy is the tendency to focus more attention on content at the beginning of a list. The items that come first are often given more attention than those that come later. In addition, the first and the last items tend to be most memorable.
Personalisation	Personalisation of messaging can increase the likelihood that someone will pay attention to the message, it can also increase a sense of reciprocity as the messenger put forth the effort to personalise the message, and finally it can make the content and the actions more salient and more helpful for remembering the precise action required.
Salience	Salience is how much a given piece of information, and a given call to action, are noticeable and how well they draw individuals’ attention. Particular components of a call to action can be made more or less salience through visual changes to written communication (size, colour, and so on), through repetition, through personalisation, and through many other strategies that draw attention to particular parts of a message and the underlying call to action.

This survey-based experiment helps clarify ways that behavioural insights can increase awareness, provide a more nuanced understanding of how to present shared responsibilities throughout levels of the safety scheme, to reduce complacency, and ultimately to contribute to a more pro-active safety culture in these and similar sectors.

Thus, the research questions addressed in this study were:

RQ1. How salient is the association of safety with actors and activities associated with the schemes? How do they differ among respondents?

RQ2. Would a more behaviourally informed INSPECTION REPORT increase the predicted accurate identification and timely resolution of non-conformances?

RQ3. Would a more behaviourally informed NON-CONFORMANCE NOTICE sent after an inspection regarding any identified non-conformances result in a better predicted response rate?

RQ4. How does the originating organisation who sends a message about safety shift the predicted response of the community of regulated entities (RECs/RGIs)?

RQ5. What is the level of safety behaviour and motivation among the respondents? How do they differ by type of respondent?

RQ6. What are the demographic characteristics of the respondents?

A different variant of the survey experiment was provided to the three key respondent groups of (a) the regulated entities (RECs and RGIs); (b) the staff of SSB; and (c) the staff of the CRU. Within each group, respondents were randomised to different arms of the study (see Table 4.6).

Table 4.6. Survey groups and experimental arms

Group	Additional stratification	Experimental arms
RECs and RGIs	(i) scheme (electrical [REC] or gas [RGI]); (ii) registered for more than 12 months (or not);3 and (iii) working as a single-person operation or within a firm.	Arm 1 Arm 2 Arm 3
SSB Staff	None	Arm 1 Arm 2
CRU Staff	None	Arm 1 Arm 2

Note: Respondents in each strata will be randomly assigned to the various experimental arms. Staff from the SSBs and the CRU will not be stratified and will only be randomised to the first two experimental arms due to the limited sample size.

Annex Table 4.B.1 in Annex 4.B provides an overview of the rationale for the six main research questions that the study addressed and how the framing changes based on the experimental arm.4

Outcome variables

For RQ1, the salience of the association between safety and a given entity was calculated as follows. Each respondent was asked to associate three words with the entity that they were randomly allocated to. These three words were then analysed using NVivo’s Text Search Query to determine how frequently the word “safety” and its synonyms appeared (“content analysis”) and the results were also coded by hand as a quality control mechanism. This frequency was then used as the outcome variable in the relevant analytical formula.

For RQ2 and RQ3, two outcomes were calculated and analysed separately. The first was the reported percentage5 of peers whom the respondent expected would resolve red, amber, and yellow non-conformances (averaged together for each respondent). The second was the reported per cent of the time that the respondent themselves expected that they would resolve the red, amber, and yellow non-conformances (averaged together for each respondent).

For RQ4, the four outcomes will be analysed. First i) was the reported percentage of entities that the respondent expects would read the message. Second ii) was the reported percentage of other entities that the respondent expects would implement the safety behaviour suggested. Third and fourth was the per cent of the time that the respondent themselves expected they would iii) read and iv) implement the message respectively.

Descriptive variables

For RQ5 and RQ6 the analysis is descriptive and not looking at the outcomes of the random allocation but only the descriptive group means.

For RQ5, two variables will be constructed. The first will be constructed from the eight items of the safety behaviour subscale of (Neal, Griffin and Hart, 2000[39]). The eight items will be averaged together to generate an average “behaviour” score with a maximum of 5 and a minimum of 1. Questions included items such as “I carry out my work in a safe manner” and “I ensure the highest levels of safety when I carry out my job”. If a respondent did not complete a single item, that item is dropped for that respondent. The second used the four items from the safety motivation subscale from the same questionnaire.6 The same procedure to generate an average construct score was used. Questions included items such as “I feel that it is important to maintain safety at all times”.

For RQ6, we present various descriptive data such as the average age, the geographic distribution of responses (by county of residence), the percentage of English speakers, the experience of the respondent, as well as other relevant data collected to understand the composition of each of the respondent groups.

Analysis

The primary analysis was conducted in the R statistical program. Ordinary least squares (OLS) regressions were used to leverage the individual randomisation and the stratification design – with separate regressions run for the various strata.7 Due to the limited number of respondents who had been registered for less than 12 months, the ‘new’ and the ‘old’ registration strata were merged for both RECs and RGIs.

For each stratum and each outcome, the outcome was regressed on a dummy treatment variable (Equation 1).

Equation 1 $\mathit{Y}\mathrm{ }~\mathrm{ }{\mathbf{\beta }}_1{\mathit{T}\mathit{r}\mathit{e}\mathit{a}\mathit{t}\mathit{m}\mathit{e}\mathit{n}\mathit{t}}_{2-1}+\mathrm{ }{\mathbf{\beta }}_2{\mathit{T}\mathit{r}\mathit{e}\mathit{a}\mathit{t}\mathit{m}\mathit{e}\mathit{n}\mathit{t}}_{3-1}$

The average treatment effect of being assigned to Treatment Arm 2 instead of Treatment Arm 1 will therefore be given by β_1. The average treatment effect of being assigned to Treatment Arm 3 instead of Treatment Arm 1 will be given by β_2.

In order to test how robust our results were, we conducted additional analysis with covariate adjustments for age and reported non-conformances as shown in Equation 2. These analyse8 did not substantively change the findings and therefore are not reported but are available upon request.

Equation 2 $\mathit{Y}\mathrm{ }~\mathrm{ }{\mathbf{\beta }}_1{\mathit{T}\mathit{r}\mathit{e}\mathit{a}\mathit{t}\mathit{m}\mathit{e}\mathit{n}\mathit{t}}_{2-1}+\mathrm{ }{\mathbf{\beta }}_2{\mathit{T}\mathit{r}\mathit{e}\mathit{a}\mathit{t}\mathit{m}\mathit{e}\mathit{n}\mathit{t}}_{3-1}+\mathrm{ }{\mathbf{\beta }}_3\mathit{A}\mathit{g}\mathit{e}\mathrm{ }+\mathrm{ }{\mathbf{\beta }}_4\mathit{N}\mathit{o}\mathit{n}\mathit{C}\mathit{o}\mathit{n}$

All findings reported below use the first treatment group as the reference group—this is usually the business-as-usual (BAU) group unless otherwise stated.

Timeline

The project timeline is presented in Figure 4.6. A pre-notification email was sent to all potential respondents on Monday 3 December 2018. This pre-notification was sent out by the CRU and the SSB to notify potential respondents that a survey will be sent out on 4 December from the OECD. The next day on Tuesday 4 December 2018 around 11am, the survey email was sent out to approximately 40 staff at the CRU, 40 staff at the SSBs, 1 500 RECs, and 1 500 RGIs. In total, 3 054 emails were sent out of which 117 were undeliverable (57 RECs and 60 RGIs) and 15 opted out. A reminder email was sent on Tuesday 18 December and again on Tuesday 8 January 2019 (both sent between 11am and noon). All responses were anonymous by design using an anonymous link to the Qualtrics-managed survey.

Figure 4.6. Project timeline

Sample characteristics

Responses were accepted from 4 December 2018 through 17 January 2019. In total, 364 potential respondents were asked for their informed consent of whom 96% (n = 349) consented. See full breakdown in Table 4.7. The vast majority (96%, n = 283/296) of RECs and RGIs had been registered more than a year and their average age was 47. Among RECs and RGIs most worked with others (59%, n = 176/296) (herein referred to as firms), and the remainder worked alone (herein individuals).

Table 4.7. Survey respondents, by type

Total number of respondents: 349

Entity	Respondents	Percentage of total respondents
RECs	164	47%
RGIs	132	38%
CRU	19	5%
Safe Electric	15	4%
RGII	10	3%

The respondents from the SSBs had been with the SSBs for an average of 8 years and had worked in the sector for an average of 28 years. The respondents from the CRU had worked at the CRU for an average of 6 years and had worked an average of 1 year in the sector prior to joining the CRU. Respondents came from counties throughout Ireland.

Figure 4.7. Consent rates and type of respondent

Results

The salience of safety (RQ1)

RECs and RGIs working alone, are more likely to associate safety with their respective SSBs. When asked what words and concepts respondents associated with either the CRU or the SSBs, those who worked as individual RECs or RGIs were more likely to associate safety with the SSBs rather than the CRU, providing 0.40 ([0.08, 0.71; p = 0.02) and 0.38 ([0.07, 0.69; p = 0.02) more words referencing safety respectively. There were no differences in the frequency of noting safety related terms among RECs or RGIs working within a firm. A combined analysis of staff from the CRU and the SSBs also found that they were more likely to associate the SSBs with safety (B = 0.29 [0.02, 0.56]; p = 0.03). See Table 4.8. An analysis combining all strata also found the same pattern in favour the stronger association of the SSBs with safety (B = 0.27 [0.14, 0.40]; p = 0.00).

Table 4.8. Salience of safety association when selecting THREE descriptors (SSBs vs. CRU)

Strata	N	Contrast	Estimate	Lower CI	Upper CI	P-Value
1_REC_SOLO	36	SSB – CRU	0.40	0.08	0.71	0.02*
2_REC_FIRM	56	SSB – CRU	0.17	-0.1	0.43	0.22
3_RGI_SOLO	35	SSB – CRU	0.38	0.07	0.69	0.02*
4_RGI_FIRM	39	SSB – CRU	0.17	-0.15	0.5	0.29
5_STAFF	42	SSB – CRU	0.29	0.02	0.56	0.03

The safety effect of primacy and implementation intentions on inspections (RQ2)

Redesigning the report template used by inspectors by placing more common non-conformances earlier in the form and adding an implementation intention resulted in RGIs in firms expecting a higher non-conformance resolution rate among their peers (15.73 ppt. [0.00, 31.46]; p = 0.05).9 Simply re-ordering the report (without the implementation intention) also suggested a higher hypothetical resolution rate (14.01 ppt.), but this latter finding was less precise ([-1.74, 29.76]; p = 0.08). The behaviourally informed redesign of the forms did not have an impact among individual RECs, RECs in firms, or individual RGIs. See Table 4.9. There were no differences when analysing all staff jointly or when analysing all strata together.

Table 4.9. Effects of primacy and implementation intentions in inspection reports on non-conformance resolutions (PEERS)

Strata	N	Contrast	Estimate	Lower CI	Upper CI	P-Value
1_REC_SOLO	51	T2-BAU	-0.13	-12.81	12.54	0.98
		T3-BAU	-13.40	-31.73	4.93	0.15
2_REC_FIRM	81	T2-BAU	-7.85	-19.18	3.47	0.17
		T3-BAU	-3.28	-14.68	8.12	0.57
3_RGI_SOLO	41	T2-BAU	11.90	-8.60	32.40	0.25
		T3-BAU	-0.67	-20.50	19.17	0.95
4_RGI_FIRM	60	T2-BAU	14.01	-1.74	29.76	0.08+
		T3-BAU	15.73	0.00	31.46	0.05*

When asked about their own hypothetical resolution behaviour, a similar pattern emerged. RGIs in firms were more likely to report that they themselves would resolve the non-conformances when presented with a redesigned inspection report with both re-ordering (primacy) and implementation intentions – although this result lacked precision (15.34 ppt. [-2.08, 32.76]; p = 0.08). Re-ordering alone was insufficient (13.51 ppt. [-3.50, 30.52]; p = 0.12). There were no effects among individual RECs, RECs in firms, or individual RGIs. See Table 4.10. There were no differences when analysing all staff jointly or when analysing all strata together.

Table 4.10. Effects of primacy and implementation intentions in inspection reports on non-conformance resolutions (SELF)

Strata	N	Contrast	Estimate	Lower CI	Upper CI	P-Value
1_REC_SOLO	48	T2-BAU	-11.37	-26.75	4.01	0.14
		T3-BAU	-12.45	-30.27	5.36	0.17
2_REC_FIRM	77	T2-BAU	-6.46	-18.17	5.25	0.28
		T3-BAU	-3.23	-14.84	8.38	0.58
3_RGI_SOLO	41	T2-BAU	-6.46	-24.26	11.33	0.47
		T3-BAU	-9.56	-28.23	9.12	0.31
4_RGI_FIRM	58	T2-BAU	13.51	-3.5	30.52	0.12
		T3-BAU	15.34	-2.08	32.76	0.08+

The safety effect of personalisation and implementation intentions on notifications (RQ3)

When considering the effects of a redesign of the non-conformance notices using the original notifications (T1), personalised notifications (T2), and notifications with both personalisation and implementation intentions (T3),10 effects only emerged among RGIs reporting their own hypothetical behaviour. The changes exerted opposing effects on individual RGIs and RGIs in firms. Individual RGIs were less likely to answer that they would resolve their non-conformances if they received a personalised notification (T2) when compared to the BAU notice (-13.89 ppt. [-27.37, -0.40]; p = 0.04). Among RGIs in firms, they were more likely (+12.65 ppt. [-1.63, 26.94]; p = 0.08) to answer that they would resolve their non-conformances if they were assigned to consider a notification that was both personalised and included implementation intentions. There were no effects among RECs (see Table 4.11). There were no differences when analysing all staff jointly or when analysing all strata together.

Table 4.11. Effects of personalisation and implementation intention in notifications on non-conformance resolution (SELF)

Strata	N	Contrast	Estimate	Lower CI	Upper CI	P-Value
1_REC_SOLO	42	T2-BAU	-3.25	-16.38	9.87	0.62
		T3-BAU	-6.89	-21.31	7.53	0.34
2_REC_FIRM	67	T2-BAU	0.21	-13.29	13.72	0.98
		T3-BAU	0.07	-12.85	12.99	0.99
3_RGI_SOLO	36	T2-BAU	-13.89	-27.37	-0.4	0.04*
		T3-BAU	-6.67	-15.73	2.4	0.14
4_RGI_FIRM	52	T2-BAU	8.26	-7.88	24.4	0.31
		T3-BAU	12.65	-1.63	26.94	0.08+

The analysis using reference shifting in which respondents stated how they expected their peers to behave upon receipt of the notification revealed no differences between those assigned to the three types of notification. See Table 4.12. There were no differences when analysing all staff jointly or when analysing all strata together.

Table 4.12. Effects of personalisation and implementation intentions in notifications on non-conformance resolution (PEERS)

Strata	N	Contrast	Estimate	Lower CI	Upper CI	P-Value
1_REC_SOLO	45	T2-BAU	2.64	-12.74	18.02	0.73
		T3-BAU	-6.94	-26.18	12.3	0.47
2_REC_FIRM	72	T2-BAU	-0.19	-10.54	10.15	0.97
		T3-BAU	0.74	-10.14	11.63	0.89
3_RGI_SOLO	41	T2-BAU	3.17	-12.2	18.53	0.68
		T3-BAU	-9.41	-26.7	7.88	0.28
4_RGI_FIRM	54	T2-BAU	5.67	-7.57	18.91	0.39
		T3-BAU	3.21	-9.79	16.21	0.62

The safety effect of messengers (RQ4)

The study next considered the effect of the messenger when receiving safety information from either the CRU (T1), their SSB (T2), or their peer trainer (T3). Once again, respondents were asked to consider how their peers would respond and how they would respond. They were asked both how likely it would be that their peers (or themselves) would read the information and separately how likely they would be to implement the suggestion.

When considering their peers reactions, RGIs in firms assigned to the scenario with a peer as the messenger suggested that other RGIs would be 15 ppt. more likely to read such information (p = 0.052) and 22 ppt. more likely to implement the suggestion (p = 0.004). There were no messenger effects for the other three groups (individual RECs, RECs in firms, and individual RGIs). See

Table 4.13. There were no differences when analysing all staff jointly or when analysing all strata together.

Table 4.13. Effects of messengers for reading and acting on safety improvement suggestions (PEERS)

Strata	N	Contrast	Estimate	Lower CI	Upper CI	P-Value
PEERS would READ
1_REC_SOLO	43	SSB-CRU	15.50	-5.64	36.64	0.15
		TRAINER-CRU	11.96	-7.95	31.87	0.23
2_REC_FIRM	69	SSB-CRU	-1.30	-17.52	14.93	0.87
		TRAINER-CRU	1.11	-12.20	14.42	0.87
3_RGI_SOLO	36	SSB-CRU	8.97	-4.42	22.37	0.18
		TRAINER-CRU	-5.87	-23.34	11.60	0.50
4_RGI_FIRM	52	SSB-CRU	6.63	-9.77	23.03	0.42
		TRAINER-CRU	15.11	-0.11	30.33	0.05*
PEERS would ACT
1_REC_SOLO	43	SSB-CRU	7.67	-13.66	28.99	0.47
		TRAINER-CRU	-3.79	-24.84	17.25	0.72
2_REC_FIRM	69	SSB-CRU	-3.33	-19.94	13.27	0.69
		TRAINER-CRU	-0.56	-15.04	13.93	0.94
3_RGI_SOLO	36	SSB-CRU	6.92	-9.03	22.87	0.38
		TRAINER-CRU	-5.80	-27.44	15.84	0.59
4_RGI_FIRM	53	SSB-CRU	13.12	-1.49	27.74	0.08+
		TRAINER-CRU	21.54	7.09	36.00	0.00***

When considering their own hypothetical reactions, RGIs in firms assigned to the peer-trainer messenger still reported the highest likelihood of reading and implementing the suggestion compared to the RGIs in firms assigned to scenarios with either the CRU (T1) or their SSB (RGII) (T2) as the messenger. However, the finding was no longer statistically significant for either reading (+8 ppt.; p = 0.28) or implementing the suggestion (+11 ppt.; p = 0.24). However, among RECs in firms, the opposite pattern emerged—they were 12 ppt. more likely to report that they would implement the suggestion when assigned to receiving the message from the CRU compared to from their peers (p = 0.09). Although the finding lacked precision, the same pattern was manifest among RECs in firms and the likelihood that they read the message, with those assigned to the CRU messenger scenario having the highest point-estimate likelihood of their reading the message—7 ppt. higher than the SSB (Safe Electric) messenger (p = 0.35) and 6 ppt. higher than the peer messenger (p = 0.32). See Table 4.14. There were no differences when analysing all staff jointly or when analysing all strata together.

Table 4.14. Effects of messengers for reading and acting on safety improvement suggestions (SELF)

Strata	N	Contrast	Estimate	Lower CI	Upper CI	P-Value
PEERS would READ
1_REC_SOLO	43	SSB-CRU	6.33	-16.24	28.91	0.57
		TRAINER-CRU	10.92	-6.74	28.57	0.22
2_REC_FIRM	69	SSB-CRU	-6.85	-21.47	7.77	0.35
		TRAINER-CRU	-5.97	-17.86	5.92	0.32
3_RGI_SOLO	36	SSB-CRU	0.26	-8.32	8.84	0.95
		TRAINER-CRU	-3.99	-14.60	6.62	0.45
4_RGI_FIRM	52	SSB-CRU	0.74	-15.52	16.99	0.93
		TRAINER-CRU	8.22	-6.90	23.35	0.28
PEERS would ACT
1_REC_SOLO	43	SSB-CRU	-2.50	-23.18	18.18	0.81
		TRAINER-CRU	-5.83	-26.74	15.07	0.58
2_REC_FIRM	69	SSB-CRU	-10.74	-26.60	5.12	0.18
		TRAINER-CRU	-11.67	-25.11	1.77	0.09+
3_RGI_SOLO	36	SSB-CRU	1.99	-11.30	15.27	0.76
		TRAINER-CRU	-0.21	-14.56	14.14	0.98
4_RGI_FIRM	52	SSB-CRU	1.68	-16.55	19.92	0.85
		TRAINER-CRU	10.78	-7.34	28.89	0.24

The safety behaviour and motivation of entities (RQ5)

The self-reported safety behaviour scores were high—and very near to their maximum. The mean safety behaviour was 4.53 for the full sample. The only statistically significant difference between subgroups was that Safe Electric staff reported higher levels of safety behaviour than did staff at the CRU (B = 0.573 [95% CI 0.009; 1.137]).

The self-reported safety motivation scores were similarly high. The mean safety motivation score was 4.70 for the full sample. The only statistically significant difference between subgroups was that the Safe Electric staff reported higher safety motivation than did RECs within firms (B = 0.121 [0.013, 0.229]). Although imprecise, the staff at Safe Electric also had higher safety motivation than staff at the CRU (B = 0.595 [‑0.024, 1.214]).

The respondents’ characteristics (RQ6)

RECs and RGIs were on average 47 years old and included respondents from all 31 counties. Almost all respondents reported speaking English at home, other languages spoken at home included Irish, Swahili, German, and Latvian. Twenty-nine per cent reported having had at least one non-conformance in their most recent inspection. Most respondents (75%) believe that the SSBs and the CRU contribute to safety. However, respondents estimate that approximately a third (35%) of all electrical or gas works are done illegally.

When considering how they would prefer to receive and report information, there is a preference for electronic communication.

Among SSB respondents, the average age was 50. Respondents had extensive experience, with an average of 8.3 years working for the SSB and an average of 28.3 years in the field. Respondents believe that the SSBs and the CRU are working towards the same goals 65% of the time. Respondents believe that inspectors usually apply the same codes of severity to non-conformances but there remains a minority that does not (27% for Safe Electric and 17% for RGII).

Among staff from the CRU, the average age was 35 and respondents had an average of 6 years of experience working at the CRU and an average of 1.4 years of experience in the industry prior to joining the CRU. Respondents felt that the SSBs and the CRU were working toward the same goals 65% of the time (the same as the SSB respondents). While respondents report a good understanding of the CRU’s role in regulating safety and strongly believe that the CRU is open to hearing new ways to improve, there was a lower level of agreement about how effective the CRU has been in regulating the gas and electricity sector.

Summary findings

The study revealed several key findings by comparing the different responses of respondents who were randomly allocated to different scenarios designed to investigate the potential impact of behavioural insights. The study also collected key descriptive information about the safety schemes. Respondents were grouped as 1) RECs working individually; 2) RECs working in firms; 3) RGIs working independently; 4) RGIs working in firms; 5) Safe Electric staff; 6) RGII staff; and 7) the CRU staff. Respondents were randomised within groups and we focus primarily on the results for the regulated entities (the first four groups). The following provides a summary of key findings.

• The different groups (‘strata’) showed different results, indicating the importance of targeting behaviourally informed interventions differently with the groups.

• Individual RECs and individual RGIs who were randomly allocated to reply with the three words they associate with the SSBs were more likely to list words associated with safety than the group of respondents who listed words associated with the CRU. This revealed that RECs and RGIs working individually are more likely to associate SSBs with safety than to associate the CRU with safety—sharing 0.40 and 0.38 more safety related words respectively for SSBs on average.

• Safety behaviour and safety motivation among respondents were high. The full sample had an average of 4.58 and 4.70 respectively on a 5-point scale. The staff of the CRU reported the lowest safety behaviour and safety motivation (4.19 and 4.38 respectively) while Safe Electric reported the highest levels (4.77 and 4.98).

The study also tested several scenarios in which the inspection report and the non-conformance notice included or excluded behavioural insights. Respondents were assigned to three groups. For reports, one group was presented with the current report, the second group was presented a version of the report that applied primacy by moving inspection areas with common non-conformances to the beginning of the form,11 and the third group was shown a report that included primacy combined with a prompt to make a specific plan for addressing any non-conformance (an implementation intention).

For the notice, group one saw the current version, group two was shown a more personalised version, and group three was shown a notice that combined the personalisation with an implementation intention.

Finally, the three groups were randomly assigned to scenarios in which safety messages were received from either the CRU, the SSB, or a trainer in order to test messenger effects.

• The experiment found that combining primacy and implementation intentions in inspection reports improved the expected likelihood of resolving non-conformances (+15 ppt. for themselves to +16 ppt. for peers), but only among RGIs who work in firms. Other types of respondents showed no impact from the changes.

• For notices, RGIs in firms were also more likely to expect timely resolutions of non-conformances when they were presented with notices that used both personalisation and implementation intentions (+13 ppt.). Individuals working alone as RGIs expected worse resolution rates with personalised notifications (-14 ppt.). Other types of respondents did not experience any impact.

• Finally, RGIs in firms were also more likely to incorporate suggestions to improve safety if the messenger was a previous trainer (+15 ppt.). RECs in firms responded more favourably to messages from the CRU (+12 ppt.) compared to trainer.

Implications for safety policy

The following section summarises the findings according to each of the research questions.

RQ1: How salient is the association of safety with actors and activities associated with the schemes? How do they differ among respondents?

Finding: SSBs are more closely associated with safety than the CRU—especially among RECs and RGIs who are working individually (not in firms). This may be due to the closer interaction between individual RECs and RGIs and their respective SSBs. Interestingly, this was also true for the staff of the CRU who were more likely to choose words associated with safety when they were assigned to describe the SSBs than when they were assigned to describe the CRU.12

Implication: Given the prominent role of the SSBs, this finding appears to confirm that the regulatory safety scheme in Ireland is associated with the SSBs—with RECs, RGIs, and the CRU all being more likely to associate SSBs with safety than to associate the CRU with safety. This may point to the need for additional awareness raising of the role of the CRU regarding safety among RECs and RGIs working individually to help better associate the CRU with safety.

RQ2: Would a more behaviourally informed INSPECTION REPORT increase the accurate identification and timely resolution of non-conformances?

Finding: The behaviourally informed changes only had an impact on RGIs in firms. This group reported higher expected rates of resolving non-conformances when assigned to the group that viewed a revised inspection report in which common non-conformances were moved earlier in the inspection report (primacy) and implementation intentions were also added. This was true when they reported based on their own predicted behaviour and that of their peers. When considering their peers, only changing the order of inspection items (primacy) also had an impact on expected non-conformance resolution rates, albeit with less precision.

Implication: New inspection forms that are re-ordered based on common violations and include an implementation intention planning prompt can have a positive effect on RGIs in firms and should not have any negative effect on RGIs who work on their own. Accordingly, new forms should be field tested with RGIs measuring observed behaviour. There does not seem to be any added value in changing the REC report form along the lines of the tested behavioural insights.

RQ3: Would a more behaviourally informed NON-CONFORMANCE NOTIFICATION sent after an inspection regarding any identified non-conformances result in a better predicted response rate?

Finding: RGIs were impacted by the behaviourally informed changes to the non-conformance notice; however, these effects differed dramatically between RGIs in firms and RGIs working individually. RGIs working individually reported the highest non-conformance resolution rates with the current business-as-usually notice and the worst rates with the personalised notice. RGIs working in firms showed the best expected rates of resolution if they were shown the notice with both personalisation and implementation intention. This may suggest that RGIs who are working individually may feel that personalisation raises privacy concerns while firms expect to have their details used in correspondence.

Implication: The notification of non-conformances should be differentiated for RGIs working individually and those working in firms. For individuals it may be counter-productive to include personalisation while for RGIs in firms a new notice combining personalisation and an implementation intention should be tested and potentially rolled out. For both groups, it may be useful to test combining implementation intentions with a different behavioural insight, such as feedback.

RQ4: How does the originating organisation (the messenger) who sends a message about safety shift the predicted response of the community of regulated entities (RECs/RGIs)?

Finding: The effect of different messengers on the likelihood of attending to and implementing non-binding suggestions for improved safety was noted for RGIs working in firms.13 RGIs in firms were more likely to read and implement suggestions coming from the person who had previously trained them (a more experienced peer).

Implication: This suggests that RGII and the CRU should consider putting in place systems for trainers to keep in contact with RGIs who work in firms and to utilise these individuals to deliver safety related suggestions. Such a change could be as simple as obtaining permission from trainers to use their likeness and names in subsequent communication materials that are sent to the RGIs they trained. However, this change should not be done for RGIs working individually because such a strategy seems to be ineffective and there is a risk that the effect may even be negative for this group. Therefore reliable systems for targeting RGIs working alone and those working in firms separately should be put in place.

RQ5: What is the level of safety behaviour and motivation among the respondents? How do they differ by type of respondent?

Finding: The level of safety behaviour and safety motivation reported were extremely high among all types of respondents. In all cases the average self-reported rating was over 4 on a 5-point scale. Safe Electric staff had the highest safety motivation (4.98) and safety behaviour (4.77) and the CRU staff had the lowest (4.38 and 4.19 respectively).

Implication: Using this limited metric, the safety behaviour and safety motivation in the sector appears to be high. However, the metric is not sufficiently sensitive and future research should identify more sensitive measures considering the fact that 29% of RECs and RGIs reported having a non-conformance in the previous year.

Conclusions

Together, these findings highlight the importance of taking into consideration issues related to individuals’ understanding and capacity of relating to and absorbing information when designing regulatory and enforcement schemes. It also highlights the importance of tailoring behaviourally informed interventions for different populations and avoiding broad generalisations. Findings also suggest that most of the tested behaviourally informed changes (primacy, personalisation, implementation intentions, and experienced peer messengers who previously conducted training) would be effective for RGIs working in firms but not for the other respondent groups. The study also illustrates the utility of testing behavioural principals and their effect on safety related processes and procedures through a stratified survey experiment prior to testing changes in practice.

Results from the study suggest that the CRU and RGII consider a field trial to further test the effects of these behaviourally informed adjustments for RGIs working in firms in Ireland while measuring observed non-conformances. For the other groups, results suggest that further behavioural principles be tested. For example, considering the potential negative effect of personalisation among individual RGIs and RECs, future tests could remove personalisation and attempt to pair implementation intentions with another behavioural insight, such as feedback — which showed promise in the Phase I study.

Together, the two phases of this project demonstrate the value of the iterative testing approach that is foundational to the behavioural insights methodology. This was especially true when comparing comparative to national-level results, which gives encouragement to other countries to engage in further research to gain a more holistic understanding of the drivers of safety culture in the energy sector. This study has also highlighted the potential for the continuous testing and improvement of the procedures and communications through the application of behavioural insights within the safety schemes.

References

[33] BETA (2018), Nudge vs Superbugs: A behavioural economics trial to reduce the overprescribing of antibiotics, The Behavioural Economics Team of the Australian Government, Department of the Prime Minister and Cabinet, https://www1.health.gov.au/internet/main/publishing.nsf/Content/Nudge-vs-Superbugs-behavioural-economics-trial-to-reduce-overprescribing-antibiotics-June-2018 (accessed on 2 March 2020).

[16] Bicchieri, C. (2005), The grammar of society: The nature and dynamics of social norms, Cambridge University Press, Cambridge, http://dx.doi.org/10.1017/CBO9780511616037.

[8] CER (2017), Requirements of the Petroleum Safety Framework, Commission for Energy Regulation, Dublin, https://www.cru.ie/wp-content/uploads/2017/11/CER16023-Requirements-of-the-Petroleum-Safety-Framework-V1.1.pdf (accessed on 2 March 2020).

[14] Clark, J. et al. (2013), “Evaluating the message or the messenger? Implications for self-validation in persuasion.”, Personality & social psychology bulletin, Vol. 39/12, pp. 1571-84, http://dx.doi.org/10.1177/0146167213499238.

[23] Conte, R. et al. (2007), Emergent and Immergent Effects in Complex Social Systems, Proceedings of AAAI Symposium, Social and Organisational Aspects of Intelligence, https://www.aaai.org/Papers/Symposia/Fall/2007/FS-07-04/FS07-04-007.pdf (accessed on 2 March 2020).

[10] Cooper, D. (2000), “Towards a model of safety culture”, Safety Science, Vol. 36/2, pp. 111-136, http://dx.doi.org/10.1016/S0925-7535(00)00035-7.

[13] Corkindale, G. (2010), Five Leadership Lessons from the BP Oil Spill, https://hbr.org/2010/06/five-lessons-in-leadership-fro (accessed on 19 March 2020).

[15] Eckel, C. and H. Gintis (2010), “Blaming the messenger: Notes on the current state of experimental economics”, Journal of Economic Behavior and Organization, Vol. 73/1, pp. 109-119, http://dx.doi.org/10.1016/j.jebo.2009.03.026.

[19] Fehr, E., U. Fischbacher and S. Gächter (2002), “Strong reciprocity, human cooperation, and the enforcement of social norms”, Human Nature, Vol. 13/1, pp. 1-25, http://dx.doi.org/10.1007/s12110-002-1012-7.

[17] Goldstein, D. et al. (2008), Nudge Your Customers Toward Better Choices, https://hbr.org/2008/12/nudge-your-customers-toward-better-choices (accessed on 2 March 2020).

[37] Goldstein, N., R. Cialdini and V. Griskevicius (2008), “A Room with a Viewpoint: Using Social Norms to Motivate Environmental Conservation in Hotels”, Journal of Consumer Research, Vol. 35/3, pp. 472-482, http://dx.doi.org/10.1086/586910.

[24] Henning, J. et al. (2009), “The influence of individual differences on organizational safety attitudes”, Safety Science, Vol. 47/3, pp. 337-345, http://dx.doi.org/10.1016/j.ssci.2008.05.003.

[26] Hofstede, G. (2001), Culture’s Consequences: Comparing Values, Behaviors, Institutions, and Organizations Across Nations, Sage, Beverly Hills, California.

[28] Hofstede, G. (n.d.), Compare countries, Hofstede Insights, https://www.hofstede-insights.com/product/compare-countries/ (accessed on 2 March 2020).

[7] ISB (2012), Energy (Miscellaneous Provisions) Act, Irish Statute Book, http://www.irishstatutebook.ie/eli/2012/act/3/enacted/en/html (accessed on 2 March 2020).

[6] ISB (2006), Energy (Miscellaneous Provisions) Act, Irish Statute Book, http://www.irishstatutebook.ie/eli/2006/act/40/enacted/en/html (accessed on 2 March 2020).

[38] Lindhout, P. and G. Reniers (2017), “What about nudges in the process industry? Exploring a new safety management tool”, Journal of Loss Prevention in the Process Industries, Vol. 50, pp. 243-256, http://dx.doi.org/10.1016/j.jlp.2017.10.006.

[1] Lunn, P. (2014), Regulatory Policy and Behavioural Economics, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264207851-en.

[31] Mearns, K. and S. Yule (2009), “The role of national culture in determining safety performance: Challenges for the global oil and gas industry”, Safety Science, Vol. 47/6, pp. 777-785, http://dx.doi.org/10.1016/j.ssci.2008.01.009.

[39] Neal, A., M. Griffin and P. Hart (2000), “The impact of organizational climate on safety climate and individual behavior”, Safety Science, Vol. 34/1, pp. 99-109, http://158.132.155.107/posh97/private/behavioral-safety/organizational-climate-Neal.pdf (accessed on 12 November 2019).

[34] Noort, M. et al. (2016), “The relationship between national culture and safety culture: Implications for international safety culture assessments”, Journal of Occupational and Organizational Psychology, Vol. 89/3, pp. 515-538, http://dx.doi.org/10.1111/joop.12139.

[5] OECD (2019), Tools and Ethics for Applied Behavioural Insights: The BASIC Toolkit, OECD Publishing, Paris, https://dx.doi.org/10.1787/9ea76a8f-en.

[4] OECD (2017), Driving Performance at Mexico’s Agency for Safety, Energy and Environment, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264280458-en.

[3] OECD (2017), Driving Performance of Mexico’s Energy Regulators, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264267848-en.

[2] OECD (2016), Protecting Consumers through Behavioural Insights: Regulating the Communications Market in Colombia, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264255463-en.

[21] OECD (2016), “Research Ethics and New Forms of Data for Social and Economic Research”, OECD Science, Technology and Industry Policy Papers, No. 34, OECD Publishing, Paris, https://dx.doi.org/10.1787/5jln7vnpxs32-en.

[11] OECD/NEA (2013), The Fukushima Daiichi Nuclear Power Plant Accident OECD/NEA Nuclear Safety Response and Lessons Learnt, OECD/Nuclear Energy Agency (NEA), Paris, https://www.oecd-nea.org/pub/2013/7161-fukushima2013.pdf (accessed on 19 March 2020).

[29] Parand, A. et al. (2010), “The disparity of frontline clinical staff and managers’ perceptions of a quality and patient safety initiative”, Journal of Evaluation in Clinical Practice, Vol. 17/6, pp. 1184-1190, http://dx.doi.org/10.1111/j.1365-2753.2010.01506.x.

[20] Rand, D., E. Yoeli and M. Hoffman (2014), “Harnessing Reciprocity to Promote Cooperation and the Provisioning of Public Goods”, Policy Insights from the Behavioral and Brain Sciences, Vol. 1/1, pp. 263-269, http://dx.doi.org/10.1177/2372732214548426.

[27] Reader, T. et al. (2015), “Safety sans frontières: An international safety culture model”, Risk Analysis, Vol. 35/5, pp. 770-789, http://dx.doi.org/10.1111/risa.12327.

[12] Reader, T. and P. O’Connor (2014), “The Deepwater Horizon explosion: non-technical skills, safety culture, and system complexity”, Journal of Risk Research, Vol. 17/3, pp. 405-424, http://dx.doi.org/10.1080/13669877.2013.815652.

[25] Sharpe, J., N. Martin and K. Roth (2011), “Optimism and the Big Five factors of personality: Beyond Neuroticism and Extraversion”, Personality and Individual Differences, Vol. 51/8, pp. 946-951, http://dx.doi.org/10.1016/j.paid.2011.07.033.

[22] Tear, M. and M. MacLennan (2018), OECD Behavioural Insights for Safety Culture, Open Science Framework, https://osf.io/r482d (accessed on 2 March 2020).

[30] Tear, M. et al. (2020), “Safety culture and power: Interactions between perceptions of safety culture, organisational hierarchy, and national culture”, Safety Science, Vol. 121, pp. 550-561, http://dx.doi.org/10.1016/j.ssci.2018.10.014.

[36] Terry, D. and M. Hogg (2000), Attitudes, Behavior, and Social Context: The Role of Norms and Group Membership, L. Erlbaum Associates, Mahwah, N.J.

[35] Terry, D., M. Hogg and K. White (1999), “The theory of planned behaviour: Self- Identity, social identity and group norms”, British Journal of Social Psychology, Vol. 38/3, pp. 225-244, http://dx.doi.org/10.1348/014466699164149.

[32] The Behavioural Insights Team (2011), Behaviour Change and Energy Use, The Behavioural Insights Team, London, https://www.bi.team/wp-content/uploads/2015/07/behaviour-change-and-energy-use.pdf (accessed on 2 March 2020).

[9] Transportation Research Board (ed.) (2016), Strengthening the Safety Culture of the Offshore Oil and Gas Industry, The National Academies Press, Washington DC, https://www.nap.edu/catalog/23524/strengthening-the-safety-culture-of-the-offshore-oil-and-gas-industry.

[18] Zohar, D. (1980), “Safety climate in industrial organizations: Theoretical and applied implications”, Journal of Applied Psychology, Vol. 65/1, pp. 96-102, http://dx.doi.org/10.1037/0021-9010.65.1.96.

Annex 4.A. Data from country analysis in Phase 1

Ireland: Results of safety culture by occupational role moderation on vignette effectiveness

Annex Figure 4.A.1. The effect of safety culture (sc) on overall vignette effectiveness (scale) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.2. The effect of safety culture (sc) on vignette effectiveness for attracting attention of workers (item1) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.3. The effect of safety culture (sc) on whether vignette should affect behaviour of workers (item2) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.4. The effect of safety culture (sc) on whether vignette would affect behaviour of workers (item3) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line).The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.5. The effect of safety culture (sc) on whether vignette should affect behaviour of managers/the entity overall (item4) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.6. The effect of safety culture (sc) on whether vignette would affect behaviour of managers/the entity overall (item5) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Mexico: Results of safety culture by occupational role moderation on vignette effectiveness

Annex Figure 4.A.7. The effect of safety culture (sc) on overall vignette effectiveness (scale) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.8. The effect of safety culture (sc) on vignette effectiveness for attracting attention of workers (item1) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.9. The effect of safety culture (sc) on whether vignette should affect behaviour of workers (item2) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line)). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.10. The effect of safety culture (sc) on whether vignette would affect behaviour of workers (item3) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.11. The effect of safety culture (sc) on whether vignette should affect behaviour of managers/the entity overall (item4) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.12. The effect of safety culture (sc) on whether vignette would affect behaviour of managers/the entity overall (item5) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line)). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Oman: Results of safety culture by occupational role moderation on vignette effectiveness in Oman

Annex Figure 4.A.13. The effect of safety culture (sc) on overall vignette effectiveness (scale) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.14. The effect of safety culture (sc) on vignette effectiveness for attracting attention of workers (item1) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.15. The effect of safety culture (sc) on whether vignette should affect behaviour of workers (item2) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.16. The effect of safety culture (sc) on whether vignette would affect behaviour of workers (item3) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.17. The effect of safety culture (sc) on whether vignette should affect behaviour of managers/the entity overall (item4) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex Figure 4.A.18. The effect of safety culture (sc) on whether vignette would affect behaviour of managers/the entity overall (item5) at levels of occupational role

Note: a4: 1 frontline staff (blue line); 2 managers (red line); 3 senior managers (green line); or 4 other staff (orange line). The steeper a line, then the more that occupational role is affected by the perceived culture of the entity in estimated the effectiveness of the vignettes.

Annex 4.B. Additional information from Phase 2

Research questions

Annex Table 4.B.1. Research questions

Research question	Rationale	Experimental arms
		Arm #1	Arm #2	Arm #3 (RECs and RGIs only)
RQ1. How salient is the association of safety with actors and activities associated with the schemes? How do they differ among respondents?	The focus on following procedures and standards may crowd out the salience of the focus on safety and replace it with a more bureaucratic focus on rule-following which could weaken safety culture. Behavioural insight(s): salience, priming	Prime: CRU	Prime: SSB	N/A
RQ2. Would a more behaviourally informed INSPECTION REPORT increase the predicted accurate identification and timely resolution of non-conformances?	The current report format has a different design for RECs and RGIs, even in elements that share content (such as sections for basic information, non-conformances, and signatures). This difference might account for a proportion of the differences in rates of non-conformance identification and resolution. Making use of the colour coding and primacy effects (what is presented at the beginning, middle, and bottom of the page) or an implementation intention (an explicit action plan) could shift the prioritisation of inspectors and the understanding of RECs/RGIs. Behavioural insight(s): primacy, implementation intentions	Control (BAU)	Default Order Changed to Test Primacy Effects (Primacy)	Prompted Implementation Intention Commitment Added (Primacy + Implementation Intention)
RQ3. Would a more behaviourally informed NON-CONFORMANCE NOTICE sent after an inspection regarding any identified non-conformances result in a better predicted response rate?	The current notice is generic and does not contain many of the common best practices for behaviourally informed communication designed to improve behavioural follow-through—such as personalisation and clear action steps. This survey can provide an initial test of the predicted effect of a change in design on hypothetical behaviour that could inform a change in the notices used by the SSBs. Behavioural insight(s): salience, personalisation, implementation intentions	Control (BAU)	Increased Salience & Ease of Use Through Personalisation (Personalisation)	Prompted Implementation Intention Commitment Added (Personalisation + Implementation Intention)
RQ4. How does the originating organisation who sends a message about safety shift the predicted response of the community of regulated entities (RECs/RGIs)?	Messages may be more salient if they come from an authoritative governmental body (the CRU), a known and trusted organisation with direct responsibility for the scheme (SSB), or a peer with more social proximity and perceived expertise (a REC / RGI who previously trained them). If the community is more responsive to a particular messenger, that information could shift the design, branding, and dissemination of key resources and messages. Behavioural insight(s): salience, messenger effects	Messenger: CRU	Messenger: SSB	Messenger: Peer (trainer)
RQ5. What is the level of safety behaviour and motivation among the respondents? How do they differ by type of respondent?	First, this information will provide a descriptive overview of the safety culture and safety environment in the sector. Second, this information will enable us to investigate the degree to which these variables differ among the various respondent groups (i.e. the RECs, RGIs, the CRU, and the SSBs).	n/a	n/a	n/a
RQ6. What are the demographic characteristics of the respondents?	Similarly to RQ5, this information will provide a descriptive overview and enable us to investigate if there are unique interactions between demographics and observed results.	n/a	n/a	n/a

REC Reports

Annex Figure 4.B.1. Group 1 (Business-as-Usual)

Annex Figure 4.B.2. Group 2 (Primacy)

Annex Figure 4.B.3. Group 3 (Primacy + Implementation Intention)

RGI Reports

Annex Figure 4.B.4. Group 1 (Business-as-usual)

Annex Figure 4.B.5. Group 2 (Primacy)

Annex Figure 4.B.6. Group 3 (Primacy + Implementation Intention)

REC Notices

Annex Figure 4.B.7. Group 1 (Business-as-Usual)

Annex Figure 4.B.8. Group 2 (Personalisation)

Annex Figure 4.B.9. Group 3 (Personalisation + Implementation Intention)

RGI Notices

Annex Figure 4.B.10. Group 1 (Business-as-Usual)

Annex Figure 4.B.11. Group 2 (Personalisation)

Annex Figure 4.B.12. Group 3 (Personalisation + implementation intention)

Notes

← 1.  The time given to resolve a non-conformance ranges from 24 hours for severe non-conformances to within 21-days for less severe non-conformances.

← 2.  Some electricians work under a registered REC who is responsible for certifying their electrical works.

← 3.  As noted below, this stratification variable was collapsed because so few respondents were in the scheme for less than 12 months.

← 4.  The study was reviewed and exempted by the Teachers College, Columbia University ethical review board (Protocol #19-069) and the analysis plan for the primary results was registered with the American Economics Association’s Social Science Trial Registry (#0003796) prior to downloading and analysing the survey data. This process follows best practice research protocols that are adhered to when conducting experiments involving samples of individuals.

← 5.  Note that for each respondent percentages were rounded to the nearest 10 digit to reduce variance that is unlikely to be meaningful for the respondent.

← 6.  While safety culture has many more facets beyond behavior and motivation, we prioritised creating a concise survey that would encourage higher respondent engagement instead of using a longer tool

← 7.  We originally planned to conduct an overall test across all strata; however, because different pre-specified strata reacted differently (even in opposite reactions) we primarily report the results by strata and only briefly note the results of the model that includes all groups with strata as a control variable.

← 8.  The study was reviewed and exempted by the Teachers College, Columbia University ethical review board (Protocol #19-069) and the analysis plan for the primary results was registered with the American Economics Association’s Social Science Trial Registry (#0003796) prior to downloading and analysing the survey data. This process follows best practice research protocols that are adhered to when conducting experiments involving samples of individuals.

← 9.  See ‎Annex 4.B for the version of the inspection reports tested with the three respondent groups.

← 10.  See ‎Annex 4.B for notices presented to the three different respondent groups.

← 11.  Common non-conformances were identified using previous quarterly reports from the SSBs.

← 12.  This may be due, in part, to the wider remit of the CRU.

← 13.  The primary trend was among RGIs in firms in favour of peer-trainers as messengers. However, RECs in firms did show some preference for messages being sent by the CRU instead of a peer-trainer.