Nurturing Social and Emotional Learning Across the Globe

Annex A. Technical background

The OECD would like to acknowledge and thank the Ayrton Senna Institute (São Paulo, Brazil), the institute that supported the development phases of the psychometric work in this assessment, and 2E Estudios y Evaluaciones, the contractor that conducted data processing and scaling for the Survey on Social and Emotional Skills (SSES) 2023 Main Survey.

Construction of social and emotional skill assessment scales

Social and emotional skill scales in SSES are scaled to fit approximately normal distributions with means around 500 and standard deviations around 100. In statistical terms, a one-point difference on a skill scale therefore corresponds to an effect size (Cohen’s d) of 0.01; and a ten-point difference to an effect size of 0.10.

The SSES assessment, like all assessments, is susceptible to several possible measurement errors. Despite the extensive investments SSES makes in monitoring the translation process, standardising the administration of the assessment, selecting questions, and analysing the data quality, complete comparability across countries and subpopulations cannot always be guaranteed. While self-reported questionnaires are an established method for measuring social and emotional skills, they can be affected by the respondent’s interpretation of the questionnaire item. Self-reported measures are also susceptible to multiple biases: social desirability bias, where students provide answers they think are more socially acceptable; reference group bias, where students compare themselves to the group of persons around them while answering questions, and when the reference group itself can differ from one student to another, and from school to school; response style bias, where students from different cultures provide different patterns of responses, such as providing more extreme or more modest responses.

SSES acknowledges these potential biases and tries to minimise the effect of these potential biases on the variables and relations between variables presented in this report. For this, the SSES controls for acquiescent response tendencies in students’ social and emotional skills. In 2019, the SSES used anchoring vignettes to examine reference group bias and assessed students’ social and emotional skills via direct (student) and indirect (parent and teacher) assessment (OECD, 2021[1]). The pattern of results was similar for the direct and indirect assessment of social and emotional skills, and unlike adjustments by acquiescence, anchoring vignettes did not generally improve the assessment beyond what was already done. Therefore, only a direct assessment of social and emotional skills via students’ self-reports was used in 2023, and responses were controlled for acquiescence.

Acquiescent response style

Acquiescence refers to tendencies among respondents to provide their agreement or disagreement to different positively and negatively worded statements irrespective of the content, wording and direction. Such response styles may result in biased measures, and the calculation of acquiescence response sets (ARS) has been suggested as a way of modelling such response tendencies for Likert-type items (Primi et al., 2020[2]). One way to control for acquiescence is using a balanced set of items per scale in which positively and negatively worded items are paired within scales. One of the design features of the SSES assessment was to have both positively and negatively worded items within each item set measuring a particular construct scale. However, the items were not evenly balanced. To derive an acquiescence response set, 34 pairs of items across all scales were selected. To control for acquiescent response styles, multiple group confirmatory factor analysis (MGCFA) models were estimated using ARS as control variables as part of multiple indicator multiple cause (MIMIC) models, which generally showed improved model fit and higher levels of measurement invariance.

Trend scales

To refine the social and emotional skills assessment, some items were replaced with new items between 2019 and 2023. Most of the analysis on social and emotional skills in this report used the “main scales” constructed from all items of the SSES 2023 skills assessment. “Trend scales” are used for all analyses that include participating countries and subnational entities (hereafter, “sites”) in SSES 2019 and 2023. These scales were constructed using only items in common between the two years (“trend items”) to allow results to be compared between sites in SSES 2019 and 2023. Trend scales were only used for the analysis where trend items or indices relating to the school or home environment were available – for example, the index of students’ sense of belonging was measured in 2019 and 2023. Achievement motivation was measured in SSES 2019 as a “compound skill” created from items used to evaluate other skills (OECD, 2021[1]). In SSES 2023, achievement motivation is measured using a new set of dedicated items, and no trend scale is available for this skill.

Cross-site comparability of social-emotional assessment scales

The SSES 2019 Technical Report (OECD, 2021[3]) and the SSES 2023 Technical Report (OECD, forthcoming[4]) explain the scaling procedures and the construct validation of all social-emotional assessment scales in detail. This section summarises the analyses carried out to validate the cross-site comparability of the social and emotional skill assessment scales used in this report. The internal consistency of scaled indices, factor analysis to assess construct dimensionality, and the invariance of item parameters are the three approaches SSES 2019 and 2023 used to examine the comparability of scaled indices across sites. Based on these three approaches, all indices examined in this report meet the reporting criteria. Internal consistency refers to the extent to which the items that make up an index are interrelated. Cronbach’s Alpha was used to check the internal consistency of each scale within the sites and to compare it among sites. The coefficient of Cronbach’s Alpha ranges from 0 to 1, with higher values indicating higher internal consistency. Similar and high values across sites are an indication of reliable measurement across sites. Commonly accepted cut-off values are 0.9 for excellent, 0.8 for good, and 0.7 for acceptable internal consistency. The reliability for each of the social and emotional skills assessment scales was higher than 0.7 in each site and for each scale (concretely in 178 of the 225), with the following exceptions in SSES 2023:

• Achievement motivation: Delhi (India) (0.65)

• Assertiveness: Bogotá (Colombia) (0.66), Delhi (India) (0.42), Kudus (Indonesia) (0.60), Sobral (Brazil) (0.67)

• Creativity: Delhi (India) (0.58)

• Curiosity: Delhi (India) (0.69), Kudus (Indonesia) (0.66)

• Emotional control: Delhi (India) (0.63)

• Empathy: Bogotá (Colombia) (0.68), Delhi (India) (0.53), Kudus (Indonesia) (0.60), Sobral (Brazil) (0.65)

• Energy: Bulgaria (0.67), Bogotá (Colombia) (0.68), Delhi (India) (0.40), Kudus (Indonesia) (0.64), Sobral (Brazil) (0.60), Ukraine (0.67)

• Optimism: Delhi (India) (0.53), Kudus (Indonesia) (0.68)

• Persistence: Delhi (India) (0.60), Kudus (Indonesia) (0.69)

• Responsibility: Delhi (India) (0.59), Sobral (Brazil) (0.68)

• Self-control: Bulgaria (0.61), Bogotá (Colombia) (0.66), Delhi (India) (0.51), Kudus (Indonesia) (0.47), Mexico (0.69), Peru (0.69), Sobral (Brazil) (0.62), Ukraine (0.64)

• Sociability: Delhi (India) (0.66), Kudus (Indonesia) (0.68)

• Stress resistance: Bogotá (Colombia) (0.69), Delhi (India) (0.42), Kudus (Indonesia) (0.51), Sobral (Brazil) (0.65)

• Tolerance: Bogotá (Colombia) (0.64), Delhi (India) (0.56), Kudus (Indonesia) (0.61), Mexico (0.69), Sobral (Brazil) (0.66), Ukraine (0.64)

• Trust: Bulgaria (0.69), Delhi (India) (0.50).

Exceptions for SSES 2019 are noted in the SSES 2019 Technical Report (OECD, 2021[3]).

The analyses of the SSES data involved a series of iterative modelling and analysis steps. These steps included the application of confirmatory factor analysis (CFA) to evaluate constructs and an MGCFA to review measurement invariance across groups (gender, age cohorts and sites). In assessing measurement equivalence for SSES trend scales, comparisons were made between cycle groups (Round 1 and Round 2). In addition, MGCFA models were estimated using ARS as control variables as part of MIMIC models, which generally showed improved model fit and higher levels of measurement invariance.

All items had a Likert-type format with five categories and included both positively and negatively worded statements. The five categories were “strongly disagree”, “disagree”, “neither agree nor disagree”, “agree” and “strongly agree”. Each item was scored from 0 to 4 for items with positively worded statements and reverse scored for the negatively worded ones.

The SSES student surveys in Delhi (India), Helsinki (Finland), Mexico and Ukraine were conducted in the third quarter of 2023 and were therefore not included in the data for estimating the scaling parameters for the student direct assessment.

In testing for measurement invariance, three different models were specified and compared:

• Configural invariance is the least constrained model. This model assumes that the items measuring the underlying latent construct are equivalent across all reference groups (e.g. sites). If the strength of the associations between the groups is the same, then the latent construct is assumed to have the same meaning for all groups (i.e. the structure of the construct is the same). Configural invariance would make it possible to examine whether the overall factor structure stipulated by the measures fits well for all groups in the sample. However, for scales reaching configural invariance, neither scores nor their associations can be directly compared across groups.

• Metric invariance is achieved if the structure of the construct is the same across groups (i.e. configural invariance is achieved) and the strength of the association between the construct and items (factor loadings) is the same across groups. Metric invariance would allow for comparisons of within-group associations among variables across groups (e.g. correlations or linear regression) but not for the comparison of scale mean scores.

• Scalar level invariance is achieved when metric invariance has been achieved, and the intercepts/thresholds for all items across groups are equivalent. When scalar invariance is achieved, it is assumed that differences in scale means across groups are free of any cross-group bias. At this level of measurement equivalence, scale scores can be directly compared across groups.

Results of the MGCFA are presented in Table A.1. Finally, the item response theory (IRT) generalised partial credit model (GPCM) was used to scale items and generate scores.

Table A.1. Levels of measurement invariance for social and emotional skills scales

	Age cohorts	Gender	Sites
Curiosity	Metric	Metric	Metric
Tolerance	Metric	Scalar	Metric
Creativity	Scalar	Scalar	Metric
Responsibility	Metric	Scalar	Metric
Self-control	Metric	Scalar	Metric
Persistence	Metric	Scalar	Metric
Achievement motivation	Metric	Scalar	Metric
Sociability	Metric	Scalar	Metric
Assertiveness	Scalar	Scalar	Metric
Energy	Metric	Metric	Metric
Empathy	Metric	Metric	Metric
Trust	Metric	Scalar	Metric
Stress resistance	Scalar	Metric	Metric
Optimism	Scalar	Scalar	Metric
Emotional control	Scalar	Metric	Metric

Construction of the questionnaire indices

Several SSES measures reflect indices that summarise responses from students, principals or teachers to a series of related questions. There are three different types of indices:

• Simple indices are constructed using an arithmetic transformation or recoding of one or more items in exactly the same way across assessments. Here, item responses are used to calculate meaningful variables, such as the recoding of the four-digit International Standard Classification of Occupations (ISCO) 2008 codes into “highest parents’ socio-economic index (HISEI)”.

• Complex composite indices are based on a combination of two or more indices. The index of economic, social and cultural status (ESCS) is a composite score derived from three indicators related to family background.

• Scale indices are constructed by combining multiple items intended to measure an underlying latent construct. The indices were scaled using GPCM unless otherwise indicated.

Simple indices

Student age

Student age (Age_Std) was calculated as the age in months at the time of the questionnaire administration. It is the difference between the date the student questionnaire was administered and the student’s date of birth. Student age was derived from information about the student’s date of birth and the actual start date of the administration of the student questionnaire. Generally, data from the student tracking forms (STF) were given priority over information provided by students when responding to the questionnaire.

Gender

A student gender variable (Gender_Std) was computed by using valid codes (i.e. not missing) from the student questionnaire variable STQM00401 STF (1 for girls, 2 for boys and 3 for other). When Gender_Std had a missing value, STF_Gender from the STF was used.

Grades

SSES collected information on school grades in three subjects: reading (Sgrade_Read_Lang), mathematics (Sgrade_Math) and the arts (Sgrade_Arts). As different sites used different grading systems, all grades were transformed on a scale from 1 to 50.

Immigrant background

Information on the country of birth of students and their parents was also collected. Included in the database are three country-specific variables related to the country of birth of the student and their mother and father (STQM11901, STQM11902 and STQM11903). The variables indicate whether the student, mother and father were born in the country of assessment or elsewhere. The index on immigrant background (IMMBACK) is calculated from these variables. It has the following categories: 1) native students (students who are born in the country of assessment and students who had at least one parent born in the country of assessment); and 2) non-native students (students who are born abroad and/or parents who are born abroad). Students with missing responses for either the student or for both parents were given missing values for this variable.

Parents’ level of education

In the student questionnaire, respondents were asked about the highest level of education of each of their parents with questions using nationally appropriate terms according to the International Standard Classification of Education (ISCED) (UNESCO, 2017[5]). Respondents were asked to select from ten levels ranging from no completion of ISCED Level 1 (primary education) through to completion of ISCED Level 8 (doctoral or equivalent level). An index, HISCED, was derived by taking either parent's highest level of education from the student questionnaire. If data were only available for one parent, then that is used as the highest level.

Parents’ highest occupational status

Occupational data were collected using open-ended questions in the student questionnaires (STQM011-STQM014). The responses were coded to four-digit ISCO codes and then mapped to the International Socio-economic Index of Occupational Status (ISEI) (Ganzeboom and Treiman, 2003[6]). The highest occupational status of parents (HISEI) corresponds to the higher ISEI score among parents or to the only available parent’s ISEI score. A higher ISEI score indicates higher levels of occupational status.

Shared mindset on the impact of social and emotional skills

A school was categorised as having a shared mindset among school staff about the impact of social and emotional skills if all teachers and the school's principal (strongly) agreed on a particular positive outcome of social and emotional skills. This categorisation was done separately for each item of the scales on the impact of social and emotional skills (TCQM10601/PRQM11801-TCQM10610/PRQM11810). Caution is warranted when interpreting results as the number of schools and teachers from each school participating in the SSES 2023 varied across participating sites (see Table T2_avgteachsch_mean).

Shared mindset on the responsibility for developing social and emotional skills

A school was categorised as having a shared mindset among school staff about the responsibility for developing social and emotional skills if all teachers and the principal of the school responded “yes” to the responsibility of a particular group (e.g. “school teachers in general”, “parents or guardians”). This categorisation was done separately for each item of the scales on the responsibility for developing social and emotional skills (PRQM11901/TCQM10701-PRQM11907/TCQM10707). Caution is warranted when interpreting results as the number of schools and teachers from each school participating in the SSES 2023 varied across participating sites (see Table T2_avgteachsch_mean).

Scaled indices

Bullying perpetration

To measure students' involvement in bullying as a perpetrator (ST_BULLYPERP), students were asked how often (“Never or almost never”, “A few times a year”, “A few times a month”, “Once a week or more”) they had done the following in the past 12 months: “I made fun of other students”, “I spread nasty rumours about other students”, “I left other students out of things on purpose”, “I threatened another student”, “I took away or destroyed things that belonged to other students”, and “I hit or pushed other students around” (Items STQM15801-STQM15806). Students received higher scores on this scale if they indicated a higher frequency of occurrence of these situations.

Bullying victimisation

To measure students' involvement in bullying as a victim (ST_BULLY), students were asked how often (“Never or almost never”, “A few times a year”, “A few times a month”, “Once a week or more”) they experienced the following in the past 12 months: “Other students made fun of me”, “Other students spread nasty rumours about me”, “Other students left me out of things on purpose”, “I was threatened by other students”, “Other students took away or destroyed things that belonged to me”, and “I got hit or pushed around by other students” (Items STQM15701-STQM15706). Students received higher scores on this scale if they indicated a higher frequency of occurrence of these situations.

Emotions at school

Students rated the frequency in which they felt certain emotions at school in the previous year (“Never or almost never”, “Less than half of the time”, “About half of the time”, “More than half of the time”, “All or almost all of the time”). Four items referring to positive emotions (e.g. “confident”, “motivated”, “happy”) from STQM155 were used to build the index of positive emotions (ST_ POSEMOT). Three items referring to negative emotions (e.g. “anxious”, “upset”, or “angry”) were used to build the index of negative emotions at school (ST_ NEGEMOT). Higher scale scores correspond to a high frequency of positive and negative emotions at school.

Engagement in extra-curricular activities

To measure students’ engagement in extra-curricular activities (ST_EXTRACUR), students were asked how often they participate in several extra-curricular activities (e.g. “School play or school musical”, “Volunteering or service activities”). The responses to the 11 items (STQM15901-STQM15911) with 4‑point response options (“I don’t”, “Once a month”, “Once a week”, “More than once a week”) were scaled to the index of ST_EXTRACUR. Students indicating a higher frequency of extra-curricular activities obtained higher scores on the scale.

Gender stereotypes and biases

Students rated their agreement with the following gender stereotypes and biases: “Boys are more ambitious than girls”, “Men make better political leaders than women do”, “Boys are better at technology than girls”, “Boys are more aggressive than girls”, “Women are better prepared to care for children than men”, “Girls are more empathetic than boys”, “Women are better in visual arts than men”, and “Girls are more sensitive than boys” (Items STQM14601-STQM14608). Each of the items included in this scale had five response options (“Strongly disagree”, “Disagree”, “Neither agree nor disagree”, “Agree”, or “Strongly agree”). Higher scores on the gender stereotypes and biases scale (ST_GENBIAS) indicate greater agreement with gender stereotypes and biases, while lower scores indicate greater disagreement.

Home gender roles

To measure how responsibilities for domestic tasks in students' homes are distributed, students were asked whether male relatives, female relatives or both are mainly responsible for the following domestic tasks: preparing meals, looking after children, cleaning the house and caring for sick family members (Items STQM14901, STQM14903, STQM14905 and STQM14907). Higher scores on the home gender roles scale (ST_HOMGEN) indicate that female relatives are mainly responsible for these tasks, while lower scores indicate that these are shared between male and female relatives.

Impact of social and emotional skills

Teachers and principals rated their agreement to the question if social and emotional skills have an impact on different positive outcomes (e.g. “Decreased absenteeism and truancy”, “Increased student participation and engagement in school”, “Improved well-being of students). The TC_SESIMPAC and PR_SESIMPACT scales consisted of the same ten items answered by principals and teachers (TCQM10601-TCQM10610 and PRQM11801-PRQM11810). Each of the ten items included in this scale had five response options (“Strongly disagree”, “Disagree”, “Neither agree nor disagree”, “Agree”, or “Strongly agree”). All items in the scale were positively worded. Teachers and principals indicating a greater impact of social and emotional skills received higher scores on the scales.

Inclusion of social and emotional learning in teacher training

Teachers were asked if any topics listed below were included in their teacher education, in-service training programme, training for their other professional qualifications or professional development activities. Teachers could choose between the following response options: “Included in my teacher education or in-service training programme or training for other professional qualifications”, “Included in my professional development activities during the last 12 months”, or “Not included”. Two scales were built from teachers’ responses to the eight items (TCQM10801-TCQM10808). Teachers receiving higher scores on the scale of inclusion of social and emotional learning in teacher training (TC_SESITE1) had received more training on the eight items relating to social and emotional teaching, whether they were included in training in the last 12 months or in previous trainings. Teachers receiving higher scores on the scale of inclusion of social and emotional learning in recent teacher training (TC_SESITE2) had received more training on the eight items relating to social and emotional teaching in training in the last 12 months.

Opportunities for developing student´s social and emotional skills

Teachers were asked how often they promote different skills or characteristics in their target class (“Never”, “In some lessons”, “In most lessons”, or “In every lesson”). Items referred to skills of each of the domains assessed in the SSES 2023 framework, including “Being assertive, sociable and enthusiastic around other people” (engaging with others) or “Being persistent, responsible and self-disciplined” (task performance). All seven items (TCQM11601-TCQM11607) in the scale of teachers’ opportunity to develop social and emotional skills (TC_OPPODEV) were positively worded. Higher scale scores correspond to providing more opportunities for developing social and emotional skills in class.

Peer-to-peer relationships

Question STQM151 collected the students’ ratings about the extent to which they agreed or disagreed with the statements about their relationships at school (e.g. “My classmates are respectful towards me”, “If I walked into my classes upset, my classmates would be concerned about me”). Six items were used to scale the students' responses into the index of ST_STUCLASS. Each of the six items included in this scale had four response options (“Strongly disagree”, “Disagree”, “Agree”, or “Strongly agree”). One item in the scale was negatively worded, and it was reverse-coded prior to scaling. Higher scale scores correspond to better student-classmate relationships.

Promotion of social and emotional learning

Principals were asked in which ways the development of students' social and emotional skills were promoted in their school (e.g. “The development of these skills is one of the objectives included in the school educational plan”, “Implementation of the development of these skills is part of the school's disciplinary rules”). Eight items (PRQM1201-PRQM1208) were included in this scale (PR_SESPRO), with two response options (“No” or “Yes”). Principals indicating a high level of promotion of social and emotional learning at their school obtained higher scores on the scale.

Sense of belonging

The ST_BELONG scale consisted of six items from the STQM154 question, three of which were positively worded (e.g. “I make friends easily at school”) while the other three were negatively worded (e.g. “I feel like an outsider [or left out of things] at school”). Each of the six items included in this scale had four response options (“Strongly disagree”, “Disagree”, “Agree”, or “Strongly agree”). Students indicating a greater sense of belonging obtained higher scores on the scale.

Skills and online and remote teaching

Teachers who indicated that they had taught students online or remotely during the previous year (TCQM12001) were asked about the extent to which the development of different skills or characteristics were hindered or fostered due to online or remote teaching in their opinion (“Hindered a lot”, “Hindered a bit”, “No effect”, “Fostered a bit” or “Fostered a lot”). The six items (TCQM12301-TCQM12307) referred to each of the skills assessed in the SSES 2023 assessment, e.g. “Controlling one's emotions, staying optimistic, and coping with stress” (emotional regulation), “Setting high standards for oneself and working hard to meet them” (achievement motivation). All items in the scale were positively worded. Higher values on the index mean that the teachers had a stronger belief that online and remote teaching can foster rather than hinder social and emotional learning.

Student-teacher relationships

A multi-dimensional CFA model was constructed using items related to student perceptions of their teachers. For the first dimension, ST_RELTEACH, students rated the agreement or disagreement with some statements about their perception of the relationship between students and teachers. Some of these statements were “The teachers at my school are respectful towards me” or “The teachers at my school are friendly towards me”. Six items were used to scale the student's responses into the index. Each of the six items included in this scale had four response options (“Strongly disagree”, “Disagree”, “Agree”, or “Strongly agree”). One item in the scale was negatively worded, and it was reverse-coded prior to scaling. Higher scale scores correspond to better relationships with teachers as perceived by the students.

Teacher coping strategies

Question TCQM128 asks teachers to rate the extent to which they used some strategies to cope with work-related stress (e.g. “Maintain a sense of humour”, “Practice good human relations skills”). Based on nine items, the index TC_COPING was constructed with the responses to a four-point scale (“Not at all”, “To a small extent”, “To a moderate extent”, or “To a large extent”). Higher scale scores correspond to a greater number of strategies to cope with work stress. All items in the scale were positively worded.

Teacher efficacy

Teachers were asked to indicate the extent to which they could do different tasks in teaching the target class (“Not at all”, “To a small extent”, “To a moderate extent”, or “To a large extent”). Items of the efficacy scale included tasks relating to the perceived efficacy of engaging students (e.g. “Help students value learning”) and in managing the classroom (e.g. “Get students to follow classroom rules”). The scale also included tasks related to socio-emotional teaching (e.g. “Understand students’ feelings and emotions”). All items in the scale were positively worded.

Three scales were built from the 11 items (TCQM11801-TCQM118011). While the teachers’ overall efficacy (TC_SELFEFF) index was derived from all items, two other scales were built from four items each to measure teachers’ efficacy in student management (TC_STSELFEFF) and teacher efficacy in classroom management (TC_CLSELFEFF). Higher scale scores correspond to greater perceived levels of overall teaching efficacy, as well as efficacy in student engagement and classroom management.

Teacher feedback

To measure teacher feedback (ST_FEEDBACK), students were asked how often the following things happen in their school (STQM15201-STQM15203): “The teacher gives me feedback on my strengths”, “The teacher tells me in which areas I can still improve” and “The teacher tells me how I can improve my performance”. Students were given the following response options according to a four-point frequency scale: “Never or almost never”, “Some lessons”, “Many lessons”, and “Every lesson or almost every lesson”. Three positively worded items contributed to the index. Higher values on the index mean that students reported more teacher feedback.

Scaling related to the index of socio-economic status

A measure of parental socio-economic status (SES) was derived for each site, based on three indices: the highest level of parental occupation (HISEI), the highest level of parental education (PARED) and household possessions (HOMEPOS). The household possessions index (HOMEPOS) consists of student-reported possessions at home, resources available at home and the number of books at home. HOMEPOS is a summary index of all household and possession items (STQM130, STQM131, STQM134 and STQM134). The computation of missing values for respondents with missing data for only one index variable was imputed with predicted values plus a random component based on a regression of the other two index variables within sites. If there were missing data on more than one index variable, the index was not computed for that student, and a missing value was assigned. After imputation, all three components were standardised for SSES sites with a mean of zero and a standard deviation of one. Then, the ESCS was constructed as the arithmetic mean of the three indicators after their imputation and standardisation.

Single items

In addition to the indices listed above, the following single items were used in this report:

• Importance of social and emotional skills and cognitive skills, as perceived by parents (PAQM12101-PAQM12107)

• Responsibility for developing social and emotional skills, as perceived by teachers, principals and parents (PRQM11901/TCQM10701/PAQM12201-PRQM11907/TCQM10707/PAQM12207)

• Social and emotional skills education in school (PRQM12101 and PRQM12104)

• Teachers’ experience with online/remote teaching (TCQM12001)

• Teachers’ sources of information (TCQM11701-TCQM11707)

• Teachers’ working hours (TCQM10401-TCQM10409).

Cross-site comparability of background scaled indices

While the SSES 2019 Technical Report (OECD, 2021[3]) and the SSES 2023 Technical Report (OECD, forthcoming[4]) explain in detail the scaling procedures and the construct validation of all contextual questionnaire data, this section presents a summary of the analyses carried out to validate the cross-site comparability of the main scaled indices used in this report. The internal consistency of scaled indices, factor analysis to assess construct dimensionality, and the invariance of item parameters are the three approaches SSES used to examine the comparability of scaled indices across sites. Based on these three approaches, all indices examined in this report met the reporting criteria.

Internal consistency refers to the extent to which the items that make up an index are interrelated. Cronbach’s Alpha was used to check the internal consistency of each scale within the sites and to compare it among sites. The coefficient of Cronbach’s Alpha ranges from 0 to 1, with higher values indicating higher internal consistency.

Similar and high values across sites are an indication of reliable measurement across sites. Commonly accepted cut-off values are 0.9 for excellent, 0.8 for good, and 0.7 for acceptable internal consistency.

For the samples of 15-year-old students, their teachers and principals, the average reliability for each of the scale indices described above was higher than 0.70, and by site only in the following exceptions:

• Bullying perpetration: Gunma (Japan) (0.67)

• Engagement in extra-curricular activities: Gunma (Japan) (0.38), Spain (0.67), Ukraine (0.70)

• Emotions at school (positive): Delhi (India) (0.65)

• Emotions at school (negative): Gunma (Japan) (0.65), Peru (0.60), Spain (0.66), Turin and Emilia-Romagna (Italy) (0.69)

• Home gender roles: Delhi (India) (0.65), Dubai (United Arab Emirates) (0.69), Kudus (Indonesia) (0.63), Turin and Emilia-Romagna (Italy) (0.66), Ukraine (0.64)

• Peer-to-peer relationships: Delhi (India) (0.65)

• Sense of belonging: Bulgaria (0.69), Delhi (India) (0.56)

• Teacher coping strategies: Bulgaria (0.64), Chile (0.67), Gunma (Japan) (0.69), Turin and Emilia-Romagna (Italy) (0.66)

• Teacher efficacy in student engagement: Mexico (0.67), Peru (0.55)

• Teacher efficacy in classroom management: Peru (0.69)

• Teacher feedback: Delhi (India) (0.62).

For the samples of 10-year-old students, their teachers and principals, the average reliability for each of the scale indices described above was higher than 0.70, and by site only in the following exceptions:

• Emotions at school (negative): Bogotá (Colombia) (0.69), Sobral (Brazil) (0.67)

• Gender stereotypes and biases: Sobral (Brazil) (0.69)

• Home gender roles: Kudus (Indonesia) (0.68), Ukraine (0.69)

• Peer-to-peer relationships: Kudus (Indonesia) (0.69), Sobral (Brazil) (0.68)

• Sense of belonging: Bogotá (Colombia) (0.62), Kudus (Indonesia) (0.54), Sobral (Brazil) (0.62), Ukraine (0.48)

• Teacher coping strategies: Bogotá (Colombia) (0.69), Helsinki (Finland) (0.68)

• Teacher efficacy in student engagement: Bogotá (Colombia) (0.67)

• Teacher feedback: Sobral (Brazil) (0.62), Ukraine (0.69).

Exceptions for SSES 2019 are noted in the SSES 2019 Technical Report (OECD, 2021[3]).

The analyses of the background scale indices also involved a series of iterative modelling and analysis steps. Items from all scales were initially evaluated through an exploratory factor analysis (EFA). A CFA was then carried out on the scales, with only acceptable items from the EFA, to assess the constructs. Generally, maximum likelihood estimation and covariance matrices are not appropriate for analyses of categorical questionnaire items because the approach treats items as if they are continuous. Therefore, the SSES analysis relied on robust weighted least squares estimation (WLSMV) models (Muthén, du Toit and Spisic, 1997[7]; Flora and Curran, 2004[8]) to estimate the CFA.

For ease of interpretation, all reversely worded items were recoded, so the highest value for each item represents a higher attribute.

The SSES student surveys in Delhi (India), Helsinki (Finland), Mexico and Ukraine were conducted in the third quarter of 2023 and were therefore not included in the data for estimating the scaling parameters for the student background questionnaire. Furthermore, a MGCFA was used to test measurement invariance. For the student questionnaire, the MGCFA was evaluated for the following groups: gender, age cohorts and sites.

In testing for measurement invariance, three different models were specified and compared: configural invariance, metric invariance and scalar invariance (see Cross-site comparability of social-emotional assessment scales

for a description).

Results of the MGCFA are presented in Table A.2. Finally, items were scaled using the GPCM. More detailed information on the measurement invariance of the scales in the background questionnaires can be found in Chapter 14 of the SSES 2019 Technical Report (OECD, 2021[3]) and in the SSES 2023 Technical Report (OECD, forthcoming[4]).

Table A.2. Levels of measurement invariance: Scales in the student, teacher and principal questionnaires

	Age cohort	Gender	Sites
Bullying perpetration	Scalar	Scalar	Scalar
Bullying victimisation	Scalar	Scalar	Scalar
Emotions at school	Scalar	Scalar	Metric
Engagement in extra-curricular activities	Scalar	Scalar	Metric
Home gender roles	Metric	Scalar	Scalar
Gender stereotypes and biases	Not invariant	Not invariant	Not invariant
Impact of social and emotional skills, as reported by teachers	Not invariant	-	Not invariant
Impact of social and emotional skills, as reported by principals	Scalar	-	Not invariant
Inclusion in teacher training	Scalar	-	Not invariant
Inclusion in recent teacher training	Scalar	-	Not invariant
Opportunities for developing social and emotional skills	Scalar	-	Scalar
Peer-to-peer relationships	Scalar	Scalar	Metric
Sense of belonging	Not invariant	Not invariant	Not invariant
Student-teacher relationships and teacher feedback	Scalar	Scalar	Metric
Teacher coping	Not invariant	-	Not invariant
Teacher efficacy (overall)	Scalar	-	Not invariant
Teacher efficacy in classroom management	Scalar	-	Metric
Teacher efficacy in student engagement	Scalar	-	Metric
Skills and on line teaching	Scalar	-	Not invariant

Note: In instances where there were only three items for the scale (i.e. teacher feedback and negative emotions), the models indicated a perfect fit and could not be evaluated due to the limited number of degrees of freedom. Therefore, an MGCFA was evaluated on a combination of these scales with related scales using multi-dimensional models.

Description of each site, their target population and cautionary notes

The group of students the survey results should represent, the target population, differed slightly across sites. A random sample of students was surveyed from the target population. Table A.3 provides a list of each site that participated in SSES 2023, their target population definition and any cautionary notes that should be considered when interpreting their data. The target population varies between sites, and these differences should be considered when interpreting analyses.

Table A.3. SSES 2023 site descriptions, target population definitions and cautionary notes

Site description	Target population definition	Cautionary notes
Bulgaria is an OECD accession candidate country located in Europe.	15-year-old students in public and private schools. There were 57 373 SSES-eligible students in 1 091 schools.	None.
Chile is an OECD member country located in South America.	15-year-old students in public and private schools. There were 229 026 SSES-eligible students in 5 753 schools.	None.
Mexico is an OECD member country located in North America.	15-year-old students in public schools. There were 633 576 SSES-eligible students in 16 284 schools. However, these estimates are not consistent with those from PISA 2022.	The data do not fully represent the target population and present major deviations from several technical standards. For this reason, data for Mexico are excluded from the international average and reported separately.
Peru is an OECD accession candidate country located in South America.	15-year-old students in public and private schools. There were 543 882 SSES-eligible students in 16 977 schools.	None.
Spain is an OECD member country located in Europe.	15-year-old students in public and private schools. There were 487 622 SSES-eligible students in 7 876 schools.	None.
Ukraine is a prospective OECD member country located in Europe.	10-year-old and 15-year-old students in public and private schools from 19 of 27 Ukrainian regions. There were 415 927 SSES younger cohort eligible students enrolled in 11 963 schools and 289 953 SSES older cohort eligible students in 11 038 schools.	The Russian Federation’s war of aggression against Ukraine meant that a minority of Ukrainian regions where it was not safe to conduct the survey are not covered. Data are representative of 19 of 27 Ukrainian regions. For this reason, data for Ukraine are labelled “Ukraine (19 of 27 regions)”. In addition, the consequences of the war also had an impact on students’ participation rates. Data for 10-year-old students should be interpreted with some caution, as student response rates were lower than expected (72%). Data for 15-year-old students should be interpreted with caution, as student response rates were much lower than expected (57%).
Bogotá is the capital of Colombia, an OECD member country.	10-year-old and 15-year-old students in public and private schools. There were 87 501 SSES younger cohort eligible students in 1 679 schools and 91 501 SSES older cohort eligible students enrolled in 1 146 schools.	None.
Delhi is the capital of India, an OECD Key Partner country.	15-year-olds in public schools managed by the Directorate of Education in the Government of the National Capital Territory of Delhi. There were 244 856 SSES-eligible students in 964 schools.	Data should be interpreted with some caution as student response rates were lower than expected (72%).
Dubai is a city in the United Arab Emirates, a non-member economy.	15-year-old students in private schools. There were 18 100 SSES-eligible students in 170 schools.	None.
Emilia-Romagna is a region located in northern Italy, an OECD member country.	15-year-old students in public and private schools. There were 22 594 SSES-eligible students in 172 schools.	None.
Gunma is a prefecture located in central Japan, an OECD member country.	15-year-old students in public and private schools. There were 14 757 SSES-eligible students in 79 schools.	None.
Helsinki is the capital of Finland, an OECD member country.	10-year-old and 15-year-old students in public schools. There were 5 883 SSES younger cohort eligible students in 96 public schools and 4 090 SSES older cohort eligible students in 68 schools.	Data for 15-year-old students should be interpreted with some caution, as student response rates were lower than expected (70%).
Jinan is the capital city of Shandong province in eastern People’s Republic of China, an OECD Key Partner country.	10-year-old and 15-year-old students in public and private schools. There were 105 510 SSES younger cohort eligible students in 708 schools and 71 167 SSES older cohort eligible students in 338 schools.	None.
Kudus is a city in the Central Java province of Indonesia, an OECD Key Partner country.	10-year-old and 15-year-old students in public and private schools. There were 13 716 SSES younger cohort eligible students in 570 schools and 10 470 SSES older cohort eligible students in 207 schools.	Data for both 10-year-olds and 15-year-olds should be interpreted with some caution as the samples drawn may not be fully representative of the target population. The data are estimated to be representative of 9 199 10-year-old students and 4 697 15-year-old students in Kudus.
Sobral is a municipality in the state of Ceará in the northeast region of Brazil, an OECD accession candidate and Key Partner country.	10-year-old and 15-year-old students in public schools. There were 2 339 SSES younger cohort eligible students in 55 schools and 2 586 SSES older cohort eligible students in 34 schools.	None.
Turin is a city located in northern Italy, an OECD member country.	15-year-old students in public and private schools. There were 14 647 SSES-eligible students in 150 schools.	None.

References

[8] Flora, D. and P. Curran (2004), “An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data”, Psychological Methods, Vol. 9/4, pp. 466-491, https://doi.org/10.1037/1082-989x.9.4.466.

[6] Ganzeboom, H. and D. Treiman (2003), “Three Internationally Standardised Measures for Comparative Research on Occupational Status”, in Advances in Cross-National Comparison, Springer US, Boston, MA, https://doi.org/10.1007/978-1-4419-9186-7_9.

[7] Muthén, B., S. du Toit and D. Spisic (1997), Robust Inference Using Weighted Least Squares and Quadratic Estimating Equations in Latent Variable Modelling with Categorial Outcomes, http://www.statmodel.com/bmuthen/articles/Article_075.pdf.

[1] OECD (2021), Beyond Academic Learning: First Results from the Survey of Social and Emotional Skills, OECD Publishing, Paris, https://doi.org/10.1787/92a11084-en.

[3] OECD (2021), Survey on Social and Emotional Skills (SSES) Technical Report 2019, OECD Publishing, Paris, https://www.oecd.org/en/about/programmes/oecd-survey-on-social-and-emotional-skills.html.

[4] OECD (forthcoming), Survey on Social and Emotional Skills (SSES) Technical Report 2023, OECD Publishing, Paris.

[2] Primi, R. et al. (2020), “Classical Perspectives of Controlling Acquiescence with Balanced Scales”, in Springer Proceedings in Mathematics and Statistics, Quantitative Psychology, Springer International Publishing, Cham, https://doi.org/10.1007/978-3-030-43469-4_25.

[5] UNESCO (2017), Data Mapping, http://uis.unesco.org/en/isced-mappings.