Providing Quality Early Childhood Education and Care

Annex B. Technical notes on sampling procedures, response rates and adjudication for TALIS Starting Strong 2018

The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.

Sampling procedures and response rates

The objective of the Starting Strong Teaching and Learning International Survey (TALIS Starting Strong) 2018 was to obtain a representative sample in each participating country of staff and leaders providing early childhood education and care (ECEC) for each level of ECEC in which the country participated. The international sampling plan for TALIS Starting Strong used a stratified two-stage probability sampling design. This means that staff (second stage units, or secondary sampling units) were randomly selected from the list of in-scope staff in each of the randomly selected ECEC centres (first stage units, or primary sampling units). The leader at each centre (i.e. the person with the most responsibility for administrative, managerial and/or pedagogical leadership) was automatically selected for participation as well. For countries with integrated ECEC systems that participated in data collection for both pre-primary education and settings for children under age 3, programmes serving both age groups were split between the two samples so that each programme could be selected for participation in only one level of ECEC. A more detailed description of the survey design and its implementation can be found in the TALIS Starting Strong 2018 Technical Report (OECD, 2019[10]).

Staff in pre-primary education settings (ISCED level 02) are those who, as part of their regular duties in their centre, provide learning opportunities in programmes at ISCED level 02. Staff for children under age 3 are those who, as part of their regular duties in their centre, provide learning opportunities for children in this age group. Staff who provide learning opportunities for both target populations in their centres are included in the TALIS Starting Strong universe. There is no minimum cut-off for how much time staff need to be engaged at either level of ECEC.

The international target population of TALIS Starting Strong restricts the survey to those staff and leaders who work in officially registered settings providing ECEC. Centres exclusively for children with special educational needs are deemed out of scope. Also considered out of scope are: short-term substitute educators (to replace staff on sick leave); nannies and other people involved in informal arrangements; volunteers who occasionally came in to provide a special activity; auxiliary staff (e.g. cleaners, cooking staff) who did not interact regularly in a pedagogical manner with the children; and medical and therapeutic staff (e.g. speech therapists, occupational therapists) whose work was primarily non-pedagogical.

For national reasons, participating countries could choose to restrict the coverage of their national implementation of TALIS Starting Strong. For example, all participating countries decided to exclude home-based settings (within the homes of the respective staff) from the pre-primary education universe, while some countries included these settings in the universe of settings providing services for children under age 3. This report excludes home-based programmes for children under age 3, focusing only on centre-based ECEC to enhance comparability across the two levels of ECEC provision. Participating countries were invited to keep sample exclusions to a minimum by keeping the national survey population to at least 95% of ECEC staff. The national project manager for each country was required to document the reasons for any exclusions.

Sample size requirements

For both levels of ECEC, the same requirements for sample size and precision of estimates were established. To allow for reliable estimation and modelling while permitting some amount of non-response, TALIS Starting Strong 2018 set the minimum number of centres per country for each population of interest (pre-primary education and services for children under age of 3) at 180. Within each centre, the minimum number of staff members selected was eight. If there were fewer than eight staff members in a centre, then all staff members were selected. Participating countries could choose to augment their national sample by selecting more centres, or by selecting more staff within each selected centre, or by increasing both. In some cases, because the average number of staff in the centres was lower than the number expected in the international plan, the number of centres sampled was increased.

Adjudication process

The basic principle that guided the adjudication was to determine, for each participating country and for each level of ECEC, whether the data released to the countries are fit to provide policy-relevant, robust international indicators and analysis on staff and leaders. To establish fitness for use, a number of quality-assurance processes were designed and activated throughout the survey process. Some processes relied on expert advice and opinion, some on qualitative information and learned judgement, some on quantitative information. More detailed information is available in the TALIS Starting Strong 2018 Technical Report (OECD, 2019[10]).

During the adjudication session, each individual dataset (i.e. the combination of participating countries and levels of ECEC) was submitted to the same examination. In addition, both staff participation and leader participation were adjudicated for each combination of participating countries and levels of ECEC.

The issues evaluated concerned the questionnaire’s adaptation to national context, translation and verification, quality of the sampling frame, handling of out-of-scope and refusal units (i.e. staff and/or centres), within-centre sampling, data collection, data cleaning, the reports of quality observers, participation rates and overall compliance with the technical standards (see the TALIS Starting Strong 2018 Technical Report (OECD, 2019[10])). Once each survey process was assessed, a recommended rating was formulated, accounting for the participation rates and for any unresolved issues. The adjudication rules, based on participation rates for leaders and staff, are shown in Table A B.1 and

Table A B.2.

Table A B.1. Adjudication rules for centre or centre leader data in TALIS Starting Strong 2018

Centre participation (returned leader questionnaires)		Risk of centre non-response bias	Rating
Before replacement	After replacement
≥ 75%	≥ 75%		Good
≥ 50% but < 75%	≥ 75%		Fair (A)
	≥ 50% but < 75%	Low	Fair (C)
		High	Poor (D)
< 50%			Insufficient

Table A B.2. Adjudication rules for staff data in TALIS Starting Strong 2018

Centre participation (minimum of 50% staff participation)		Staff participation after centre replacement	Risk of staff non-response bias	Rating
Before replacement	After replacement
≥ 75%	≥ 75%	≥ 75%		Good
		≥ 50% but < 75%		Fair (A)
≥ 50% but < 75%	≥ 75%	≥ 75%		Fair (B)
		≥ 50% but < 75%	Low	Fair (C)
			High	Poor (D)
≥ 50% but < 75%	≥ 50% but < 75%			poor (E)
< 50%	≥ 75%			Poor (F)
< 50%	< 75%			Insufficient

The following is a guide to help data users appreciate the limitations on use or quality:

• Good: The participating country’s data can be used for all reporting and analytical purposes and should be included in international comparisons.

• Fair (A): National and subnational estimates can be produced. Some staff characteristics may suffer from larger standard errors (s.e.), hence the warning “Fair”. No additional warnings to users appear necessary.

• Fair (B, only for staff data adjudication): National and subnational estimates can be produced. Some subnational estimates may be of lower precision (larger s.e.) if sample size is locally low, hence the warning “Fair”. No additional warnings to users appear necessary.

• Fair (C):

  • National and subnational estimates can be produced.

  • Some subnational estimates may be of lower precision (larger s.e.) if sample size is locally low, hence the warning “Fair”. But a note on data quality could appear pointing to the outcome of the non-response bias analysis.

  • Since centre participation is somewhat lower than under (B), comparing subnational estimates should be done with care, as some of those results are based on few centres.

  • Comparing small subnational estimates with similar groups from other participating countries is unlikely to uncover any statistically meaningful differences, as s.e. are likely too large.

• Poor (D):

  • In addition to the warnings issued for the previous category, a note should warn users of indications of non-response biases in some estimates.

  • Comparisons of subnational estimates should be limited to groups with larger sample sizes.

  • At this point, the sample represents between 37% and 56% of the workforce, from a rather small sample of centres.

  • Comparisons with similar groups in other participating countries would not be encouraged.

• Poor (E, only for staff data adjudication): Subnational estimates would not be recommended. There should be a note pointing out the difficulty of obtaining a representative sample of centres.

• Poor (F, only for staff data adjudication): Limitations similar to those of line E, but there should be a note pointing out the difficulty of obtaining at least 50% participation of the selected sample of centres. There are risks of having a non-representative sample of centres.

• Insufficient: Weights should not be calculated for any official tabulations. Hence, data should not be incorporated in international tables, models, averages, etc.1

The participation rates and the adjudication rating per participating country and level of ECEC are presented in Table A B.3 to Table A B.6. These tables display the participation rate estimates that were the most favourable for the adjudication rating. The most favourable estimates could have been weighted or unweighted depending on the characteristics of the country, the staff and leader population and the level of ECEC.

Notes regarding use and interpretation of the data

This section lists issues to be noted regarding the sampling or field operations that should be considered when interpreting the data reported for the following countries:

• Denmark

  • Low response rates during the survey may have resulted in bias in the estimates reported, thus limiting comparability of the data.

  • The data collection period was extended due to a public strike.

• Germany

  • The data collection period was re-opened for a few weeks to encourage additional centres and staff to participate.

  • Non-response bias analysis failed to show that there is not a high risk of centre and staff non-response bias.

  • In two centres serving children under age 3, staff listings were found to be incorrect; these centres were considered as “non-participant”.

• Iceland

  • In Iceland, a shortage of certified ECEC teachers means that staff without this credential (i.e. assistants) may be serving as teachers in some settings. Although data were collected on these role divisions, the distinction was unclear in many instances. Therefore, this overall role distinction is not used in TALIS Starting Strong because it is not meaningful for Iceland.

• Israel

  • For pre-primary education settings (ISCED level 02), ultra-orthodox centres were excluded after the survey because of the low participation rates in this sector. The exclusion rate therefore exceeded 5%.

  • The data collection period was extended for centres serving children under age 3 to accommodate the split system in Israel.

• Korea

  • The data collection period was extended to encourage additional centres and staff to participate.

• Norway

  • Home-based ECEC settings were excluded. The exclusion rate therefore exceeded 5%.

• Turkey

  • Centres under the responsibility of the Ministry of Family, Labour and Social Services were excluded. The exclusion rate therefore exceeded 5%.

Table A B.3. Services for children under age 3: Centre leader participation rates and recommended ratings

Participating country	Number of participating leaders	Estimated size of leader population	Leader participation before replacement (%)	Leader participation after replacement (%)	Recommended rating
Germany	273	48 699	50.7	57.2	Poor
Israel	226	5 042	93.3	97.4	Good
Norway	163	4 916	66.8	92.6	Fair
Denmark**	93	2 852	35.4	47.5	Insufficient

** Low response rates in the survey may result in bias in the estimates reported and limit comparability of the data.

Table A B.4. Services for children under age 3: Staff participation rates and recommended ratings

Participating country	Number of participating centres	Number of participating staff in participating centres	Estimated size of staff population	Centre participation before replacement (%)	Centre participation after replacement (%)	Staff participation in participating centres (%)	Overall staff participation (%)	Recommended rating
Germany	272	1 171	268 310	50.3	57.0	89.7	51.1	Poor
Israel	225	1 113	23 201	90.7	95.3	97.4	92.8	Good
Norway	161	938	35 514	67.1	91.1	86.5	78.8	Fair
Denmark**	87	563	28 303	31.7	43.7	86.1	37.6	Insufficient

** Low response rates in the survey may result in bias in the estimates reported and limit comparability of the data.

Table A B.5. Pre-primary education (ISCED level 02): Centre leader participation rates and recommended ratings

Participating country	Number of participating leaders	Estimated size of leader population	Leader participation before replacement (%)	Leader participation after replacement (%)	Recommended rating
Chile	228	9 426	94.8	98.6	Good
Germany	247	51 942	52.8	69.0	Poor
Iceland	178	236	75.4	75.4	Good
Israel	416	12 175	97.4	98.3	Good
Japan	216	35 577	87.2	98.6	Good
Korea	188	22 722	61.5	76.1	Fair
Norway	152	4 877	64.6	83.8	Fair
Turkey	340	22 380	99.8	99.8	Good
Denmark**	102	3 034	40.9	55.3	Insufficient

** Low response rates in the survey may result in bias in the estimates reported and limit comparability of the data.

Table A B.6. Pre-primary education (ISCED level 02): Staff participation rates and recommended ratings

Participating country	Number of participating centres	Number of participating staff in participating centres	Estimated size of staff population	Centre participation before replacement (%)	Centre participation after replacement (%)	Staff participation in participating centres (%)	Overall staff participation (%)	Recommended rating
Chile	228	1 349	58 060	94.0	98.3	98.8	97.1	Good
Germany	250	1 401	404 202	50.3	57.0	89.7	51.1	Poor
Iceland	204	1 378	3 624	87.0	87.0	84.8	73.8	Good
Israel	409	1 987	43 478	95.7	96.7	97.4	94.1	Good
Japan	216	1 616	307 070	87.2	98.6	99.6	98.2	Good
Korea	182	927	91 586	59.7	75.4	96.8	72.9	Fair
Norway	144	815	39 107	60.7	78.7	83.8	65.9	Fair
Turkey	340	1 605	65 191	99.7	99.7	99.6	99.2	Good
Denmark**	85	544	28 831	32.8	45.7	84.3	38.5	Insufficient

** Low response rates in the survey may result in bias in the estimates reported and limit comparability of the data.

Reference

[10] OECD (2019), TALIS Starting Strong 2018 Technical Report, OECD Publishing, Paris.

Note

← 1.  At their November 2018 meeting in Paris, the TALIS Starting Strong 2018 Technical Advisory Group recommended that data from participating countries that had not reached 50% participation should nonetheless be weighted and displayed in tables.