This chapter looks at how Saudi Arabia’s curriculum and assessment practices influence student learning. In the past, Saudi Arabia has typically relied on textbook content to direct what students learn. Assessment was based upon the amount of factual recall that students could demonstrate, often also based on textbook content. These strategies, however, are not supporting the country’s aims to train a labour force that is highly skilled and able to apply their knowledge to solve problems in novel situations. In response to these circumstances, Saudi Arabia has developed a national curriculum framework that focuses on skills and competences. It has also created a National Assessment Programme to assess student learning against the curriculum and help teachers understand how they can do so as well. This chapter makes suggestions about how the curriculum framework can be further strengthened, and how teacher appraisal methods can be reconfigured to help teachers implement the new curricula. Regarding assessment, this chapter recommends that Saudi Arabia create a national assessment framework to help co-ordinate the numerous assessments that students take, support teachers to adopt more modern assessment practices, and refine national assessments and examinations to reflect and support the educational goals of the country.
Education in Saudi Arabia
Chapter 4. Modernising the curriculum and assessment practices
Abstract
Main features of curriculum and assessment in Saudi Arabia
A curriculum is the strongest expression of the goals of an education system. What students should know, how they should be taught and how their learning should be assessed are articulated through the curriculum’s framework, learning standards and accompanying materials. Through its continuous curriculum reforms, Saudi Arabia has communicated its aims to modernise its education system. The country envisions a departure from a focus on learning content, teacher‑led instruction and assessment through memorisation and movement towards a focus on developing complex skills, student‑centred learning and assessment through applying knowledge in novel situations. Saudi Arabia’s reform efforts, however, have not fully succeeded in transforming the nature of education in the country. Teachers still largely rely on textbooks to guide their instruction and students sit through in upwards of one hundred hours of multiple‑choice testing per year.
Saudi Arabia is about to introduce its first national curriculum framework, which will not only guide the creation of a more modern curriculum, but also provide an opportunity to address several of the systemic issues that have prevented previous reforms from being embedded in all classrooms across the country. This chapter analyses and makes recommendations about how the curriculum framework can more clearly communicate the country’s educational aims, how teacher supervision procedures can be revised to reinforce implementation of the curriculum and how student assessment structures can be transformed to ensure that students have acquired the most important skills they need to be successful in a modern, knowledge‑driven economy.
This chapter looks at Saudi Arabia’s curriculum reforms from the perspective of three dimensions: the intended curriculum, the implemented curriculum and the assessed curriculum. These dimensions are illustrated in Figure 4.1 and are discussed in detail next.
Figure 4.1. The intended, implemented and assessed curriculum
Source: Adapted from Taguma (2017), Preliminary Findings from the OECD Education 2030 project. OECD. https://www.nier.go.jp/kankou_kiyou/146/b08.pdf
Intended curriculum
The intended curriculum refers to what the curriculum expects students to learn and, through these expectations, expresses nationally held beliefs about underlying values, pedagogical methods and assessment aims (OECD, 2013[1]). The intended curriculum increasingly includes objectives for learning attainment, often termed learning standards, which clarify what students are expected to have mastered at the end of a cycle of education. These elements can be further be disaggregated through phases or grades, and specific learning areas.
Saudi Arabia’s national curriculum remains strongly textbook centred, but has been updated continuously to integrate modern learning concepts
The curriculum in Saudi Arabia is strongly textbook centred. Until recently, other references that might guide how teachers and schools organise learning, such as learning objectives or performance standards, have been lacking. As a result, the central resource for determining what students learn have been the content and outcomes contained in national textbooks. In interviews with teachers, supervisors and policy makers, when the OECD review team asked what was understood by “the curriculum”, they referenced the contents of their textbooks.
Over the past two decades, the national textbooks provided to schools have been updated continuously to integrate more modern concepts and student‑centred approaches. In each iteration, the goal has been to move education away from the strict memorisation of facts and towards the development of higher‑order thinking skills and applied knowledge (World Bank and ETEC, 2016[2]). Content and objectives have also been more strongly aligned with international standards, notably through benchmarking with TIMSS. This process of curriculum renewal has been constant and affected all subjects, in particular the sciences and mathematics.
Curriculum development is centralised and split across entities
Historically, the development of textbooks and other curriculum materials (teacher resources, student workbooks, sample tests and quizzes) has been led by the MoE. In 2007, Tatweer for Education (Tatweer) was established as an executive arm of the MoE and assumed responsibility for producing most textbooks and other resources. In the past few years, Tatweer has invested significantly in the development of digital materials and created a national education portal (iEN) to host materials that educators can access online (see Main policy initiatives underway).
Since 2013, the Education and Training Evaluation Commission (ETEC), a standards setting and assessment organisation, has assumed some of the main responsibilities for creating overarching curriculum documentation and guiding the development of curriculum materials. Notably, ETEC has developed the first national curriculum framework and is in the processes of developing subject specifications, including performance standards that align with the framework (see Main policy initiatives underway). At the time of writing this chapter, the national framework was not yet been approved by MoE.
Tatweer carries out regular revision of curriculum materials using a ‘corrective constructive approach,’ which includes correcting mistakes or misconceptions and updating information and statistics. Pursuant to this approach, Tatweer has already begun to review existing resources to see what adaptations and new materials will be required in accordance with the new framework.
While focus groups of teachers and supervisors have been consulted on these developments, overall it appears that stakeholder consultation was relatively limited (Alnefaie, 2016[3]). Similarly, it does not seem that there are plans to pilot the curriculum framework in select schools before it is fully introduced. In the past, changes to the curriculum and accompanying materials have been introduced rapidly and across all grades simultaneously. During these review cycles, channels for collecting teacher feedback during the review process also appear to have been limited.
Important efforts have also been made to review how learning time is used
There are 160 days of instruction per year in Saudi Arabia, compared to 185 across OECD countries (Kingdom of Saudi Arabia, 2018[4]). The shorter school years, plus expectations that schools follow strict curricular time tables, give schools little freedom to adapt the curriculum and, until recently, gave students very limited curricular choice. A number of recent reforms intend to provide more flexibility and time for student‑led activities. Notably, the structure of secondary schooling is being adapted to a credit system that allows students to choose which subjects they take each academic term. The MoE has also added an extra hour of instruction to the school day in order to encourage students to participate in extracurricular activities, with a focus on applied academic activities, such as science clubs and sports.
Implemented curriculum
The term implemented refers to the actual teaching and learning activities that occur in schools. In effect, the implemented curriculum is a manifestation of how the intended curriculum is translated into practice
Despite concerted efforts to modernise teaching and learning, classroom practices in Saudi Arabia remain very traditional
The constant renewal of Saudi Arabia’s curriculum has been accompanied by a broad array of training initiatives that are intended to support teachers in adopting modern pedagogies. Many of these initiatives have been designed in conjunction with the release of new materials, with a strong focus on the type of job‑embedded support that international experiences shows is most effective in helping teachers embrace new methods (Darling-Hammond and Burns, 2017[5]). This is the case, for example, of the Maths and Science Blended Professional Development Programme that promotes student‑centred instruction (see Main policy initiatives underway).
However, there is consensus that these efforts have not yet translated into significant change in instructional practices in most Saudi Arabian classrooms. Research shows that, in many schools, teaching and learning is still very focused on memorising facts to pass tests rather than developing deep learning (Alhareth, 2014[6]). Pedagogy is characterised by teacher‑led lectures with students taking notes and students are not systematically encouraged to be critical, reflective learners. These observations were confirmed in OECD interviews with teachers, principals and supervisors.
Teachers rely heavily on the provided resources, especially textbooks
An important reason why the intended curriculum is not widely reflected in the implemented curriculum is that teachers in Saudi Arabia rely heavily on the resources they are given, especially textbooks, to plan their lessons (Albedaiwi, 2014[7]). While textbooks are a central resource for teaching in most education systems, what is distinct in Saudi Arabia is the extent to which teachers are reliant upon them to structure their lessons. Instruction frequently follows the textbook from cover to cover, with teachers exercising little discretion as to what material might need to be repeated, adapted or put aside in response to learner needs.
Several factors explain teachers’ heavy reliance on textbooks. These include the relatively weak pedagogical knowledge and skills of many teachers (see Chapter 3) as well as the lack of understanding of the overall learning objectives and standards that they should be working towards. The fact that teacher supervisors monitor teacher compliance with the textbook and assess students against lesson unit outcomes further reinforces teachers’ dependence on textbooks.
Assessed curriculum
The assessed curriculum is the knowledge, skills and understanding that learners actually acquire as a result of teaching and learning, as demonstrated through different means of evaluating student learning. This includes overall policy around assessment and types of assessment, such as classroom assessments (e.g., tests and quizzes administered by teachers), national standardised assessments (e.g., standardised tests with no formal consequences for students) and examinations (e.g., standardised tests with consequences, such as to determine a student’s entry into tertiary institutions). Assessment literature traditionally makes a distinction between assessment for summative and formative purposes. Regarding the instruments used to assess students, two important qualities are to what extent the instruments are valid and reliable. These terms will be used to discuss the assessed curriculum in Saudi Arabia and their definitions can be found in Box 4.1 .
Box 4.1. Key definitions
Formative assessment ‑ assessment for learning, identifies aspects of learning as they are still developing in order to shape instruction and improve subsequent learning. Formative assessment frequently takes place in the absence of marking.
For example, a teacher might ask students questions at the end of lesson to collect information on how far students have understood the content, and use the information to plan future teaching.
Summative assessment ‑ assessment of learning, summarises learning that has taken place, in order to record, mark or certify achievements.
Validity ‑ focuses on how appropriate an assessment is in relation to its objectives. A valid assessment measures what students are expected to know and learn as set out in the national curriculum.
Reliability ‑ focuses on how consistently the assessment is measuring student learning. A reliable assessment produces similar results despite the context in which it is conducted, for example, across different classrooms or schools. Reliable assessments provide comparable results.
Source: OECD (2013), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264190658-en.
Assessment policy and governance
Assessment policy and governance refers to the definition of the objectives of a student assessment system and the regulatory framework in place to ensure that student assessment results are used in a way that helps achieve the objectives. Assessment policy and governance also includes the distribution of responsibilities for implementing the assessment framework and how different levels of governance interact to form a comprehensive assessment system ‑ for example, how national assessments and examinations relate to classroom practice (OECD, 2013[8]).
Both MoE and ETEC set assessment policy in Saudi Arabia, though their activities are not coordinated
In Saudi Arabia, both MoE and ETEC are responsible for setting national policy related to assessment. MoE, through the General Directorate of Educational Supervision, has developed school supervision (inspection) standards. These standards set out how schools and teachers are to be evaluated, including through the direct testing of students by supervisors (inspectors) in order to appraise teacher performance. Separately, the Department of Measurement of Cognitive Achievement will also administer census‑based assessments for system monitoring and school accountability purposes (see Main policy initiatives underway).
QIYAS, the former national assessment and examinations agency, was recently integrated into ETEC, bringing all of QIYAS’s assessments under the latter’s remit. These assessments include a recently developed, sample‑based National Assessment Programme (NAP) and two entrance examinations into tertiary education institutions (see Main policy initiatives underway). ETEC is also developing the new Teacher Standards and Professional Pathways, which are Saudi Arabia’s first ever teacher standards, which include expectations for how teachers assess students.
There is currently no assessment framework that defines assessment activity at a national level and how different assessments, in particular ETEC’s examinations, work together to advance student learning. As a result, and in the absence of any overarching co‑ordinating body, MoE and ETEC set assessment policy largely in parallel, which can result in conflicting priorities and overlapping assessments. This is evident, for example, in the aforementioned duplication of national assessments that arose from a lack of agreement between the two organisations as to whether the assessments should be census ‑ or sample ‑ based.
Classroom assessment
Teachers’ classroom assessment is the most important form of assessment for student learning. Classroom assessment is most effective when it balances regular high-quality formative feedback with periodic summative judgements of student performance in relation to national standards. Modern curricula, like Saudi Arabia’s, that expect students do more than know discrete facts also require teachers to use a broad array of assessment instruments to measure students’ complex skills, such as open‑ended tests, task‑oriented projects and portfolios.
Teachers’ classroom assessment is predominantly summative and assessment materials strongly feature memorisation and closed‑ended item types
In Saudi Arabia, teacher assessment is heavily summative. Teachers and students are very focused on the marks that students receive and consider assessment without marks to be meaningless. Conversations with teachers suggested that students are motivated primarily by the marks that they receive and that there is a culture of “learning for the test”. Teachers mentioned that, in schools that eliminated end of term tests, students became less interested to learn and, in some cases, attended class less.
Reviews of classroom assessment materials showed that they are very content‑focused and do not evaluate complex skills such as problem solving and critical thinking (Almuntasheri, 2016[9]). When asked by the review team to show examples of literacy assessments that evaluated student skills, teachers showed tests that evaluated spelling and punctuation. Most test questions only had one correct answer choice and did not encourage deeper learning.
Formative assessment is encouraged by recent education initiatives, but is not readily practised
Recent initiatives in Saudi Arabia have clearly encouraged the use of formative assessment. These include:
The new curriculum framework, which explicitly emphasises formative assessment as part of student assessment processes;
The growing set of assessment resources created by ETEC to help teachers reliably assess individual student learning in relation to national standards;
Local initiatives by some Education Offices to increase the use of formative assessment in classrooms;
The national guidance given to schools to not assess students using marks until Grade 4.
Despite these efforts, implementing meaningful formative practices remains a challenge. In conversations with teachers, the review team found that few were providing regular formative feedback to students and the majority said that they did not adapt their instruction based upon previous assessment results. These findings were confirmed by several stakeholders who reiterated that teachers had weak overall assessment literacy, and did not understand that assessment went beyond marking and should inform teaching. One administrator summarised the situation as teachers have heard about formative assessment, but it remains “just a noise”.
Principals and teacher supervisors test students as part of teacher appraisal and for monitoring purposes
Teacher appraisal is conducted by principals and teacher supervisors in tandem. To inform their opinions about teacher performance, both principals and teacher supervisors test students independently using different instruments. Principals develop their student assessments by themselves, while teacher supervisors’ assessments are created by selecting questions from an item bank that is developed by their respective Directorates. While these tests may not directly count towards students’ grades, they do mean that students are tested frequently by external authorities, which can take time away from regular teaching and learning. Furthermore, because these assessments contribute to a teacher’s appraisal rating, they also influence classroom teaching by motivating teachers to prepare their students strictly for the tests (Alhareth, 2014[6]).
In Grades 4, 7 and 10 teacher supervisors administer “stage assessments” to all students in maths, science and Arabic. These assessments are conducted in addition to the previously mentioned teacher appraisal assessments. The purpose of the stage assessments is to understand the performance of the students over time and each Directorate administers the same stage assessments in all of its schools.
Assessment resources are centrally provisioned and are not meant to be adapted locally
Similar to general instructional materials, teachers’ assessment resources are provided to them from a central level and teachers have little agency in helping to create them. These include the previously discussed textbooks, which often contain assessment material, but also overall guidance around assessment. For instance, MoE publishes guidelines for how teachers should develop examinations for their students at the end of academic terms. These guidelines give instructions about what types of items should be used, how long the items should be, how the items should be marked and how each set of items should be weighted to produce a student’s final grade.
Providing teachers with rigid assessment resources is problematic in the Saudi Arabian context. As discussed previously, teachers are already inclined to closely follow the resources they have without consideration of their students’ individual needs. Giving them strict assessment guidelines exacerbates this situation.
Teacher professional development does not strongly emphasise improving teacher assessment practices
In Saudi Arabia, teacher professional development is primarily provided through local training centres that are affiliated with Education Offices or Directorates. Some large Education Offices and Directorates might have several training centres. Conversations with training centres revealed that specific pedagogical practices (e.g., active learning and co‑operative learning) are the most frequently demanded training modules. While training in assessment is available, they are underutilised and tend to focus on certain activities (e.g., developing a test) rather than helping teachers improve their assessment literacy in general.
National standardised assessments
National assessments are centrally developed, standardised instruments that are administered to monitor student progress and evaluate educational programmes (OECD, 2013[8]). The vast majority OECD countries administer regular national assessments. While the design of the assessments vary, there is increasing use of census‑based testing as a means to monitor progress and support instructional improvement. It is common for OECD countries to use national assessment data as a source of information for school accountability, but most countries take steps to avoid consequences for teachers or students.
ETEC has created a sample‑based national assessment while MoE has developed census‑based national assessments
The primary national assessment in Saudi Arabia is the NAP, which began in 2018. This assessment is sample‑based and is intended to be used for monitoring student attainment at the national level (see Main policy initiatives underway). It builds on a previous attempt by ETEC ‑ then EEC ‑ to establish a regular national assessment, which was halted in part because of unstable funding.
Because NAP is a sample‑based national assessment, its results cannot be used to monitor sub‑national entities or individual schools. MoE, through the General Department of Measurement of Cognitive Achievement, is developing census‑based assessments that will be used for school accountability purposes. This assessment is currently planned to be administered in Grades 3, 6 and 9.
MoE’s supervisor assessments serve a Directorate‑level monitoring function
While supervisor assessments, both those administered to appraise teachers and the stage assessments, are not standardised, they are nevertheless used to compare school performance. Based on the results of these assessments, Directorates rank schools and these rankings are used for internal monitoring and accountability purposes (these assessments were completed and implemented in 2019, after the analysis for this report was completed).
Saudi Arabia participates in both TIMSS/PIRLS and PISA to help monitor national performance
Saudi Arabia has participated in TIMSS since 2011 for Grade 4 and 2003 for Grade 8, and in PIRLS since 2011 (IEA, 2019[10]). Saudi Arabia joined PISA in 2018 and received its first PISA results in late 2019. Results from these three international assessments are considered key performance indicators in the National Transformation Plan (NTP), which outlines goals for every government Ministry in accordance with the aims expressed in Vision 2030, the national initiative to redevelop the country’s economy.
National examinations
National examinations are standardised assessments that are developed at the national or state level with formal consequences for students. A common practice for almost all OECD countries is to require students to pass an examination in order to be certified as having completed upper secondary school (OECD, 2013[8]). These examinations ensure that graduates demonstrate basic minimum competencies and exert pressure on students to apply themselves in order to pass the exam. Research shows that students who attend schools in areas with certification examinations demonstrate higher academic outcomes and future earnings potential than students who do not (Bishop, 2006[11]).
Two national examinations are administered at the end of upper secondary school to help determine entrance into tertiary institutions
Saudi Arabia does not have an upper secondary certification examination. Such an exam used to be administered but was discontinued in order to increase completion rates. Currently, Saudi Arabia has two national examinations, the General Aptitude Test (GAT) and the Scholastic Achievement Admission Test (SAAT). They are administered starting in Grade 11 and help select students for entrance into higher education institutions.
GAT is not aligned with the curriculum and aims to assess deductive, logical and critical thinking skills through verbal and quantitative sections. The verbal section includes sub‑sections about reading comprehension, sentence completion, analogies, contextual error and relationship and difference. The quantitative section for science majors contains questions about arithmetic, geometry, algebra, statistics and comparison. The quantitative section for arts majors contains geometry, arithmetic and mathematical analysis. All of the questions are multiple-choice and tests are machine scored.
SAAT is also not aligned with the curriculum but is more difficult and is more focused on subject domains. SAAT for science majors tests mathematics, biology, physics and chemistry. Some questions measure comprehension and others measure application and inference. SAAT for arts majors consists of general aptitude questions in reading comprehension, logical relations, solving problems based on basic mathematics, inference skills and measuring capacity. It also tests students in Islamic culture, monotheism, jurisprudence, Arabic grammar, rhetoric and criticism, literature, history and geography. All questions on the SAAT are multiple‑choice.
In addition to selecting students for higher education institutions, the results of GAT and SAAT are used for performance monitoring purposes. Student results are aggregated at the school level and released on ETEC’s website. Schools are ranked according to their average student performance and users can search for and sort schools. Results are published only for schools that have had at least 10 students take the tests in three consecutive years. Figure 4.1 shows the various external assessments and examinations that are administered in Saudi Arabia.
Table 4.1. External assessments and examinations in Saudi Arabia
Main policy initiatives underway
ETEC is developing a new national curriculum
One of ETEC’s primary responsibilities is to develop a new national curriculum. This is being completed in two phases. In phase one, ETEC created a National Framework for Public Education Curricula Standards, which sets out the overall expectations for student learning and achievement in school from Grades 1 to 12. In phase two, which is underway, ETEC will develop subject-specific standards in ten core learning areas.
Underpinning the curriculum framework are 10 “foundations” that represent the main pillars upon which the curriculum standards are built. They include Arabic language, the national objectives of Vision 2030 and an aspiration to align Saudi Arabian education with modern educational trends and practices such as active, student‑centred pedagogy. The framework specifies an overall vision that learners be proud of their religion and language, contribute to their nation’s development, have a constructive and balanced personality and be creative and productive. The curriculum framework is intended both as a guide for the development of teaching and learning materials and as a key lever to align the education system with the country’s overall transformation agenda. To this end, the curriculum framework focuses not only on learning areas, but equally on developing skills, such as critical thinking and creativity, and also on values, such as self‑esteem and responsibility.
ETEC is creating a National Assessment Programme that will monitor student learning nationwide
Developed by ETEC, NAP is a sample‑based, criterion‑referenced assessment in Grades 4 and 8 in maths, science, reading and writing. ETEC intends to administer the assessments annually with the primary purpose of monitoring the academic achievement of students at a national level. The subjects will be rotated in each administration, with each subject expected to be tested every three years. NAP was tested in 2018 in a field study. Roughly 60 000 Grade 4 and Grade 8 students were sampled from around the country and were tested in maths and science. The NAP assessment framework has been designed to enable the benchmarking of student performance against international standards, notably those used in TIMSS and PIRLS. Because NAP was developed before the new curriculum standards will be released, it is not aligned with the upcoming standards and it is unclear if it will be.
The representative sample of NAP will enable analysis according to several different dimensions, such as geographic location and gender. This analysis will be presented in five different reports for various audiences:
An executive summary;
A report for MoE;
A report for principals and teachers;
A report for the media;
A technical report.
Tatweer has created an online portal to hold educational resources
To facilitate the development and delivery of high quality instruction resources, Tatweer has created the National Education Portal (iEN). This portal acts as an online repository that provides teaching and learning opportunities (e.g., online videos and instruction) and educational resources to students, teachers, principals, supervisors and parents. In addition to centrally developed resources, teachers submit their own materials directly to iEN for other teachers to browse and use. The resources held by iEN include lesson plans, worksheets, homework assignments and a question bank for classroom assessments. Currently, iEN holds over 42 000 resources and over 100 000 assessment items. Tatweer wishes to expand iEN’s assessment item bank to over one million.
Intended curriculum
Strengths
The national curriculum increasingly encourages modern instructional approaches
Saudi Arabia has continuously revised its curriculum to encourage modern instructional approaches that have been shown to be effective in helping students learn. Many of the curriculum materials examined by the review team encouraged teachers to engage in instructional activities that went beyond asking students to perform lower order tasks such as memorisation. Instead, these materials promoted the use of more demanding activities that asked students to reflect on their learning and construct their own conclusions. These expectations for student learning align broadly with those found in OECD countries in terms of rigour and coherence. Policy reforms have also reinforced curriculum revisions. There have been notable attempts to reduce content‑overload and to give students more flexibility and ownership over what they learn, such as through extending the school day and introducing the credit system in secondary school.
Tatweer has strong capacity to develop curriculum materials and deliver them to schools
Tatweer has strong organisational capacity to develop curriculum materials. Staff at Tatweer have considerable subject‑matter expertise, which informs the development of their resources, and also technological proficiency, which helps with resource delivery. The creation of iEN is a very positive initiative because it allows teachers to share their own materials with each other. These materials are more likely to be more relevant to classroom contexts than those developed centrally. The fact that iEN is online also allows centrally developed and teacher developed resources to be accessible to more teachers.
Twenty‑first century skills are firmly embedded in the new curriculum framework
Around the world, education systems have increasingly emphasised the development of complex and applied skills. These key skills are often called 21st century skills and can include creativity, communication and use of technology (OECD, 2013[8]). They are seen as increasingly important to student success and national competitiveness in era of knowledge‑driven economic growth.
Saudi Arabia’s new curriculum framework explicitly focuses on developing these transferrable, 21st century skills. The document is underpinned a set of curriculum priorities, values and skills that include creative thinking, communication and self‑learning. Using these principles to guide the creation of subject‑specific standards indicates that that students are expected to use their knowledge in new situations, and not just memorise answers to take a test.
The new curriculum framework has the potential to establish consistency across subjects and address gaps in standards and Arabic language materials
Recent decades have seen almost constant curriculum development in Saudi Arabia, with materials being adapted from a range of national and international sources and revised based on continually evolving specifications. This has inevitably resulted in a degree of pedagogical inconsistency across subjects and core concepts. The creation of the new curriculum framework, and its position as the main reference for developing curriculum materials, has the potential to improve the coherence of students’ learning experiences. It can also support the establishment of a more structured cycle for curriculum revision in the future.
The new curriculum framework also allows for the development of clear standards for student achievement across subjects and grades. These standards would give teachers a broader reference for student learning than the discrete learning objectives contained in textbooks. The standards can further support the use of a wider range of instructional materials than are presently employed. Importantly, the planned new standards for Arabic are intended to address many of the concerns in the current approach to language learning. These include the heavy focus on grammar and other language construction rules and a limited focus on overall understanding (see Chapter 5).
There is broad alignment between the curriculum framework and standards for teachers and schools
It is crucial that the reference standards for student learning, teachers and schools be aligned so the entire education system acts to achieve the same goals. In Saudi Arabia, the OECD review team noted that there is strong alignment between the new curriculum framework and the standards that are being developed for teachers and schools. The documents shared with the review team signal a consistent emphasis on active learning, student‑centred pedagogy and context‑sensitive teaching and learning.
This type of alignment in expectations provides a strong basis for creating alignment in pedagogy and assessment and more broadly in how teaching and learning is organised in schools. This alignment will be further strengthened when ETEC begins developing accompanying instruments to support the evaluation of teaching practices, such as classroom observation protocols and indicators of teaching and learning.
Challenges
The new curriculum framework lacks some internal coherence and might be difficult for users to understand
The general content of the curriculum framework suggests a clear shift of teaching and learning towards contemporary education principles. However, the document is dense and not always internally coherent. For example, the framework is organised into the following sections, intended to guide the development of subject‑specific standards:
Vision of the standards
Structure of the standards (learning areas, curriculum priorities, values, skills)
Levels of learning
Guiding principles of the standards (curriculum content, learning processes, information and communication technology, assessment processes)
Independently, these sections are coherent. Read together, however, it is difficult to determine how they are integrated. Levels of learning ostensibly refers to levels of student progression in learning areas and/or skills. However, a close review of this section suggests that it refers to grade levels or stages of schooling. Further, this section only makes reference to learning areas (subjects), but not values and skills, even though these were explicitly mentioned earlier in the framework. Thus, while values and skills are clearly an important part of the framework, it is unclear how students are expected to progress in these areas.
Similarly, the sections on learning and assessment processes are independently logical, but are not aligned with each other or the rest of the framework. It is uncertain, for instance, how the seven principles for assessment support the six principles for learning. It is equally uncertain how assessment processes will assess values and skills, especially across different levels of learning.
This lack of internal coherence could negatively impact the users of the curriculum framework. The primary audience for the curriculum framework is developers of learning standards and instructional materials. Given the lack of coherence between different parts of the framework, however, a test developer will be unsure about how he/she should use the framework to, for instance, create a test that assesses student’ skills across different levels of learning.
Plans for how the new curriculum will be introduced could be clearer and might underestimate the need for significant two‑way communication
International experience shows that effective curriculum reform begins with inclusive consultation of stakeholders (Fullan, 2007[12]). It continues with constant two‑way communication to align implementation efforts across central, local and school leadership and feed lessons from practice back into curriculum policy (Barber, 2009[13]; Persson, 2016[14]).
In Saudi Arabia, conversations with schools and teachers, some of which occurred online through the Tatweer portal “Your Opinion is Important”, revealed that teachers feel uninformed about the curricular changes underway. Some were unaware that a new curriculum framework is about to be introduced. While input from teachers, supervisors and local education authorities has been solicited as the framework has been developed, this is not visible and does not appear to have been adequate to ensure that materials are contextually relevant or that teachers have “bought into” the changes.
The curriculum framework itself contains a section about implementation. However, this section only articulates two initiatives-further professional development for teachers and aligning education system components with the new framework. Including teachers in a pilot of the curriculum or consulting them about implementation challenges is not mentioned. Without such engagement by teachers and school leaders, there is a possibility that the new curriculum will not be widely understood or used properly.
Curriculum resources have been created but might not be sufficient, and can be difficult to access for some populations
Tatweer has made tremendous strides in creating materials to accompany the national curriculum. Nevertheless, given the large number of schools in Saudi Arabia and the variation in teacher capacity, the educational resources available might still be insufficient. In visits to schools, the review team was told that inadequate resources is still a significant concern, both in terms of quality and quantity.
The development of iEN is a step forward in creating and disseminating teaching and learning resources in Saudi Arabia. However, OECD school visits revealed that, while some teachers are using iEN, others lack sufficient digital literacy to access the portal and make best use of the resources in the classroom. This is particularly true for teachers in remote, rural areas. Furthermore, there is an enormous amount of content on iEN, and Tatweer intends on adding even more. With limited time and capacity, teachers cannot easily identify which resources are best and most relevant to their needs.
Implemented curriculum
Strengths
A new centre for professional development will provide more support to teachers in implementing the curriculum
The National Institute for Professional Educational Development (NIPED) is responsible for the professional development of teachers, principals and supervisors. Teacher professional development has been designed to focus on modern pedagogical skills that are needed to engage in the student‑centred, outcomes‑based instruction that curriculum reforms have demanded. In particular, NIPED has taken strides to lessen teachers’ rigid reliance on using the textbook in class, which impedes them from adopting modern pedagogical approaches. For example, NIPED has collaborated closely with educators from the United States and developed Lesson Plan Study activities in partnership with Japan. This training encourages teachers to study other teachers’ lesson plans in order to increase their capacity to create lessons that are less dependent upon on their textbooks and more adaptive to the needs of students.
Challenges
Teacher appraisal structures discourage teachers from implementing the intended curriculum
One factor that is preventing teachers from implementing the intended curriculum more fully is how they are appraised. An important criteria against which teachers are evaluated is how much of the curriculum they have covered, as measured by student results on supervisors’ tests (Ministry of Education - World Bank, 2017[15]). This system incentivises teachers to race through the curriculum in order to receive a good appraisal rating. This is contrary to the modern instructional approaches promoted by the intended curriculum, which emphasise developing deeper learning and tailoring instruction to the needs of individual students.
Teachers’ strict adherence to classroom resources contradicts the aims of the intended curriculum, which expects teachers to adapt materials for student needs
Textbooks exist, alongside many other resources, to support teachers in their lesson planning. They should not be seen by teachers as prescriptive instructions to be followed word‑for‑word without any regard to the classroom context and their students’ specific needs (Isaacs, unpublished[16]). However, in Saudi Arabia, many teachers believe that textbook content is synonymous with the curriculum and follow textbooks very closely.
Teachers interviewed by the OECD review team understood the learning outcomes expected at the end of each lesson unit, but lacked an appreciation of overarching content and standards, which are critical to know if teachers are to make more discerning use of textbooks and other resources. This rigid adherence to the textbooks prevents teachers from focusing on individual students and adapting their instruction based upon their students’ ability and interests. In this context, the new curriculum framework, which encourages teachers to focus on standards, rather than the textbook, and draw on a range of instructional resources, will encounter challenges in school‑level implementation. Box 4.2 describes the challenges that were encountered in South Africa, a large country with significant geographical disparities, as it tried to implement a more modern curriculum.
Box 4.2. Lessons learned from introducing a modern curriculum in South Africa
In South Africa, curriculum reform in 1997 sought to replace a knowledge-based curriculum with an outcomes-based curriculum that integrated broader learning areas (e.g., teamwork, critical thinking and problem solving). However, a review of the curriculum’s implementation noted that there was a lack of alignment between what the curriculum intended and what was implemented in schools and the reform was not pursued further. Researchers studied the situation and noted that limited teacher capacity contributed significantly to why the reforms were unsuccessful. Specific reasons include:
A lack of assessment knowledge among teachers, especially on formative assessment, due to the absence of guidelines about the fundamental principles of good assessment practice;
Teachers did not have access to adequate resources, such as revised curriculum. Syllabi and textbooks, and did not know how to use what was available;
Teachers did not understand the curriculum and did not receive sufficient training and development around implementing it;
Source: Kanjee & Sayed (2013[17]) Assessment policy in post-apartheid South Africa: challenges for improving education quality and learning, Assessment in Education: Principles, Policy and Practice, Vol.20/4, pp 442-469. https://doi.org/469. 10.1080/0969594X.2013.838541
Schmidt (2017[18]), The Perils of Outcomes-Based Education in Fostering South African Educational Transformation. Open Journal of Political Science, 7, 368-379. https://doi.org/10.4236/ojps.2017.73030
Assessed curriculum
Assessment policy and governance
Strengths
Assessment is a clear priority of the education system
Assessment has been a longstanding focus of the Saudi Arabian education system. All stakeholders recognise the importance of assessment, which is evinced by the quantity of external assessments that are administered to students (see Table 4.1). The new curriculum framework reserves an entire sub‑section to principles of assessment, further emphasising the importance of assessment in the instructional process.
Considerable assessment expertise has been developed at the national level
ETEC is a highly technical organisation with considerable assessment expertise. ETEC is administering several national examinations and assessments. These assessments are delivered via paper‑and‑pencil and computer mediums and some assessments use sophisticated item response theory models. ETEC is also helping to build the assessment capacity of Directorates, Education Offices and schools. It is developing materials that teachers and principals can use to assess the performance of schools in a standardised manner and will also create resources to aid the school evaluation framework.
Challenges
Not all assessment activities are aligned through national policy, such as a national assessment framework
A common, coherent vision for student assessment needs to be disseminated to stakeholders in order to create consensus around what assessment is and why it is important. In many countries, this is accomplished through the creation of an assessment framework, which articulates a clear assessment pathway for students from primary through upper secondary education. This would include the types of assessments to be used, when and how they should be used and the subjects to be assessed. In high-performing countries, assessment frameworks cover both formative and summative assessment and specify how both purposes are achieved through classroom assessments, national testing programmes and examinations. Box 4.3 describes the integrated assessment framework from Hong Kong.
Box 4.3. Assessment framework in Hong Kong
Hong Kong’s assessment framework combines school‑based assessment, a territory‑wide system of assessment and subject examinations in upper secondary school. School‑based assessment emphasises formative assessment but also helps guide teachers’ summative judgement, especially in upper secondary school. The assessment framework can be easily accessed on the website of the Hong Kong Examinations and Assessment Authority (HKEAA) (http://www.hkeaa.edu.hk/en/), which allows the public to easily know what assessments students are expected to take.
National tests include the Territory‑wide System Assessment (TSA) tests of basic competency that are administered in each school at years 3 and 6 in primary school and at year 3 in secondary school. These provide schools with objective data on students' performances in Chinese language, English language and mathematics. TSA reports and school reports provide information about students' strengths and weaknesses against specific basic competencies and key learning areas. The outcomes of the tests aid schools and teachers in their teaching and learning plans. Government uses the territory‑wide data to review policies and provide focused support to schools. These tests are for internal use only; comparisons on the performance among schools and students are not supposed to be made and access to data is restricted, with schools following a strict protocol to avoid information misuse. A section of the HKEAA website makes available various resources related to the TSA, such as question papers, marking schemes and reports.
Students in upper secondary school prepare for the Hong Kong Diploma of Secondary Education (HKDSE) examinations held at the end of year 6 of secondary school. HKDSE tests four core subjects ‑ Chinese, English, mathematics and liberal studies ‑ and up to three selected subjects. The results at this examination contribute to students’ certification requirement from secondary school.
Source: Isaacs & Creese (2014[19]), Aligned Instructional Systems: Cross-Jurisdiction Benchmarking Report. National Center on Education and the Economy. http://www.ncee.org/wp-content/uploads/2015/09/AIS-Cross-Jurisdiction-Report.pdf
HKEAA (2019[20]), Website of the Hong Kong Examinations and Assessment Authority. http://www.hkeaa.edu.hk/en/
In Saudi Arabia, a national assessment framework that oversees all assessment in the country has not been developed. Both ETEC and MoE engage in substantial assessment activities, but these are not regulated by a single point of reference. Assessment activities are not co‑ordinated and most stakeholders do not have an accurate picture of all the assessment that occurs in the country. As a result, there is duplicate testing (e.g., the external assessments by Directorates, ETEC and MoE) as well as assessment gaps in key domains during critical period of student learning (e.g., there is no national, census‑based assessment during primary grades). There are also tests that are not fulfilling their stated purposes, as is the case with supervisor assessments that distort instead of supporting teaching and learning.
The assessment principles in the curriculum framework are theoretically sound but will be hard to implement
In any curriculum reform, the assessments that will provide evidence of student achievement of the curriculum should be developed simultaneously with the curriculum standards and content (Ahmed and Pollitt, 2008[21]; Crisp, Johnson and Constantinou, 2018[22]). In Saudi Arabia, central education policy advocates for a modern understanding and practice of assessment. The new curriculum framework encourages teachers to use a diversity of assessment methods and to utilise the data from assessments to improve instruction. These principles are in line with international best practices supported by the OECD review team (Isaacs and Creese, 2014[19]).
However, as with other aspects of the curriculum, it does not appear that assessment practice in Saudi Arabian classrooms fully reflects the modern principles advocated by central policy. Teachers often ask students to memorise and recite passages in order to test their understanding of a concept. Written tests and quizzes, whether developed by teachers, principals or supervisors, tend to be composed mainly of multiple‑choice questions. These common practices suggest that the modern vision of assessment that is espoused centrally has not embedded itself into classrooms.
Classroom assessment
Strengths
Regular summative assessment procedures are established and followed
Summative assessment features prominently in Saudi Arabian education. Teachers record individual student achievement in their classroom grade books and produce report cards that are shared regularly. These practices are well-established and monitored closely by principals and supervisors. Teachers receive some support, in the form of criteria and rubrics, to improve the reliability of their judgements and ETEC intends to provide teachers with more assessment resources to further improve the quality of their summative instruments and the validity of their marking.
Formative assessment has gained prominence in recent initiatives
While Saudi Arabian classrooms have typically been dominated by summative activities, formative assessment has gained recent attention. Some local training centres now provide courses in assessment for learning, which the NIPED intends to strengthen and expand. Tatweer has launched a study to determine to what extent formative assessment practices are being used, including asking students if they know how well they are doing based upon their teachers’ feedback. This will generate valuable evidence to inform the expansion of professional development. Tatweer has also developed several resources to aid teachers in using formative assessment methods. These include tools such as observation protocols and guidelines around student journals, student self‑assessment and assessment by more than one teacher. The new school evaluation framework should give further impetus to these efforts and positively includes providing feedback as one of the two sub‑standards for evaluating the quality of assessment.
Challenges
There is too much assessment of little education value for students
Summative assessment is an important part of assessment and education in general. Research shows, however, that well‑conceived formative assessment has the potential to more positively impact student learning (Black and Wiliam, 1998[23]). For this reason, many countries are trying to emphasise formative assessment while exercising caution with respect the amount of summative assessment that students encounter (OECD, 2013[8]).
The lack of a national assessment framework in Saudi Arabia and the strong culture of testing have created a situation in which a great deal of predominantly summative assessment occurs without alignment or a clear sense of the educational purpose behind the assessments. Depending upon the grade a student is in, the review team calculated that some students might be subject to upwards of 100 hours of quizzes and tests per academic year. Many of these assessments duplicate each other, with teachers, principals and supervisors all testing students on similar areas of knowledge.
Importantly, the large volume of assessment that students undergo adds little value to their learning. Much of the testing occurs to help produce a rating for teachers or schools, not to directly help students learn. Further, such a heavy testing burden actually has a negative effect on students’ education. Teachers and students spend so much time preparing for and taking tests that they do not have enough time to cover what is already a dense curriculum in a relatively short instructional year. The fact that persons other than a student’s teacher enter the classroom to test students on a regular basis, a very uncommon practice internationally, also undermines the professional autonomy of teachers.
Teachers’ assessment judgement lacks accuracy and reliability and no moderation practices are in place to improve these areas
At its core, the purpose of assessment is for teachers to form reliable judgements about students’ progress. If teachers’ assessment judgements are not accurate, the purpose of assessment and what can be done with assessment results is undermined. Teachers will not know what their students are capable of and thus cannot adapt their instruction accordingly, or will do so incorrectly. Students might be under the impression that they have mastered a subject and lose motivation to continue applying themselves when, in fact, there is much that they can still learn.
In Saudi Arabia, stakeholders do not always trust teachers’ judgement, which largely explains why principals and supervisors also test students. Resources available to improve this situation, while expanding, are still limited overall. In particular, they do not address adequately the need to strengthen in‑school moderation practices, in which teachers review each other’s assessments and mark students’ work against a shared set of criteria (OECD, 2013[8]). Such practices expose teachers to a wider variety of student work and help them identify their own sub‑conscious biases when marking. They thereby support teachers in establishing a judgement of student achievement that is more consistent, fair and valid in relation to expected curriculum standards. The OECD review team did not find evidence of moderation in schools visited, nor of policies in place to address this gap.
Saudi Arabia’s assessment resources do not capture information about students’ higher‑order skills
It is important for assessments to evaluate the most critical skills that students need in order to be successful. In knowledge economies, this means that assessments should evaluate higher‑order and transversal skills, which are more effectively assessed through the usage of open‑ended assessment questions and performance‑based tasks (e.g., experiments and projects) (OECD, 2013[8]). Developing such skills and assessments are emphasised strongly in most modern curricula, including in Saudi Arabia’s new curriculum framework.
However, research into assessment in Saudi Arabian schools suggests that the assessment methods used by teachers assess rudimentary, lower‑order skills, such as finding a word in a passage (Alafaleq, 2014[24]). One main reason for this is the types of assessment resources provided to teachers. The review team examined a selection of these and found them to be overly focused on factual recall and the completion of simple tasks and inadequate in guiding the evaluation of higher‑order skills.
For example, the recently introduced Exam Guidelines for Teachers (2017) gives information on how teachers should develop, implement and mark end‑of‑term tests at the intermediate and secondary stages. The review team studied the English subject guidelines and found that they would require significant further revision to support the new curriculum framework in helping students master higher‑order thinking and communication skills.
For instance, the exam guidelines for English mostly propose selected‑response type items, such as multiple‑choice, one word or very short answers. Marking student responses is largely based on looking for demonstration of lower order thinking skills, such as defining basic concepts with little application of those concepts. Although these guidelines require that students complete an open‑ended writing exercise, this does not receive adequate assessment weight. For intermediate students, the writing section is worth five marks out of a total 30. For senior students, it is worth 12 marks out of a total 50. How the writing task is marked also focuses on lower‑order skills. In the intermediate level, 60% of a student’s 5‑point mark on the open writing task is from grammar, vocabulary, mechanisms and spelling, In other words, 28 of 30 points on an intermediate student’s end‑of‑term English test is derived from a simple, mechanistic understanding of the language
How teachers are appraised prevents teachers from adopting formative assessment more widely
Research shows that embedding formative assessment practices into schools is very challenging. A key factor in successfully promoting the use of formative assessment is aligning national policies so teachers are supported in their use of formative assessment. In many countries, however, policies have acted as barriers. These policy barriers include a dense curriculum, the pressure on the teachers to cover all of the curriculum, the lack of support to schools for implementing formative assessment and accountability systems that are focused on summative assessment scores (Box, Skoog and Dabbs, 2015[25]; OECD, 2005[26]). In Saudi Arabia, all these obstacles are apparent.
Teachers have weak assessment literacy, but training agencies do not have the capacity to provide adequate professional development in this area
Many of the aforementioned policy barriers were actually created in response to concerns about teachers’ overall weak assessment literacy. In conversations with the review team, teachers seemed to interpret formative assessment as simply not giving a grade, thus mistaking a technique for a principle. Some also mistook continuous summative assessment as formative assessment. Researchers have noted that teachers in Saudi Arabia do not feel comfortable adapting assessment resources for their students’ needs or creating their own (Albedaiwi, 2014[7]). Thus, policies were established to strictly regulate how teachers teach and assess in an effort to mandate the use of certain activities.
While well‑designed accountability structures can play a role in mitigating weak assessment literacy, if assessment is to support learning there is no way around developing the professional assessment knowledge and skills of teachers. Until recently, such support in Saudi Arabia has been relatively limited. There is consensus that most teachers receive little preparation in assessment in their initial teacher education and in‑service training capacity is highly constrained. Tatweer, NIPED and ETEC have created some valuable resources that teachers can draw upon. However, in order to make meaningful use of these, teachers need qualified and sustained support within their schools and classrooms, either from their peers, their principals or qualified external coaches. Such support is not currently available in most schools.
National standardised assessment
Strengths
Saudi Arabia has made considerable efforts to collect more consistent data on student learning
Saudi Arabia recognises the need to assess students in a consistent manner in order to monitor student performance in relation to national curriculum standards. Both ETEC and MoE have created national standardised assessment initiatives to help accomplish this purpose. ETEC’s NAP will test a representative sample of students, while a new MoE programme might administer a standardised, census‑based national assessment. MoE also tests all students through supervisors’ assessments, which are used to compare student achievement across schools.
Saudi Arabia is supplementing national data with international survey results
Participation in international surveys provides comparative information in key areas of schooling. The results provide a frame of reference that helps countries determine their strengths and weaknesses, allows them to better understand their own system and provides ideas to inform further research and policy development (Tamassia and Adams, 2009[27]; OECD, 2013[8]).
MoE and ETEC have been making intelligent use of Saudi Arabia’s international survey data (both the performance results and contextual questionnaire information) to help develop the curriculum and inform the assessment items for NAP. In 2018, Saudi Arabia joined PISA for the first time with the objective of focusing more attention on broader student competencies and 21st skills, such as creativity and digital literacy.
Challenges
Large‑scale, census-based national assessments are not yet in place
Census‑based, national testing allows countries to analyse data at several levels and according to different dimensions. School and regional level data can be produced and examined according to students’ gender, family background and other factors. A majority of OECD countries administer census‑based national assessment at either the primary or lower‑secondary levels (OECD, 2013[8]).
At the time of writing, Saudi Arabia does not have a census‑based national assessment. The only wide‑ranging assessment data comes from student results on GAT and SAAT. NAP partly addresses this situation by testing a representative sample of students at several grade levels. However, its sample‑based strategy will not allow for results to be collected at the school‑level or, depending on how sampling occurs, at the Education Office‑ or Directorate‑level. MoE’s supervisor assessments do test all students, but the instrumentation is not standardised. This means that students and teachers have no reliable, external benchmarks of learning outcomes until Grade 12. It also weakens accountability at the school level, which is especially problematic in a country where comprehensive school evaluation is still under development and will take a long time to roll‑out to all schools.
National performance is measured primarily by results on international surveys, but these results do not capture the Saudi Arabian context
In the National Transformation Plan, national goals related to student achievement only focus on using international survey results. While it is positive to use these outcomes for benchmarking purposes, these instruments do not fully reflect the educational context or curriculum of Saudi Arabia. National assessments, such as NAP, have been introduced, but student achievement according to this assessment has not yet been translated into national goals. This also creates uncertainty regarding the future of NAP and its position as a key performance metric (a lack of sustained political and financial support stymied previous attempts to establish a regular, standardised national assessment).
National examinations
Strengths
End of secondary examinations are established and well understood by stakeholders
National examinations serve several purposes. Being centrally produced, they provide an objective signal of student achievement that can help inform students’ future education and career options (Bishop, 1999[28]). Research shows that high‑stakes examinations can also positively impact students’ motivation and reinforce rigour in classroom learning by encouraging students and teachers to prepare for the test (Sukyadi and Mardiani, 2011[29]; Roderick and Engel, 2001[30]). Finally, a national examination that is aligned with the curriculum also helps concentrate teaching and learning on the curriculum, which helps ensure alignment between the intended and implemented curricula.
In Saudi Arabia, the GAT and SAAT are administered at the end of upper secondary education for the purpose of university admissions. They are administered by ETEC and their methodology has been tested and refined over several administrations. Students and teachers are well aware of these examinations and perceive the examinations to be fair. Although these examinations select students for entrance into for higher education, national priorities to admit more students into tertiary institutions have lowered the stakes of these examinations. This, in turn, also limits how much the exams can exert pressure and influence teaching and learning in classrooms.
Challenges
No national examination is aligned with the curriculum, which prevents the examination system from supporting implementation of the curriculum
International experience shows that what is tested shapes what is taught. Teachers and students, wary of the importance attached to examinations at the end of upper secondary school, tailor their teaching and learning to the content of those examinations. Tests that are aligned with the curriculum help ensure that teachers and students focus on the curriculum in class. This is particularly true for certification examinations that directly assess whether students have mastered the curriculum. Tertiary entrance examinations, especially in the presence of a certification examination, generally have a little more flexibility with respect to curriculum alignment in order to accommodate the needs of tertiary institutions. However, it is highly unusual for the needs of tertiary institutions to be strongly misaligned with what upper secondary curricula deem to be the most important competences that students should master.
Since Saudi Arabia currently does not have an upper secondary certification examination, the GAT and the SAAT play a de facto role of helping to steer the focus of teaching and learning. However, the review team was told that neither the GAT nor the SAAT are aligned with the curriculum and that this is by design. Currently, ETEC does not believe that the curriculum and university expectations are co‑ordinated. Therefore, ETEC views its university entrance examinations as bridging a gap between what students learn in school and what universities expect them to be able to do.
Because no exam is fully aligned with the curriculum, Saudi Arabia does not have an external lever than can support teachers in implementing the intended curriculum. Teachers who do decide to use classroom instruction time to help students prepare for the tests that are administered must actually deviate from the intended curriculum to do so. This is problematic in a context in which the implemented curriculum is already misaligned with the intended curriculum.
Success on the GAT and SAAT is generally expected, which might prevent the exams from applying positive pressure on students to apply themselves
Saudi Arabia’s tertiary enrolment rate has steadily risen in recent years. While taking the GAT and the SAAT is required in order to enter tertiary education institutions, most students who take them end up enrolling in tertiary education. This suggests that the exams are either rather easy for students, or entrance requirements are set fairly low, or both.
Without high‑stakes, the GAT and the SAAT are unable to impact strongly student and teacher motivation in upper secondary school. In other countries, examinations motivate students to apply themselves and teachers to help students prepare. In Saudi Arabia, because most students who take the exams expect to enrol in university, they feel less compelled to apply themselves. Consequently, the education system loses a powerful lever for improving student achievement.
Items from the GAT and SAAT test a limited range of skills and are sometimes internally incoherent
The review team analysed items from the GAT and SAAT to better understand the content of these examinations. While many items are well designed, the review team also identified several weak items. These include items that assess skills that are too simple, internal incoherence of items (e.g., there were factual errors in the questions or why an answer is correct is unclear) and item bias. A selection of specific limitations that the review team identified from these two examinations can be found in the annex of this chapter.
Recommendations
Intended curriculum
Improve the internal coherency of the curriculum framework
Before the new curriculum framework is officially released, the OECD review team recommends that it be reviewed and made more internally coherent. In particular, the section on curriculum standards should be further developed so learning areas, curriculum priorities and values explicitly relate to each other instead of being separate lists of ideas and issues. Box 4.4 describes some key elements of curriculum frameworks and provides digital links to curriculum resources from high-performing countries.
Saudi Arabia’s review of the curriculum framework should be done in conjunction with the teacher and school standards so that common language is used and complex concepts are defined in the same way. While broadly aligned, there is space for improvement in Saudi Arabia’s materials. For example, the same student skills that are mentioned in the curriculum framework should also appear in the teacher standards and in the new school evaluation framework so teachers and schools are appraised based upon how well they help students develop these skills.
Box 4.4. Curriculum frameworks
The table below provides broad guidelines about what is usually found in a national curriculum framework.
Examples of curriculum frameworks from around the world can be found through the links below.
Source: International Bureau of Education (2013[31]), Training tools for curriculum development: a resource pack; 2014, http://www.ibe.unesco.org/fileadmin/user_upload/Publications/Training_tools/IBE-CRP-2014_eng.pdf.
Solicit feedback from teachers about the new curriculum before it is fully implemented to ascertain its usability
Teachers should be consulted extensively about the new curriculum before it is fully implemented. Curriculum development and implementation can only be successful if the final users, a country’s teachers, are confident that they can understand the curriculum and apply it in their classrooms (Solini, Pyhalto and Pietarinen, 2017[32]). Internationally, Japan is a high‑performing school system in which teachers and other stakeholders are integrally involved in the reform of curriculum. This process is called “Materialization of society‑wide commitment to improve education” and, as part of this process, meetings are organised at schools during which school staff receive a guide to the revised curriculum and provide feedback before the reforms are finalised (MEXT, n.d.[33]).
Saudi Arabia should establish procedures to formally involve teachers in a review of the draft curriculum and its implementation strategy. This can be accomplished by forming separate working groups of teachers from different areas of the country and who teach different grades and subjects. These teachers would be responsible for reviewing the curriculum as it has been developed thus far and providing their opinions about how easy to understand it is and how easy to use it would be. They would also provide input about what kind of resources need to be provided alongside the curriculum to help teachers implement the curriculum in their classes.
Create a transparent, regular process for curriculum review
Developing a modern curriculum framework is a tremendous accomplishment. As the education landscape changes, however, it will be important that the curriculum and learning standards be subject to periodic review to reflect the latest developments in education and evolving skills needs. A key element of curriculum review is to establish a regular timeline and procedures, which help create transparency around the process and legitimise future reforms. This process helps to maintain the relevance of curricula while avoiding the destabilising effects of the type of unexpected curriculum changes that have occurred in Saudi Arabia previously. Internationally, this type of regular review occurs in England, Finland, Japan, the Netherlands and Singapore (Sargent et al., 2010[34]).
In Saudi Arabia, the OECD review team recommends that a regular curriculum review process be established that is a joint effort between MoE, ETEC, Tatweer and NIPED. Such a process would incorporate perspectives from organisations responsible for education policy, curriculum development, resource provision and training delivery. This consortium should establish a clear timeline that specifies how frequently reviews should occur, how long they should take, which entities are responsible for which tasks and how the review should be conducted.
Develop more effective classroom instructional materials by including teachers in their creation
Tatweer is responsible for developing materials to accompany curricula. While the effectiveness of the resources has steadily improved, there remains some concerns about the types of pedagogical approaches that the materials support and their level of rigour. Internationally, countries often enlist teachers in the development of resources as they are most familiar with the needs of their classrooms and the inadequacies of their current materials. This partnership is critical in Singapore, where teachers provide annual feedback on textbooks and other resources and their suggestions are taken into account for the next version of the materials (Isaacs et al., 2010[35]).
To strengthen the quality of resources provided to teachers, Tatweer should enlist teachers to help them develop materials more directly than in the past. Teachers should review the resources and communicate if they are relevant to teachers’ needs and accessible to teachers with varying levels of experience. Teachers can also help identify which parts of the textbooks might not be relevant for some classroom contexts and suggest that Tatweer, through textbook guides or the textbook content itself, prompt teachers at these moments to exercise flexibility and draw on other materials.
Given the size of the Saudi Arabian workforce, it will not be possible to involve a significant percentage of the country’s teachers in resource development. What will be important is for the teacher workforce to know that their fellow teachers contributed to the development of the resources, which would make all teachers more likely to embrace and use the resources (Castro Superfine, Marshall and Kelso, 2015[36]; Persson, 2016[37]).
Implemented curriculum
Develop teacher appraisal processes that are aligned with the curriculum’s pedagogical expectations
Other chapters recommend an overhaul of Saudi Arabia’s teacher appraisal system, with a new cadre of assessors being responsible for appraisal for promotion and principals being responsible for regular appraisal, but only for formative purposes. Chapter 3 also recommends that how teachers are regularly appraised be fundamentally revised, and that the direct testing of students to appraise teachers be stopped.
This chapter recommends that the reformed, school‑based regular appraisal be designed in a way that reinforces the new curriculum’s expectations with respect to instructional practice. The new teacher standards, which should guide appraisal, are broadly well aligned with the curriculum, emphasising the importance of using a diverse range of educational resources, the need to set high expectations for every student and foster higher‑order thinking skills. However, the standards and sub‑standards are set at a general level and principals will need additional guidance on how to evaluate teachers against these expectations.
As the developer of the teacher standards and curriculum framework, ETEC is well positioned to create materials that will help principals perform regular teacher appraisal. For example, ETEC should develop classroom observation guidelines, along with performance descriptors and videos of practice. These tools can help principals reliably evaluate how teachers are performing in areas that are expressed in the curriculum, such as using a variety of curriculum and assessment materials that are related to national standards. Internationally, the New Zealand Teaching Council (NZTC) has created high‑quality resources to guide principals through conducting effective teacher appraisal. These include a quality practice template to illustrate what types of evidence can demonstrate that good practice is occurring and an alignment matrix that shows how the appraisal system is co‑ordinated with the curriculum and teacher standards (OECD, 2013[8]; Nusche et al., 2012[38]; Teaching Council New Zealand, n.d.[39]).
Align teacher continuous professional development with the new curriculum and its emphasis on modern teaching and learning
Chapter 3 makes extensive recommendations about continuous professional development for teachers. This chapter highlights the importance of ensuring that the aims of the curriculum, particularly its emphasis on modern pedagogy and assessment methods, be considered centrally when designing the training offered through NIPED. The centralisation of the Saudi Arabian system means that the MoE has strong levers to align teacher supports with curriculum goals, especially if there is more structured collaboration with ETEC, Tatweer and NIPED, as this review strongly recommends.
An area of focus for teacher training that is particularly relevant to Saudi Arabia is how to properly use textbooks to complement instruction. Research shows that training teachers in how to use textbooks can stimulate a change in teachers’ thinking about what are effective instructional practices and how students should learn in relation to the books (Arbaugh et al., 2015[40]). Creating and delivering this type of training, alongside developing new teacher appraisal procedures (see next recommendation), can help teachers view and use textbooks as a complementary educational tool as opposed not a script to be followed. The Lesson Plan Study activities that NIPED is promoting could be used for this purpose. As part of participation in this training, teachers could be shown how to incorporate their own materials into their lesson plans and evaluate the efficacy of those lessons. The OECD review team recommends that this training be delivered via a blended methods approach, in which some training is delivered in‑person and some online, which has proven to be effective in training in successful models such as Pearson’s.
Assessed curriculum
Assessment policy and governance
Create a national assessment framework that clarifies the purpose, methods and relationship between classroom assessment, national assessments and examinations
Saudi Arabia has a wide variety of national assessments, national examinations and classroom assessments. What is missing, however, is co‑ordination of all these assessments so their aims are clear, complementary and aligned with curriculum goals. To accomplish this, and also address the duplication and redundancy that currently exists in relation to some assessments, it is recommended that Saudi Arabia create a national assessment framework.
Saudi Arabia’s national framework should be based on national educational goals. Assessments should not be articulated in the framework unless their value towards achieving the goals can be clearly explained. Standardised testing within the framework should not be used in an overly punitive manner, even if accountability is one of its purposes. Classroom assessment should require an array of assessment types that achieve both formative and summative purposes.
To develop its national assessment framework, Saudi Arabia should begin by looking at the principles of assessment processes that are found in the national curriculum framework. Contained within these principles are ideas around assessment planning, diversity of methods, providing feedback to students and measuring progress and data analysis. Saudi Arabia’s assessment framework should be built around these principles and would provide high‑level direction for other important reforms related to assessment, such as the development of NAP and revisions to the national examinations. An example of a national assessment framework from Hong Kong was provided in Box 4.3, and another from Norway is provided in Box 4.5.
Box 4.5. A national assessment framework in Norway
The Norwegian National Testing Framework was introduced in 2017 and is available on the website of the Directorate for Education and Training. It acts as the central reference point for all classroom and national assessment practices, thus ensuring that all assessment activities are oriented towards a common purpose without duplication of efforts.
Regarding classroom assessment, the framework places a strong emphasis on formative assessment. The framework specifies that the purpose of assessment is to provide schools with information about student learning that can be used by teachers to improve their practice.
Regarding national assessments, the framework requires national testing in English, reading and numeracy in years 5 and 8 and reading and numeracy in year 9. The National Testing Framework articulates the content and form of the tests and includes instructions and specifications for student sampling, as well as reporting and use of the test results. National assessments are developed by university experts based upon learning objectives as stated in national curricula. Most items are marked automatically. Open‑ended questions, however, are marked by teachers based upon external guidelines.
Source: Tveit (2018[41]), Ambitious and ambiguous: shifting purposes of national testing in the legitimation of assessment policies in Norway and Sweden (2000-2017), Assessment in Education: Principles, Policy & Practice, 25(3), 327-350. https://www.tandfonline.com/doi/pdf/10.1080/0969594X.2017.1421522?needAccess=true
Position ETEC in an overall co‑ordination role to facilitate communication between the various organisations responsible for assessment policy
Creating a national assessment framework is an important step towards achieving alignment between Saudi Arabia’s assessment practices and national education goals. An equally important step is facilitating the collaboration of institutions who play a role in assessment policy and practice such that the intent of the assessment framework is widely understood and all institutions are aware of their responsibilities.
It is recommended that ETEC be responsible for the overall co‑ordination of assessment policy in Saudi Arabia. ETEC is well placed to assume this role because it is already responsible for creating standardised assessments and developing classroom assessment tools and is the locus of most assessment expertise in the country. In this position, ETEC would not unilaterally determine assessment policy, but should work closely with MoE and Tatweer to map our clear roles and responsibilities to avoid duplicating efforts and make sure that all important tasks are assigned. One important area for communication is with Directorates, Education Offices and supervisors. As Saudi Arabia moves forward to stop the direct testing of students by supervisors and replace the stage assessments with the new standardised tools developed by ETEC, as this review also recommends (see below), the supervisors themselves will need to be aware of this change and understand how it affects their roles.
In the new teacher standards, set clear expectations for the assessment literacy of teachers at different levels of the professional pathways
Assessment literacy refers to teachers’ knowledge and use of effective assessment practice and is an essential pedagogical skill in contemporary education. Internationally, the Australian Professional Standards for Teachers prominently feature assessment literacy. In the domain “Professional Practice”, assessing students is a standalone standard that includes five focus areas. Expectations in these areas are then specified according to the different levels of the teaching profession. (Australian Institute for Teaching and School Leadership, 2011[42]).
In Saudi Arabia, the new teacher standards dedicate an entire standard to evaluating student performance and this standard is further disaggregated into five sub‑standards. A separate standard also contains a sub‑standard about creating assessments that are aligned with the aims of the curriculum. These are positive developments and clearly signal the importance of assessment literacy. However, what is missing and should be introduced are expectations of how teachers at different levels of the profession should demonstrate different levels of assessment literacy. An Expert Teacher would be expected to have a different level of assessment literacy than a Practitioner Teacher, but this distinction is not clear in the standards. Introducing this type of differentiation would signal the importance of continuously improving one’s assessment literacy.
Classroom assessment
Use teacher appraisal and school evaluation to promote the use of more diverse, especially formative, assessment practices
Improving classroom assessment is a critical need in Saudi Arabia and successfully reforming teacher practices will require co‑ordinated efforts in the areas of policy initiatives, resource provision and teacher training. Regarding policy, effectively designed evaluation and appraisal procedures can be powerful levers for driving changes in instructional practice. They can help to reinforce a better balance between summative and formative classroom assessment and encourage the use of more diverse assessment tools.
Saudi Arabia’s new school evaluation framework and envisioned changes to the teacher appraisal system will help align evaluation and appraisal with the curriculum’s aims regarding student assessment. For example, the new school standards contain a key standard about assessment, which will ensure that this important element of school performance is evaluated, and the new teacher standards require teachers to compile portfolios that demonstrate their effective use of a range of assessment instruments.
Given the habits that the supervision system has engrained, ETEC should consider also including clear statements in school evaluation and teacher appraisal documents of what will not happen. For example, principals and external evaluators will not test students or use progression through a textbook to determine teacher performance. Instead, documentation should explicitly require that evaluators look for evidence of positive practice, such as school‑level moderation of marking (described in greater detail below).
Create moderation programmes in order to improve teachers’ assessment judgements and make them more reliable
An initiative that would particularly useful in Saudi Arabia is to encourage assessment moderation programmes. There is widespread concern that the assessment judgements of teachers are unreliable and that assessments are not well designed to yield consistent results. Neither principals, supervisors nor parents trust that the marks given to students accurately represent those students’ levels of achievement. Most OECD countries have policies in place to encourage and support schools to introduce moderation activities to improve reliability.
This review recommends that Saudi Arabia use policy levers to require moderation. First, under the “management of learning process” key standard in the new school standards, an indicator should be developed that references organising moderation in schools and school principals should be held accountable for meeting this standard. Allocating time for moderation during the school day would also help support its implementation. Second, to build understanding around moderation, it is recommended that the national assessment framework specifically define moderation, explain what it is, why it is important to conduct and why it matters. Case studies of schools that perform moderation (these could be international at first and then feature Saudi Arabian schools) should accompany the description to help schools visualise moderation.
Study student testing time to help monitor and reduce the quantity of testing that students undergo
Saudi Arabia should reduce the amount of testing that occurs in Saudi Arabia be reduced in order to focus classroom time more on teaching and learning the curriculum. The policy levers to do this have been addressed elsewhere, such as creating an assessment framework that maps testing activities and eliminating principal and supervisor testing of students as part of teacher appraisal. While these policy initiatives will reduce testing, it is important to monitor if they achieve the desired change in a way that supports the more effective use of instructional time and a better balance between formative and summative approaches.
One of the challenges to monitoring the situation is a lack of clarity about how much testing students currently undergo. Therefore, it is recommended that MoE perform a baseline study to measure how much time a representative sample of students (i.e., according to grade level and Directorate) spends on taking tests in an academic year, and the different purposes and sources (central, local, school) of these different tests. This study can be repeated after the recommended reforms have been implemented to determine how much testing has reduced and if certain policies are more impactful than others. The policies can then be re-evaluated in light of this information.
Create diagnostic assessments and eliminate the stage assessments that are being administered by supervisors
Policy initiatives can help incentivise teachers to adopt better and more formative classroom assessment practices. However, teachers will need to be supported in their efforts through high‑quality assessment resources in order for policy aims to be embedded. In Saudi Arabia, a critical concern is the weak capacity of teachers to determine accurately their students’ levels of learning. Without such information, teachers are not sure where their students are with respect to the curriculum’s learning expectations and will not be able to adapt their instruction accordingly. To address this challenge, the OECD review team recommends that Saudi Arabia develop high-quality diagnostic assessments to aid teachers in understanding the abilities of their students.
Diagnostic assessments are a type of formative assessment, which often takes place at the beginning of a study unit in order to find a student’s starting point, or baseline, for learning and to develop a suitable learning programme for that student. Diagnostic assessments may also serve to identify students who are at risk of under achievement, uncover the reasons for their learning difficulties and create an appropriate intervention.
ETEC is already developing a set of assessments in relation to the new school evaluation framework that schools can either download or take online to assess how students are performing in relation to national standards. The OECD review team recommends that some of these tests should be designed with an explicit student‑level, diagnostic function. This would entail developing enough questions for the tests to measure individual student learning and potentially making some units voluntary so teachers can elect to diagnose only specific skills. These assessments could be designed first for core subjects at the start of each stage and eventually each grade in all subjects.
ETEC will not have to create the diagnostic assessment items from scratch because some suitable resources already exist. For example, the stage assessments that are being administered by supervisors can also be repurposed to be used by teachers. With classroom‑level diagnostic assessment helping teachers to understand student learning and a nationwide standardised assessment (see National standardised assessments) school‑level performance, there will no longer be any need for the stage assessments. They should be reduced and, eventually, eliminated.
It should be noted that these diagnostic assessments will not be standardised and comparable nationally. While the content of the tests is developed centrally, their administration and marking will be conducted locally. As the purpose of diagnostic exams is to help individual teachers understand their students’ level of learning, not to monitor nationwide performance, it will not be necessary to spend the resources to standardise these tests.
Develop materials to support the assessment of complex and higher‑order skills
In Saudi Arabia teachers are not familiar with how to assess the higher‑order thinking skills that are expressed in the curriculum standards and need materials to assist them in this area. To support teachers, Tatweer should prioritise the development of sample quizzes, worksheets and class projects that would help teachers determine student development of complex skills. Examples of assessment resources from other countries that test higher‑order skills can be found in Box 4.6.
Box 4.6. Assessment formats that measure broader competencies
In the Flemish Community of Belgium, a range of institutions including educational centres, academic institutes and umbrella organisations, have developed tools that teachers can draw on to assess non-cognitive aspects of learning, such as attitudes, well-being and involvement. Some of these tools are used quite widely while others are applied rather sporadically and in a limited number of schools. The most commonly used tool for assessment of non-cognitive performance is the SAM-scale (Scale for Attitude Measurement; Schaal voor Attitude Meting). The SAM-scale measures students’ attitudes and has been developed to assess, guide, stimulate and orientate students. By means of this tool a teacher can determine to what extent a pupil scores high or low for certain attitudes, e.g. flexibility, diligence and responsibility.
In Ontario (Canada), the document “Growing Success” provides a description of all the dimensions of evaluation in Ontario schools and includes the assessment of non-cognitive skills, called “Learning Skills and Work Habits”. All teachers from Grade 1 to Grade 12 are required to assess and report on these six non-cognitive skills. “Growing Success” includes a non-exhaustive list of sample behaviours for each skill to support teachers’ understanding and identification of each skill in the classroom. It also includes a 4-point rating scale (Excellent, Good, Satisfactory, and Needs Improvement) used for the evaluation of non-cognitive skills and a description of how this information should be reported on student report cards, which are standardised throughout the province.
In New Zealand primary schools, progress towards the achievement of national curriculum goals is measured via the National Education Monitoring Project (NEMP), which incorporates the assessment of competencies and values. Many of the NEMP assessment tasks are performance-based, requiring students to transfer learning to authentic close-to-real life situations. There are different assessment situations including one-to-one interviews, work stations and teamwork. Most assessment tasks are carried out orally and some are videotaped to allow for an in-depth analysis of student responses and interaction with teachers. While NEMP is designed for system monitoring, examples of previous assessment tasks are available for teachers and may be used in the classroom.
Sources: Flemish Ministry of Education and Training (2010[43]), OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes Country Background Report for the Flemish Community of Belgium. http://www.oecd.org/education/school/46974684.pdf .
Nusche et al. (2012[38]), OECD Reviews of Evaluation and Assessment in Education New Zealand, OECD Publishing. http://dx.doi.org/10.1787/9789264116917-en .
Merchant, Klinger & Love (1995[44]), Assessing and Reporting Non-Cognitive Skills: A Cross-Canada Survey. Canadian Journal of Educational Administration and Policy, 187, 2-17. https://journalhosting.ucalgary.ca/index.php/cjeap/article/view/43135/43931
Disseminate assessment resources through iEN and establish curation procedures to make the best resources easier to find
After high‑quality assessment resources are created, it is recommended that they be distributed through Tatweer’s iEN system. However, while easily accessible, iEN is not always easily navigable. Its item bank already has over 100 000 questions and this large quantity of items is intimidating to teachers who do not have time to sift through so many questions. Adding even more resources to iEN, even high quality ones, does not guarantee that those resources will be used if teachers cannot find them. iEN’s assessment resources need to be curated to make it easier for teachers to identify the most useful materials and for Tatweer to remove those of low quality.
A first step is to require more metadata be associated to iEN resources so they can be digitally sorted by users. Anyone who uploads materials should specify, for example, if the material is centrally created or by a user, what grade level the resources are intended for, which content area of the curriculum is assessed, which skills are assessed and what types of questions are contained (i.e., short answer, multi-stage or essay questions). This information must be inputted before the resource is available on iEN. Users can then more easily browse for materials that are relevant to them. Tatweer can also generate assessment maps that link materials on iEN directly to curriculum objectives and standards.
The next step is to manage more actively the quality of resources that are stored in iEN. This can be accomplished by a combination of education expert reviews and “crowd sourcing”, in which the materials are reviewed by the users themselves. Curators would be responsible for determining if the resources are well aligned with the curriculum framework and if they assess higher‑order skills. Users would share their experiences in using the resources and suggest what can be done to make the resources more pertinent in specific contexts.
Provide targeted pre‑ and in‑service development to enhance teachers’ assessment capacity
Resources will help guide teachers in using more effective assessment practices, but they will also need training in how to use and adapt those resources for their classroom contexts. Internationally, the higher education sector often plays an important role in strengthening teacher assessment literacy. Several universities offer programmes that specifically focus on student assessment. In the United Kingdom, University College London’s Institute of Education offers a Master’s degree in Educational Assessment (several Saudi Arabian students have completed this programme), a more general Master’s degree in Curriculum, Assessment and Pedagogy and in-service teacher training (University College London, n.d.[45]).
Saudi Arabia’s leading higher education institutions, such as King Saud University, should offer similar specialisations in student assessment. Not all graduates of such a programme would necessarily become teachers, but they could work for institutions like Tatweer and ETEC to develop useful assessment materials for teachers. Faculty of this programme could also contribute their expertise to Saudi Arabia’s initial teacher preparation programmes, which would improve teacher assessment literacy before candidates are posted to positions.
In‑service training in Saudi Arabia will also need to focus on improving teacher assessment literacy. Teachers need to understand concepts such as validity, reliability and test standards. They need to be trained to assess higher‑order thinking and not just rote memorisation of knowledge. To this end, NIPED should make the provision of this type of training (and accreditation of third parties that offer this kind of training) a national priority. Faculty of the national specialised programmes in student assessment would be well placed to help NIPED develop and deliver the training modules, and also to make graduate programmes more relevant to teacher classroom needs.
National standardised assessments
Extend the National Assessment Programme to be census‑based and consider several design elements in light of the national assessment framework
While the sample‑based, national testing that Saudi Arabia conducts through NAP provides important information on how the system as a whole is performing, it does not generate school‑ or student‑level results. In order to provide schools and teachers with reliable information on how well their students perform with respect to the curriculum’s expected learning outcomes, it is recommended that Saudi Arabia should administer census‑based assessments for at least two grades, one at the primary level and another at the secondary level. Given the need to focus on basic numeracy and literacy, it is suggested that the subjects tested include mathematics, reading and writing. Administering NAP as census‑based assessments would also eliminate the need for MoE to conduct its own census‑based testing.
As well as generating data for system monitoring purposes, the value of having census assessments would be the information provided to teachers and schools on student performance in relation to national expectations. At a time of curriculum reform, national student‑level assessments can be particularly effective at building understanding of new standards and how they can be measured. Furthermore, the information be used to identify which schools are in greatest need of additional support and determine instructional interventions and improvements. Through generating positive backwash effects, census‑based assessments can strengthen external school evaluation and accountability, helping to anchor qualitative assessments in consistent metrics.
Specific design elements to be considered when moving towards census‑based assessments include:
Access to data: School‑level results should be made available centrally and Directorates and Education Offices should have access to the results of all schools within their jurisdictions. Certain groups, such as relevant government agencies, educational researchers and universities, should also have access to the micro-data. Schools themselves should only have access to their own data, and to relevant benchmarks (e.g., national performance indicators). School‑level results should not be made public to avoid schools being ranked against each other.
National‑level results should be published and disaggregated by several dimensions, such as by Directorate, Education Office and gender. These indicators would be used to monitor the system and could form a composite indicator that appears as a goal in NTP (e.g., increase the number of Education Offices that perform above a certain threshold).
How data is used: Results should not be used to sanction teachers or schools. As discussed in Chapter 2, school data should be used to prioritise schools in need of support. Within the school evaluation framework, it would provide one measure of quality alongside others that ETEC is developing. Nationally, however, the data should be used more consequentially to hold Directorates accountable. Those with low outcomes should be closely monitored and provided with greater support to help their schools improve. Data should also be used deliberately in the development of curriculum materials. The assessment will reveal where teachers and students are struggling and this information should be used to direct the development of supporting instructional resources.
Box 4.7 describes the National Assessment Program from Australia, which could act as a model for Saudi Arabia’s.
Box 4.7. The National Assessment Program from Australia
Australia’s National Assessment Program (NAP) is a battery of two tests, both aligned with the Australian National Curriculum, that are administered nationally to collect information about student learning across the country. The two types of tests are:
The National Assessment Program – Literacy and Numeracy (NAPLAN), an annual, census-based assessment in literacy and numeracy that is administered to students in Years 3, 5, 7 and 9;
Three NAP sample-based assessments: science literacy, civics and citizenship and the information and communications technology, administered nationally to a sample of chosen students once every three years on a rolling basis.
Most items from these tests are multiple-choice, which makes them cost-effective to mark. However, the items were specifically designed to test students’ higher-order skills and not just factual recall. For example, the civics and citizenship sample tests cover not only content knowledge (e.g., vocabulary around government bodies), but also the cognitive processes that students need to understand and draw inferences from the content. This is accomplished by asking students to think about the type of social behaviours that would encourage and discourage participatory governance.
Test development, implementation and marking procedures are fully transparent. NAP’s website (https://www.nap.edu.au/) provides easily accessible assessment frameworks, technical reports, national reports and sample assessments. For both tests, national reports and associated data are provided on NAP’s website. National reports include a comparison of students’ results by jurisdiction, by demographic group, by location of the school and by gender.
For NAPLAN only, school level results are released on the My School website and reports on individual students’ results are provided to all students and parents. For the sample-based assessments, a basic report about the performance of their students is provided to each participating school.
Source: ACARA/Australian Curriculum/My school (n.d.[46]), National Assessment Program. https://www.nap.edu.au/
Australian Government (2018[47]), National Assessment Program - sample assessments. www.education.gov.au/national-assessment-program-sample-assessments
ETEC should be responsible for overseeing the assessment, but needs to receive adequate, sustained support
As the national testing agency, ETEC should be responsible for developing and overseeing the expanded NAP. Nevertheless, there are legitimate concerns about whether ETEC presently has the capacity to manage such an ambitious project. A previous census‑based assessment programme was abruptly halted due to a lack of sustained funding. Therefore, the OECD review team recommends that the budget for the expanded NAP be guaranteed so ETEC feels supported in devoting resources to developing and administering it. Including results from this assessment in NTP and building accountability measures around local‑level indicators from the assessment would help establish it as the authoritative source of performance data, which would help guarantee its funding.
MoE should assist ETEC with the administration of the expanded NAP because of their well‑established relationships with schools. This would entail overseeing all logistical matters (which will be considerable as the assessment will have to be administered in paper‑and‑pencil format) such as transportation, proctoring, marking and data input. It will be the responsibility of ETEC to train MoE staff (e.g., supervisors and Directorate/Education Office staff) to perform these tasks, but MoE must assure its support. Furthermore, MoE should show its support for the centrality of this assessment by ceasing its own external assessment activities, such as those administered by supervisors and MoE departments.
Align the assessment with the curriculum and use reporting from the assessment to build understanding of the curriculum standards
A national assessment is a powerful tool for not only measuring performance, but improving understanding of key education resources. An assessment that is aligned with the curriculum standards, for example, helps educators understand what the standards are and their critical significance to teaching and learning. For these reasons, the majority of OECD countries have central assessments that are aligned with national curriculum goals and/or standards (OECD, 2013[8])
Given the need to improve teachers’ assessment literacy, the expanded NAP should be aligned with the new curriculum standards to help teachers better understand the curriculum standards. Teachers will be able to see the types of questions that NAP asks students and what types of skills they are intended to measure. This will help teachers differentiate between teaching and assessing surface‑level knowledge versus a deep mastery of skills.
As ETEC is already developing a teachers’ report based on NAP results, the OECD review team recommends that, in addition to reporting relevant assessment results, this report should also contain formative information that helps teachers understand the standards. An example of this type of report are report cards from the National Assessment of Educational Progress (NAEP) in the United States. They include a description what students are expected to know and be able to do at three different achievement levels across subjects. The report cards also include informative digital links that contain further detailed descriptions of the standards and achievement levels (The Nation’s Report Card, n.d.[48]).
Examinations
Set, as a medium-term (five years) goal, the development of an examination that is aligned with the curriculum and certifies completion from upper secondary school and selects students for entrance into tertiary education
Saudi Arabia’s examinations system faces several challenges. There does not exist a school‑leaving examination that is aligned with the curriculum, which prevents the system from exerting positive pressure on teachers to implement the intended curriculum. The two tertiary‑entrance examinations, the GAT and the SAAT, are not fully aligned with the curriculum and are generally perceived as easy with low‑stakes. They thus do not act as a significant motivating factor for students.
The development of a new curriculum framework represents an opportunity for Saudi Arabia to revisit its examinations system and make it more fit‑for‑purpose. The new system should encourage students to apply themselves and reinforce the aims of the curriculum by assessing higher‑order thinking skills, especially in core learning areas such as Arabic and mathematics. To achieve these goals, the OECD review team recommends that Saudi Arabia set a goal to create a single, curriculum‑aligned examination that would both certify completion from upper secondary school and select students for entrance into higher education institutions. Once introduced, this examination would incorporate many of the elements that are currently in the GAT and SAAT, which would eliminate the need for the GAT and SAAT as they are currently configured.
Establishing an examination that certifies graduation from upper secondary school would motivate students to apply themselves
The OECD review team understands that a previous upper secondary certification examination in Saudi Arabia was eliminated in order to increase completion. Currently, completion rates have increased and now a greater concern is that schooling lacks rigour and students need to be motivated to improve their learning. The GAT and SAAT are not fulfilling this purpose as strongly as they could because entering tertiary education is not overly difficult and therefore not all students need to prepare for these exams. Creating a certification examination would introduce much needed stakes into upper secondary school, which would incentivise students to focus on their studies so they can pass the exam.
Aligning the examination with the curriculum would support teachers in implementing the curriculum in their classrooms
Similar to aligning national assessments with the curriculum, aligning examinations with the curriculum helps teachers better understand national learning standards and reinforces the implementation of the curriculum in the classroom (OECD, 2013[8]). In Saudi Arabia, neither the GAT nor the SAAT are aligned with the curriculum. This can create conflicting messages for teachers who feel responsible for not only teaching the curriculum, but helping their students enter higher education institutions.
To further support classroom instruction and implementation of the curriculum, the OECD review team recommends that the proposed examination be aligned with the curriculum. In particular, the types of items that appear in the examination need to be able to assess the higher‑order thinking skills that are expressed in the curriculum standards. The vast majority of certification examinations employ a combination of closed-ended (multiple‑choice and single response) and open‑ended item types, which are generally more capable of assessing complex skills. Teachers will already be required by the curriculum and the teacher standards to develop these skills in students and use proper assessment methods to evaluate them. Assessing these skills on a national examination would reinforce teachers’ efforts and, through teachers’ viewing the examination items, help them improve their own assessment capacity.
Requiring a limited number of subjects while making others voluntary balances the need to assess the important basic skills with the requirements of individual tertiary programmes
It is important that examinations balance the need to assess the most important skills while not becoming too burdensome for test takers. The OECD review team generally supports examinations that include a limited number of core compulsory subjects, usually including mathematics and the language of instruction, and a significant elective component. Some subjects, in particular mathematics, should have differentiated versions depending upon student interest and need (e.g., functional mathematics for students who will not be engaging further in technical subjects and advanced mathematics for those who intend to pursue technical fields). In 30 OECD education systems that have entrance examinations to higher education institutions, those examinations include at least one compulsory examination subject and a range of optional subjects (OECD, 2015[49]; OECD, 2013[8]).
For Saudi Arabia’s examination, the OECD review team recommends that Arabic and mathematics be mandatory subjects and potentially a foreign language, such as English. Mathematics should also be divided into basic and advanced versions. Students would take one or the other depending upon interests and the requirements of the university programmes they wish to enter. In addition to the mandatory subjects, students would be required to take a small number of voluntary subjects (e.g., in science, social or Islamic studies), again depending upon their interests and programme requirements. This elective dimension would also align with current reforms that give upper secondary students more choice over their subjects of study.
The advantage of this structure is that it focuses student testing on those domains that matter most for a student’s future. It also reduces the amount of oversight and development required from ETEC, enabling staff to concentrate on developing fewer, but higher quality items. The current GAT and SAAT are already showing item‑level errors, and this is likely related to the fact that ETEC currently oversees over 25 assessments and examinations with constrained capacity. Decreasing the number of items that ETEC needs to develop through limiting the subjects tested and through the development of a single examination would help to improve the quality of examination items.
A single examination model in the Republic of North Macedonia is described in Box 4.8. It has been highlighted as it represents a good example of a relatively newly introduced examination, which combines the strong features of more mature systems, without some of the legacy constraints of older models. An additional feature relevant to Saudi Arabia is the inclusion of a project assignment, which would can help to support the emphasis the curriculum gives to applied learning.
Box 4.8. Single examinations model in North Macedonia
In North Macedonia, the State Matura examination, which has been a model for other countries’ examination systems, certifies students as having completed upper secondary school and selects them for entrance into universities. Students must take their native language, mathematics or a foreign language and must choose from a list of electives for the remaining subjects. The State Matura also includes a project assignment, which allows students to demonstrate a broader range of competencies than they could via a written examination. The State Matura can be taken by students from both gymnasium (general education) schools and vocational schools. The table below summarises the design features of the State Matura.
In the meantime, conduct a thorough and cohesive review of the GAT and SAAT tests, involving a large range of assessment experts
The proposed changes to the examinations systems cannot be made quickly, which is why the OECD review team suggests that they be set as a medium-term goal. While preparations are being made for the new examination system, the OECD review team recommends that ETEC conduct a thorough review of the current examinations, including their construction and the development of their items.
To conduct the review in accordance with international practices, ETEC should appoint a chief examiner who reviews the examinations and writes a draft paper about his/her findings. Another group of examiners would meet to discuss the draft paper and modify the findings. This dual‑stage process helps identify and resolve internal errors found in the exams. After the items have been modified, the examiners should take the tests as if they were students, which would help them identify mistakes that may have been missed during the first round of review. Once the exams have been taken by students, ETEC should proceed with its quantitative tests of item functioning to determine if the items behaved as expected. Any inconsistencies should prompt a review of the items that produced them and, if necessary, dismissal of the items (Assessment and Qualifications Alliance, n.d.[51]; Cambridge Assessment, 2017[52]).
References
[46] ACARA/Australian Curriculum/My school (n.d.), National Assessment Program, https://www.nap.edu.au/ (accessed on 9 May 2019).
[21] Ahmed, A. and A. Pollitt (2008), Outcome Space Control and Assessment.
[7] Albedaiwi, S. (2014), EFL Materials in public school classrooms in Saudi Arabia: an investigation of the extent to which teachers engage in, University of Glasgow.
[6] Alhareth, Y. (2014), “The Assessment Process of Pupils’ Learning in Saudi Education System: A Literature Review”, American Journal of Educational Research, Vol. 2/10, pp. 883-891.
[9] Almuntasheri, S. (2016), “Saudi teachers’ practices of formative assessment: a qualitative study”, Problems of education in the 21st century, Vol. 74/6, http://www.scientiasocialis.lt/pec/files/pdf/vol74/6-15.Almuntasheri_Vol.74_PEC.pdf (accessed on 7 May 2019).
[3] Alnefaie, S. (2016), “Teachers’ role in the development of EFL curriculum in Saudi Arabia: The marginalised status”, Cogent Education, Vol. 3.
[51] Assessment and Qualifications Alliance (n.d.), How a question paper is created, https://www.aqa.org.uk/about-us/what-we-do/getting-the-right-result/how-exams-work/making-an-exam-a-guide-to-creating-a-question-paper.
[47] Australian Government (2018), National Assessment Program - sample assessments, https://www.education.gov.au/national-assessment-program-sample-assessments (accessed on 13 May 2019).
[42] Australian Institute for Teaching and School Leadership (2011), Australian Professional Standards for Teachers, https://www.aitsl.edu.au/docs/default-source/teach-documents/australian-professional-standards-for-teachers.pdf (accessed on 7 May 2019).
[11] Bishop, J. (2006), “Drinking from the Fountain of Knowledge: Student Incentive to Study and Learn – Externalities, Information Problems and Peer Pressure”, Handbook of the Economics of Education, Vol. 2, pp. 909-944, http://dx.doi.org/10.1016/S1574-0692(06)02015-0.
[28] Bishop, J. (1999), “Are National Exit Examinations Important for Educational Efficiency?”, Articles and Chapters, https://digitalcommons.ilr.cornell.edu/articles/23 (accessed on 7 May 2019).
[23] Black, P. and D. Wiliam (1998), “Assessment and Classroom Learning”, Assessment in Education: Principles, Policy & Practice, Vol. 5/1, pp. 7-74, http://dx.doi.org/10.1080/0969595980050102.
[25] Box, C., G. Skoog and J. Dabbs (2015), “A Case Study of Teacher Personal Practice Assessment Theories and Complexities of Implementing Formative Assessment”, American Educational Research Journal, Vol. 52/5, pp. 956-983, http://dx.doi.org/10.3102/0002831215587754.
[52] Cambridge Assessment (2017), The Cambridge Approach to Assessment, Cambridge University, https://www.cambridgeassessment.org.uk/Images/cambridge-approach-to-assessment.pdf (accessed on 7 May 2019).
[36] Castro Superfine, A., A. Marshall and C. Kelso (2015), “Fidelity of implementation: Bringing written curriculum materials into the equation”, Curriculum Journal, Vol. 26/1, pp. 164-191.
[22] Crisp, V., M. Johnson and F. Constantinou (2018), “A question of quality: Conceptualisations of quality in the context of educational test questions”, Research in Education, p. 003452371775220, http://dx.doi.org/10.1177/0034523717752203.
[55] DET (2017), Rammeverk for nasjonale prøver [The national testing framework], Oslo: Directorate for Education and Training, https://www.udir.no/eksamen-og-prover/prover/rammeverk-for-nasjonale-prover/hva-er-nasjonale-prover/#formal (accessed on 9 May 2019).
[54] Downing, S. (2006), “Twelve Steps for Effective Test Development”, in Haladyna, T. and S. Downing (eds.), Handbook of Test Development, Lawrence Erlbaum Associates, Inc.
[43] Flemish Ministry of Education and Training and the University of Antwerp Edubron Research Group (2010), OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes Country Background Report for the Flemish Community of Belgium, http://www.oecd.org/edu/evaluationpolicy (accessed on 22 May 2019).
[13] Fullan, A. (ed.) (2009), From system effectiveness to system improvement: reform paradigms and relationships, Solution Tree.
[12] Fullan, M. (2007), The new meaning of educational change (fourth edition). New York: Routledge., Routledge.
[53] Haladyna, T. and S. Downing (eds.) (2006), Twelve Steps for Effective Test Development, Lawrence Erlbaum Associates, Inc.
[20] HKEAA (2019), Website of the Hong Kong Examinations and Assessment Authority, http://www.hkeaa.edu.hk/en/ (accessed on 13 May 2019).
[14] Honeyman, C. (ed.) (2016), Curriculum and expected learning outcomes, International Institute for Educational Planning. learningportal.iiep.unesco.org.
[10] IEA (2019), TIMSS and PIRLS International Study center Website, https://timssandpirls.bc.edu/ (accessed on 20 May 2019).
[31] International Bureau of Education (2013), “Training tools for curriculum development: a resource pack; 2014”, http://www.ibe.unesco.org/fileadmin/user_upload/Publications/Training_tools/IBE-CRP-2014_eng.pdf.
[16] Isaacs, T. (unpublished), FUTURE OF EDUCATION AND SKILLS: THE OECD EDUCATION 2030 POLICY QUESTIONNAIRE ON CURRICULUM REDESIGN: Literature Review on Planning for Implementation, OECD.
[19] Isaacs, T. and B. Creese (2014), Aligned Instructional Systems: Cross-Jurisdiction Benchmarking Report, National Center on Education and the Economy.
[35] Isaacs, T. et al. (2010), Aligned Instructional Systems: Singapore, National Center on Education and the Economy, http://www.ncee.org/wp-content/uploads/2015/09/Singapore-Jurisdiction-Report.pdf (accessed on 7 May 2019).
[5] John Wiley & Sons (ed.) (2017), Empowered educators : how high-performing systems shape teaching quality around the world, https://www.wiley.com/en-us/Empowered+Educators%3A+How+High+Performing+Systems+Shape+Teaching+Quality+Around+the+World-p-9781119369608 (accessed on 12 April 2019).
[17] Kanjee, A. and Y. Sayed (2013), “Assessment policy in post-apartheid South Africa: challenges for improving education quality and learning”, Assessment in Education: Principles, Policy and Practice, Vol. 20/4, pp. 442-469, http://dx.doi.org/10.1080/0969594X.2013.838541.
[4] Kingdom of Saudi Arabia (2018), Human Capability Development Program - SMC Pulse Check Presentation.
[56] Kingdom of Saudi Arabia (n.d.), Saudi Vision 2030, https://vision2030.gov.sa/download/file/fid/417.
[40] Lott, J. and J. Luebeck (eds.) (2015), Textbook-Specific Professional Development: Impacting Teachers’ Knowledge and Views., Association of Mathematics Teacher Education.
[44] Merchant, S., D. Klinger and A. Love (1995), “Assessing and Reporting Non-Cognitive Skills: A Cross-Canada Survey”, Canadian Journal of Educational Administration and Policy, Vol. 187, pp. 2-17, https://journalhosting.ucalgary.ca/index.php/cjeap/article/view/43135/43931 (accessed on 22 May 2019).
[33] MEXT (n.d.), Improvement of Academic Abilities （Courses of Study）, http://www.mext.go.jp/en/policy/education/elsec/title02/detail02/1373859.htm (accessed on 10 May 2019).
[15] Ministry of Education - World Bank (2017), Component 3: 21st Century Competencies in Teaching and Learning Practices: A Study on 21st Century Skills and Education in the KSA.
[50] National Examination Centre (n.d.), State matura, http://dic.edu.mk/%D0%B4%D1%80%D0%B6%D0%B0%D0%B2%D0%BD%D0%B0-%D0%BC%D0%B0%D1%82%D1%83%D1%80%D0%B0/?lang=en.
[38] Nusche, D. et al. (2012), OECD Reviews of Evaluation and Assessment in Education New Zealand, OECD Publishing, https://www.oecd-ilibrary.org/docserver/9789264116917-en.pdf?expires=1558519044&id=id&accname=ocid84004878&checksum=E1F85BEDC5E0CA10BCC05804E50543CD (accessed on 22 May 2019).
[59] OECD (2019), PISA 2018 Database, https://www.oecd.org/pisa/data/2018database/ (accessed on 7 January 2020).
[49] OECD (2015), Education at a Glance 2015: OECD Indicators, OECD Publishing, Paris, https://dx.doi.org/10.1787/eag-2015-en.
[1] OECD (2013), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, http://www.oecd.org/edu/school/synergies-for-better-learning.htm.
[8] OECD (2013), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264190658-en.
[26] OECD (2005), Formative Assessment Improving learning in secondary classrooms, OECD Publishing, Paris, http://www.SourceOECD.org, (accessed on 22 May 2019).
[37] Persson, A. (2016), “Curriculum and expected learning outcomes”, in Honeyman, C. (ed.), Textbooks and teachers’ guides. 25 Ways to Improve Learning, International Institute for Educational Planning.
[24] Pope, S. (ed.) (2014), A New Assessment Approach in Mathematics Classrooms in Saudi Arabia.
[30] Roderick, M. and M. Engel (2001), “The Grasshopper and the Ant: Motivational Responses of Low-Achieving Students to High-Stakes Testing”, Educational Evaluation and Policy Analysis, Vol. 23/3, pp. 197-227, http://dx.doi.org/10.3102/01623737023003197.
[34] Sargent, C. et al. (2010), Thematic Probe Curriculum review in the INCA countries: June 2010, NFER, https://webarchive.nationalarchives.gov.uk/20130220111913/http:/www.inca.org.uk/Curriculum_review_probe_final_01_dec_2010.pdf (accessed on 7 May 2019).
[18] Schmidt, M. (2017), “The Perils of Outcomes-Based Education in Fostering South African Educational Transformation”, Open Journal of Political Science, Vol. 7, pp. 368-379, http://dx.doi.org/10.4236/ojps.2017.73030.
[32] Solini, T., K. Pyhalto and J. Pietarinen (2017), Does it matter how the change is done? National curriculum reform in Finland | BERA, https://www.bera.ac.uk/blog/does-it-matter-how-the-change-is-done-national-curriculum-reform-in-finland (accessed on 7 May 2019).
[29] Sukyadi, D. and R. Mardiani (2011), “The Washback Effect of the English National Examination (ENE) on English Teachers’ Classroom Teaching and Students’ Learning”, k@ta, Vol. 13/1, pp. 96-111, http://dx.doi.org/10.9744/kata.13.1.96-111.
[27] Tamassia, C. and R. Adams (2009), ““International assessments and indicators – How will assessments and performance indicators improve educational policies and practices in a globalised society?”, in Ryan, K. (ed.), The SAGE International Handbook of Educational Evaluation, SAGE Publications, Thousand Oaks, California.
[39] Teaching Council New Zealand (n.d.), Appraisal, https://teachingcouncil.nz/content/appraisal (accessed on 23 May 2019).
[48] The Nation’s Report Card (n.d.), NAEP Mathematics Report Card, https://www.nationsreportcard.gov/math_2017?grade=8 (accessed on 24 May 2019).
[41] Tveit, S. (2018), “Ambitious and ambiguous: shifting purposes of national testing in the legitimation of assessment policies in Norway and Sweden (2000-2017)”, Assessment in Education: Principles, Policy & Practice,, Vol. 25/3, pp. 327-350, http://dx.doi.org/10.1080/0969594X.2017.1421522.
[58] UNESCO Institute for Statistics (2020), Education database, http://data.uis.unesco.org/.
[45] University College London (n.d.), Short courses and CPD | Institute of Education, https://www.ucl.ac.uk/ioe/courses/short-courses-and-cpd (accessed on 7 May 2019).
[57] World Bank & Education Evaluation Commission (2016), Measuring the State of Education (Unpublished).
[2] World Bank and ETEC (2016), Measuring the State of Education (Unpublished).
Annex 4.A. Review of selected GAT and SAAT items
GAT
While the GAT claims to require deductive, logical and critical thinking skills, the review team found that most questions could be answered by memorisation. The reading section assesses lower‑level skills, meaning that most of the answers can be derived from simply reading the text, rather than inferring from it.
The “odd one out section,” in which students must identify which word does not belong in a group, presented two difficulties. Question 7’s group of words is golf, billiards, cricket and volleyball. The correct answer is billiards, either because billiards is the only indoor game or because in billiards several balls are in play simultaneously while only one ball is in play at a time in the other three sports. However, to answer this question correctly, test takers must know the rules of some fairly specific activities. The item might also be culturally inappropriate and could discriminates against girls, who are less likely to know the details of each sport.
Question 8’s group of words is four, book, silk and milk. According to the answer key, the correct response is silk. It is unclear why this is. The review team thought the most correct response was four, either because the rest are tangible items and “four” is abstract, or because the rest contained the letter “k.”
Another set of questions highlights four words in a passage and asks students to identify which words is used incorrectly. The answer key for Question 11 in this section is incorrect. The answer should be A (pusillanimity), not C (chastised). Question 15 has two responses that could be considered correct – B (perpetual, which should be permanent) and C (eradicable, which should be ineradicable), which is the identified as correct in the answer key.
The quantitative section of the GAT tests arithmetic, algebra, geometry and interpretation of graphs and tables. This set of assessed competencies is slightly less challenging than examinations internationally. For example, the General Certificate of Secondary Education in England, which is taken by over 90% of 16 year old students, assesses algebra, ratios, proportion and rates of change, geometry and measures (including sine, cosine and tangent) and probability and statistics.
Specific questions also raised concerns. For example, question 14 asks students to determine which column of a column chart is higher, but two columns are of such similar height that it is difficult to ascertain which is higher. Question 20 asks students to compare two values, both of which are the same. The correct response should be “the two values are equal.” However, the answer key states that the correct response is the first value is greater than the second value.
SAAT
The sample SAAT was expectedly more difficult than the GAT contained questions that were more appropriate for Grade 12 students (the test covers content from Grades 10, 11 and 12). However, answering the questions for the most part relied on memorisation of facts rather than critical thinking skills. One biology question, for instance, simply tested students’ ability to memorise “which worm belongs to phylum annelida”.
There are also inconsistencies regarding SAAT questions. Question 7 for the biology question uses the phrase ‘dormant A blood type’, which is not possible. Given the answer choice, presumably the item writer meant homozygous (i.e. AA). Chemistry question 2 has two distracters that rely on the test taker knowing how to spell “molar”, rather than knowledge related to chemistry. Chemistry question 8 is formatted in a confusing way because one of the choices precedes the end of the question. Furthermore, the first of the answer choices does not exist.