Olivia Cortellini
Council for Aid to Education, United States
Tess Dawber
Council for Aid to Education, United States
Does Higher Education Teach Students to Think Critically?
5. General CLA+ International results
Abstract
The CLA+ International Database comprises data from studies conducted by the Italian National Agency for the Evaluation of Universities and Research Institutes (ANVUR) covering 23 campuses of exiting students from the University of Guadalajara system; entering and exiting students from a consortium of four “post-1992” (Hannah, 1996[1]) institutions and a public university in the United Kingdom; several individual institutions across Chile; and entering and exiting students from 18 universities and universities of applied sciences in a study sponsored by the Ministry of Culture and Education of Finland. The dataset also includes all of the entering and exiting students in the United States between fall 2015 and spring 2020. This section presents the results from the aggregated database of all student participants. Some analyses separate students by those from, and those outside, the United States due to disparity in sample size. Part III provides information on each participating country or region.
Introduction
Chapter 5 addresses the distribution of scores and mastery levels across the student population. Before discussing the scores and mastery levels, information about the data are provided. Table 5.1 shows the number of students for each country by year and administration. As you can see, the United States has data for all the administrations from 2015 to 2020 and has the largest total sample by far. All countries except Italy have participated in multiple administrations of the Collegiate Learning Assessment (CLA+).
Table 5.1. CLA+ samples by country and administration
Table 5.2 shows the number of students for each country by entering (Year 1) and exiting (Year 4) university status. Overall, 52% of the sample are entering students, and 48% of the sample are exiting students. However, the entering and exiting percentages varied greatly by country. There are more entering than exiting students for each country, except for Italy, which only tested exiting students. The countries with a high percentage of entering students in the sample are the United Kingdom and Chile (93 and 81, respectively). In contrast, 52% of the U.S. sample are entering students.
Table 5.2. Sample by country and year of study
Because the countries have differing numbers of participants, the summary information is based on equal weighting of the countries, so the results are not heavily influenced by countries with higher student counts. That is, the mean scores presented are the average of the six country means. We assume that the population variances are equal across countries, so the standard deviations presented are the average of the country standard deviation values.
Mastery levels
Entering students in the combined international dataset received, on average, a total CLA+ score of 1 086 (SD = 134; see Table 5.3), which corresponds with the Developing mastery level. Exiting students, on average, received a total CLA+ score of 1097 (SD = 138), which corresponds with the Proficient mastery level. Although the effect size is small (d = .10), it is pertinent to note that the average score for exiting students passed the criterion-referenced threshold for “proficient” performance, whereas the entering students’ average total score did not. Furthermore, it is important to acknowledge that the entering and exiting student samples differed across countries, given that institutions in different countries tested different student populations. Nevertheless, it is encouraging to see the improvement in average mastery level. Further results broken down by section score are reported in Table 5.3. Given widely varying participation rates across countries, Table 5.3 reports the grand mean computed from each country mean and the mean standard deviation rather than basing the results on all students in the dataset to prevent the large U.S. sample from skewing the results.
Table 5.3. Average total CLA+ scores and section scores, by class
Note: Standard deviations are listed below means, in parentheses.
Using the integrated international database, results are presented for the proportion of students classified into the five CLA+ mastery levels for all students (Table 5.4) and for all students by class (Table 5.5). Table 5.4 shows the average percentage of students within the countries at each performance level. The minimum and maximum values represent individual country percentages. For example, the average percentage of students classified as Emerging was about 21% across the six countries. To show the range, one country had 13% of students (Minimum column) and one country had 42% of students (Maximum column) classified as Emerging. The results highlight the variability of student performance across countries.
Table 5.4. Mastery level distribution
Distribution of mastery levels was somewhat similar between class levels, as indicated in Table 5.5. Like the results in Table 5.4, the average for entering and exiting students is the average of the country percentages rather than the average across students. Overall, a slightly lower percentage of exiting students performed below the Proficient mastery level compared to entering students, and a slightly higher percentage of exiting students performed above the Proficient mastery level compared to entering students. At both class levels, student scores were clustered around the Developing and Proficient mastery levels.
Table 5.5. Mastery level, by class
CLA+ total scores
Overall, total CLA+ score distributions were similar for entering and exiting students. The distribution of CLA+ scale scores for entering students and exiting students are presented in Figure 5.1 and Figure 5.2, respectively. The exiting student distribution shifted slightly to the right compared to the entering student distribution, suggesting some improvement in the CLA+ total scores for the exiting university students.
Figure 5.1. CLA+ total score distribution, entering students
Figure 5.2. CLA+ total score distribution, exiting students
CLA+ subscores
In addition to receiving total and section scores, students who complete CLA+ also receive subscores for each section. For the Performance Task (PT), students receive subscores for Analysis and Problem Solving (APS), Writing Effectiveness (WE) and Writing Mechanics (WM). PT subscores range from 1-6 points each. Among entering students, the average subscores for APS, WE and WM, respectively, were 3.3 (SD = 0.8), 3.2 (SD = 0.8) and 3.6 (SD = 0.7). Among exiting students, subscores were slightly higher and slightly more variable. The average for APS was 3.4 (SD = 0.8), the average for WE was 3.4 (SD = 0.8) and the average for WM was 3.7 (SD = 0.8). These findings, illustrated in Figure 5.3, show that exiting students obtained slightly higher PT subscores.
Figure 5.3. Performance Task subscores among entering and exiting students
Note: PT = Performance Task; APS = Analysis and Problem Solving; WE = Writing Effectiveness; WM = Writing Mechanics
Like the PT, the Selected-Response Question (SRQ) section is composed of three subscores. The subscores are Scientific and Quantitative Reasoning (SQR), Critical Reading and Evaluation (CRE) and Critique an Argument (CA). For both the SQR and CRE subscores, exiting students outperformed entering students. Entering students received an average score of 521 (SD = 101) on the SQR section, compared to exiting students who scored on average 532 (SD = 103). In the CRE section, entering students received an average score of 507 (SD = 100), whereas exiting students received an average score of 513 (SD = 100). On the CA section, entering students, with an average score of 522 (SD = 98), slightly outperformed exiting students, who scored on average 512 (SD = 102). Results are illustrated in Figure 5.4. Although the results are not as consistent as the PT subscore results, the exiting students obtained slightly higher scores on two of the three subscores compared to the entering students.
Figure 5.4. Selected-Response Question subscores among entering and exiting students
Note: SRQ = Selected-Response Question; SQR = Scientific and Quantitative Reasoning; CRE = Critical Reading and Evaluation; CA = Critique an Argument
Student self-reported effort and engagement
After students complete the CLA+, they typically receive a questionnaire in which they report the amount of effort they spent on each section of the assessment as well as how engaging they found each section of the assessment was. Both effort and engagement are reported on 5-point Likert scales, with higher values indicating higher levels of effort and engagement. On the PT, the average effort rating given by entering students was 3.6 (SD = 0.9), and the average rating given by exiting students was 3.7 (SD = 0.9). For the SRQ section, entering students reported an average of 3.1 points on the effort scale (SD = 0.9), and exiting students reported an average of 3.3 (SD = 0.9.) Table 5.6 summarises the distribution of self-reported effort ratings by class and section. Important to note is that these data are not available for all students because not every country included this survey at the end of the assessment.
Table 5.6. Students’ self-reported effort on each CLA+ section
Note: PT = Performance Task; SRQs = Selected-Response Questions
As shown in Table 5.7, for each increase in self-reported effort, there was an increase in the average score on the applicable section. In other words, students who reported making more effort on the PT generally received higher PT scores than those who reported less effort, and students who reported higher levels of effort on the SRQ section tended to score higher on that section than did their peers who reported less effort. One exception to this is that SRQ scores did not increase for students who reported making “[their] best effort” compared to students who reported “a lot of effort”. This pattern held for both entering and exiting students. The relationship between CLA+ section score and the amount of effort expended is illustrated in Figure 5.5. The PT results for the exiting students showed a different trend line compared to the other results. A steeper slope was observed from “a little effort” to “a lot of effort” and “best effort”, demonstrating a more dramatic increase in PT scores for the highest two effort ratings.
Table 5.7. Average CLA+ section score by self-reported effort
Note: Standard deviations are listed below means, in parentheses. PT = Performance Task; SRQs = Selected-Response Questions
Figure 5.5. Average CLA+ section score by amount of effort
Note: PT = Performance Task; SRQs = Selected-Response Questions
Compared to self-reported effort, there was more variability between the PT and the SRQ section in self-reported engagement. The average rating that entering students reported for their engagement with the PT was 3.1 (SD = 1.0), and that given by exiting students was 3.2 (SD = 1.0). However, entering students reported an average engagement level of 2.6 (SD = 1.0) for the SRQ section, and exiting students reported an average of 2.8 (SD = 1.1). Distributions are summarised in Table 5.8.
Table 5.8. Students’ self-reported engagement on each CLA+ section
Note: PT = Performance Task: SRQs = Selected-Response Questions
There was a similar relationship between engagement and CLA+ section score to the previously described relationship between effort and section score. As with effort, students who reported higher levels of engagement on a section tended to receive higher scores on that section than their peers who reported less engagement. Also similar to the previous findings on effort and section score, the pattern did not hold for students who reported being “very engaged” to “extremely engaged” with the SRQ section (see Table 5.9). The mean SRQ scores tended to plateau and decrease for the higher engagement ratings whereas the mean PT scores continued to rise across the range of engagement ratings. The relationship between CLA+ section score and level of engagement is illustrated in Table 5.9.
Table 5.9. Average CLA+ section score by self-reported engagement
Note: Standard deviations are listed below means, in parentheses. PT = Performance Task; SRQs = Selected-Response Questions
Figure 5.6. Average CLA+ section score by level of engagement
Note: PT = Performance Task; SRQs = Selected-Response Questions
Summary
Chapter 5 addressed the distribution of scores and mastery levels across the student population. Because the countries have differing numbers of participants, the summary information was based on equal weighting of the countries, so the results are not heavily influenced by countries with higher student counts.
Overall, the exiting students performed better than their entering peers on average. This is consistent across all countries in the sample and offers some evidence that higher education contributes to the improvement of students’ generic skills. However, the learning gains are not large and there is room for improvement of these skills globally.
Based on mean scores, entering students performed at the Developing mastery level and exiting students performed at the Proficient mastery level. Subscore results showed slightly higher scores for exiting students compared to entering students. The relationship between CLA+ scores and self-reported effort/engagement on each section was examined. Generally, for each rating increase in self-reported effort/engagement, there was an increase in the average score on the applicable section. It is encouraging to see the improvement in scores and to see the linear relationship between effort/engagement and test results.
Reference
[1] Hannah, S. (1996), “The higher education act of 1992: Skills, constraints, and the politics of higher education”, Journal of Higher Education, Vol. 67/5, pp. 498-527, https://doi.org/10.1080/00221546.1996.11780274.