Olivia Cortellini
Council for Aid to Education, United States
Tess Dawber
Council for Aid to Education, United States
Does Higher Education Teach Students to Think Critically?
6. CLA+ International demographic variables
Abstract
This chapter explores relationships between demographic variables and Collegiate Learning Assessment (CLA+) performance. The demographic variables of interest in this dataset are primary language, gender, and parental education level. To allow for inferential analyses, overall means are reported in this chapter rather than grand means of country means. However, due to the large sample from the United States compared to the sample sizes of other participating countries, results for the U.S. domestic data are reported separately from results from the rest of the combined international data.
Primary language
After completing CLA+, most students responded to a series of demographic survey questions. In one question, students were asked to identify whether their primary language was the same as the language of instruction at their institution, or whether their primary language was different from the language of instruction at their university. Tables 6.1-6.4 summarise average total CLA+ score as well as CLA+ section scores broken down by primary language, class level and sample.
Table 6.1. CLA+ score by primary language, entering students: International sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Table 6.2. CLA+ score by primary language, entering students: U.S. sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Table 6.3. CLA+ score by primary language, exiting students: International sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Table 6.4. CLA+ score by primary language, exiting students: U.S. sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Independent samples t-tests were used to determine whether any differences in CLA+ scores between primary language groups were significant (see Tables 6.5-6.7). Most t-tests yielded significant results, except for the comparison of Selected-Response Question (SRQ) section scores among entering students from the U.S. domestic dataset. However, although most results were statistically significant, the effect sizes ranged from negligible to small. Generally, small differences were found between primary language groups among international entering students and among U.S. exiting students. In both samples, students whose primary language was the same as the language of instruction on average performed slightly better than their peers who had a different primary language. Among entering U.S. students and exiting international students, any differences found were too small to be practically meaningful.
When examining the differences between primary language groups more closely, it becomes evident that neither portion of the assessment is uniquely driving these differences. However, in subsamples where there were meaningful differences (i.e., international-entering and U.S.-exiting), there were found to be slightly larger effect-sizes with respect to the SRQ section than to the PT section. In some ways this is counterintuitive, as the PT section requires a written response whereas the SRQ section does not. One possible explanation for this unexpected finding is that, on average, students receiving instruction in their non-native language may be more adept at writing in their language of instruction than they are at comprehending documents in their language of instruction. Although both sections are document-based, the content of the PT is broader in scope than that of the SRQ section. Thus, when completing the PT, students may be better able to comprehend the information because they are given more context. Further research is needed to fully investigate the differences in CLA+ performance between students receiving instruction in their primary language versus a different language.
In the international sample, differences between primary language groups were found only among entering students. There are several possible explanations for this finding. One possibility is that students who are receiving instruction in their non-primary language may face a greater learning curve at the beginning of their higher education careers, which they adapt to over the course of their education. Another possibility is that there is an attrition effect. That is, it is possible that, if some students who receive instruction in their non-primary language are struggling more than their peers, they may be less likely to continue in their higher education careers. Thus, the students who were struggling upon entrance would not be included in the exiting student sample.
Meanwhile, students in the United States showed the opposite pattern in CLA+ performance. Specifically, a language-based difference in CLA+ performance emerged only among exiting students in the U.S. sample. Similar to the International sample, it is possible that this result was affected by student attrition. However, further investigation is needed to examine potential factors influencing this unexpected finding.
Table 6.5. Independent samples t-test results: Total CLA+ score by primary language
Table 6.6. Independent samples t-test results: Performance Task score by primary language
Table 6.7. Independent samples t-test results: Selected-Response score by primary language
Gender
Similar to primary language, students also identified their gender after concluding the assessment. The answer options presented to students were: male, female and decline to state. Since the framing of the gender survey question was consistent across participating countries, gender is a key variable for drawing comparisons across the overall sample. However, once again, U.S. data is reported separately from the rest of the international dataset due to the large sample sizes in the United States. Table 6.8-Table 6.11 summarise CLA+ total and section scores by gender, class and dataset.
Table 6.8. CLA+ score by gender, entering students: International sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Table 6.9. CLA+ score by gender, entering students: U.S. sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Table 6.10. CLA+ score by gender, exiting students: International sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Table 6.11. CLA+ score by gender, exiting students: U.S. sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
One-way analysis of variance (ANOVAs) were used to further examine potential gender difference in CLA+ performance (see Table 6.12-Table 6.14). Similar to the findings with primary language, most differences found were statistically significant but negligibly small. In most cases, post-hoc analyses revealed that the difference between males and females was driving the significant results rather than the students who declined to state their gender. This may be due to a relatively small sample of students in the latter group. Overall, there were no consistent patterns found within specific subsets of the sample. In other words, there was not one subsample in which males consistently outperformed females or vice versa. Similarly, for total CLA+ score and Performance Task (PT) score, there was not a consistent pattern of one gender outperforming another across subsamples.
The only consistent finding was the difference in average CLA+ performance on the SRQ section. On the SRQ section, all groups yielded a significant difference in which male students slightly outperformed female students on average. However, in most subsamples, the effect-size between males and females in average SRQ performance was negligible. One exception to this is the International-Exiting subsample, in which there was a small difference between males and females in average SRQ performance.
Overall, there were few clear patterns regarding gender-based differences in CLA+ performance. The one finding of note was the difference between males and females on the SRQ section among exiting international students. Further research is needed to examine the factors that may have contributed to this finding.
Table 6.12. One-way ANOVA results comparing total CLA+ score by gender
Table 6.13. One-way ANOVA results comparing Performance Task score by gender
Table 6.14. One-way ANOVA results comparing Selected-Response score by gender
In addition to providing their primary language and gender, students also responded to a survey question about their parents’ highest level of education. However, the response options differed based on whether the students tested on the international platform or the domestic platform. For students who tested on the international platform, except for the Italian students, the response options were based on the UK education system. Translations for other countries were kept parallel so that each answer choice would indicate an equivalent level of education to the UK sample. For the purpose of these analyses, response categories have been converted to map onto ISCED levels. Students’ average scores by class and parental education level are reported in Figure 6.1-Figure 6.6 for the International and U.S. samples.
CLA+ results by parental level of education were further examined via one-way ANOVAs (see Table 6.15-Table 6.16). Broadly speaking, higher levels of parent education were associated with higher CLA+ scores. In the international sample, each successive level of parent education was often associated with a statistically significant average score increase up until the bachelor’s degree level. For degrees beyond a bachelor’s, there were fewer significant score differences between education levels. In the U.S. sample, each successive level of parent education was associated with a significant average score increase.
In conclusion, students whose parents had at least a bachelor’s degree performed better on CLA+ than did those whose parents had less than a bachelor’s degree. In the international sample, the benefit of parent education diminished after the bachelor’s degree level. In the U.S. sample, however, the benefit continued to the graduate/post-graduate education level. The difference between the international and U.S. samples may be due to nuances in the relationship between education attainment and socio-economic status. Alternatively, these differences also may result from sampling discrepancies among countries.
Figure 6.1. Average CLA+ score by class and parent education, International Sample
Figure 6.2. Average performance task score by class and parent education, International sample
Figure 6.3. Average selected-response score by class and parent education, International sample
Figure 6.4. Average CLA+ score by class and parent education, U.S. sample
Figure 6.5. Average performance task score by class and parent education, U.S. sample
Figure 6.6. Average selected-response score by class and parent education, U.S. sample
Table 6.15. One-way ANOVA results for comparisons by parent education, international sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Table 6.16. One-way ANOVA results for comparisons by parent education, U.S. sample
Note: PT = Performance Task; SRQ = Selected-Response Questions
Given the differences found in CLA+ performance based on parent education levels, it is useful to investigate any potential interaction between parent education and CLA+ performance. Specifically, it is important to address the concern that performance differences between entering and exiting students may be due to selection rather than educational effect. Although the available data does not allow for a conclusive causal inference to be made, it does allow for deeper exploration. To disentangle the relationship between parent education and student CLA+ performance, entering and exiting students’ average total CLA+ scores were compared after controlling for parent education. The first step of this procedure entailed running a simple regression between parent education and total CLA+ score (see Table 6.17-Table 6.18).
Table 6.17. Simple regression results, international sample
Table 6.18. Simple regression results, U.S. sample
Next, independent-samples t-tests were used to compare entering and exiting students within each sample. The residuals from the simple regressions were used as the dependent variables (see Table 6.19). The t-tests showed significant results for both samples; however, the effect-size for the international sample was negligibly small. From a practical standpoint, these results are inconclusive as to whether performance differences between entering and exiting students can be traced back to selection or education effect. On the one hand, significant differences between class levels after controlling for parent education support the notion that there may be an education effect beyond selection. This is further enhanced by the modest but meaningful effect size seen in the U.S. sample. However, the small effect size among the international sample points to initial selection as a more important indicator than education effect.
Table 6.19. Independent samples t-test results: Total CLA+ score residual by class level
Overall, these results must be interpreted with caution given the limitations of these analyses. Most notably, unlike analyses from previous chapters, the residuals in the international sample were not weighted by country. This may have masked effects that were more prominent in some countries but less prominent than others. Furthermore, there is not sufficient information to draw a causal inference about the impact of education versus selection. Future research is needed to tease out this complex relationship.