Doris Zahner
Council for Aid to Education, United States
Jessalynn James
TNTP, United States
Jonathan Lehrfeld
ETS, United States
Does Higher Education Teach Students to Think Critically?
7. Predictive validity of CLA+
Abstract
The research results presented in this chapter are from two studies using a longitudinal data set that examined the validity of CLA+ as a predictor of post-higher education outcomes for students transitioning from higher education to career. CLA+ data from students who graduated in 2014 and 2017 and survey results from their employers and advisors help answer questions about the importance of these skills in post-higher education and whether they can be predicted by CLA+ test scores.
Introduction
Note: The data on the predictive validity of Collegiate Learning Assessment (CLA+) pertain to the data from CLA+ for students from the United States. There is currently insufficient CLA+ International data for a study of the predictive validity of the instrument.
Fact- and content-based knowledge is no longer sufficient for success in higher education and career. Students need generic skills such as critical thinking, problem solving and written communication to achieve their full potential. Although parents and students often believe that gaining admission to higher education is a clear step toward success, today’s students face an enormous challenge in successfully navigating higher education, as reflected in national graduation rates within the United States. Only 41% of first-time, full-time higher education students within the United States graduate within four years and only 59% do so within six years (de Brey et al., 2019[1]), statistics that paint a concerning picture. Persistence and retention are long-standing challenges – with little recent improvement – particularly for minority and low-income students (Banks and Dohy, 2019[2]; Hernandez and Lopez, 2004[3]). The most recent data indicate that among students who enrolled in higher education for the first time in fall 2017, only 62% were retained at their original institution in fall 2018 (National Student Clearinghouse Research Center, 2019[4]). Although many students cite non-academic reasons such as financial difficulties, health or family obligations as the primary causes for dropping out or deferring their education (Astin and Oseguera, 2012[5]), academic failure is also a significant factor contributing to lack of persistence and retention of students in higher education.
Once students do graduate, their next challenge is finding a career that leverages their knowledge, skills and abilities. As stated in Chapter 2, while content knowledge is a requisite part of a student’s education, it alone is insufficient for a student to thrive academically and professionally (Capital, 2016[6]; Hart Research Associates, 2013[7]; National Association of Colleges and Employers, 2018[8]; Rios et al., 2020[9]; World Economic Forum, 2016[10]). The question of whether these generic skills are empirically predictive of post-higher education outcomes remains. CLA+ data from graduating seniors help answer questions about the importance of these skills and the effectiveness of using CLA+ as a tool for identifying students’ strengths and areas of improvement.
The research results presented in this chapter are from two studies using a longitudinal data set that examined the validity of CLA+ as a predictor of post-higher education outcomes for students transitioning from higher education to career. CLA+ data from students who graduated in 2014 and 2017 and survey results from their employers and advisors help answer questions about the importance of these skills in post-higher education and whether they can be predicted by CLA+ test scores.
Study 1: CLA+ Predictive Validity of Post-Higher Education Outcomes
This study examined the validity of CLA+ as a predictor of post-higher education outcomes for students’ transitions to their careers. A longitudinal survey was administered to spring 2014 graduates to follow their post-higher education experiences.
Method
Participants
A total of 12 752 seniors tested in spring 2014. They came from 149 four-year institutions of higher education that included a mix of public and private research universities, master’s colleges and universities, and baccalaureate colleges. Admissions rates from the 149 institutions ranged from 18% to 100% (median 66%), six-year graduation rates ranged from 19% to 92% (median 55%), and percentage White ranged from 5% to 95% (median 68%). Criterion-referenced standards for the CLA+ were established (Zahner, 2014[11]) using the bookmark methodology (Lewis et al., 1999[12]). Table 7.1 shows the demographic information for the entire cohort and for those who earned the proficient, accomplished, or advanced level of mastery, which was 61.6% of the cohort.
Table 7.1. Demographic descriptive statistics
Data sources and materials
CLA+
Students took the CLA+ in spring 2014.
Survey #1
A longitudinal survey was administered to the 2014 cohort to follow their post-higher education experiences. Surveys were administered to participants three, six, and 12 months following graduation. Of approximately 13 000 students, 1 585 agreed to participate in the survey, and 993 persisted through all three phases. It should be noted that the registration for the survey was sent in August, three months after many of the participants had graduated, potentially limiting reach due to defunct or unattended email addresses.
Results
Logistic regressions were used to analyse the predictive validity of CLA+ scores on post-higher education outcomes (Table 7.2). These included:
positive post-higher education outcomes in general (0 = no, 1 = yes to full- or part-time employment, enrolment in continuing education, military service, or participation in a service or volunteer programme such as AmeriCorps)
annual salary (0 = below USD 45 000, 1 = above USD 45 000), using the national median for 2014 graduates from higher education (National Association of Colleges and Employers, 2015[13]) to determine the dichotomy
employment (0 = unemployed, 1 = employed full or part time)
full-time employment (0 = no, 1 = yes)
continuing education (0 = not currently enrolled in a programme of continuing education, 1 = enrolled in a programme of continuing education).
The CLA+ score was used to predict the five dichotomous variables in separate analyses. The results are found in Table 7.2. CLA+ was found to be a significant predictor of all post-higher education outcomes for students one year following their graduation.
Table 7.2. Logistic regression results for predicting post-higher education outcomes by CLA+ scores
Note: *p < .05; **p < .01
Race/Ethnicity by generic skills by institution competitiveness (Figure 7.1)
Race/ethnicity was self-reported by students in the demographic survey from the CLA+. Four categories were selected for analysis: Asian; African-American/Black, non-Hispanic; Hispanic or Latino; and White, non-Hispanic. Students were also categorised into two groups based on their mastery of the skills measured on CLA+: those proficient in critical thinking and written communication and those with developing or emerging skills. The final variable was whether the student attended a competitive or non-competitive institution (Barron's Profiles of American Colleges, 2014[14]).
Figure 7.1. Distribution of CLA+ proficiency and institution competitiveness by race/ethnicity; n = 12 476.
Basically, there are large proportions of minority students who have proficient and above mastery of the critical-thinking and written-communication skills attending non-competitive institutions. Approximately 35% of African-American/Black (non-Hispanic) and 25% of Hispanic or Latino students attending these non-competitive institutions have proficient, accomplished, or advanced skills. Although these proportions may not seem large, the number of minority students in less or non-competitive institutions far exceeds the number who attend the competitive institutions. This means that there is a significantly large group of qualified university graduates from under-represented or minority groups who may be overlooked as viable candidates due to the school they attended.
There are potentially millions of students graduating from less and non-competitive institutions (Benjamin, 2020[15]) who are proficient in the skills that employers say they desire (Hart Research Associates, 2013[7]), (2015[16]). Given that there is increasing enrolment at these less and non-selective institutions, which have higher proportions of minority students (Benjamin, 2020[15]), employers should expand their recruitment searches beyond the elite colleges and universities in order to have a more representative and diverse workforce.
Findings from this study offer support for the conclusion that critical-thinking and written-communication skills are important in predicting career placement and workplace success (Arum and Roksa, 2014[17]). Additionally, CLA+ can serve as both an effective instrument for identifying high-achieving students from less and non-competitive institutions and for making their skills more visible to perspective employees. The high-performing students who attend less and non-competitive institutions (Hoxby and Aver, 2012[18]) do in fact have the same critical-thinking skills as their peers at competitive institutions, which can potentially lead to positive post-college outcomes.
Study 2: CLA+ Predictive Validity of Employers’ and Advisors’ Assessments
This study follows the first study on the predictive validity of CLA+ on post-higher education outcomes and further answers the question of the importance of generic skills and the utility of an instrument for measuring these skills. In 2015, researchers contacted the employers or graduate advisors of the original student cohort and surveyed them. A second cohort of students from spring 2017 was also contacted for this study. They were not included in Study 1.
Method
Participants
From the spring 2014 cohort, 52 employers and 23 advisors responded to the survey for a total of 75 participants. An additional 10 employers and 4 advisors responded to a separate survey for the 2017 cohort, for a grand total of 89 participants. Given the small sample size, the employers’ and advisors’ survey results were analysed together. Table 7.3 shows the demographic information of the students whose employers and advisors responded to the survey and for all students who tested in spring 2014 as well as spring 2017.
Table 7.3. Demographic descriptive statistics
Analysis
Descriptive statistics and chi-square tests were used to investigate whether employers and advisors care about the skills measured by CLA+. Ordinal logistic regression models were then used to illustrate the relationship between CLA+ total score and employers’ and advisors’ ratings of the participants on said skills, as well as the relationship between CLA+ total score and employers’/advisors’ rating of how the participant ranked compared to other recent higher education graduates in the workplace/graduate programme. The proportional odds assumption was tested by comparing the fit of the ordinal logistic regression models with multinomial regression models. Both sets of models were found to result in very similar fit for each question.
Data sources and materials
Survey #2
In 2015, one year following graduation from university, a survey was administered to employers and advisors of students who took CLA+ in spring 2014. The survey was also administered to employers and advisors from the 2017 cohort. It should be noted that there is bias in the sample since students self-selected to provide their employers’ and graduate advisors’ information. However, the students did not significantly differ demographically from the total cohort of students (Table 7.3). There were slightly more female students, students who identified as white, students who spoke English as a primary language at home and students with parents with at least an undergraduate degree in the participating group than the total cohort.
The survey consisted of a series of questions (Table 7.4) regarding how important critical thinking and written communication skills are to successful performance by an employee or student, the employer’s or advisor’s perceptions of how proficient their employee or student is, and how the employee or student ranked in comparison to peers in the workplace or graduate programme.
Table 7.4. Employer survey questions
Results
Importance of CLA+ skills
Results indicate that employers and graduate advisors indeed find critical thinking and written communication skills, as measured by analysis and problem solving, writing effectiveness, and writing mechanics, important. Table 7.5 shows the distribution of responses to the first three questions in Table 7.4. Since only a few employers or graduate advisors responded “Unimportant” or “Of little importance”, these two categories and “Moderately important” were collapsed into one “Moderately important or less” category in subsequent analyses. However, for descriptive purposes, we show the original five response categories.
As might be expected given the observed percentages reported in the table, the chi-square tests confirmed that the responses were significantly different from chance (i.e. there was not an equal chance that employers/advisors would choose any of the three responses to each question). Clearly, employers and graduate advisors deemed analysis and problem solving, writing effectiveness, and writing mechanics to be important or very important.
Table 7.5. Distribution of responses to “Importance” questions
CLA+ scores predicting participants’ workplace or graduate school performance
Next, we used ordinal logistic regression models to examine the predictive ability of CLA+ total score on four ratings given by the participants’ employer or graduate advisor (questions 4-7 in Table 7.4). Given that analysis and problem solving, writing effectiveness and writing mechanics are important or very important skills, how well does CLA+ total score predict participants’ subsequent use of these skills in the workplace or graduate school? Also, how well does the CLA+ score predict relative rankings of the participants by the employer or graduate advisor?
Table 7.6 shows the ordinal logistic regression coefficients, their standard errors, 95% confidence intervals and the t-statistics (p < .001 for all analyses). The regression coefficients can be interpreted as the log-odds of being rated higher given a 1-point increase in CLA+ total score. For instance, in the analysis and problem solving model, the estimated coefficient is given as .0033. Thus, for a 1-point increase in CLA+ total score, the log-odds of “jumping” to a higher rating category (“Good” instead of “Satisfactory or worse”, or “Outstanding” instead of “Good”) increases by .0033. The regression coefficients are small because CLA+ total scores are on a large scale (400-1600), so one extra point is not expected to make much of a difference. Two factors would increase the interpretability of the results: 1) using a more meaningful score increase, such as 50 points, and 2) converting the log-odds to odds by exponentiating the coefficient. Thus, if one student scores 50 points higher than a second student, the log-odds of being rated one category higher than the second student is 50*.0033 = .165, and the odds are exp (.165) = 1.18. This first student is 18% more likely than the second student to be rated one category higher (“Good” rather than “Satisfactory or worse”, or “Outstanding” rather than “Good”) due to the higher CLA+ total score.
Table 7.6. Ordinal logistic regression models for predicting participants’ post-higher education performance
Note: Estimated coefficients are log-odds of being rated one category higher given a 1-point increase in CLA+ total score.
Conclusion
Employers and advisors find critical thinking and written communication skills to be important or very important for entry-level positions in the workforce and graduate programmes. CLA+ is predictive of positive post-higher education outcomes as measured by employers’ survey responses. This is important to note because despite approximately 1.8 million individuals graduating each year, employers are still finding a skills gap (Arum and Roksa, 2014[17]; Capital, 2016[6]; Hart Research Associates, 2013[7]; National Association of Colleges and Employers, 2018[8]; Rios et al., 2020[9]; World Economic Forum, 2016[10]). Recent graduates struggle to find appropriate entry-level jobs and wonder if they are getting a good return on their investment (Abel, Deitz and Su, 2014[19]). And traditional career services and job-search resources typically do not provide students with a platform to demonstrate higher-order skills to employers.
The impact to students who either do not graduate or graduate and are not able to find appropriate employment is huge for students and parents as well as institutions. The most recent data from the US Department of Education indicate that many low- and middle-income families have taken on a substantial amount of debt to finance their child’s college education (Fuller and Mitchell, 2020[20]). The OECD (2013[21]) Survey of Adult Skills (PIAAC), assessing foundation skills such as literacy, numeracy and problem solving in digital environments has demonstrated that higher education qualifications, the most commonly used measure of human capital, are a poor indicator of the actual skills level of the population. There is growing evidence that qualifications do not match skills (McGowan and Andrews, 2015[22]). Helping students improve and showcase their critical thinking, problem solving and communication skills improves their chances for positive academic and career outcomes.
Findings from this study offer support for the conclusion that generic skills such as critical thinking and written communication are important in predicting career placement and workplace success. Additionally, the CLA+ can serve as an effective instrument not only for identifying high-achieving students but also for making their critical thinking and written communication skills more visible to prospective employers and graduate school admissions officers.
References
[19] Abel, J., R. Deitz and Y. Su (2014), “Are recent college graduates finding good jobs?”, Current Issues in Economics and Finance, Vol. 20/1.
[17] Arum, R. and J. Roksa (2014), Aspiring Adults Adrift, University of Chicago Press, Chicago, IL.
[5] Astin and L. Oseguera (2012), “Pre-College and Institutional Influences on Degree Attainment”, in Seidman, A. (ed.), College Student Retention: Formula for Student Success, Rowman & Littlefield Publishers, New York.
[2] Banks, T. and J. Dohy (2019), “Mitigating Barriers to Persistence: A Review of Efforts to Improve Retention and Graduation Rates for Students of Color in Higher Education”, Higher Education Studies, Vol. 9/1, pp. 118-131, https://doi.org/10.5539/hes.v9n1p118.
[14] Barron’s Profiles of American Colleges (2014), Barron’s profiles of American colleges, 31st edition, Barron’s Educational Series.
[15] Benjamin, R. (2020), “Leveling the Playing Field From College to Career”, in Collective Goods and Higher Education Research, https://doi.org/10.4324/9780429453069-6.
[6] Capital, P. (2016), 2016 Workforce-Skills Preparedness Report, http://www.payscale.com/data-packages/job-skills.
[1] de Brey, C. et al. (2019), “Status and trends in the education of racial and ethnic groups 2018”, National Center for Educational Statistics, Vol. NCES 2019-038, https://nces.ed.gov/programs/raceindicators/index.asp.
[20] Fuller, A. and J. Mitchell (2020), “Which Schools Leave Parents With the Most College Loan Debt?”, The wall street journal, https://www.wsj.com/articles/which-schools-leave-parents-with-the-most-college-loan-debt-11606936947.
[16] Hart Research Associates (2015), Falling short? College learning and career success, Hart Research Associates, Retrieved from Washington, DC, http://www.aacu.org/sites/default/files/files/LEAP/2015employerstudentsurvey.pdf.
[7] Hart Research Associates (2013), “It takes more than a major: Employer priorities for college learning and student success”, Liberal Education, Vol. 99/2.
[3] Hernandez, J. and M. Lopez (2004), “Leaking Pipeline: Issues Impacting Latino/A College Student Retention”, Journal of College Student Retention: Research, Theory & Practice, Vol. 6/1, pp. 37-60, https://doi.org/10.2190/fbly-0uaf-ee7w-qjd2.
[18] Hoxby, C. and C. Aver (2012), “The Missing One-Offs: The hidden supply of high-achieving, low-income students”, National Bureau of Economic Research Working Paper 18586.
[12] Lewis, D. et al. (1999), The Bookmark Standard Setting Procedure, McGraw-Hill, Monterey, CA.
[22] McGowan, M. and D. Andrews (2015), Skill mismatch and public policy in OECD countries, OECD Publishing, Paris.
[8] National Association of Colleges and Employers (2018), Are college graduates “career ready”?, https://www.naceweb.org/career-readiness/competencies/are-college-graduates-career-ready/.
[13] National Association of Colleges and Employers (2015), Average starting salary for college class of 2014, http://www.naceweb.org/about-us/press/average-starting-salaries-class-2014.aspx.
[4] National Student Clearinghouse Research Center (2019), First-Year Persistence and Retention for Fall 2017 Cohort, https://nscresearchcenter.org/snapshotreport35-first-year-persistence-and-retention/.
[21] OECD (2013), Technical report of the Survey of Adult Skills (PIAAC), OECD, Paris, http://www.oecd.org/skills/piaac/_Technical%20Report_17OCT13.pdf.
[9] Rios, J. et al. (2020), “Identifying Critical 21st-Century Skills for Workplace Success: A Content Analysis of Job Advertisements”, Educational Researcher, Vol. 49/2, pp. 80-89, https://doi.org/10.3102/0013189X19890600.
[10] World Economic Forum (2016), Global Challenge Insight Report: The Future of Jobs: Employment, Skills and Workforce Strategy for the Fourth Industrial Revolution, World Economic Forum, http://www3.weforum.org/docs/WEF_Future_of_Jobs.pdf.
[11] Zahner, D. (2014), CLA+ Standard Setting Study Final Report, Council for Aid to Education, New York, NY.