Mark Keough
Council for Aid to Education, Australia
Does Higher Education Teach Students to Think Critically?
Report
Edited by Dirk Van Damme and Doris Zahner
Table of contents
Share
Facebook
Twitter
LinkedIn
16. Assessing students' generic learning outcomes in Australia and New Zealand
Abstract
This chapter deals with the outlooks for implementing the CLA+ assessment in professional and vocational colleges across Australia and New Zealand.
Introduction
Post-secondary education operates in two primary systems. The higher education sector incorporates universities and private not-for-profit and for-profit providers. This system is standardised and governed by the federal government where qualifications are defined by a national structure, the Australian Qualifications Framework.
Most professional education in the traditional professions is delivered in collaboration between universities and professional colleges with a university and industry accreditation system for professional practice. A national system of accredited qualifications governs technical and vocational education and training.
There are 39 public universities and approximately 130 private higher education providers (for-profit and not-for-profit) across the seven states and territories. According to the Australian Skills Quality Authority (ASQA), close to 4 000 registered training organisations address the technology and vocational education and training (TVET) market for domestic and international students.
Universities are clustered around research interests and profile. For example, a cluster known as the Group of Eight (G8) Universities focuses on the traditional professions (for example law and medicine) and formal research studies while the Australian Technology Network clusters five universities focussed on technology. Some universities offer dual-sector provision of both higher degrees and vocational education and training. Dual-sector provision is particularly prevalent in one state of Australia, Victoria.
Since 1985, the federal government has allowed universities to charge overseas students for enrolments. The post-secondary sector has grown to become the most extensive service-based export industry in Australia, contributing AUD 37.6 billion to the total Australian GDP of AUD 1 397 billion in 2019. The ratio of domestic to international students is close to 70:30.
The Australian system is regulated by the Tertiary Education Quality and Standards Agency (TEQSA) for higher degrees and the Australian Skills Quality Agency (ASQA) for vocational levels qualifications. The overarching framework is a structure of nationally regulated qualifications under the Australian Qualifications Framework (AQF). There are 10 levels of the framework, with a standardised nomenclature for qualifications. The TVET sector is represented by Levels 1 to 6 and uses a national qualifications framework that specifies detailed competency outcomes. The competency 'warehouse' is known as the Australian Quality Training Framework (AQTF).
Table 16.1. Australian Qualification Names
|
Level 1 |
Certificate 1 |
|---|---|
|
Level 2 |
Certificate 2 |
|
Level 3 |
Certificate 3 |
|
Level 4 |
Certificate 4 |
|
Level 5 |
Diploma |
|
Level 6 |
Advanced Diploma/Associate Degree |
|
Level 7 |
Bachelor’s Degree |
|
Level 8 |
Bachelor’s Honours, Graduate Certificate, Graduate Diploma |
|
Level 9 |
Master’s Degree |
|
Level 10 |
Doctoral Degree |
In New Zealand, there are eight public universities1, 16 public polytechnic institutes and a similar profile of domestic and international students. Connections to the extended Association of Southeast Asian Network (ASEAN; 16 countries) co-operation group includes many Indo-Pacific nations2. While there are many detailed differences between the Australian and New Zealand systems, the New Zealand post-secondary system operates in a very similar manner to the Australian system. It is governed by the New Zealand Qualifications Authority (NZQA).
Policy context
Critical Thinking and Problem Solving are among the ecto-curricula3 skills identified by each sector as contributing to employability, a key focus for the domestic education component of all education providers.
Employability influences the international student sector, where support for career mobility and options is vital in choosing education pathways. One of the key themes in the region defined as the greater ASEAN region, or what is increasingly identified as the Indo-Pacific, is the prevalence of student mobility. Students in this region actively seek education experiences from countries with a Western education tradition, such as Malaysia and Singapore, Australia and New Zealand.
A few authors and research bodies contribute to this field of study in Australia and New Zealand. Deakin University's Centre for Research in Assessment and Digital Learning in Education (CRADLE) is a dedicated research body with a peak interest in the field. Emeritus Professor Beverly Oliver is among the eminent writers on employability and graduate attributes, especially in the higher education sector. Australian vocational education focuses on employability and ecto-curricula learning outcomes on what are known as foundation skills. Sources of research include the National Council for Vocational Education Research and the National Skills Commission. In New Zealand, interest in this field is concentrated at the research institute, AKI Aotearoa4.
There is currently a significant suite of reform processes underway in Australia for the TVET sector. There is wide-ranging discussion among practitioners, academics and public policy makers about micro-credentials. Like any over-arching or jargon term, though, the question may well be raised about whether the word is itself a panacea for a deeper problem. It may be offering a convenient label for a set of ideas when more profound examination and insight is needed of what immediate and small credentials may provide in terms of skills development and enhancement.
Whether one considers the micro-credential discussion a trend or a fad, small amounts of learning will always need evaluation to ensure that learning utility is achieved. Education does not require a specifically tangible employment outcome context to be relevant. No matter how small, a learning outcome may find its context in ecto-curricula analysis, successful skills for career entry and/or career transition or progression. This kind of context suits such skills as critical thinking, creative thinking, critical analysis, comprehension, synthesis, and communication.
Technical and vocational education and training sector initiatives
In this light we can observe the public discourse in Australia and New Zealand on vocational training, where employability as a trend has been part of the general commentary for many years.
The Employability Skills Framework developed by the Australian Chamber of Commerce and Industry and the Business Council of Australia in 2002 remains in current use. The eight skills identified within that framework are: communication, teamwork, problem solving, initiative and enterprise, planning and organising, self-management, learning, and technology.
Building on this earlier industry research-based initiative, the Core Skills for Work Developmental Framework was created in consultation with employers in 2013. The aim was to assist training providers, employment services and any other organisations providing services to groups seeking to help people become work-ready. These core skills have also found expression as 10 core work-ready behaviours: manage career and work-life; work with roles; rights and protocols; communicate for work; connect and work with others; plan and organise; make decisions; identify and solve problems; create and innovate; recognise and utilise diverse perspectives, and work in a digital world.
In 2016, while seeking to address the employability needs of young people, the Australian Federal Department of Education, Skills and Employment drew on this overall body of work to identify core skills for employability. The Business Council of Australia released the publication Being Work Ready: A Guide to What Employers Want, (Business Council of Australia, 2016[1]) which groups skills and attributes desired by employers into three categories: values, behaviours and skills. Being Work Ready is designed to show the minimum standard of skills employers expect from job applicants soon after they have started the job.
Table 16.2. Summary of skills adapted from Business Council of Australia publication
Being Work Ready
|
Values |
Behaviours |
Skills |
|---|---|---|
|
Accountability Honesty Respect Work ethic |
Adaptable Business-minded Customer-focused Globally aware Authentic Collaborative Flexible Self-aware Resilient |
Business literacy Data analysis Literacy Problem-solving Critical analysis Digital technology Numeracy Technical skills |
Note: Copyright Business Council of Australia (2016[1]).
As part of a significant set of reforms for the vocational education sector, the Australian federal government in July 2020 created a National Skills Commission.
In 2020, this new statutory authority established a data science-driven approach to skills to develop job and training matching services. Project JEDI (Job and Education Data Infrastructure) seeks to create a common language for skills; link jobs to appropriate training; forecast future needs based on analytics; and use a single data engine to support many outcomes and services in the Australian context. The Commission classifies core competencies, including employability skills, soft skills, foundational skills and transferable skills.
There has been a strong emphasis on definitions, stakeholder input and the establishment of overarching strategy. Unfortunately, this has led to many different meanings, confusing public dialogue. We can observe substantial inputs but limited outcomes or clarity.
Some describe scholarly work in employability research in Australia as under-developed, lacking, and nascent (McArthur et al., 2017[2]). This perspective likely arises because the published work has often focussed on descriptive analysis rather than an inquiry about economic impacts and social perspectives.
There is no doubt this is an emerging area of interest as the regional population becomes more transient in their job and career experience. New initiatives such as Project JEDI will take some time to find their feet. There is hope that there will be interest among academics to investigate the effectiveness of these initiatives in terms of economic and social outcomes.
In the New Zealand context these skills are defined through the careers promotion context. The site offers guidance through self-assessment resources including the “skill matcher” tool.5
Higher education sector policy initiatives
TEQSA has defined graduate attributes in this way: “Generic learning outcomes refer to transferable, non-discipline-specific skills that a graduate may achieve through learning that have application in the study, work and life contexts.” Graduate employability and citizenship are critical outcomes attributed to the development of university graduate attributes in the Australian context (Oliver and Jorre de St Jorre, 2018[3]). In 2000, Bowden et al. (2000[4]) acknowledged that graduate attributes were distinguished from the technical or domain knowledge and skills in a curriculum by their capacity to prepare graduates as agents of social good in an unknown future.
Oliver et al. suggest three pivotal questions regarding graduate preparation from Australian universities for 2020 and beyond:
1. Which graduate attributes should be emphasised given the massive changes occurring in society?
2. Beyond embedding, how are attribute outcomes assured?
3. Which attributes equip for employability?
Measurable outcomes are not guaranteed. Where measurement occurs, it is likely based on generic, opinion-based surveys rather than empirical and objective data. The core reasons why universities promote graduate attributes are perhaps open to question. Oliver rightly considers whether it is more about marketing potential outcomes to potential students than a steadfast commitment to extrinsic and measurable results after graduation.
There has been considerable work undertaken in the interest of teaching and learning standards funded by the Australian Teaching and Learning Council. These are standards of delivery but not yet a measure of outcomes. Oliver and Jorre de St Jorre continue to find that graduate attributes are consistent in their emphasis in broad terms. If more explicit definitions can be achieved, a more significant analysis of objective outcomes may be possible. There seems to be an emerging place in the Australian and New Zealand context for the empirical and longitudinal study of effects rather than the current emphasis on inputs and satisfaction surveys.
Follow-up work in 2015 indicated that while most universities published graduate attributes, fewer than half of Australian non-university higher education providers followed suit. Universality and consistent definition remain a challenge in terms of measurement.
Currently, despite emphasising employability skills and graduate qualities in public policy and policy-associated rhetoric, the outcomes for students are not clear. Graduate outcomes supporting employability skills are largely unmeasured.
Indicators from Oliver's work suggest that much has been done to understand the types of assessment that engage students. The consistent application of those principles to graduate attributes may be ahead of us still, rather than in everyday practice (Oliver and Jorre de St Jorre, 2018[3]).
Qualification relevance to employability under question
Micro-credentialling is a trend as industry representatives seek to address immediacy and job relevance in accredited training and education options. The benefit of just-in-time learning related to current or prospective job needs has significant appeal to employers and potential job candidates. It is also relevant for retraining existing employees. Agility in enterprise for both employers and potential employees is key to success as technology enhancements increasingly accelerate the pace of change.
It is difficult to find direct evidence of benefit for micro-credentialling as a trend or clear benefits of the emphasis on employability skills, however. Progress on the development and implementation of 21st-century skills has been reported by the Brookings Institution in October 2020 (Taylor et al., 2020[5]). Their Center for Curriculum Redesign (CCR) has defined 12 competencies and are monitoring their progressive inclusion in curriculum on a jurisdictional basis. The 12 competencies are defined under the CCR framework.6 The report concludes that institutions are trending toward including these competencies in curriculum, moving beyond pure academic pursuits and focussing on the needs of the 21st-century learner. The report (p9) indicates that Australia is a leader in the comprehensive consideration of all 12 competencies in the framework.
Quality Indicators of Learning and Teaching (QUILT) is a group of Australian surveys supported by the government which seeks to measure student experience, graduate outcomes, longitudinal graduate outcomes and employer satisfaction. The data sets from these surveys underpin the CompareED website. (www.compared.edu.au). This website provides comparison data against national averages based on the survey outcomes. Oliver identifies that the challenge with the QUILT surveys is that they are national and generic across whole institutions rather than specific to particular areas. The self-reported outcomes of students and employers are reported rather than objective views based on data-driven measures. Nonetheless, they do provide some useful comparisons of satisfaction and a basis of institutional comparison in the higher education sector. The QUILT surveys are an opt-in measure and not compulsory.
What about secondary school curriculum?
Employability skills are also the focus in the late secondary years. Problem solving and critical thinking have been identified as essential in the transition from secondary education into workplaces for further study. The Australian curriculum for secondary schools describes these skills as “General Capabilities”. Dr Paul Weldon of the Australian Council of Education Research (ACER), writing in April 20207, laments that there is a lack of agreement about the fundamentals of the construct of general capabilities. He further emphasises that job automation is a driver of change away from domain-specific skills and knowledge, and towards such critical capabilities as critical thinking and problem solving (Weldon, 2020[6]). The emphasis from the OECD Learning Framework 20308 favours “the mobilisation of knowledge, skills, attitudes and values through a process of reflection, anticipation and action, to develop the inter-related competencies needed to engage with the world”.
COVID-19 impacts on policy
International student numbers have been decimated by COVID-19 as the Australian border has remained closed since the first quarter of 2021 despite (mostly) an absence of community spread of COVID-19 cases in Australia and New Zealand. The policy emphasis in a post-COVID-19 world seems to be on investment in skills and training for employment. Student mobility will likely be de-emphasised for some time, perhaps several years. The emphasis on employment perhaps comes as an unfortunate focus when employability is a subtly different motive or outcome. Employability is a state of being whereby societal contribution focuses on productivity in a broader social sense than simply an employment outcome. Successful employment is a direct outcome of employability.
The discourse about skills and know-how has moved to focus on the domestic employment market in response to the pandemic. Staying job-ready for those challenged by employment opportunity in this new era seems a common motif in governmental response. Some authors see some risks and perhaps unsupported assumptions that maintaining skills rests entirely with the learner, student or the unemployed. There appears to be an emphasis on responding solely to the continuing automation of work and developing skills for labour markets as a simple supply and demand arrangement. Increasing amounts of work will become task-oriented rather than career-oriented (O’Keeffe and Papadopoulos, 2021[7]).
Indications are that this contracting economy, often called the gig economy, will grow significantly in the next five years. Casualisation and gig- or bid-based work are factors leading to the importance of transferable skills for participants. Just how important and how quickly the sector is growing is often in the realm of market forecasters such as Deloitte, Gartner, and Forbes rather than academic research so it is often highly speculative. From these sources, indications suggest we might consider growth from 25% of the adult workforce in Western developed economies in 2020 and 2021 to 40% in 2025. Research commissioned by the Institute of Actuaries of Australia show that the gig economy grew ninefold in 2015-2019.9 For many workers, employment can consist of more than one contract (or gig) at a time, perhaps even using entirely unrelated skills. Developing and curating a lifelong warehouse of skills is critical to employment success.
Gig workers also need these skills to be transferrable between roles and, preferably, measurable. To win contract roles, a gig worker relies on referrals from previous clients and projects. Embedded in those referrals is an acknowledgement of a worker's employability skills or transferrable skills. But, unfortunately, there is little objective measurement of those skills. As it stands, subjective views on a worker’s skills is such that good client or customer reviews can bring work while negative ones deny employment.
Taking an ASEAN and Indo-Pacific view, we revisit the question of mobility. Aspirational economies in these regions thrive on encouraging their students to pursue an academic career overseas. While many repatriate to their country of origin, others find new homes as citizens of the countries they chose to study in. The notion that citizenship is as essential as employability is a clear finding by Oliver (Oliver and Jorre de St Jorre, 2018[3]). In this manner the CLA+ assessment provides a value-added benefit to students who may seek to use 21st-century skills in seeking to work, or even settle, outside of their home country.
Next steps and prospects
The employability of vocationally focussed lifelong learners and university graduates is the policy driver for the focus on foundation skills in Australian and New Zealand vocational post-secondary learning. But there appears to be a significant gap in implementation and measurement of both learning and learning outcomes. The policy is deep in desire but not intense in terms of the methodology or measurement of learning and learning outcomes.
The CLA+ and its derivative analysis tools offer a ray of light in empirical analysis and detailed benchmarking for skills measurement in foundation or employability skills. Through potential pilot programmes in a range of industry settings, there is a likely benefit in introducing the CLA+. Under consideration are skills transition projects in military veterans moving from service into civilian employment, healthcare management, and the late secondary years when students transition to either work, vocational education, or higher education. In this setting, the CLA+ offers insights regarding employability for younger people.
The Council for Aid to Education (CAE) has established a Memorandum of Understanding with the Australian National Council for Vocational Education Research. Under this arrangement there are plans in the latter half of 2021 to undertake a pilot project with a regional public vocational education provider (Goulburn and Ovens Valley TAFE) in Victoria. This project will afford students an empirical measure of their foundation skills. It will provide validated insight into their relative strength and weaknesses in those skills. We look forward to understanding the impact of this approach.
The secondary graduation examinations and learning pathways are currently under review with several research projects underway. Unlike national regulation of post-secondary education, secondary education in Australia is overseen by state authorities.
Currently, QUILT surveys serve the university sector. The challenge with these surveys is that they are used to target potential students in the market for a university rather than providing empirical evidence on student learning. Many universities are autonomous in their scope and automated in terms of quality, and so working with university partnerships is highly desirable. The CLA+ would offer an established data set to allow immediate benchmarking.
Given employability is a key theme, connecting graduate attributes and foundation skills with indicators of employment success would seem a valid connection to pursue in the Australia and New Zealand context.
Conclusion
Transferability of ecto-curricula skills is crucial in the post-pandemic economies of the Indo-Pacific region, encompassing a gig economy, plural careers, transferability and regional mobility. As Australia and New Zealand re-establish their role as providers of post-secondary and pre-employment education for the region, measurement and assurance of outcomes for cohorts of students is of increasing importance. As individual students take more responsibility for curating their skills throughout their career, data-based measures of relative learning progression such as the CLA+ are likely to increase in importance.
References
[4] Bowden, J. et al. (2000), Generic capabilities of ATN university graduates, Australian Government Department of Education, Training and Youth, Canberra.
[1] Business Council of Australia (2016), Being Work Ready: a guide to what employers want, https://www.bca.com.au/being_work_ready_a_guide_to_what_employers_want.
[2] McArthur, E. et al. (2017), “The Employers’ View of “Work-Ready” Graduates: A Study of Advertisements for Marketing Jobs in Australia”, Journal of Marketing Education, Vol. 39/2, pp. 82-93, https://doi.org/10.1177/0273475317712766.
[7] O’Keeffe, P. and A. Papadopoulos (2021), “The Australian Government’s business-friendly employment response to COVID-19: A critical discourse analysis”, Economic and Labour Relations Review, Vol. 32/3, https://doi.org/10.1177/1035304621997891.
[3] Oliver, B. and T. Jorre de St Jorre (2018), “Graduate attributes for 2020 and beyond: recommendations for Australian higher education providers”, Higher Education Research and Development, Vol. 37/4, pp. 821-836, https://doi.org/10.1080/07294360.2018.1446415.
[5] Taylor, R. et al. (2020), Competencies for the 21st Century, Center for Curriculum Redesign, and Brookings Institution, Boston, MA.
[6] Weldon, P. (2020), Defining Skills for the furure: What’s in a name?, https://www.acer.org/au/discover/article/defining-skills-for-the-future-whats-in-a-name (accessed on 21 October 2021).
Notes
← 3. Ecto-curricula skills are skills that are gained through the process of learning, or participation in a programme of learning. They are, however, not a direct part of the curriculum or learning objectives. These skills may be colloquially known by a number of terms in common use such as soft skills, foundation skills, employability skills or 21st-century skills.