Rethinking Quality Assurance for Higher Education in Brazil

Executive Summary

Brazil has well-established systems to assess, monitor and assure the quality of learning and teaching in private and public higher education providers in the federal higher education system. The Ministry of Education (MEC) makes regulatory decisions about accreditation of institutions and authorisation and recognition of undergraduate programmes based on the results of evaluations coordinated by the Anísio Teixeira National Institute for Educational Studies and Research (INEP). The Foundation for the Coordination of Improvement of Higher Education Personnel (CAPES) implements a separate system of quality assurance for academic postgraduate programmes.

The external quality processes for higher education institutions (HEIs) and undergraduate programmes are mandatory and apply to private and federal public institutions. These account for 90% of the over 2 400 HEIs in Brazil and enrol 91% of undergraduate students in the country. Three quarters of undergraduate enrolment in Brazil is in the private sector. The remaining 9% of enrolment is in public state and municipal institutions, which are subject to regulation and quality assurance by state governments. CAPES evaluation applies to all academic postgraduate education in the country.

This review assesses the relevance, effectiveness and efficiency of the quality assurance procedures in place. The OECD review team was asked to consider the effectiveness and efficiency of the systems in ensuring minimum quality standards, providing differentiated measurement of quality and promoting improvement of quality and quality-oriented practices in HEIs, providing recommendations for improvement.

Regulating “market entry”: ensuring providers of higher education meet high standards, while streamlining quality assurance procedures

In contrast to some other countries in the Latin America and Caribbean region, compliance with Brazil’s system of accreditation for private higher education institutions and official recognition of undergraduate programmes is nearly universal. The requirements of institutional accreditation are sufficiently rigorous to limit fraudulent or grossly unqualified private institutions from entering the higher education marketplace. The formal requirement for all new undergraduate programmes to obtain official recognition in the early stages of their operation provides a basic guarantee of quality.

Nevertheless, Brazilian authorities could improve the relevance and effectiveness of the regulatory and evaluation processes that govern the “market entry” of new private higher education providers and undergraduate programmes. First, current quality assurance systems and frameworks can be adapted to make them more effective. In the short-term, measures should include creating a user-friendly online platform to provide students and families with reliable information on the accreditation status and quality of higher education programmes; developing more sophisticated indicators to assess the quality of distance education and monitor its expansion; improving the selection and preparation of peer review commissions; and taking greater account of pedagogical processes and initial results in on-site reviews of recently created programmes.

Second, there is scope to target the finite resources available for external evaluation of higher education on institutions and programmes that present the greatest quality risks for students and society. HEIs with demonstrated capacity in internal quality assurance could be permitted to “self-accredit” their own programmes, following rigorous institutional reviews. For institutions that remain subject to programme-level reviews, Brazilian authorities could consider allocating certain tasks to a professional inspectorate, allowing academic peer reviewers to focus on evaluating core aspects of the learning process.

Ongoing quality assurance of undergraduate programmes: improving measurement of quality and better targeting of resources

Each year, students graduating from undergraduate programmes registered in particular disciplines take a mandatory competency assessment: the National Examination of Student Performance (ENADE). The formal objective of ENADE is to assess students’ acquisition of knowledge and skills specified in National Curriculum Guidelines (DCN) and their understanding of unspecified broader societal themes. These objectives are too broad for a test with the scope of ENADE and the general knowledge component of the tests is unrelated to the content of the programmes it is supposed to evaluate.

There are also weaknesses in the way ENADE is designed and implemented, which hinder its ability to generate reliable information on student performance and programme quality. It is a low stakes exam, which reduces the motivation of students; test items are not standardised, meaning tests are not of equivalent difficulty between years and subjects and there are no explicit quality thresholds to indicate what good performance is. Results for students on each programme are standardised to generate a score on a scale of one to five, but is a relative measure of average student performance, not a clear indication the level of their knowledge and skills. Brazilian authorities should undertake a thorough assessment of the objectives, costs and benefits of large-scale student testing to identify how weaknesses can be addressed and how, in contrast to the current situation, ENADE could be made into a useful tool and feedback mechanism for teachers and institutions.

The results of ENADE feed into a composite indicator of quality for each programme: the Preliminary Course Score (CPC). This also includes scores for the profile of the teaching staff, student feedback and an indicator of assumed learning gain (IDD). The IDD is based on a number of bold assumptions about the influence of programmes on student performance, which make it hard to justify its weight (35%) in the CPC. INEP should move to monitoring programme performance using an “indicator dashboard”, with a broader range of disaggregated indicators, including measures of student drop-out and, ultimately, graduate employment outcomes.

Site visits for established undergraduate programmes, which are currently used only when programmes perform poorly in relation to the CPC, use a review template and scoring system that do not focus on identifying the causes of poor observed performance and do not consider graduation rates and graduate destinations. Site visits should be retargeted to focus on the root causes of poor performance highlighted by indicators, while peer reviewers should also visit good programmes to help them understand the factors that affect good performance.

Quality in postgraduate programmes: fine-tuning existing practice and planning for the future

The system of external quality assurance for academic postgraduate education in Brazil sets a comparatively high bar for academic postgraduate training to enter the system. However, there is scope for the peer review committees that evaluate new programme proposals to focus more on the relevance of programmes to expanding knowledge fields and on the design of the training provided to postgraduate students. The four-year periodic reviews involve resource-intensive review of staff outputs, alongside consideration of other factors, but neglect training conditions, student output and graduate destinations. CAPES should rebalance the evaluation criteria to focus more on student outputs and outcomes. The reliance on peer review will make the system harder to scale as postgraduate education expands, while inbreeding creates risks for objectivity and quality. It is important to involve international peers in assessment of the top-rated programmes and plan for the future, notably though evaluating the role of academic Master’s programmes and the real costs of the current peer review processes.

Quality higher education institutions: increasing focus on internal quality assurance and the role of external institutional reviews

Although private and federal public institutions are subject to periodic re-accreditation, with institutional reviews, “de-accreditation” is rare for private institutions and effectively impossible for public institutions. The General Course Index (IGC) – another composite indicator used by INEP - provides limited signals about institutional quality. On-site re-accreditation reviews pay limited attention to evidence of institutional performance, internal quality processes and their practical implementation. Brazilian authorities should reduce the period of accreditation for universities and university centres (currently eight or ten years) and, in more robust institutional reviews, increase the focus on outputs, outcomes and internal quality assurance procedures. A greater focus on these issues would allow Brazil to move to a system where institutions with demonstrably strong internal quality assurance capacity and a proven record of delivering quality can accredit (authorise and recognise) their own programmes.

Governance of the quality assurance system: ensuring greater transparency, improved steering and engagement of sector organisations

The basic legitimacy of external quality assurance in the federal higher education system is not questioned and INEP has developed significant experience and capacity in evaluation. However, challenges in system governance include a potential conflict of interest in MEC, which both steers and regulates federal public institutions. CONAES – the National Commission for Evaluation of Higher Education - lacks resources and the capacity to oversee the quality assurance system, while higher education intermediary organisations have weak capacity in quality-related matters. Drawing inspiration from international examples, Brazil should explore ways to establish an independent quality assurance agency to take on roles currently in MEC and INEP, considering the option of combining a professional inspectorate and academic peer review. CONAES should receive dedicated resources and sector associations in higher education should be incentivised to promote quality across the system. In cooperation with state governments and the higher education sector, federal authorities should explore how a reformed external quality assurance system could also apply to state and municipal institutions.