Effective systems of external quality assurance for higher education require effective governance. International experience points to the necessity of three key characteristics in the design of institutions for the governance and implementation of quality assurance. First, quality external assurance bodies need to be independent of government and the demands of party politics, and of the higher education sector itself. Second, responsible authorities must have sufficient resources - financial, human, and intellectual - to meet properly their responsibilities. Third, quality assurance bodies should have carefully developed procedures for engaging with the broader society that higher education systems have a responsibility to serve. This chapter briefly examines the governance and implementation of quality assurance in Brazil in light of these considerations and provides recommendations for the further development of system governance.
Rethinking Quality Assurance for Higher Education in Brazil
8. Governance of external quality assurance
Abstract
8.1. Focus of this chapter
Effective systems of external quality assurance for higher education require effective governance. International experience points to the necessity of three key characteristics in the design of institutions for the governance and implementation of quality assurance. First, quality external assurance bodies need to be independent of government and the demands of party politics, and of the higher education sector itself (ESG, 2015[1]; INQAAHE, 2016[2]). Second, responsible authorities must have sufficient resources - financial, human, and intellectual - to meet properly their responsibilities. Third, quality assurance bodies should have carefully developed procedures for engaging with the broader society that higher education systems have a responsibility to serve (CHEA, 2016[3]). Below, we briefly examine the governance and implementation of quality assurance in Brazil in light of these considerations.
8.2. Strengths and weaknesses of the current system
SERES, INEP and CAPES together regulate, evaluate and supervise the federal higher education system
An elaborate system of quality assurance has evolved in Brazil, in which responsibility for quality assurance rests principally with the Ministry of Education (MEC) and, in particular, the Ministry’s regulatory department, the Secretariat for Regulation and Supervision of Higher Education (SERES), and executive agency for evaluation, INEP. In its evaluation of academic postgraduate programmes, CAPES, which is also answerable to MEC, focuses on a specific sub-sector of activity operating within an institutional landscape much of which is ultimately regulated and evaluated by SERES and INEP1.
INEP is a semi-autonomous federal institute (literally an “autarchy”, Brazilian administrative terminology), linked to MEC and bears responsibility for planning, coordinating, collecting, and analysing evidence upon which the quality assurance of undergraduate education rests. This includes planning and managing the collection of data for the census of higher education institutions; designing and supervising, and analysing the results of the ENADE (and ENEM); planning and managing on-site evaluations; and developing and managing the indicators that inform and support the regulatory work of government, such as the CPC and IGC.
The evidence of institutional and programme quality produced by INEP’s evaluation work provides the evidentiary basis upon which SERES supervises and regulates the federal system of higher education, both federal public and private institutions. SERES recommends the accreditation, re-accreditation and termination of accreditation of higher institutions to the National Council of Education, the deliberations of which are subsequently sanctioned by the Minister of Education.
CONAES formally supervises the evaluation work of INEP
Under the provisions of SINAES, the National Commission for Evaluation of Higher Education (CONAES) is responsible for proposing and assessing the procedures for the evaluation of institutions, courses and students; for approving the list of courses to be assessed by ENADE, and, more generally, for advising on the articulation federal and state education systems. CONAES is composed of representatives of INEP, CAPES, SERES; representatives of student, teacher and higher education administrative personnel associations; and five external members of recognised expertise appointed the Minister of Education.
This system has strengths…
There are important strengths to the governance and implementation of quality assurance in Brazil. For example,
INEP is recognised internationally as a leading public agency for educational assessment. Its wide experience with large-scale assessment and its capacity to manage data collection systems provides the nation’s higher education quality assurance system with a high level of competence.
CONAES has succeeded in attracting experts to its council, and through them has been able to mobilise higher education research from across the nation to inform the further development of SINAES.
The basic legitimacy and integrity of the quality assurance system is widely accepted across the higher education system, by public and private institutions alike, and by representatives of academic staff and the administrators and owners of higher education institutions.
In the course of its implementation, SINAES has used a range of evaluation techniques – including self-assessment, peer review, and external review grounded in student assessment - that has been widely welcomed.
Some higher education institutions in Brazil now closely monitor the experience of their students and their readiness to participate in external assessments. Others are making efforts to use compulsory self-assessment and peer review processes as opportunities for improvement, and to engage broadly their university community in the assurance of quality.
…but the current system of governance faces three main challenges
Nonetheless, there are three fundamental challenges facing the institutions of quality assurance that merit attention and improvement. First, the design of quality assurance institutions creates conflicting responsibilities for the Ministry of Education. MEC establishes, funds, and steers the federal university system, through its Secretariat for Higher Education (SESu). At the same time, it is responsible, through SERES and, indirectly, INEP, for evaluating their performance and for regulatory actions concerning the programmes they offer. These conflicting responsibilities lead the nation’s higher education institutions, especially its private institutions, to view the Ministry as a champion of one sector rather than a neutral arbiter among all. As one representative of a private higher education institution told to the review team, “For the people at MEC, the federal universities are their children, and we are their bastards.”
Second, while CONAES is responsible for providing guidance and feedback on the functioning of SINAES, it is not properly resourced and organised to do so. CONAES does not have its own professional staff or a dedicated budget, and lacks a capacity to undertake the sort of detailed and sustained analytical work that is needed to evaluate how SINAES is working. Instead, it depends upon the input of implementing bodies whose work it is to supervise and guide, most especially INEP. This dependence is exacerbated by the participation of the implementing bodies on the council itself. It lacks sufficiently wide input – from professional bodies, employer associations, and other centres of government - to take into account the broader social responsibilities of higher education.
Finally, in most higher education systems, responsibility for promoting and sharing quality improvement practices lies with bodies outside of government - with associations that represent sub-sectors (such as research, confessional, or polytechnic universities), and with bodies that represent professional groups within higher education institutions, including institutional research, curriculum design, assessment, and quality assurance. Examples of such bodies with such a role in other OECD countries include the German Rectors’ Conference (HRK, 2018[4]), Universities UK (UUK, 2018[5]) and the National Association of Universities and Higher Education Institutions in Mexico (ANUIES, 2018[6]). The review found few examples of the engagement of equivalent bodies in Brazil in research, advocacy, and training in support of quality improvement, and little attention on the part of public authorities to their potentially important role.
In addition to these three structural issues, there is the question of the coverage of the federal quality assurance system. As noted earlier, the systems for external quality assurance of HEIs and undergraduate programmes analysed in this report apply only to private HEIs and federal public HEIs. State and municipal public institutions – which account for almost 10% of enrolment - are not subject to SINAES, but rather to state-level regulatory and quality assurance rules. Although this situation reflects the constitutional distribution of competences in the Brazilian state, which allow considerable autonomy to states and municipalities, it leads to a fragmented system and means there is no single national benchmark of higher education quality. A single quality reference framework would make external quality assurance for higher education more transparent and understandable for students and their families.
8.3. Key recommendations concerning governance
To respond to the challenges outlined above, we believe that four related policy choices merit consideration.
1. Create an independent quality assurance agency
To address the conflicting responsibilities of MEC – or indeed any future ministry responsible for higher education - Brazilian authorities should consider creating an independent quality assurance body that stands outside the Ministry, in line with practice in many OECD and partner countries. This agency would take the lead in implementing the reformed system of quality assurance proposed in this report. Good international models of bodies with strong legal, financial, and administrative independence exist. In systems with a similar legal tradition to Brazil, such agencies include, for example, Portugal’s Agency for the Assessment and Accreditation of Higher Education (A3ES).
The work to design and create any new agency for quality assurance in Brazil will need to address some key questions:
Which existing functions should be transferred to the new agency? In principle, the new agency would combine the evaluation functions coordinated by INEP’s higher education evaluation directorate (DAES) and the regulatory and supervisory roles of SERES. The changes to the overall model of regulation, evaluation and supervision proposed in this report – such as increased focus on institutional review, reduced numbers of programme-level reviews, a reformed ENADE and a new indicator dashboard - will affect requirements for staff in different roles. The advantages and disadvantages of creating specific evaluation units for different sets of disciplines (natural sciences, social sciences etc.) should be considered. Such units, integrated within the agency, could potentially allow evaluation to be better tailored to individual disciplines and work more closely with the discipline-specific CAPES evaluations.
Should some tasks be devolved to decentralised offices in the states? The current system of quality assurance in the federal higher education system is highly centralised, with all evaluation and regulation activities coordinated from Brasilía. Devolving responsibility to regional departments might theoretically allow a more differentiated approach to quality assurance, with better consideration of the large regional differences in Brazil. However, in the view of the OECD team, distinct quality assurance procedures in different parts of the country would risk creating a two- (or multi-)tier system and undermining national recognition of quality standards. It could be possible, however, to establish regional offices to house professional inspectorates to undertake inspection of infrastructure and institutional management, freeing academic peer reviewers to focus on assessment of academic performance, potentially remotely (see above). The costs of the current system of peer review and the potential costs of a permanent inspectorate would need to be assessed in detail.
How should the new agency be funded? The current system of external quality assurance in Brazil is funded by a combination of public resources (paying the salaries of public servants, for example) and fees paid by institutions for evaluation activities. Quality assurance agencies in a number of systems, including the Portuguese example mentioned above, are funded primarily through fees from institutions. To ensure efficient use of public resources, this should be the long-term aim in Brazil. A thorough analysis will be required to determine the costs of a new agency and the level of fees needed to finance its operation.
The OECD team recognises that there is an existing proposal to create a National Institute for the Supervision and Evaluation of Higher Education (INSAES), that was introduced as a draft bill in Congress in 2012 (Congreso Nacional, 2012[4]), but not pursued. This initiative effectively also proposed a merger of the functions of SERES and INEP, but was criticised for its potential cost and limited added value. The OECD team believes that a new agency would be the most effective way to implement a reformed system of external quality assurance. The reforms proposed in this report are vital to improve the effectiveness and efficiency of the system and any future agency must be designed to operate as efficiently as possible and with limited direct public subsidy.
2. Strengthen CONAES
To ensure that the quality assurance agency has an advisory council that brings a wide social vision to its work, CONAES could take on this responsibility, after substantially modification. CONAES would be a council holding fixed and staggered terms to ensure their independence of government, and encompass balanced representation from students, public and private sector employers, instructors from public and private higher education institutions, higher education administrators, leading researchers, and the senior policy official in MEC with responsibility for taking a comprehensive view of higher education.
3. Restructure the government departments that are responsible for higher education
MEC – or any future ministry responsible for higher education - can support the improvement of quality assurance by restructuring its responsibilities for higher education. This could entail creating a post for a principal policy officer who takes a comprehensive and strategic view of the entire Brazilian higher education system – which the Ministry presently lacks. Units organised along sectoral lines, for example, could support the work of a senior official. These might include groups responsible for (a) federal universities; (b) private universities; (c) technical higher education; and (d) coordination with state and municipal higher education institutions. This scheme of organisation would benefit the nation’s quality assurance system by supporting a strategic and comprehensive vision for the higher education system, by clarifying the role of private provision within the system, and by encouraging continued differentiation of institutions and policies.
4. Incentivise the development of expertise in quality assurance in sector organisations
In monitoring and evaluating the nation’s quality assurance system, a reconstituted quality assurance agency and advisory council (i.e. CONAES) should focus on supporting the development of quality enhancing organisations outside of government. For example, it could support collaboration among state and national bodies of institutional evaluation offices (CPAs), so they share experiences of quality management and improvement practices with one another.
5. Explore how a reformed external quality assurance system could also apply to state and municipal institutions
A single system of external quality assurance applying to all higher education institutions in the country would be more transparent for students and the public than that current co-existence of a large federal system and individual systems for state and municipal institutions in each state. The federal and state authorities, working with the higher education sector, should explore how – and under what conditions - a reformed federal system of quality assurance could be applied to state and municipal institutions, while respecting the distribution of competences enshrined in the constitution of the Union.
References
[6] ANUIES (2018), Asociación Nacional de Universidades e Instituciones de Educación Superior, Asociación Nacional de Universidades e Instituciones de Educación Superior, http://www.anuies.mx/# (accessed on 18 November 2018).
[3] CHEA (2016), The CIQG International Quality Principles: Toward a Shared Understanding of Quality, Council for Higher Education Accreditation , Washington, DC, http://www.chea..
[7] Congreso Nacional (2012), Projeto de lei: cria o Instituto Nacional de Supervisão e Avaliação da Educação Superior - INSAES, e dá outras providências., Secretaria de Assuntos Parlamentares, http://www.planalto.gov.br/ccivil_03/Projetos/PL/2012/msg398-31agosto2012.htm (accessed on 30 November 2018).
[1] ESG (2015), Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG).
[4] HRK (2018), The HRK - German Rectors' Conference, https://www.hrk.de/hrk-at-a-glance/ (accessed on 18 November 2018).
[2] INQAAHE (2016), INQAAHE Guidelines of Good Practices 2016 - revised edition, International Network for Quality Assurance Agencies in Higher Education (INQAAHE), Barcelona, http://www.inqaahe.org/.
[5] UUK (2018), Universities UK, https://www.universitiesuk.ac.uk/ (accessed on 18 November 2018).
Note
← 1. CAPES regulates all academic postgraduate provision in Brazil, including in public State and municipal universities and university centres. SERES and INEP only regulate institution in the Federal higher education system.