Rethinking Quality Assurance for Higher Education in Brazil

4. Market entry: New institutions and undergraduate programmes

Abstract

This chapter focuses on the processes in place in Brazil to regulate the establishment of new private higher education institutions and of new undergraduate programmes in new and existing institutions in the federal higher education system. The establishment of new private institutions and the creation of new undergraduate programmes in all types of institution require higher education providers to seek regulatory approval from the Ministry of Education (MEC). This approval depends on compliance with administrative procedures and a positive outcome from external peer reviews designed to assess the quality of proposed new institutions and new and recently established programmes. The chapter analyses these regulatory and evaluation processes, highlighting strengths and weaknesses, and provides recommendations for their improvement

4.1. Focus of this chapter

This chapter focuses on the processes in place in Brazil to regulate the establishment of new higher education institutions (HEIs), and of new undergraduate programmes in both new and existing institutions in the federal higher education system.

Specific processes exist for institutional accreditation for private institutions…

Public HEIs in Brazil are created by federal, state or municipal governments, through legislation that automatically conveys authorisation to operate for the institution in question. As explained earlier in this report, once established, public institutions created and funded by the federal authorities form part of the federal higher education system and are legally subject to regulation and supervision by the Ministry of Education (MEC) and institutional and programme evaluation by INEP as part of the National System of Higher Education Evaluation (SINAES). Public institutions created and funded by state or municipal governments (which include some large institutions such as the University of São Paulo (USP) or the University of Campinas (UNICAMP) in the State of São Paulo) are regulated by state governments and are not obligated to follow the rules of SINAES.

All private higher education providers are considered part of the federal system and are required to obtain formal external accreditation (credenciamento) from MEC to allow them to begin operation.

HEIs in Brazil are legally classified as colleges (faculdades), university centres or universities. In broad terms, colleges are undergraduate teaching institutions; university centres are required to have more extensive undergraduate provision, more permanent staff and developed outreach activities; and universities, in addition to meeting these criteria, must have research activity and provide postgraduate education. Public HEIs created by federal, state, or municipal governments may be established in any of these three legal forms from the outset. In contrast, all new private institutions must first be established as colleges and may subsequently transition to the status of university centre or university if they meet the relevant criteria and successfully undergo a new external re-accreditation process. The specific criteria that private HEIs must meet to qualify for each institutional status (organização acadêmica) are set out in articles 16 and 17 of a 2017 decree governing quality assurance processes in the federal higher education system (Presidência da República, 2017[1]).

…and approval of new programmes

As a general rule, colleges in the federal higher education system, which are almost exclusively private1, are required to obtain formal authorisation from MEC (autorização) to start new programmes. All applications for the establishment of a new higher education institution must be accompanied by at least one (and up to five) application(s) to create a new undergraduate programme. Existing colleges must submit applications for authorisation for each new programme they wish to create. As discussed below, the complexity of the procedures followed to obtain authorisation varies depending on the institutional quality score held by the college submitting the request for authorisation and field of study of the proposed programme. University centres (all of which are private) and universities2 have a greater degree of autonomy and are not generally required to obtain authorisation in advance to start new programmes, but must notify MEC of the creation of all new programmes. Universities and university centres do require prior authorisation to start new programmes in medical fields and law (see below).

All HEIs in the federal system, whatever their legal form, are required to submit new programmes to an external quality assurance process called “recognition” (reconhecimento), once half of total teaching hours have been completed (in the second or third year, for example). All programmes offered by HEIs need to complete the recognition process successfully for the degrees they award to be valid in Brazil. As a result, reconhecimento is effectively part of the initial approval process for programmes. Failure to obtain recognition would mean that the diplomas obtained by graduates would not be valid nationally and may thus lead to the closure of the programme.

The approval of new institutions and programmes forms a key part of the wider regulatory work of Secretariat for Regulation and Supervision of Higher Education (SERES) in the Ministry of Education (MEC), the full scope of which is illustrated in Table 4.1 below.

Table 4.1. Regulatory acts by SERES in 2017

Regulatory Acts	Number
Regulatory acts relating to HEIs
Credenciamento (Accreditation)	208
Credenciamento EaD (Accreditation, Distance Provider)	70
Recredenciamento EaD (Re-accreditation, Distance Provider)	36
Transferências de Mantença (Transfer of Ownership)	273
Regulatory acts relating to programmes
Autorização não-vinculada (Authorisation, not linked to accreditation)	1361
Reconhecimento (Programme Recognition)	1686
Renovação de Reconhecimento (Renewal of Programme Recognition)	6781
Autorização EaD (Authorisation of New Programme, Distance Education)	225
Reconhecimento EaD (Renewal of programme recognition, distance education)	266
Renovação de Reconhecimento EaD (Renewal of Programme Recognition, Distance Education)	188
Aditamento de polos (Addition of Distance Education Sites)	88
Aumento de vagas (Authorisation for increase in vagas)	267
Other regulatory acts
Aditamento (Modification of existing regulatory acts)	645
Chamamento Público (Public call)	2
Total Acts	12 096

Source: SERES, “Regulação e Supervisão da Educação Superior” p.17, December 2017, presentation to OECD Review Team.

4.2. Strengths and weaknesses of the current system

This section analyses the strengths and weaknesses of the current systems of regulation and evaluation for the accreditation of new private providers and the authorisation and recognition of new undergraduate programmes.

Establishment of new institutions: accreditation

Relevance: the rationale and objectives of the current system

In all systems of external quality assurance, processes of “market entry” by new higher education institutions must take care to balance quality and quantity. Entry requirements must be sufficiently restrictive to prevent bad programmes, such as those in which large numbers of students are unable to complete their studies, or complete their studies but acquire few skills and have poor prospects for employment. At the same time, these requirements must not create barriers to entry that unnecessarily limit supply and leave students without access to beneficial study opportunities. In addition, badly designed arrangements for the establishment of new institutions may require new institutions to adopt governance, management, or staffing arrangements that hamper innovation and efficiency.

Brazil’s system of accreditation (credenciamento) for new private HEIs exists to ensure that new institutions meet minimum standards of quality in providing higher education programmes to undergraduate students. The arrangements for the establishment of new private HEIs, which must always be coupled with the creation of programmes, strike a balance between ex ante and near-term ex post review to assure quality. Like other higher education systems, Brazil applies a lower level of scrutiny to institutions established by public authorities – in practice, institutions established by federal, state and municipal governments. Public institutions are not subject to the same accreditation requirements as private institutions, may be created as universities or university centres, and are exempt from any other forms of ex ante review. Public law authorises heightened control over the creation of private provision, permitting newly established private institutions to take only the form of “college” (faculdade), rather than fully developed university institutions.

Effectiveness: quality indicators used and division of responsibilities

MEC necessarily carries out accreditation of new private institutions based on planned provision rather than functioning programmes and institutions, and thus must focus on the planned inputs and processes, rather than observed processes, outputs or outcomes. New private higher education institutions begin the process of accreditation when a legal representative of the maintaining institution submits to SERES documents outlining various dimensions of the planned institution and a self-evaluation of its expected performance. The information submitted (Presidência da República, 2017, pp. art. 20-21[1]) includes:

1. Information about the legal, financial and tax status of the operating organisation (mantenedora).

2. An Institutional Development Plan (PDI), containing:

i. A description of the institutional mission, goals and profile,

ii. An institutional pedagogical programme describing educational, research and outreach policies,

iii. Details of the number and nature of planned study programmes, including the number of study places (vagas),

iv. Details of (planned) teaching staff,

v. Planned digital document and records management systems,

vi. Details of physical infrastructure, including library, laboratories etc., where relevant,

vii. Details of financial capacity and a financial sustainability plan,

viii. Where relevant, specific information relating to distance education.

3. Internal statutes.

4. Information about the proposed management staff, including their academic experience.

5. Proof that suitable premises and facilities (imóvel) are available, including technical appraisal of their accessibility for students with special needs and compliance with fire and safety regulations.

Institutional accreditation is linked to a review of proposed programmes. Private institutions seeking initial accreditation have up to three years following approval of their initial institutional plan to submit and obtain authorisation of between one and five programmes (autorização vinculada a credenciamento). Additionally, campuses that are located outside of the municipality in which a private institution has its seat of operations must undergo a separate process of accreditation.

Following review and approval by SERES of documents submitted to it through the e-MEC online platform, a process of peer review is undertaken. Three external evaluators drawn from INEP’s BASis database undertake an on-site inspection and review visit, evaluating the conditions for the proposed institution. This institutional review may be combined with review of up to two programmes (see below) by the same review commission (Presidência da República, 2017, p. art.19[1]).

For the institutional review, the evaluation commission uses specific evaluation criteria and scoring detailed in an evaluation template (“instrument”) for accreditation developed by INEP (INEP, 2017[2]). The evaluation process implemented by peer reviewers is organised around five axes, and assesses the proposed institution against 45 qualitative indicators, each of which is evaluated on a five-point scale. Some of the indicators apply only to distance education and others only to physical campuses, meaning a given institution is only ever assessed against a maximum of 41 indicators.

Of the 45 indicators in total, 30 could best be classified as planned inputs to institutional management and the educational process (general institutional policies, staffing, infrastructure and equipment) and another 15 as planned processes (policies for more specific institutional processes and proposed activities). For obvious reasons, real outputs cannot be considered in this form of ex-ante evaluation. The principal foci of institutional accreditation are the Institutional Development Plan (PDI) (30%), planned academic policies (20%), planned management policies (20%) and infrastructure (20%). The final score generated by this evaluation, on a scale of one to five, is referred to as the “institutional score” or Conceito Institucional (CI). Institutions need a score of at least three to receive accreditation from SERES.

Table 4.2. Indicators used for on-site inspections for institutional accreditation

Axis	Number of indicators	Weight
Planning and institutional evaluation	3	10
Institutional development	7*	30
Academic policies	10	20
Management policies	7	20
Infrastructure	18**	20
Total	45	100

Note: * 2 indicators apply only to distance education institutions; ** 4 indicators apply only to distance education institutions and 4 only to campus-based institutions.

Source: OECD calculations based INEP (2017) External institutional evaluation instrument - classroom-based and distance – accreditation, adopted by INEP in October 2017 (INEP, 2017[2]).

According to SERES, 829 institutional accreditation reviews were performed in the period 2015-2017, and on average two to three courses associated with each institutional proposal were reviewed (2 013 courses in total). SERES promulgated 208 acts of accreditation in 2017. It has not reported the number of accreditation proposals that it modified or rejected.

Institutional accreditation for private colleges is valid for a period of three to five years; depending on the CI score (three to five) they receive. After this period, colleges must undergo a process of re-accreditation (recredenciamento), which formally applies to all types of private and public institution, and which we discuss in Chapter 7.

Effectiveness and efficiency: use and effects of the current system

The cost of institutional accreditation falls exclusively on private higher education institutions, since public institutions are exempt from its requirements. Private institutions bear the direct financial costs of complying with accreditation requirements, including costs for staff and renting premises in the period between submission of an accreditation request and approval to begin operations.

Overall, however, the institutional accreditation requirements in place since the adoption of the 2004 legislation on SINAES do not appear to have created excessive barriers to the market entry of private higher education providers in Brazil. As highlighted in Chapter 3, data on enrolment and the number of HEIs show that Brazil’s higher education system has grown swiftly over the last decade, and private sector institutions have provided the majority of new study places. Moreover, higher education leaders with whom the review team met, including representatives of private institutions, did not report that accreditation requirements resulted in unmet demand among prospective students.

Well-designed accreditation of new higher education providers brings benefits to society, protecting students, as consumers, from poor or fraudulent provision. In contrast to some other countries in the Latin America and Caribbean region (IESALC, 2017[3]), compliance with Brazil’s system of institutional accreditation appears to be nearly universal. Private institutions do not frequently operate without institutional accreditation. Moreover, the requirements of institutional accreditation appear to be sufficiently rigorous to limit fraudulent or grossly unqualified private institutions from entering the higher education marketplace.

Nonetheless, there are examples of accredited higher education institutions offering programmes that are not authorised, and organisations that are not accredited higher education institutions offering fraudulent diplomas (Governo do Brasil, 2017[4]). While the Ministry’s e-MEC platform provides a single national registry of accredited institutions and authorised programmes, it is primarily an administrative database. Incidents of allegedly fraudulent provision suggest that not all students have ready access to information that allows them to confirm the validity of the institutions and programmes in which they plan to study. While the layout and functionality of the e-MEC site are not designed to be used by students and their families, the information contained in the system could easily be exploited as part of a more user-friendly information service for students.

There are specific concerns in Brazil – as in other countries – regarding the expansion of distance education, including HEIs that only provide distance programmes. Distance education now accounts for almost 20% of total enrolment in Brazil, with over 90% provided by the private sector. Private distance education institutions and the programmes they provide are subject to the same procedures for institutional accreditation (and programme-level authorisation and recognition – see below) as providers of traditional classroom-based higher education. A limited number of qualitative indicators relating specifically to distance education have been incorporated into the evaluation templates used for accreditation, covering pedagogical approaches, digital technologies and infrastructure.

Brazilian legislation requires distance education programmes to respect the requirements of national curriculum guidelines (DCNs), for fields where these exist, and distance programmes have hitherto mostly been blended programmes, with some face-to-face instruction and assessments, often conducted in decentralised distance education learning centres (referred to as “poles”). This pattern might be considered to be positive, as internationally, blended programmes have been shown to be more effective than fully online programmes (Escueta et al., 2017[5]).

However, recent legislative changes have made it easier for private higher education providers to establish large numbers of distance education “poles” (up to 250 a year), in multiple locations, without the need for the facilities in each location to be inspected by INEP evaluators (Presidência da República, 2017[6]; MEC, 2017[7]). Some stakeholders in Brazil are concerned that this will promote the uncontrolled expansion of distance education, without adequate quality guarantees (Estadão, 2018[7]). Furthermore, the specific evaluation criteria for distance education institutions (and programmes) used currently are few in number and underdeveloped in light of the risks associated with this kind of provision (limited staff-student interaction, the risk students are isolated, the challenges of organising fair and rigorous assessments and examinations, etc.).

Creation of new programmes: authorisation and recognition

Relevance: the rationale and objectives of the current system

The principal focus of quality assurance with respect to new provision or “market entry” in Brazil is the offer of new undergraduate study programmes by private HEIs. Recent data show that the majority of new programmes are created in the private sector. Indeed, between 2015 and 2016, the number of registered classroom-based and distance undergraduate programmes in private HEIs increased by 1 092 to 23 824 (a 5% increase), while the number of programmes in federal universities fell by 161 to 4 876 (a 3% fall) (INEP, 2018[6]).

MEC closely supervises the establishment of programmes through processes of ex-ante authorisation (autorização) for a proportion of new programmes in colleges and recognition (reconhecimento) of all new programmes in all types of HEI (in practice, mostly in private HEIs), once they have completed at least half the teaching hours of their first cycle of operation.

In principle, new courses enter the higher education system after careful and integrated scrutiny: on-site reviews at the stage of authorisation (in some cases) and recognition (in all cases) examine the conditions in place that affect the supply of the programme. These initial checks of the supply conditions for programmes remain the primary guarantor of the quality of undergraduate programmes in Brazil. They are later complemented by ongoing monitoring of a small set of indicators, based primarily on the results of assessment of student learning outcomes using the National Examination of Student Performance (ENADE) and periodic “renewal of programme recognition” that follows a three-year cycle. In cases where quality problems are detected through the monitoring indicators, or through other sources, such as complaints, INEP conducts further on-site visits for renewal of recognition. We discuss these later quality checks in Chapter 5.

Authorisation

Colleges must seek authorisation of all new programmes before instruction begins. The standard authorisation process starts with analysis by SERES of documents relating to the new programme submitted by the HEI. These documents include a “Programme Pedagogical Project” (projeto pedagógico do curso, PPC), setting out the programme structure, number of study places (vagas), proposed curriculum, teaching methods and use of technology. In addition, HEIs provide details of proposed teaching staff and proof of available teaching spaces, including, where relevant, decentralised learning centres (“poles”) for distance education (Presidência da República, 2017, p. art.43[1]). Subsequently, in the standard authorisation procedure, an on-site review is organised by INEP. This review is undertaken by external reviewers from the BASis database, using the dedicated evaluation template for authorisation (INEP, 2017[7]), discussed below. The evaluation report and a score, referred to as a Conceito de Curso (CC) and awarded on a scale of one to five, are then transmitted to SERES. Finally, SERES reviews the evaluation report, approves the CC and makes a decision to authorise the programme, if the CC is three or above, or to refuse authorisation.

The authorisation of new programmes proposed by colleges is a risk-adjusted process. Recent changes to the regulatory regime allow colleges to obtain authorisation for new courses under certain circumstances based exclusively on a desk-based analysis by SERES of the programme documents submitted by the HEI, without undergoing an on-site inspection. Colleges with the minimum institutional quality score (CI) of three can start up to three new programmes a year without on-site reviews, provided they already have officially recognised (i.e. quality assured) programmes in the same disciplinary field. Colleges with institutional quality scores of four and five are permitted to create, respectively, up to five and ten new programmes a year under the same lighter regulatory conditions, in fields where they have existing quality assured programmes (MEC, 2017[8]).

Programmes in law, medicine, dentistry, psychology and nursing form a major exception to the general principles guiding authorisation. Programmes in these fields now always require prior authorisation from MEC and an on-site authorisation review, even in university centres and universities (Presidência da República, 2017, p. art.41[1]). To inform its decisions in these fields, MEC takes advice from, respectively, the federal council of the Brazilian Bar Association (Ordem dos Advogados do Brasil, OAB) and the National Health Council (Conselho Nacional de Saúde). Compared to previous legislation, the 2017 decree on quality assurance (Presidência da República, 2017[1]) extended the requirement for systematic authorisation with on-site inspections to nursing programmes and also made increases in the number of study places in all undergraduate law and medicine programmes dependent on MEC authorisation (SEMESP, 2017[9]).

Recognition

All HEIs, including university centres and universities, must seek regulatory “recognition” (reconhecimento) from SERES for every undergraduate programme when a programme’s first cohort of students has completed between 50 and 75 percent of the workload of the course. Recognition is needed for the diplomas issued to graduates of the programme to be valid nationally in Brazil. A separate recognition process is required for each programme offered in campuses outside the municipality where the HEI has its headquarters. Formally, the process of recognition also applies to federal public institutions. In practice, the low levels of programme creation in the federal public sector mean federal public institutions are comparatively rarely involved in processes of recognition.

HEIs are required to submit requests for the recognition of their programmes to SERES, providing the same set as documents as is required for authorisation of programmes (programme pedagogical project etc.). SERES undertakes a desk-based analysis of these documents and INEP organises an on-site review process by external evaluators, who use a separate evaluation instrument to rate programmes and generate a programme quality score (Conceito de Curso, CC) on a scale of one to five (INEP, 2017[10]). For programmes in law, SERES seeks an opinion from the Brazilian Bar Association, and for programmes in Medicine, dentistry, psychology and nursing, from the National Health Council, on the decision to recognise the programme.

If the CC score is at least three and, where relevant, the Bar Association and Health Council issue positive opinions, SERES confirms the CC and issues an official recognition of the programme. For programmes previously subject to authorisation, the CC resulting from the recognition process becomes the new quality score for the programme.

If the result of the on-site evaluation is negative (a score of two or less), SERES requires the HEI to draw up a “Commitment Protocol” (protocolo de compromisso) which sets out how the quality problems detected will be addressed within a 12-month timeframe (Presidência da República, 2017, p. art.54[1]). This stage of the supervisory process is referred to as a “remediation procedure” (procedimento saneador). If it considers there is an immediate risk for students, SERES may also impose one or more sanctions (medidas cautelares), including suspension of new student intakes (see Box 4.1). Internal data transmitted to the OECD team by SERES suggests these kinds of sanction are virtually never applied in remediation procedures.

At the end of the period established by the Commitment Protocol, the programme is subject to another on-site inspection by INEP evaluators. If it still fails to meet minimum quality requirements, SERES launches a “sanctioning procedure” (procedimento sancionador), which may entail the same sanctions mentioned in Box 4.1 (Presidência da República, 2017, p. art.73[1]). For serious cases in private institutions, the relevant legislation allows for the withdrawal of institutional accreditation, which would effectively lead to the closure of the institution. Again, in practice, such cases are rare. Legally, some of the sanctions can be applied to public institutions, but the legal status of these institutions as public bodies means they may not have their institutional accreditation withdrawn.

Box 4.1. Sanctions used by SERES in supervision of (private) higher education

I - suspension of admission of new students;

II - suspension of the offer of undergraduate or lato sensu postgraduate programmes;

III - suspension of the autonomy of the HEI;

IV - suspension of the prerogative to create new distance education poles by the HEI;

V - suspension of regulatory processes that the HEI or other HEIs owned by the same operating organisation have submitted to SERES;

VI – prohibition of filing new regulatory processes to SERES by the HEI or other HEIs owned by the same operating organisation;

VII - suspension of the HEI’s right to enter into new Student Financing agreements as part of the FIES system;

VIII - suspension of the HEI’s right to participate in a selective process for the offer of scholarships from the University for All Programme (PROUNI);

IX - suspension or restriction of the HEI’s right to participate in other federal access programmes.

Source: Article 63 of Decree 9 235 of 15 December 2017 (Presidência da República, 2017[1])

Effectiveness: indicators used for authorisation and recognition of new courses

The on-site evaluation templates (“instruments”) used by external reviewers for the processes of authorisation and recognition were revised by INEP in October 2017. They establish nearly identical review templates. The judgement criteria in the template for authorisation (INEP, 2017[7]) focus on planned inputs and activities, while those in the template for recognition (INEP, 2017[10]) refer to real inputs and activities, verified in practice by the external review commission sent by INEP.

Both templates direct reviewers to evaluate programmes on three dimensions: the proposed pedagogical approach and organisation of the programme (Organização Didático-Pedagógica); the instructional workforce (Corpo Docente e Tutorial); and infrastructure (Infraestrutura). The assessment of the pedagogical approach and organisation of the programme considers the extent to which the planned curriculum meets the requirements of subject-specific National Curriculum Guidelines (Diretrizes Curriculares Nacionais, DCN) approved by the National Education Council, which exist for many, but not all, disciplines in higher education. Together, these dimensions contain more than 50 indicators, some of which apply to all programmes, others specifically to distance education or classroom-based programmes, and still others to programmes offering clinical or field-based learning. Crucially, peer reviewers are responsible for scoring the indicators on a five-point qualitative scale.

The 50+ indicators used in each on-site evaluation template focus either on programme inputs (teaching staff, infrastructure) and processes (pedagogical processes, support to students etc.). Although both instruments assess the expected “profile” specified for the graduates that the programmes educate, they do not consider programme outputs. This choice is necessary because the reviews take place prior to, or midway through, the study programme for the first student cohort. Other than the distinction between planned and real inputs and activities, the principal difference between the two evaluation instruments rests with the weight they assign to the indicators, with the instructional workforce taking on greater weight in the process of recognition (40%) than in authorisation (20%).

The indicators assessed under the “teaching staff” dimension of the evaluation instruments focus primarily on the qualifications of staff, their employment status and the extent to which their profiles match the needs of the programme. The choice to assign greater weight to assessment of these factors during the recognition process than in authorisation reflects the fact that the staff will actually be in place and working at the time of recognition, so the composition of the teaching workforce can be judged more accurately. However, the judgement criteria reward the presence of full-time staff with doctoral degrees and attach little value to professional experience, thus disadvantaging professionally oriented programmes. At the same time, relatively little weight is attached to assessment of the pedagogical and didactic approaches implemented by the programme, despite their crucial role in supporting students to acquire relevant learning outcomes.

The weighted sum of scores provided in the on-site review are used to calculate the programme score, the Conceito de Curso (CC), the value of which ranges from one to five, and provides the basis for SERES to authorise or recognise new programmes, or not.

Table 4.3. Indicators used for authorisation (autorização)

Dimension	Number of indicators	Weight
Pedagogical and didactic organisation of the programme	24	40
Instructional workforce	14	20
Infrastructure	16	40
	54

Source: INEP (2017) Evaluation instrument for undergraduate programmes - classroom-based and distance – authorisation (INEP, 2017[7])

Table 4.4. Indicators used for recognition (reconhecimento)

Dimension	Number of indicators	Weight
Pedagogical and didactic organisation of the programme	24	30
Instructional workforce	16	40
Infrastructure	18	30
	58

Source: INEP (2017) Evaluation instrument for undergraduate programmes - classroom-based and distance – recognition and renewal of recognition (INEP, 2017[10])

The on-site evaluation templates now make special provision for the authorisation and recognition of distance education courses. For example, in evaluating the pedagogical and didactic organisation of the programme, three indicators consider respectively the tutoring approach, tutoring staff and virtual learning environment for distance education programmes (INEP, 2017[7]; INEP, 2017[10]). Under the section on instructional workforce, peer reviewers are also to assess whether programmes have staff with “experience in teaching in distance education” and “experience of tutoring in distance education” (in this context, “tutor” is used to refer to teaching assistants, who support core academic teaching staff in delivering a programme). Infrastructure indicators have, likewise, been augmented to take account of distance education programmes.

However, 45 out of 55 indicators in the templates are applicable to both classroom-based and distance programmes. The specific indicators of programme quality related to curriculum, instruction, learning support, and assessment in distance programmes are less developed than those used in accreditation systems in other OECD and partner countries, including the United States (DEAC, 2018[11]). Developing appropriate measures of quality that reflect the specific characteristics of distance education is, however, a challenge shared by many higher education systems.

Effectiveness and efficiency: use, effects and efficiency of programme authorisation and recognition

What are the principal effects of programme authorisation and recognition for Brazilian higher education? First, authorisation and recognition play a critical role in regulating the enrolment capacity of the nation’s higher education system. Through these processes, private higher education institutions propose the number of study places they believe programmes can properly accommodate in their Programme Pedagogical Project (PPC). The on-site reviews evaluate supporting evidence for this claim3, and their assessment is confirmed by SERES, which officially determines how many study places the programme may have.

The impact of the authorisation and recognition processes on the quality of programmes provided is much more difficult to ascertain. There are no programmes in private institutions in Brazil that have not been subject to recognition that could form a comparison group with which to compare recognised programmes in an effort to analyse the quality effects of the regulatory and evaluation processes. There are no other readily available and comparable indicators of programme quality in Brazil that would provide an alternative means to assess programme quality - and make it possible to judge whether recognition provides an effective guarantee of quality.

The formal requirement for all courses to obtain official recognition in the early stages of their operation provides a basic guarantee of the quality of programmes. The procedures in place force higher education providers to reflect seriously about the design of the programmes they are providing and put in place a range of documents and processes – described in the programme pedagogical project – that should contribute positively to the delivery of relevant and high quality programmes. Nevertheless, the factors verified through the on-site evaluation at the stage of recognition are all conditions for the delivery of quality programmes, but do not provide a guarantee that programmes deliver high quality education in practice.

Moreover, the processes used to evaluate the quality of new programmes are subject to four principal lines of criticism with respect to their reliability, usefulness, and cost effectiveness.

First, representatives of private institutions consulted by the OECD review team complained that external reviewers appointed by INEP to implement on-site reviews very frequently come from public universities, while the vast majority of authorisation and recognition processes occur in private institutions. More generally, institutional representatives argued that those who are called upon to carry out reviews sometimes lack expertise with respect to the programme under review. In the first case, reviewers may bring to private institutions unfamiliarity with their circumstances, or even an active hostility to their institution. In the second, the risk is that reviewers lack sufficient expert knowledge to make sound judgments about the didactic and pedagogical profiles of the programmes they review.

INEP claims that recent improvements to the BASis database of reviewers and the rules governing the allocation of external experts should address both concerns. The ordinance governing on-site evaluation requires that review commissions for different types of programme have direct experience with programmes in the same field and mode of provision (MEC, 2017[12]). Moreover, a recent “administrative instruction” (MEC, 2017[13]) requires that at least one reviewer in institutional reviews in private HEIs has experience in a private HEI. It has not been possible to determine, however, to what extent these rules are applied in practice and, in the latter case, whether experience in the private sector is required for programme-level reviews in private institutions.

Second, there is concern about the subjectivity or unreliability of qualitative assessments. The process of on-site review for programme authorisation and recognition (as revised in late 2017) asks reviewers to make qualitative judgments on a five-point Likert scale, using pre-formulated judgement criteria. Despite the attempts by INEP to formulate the judgement criteria clearly, these scales still leave considerable room for interpretation. They call upon reviewers to make distinctions that that are likely to be inconsistent between individuals. The OECD review team was told by campus officials that the same programme offered in different campuses with otherwise near-identical supply conditions received different marks from on-site reviewers.

Third, the OECD review team heard frequent criticisms from institutional representatives of the delay and burden associated with the on-site review process for authorisation and recognition. Describing the whole system of on-site reviews as it functioned in 2012 (before recent reforms), a document written by the Association of Private Higher Education, ABMES, notes:

The evaluation system is nearing collapse. INEP holds approximately 5 000 assessment visits per year, or about 100 per week. The logistics to support an operation of this size, nationwide, and every day is overwhelming. For example, there are more than 400 flights per week to be scheduled, budgeted, accounted for and issued by INEP. Yet, for a system with nearly 30 000 undergraduate programs and 3 000 institutions, not counting new authorization and accreditation procedures for courses and institutions, 5 000 visits are insufficient. This causes crowding of the evaluation system and a growing backlog. There are higher education institutions with applications for recognition awaiting for years the visits of committees. (de Magalhães Castro, 2015[14])

INEP and SERES argue that the situation has improved since 2012. In particular, they point to the fact that HEIs that have received adequate quality scores (a CI of three or above) are exempted from on-site reviews at the stage of authorisation for programmes in fields where they already have courses (within certain limits). They argue that the most recent regulatory changes in Decree 9 235/2017 (Presidência da República, 2017[1]) reduce burden for institutions with an established quality record, allowing them to create additional study places more easily, for example.

While there has indeed been a shift in the regulatory approach, the market entry process for new undergraduate programmes in the federal higher education system remains administratively burdensome for private HEIs and the evaluation agency (INEP) when compared to equivalent processes in many OECD countries. In Brazil, despite the recent changes, all new programmes are required to go through the recognition process, with on-site reviews that depend on peer review and are logistically complex to organise.

In Anglophone OECD countries and a number of non-Anglophone European higher education systems, HEIs can create programmes and issue valid diplomas without prior authorisation (European University Association, 2018[15])4. In these and other systems, authorities often link quality review procedures more closely to risk of poor quality than is the case in the Brazilian system, with less complex procedures in place for institutions that can demonstrate they present a lower risk. Although the large private higher education sector in Brazil creates specific risks, which are not found in all higher education systems, there is certainly scope for Brazil to draw on risk management practice in other quality assurance systems5.

Finally, on-site visits carried out in support of programme recognition permit higher education institutions initially to award degrees without providing evidence about the initial performance of the programme, such as rates of attrition among its students. Additionally, the process of recognition does not systematically elicit information from the students who the programmes serve (as the ENADE process does at a later stage), or external stakeholders who have experience of working with the programme and its students, such as public sector employers and private firms which provide internships.

4.3. Key recommendations

1. Improve the reliability and visibility of information about institutions’ accreditation status to ensure students and families are well informed

Although MEC, with the support of evaluations coordinated by INEP, regulates the entry of new institutions into the Brazilian higher education marketplace more comprehensively than in other systems undergoing rapid expansion, the quality assurance system is not fully effective in preventing fraudulent and unauthorised provision. The first line of defence against unaccredited higher education providers is students themselves. Informed students understand which institutions are accredited and not, and why this matters to them, and are able to identify and avoid unaccredited institutions. In principle, comprehensive information about accredited institutions and recognised programmes is available through the online e-MEC. However, e-MEC is not a user-friendly source of accreditation information. More accessible public Internet resources found in other higher education systems could serve as references for the Brazilian authorities in this regard (UK Government, 2018[16]). In the medium-term, the aim should be to develop a comprehensive online portal providing students and prospective students not only with programme-level information on quality assurance results, but also on issues such as graduation rates and graduate employment outcomes (see discussion on programme indicators below).

2. Over time, increase the focus on institutions as units of evaluation in the external quality assurance system to reduce burden, while maintaining effectiveness

We have noted four main concerns about the processes used to authorise and recognise new study programmes. Despite attempts to address concerns about the composition of review commissions and reduce requirements for authorisation in some cases, the Brazilian system of programme review at market entry remains complex and burdensome and may not represent the best use of the country’s resources. There is a need for a system in which the burden and benefit of new programme approval are re-balanced.

Permitting institutions themselves to play a wider role in assuring the quality of programmes, while maintaining an enhanced system of programme level monitoring indicators, could significantly reduce the burden of programme approval through authorisation and recognition. Programme-focused regulatory decisions – for new and existing programmes - account for more than 10 000 of the 12 000 acts that SERES handles annually.

The Brazilian system of quality assurance currently focuses proportionally more efforts on the programme-level than on the institutional level as a unit of evaluation and monitoring. To some extent, the current system regards HEIs as “holding units” for programmes, which are then the main focus of detailed analysis in the quality assurance system. In Chapter 7 we take up the question of institutional quality, and make suggestions for a more rigorous and comprehensive process of institutional reaccreditation, with a view to permitting higher education institutions with demonstrated capacity to assume responsibility for quality of the programmes that they offer and to become “self-accrediting institutions”.

3. In the near term, take steps to improve the evaluation process for programmes that remain subject to programme-level authorisation and recognition

The OECD review team sees a clear case for maintaining programme-level authorisation and strict market entry requirements at programme level for HEIs that lack a strong track record of good quality provision and are not able to demonstrate adequate capacity to self-accredit their own programmes. It is thus important to increase the effectiveness of these processes in promoting quality practices for institutions that remain subject to programme-level authorisation and/or recognition. Priorities for improving current practice in the short-term include:

• Further improving the criteria used to select and assign peer reviewers for on-site reviews to increase the fit between reviewer expertise and programme review responsibilities. It is particularly important that reviewers for professionally oriented programmes have adequate understanding of the objectives and operation of such programmes and are able to make robust assessments about the quality of provision in teaching institutions that lack a traditional academic focus and research activities.

• Continuing and increasing efforts to improve the training of peer reviewers, with a view to improving the reliability and impartiality of scoring.

• Increasing the weight attached to the organisation and implementation of teaching and learning in the evaluation instrument for recognition, reflecting the importance of these factors for students.

• In cooperation with international peers, refining and expanding the specific indicators used for the evaluation of distance education programmes, so that these address the particular risks associated with this type of provision. This should consider how best to evaluate decentralised distance education centres (“poles”).

• Using the recently introduced process of feedback about the performance of peer reviewers to monitor and revise selection and training.

4. In the longer term, take steps to reduce further the burden and improve the effectiveness of quality assurance processes for programmes outside self-accrediting institutions

In the longer term, two issues should be considered in particular. First, the procedures for on-site visits could be fundamentally reformed. Responsibility for reviewing institutional infrastructure and basic institutional policies could be assigned to a well-trained and professionalised inspectorate. The expert judgment of academic peers (who currently review all aspects of institutions and programmes) could then be applied to a more limited set of indicators than at present, focused on core teaching and learning activities. A sequenced process of accreditation and authorisation could be implemented in which a professional inspectorate initially carried out its work, and academic peers would be engaged only for institutions and programmes that have passed a first stage of review. Second, it will be important to identify ways in which the more extensive, quantitative, and comparable information about intermediate programme performance can be incorporated into the process of programme recognition. Examples include student attrition from programmes, and student feedback concerning the teaching and learning environment.

References

[17] de Magalhães Castro, M. (2015), “Higher Education Policies in Brazil: A Case of Failure in Market Regulation”, in Schwartzman, Simon Pinheiro, Rómulo Pillay, P. (ed.), Higher Education in the BRICS Countries - Investigating the Pact between Higher Education and Society, Springer, Dordrecht, http://dx.doi.org/10.1007/978-94-017-9570-8_14.

[14] DEAC (2018), DEAC Accreditation Handbook, Distance Education Accrediting Commission (DEAC), Washington, D.C., http://www.deac.org (accessed on 15 November 2018).

[5] Escueta, M. et al. (2017), “Education Technology: An Evidence-Based Review”, NBER Working Paper Series, No. 23744, National Bureau of Economic Research, Cambridge, http://www.nber.org/papers/w23744 (accessed on 23 November 2018).

[8] Estadão (2018), Governo vai fiscalizar oferta de curso a distância (Government will investigate supply of distance education courses), Estadão, https://educacao.estadao.com.br/noticias/geral,governo-vai-fiscalizar-oferta-de-curso-a-distancia,70002327119 (accessed on 02 December 2018).

[18] European University Association (2018), University Autonomy in Europe, https://www.university-autonomy.eu/ (accessed on 15 October 2018).

[4] Governo do Brasil (2017), MEC identifica irregularidades em cursos de 27 instituições de ensino superior (MEC identifies irregularities in programmes in 27 HEIs), News site of the Brazilian Government, http://www.brasil.gov.br/noticias/educacao-e-ciencia/2017/06/mec-identifica-irregularidades-em-cursos-de-27-instituicoes-de-ensino-superior (accessed on 14 November 2018).

[3] IESALC (2017), Educación superior y sociedad : Aseguramiento de la Calidad en America Latina, Instituto Internacional de Unesco para la Educación Superior en América Latina y el Caribe (IESALC), Caracas, http://www.revencyt.ula.ve (accessed on 14 November 2018).

[9] INEP (2018), Sinopses Estatísticas da Educação Superior 2017 – Graduação - INEP (Statistical Sinopsis for Higher Education 2017), http://portal.inep.gov.br/web/guest/sinopses-estatisticas-da-educacao-superior (accessed on 13 November 2018).

[13] INEP (2017), Instrumento de Avaliação de cursos de graduação Presencial e a distância - Reconhecimento e Renovação de Reconhecimento (Evaluation instrument for undergraduate programmes - classroom-based and distance - recognition and renewal of recognition), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

[10] INEP (2017), Instrumento de Avaliação de cursos de gradução - Presencial e a distância - Autorização (Evaluation instrument for undergraduate programmes - classroom-based and distance - authorisation), http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

[2] INEP (2017), Instrumento de Avaliação Institucional Externa - Presencial e a distância - Credenciamento (External institutional evaluation instrument - classroom-based and distance - accreditation), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira , Brasília, http://download.inep.gov.br/educacao_superior/avaliacao_institucional/instrumentos/2017/IES_credenciamento.pdf (accessed on 11 November 2018).

[16] MEC (2017), Instrução normativa No 2, de 18 de dezembro de 2017 (Normative instruction 2 of 18 December 2017), Ministério da Educação / Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://portal.imprensanacional.gov.br/web/guest/consulta?p_p_id=101&p_p_lifecycle=0&p_p_state=maximized&p_p_mode=view&_101_struts_act…1/5 (accessed on 15 November 2018).

[11] MEC (2017), Portaria Nº 20, de 21 de dezembro de 2017. (Ordinance 20 of 21 December 2017, relating to procedures and decision-making standards for the processes of accreditation, re-accreditation, authorisation, recognition and renewal of recognition of classroom-based and distance higher education programmes, as well as modifications to these, in HEIs in the federal education system), http://www.in.g (accessed on 14 November 2018).

[15] MEC (2017), Portaria Normativa No 19, de 13 dezembro de 2017 Dispõe sobre os procedimentos de competência do Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira - INEP referentes à avaliação de instituições de educação superior, de cursos de graduação e de desempenho acadêmico de estudantes., Ministério da Educação, Brasília, http://www.abmes.org.br (accessed on 15 November 2018).

[7] MEC (2017), Portaria Normativa no.11, de 20 de junho de 2017 - Estabelece normas para o credenciamento de instituições e a oferta de cursos superiores a distância, em conformidade com o Decreto nº 9.057, de 25 de maio de 2017., http://www.in.gov.br/autenticidade.html (accessed on 04 December 2018).

[6] Presidência da República (2017), Decreto Nº 9.057, de 25 de maio de 2017 Regulamenta o art. 80 da Lei nº 9.394, de 20 de dezembro de 1996, que estabelece as diretrizes e bases da educação nacional. (Decree 9057 concerning distance education), http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9057.htm (accessed on 02 December 2018).

[1] Presidência da República (2017), Decreto Nº 9.235, de 15 de dezembro de 2017 - Dispõe sobre o exercício das funções de regulação, supervisão e avaliação das instituições de educação superior e dos cursos superiores de graduação e de pós-graduação no sistema federal de ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of regulation, supervision and evaluation of higher education institutions and undergraduate and postgraduate courses in the federal education system), http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on 10 November 2018).

[12] SEMESP (2017), Decreto no 9235/17: Análise comparativa, SEMESP, São Paulo, http://www.semesp.org.br/wp-content/uploads/2017/12/Comparativo-Decretos-21.12.pdf (accessed on 15 November 2018).

[20] TEQSA (2018), Risk Assessment Framework - Tertiary Education Quality and Standards Agency, Tertiary Education Quality and Standards Agency, https://www.teqsa.gov.au/risk-assessment-framework (accessed on 15 November 2018).

[19] UK Government (2018), Check if a university or college is officially recognised - GOV.UK, https://www.gov.uk/check-a-university-is-officially-recognised (accessed on 15 November 2018).

Notes

← 1.  In 2016, there were 1 866 private colleges (“faculdades”) in Brazil and only four federal public colleges. In addition, there were 134 state and municipal public colleges, but these are not subject to the federal regulatory and quality assurance system for institutions and undergraduate programmes (Source: INEP).

← 2.  In 2016, there were 63 federal public universities, 89 private universities and 156 private university centres in Brazil. No federal public institutions had the status of “university centre”.

← 3.  Peer reviewers use indicator 1.20, “Número de vagas” (number of study places), to evaluate programme capacity. An on-site evaluation with a score of three or higher authorises the programme to have the number of study places proposed in the PPC. The programme may subsequently seek additional study places by submitted a request to SERES, a new on-site evaluation and another regulatory act called “Aumento de vagas” (increase in study places).

← 4.  This is the case in Austria, Ireland, Luxembourg, Norway, Poland, Sweden, Switzerland, United Kingdom (European University Association, 2018[15])

← 5.  Australia, for example, has a highly developed approach to risk in quality assurance, outlined by the Tertiary Education Quality and Standards Agency in their Risk Assessment Framework (TEQSA, 2018[20])