Chapter 14: Institutional management and quality audit: the experience in Chile – External Quality Audit

14

Institutional management and quality audit: the experience in Chile

María José Lemaitre, Mario Maturana, Elisa Zenteno and Andrea Alvarado

Abstract:

After a decade and several cycles of institutional accreditation carried out under a quality audit approach, it was considered necessary to find out what effects – if any – were apparent in higher education institutions. A review of accreditation decisions in ten universities in Chile, with different accreditation results, yielded interesting information about changes in governance and managerial practices. Interviews with representatives of the universities that showed a more marked improvement identified good practices with regard to the review and operationalization of mission statements, organizational structure and decision-making policies and procedures, and academic management. The study also suggested that institutions tend to follow a developmental itinerary, which can be useful in the review and implementation of quality assurance processes.

Key words

quality audits

experience of audits in Chile

quality assurance and assessment

Development of external quality assurance schemes in Chile: an answer to the challenges of higher education evolution

Traditionally, higher education in most Latin American countries developed without any thought to the need for external quality assurance mechanisms. While higher education was restricted to the most qualified students in their respective countries, taught by highly respected faculty who had degrees obtained from the most prestigious universities in Europe and the United States, quality was assured through a sort of ‘corporative self-regulation’ of institutions enjoying the privileged autonomy that comes with guaranteed public income and no interference from public officials in academic, administrative or financial matters.

Such was the situation in Chile until 1980. Higher education was the responsibility of eight universities, two of which belonged to the State,1 three to the Catholic Church and three to private foundations.2 They were organized according to the French model, with a strong emphasis on professional training, and since 1965 had developed an effective research capability. Their students were selected through a national admissions test and about 67 per cent of their budget came from the government. About 10 per cent of the 18–24 cohort were enrolled in higher education, and these students paid only token fees. Chilean universities enjoyed a high level of prestige and recognition in Latin America and their graduates entered graduate programs in the United States and Europe with very good results. As mentioned above, no external quality monitoring seemed necessary, but quality was constantly being monitored and assured from within.

In 1980 the military government that had taken power in 1973 put in place a comprehensive reform of higher education, in a context that made public debate impossible.

The main changes had to do with the establishment of new, public universities, developed from the regional branches of the state universities; the authorization to set up private higher education institutions (HEIs); the reduction of public funding and the requirement that HEIs become self-sufficient through new sources of income, one of which was student fees, established at all (state and private) institutions. The system was further diversified into three institutional tiers: universities, which granted professional and academic degrees; professional institutes, which could offer professional but not academic degrees; and technical training centers, offering two-year technical degrees.

For the next decade, regulation was left to the market, with the understanding that students would prefer ‘good’ institutions and that those offering a poor service would therefore be forced to close down. Soon it became evident that the market could not ensure quality, or provide a measure of social legitimacy to private institutions: they came to be known as institutions for ‘rich fools’, that is, for students without the qualifications for a selective, public institution and with the means to pay for a private one.

The same military government (and its neoliberal advisors) that had de-regulated higher education realized that external quality assurance was unavoidable, and in 1990, on the last day of its rule, it passed a law putting in place a mandatory licensing scheme. All new private HEIs had to go through a rigorous supervisory period of a minimum of 6 years and a maximum of 11 years. After this, they could be fully licensed (and thus authorized to offer freely all kinds of programs), or they had to be closed down.

During the 1990s enrolment grew to about 30 per cent of the age cohort and it was considered necessary to expand quality assurance processes to all institutions, both public and private. Therefore, in 1999 the government set up a National Commission for Accreditation charged with voluntary program accreditation for all HEIs.

In a context of fierce competition for students, accreditation was seen as a sound differentiation mechanism. However, it soon became clear that program accreditation was too slow and too expensive to really make a difference in terms of providing public assurance of the quality of higher education.

Institutional accreditation seemed a good solution, but since many HEIs had already gone through a thorough evaluation under the licensing process, it was designed under a quality audit approach: the focus of institutional accreditation3 is the formal establishment and the effective operation of institutional policies and mechanisms to work towards the fulfillment of their stated purposes. All institutions applying for accreditation must provide evidence that they have adequate mechanisms for self-regulation in the fields of teaching at degree level and institutional management. Institutions may add other fields, linked to their mission statement, such as graduate studies, research or continuing education (CNA-CHILE, 2011).

Institutions wishing to be accredited under the quality audit approach must provide evidence that

 they have clearly defined purposes, which provide useful guidelines for institutional decision making processes and are translated into formally established policies and mechanisms;

 these policies and mechanisms are effectively implemented at all institutional levels (all departments, all branches, all modes of teaching);

 they achieve results which are consistent with the stated purposes;

 the institution has the ability to make the necessary changes and adjustments in order to advance systematically towards achieving its stated purposes (including changing these purposes if needed).

Institutions can be accredited or denied accreditation. Accreditation has a validity of a maximum of seven years and a minimum of two years, depending on the degree of consolidation of the quality policies and mechanisms in place.

The experimental system set up in 1999 operated until 2007, when a law was passed setting up a complex quality assurance system that includes licensing, program accreditation and institutional audit. At this date, all universities and most non-university HEIs have gone through the institutional accreditation process.

A study to learn about the impact of quality audits

Methodology

In order to learn about the possible effects of quality audits on HEIs, a study was carried out covering ten universities that had gone through two accreditation cycles. These universities were divided into three groups: those that had not shown improvement between one cycle and the next; those that had improved; this second group was further divided into universities showing marked improvement and universities where improvement was clear but not sufficient to move into the higher levels of accreditation (Figure 14.1).

Figure 14.1 HEI audits-accreditation cycles

Criteria for selecting institutions for the study

The first stage of the study focused on institutional policies and managerial practices and addressed all ten universities. Information was obtained through an analysis of the accreditation reports prepared by the quality assurance agency and the responses to a questionnaire applied to institutional leaders.

The second stage was carried out through case studies, addressing three of the universities that were showing strong improvement, and one in the group showing slight improvement (see Figure 14.1).4 Here information was obtained through in-depth interviews with institutional leaders in both the academic and the administrative fields and the main purpose was to identify good practices in institutional management.

Results

An analysis of the first stage

The first stage provided evidence that there were significant differences between the more consolidated institutions and the rest. The main changes had to do with the establishment of technical units to support accreditation processes, the growing links between the assessment results and institutional planning and the improvement in internal managerial practices.

It is interesting to note that all institutions reported making changes in their organizational structure, leading to a better division of labor and increased coordination. However, while in the developing institutions these changes were mostly formal adjustments, the more consolidated ones actually introduced new practices, allocated human and financial resources, trained their academic and support staff and increased the participation of internal and external stakeholders in their decision-making processes. Less consolidated HEIs tended to limit participation to gathering the opinions of students and staff, thus reducing the opportunities for increased ownership of the results of the evaluation.

Results were less evident regarding the teaching and learning process; however, the more consolidated HEIs stated that they applied systematic reviews to their programs, mostly through the development of substantive teaching policies and curricular changes. They also reported increased allocation of resources to teaching. Developing institutions showed some actions in this respect, but these were not implemented systematically, and the lagging institutions provided no evidence of explicit policies regarding the improvement of teaching.

These results were consistent with an international study that was being carried out on the impact of quality assurance processes (CINDA, 2010),5 and they helped to define the issues that would be further explored in the interviews during the second stage of the project.

Good practices for institutional management: results from the second stage

The interviews showed that going through the institutional accreditation process – which follows the quality audit model – helped institutions to develop a shared set of good practices. These are summarized below.

Mission statements

These used to be mostly formal statements that did not have a direct link to institutional policies and mechanisms. Most institutions reported that going through the audit process helped them to translate their mission statements into clear and explicit goals and objectives, which were then used as a framework for action.

Organizational structure

All institutions reported significant changes in organizational structure, in order to improve alignment with redefined institutional priorities. These included the improvement of strategic planning and quality assurance processes, but one of the most interesting developments was the establishment of institutional research units. These tend to be highly placed (at the level of rectors or vice rectors), focused on the gathering, processing and provision of information to support decision making at all levels in the HEI. In most cases, these units developed initially to support self-assessment exercises, but in all of the consolidated institutions they developed into technical units with a much wider and strategic role.

Decentralization

The development of institutional research units helped to decentralize decision making and contributed to an increased professionalization of institutional management. At the same time, involvement in quality assurance practices led to increased participation of internal and external stakeholders. Internal stakeholders showed an increased role in decision making on issues closely related to their work, and external stakeholders were increasingly invited to provide inputs on curricular and other areas. This also led to a better understanding of institutional goals, and therefore a stronger commitment to their achievement.

Development of quality-related policies and mechanisms

A significant result was the increased alignment between institutional goals and internal actions and mechanisms to achieve them. Most institutions turned implicit practices into explicit policies and mechanisms, thus making it easier to monitor their implementation and evaluate whether they were actually being applied in different institutional units or levels.

Institutional planning and financial management

The management of always-scarce resources is one of the central concerns of HEIs. Most institutions reported that quality assurance helped to develop more effective and efficient administrative processes, and the combination of self-assessment and its validation through external reviews provided a sounder basis for financial decisions. Financial leaders reported a higher correlation between resource allocation and institutional priorities, which was also linked to a stronger valuation of the teaching function and greater investment in the development of pedagogic skills.

Information management

Quality assurance processes required the provision of information to support quality judgments. This in turn led to the identification of relevant information, gathering and processing valid and reliable data, and using these to support decision making. This was recognized as one of the main outcomes of quality assurance: it improved decision making at all levels, and helped to professionalize institutional management. However, it had demanded high investments (in technological tools, hiring and training of specialized staff), and therefore strong attention must be paid to its maintenance.

Academic management

As mentioned before, impact of quality assurance processes is much less evident in the teaching and learning process. However, most institutions reported changes in curricular design and adjustment practices. These could be seen in the establishment of institutionalized processes for periodic review of expected learning outcomes and curricular structure, as well as the involvement of internal and external stakeholders in these reviews. Some institutions also reported changing the criteria and procedures for assessing the quality of the teaching staff, infrastructure and learning resources, in order to make them more closely aligned with the needs of the programs. This effect could be seen in areas such as the following:

 a review of academic offer (programs, regional branches) in order to adjust them to quality criteria;

 review of the requirements for academic staff, in line with program and student needs;

 improvement of teaching support units and learning resources;

 curricular changes, introduction of institutional educational frameworks;

 decision to ask for program accreditation, as a measure of recognition of the added value of these processes for institutional quality.

Lessons learned and recommendations

The study provided significant information about the ways in which taking part in quality assurance processes affects institutions, and the perceived impact of the quality audit approach. In addition, by differentiating between HEIs in different stages of development it was also possible to map the itinerary followed by these institutions, which suggests a developmental pathway that can contribute to a better understanding of the implementation of quality assurance processes.

The first stage in this itinerary refers to a review of the institution’s mission and purposes, and the development of effective linkages between them and institutional planning. This seems obvious, but most institutions reported that the in-depth analysis of the actual meaning, implications and range of their mission statements that they performed in response to quality assurance requirements helped them to fine-tune their strategic goals and objectives and turn them into effective managerial tools. This is also a developmental process, which starts with a review of the mission statement, proceeds to its translation into institutional priorities and objectives, and proceeds to the establishment of systematic assessment tools. Initially, institutions respond to specific quality criteria established by the quality assurance agency; the more consolidated institutions showed that after this initial stage they moved to an internally motivated review, focused on the institution’s development plan.

A second stage is that of the review of institutional norms and procedures. Institutions begin with an increased formalization of implicit policies and mechanisms, which leads to the development of formal regulations (and sometimes to complaints of excessive bureaucratic demands); then institutions focus on monitoring the implementation of these new norms and regulations, and the more consolidated institutions move towards less bureaucratic and more substantive managerial practices, with increased decentralization and participation mechanisms.

At the same time, institutions work on the formalization of quality assurance practices; they normally start with the installation of a quality assurance unit, which provides basic information and support to self-assessment processes and the preparation of self-assessment reports; it also helps to prepare for external reviews and for the preparation of improvement plans.

This leads to another aspect of this developmental approach, dealing with the identification and use of quality indicators. Institutions begin with a basic set of indicators, usually those required by the quality assurance agency; their quality assurance units develop mechanisms to gather data and produce these indicators, which are used to support quality judgments in their self-assessment reports. Institutions that are able to process these data, translate them into useful indicators and develop a capacity for analyzing causal linkages are also those that have been identified as more consolidated. In these cases, information is understood as an essential managerial tool, and the quality assurance units evolve into institutional research units or departments. They are placed much higher in the institutional hierarchy, are adequately staffed and funded, have close links with institutional strategic planning, and provide information to support decision making on a regular basis (including, but not limited to, self-assessment processes).

As a main result, it is possible to conclude that quality assurance processes, if they are to be really effective, must make quality assurance a component of institutional management, closely embedded in institutional policies and mechanisms. Most statements of good practice for quality assurance agencies (the European Standards and Guidelines set up by the European Association for Quality Assurance [ENQA]; the Guidelines of Good Practice of the International Network for Quality Assurance Agencies in Higher Education [INQAAHE]; the guidelines of good practice developed by the Iberoamerican Network for Quality Assurance in Higher Education [RIACES]) state that quality is the primary responsibility of HEIs. Agencies must take this statement into account when developing their criteria and procedures, and work with institutions to make quality their main governance priority, and take the necessary actions to advance to an ever-increasing level of achievement of sound and relevant institutional purposes.

References

CINDA, Proyecto Alfa. Políticas Públicas y Gestión Universitaria, 2010. Aseguramiento de la Calidad, Available from: http://www.cinda.cl/proyecto_alfa/index.htm [[Accessed 3 November 2012]].

CNA-CHILE, Comisión Nacional de Acreditación, 2011. Available from: http://www.cnachile.cl [[Accessed 3 November 2012]].

ENQA Standards and Guidelines for Quality Assurance in the European Higher Education Area. Available from: http://www.enqa.eu/files/ESG_3edition%20(2).pdf [Accessed 3 November 2012]

INQAAHE Guidelines of Good Practice. Available from: http://www.inqaahe.org/main/professional-development/guidelines-of-good-practice-51 [Accessed 3 November 2012].

RIACES Orientaciones de buenas prácticas. Available from: http://www.riaces.net/index.php/es/desarrollo-de-las-agencias/evaluacion-de-las-agencias.html [Accessed 3 November 2012].


1.These two institutions had branches throughout the country, thus providing quality teaching to students in most regions. Their offerings were mostly centered in engineering and education.

2.All these universities, and the new institutions created from the branches of the state universities, are called ‘public institutions’ (even though some of them are private) because they are all funded by the government at the same rate, established in 1980 on the basis of their enrolment and their budget for the previous years.

3.In the following pages, we will refer to institutional accreditation; however, it must be understood that it is a process that closely follows the rationale of quality audits: it focuses on self-regulation and the main criteria refer to the ways in which institutions progress towards increased fulfilment of their own stated purposes.

4.This university was included in the second stage because the information gathered in the first stage suggested that it had experienced significant changes, even though it had not reached the higher accreditation levels.

5Project ALFA III, Quality Assurance: Public policies and institutional management, a project carried out by CINDA, with the support of the European Union and participation of 25 universities in 17 countries.