Accreditation and institutional learning: the impact interactions based on a minimaxing strategy have on the benefits from external reviews
This chapter is a personal epistemology of the potential limitations on the organizational learning that universities can actually achieve through external review processes. The goal is to engender a discussion of institutional learning from the perspective of risk and of how an understanding of the potential limitations can enhance the capacity for learning. The chapter begins with a description of the perceptual and structural challenges to institutional learning emanating from external review processes, which can help to explain the hurdles that have to be overcome in designing and implementing internal and external quality assurance processes. This is followed by a discussion of the nature of learning, of when learning happens, the limitations on organizational learning, and the institutional mechanisms and values that impact on the potential for learning vis-à-vis acceptance of the appropriateness of external demands for and the basis of accountability.
Because change is a more arresting phenomenon than persistence, we are apt to overlook the fact that in an institution or culture much remains the same even in the midst of radical change. (Daniel J. Bronstein, 1951, p. 53)
This book explores the question: have external quality audits (QAs) improved quality assurance in universities? The de rigueur answer is “yes,” but the answer is more nuanced based on (1) in regard or relation to what, (2) from whose perspective the question is addressed, and (3) the depth and extent of the impact that QAs have on the university and its academic programs, as well as the beliefs and practices of academic staff. Bronstein’s (1951) observation captures the challenge of how influential the organizational learning can be within universities and the extent of the impact that quality assurance can have internally. It also helps in understanding the tension between, on the one hand, a university’s desire to enhance and maintain institutional autonomy vis-à-vis its agility to cope with a changing environment and, on the other hand, how that desire for autonomy is questioned or minimized by external forces demanding that higher education subjugate itself to national expectations.
There are different answers to this book’s question. They range from an emphatic “yes” to a “not really” to “surely you’re joking” – this assuming that people are willing to go on record with their actual beliefs. These answers are demonstrative of the level of commitment, the willingness to actively engage and benefit from quality assurance activities. Compliance cannot be confused with commitment (Senge, 2006). How much willingness is exhibited is based on the attitude of organizations and their members regarding the degree of dissonance in values and vision of purpose between themselves and their external environment. A divide is often observed between faculty in most disciplines and what policy makers want, as is reflected by the changing external standards identified and used by accrediting bodies at the institutional and programmatic levels. The clash of views is, at its most basic, a challenge to institutional and faculty legitimacy – a status conferred and always controlled by those outside of the institutions (Pfeffer and Salancik, 1978). And this divergence of opinion results from how the external world interprets what campuses and faculty do and, conversely, what faculty, administrators, and related disciplinary/professional associations interpret the demands and expectations from policy makers, regulatory agencies, and other interested parties (aka stakeholders) to be. To paraphrase Daft and Weick (1984), the challenge for both sides in understanding each other is based on (1) each side’s belief that it can analyze the other, and (2) the extent to which each side will intrude upon the other to understand what each does and represents. It is also about the shape that the intrusion takes, because the resulting interpenetration leads to a reasonable degree of reciprocity. “The stability (= expectability) of action thus results from a combinatory play, a mixed-motive game” (Luhmann, 1995, p. 214). One would hope that the result would be a degree of interplay and understanding that goes beyond US Supreme Court Associate Justice Potter Stewart’s famous pronouncement in his concurring opinion in Jacobellis v. Ohio (378 U.S. 184 (1964)) regarding obscenity:
The intention in this chapter is to share the author’s personal epistemology in the form of a framework regarding what, in effect, leads to Herbert Simon’s (1976, 1991) notion of bounded rationality, i.e., focusing on satisfying rather than maximizing choices based on incomplete information shaped by the role that individuals play within an organization or university. This approach recognizes Cameron’s (1986) complaint that evaluations of organizational effectiveness (which, in a way, this is a discussion about) seem to be arbitrary in nature by simply suggesting that this is a personal observation. This is why, rather than call this chapter a conceptual or theoretical framework, Urman and Roth’s (2010) definition of personal epistemology comes more closely to what is being presented: a “tacit [assumption] about the nature of knowledge and how it is acquired” (p. 9) – and, in this case, about how learning is limited – based on the author’s personal discussions, experience, and observation as a participant in quality assurance processes in the USA as an external reviewer and chair of review teams, author and co-author of institutional responses to accrediting bodies, consultant to faculty members involved in writing their portions of reports, occasional academic unit administrator, and faculty member interested in researching quality assurance schemes and their impact on universities.
In a way, it can be said that this chapter presents a contrarian look at the extent and purpose of organizational learning in HEIs. The rationale is not to discredit the possibility that learning occurs and, in some instances, that external reviews have led to a lot of learning that has improved institutional quality and even quality assurance mechanisms. Rather, the desire is to demonstrate how there are beliefs and structures that limit the extent to which learning can occur and that inhibit the maximum potential for improving institutional quality and quality assurance apparatus.
Providing a description of the perceptual and structural challenges to institutional learning emanating from external review processes can help to explain the hurdles that have to be overcome in designing and implementing internal and external quality assurance processes. Policy steering may allow governments to trump institutional will because, as Lowi (1972) wrote, policy expresses a clear rule of law that sets public morality upon some action hitherto considered private. However, at least in the USA, there is a strong resentment by many academics towards the linking of regulation with academic quality (Dill and Soo, 2004) because it is seen as undermining academic freedom and diminishing the role of peer review (cf. Hamilton, 2007), while emphasizing the role of central administration over and above the participation of faculty in decision making – or giving prominence to the vertical rather than horizontal elements within a university (Padró, 2006).
Before continuing any further, it must be noted that a report by Ewell, Paulson, and Kinzie that came out in June 2011 stated that the data in their study “belie the common perception that most assessment is stimulated by forces outside the institution, such as external accountability and board mandates” (p. 8). The reason for this statement is that the study found that while the main driver for assessment at the program level was institutional accreditation, more than half of the respondents identified faculty interest and the desire to improve programs as major reasons for performing assessments. While this looks like a repudiation of the premise underlying this chapter, it can be argued that this is not because there are two aspects to the learning process within higher education that are at play: (1) learning is a process of change based on developing knowledge on a continuous basis and (2) academics define what Kanji (1996) terms their skills in pursuit of total professionalism guided by commitment and engagement founded on buy-in rather than compliance to the inevitable.
What do the external examiners and judges of QA have to be aware of in terms of the dynamics of internal organizational learning in order for meaningful learning to occur in an intended rather than stochastic (and therefore less than optimal) manner?
What do trustees, administrators, faculty, and staff have to understand about the role of QA in an era where institutional autonomy and role are legitimate societal concerns and traditional notions of academia are being challenged as the price for supporting university research and instruction?
The initial response, in looking at all these questions, begins with a quick discussion of the nature of learning, mainly in organizational settings. Four basic, interrelated elements come immediately to mind before the discussion moves on to the epistemological framework itself: (1) learning as a stochastic proposition, (2) the paradoxical nature of learning and what becomes meaningful information, (3) the relationship between perception and learning, and (4) the power of the language used by organizations in interactions with the external environment.
Bateson (1979) argued that learning is a stochastic process. For learning to occur, some sort of governor is needed. Argyris (1977) suggested that when it comes to an organization, learning is a process of detecting and correcting error, specifically “any feature of knowledge or knowing that inhibits learning” (p. 116). Luhmann (1995) also proposed that when structural change presupposes self-maintenance, continued relevance of the system is then based on a threefold difference of actions: “(1) connective action within the framework of existing structures of expectation, (2) connective action on the basis of deviant structures of expectation, and (3) cessation” (pp. 347–8). When the reciprocity within the system is absent for whatever reason, the challenge becomes optimizing the ability to provide input and, as Luhmann (1995) put it, to interpenetrate the larger activity of the social system.
Most readers will be familiar with Argyris and Schön’s (1974) double loop learning based on confirmation and disconfirmation. However, Weick’s sensemaking model explains the different steps that organizations take when it comes to learning as a self-generating governing process (how they acquire information and that information is referenced):
The challenge of these steps is in figuring out whether the interpretation of the available information becomes framed in Kantian categorical imperatives where everyone thinks their view is the morally correct one, or is reflective, where observation is based on immediate experience and “the elements available for explanation are simply, the objective content, eternal objects, and the selective concrescence of feelings whereby an actual entity becomes itself” (Whitehead, 1929, p. 232). In more prosaic terms, this is what makes the difference between using quality assurance as it is intended or, to paraphrase a colleague, to check off the points in the checklist.
Handy (1994) says that “[p]aradox does not have to resolved, only managed” (p. 18) and, ultimately, that is what this chapter is about: managing the paradox that is learning. Information as data is only part of a larger process. Ambiguity and uncertainty enter the picture when there is conflicting to outright contradicting, incomplete, or unclear information that does not allow for a straightforward determination of what is happening. Complexity is enhanced because different aspects of the external environments and contingencies framing the interaction challenge the idea of “one size fits all” in learning and actually understanding what is going on. In the end, it is all about the degree of agility to adapt and to regenerate or self-create (autopoiesis).
Luntley (2008) defines learning as those activities involved in getting one’s life in order. But learning, in keeping with Lewis and Dehler’s (2000) thinking, is a paradox in that it is perceptual and often navigates through mixed messages, conflicting demands, or opposing perspectives. These paradoxes can be based on inconsistencies real or designed, as Argyris (1996) points out. In an organizational structural setting such as a university, the capacity for paradox and learning and action increases because of attributes such as loose-coupling, high specialization of roles, continuity, deviation amplifying processes, and an expanded listening capacity for input (cf. Cameron, 1986). When it comes to problem solving, the managing of paradox is to focus not on the contradiction but on what Bohr (1963) termed complementarity, the ability to look beyond what at first seems to be disagreement in order to see what commonality might exist.
Paradoxes are present within organized structures, which, in a way, makes their presence predictable (Cameron, 1986) and manageable. What is important is to determine which data become meaningful information and to translate that into useful learning. In an era where the preference is for quantifiable data and the focus is on regression and trends, there has to be sensitivity to the idea that much useful information is nonlinear in nature and that contradictions are not necessarily aberrations as Cameron (1986) suggested. Indeed, serious consideration should be given “to manipulat[ing] and explor[ing] data in nonstatistical ways before submitting it to standard programs of analysis” (p. 551).
Learning is connected to perception because it focuses on differentiation of the “fluid organization of meaning existing … at any instant” (Combs et al., 1976, p. 22). It is a complex process dependent on a number of other factors that can be broken down into four properties distinguishable from experiences that have to be learned: effortless structure (cohesive, stable experience permitting action), determinism (avoiding vagueness), perception–action coupling (ability to take action), and pre-interpretation (perception often occurs prior to a person’s acquiring belief or knowledge about a situation) (Roeckelein, 1998; Schwartz and Heiser, 2002). Piaget (1995/1959) noted that perception is more than isolated terms brought together by association or judgment, ultimately contextualizing facts based on expectations (Bruner, 1986). Therefore, people do not behave in accordance with the facts of others, but behave according to how they see the facts themselves (Combs and Snygg, 1959).
The impact of perception on organizational learning is clearly demonstrated in Weick’s (1995) sensemaking model. The seven steps listed above all focus on the person or the group trying to figure things out. In the last two steps, extracting cues and plausibility show how learning is directed by perception, first by what is singled out and embellished as content of thought that becomes salient, and then by how an individual within the organization determines that enough information is available to proceed with what has to be done (cf. Weick, 1995, pp. 61–2).
Normative referencing as exemplified by policy formation, standards generation, legitimacy of organizational identity, and degree of autonomy suggests a recognition and understanding of role and standing reflected by language use. Bourdieu (Bourdieu and Passeron, 2000) proposed the presence of a dominant language structure that reflects the status of relationships and the shaping of meaning; in effect, how the language used helps to identify existing power relationships. Of particular interest is his interplay between field (structured space with individuous interactions) and habitus, those structured dispositions leading to what is practiced (Bourdieu, 1993). “Thus, for Bourdieu, it is the combination of institutional control over forms of capital together with processes of conversion and transmission that is crucial to the capacity of dominant classes to maintain their position – and therefore to social reproduction overall” (Goldthorpe, 2007, p. 5).
When it comes to change, especially in higher education, Kezar (2001) suggests the need to develop a common language in order to better conceptualize what change means. As Kezar (2001) points out, the issues faced here are not merely first-order change, but second-order change, which is multi-dimensional and multi-faceted, based on what has been discussed above in this section. As Quine pointed out, there is a public nature of language and this has consequences for meaning and communication (Follesdal, 1999). The issue is pretty simple: whose language dominates, ergo, which doctrine is dominant? In an environment where there is a misunderstanding or mistrust of what an organization does in relation to expectations, then the action generated is towards aligning language so that there is understanding and acceptance of task. The corollary, however, as Luhmann (1995) suggested, when there is a power game in place – and this form of alignment is a form of power striving – is that conflict politicizes power and decision-making recourse is through law. The reality of modern academia is lack of awareness of the scope and significance of the restructuring that is ongoing in higher education (Rhoades, 1998).
Figure 16.1 provides an approach to recognizing how learning occurs within the university’s enterprise. The Padró and Hawke (2003) model is meant for both public and private institutions, even if Zumeta’s (1992) findings regarding state policies do not tend to consider private HEIs to the extent that they should. The emphasis is on how perception impacts upon action; however, the model is useful in looking at:
The inherent political capacity of universities to maintain their levels of autonomy in their interactions with their external stakeholders (cf. Bolman and Deal, 2008; Rawls, 1995) as buttressed by reputation (DuBrin, 2011) and ability to act in accordance to expectations (cf. Landecker, 1952).
How institutional stability is the main hub for how the external environment judges how the institution balances its internal and external interests. As Heidegger (2008/1962) pointed out, although phenomenological, dealings with the environment have manifold concerns that are manipulated and put to use in more concrete terms, which forms its own kind of “knowledge.”
The presence of input and environmental scanning. Although mainly a role of governance, the challenge is to bring into the light the different routes of scanning occurring through natural boundary spanning and to become part of the overall decision-making process.
The distinction between output and throughput functions and the differentiation between administrative support and instructional/research activity. This difference helps to identify organizational culture differences based on function. The model suggests that quality control is mainly a performance capacity, also tied to some process interest on the part of administration. Quality assurance is a throughput function tied to administrative goals and linked to how success is identified in performance so that stability can be demonstrated and communicated within and outside the campus.
How the valence of dominance probably is issue specific and based on Luhmann’s (1995) characterization of double contingency, the ability of each side to understand the other’s communications. Thus, the temptation to give dominance to the external environment at all times negates the potential impact of reciprocity in the overall system.
Paradoxically, institutional success is shaped by the university’s capacity to participate in the formation of professional knowledge and standards that guide/reflect policy-steering activities. Blau’s (1994) study on academic work suggests that some people question whether faculty can be considered professionals. Under some technical considerations the answer is both “yes” and “no” based on what happens to their scholarship within the discipline. Nevertheless, the demand by faculty for authority over their own work, for self-regulation, for peer review, and for capacity to solely set standards for special competence frames the basis for their claim that their works and views should be the ones influencing and/or formally shaping external expectations and policy formation on what it is that faculties do (cf. Foucault, 1980, p. 108).
Toma’s (2010) model of building organizational capacity at HEIs provides a second layer of understanding of how organizational learning takes place (Figure 16.2). From a practical perspective, his eight elements serve as an “index to what constitutes the foundation needed to support the ambitions and functions of an institution” (p. 6). Wildavsky (2001) observed that change is harder to do, the deeper into the organization it goes. For Wildavsky (2001), there is a greater possibility for change to happen if it is limited to central administration. Input from the external environment comes in many forms; yet HEIs, paradoxically, impact on professional knowledge and the creation of standards because of the research and teaching that they do (Padró and Hawke, 2003). Therefore, learning occurs when there is a sense to the answers to the questions that Toma (2010) raises, and agreement based on the perception that capacity to change is predicated on mutual interest or understanding, if not full agreement:
|Purpose:||Why are we here and where are we headed?|
|Structure:||How should we be configured to do our work?|
|Governance:||Who should make what decisions?|
|Policies:||What rules should we proceed under?|
|Information:||What do we need to inform our decision making?|
|Infrastructure:||What are our human, physical, technological, and financial assets?|
|Culture:||What is our essential character? (p. 208)|
Both models bear out Senge’s (2006) point that structure influences behavior in subtle ways. It does so by shaping who receives the information and how information is turned into evidence through analysis for interpretation into action. In other words, using Whitehead’s (1929) notion of concrescence, how information comes together, is contextualized, and placed into a workable scheme. Structure provides a level of access while function gives context within the scheme of things. Culture impacts on identity (vis-à-vis ambiguity, conflict, and uncertainty) and the flow of communication. Culture also determines fitness within function as well as actor perception and filters (DuBrin, 2011).
Consequently, optimal learning happens when there is convergence between external environment preferences, campus structure, and institutional climate. Learning can happen even if convergence is not complete between these three elements because, if nothing else, Goffman’s (1959) notion of impression management implies that protective practices on the part of participants interacting on behalf of HEIs and designated external environment players and the supporting cast behind the environmental players (e.g. accrediting agencies, professional associations, standards agencies) provide for an environment where some decision will be made, albeit more by imposition than by mutual agreement.
Figure 16.3 illustrates challenges within the external environment, organizational structure, and institutional climate that have to be dealt with or overcome in order for organizational learning to occur and to get the most from external audits or reviews. One term that best describes these challenges is “defensive routines” – “actions or policy that prevents human beings from experiencing negative surprises, embarrassment, or threat, and simultaneously prevents the organization from reducing or eliminating the causes of surprises, embarrassment, and threat” (Argyris, 1996, pp. 102–3). Add to the list threat to autonomy and identity. Conversely, another idea that enters the discussion is Foucault’s (1980) balancing of sovereignty – public right – and disciplinary power – peripheral institutions. More specifically, his concept of governmentality, a neo-liberal model whose aim “is the strategic production of social conditions conducive to the constitution of Homo economicus” (Hamann, 2009, p. 37), suggests an impetus for government to ensure that this aim is achieved, in contrast to maintaining the traditional notions of academia, which in the USA is broadly stated in Justice Felix Frankfurter’s concurring opinion in Sweezy v. New Hampshire, 354 U.S. 234 (1957): to determine for themselves on academic grounds who may teach, what may be taught, how it shall be taught, and who may be admitted to study. The underlying issue that generates defensive routines from faculty is the view that training is confused for education because of the economic benefits of having a degree. This issue, in turn, places considerable attention on the relationship between interactions to describe what is happening between HEIs and governments and what Maturana (1970) termed the domain of relations.
Centermost in limiting a university’s organizational learning is the degree of consonance between its purpose based on mission and value construct and the social values put forward by government agencies (cf. Clark, 2008). The ability of HEIs and higher education systems to interpenetrate the policy-making and regulatory compliance structures in part determines the dominant values at play, as represented in the language used. At play is whether the campus response is compliance based on full agreement, compliance based on necessity, or de minimis compliance with external demands (with the possibility of open or sub rosa opposition or support). Figure 16.3 suggests that organizational learning happens best when there is convergence of external demands, and internal organizational structure and climate exhibit agreement, i.e. reciprocity is clearly represented in the relationship between an institution or system and the state. When there is structural and/or climate dissonance, organizational learning is diminished, based on the nature, type, and extent of discord.
Foucault’s governmentality represents the tactics of government that allow it to define and redefine what its competencies are (as cited in Lemke, 2007). This explains why international agencies and the US Department of Education are looking for an enhanced role for national government in determining the purpose, overall framework, and standards of quality for higher education at the system and institutional levels. And as Hamann (2009) and Lemke (2007) point out, governmentality explicitly aligns government action with the neo-liberal mindset, which links educational endeavors with economic well-being. On the other hand, when there is dissonance between academicians and state entities – when the feeling is that academic connoisseurship as Eisner (1979) defined it pertaining to education and professional/disciplinary concerns is replaced by those who are not able to “perceive the subtle particulars of educational life” (p. 195) – an opposite, defensive mixed strategy may develop: minimaxing. According to Samuelson (1976), the general theorem is:
If all persons had the same degree of risk aversion and faced the same symmetric probability distribution of fates, they would selfishly agree on a unanimous vote on an optimal compromise between redistributive taxation and the deadweight loss involved therefrom.
If no person has infinite risk aversion, they will unanimously suffer from an imposed minimaxer’s regime of complete (and costly) egalitarianism, preferring to be spared that version of non-Pareto optimal “fairness,” or if the persons differ in their risk tolerances, and even if they agree on a symmetric probability distribution, it seems unclear whether their sense of “fairness” or “justice” would lead them (a) to be able to identify and calibrate the person who is “hurt most” under any given tax system, t(xk; xk), and (b) to agree that the “best” system is to be that t**(;) which leaves the “worst-off person” in the “best feasible position” (Samuelson, 1976, p. 186).
Simply, the institution will yield what it must in order to preserve as many of its interests as possible. Minimaxing places compliance as a zero-sum game. The interactions are based on strategies that identify an institution’s maximum loss and minimize policy-maker/regulator potential gain. It is about setting and acting based on a selected security level (cf. Luce and Raiffa, 1985/1957). Savage (1951) wrote that minimaxing is based on ultra-pessimism because it demands “that the actor assume the world to be in the worst possible state” (p. 63). So, the mindset of this strategy is that regulation is a form of tax, and compliance or avoidance is based on the willingness to pay the tax.
Senge (2006) and others, such as Ulrich (1998), talk about commitment as a two-way process, one that Ulrich points out increases an organization’s intellectual capital. Commitment within the university structure, however, has to go through the filters of the campus decision-making machinery and the climate that exists before its presence can be seen. Figure 16.4 exemplifies how commitment or resistance develops. Embedded within this mechanism are what is rewarded and how, as exemplified by the promotion and tenure process, the shared governance structure, and the patterns of faculty participation in governance and other campus service (Figure 16.5). The caveat to watch out for is what Kegan and Lahey (2001) call competing commitment, when faculty seem to apply more energy to creating resistance instead of fostering change, even if they hold a sincere commitment to change and improvement.
Present as a backdrop to the issues presented in Figures 16.4 and 16.5 is the tension between departments and central administration, what has been termed vertical integration versus horizontal representation (Padró, 2004). Central administration is charged with the overall identity of the university, while “the disciplinary imperative demands that administrators be sensitive to the nuances of differences among subjects and their supporting groups” (Clark, 2008, p. 263). “The challenge … is to ensure that the vertical ‘functional silos’ and horizontal ‘process silos’ are in balance” (Dive, 2003, p. 22) in order for viable stability where interests are balanced (cf. Pascale, 1999) to be accounted for (e.g. see Figure 16.1).
Not to succeed in demonstrating stability definitely impacts on the acceptance of data and the amount of organizational learning that occurs. Figure 16.5 shows how easily organizational learning can become marginalized because of the notion that the elements of learning as described earlier in this chapter is a case of reductio ad absurdum, hence challenged and/or ignored.
This chapter is not about nay-saying; rather, it is about the presence of external agencies, higher education systems, and the capacity of universities to identify potential risks to organizational learning that are symptoms of other, more important challenges that need to be overcome. ISO standard 31000 (ISO, 2009) considers risk management as identifying events that change circumstances in order to maximize opportunity to achieve the goals and performance improvement that should be an integral part of all organizational processes (Figure 16.6). The reason why this way of thinking is important is because, as Quinn (2005) proposes, those who take on leadership roles (on both sides of the institution’s boundaries) have to move from a “normal” (comfort centered, externally directed, self-focused, and closed to other stimuli) to a “fundamental” state (results centered, internally directed, other focused, and open to change) in which leaders draw on their own fundamental values and capabilities, which, paradoxically, is not their normal state of being. If the eventual goal is to create networks that are outside the comfort zone of many of the potential participants (Ibarra and Hunter, 2007), then identifying where the limitations are for acceptance and learning is a form of risk that must be part of the accountability equation.
For those not comfortable with the interdisciplinary approach of this chapter, the purpose here has been to provide a personal epistemology of what the author sees as elements in the discussion of university performance that are not taken into account. Simply to focus on one or two disciplinary strands leaves many questions unanswered. It was forays into the different fields of study that began to provide the author with insights into how quality in universities is and should be addressed. These are the theories that have guided reflection on the experience as an examiner, as a person responsible for writing some of the reports, and as a student of the process of quality in higher education. The goal is to provide impetus to the discussion of how much learning occurs and why from the perspective of risk, and how the exercise of understanding potential limitations can enhance the opportunities for learning.
Bronstein, D.J. The principle of polarity in Cohen’s philosophy. In: Baron S.W., Nagel E., Pinson K.S., eds. Freedom and Reason: Studies in Philosophy and Jewish Culture in Memory of Morris Raphael Cohen. New York: Free Press; 1951:44–58.
Dill, D.D., Soo, M. Transparency and quality in higher education markets. In: Teixeira P., Jongbloed B., Dill D., Amaral A., eds. Markets in Higher Education: Rhetoric or Reality?. Dordrecht: Kluwer; 2004:61–86.
Ewell, P.T., Paulson, K., Kinzie, J., Down and In: Assessment Practices at the Programmatic Level – Program Level Survey Report. National Institute for Learning Outcomes Assessment (NILOA), Champaign, IL, 2011. Available from: [Accessed 28 June 2011]. http://www.learningoutcomesassessment.org/documents/NILOAsurveyreport2011.pdf
ISO. International Standard 31000: Risk Management – Principles and Guidelines. 1st ed. ISO, Geneva, 2009. Available from [Accessed 5 November 2011]. http://hsec.ir/file/pdf/ISO-31000.pdf
Kezar, A., Understanding and Facilitating Organizational Change in the 21st Century: Recent Research and Conceptualizations. ASHE-ERIC Higher Education Reports; 28. Jossey-Bass, San Francisco, 2001. .
Padró, F.F., Using continuous improvement process as a means of evaluating and documenting institutional performance. C. Turnbull, C. Pratt, C. Pratt. Proceedings of the Evaluation Forum 2005 – University Learning and Teaching: Evaluating, Enhancing the Experience. University of New South Wales Learning and Teaching Unit, Sydney, 2006:43–56.
Padró, F.F., Vertical integration v. horizontal representation September. The clash of cultures in university environments and how these impact institutional standards and their assessment of quality. Proceedings for the 7th “Toulon-Verona” Conference on Quality, September 2–4, 2004. University of Toulon-Var, Toulon, 2004:135–144.
Piaget, J. The early growth of logic in the child: Classification and seriation. In: Gruber H.E., Vonèche J.J., eds. The Essential Piaget: An Interpretive Reference and Guide. Northvale, NJ: Jason Aronson Inc., 1995.