“ASSESSMENT PROGRAMS:
THIN VENEER OR CORE ACTIVITY?”

 

Address by Dr. William B. Allen, Director
State Council of Higher Education for Virginia

To The Virginia Assessment Group

Wintergreen Resort
Wintergreen, Virginia

November 5, 1998

 

© 1998 W. B. Allen

 

Good afternoon. Thanks, Donna, for that kind introduction. I also want to express my appreciation for the way that you have taken up the banner in leading SCHEV’s interactions with the Virginia Assessment Group (VAG) and with the institutions with regard to assessment.

Let me also thank VAG for inviting me to speak with you today. It’s great to have a reason to travel to the Blue Ridge Mountains at the peak of fall color. Having lived for so many years in California and Florida, I am especially appreciative of the chance to enjoy the changing of the seasons here in Virginia.

Another of the many reasons that I am pleased to be working in the Commonwealth of Virginia is the reputation it developed as a successful pioneer in the higher education assessment movement. The large turnout for this conference tells me that Virginia’s colleges and universities continue to view assessment as vitally important. So do I.

My belief in the value of assessment was a major factor in my decision to hire Dr. Karl Schilling as SCHEV’s Deputy Director for Policy. I am sure that many of you already know Dr. Schilling and that still more know him by reputation. Also, I’ve heard this is not the first time that he has attended a VAG conference. I’m confident you and he together will continue to make the Commonwealth a leader in advancing the theory and practice of assessment in higher education. There is new ground that needs to be broken in both theory and practice and I hope to challenge you to become pioneers once again. He will sketch out some his ideas about new frontiers following my remarks.

The title of this conference— ‘Expanding the Role of Assessment”—captures a sense that many of us share. Despite the excellent work that has been done in the assessment field over the past fifteen to twenty years, and despite the insertion of assessment requirements into accreditation standards, we have only scratched the surface in terms of our potential. Assessment can be a powerful tool not only for measuring student achievement and institutional effectiveness but, more importantly, for increasing both.

In the current issue of Change, Ted Marchese concludes that the assessment activities at most institutions “remain a thin veneer.” Likewise, the dean of a prominent school of education was recently heard to say that assessment has not had much impact on college teaching. Contrast those statements with the experience of Truman State University, where assessment is so deeply embedded in the institution’s culture that their 1997 recruitment for a new president explicitly sought candidates with “strong, demonstrable commitment to . . . the proactive use of assessment data for the improvement of teaching and learning.” I might mention, as an aside, that Virginia may have contributed to the strength of Truman State’s assessment program, since I am told that their former president, Russ Warren, moved to Truman from JMU some years ago.

What accounts for the different experiences and conclusions exemplified in these statements? I think there are three primary factors that determine the effectiveness of an institution’s assessment program:

·        First, clarity about purpose. Is there a shared understanding of why assessment is being done? Does the purpose meet a compelling, internal, and widely perceived need?

·        Second, faculty involvement. I maintain that it is impossible for an assessment program to have significant impact without extensive faculty engagement. Further, leadership from the deans can make a real difference. I say this as someone who used my position as dean of James Madison College to highlight the role of assessment in advancing teaching and learning at our College.

·        Third, the pervasiveness and integration of the program. Are assessment activities systematic and integrated into the academic and student development programs of the campus, or are they sporadic and isolated?

I’d like to share a few thoughts with you about each of these factors, particularly with regard to faculty involvement and faculty evaluation.

It may sound simplistic, but you know as well as I do that the purpose behind a college’s assessment program profoundly determines the impact it can have. If assessment is being done at your campus only, or primarily, to satisfy SCHEV, then my advice to you is to “cease and desist.” I am confident, however, that no colleges in Virginia any longer carry out their assessment program only in response to SCHEV mandates, though rumor has it that some of your programs may have originated for that purpose. Donna has told me that Virginia institutions early on found assessment to be an important means for evaluating, rethinking, and improving general education—a process that continues to this day.

Sandy Astin, Tom Angelo, and others describe assessment as a powerful tool to assist the academy in our important work of talent development. In his 1991 book, Assessment for Excellence, Astin argues persuasively that the effectiveness of a college or university should be judged by how much and how well it contributes to the cognitive and affective development of students, rather than on the basis of reputation or resource acquisition.

I challenge the use of simplistic, uninformative statistics such as graduation rates that measure nothing but nominal progress without attention to the pacing—the changes in velocity—that do or do not occur within students. We need measures that focus on value added. I have charged the SCHEV staff to investigate meaningful ways for us to develop such measures and I am sure they will call on you to help in that work.

At too many institutions of higher learning, growth in numbers—such as matriculants and dollars—have become the standard for success. I say that a superior measure of success is an institution’s capacity for change—the constant refinement of procedures, resources, and facilities that is the only true path to excellence. Quality of academic life, rather than quantity of academic life, ought to be our paramount concern.

The second factor—faculty involvement—invokes the Lynchburg College student, who strolled recently into the Zen Pizza Parlor in nearby Nellysford. The student ordered a pizza with everything, paid the proprietor, a Buddhist monk, with a $20 bill, and asked for change. “Change,” the proprietor inscrutably replied, “must come from within.”

To foster change from within, we need to ensure that we assess core activities, not those that are peripheral. Student learning must be at the center of our assessment programs. This means, of course, that faculty engagement must also be at the heart of the enterprise. It is the faculty who must consider what questions to ask and what evidence to gather. There is little point in gathering information about student learning if that information is not used to improve faculty teaching.

I think that most faculty want to be reflective practitioners, who take the time to observe the impact of their teaching on student learning and who use these observations to improve the practice of their craft. There are, however, two potent forces that hinder this inclination: the tradition of faculty autonomy and the fact that research trumps teaching in the merit system at many campuses. We can see hope for change, however, with regard to both of these factors—change generated from within the academy and change stimulated from without.

Assessment can aid not only in developing student talent, but also that of the faculty. Increasing numbers of faculty see collaboration and peer review as valuable means to improve classroom performance, but on some campuses these approaches meet strong opposition. I’m a strong believer in peer review—so much so that the SCHEV staff has just added a peer review component to its own performance evaluation system. Faculty evaluation and development are, however, complex areas and deserve thoughtful consideration.

The chief criticism of peer review in teaching is the claim that it mainly involves retailing hearsay. That is an odd criticism for two reasons. First, it is usually the case that so-called hearsay emerges only in contexts in which faculties stubbornly resist classroom visitation as a routine. It hardly seems fair to blame hearsay when direct evidence is deliberately withheld. Secondly, and far more importantly, what is casually called “hearsay” is in fact the meat and potatoes of academic life, namely, deliberate, self-conscious discussion about teaching among colleagues.

As I reminded my colleagues at Michigan State University, teaching at the highest levels and with the most reliable results sets institutions apart in the field of higher education. Because this results from the purpose, rather than an accident, of the institution’s existence, we constantly bear the obligation to speak knowledgeably about the goals and methods of education.

The first question to engage our attention when we speak of the evaluation of faculty ought to be the question whether and in what way they may constitute models for others who would seek like success. Do they contribute meaningfully to the foundations of knowledge regarding undergraduate instruction? What other institutions perform comparably or better in this regard? How might they assess their own performance in this regard, by which I mean to ask, are they able to articulate standards by which they are willing to be judged in a court not of their own making? When I became a dean, I attempted—through a questionnaire distributed to the faculty—to gauge this dimension of the College. Too few faculty responded to permit any reasonable conclusion about the faculty as a whole, but the few who did respond indicated some ambiguity concerning consciousness of the relative value of the College’s contribution. Might that ambiguity reflect hesitation to engage such a judgment?

Because we value teaching—undergraduate teaching—above all other dimensions of performance within higher education, the question of how we might phrase relative contributions within this arena is a necessary prelude to questions regarding internal consistency in the application of standards of evaluation. We need to know, for example, whether we regard the least acceptable level of accomplishment in any given institution as comparable to the least acceptable level of accomplishment elsewhere. Or, is that least the average elsewhere? The best? What does it mean to say that a faculty member demonstrates sustained excellence in teaching, when compared with teaching in institutions one would regard as comparable or nearly so?

It would be easy to mistake questions of comparability as a fruitless exercise in filling out bureaucratic scorecards in place of self-conscious integrity regarding a faculty’s program and its intrinsic requisites. I submit, however, that the case is precisely the reverse; namely, mistaking one’s own sincerity of purpose as sufficient justification before disinterested observers. The fact that one’s mission is unique does not require that it be wrapped in preciosity. It is time to discard the myth of incomparability.

An example: some institutions quite plausibly claim to educate undergraduates rather by virtue of the specific discipline they require of the students, no matter what native gifts the students bring with them. By the discipline required of students, I mean theoretical sophistication, familiarity with canonical interpretations and intelligent challenges to them, and carefully integrated study across a general range of concerns, systematically developed through increasing levels of difficulty and, therefore, a coherent curriculum which uses the humanities to vector its content. These institutions may plausibly compete with institutions that prefer to work only with the most gifted students and in the mode of narrow specialization as the preferred method of undergraduate education. Comparisons between these two types of institutions ought not to be avoided because of the supposed gap between the elite and other institutions. If institutions that educate by virtue of required discipline succeed as well they think they do, then it is well time to inquire—or, better, to state—why they do so.

Every discussion of assessment contains at least implicitly a discussion of faculty evaluation. The conception I now offer of the goal and manner of our work suggests an approach to faculty evaluation. Because we begin from self-consciousness of a particular approach, is it not reasonable first to judge faculty—not by student reaction—but by their respective abilities to articulate this approach both in their teaching materials and in their own evaluative commentaries on their teaching? Moreover, would not this requirement produce in turn, as direct evidence of teaching ability, reasonable reliance on a faculty member’s capacity to persuade colleagues that the faculty member understands and can appropriately relate the conditions of success of the curriculum as a whole?

In these questions I do not aim to depreciate the value of student evaluations, on which we rely in their proper role (which is a second order role).* Perhaps we need to be more explicit about the role of student evaluations. Is it not true that, while we do not regard them as definitive evidence of good or excellent teaching, we do consider them as highly probative—that is, as having diagnostic value rather than interpretive value—with respect to bad teaching? In such a case, a faculty member’s student evaluations would have most value when indicating symptoms that may require medication, much as blood pressure or body temperature serve when abnormal to indicate a regime but offer scant assurance of health when normal.

At the center of the evaluation of faculty teaching, accordingly, ought to stand the reality that we value teaching to the point of making excellence—not mere acceptability—a sine qua non for faculty advancement. Does this mean, as it implies, that an excellent researcher but mediocre teacher ought not reasonably to rely upon promotion?

We may appreciate faculty scholarly contributions—not defined narrowly—on the conviction that scholarly growth is a necessary complement to teaching excellence. Accordingly, we may encourage faculty to engage in continuous intellectual development, not only through means of course preparation but also through independent and collaborative research and creative endeavors. Clearly, assessment of research contributions must be mission dependent. In the research university, one may rightfully be mindful to adapt teaching requirements to the criteria that characterize membership in that larger faculty. Nor should an understandable preoccupation with teaching obscure, in faculty evaluations, the obligations that descend from that research university status.

The questions to pose, however, are whether it is possible, given the research mission, that an excellent teacher with no research accomplishment might yet rely upon promotion? Should we make an allowance for teaching not only unchallengeably excellent but also beyond the research university norm? Concede that mediocre teaching cannot rescue a record of excellent research; is it also true that it requires the value added of excellent research (or at least a teaching load beyond the norm) to attain the highest merit recognition within the research university?

I will not speak of the equally interesting question, how we evaluate research excellence. Research excellence is intrinsic but secondary to our topic today, namely, the levels of intelligible inter-communication among colleagues that we demand in teaching. Insofar as research accomplishments inform classroom instruction, they are relevant to this conversation. For one could scarcely credit a claim to impart to undergraduates what cannot with greater effect be communicated to informed, if lay, colleagues.

I’ve described the purpose behind an institution’s assessment program as paramount. I’ve raised some questions about faculty involvement and faculty evaluation, which are both essential components in a strong assessment program. This brings me to the third factor that can change assessment from a superficial activity to one that affects the core of the educational enterprise. To have a real impact on student development, assessment must be systematic and thoroughly integrated into the student’s classroom and co-curricular experiences. We know from examples like Truman State and Alverno College that it is possible to build and use such assessment programs. We do not lack the technical expertise to put such assessment programs in place at every campus. And I expect that we will.

What benefits result from systematic assessment? Count at least these:

1.      Assessment creates an attitude of excellence and accountability.

2.       Assessment gives the ability to measure outcomes and creativity.

3.      Assessment points out how to achieve outcomes that are meaningful.

4.      Assessment practices provide information to interested university partners and build confidence in faculty decision making.

Which description characterizes the state of assessment at your institution? Thin veneer, or a powerful tool to break up intellectual and organizational hardpan and to develop talent? Wherever you place your institution along this continuum, you surely can see opportunity to expand and grow. Dr. Schilling will, in a minute, offer some provocative ideas on new directions for us to consider in this expansion.

        Before turning to Dr. Schilling, however, I want to issue a challenge to VAG. I believe that even those institutions that have been most successful in integrating systematic assessment into all areas of the academy have not yet found ways to assess the most profound aspects of our work. We still tend to assess those things that are easy to measure rather than what is most important.

        One of my foremost goals as director of SCHEV is to increase public awareness of the value of a college education. There is clearly a public focus today on the cost of a college education. Students, their families, and taxpayers—as well as appointed bodies such as the Governor’s Blue Ribbon Commission on Higher Education and the Legislative Joint Subcommittee on Higher Education Funding Policies—want to know why the cost is so high. Further, they want us to keep college affordable. I think the public also wants, and needs, to know more about the value of higher learning. Our collective challenge is to develop meaningful ways to assess and articulate the value that a college education adds to the lives of individuals and to society. I believe we have the expertise to break new ground and develop such measures.

        My question to you is: Do we have the will?


_____________________
* Although it may fairly be said that what actually happens to students after they leave a professor would be even more probative of teaching ability than student evaluations. This form of assessment is peculiarly difficult to establish, however, and perhaps also even more subject to the interplay of chance than any of the other evaluative techniques. Who, after all, gets the credit for the Rhodes Scholar, or the Nobel Prize winner?

 

Back to Top