Senate Ad Hoc Committee
Bill McBroom, Sociology
Irene Applebaum, Philosophy
Jim Walsh, Psychology
Susan Wallwork, Institutional Research
Julie Pilling, Graduate Student, Linguistics
STUDENT EVALUATIONS
Student evaluations of courses and of professors are commonly conducted at colleges and universities. For example, some sort of student evaluation is mandated at The University of Montana by the Collective Bargaining Agreement (section 10.220). As with many other institutions, the intended use of these evaluations at UM is not only to provide faculty with feedback from students, but also to inform the faculty evaluation process. In light of growing evidence that the evaluations themselves are flawed in many ways, it is important that guidelines be developed for student evaluations. In the sections that follow, we consider the validity of student evaluations and suggest means of standardizing the administration of evaluation forms. Since a major use of student evaluations is to evaluate teaching, alternative means to accomplish this goal are also proposed.
Validity
In any assessment, concerns with validity are of paramount importance. While reliability refers to consistency or stability of results, validity concerns the issue of whether the dimension of interest is actually being measured.
Student evaluations may be said to have “content” validity if they include questions tapping the content of teaching. The core idea of validity, however, concerns the complex, and controversial, issue of the nature of reality. Just because questions on an evaluation form deal with teaching does not in itself mean that the aggregate results of completed forms provide a valid measure of one’s teaching. The problems of validity of student evaluations are complicated by the recognition that students are likely to have one definition of “good teaching” and faculty colleagues quite another.
If one is to rely on student evaluations, one would minimally demand some “criterion-related” validity. This clumsy term means that it is important to show the correspondence between student evaluations and some external criteria or factors known (or believed) to measure the dimension of interest. If, for example, student evaluations were strongly correlated with judgments of professional educators who were subject matter experts, then the evaluation forms would be said to have some external validity.
Published reports of student evaluations strongly suggest that the validity of student evaluations is unknown. Indeed, available evidence suggests the teaching may be evaluated on political grounds (Coren, 1995) and that erroneous attributions of racism and sexism are sometimes found in evaluation forms (Coren, 1995). Trout (1996) suggests that some students stress the popularity of the instructor and the entertainment value of the course content instead of responding positively to demanding standards and high academic quality. Platt (1995) argues that student evaluations are flawed, in part, because there are no checks against inaccurate statements. That is, validity is suspect. After reviewing a number of published works, Trout (1997) goes so far as to assert that because rigorous requirements seem inversely related to course evaluations, instructors wanting good evaluations should “suck up” to students.
Administration
Anecdotal evidence suggests that the administration of student evaluation forms at UM varies widely. In some cases, forms were purportedly completed at an end-of-term social gatherings. Sadly, there seems to be more than one instance of this approach (Barrett, 1998).
In order to increase the reliability (not validity) of evaluations, it is important to standardize the administration of the forms. This involves three interrelated concerns: timing, privacy, and content.
The Faculty Senate recommends that evaluations be distributed during the last week of classes, although not necessarily at the final class meeting. Forms should be administered either at the start or at the end of the period.
It is inappropriate for the instructor to be present during any aspect of the administration of evaluation forms. Thus, in addition to the current guideline--that the instructor not be present while students are filling out forms and that the instructor not personally collect the forms--the Senate recommends that the instructor should also not hand out forms or provide instructions for their completion. Rather, a third party should be responsible for the administration of forms and for delivering them to the appropriate office. The instructor should not be present during any stage of this process.
In addition to whatever core questions are included and regardless of whether computerized forms are used, narrative comments should specifically be solicited for dimensions deemed relevant by units.
Use of Evaluations in the Faculty Evaluation Process
At The University of Montana, both a Student Evaluation Committee (SEC) and a Faculty Evaluation Committee (FEC) are called for by the collective bargaining agreement (sections 10.220 and 10.230, respectively). While the full range of activities undertaken by students is unknown, in many units the SEC relies heavily, if not exclusively, on the student evaluation forms. A common practice is for SECs to comment on the strengths and weaknesses of instructors as revealed by student written comments. In what appears to be an effort to be balanced, this often takes the form of reporting both positive and negative remarks. However, the fact that there may be far more of one than the other is often not emphasized in SEC reports.
Given the unknown validity of student evaluations, their value in the faculty evaluation process is extremely limited. Perhaps when comments on specific points occur with overwhelming frequency, they provide useful information only about extremely good and extremely poor teaching. Even with this, members of FECs should satisfy themselves of the nature and direction of comments, rather than depend on SEC reports for this information.
The collective bargaining agreement specifically requires the use of some sort of evaluation form (10.220, p. 29). However, the very openness of the charge allows for considerable latitude in evaluations. For example, the Law School practice of placing greater weight on signed evaluations, as opposed to those submitted anonymously, is not precluded by the CBA/.
Future Directions
It may be desirable to go beyond student evaluation of faculty to include also a university-wide program for the improvement of teaching. Such a program should involve both better evaluation procedures and real, available resources to help faculty improve their teaching skills.
More extensive evaluation is better conducted by peers than students and even better by a team of peers, some of whom might be knowledgeable unit colleagues and others professors well known for their instructional skills and general excellence in teaching.
Such assessment could include, for example, a review of course materials (e.g., syllabi, assignments, exams, reading lists), in-class observations of instruction over several occasions, and interviews with the instructor. Reviewers should reach agreement with the instructor beforehand with regard to the focus of the assessment; they should also be open to a variety of approaches to instruction and should be knowledgeable about them.
Given that there is certainly more than one way of teaching, an essential companion to assessing teaching is to provide a range of resources and effective strategies for improving teaching — opportunities for both the new and the experienced teacher to enhance their teaching strengths, correct or ameliorate any teaching weaknesses, and expand their repertoire of teaching techniques to respond to new and varying teaching situations. If we wish to take the task of evaluation of teaching seriously, we should do the best job we can and we should realize that the task will require new resources.
References
Barrett, Richard. 1998. “Comment on Paul Trout’s ‘How to Improve Your Teaching
Scores Without Improving Your Teaching’.” The Montana Professor 8(2):2-5.
Coren, Stanley. 1995. “When Teaching is Evaluated on Political Grounds.” The
Montana Professor 5(1):12-14.
Coren, Stanley. 1995. “Are Student Attributions of Instructor Racism and Sexism on
Course Evaluation Forms Valid?” The Montana Professor 5(1):15-16.
Platt, Michael. 1995. “What Student Evaluations Teach.” The Montana Professor
5(3):21-29.
Trout, Paul. 1996. “What Students Want: A Meditation on Course Evaluations.” The
Montana Professor 6(3):12-19.
Trout, Paul. 1997. “How to Improve Your Teaching Evaluation Scores Without Improving Your Teaching.” The Montana Professor 6(3):12-19.