the personal relationships of testing

This post is rated VT (very technical) for technical reflection on assessment, so it might get a little boring if you, Gentle Reader, are not involved in testing/assessment. Or, you might find it interesting as a little window on What Teachers Think About. In the Traditional System of assessment, a student wrote an examination essay at the end of the course that was in answer to a set question designed to express the main ideas and synthesis of course ideas through a recital of facts associated with the subject. The type of assessment we think about in the Pedagogy Biz has two forms: formative (as you go along, how is everybody doing) and summative (that trad exam above). An important idea in formative assessment is to have the test results be able to tell the instructor if the students are are getting the information that the instructor is trying to get across. It might seem simple, but in actual practice, there is whole lot more going on.

From the instructor's point of view, I want to design a test that asks the students to demonstrate their ability to work with the class material. I want to see if the course goals are being met. This purpose assumes I am honest about what my actual course goals are, so as a teacher, I need to make sure I am testing what I am teaching. It just seems fair that way, to me. However, as a student in this Irish system, I have learned about some new nuances of testing that are very interesting.

The test is part of the personal relationship that the teacher has with each student. It is sometimes the only one-to-one interaction with the professor that the student has in a big lecture of 100-400 students. It is also an expression of trust in the relationship: the professor will test me fairly. An exchange is implied where if I work hard for the professor (study, come to lecture, do the readings), then the assessments will deal with those things, or whatever the professor tells me I will be assessed. Because of the judging nature of the exams, and a score with grade consequences, I will adjust my behavior to accommodate that judgement. As a student, I understand that the only thing I have control of is my studying: the nature of the assessment is given to the person who, technically, I trust to know what I need to know. Thus, the test is also seen by the students as a measure of the professor's competence. Does this person understand her job sufficiently to be able to competently judge what I need to know and how to find out what I do know.

So studying is something I do for the course, but there is also an element that the studying I do is part of an exchange I have with the professor. There are multiple skills in test taking. Most obviously, one masters the content of the course, but there are several layers to this process. First, what type of content do I guess (and it really *is* guessing, from a student's point of view as a novice in the subject) that the professor will choose to put on the test. This is a guess most often informed by two clues: what the professor says in class relating to the test and cultural expectations of what is usually asked on tests. One might also guess from what is emphasized in the class lectures, since, presumably, this would indicate the emphasis of the course. However, all of these guesses assume that the purpose of the test is to allow the student to demonstrate her mastery of the course material. Nevertheless, there are purposes to testing such as instilling fear to make students work harder, to weed out a certain amount of the class who guesses poorly, either concerning content or structure, or even just a random fulfilling of the expectation that there be testing done. One cannot prepare for any of these purposes.

I have taken three tests at the University of Limerick so far in three different classes, and it is very clear to me that cultural knowledge is very important in studying. In all three classes, each professor was doing something novel by having a test in the middle of the course, and not having only a final exam. So, it is possible that this form of testing was new to them. In each case, they all announced in class that it would be a short test/exam/quiz (all three words were used) on the course material so far, and each one said, "Don't worry about it." In every case, this was false information, so any studying guesses based on believing the professor would have been wrong. The tests were all for at least 10% of the final grade: a student should be concerned about this. The course materal cited in some of the questions was tangential to the main points made in the lectures: one had to closely study one's notes and any ancillary material provided. They were all three "something to worry about." In all three tests, the focus was on the recitation of facts. In no case were course themes, ideas, or analysis requestsed, which had been the focus of the lectures I attended. These tests were all at the bottom of Bloom's taxonomy: memorize and identify. In all three tests, obscure details were requested in at least two or three questions that had nothing to do with the main content of the course.

I was not able to speak with students about one of the tests, but I did speak with students about the other two afterwards. In both cases, the American students were reeling. There was a clear case of complete misunderstanding of how to study and what to study for these tests. The Americans were focused on ideas and themes, not "What character in the play said this line" or "What type of flute was James O'Donnel playing in the sample played in class." The Americans had studied Big Picture but the tests were asking for the numbers on the topographic lines. Also clear in these conversations was a feeling of betrayal. I was suprised by this, but I also recognized it and shared the feeling. That the professors had not been clear about what they wanted us to know, and the apparently obscure details they were asking for was regarded as a breach of trust between student and professor. One student said she found the test she took 'insulting'. Another student commented, 'I was starting to like that guy, but now I totally don't trust him. What's the point of working hard for his class when he'll do something like this to you?"  From my point of view, these tests indicated a very different way of knowing the subject.  I lacked the cultural/social back ground to know what I needed to know.  When I asked one of the lecturers how grades fell out on his test, he said the German students did the best, American students not so well, and the Irish students were kind of up and down the scale. 

From the Irish students' point of view, they accepted this as par for the course. "You never know what they want you to know. You memorize as much as you can, and if you get a C, you still pass." "No one can get an A here, so why try?" No personal relationship with the professor was assumed: the teacher was someone who did things to you. "If you care, you'll just get disgusted so why bother?" said one Irish student in his fourth year, "Just memorize the exam papers from last year. There's no point in going to the lectures." This alienation suprised me, but the practical approach to high stakes exams made sense. But as someone who does care, , it seemed so dry, so impersonal, so sad to think about course content this way.

My other level of concern is that these tests clearly would not tell the professors anything useful about the classes, about what ideas were being understood, about what concepts might need more attention. It really was busy work for them. In one class, the professor commented to the class, "On the test it seemed like a lot of you were just guessing. That's not good," and that was the only feedback she gave. All she did was tell us we were losers. Well, I thought, if the majority of the class had trouble with the assessment, would that not tell the lecturer something about either the test (it was a bad test) or the class (something about the content needs to be clarified because clearly what you want to get across is not getting across). The professor's automatic assumption that the class was a bunch of slackers was insulting and discouraging.

So what have I learned so far from this? A test is a part of the relationship I have with my students. I need to be responsible with my end of the bargain: I must be clear and fair about what I want students to know and how I ask them to demonstrate it. A test with both identification/definition as well as analysis/synthesis will give me a better portrait of how the class is doing than just one or the other. I need to remember that my students are novices in the topic in which I am an expert. What seems easy to me might still seem hard to them. It is not cheating to be clear up front with my testing expectations for the students, and my tests should reflect what I have emphasized in class. Partial credit is not a bad thing. Testing is a demonstration of my leadership of the class, and I need to be able to demonstrate that I am a person worthy of being trusted with that power.