Teacher evaluations - can we learn from them?

A quick Facebook comment from a fellow professor made a very good point in a permanently ongoing discussion among teachers: can we learn from student evaluations? His point was that he could very easily create a course that would make the students happy and ensure good evaluations, but he didn't feel it would be an ethical act. Why? Because he would not act according to his standards of good teaching, but according to the format for the evaluations used at his University.

One of the discussants in that thread pointed out that high scores in evaluations depend on managing expectations. If my friend had made a standard course, according to the expectations of the students, the score would quite likely have been very high. A standard course makes students feel safe, it helps them deal with the insecurity of being in a learning process. For good reasons students abhor insecurity. They know they will be graded according to their performance, so they need to minimize the risk for bad grades. And so they have learned strategies for this, by learning how to score well on certain standard tests, how to learn well in certain environments, and how to study by using certain tools. When they are asked to use new tools, they react very negatively, and try to control the situation by managing our expectations. And their main tool for managing our expectations is the evaluation: if we are bad teachers, them failing will be our fault, not theirs.

I find that students who are stressed and afraid give much worse evaluations than students who feel reasonably secure. Also, students who have a wide range of experiences, and know that they can learn in very different environments, give much better evaluations when they have to relate to something new, than students who have only experienced a narrow range of teaching/learning styles. Whether or not the students are used to doing well, if they can expect to do well, or if they eventually do well doesn't matter, it's their experience of stress at the moment which influences the evaluation they give the teacher.

One term I taught the same course in three different classes. That gave me three different evaluations to read. One class was extremely negative in their evaluation, and several of the comments were pure character assassination of me and the other teachers (there were three of us). One class was critical, but not personal, and while they didn't all love the course, they didn't set out to trash the professors either. One class loved the course, loved the topics, and loved the teachers. All three classes had received the exact same lectures by the exact same teachers. The difference between them was in the dynamic in the room, how the students related to each other.

In the negative class, when we discussed something there would be two responses: The loud, main one was a demand to change things, mainly the teaching, in a way that they liked better. My experience from several years of trying out what would work didn't matter, their experience with what made them feel comfortable was more important. In the same class, several people who never spoke in class would later come up to us and say: "I love the way this is done. You are all great teachers. Don't change things just because of this. It works for us." The group obviously had an internal dynamic that was killing the openly spoken diversity. Some strong voices made the others feel afraid of speaking up, and when I did not submit to their pressure, it made them increasingly aggressive. Now I made a few errors with them - rather than saying "no" to the changes they wanted, I should have let them know that if I did what they wanted, I'd have to stop doing something else, which I knew they also wanted. I didn't have the resources to do both. Instead I just said "no, not going to happen." That was bad management of their demands, from my hand. After that, they didn't see the changes they wanted that we did make: they were all convinced I would never listen. Interestingly, the same group decided among themselves that a letter did not contain what to them was vital information, and afterwards they were not able to read that information when they read the letter. I had several students make the claim that they had not been told A, and then, when we read the letter together, I could show them where the letter said A. This kind of selective perception is interesting, and quite daunting when such a large group suffers from it. It does say something of the power of self-deception inherent in group dynamics, and the power of the negative thinking I was unable to break up.

In the fairly neutral class, the students were outspoken, vigorous and active, they took charge of the tasks they were given and approached them with a positive attitude, and worked hard at making things work. They tested out new things, asked questions, participated in discussions. When they came up after class, it was to talk about examples, ask questions, be to-the-point. They didn't think we were god's gift to teaching, but it didn't matter, they managed, they responded and they helped each other figure things out. The main characterestic of that class was that they were not afraid. They were not afraid of laughing, being laughed at, being sanctioned by each other. Their group dynamic was positive and generous, and they also addressed each other while in class, and not just us.

In the extremely positive class, they were all very quiet, but it was a contemplative silence. This group took it all in. Their questions were reasonable. They read not just the readings, but also the emails and the material describing the course. They smiled when they talked to us, and they approached us with questions about what they felt was unclear, rather than with demands to change things the way they wanted it.

So, how can I get a better evaluation, based on this experience? Obviously, the main errors were made with the negative class. I should have managed their demands better, given them a feeling of influence rather than saying "no". But both cases of "no" were based on my previous experience. The first was due to several years of positive feedback to how I presented the readings. If I was to do it the way this class wanted it, I'd have to give up the way I was doing it, a way I knew worked from several previous evaluations and conversations. The second was due to experience with student results. When I had previously done things the way these students wanted it, certain errors were a lot more prevalent than they ended up being. Sticking to my experience helped the class as a whole to maintain their focus on the correct task at the time, rather than mixing up the tasks, as bad timing had caused earlier.

I was in a situation where I knew that not giving in would make the students annoyed, but giving in would cause their results to deteriorate. At that point I chose to get a lower score on their evaluation of me, in order to ensure that they performed better in the long run.

And this is the reason why my friend on Facebook doesn't want to design the "perfect score" course. While the students are experts on what makes them feel they control their individual learning process, we, the teachers, know a thing or two about how to manage courses. Sadly, some, like me, also have a temper, and will in a stressed situation answer directly and honestly, rather than pause and consider how to make students feel good about what I am about to tell them. And this is actually a bad thing for a teacher, because students are genuinely nervous, with good reason. They are in a tense situation, and I need to address that fear as much as I address my own truth. I need to make them feel that I am on their side, and my choices are made to help them, not to make their lives more difficult.

Sadly, that doesn't always work. But all I can do is to try again. Redesigning the course to get a good score on the evaluation of me is not going to happen. Everything I do is intended to make sure the students get a good grade, and that, in my experience, they do. If it means they have to be angry with me to do it, well, so be it.

Quiet time to do research

Denmark, like most countries, has an ongoing debate about how Universities are to get as much as possible out of the Universities. Due to very good economic support for students, many young people try to ride out the crisis by taking an education. That is not the worst way to go, and I am all for offering education as an alternative to unemployment.

However: the Universities expected to deliver this education get very little extra funding for it, no new positions and also a change of policy which rewards externally funded research, and takes teaching for granted (link in Danish, it's about the demand for more teaching in the Universities.) Now, professors are not stupid (if we were, that would indeed be a sad state of affairs), even if they sometimes are idealistic. And so, when the government says: get more research funding - well, we get more research funding, putting researchers on the case, while finding ways to keep the teaching rolling at the side.

This leads to teaching which is not research based, because all the professors are busy doing research for external funding, just like the government ordered! In order to fix this, the government insists that students need to have more classes, and these classes have to be with the professors who actually do research: research based teaching.

I am very much in favour of this, so I have no problems with the demand for more research based teaching. It's the one benefit Universities have over private research organisations: I get to meet students. Over the years, students have challenged me, pushed me and rewarded me in ways no other part of my job does. But they challenge me into doing research. Without research, which gives me something new and interesting to share and each, facing the students become a repetitive chore.

At the same time, research appears to be viewed as a time of "quiet", of "peace". "We need to give the staff quiet, calm times to do research," professor Sune Auken from the blog Forskningsfrihed says. The error in this is how it puts teaching, apparently a stressful, draining chore, up against research, a calm, peaceful, contemplative activity. In my experience, this is wrong. Of course, included in research is long hours of reading, days of checking for literature, months of analysing data, sometimes years of organising and writing up theories and findings. It may look peaceful. Still, if you're doing research worth the paper it's printed on, it's stressful, tense, fraught with doubt and built on meticulous, tedious fact-checking and proofing, in all possible ways.

Research isn't a time of calm or quiet, it's a time of immersion, introspection and tension, put up against a time of activity, communication and attention to others. In universities these two positions do not oppose each other, they play together and rejuvenate each other. The teacher needs long stretches of time protected from teaching in order to immerse herself into research. The researchers needs to come out of the immersion and communicate, be challenged and be pushed, in order to have something new to bring into her time of immersion.

Of these activities, teaching is the one that it is possible to hire others for though. Entry-level teaching can easily be done by somewhat more advanced teachers, that's not where researchers really have to engage. And so in order to meet the demands for research, universities look at the teaching hours, and move the researchers from the least demanding classes - which also happen to be the most populated. It's a matter of trying to work more efficient - as the government constantly insist the Universities should do. This way some students will get research-based teaching. Hiring non-teaching researchers to do the research while the professors teach means NO students will get research based teaching, as their professors will always be teaching, and never researching.

In order to create a good policy of teaching and research, the government needs to remember that it's the same people who have to do both. It's impossible to drain the university brains into externally funded research, and expect the same heads to have time to teach. And it's impossible to demand research based teaching, and not give the teachers time and opportunity to do research. There are only two answers to this: Fewer students, or more professors. Barring some groups in different organisations - there are always some groups that have more slack than they really ought to have, I have no idea where, but they exist, it's the law of organisations - that is the only way the government can have both more teaching for each student, and more research for each professor. I am sorry. There are no easy ways to split the professorial heads, and neither research based teaching, research itself happens without effort.

When teaching works

One of the things about being a teacher is that it's pretty hard to know when teaching works. I tend to believe that the reason things work out is because I have such brilliant students. I am a pretty privileged teacher, I have a lot of brilliant students, they participate, they share, the listen and they learn. It's a wonderful place to be in.

Sometimes, however, I get this sneaking suspicion that I have influenced them. And I am not talking about the many, many students traumatised to the point of never being able to squeeze another ketchup bottle after advertising lectures with graphic images. No, you see, sometimes I do something with my teaching because I want students to get better at one particular skill, and they get better at it!

I have just come out of a marathon of paper grading, the curse of every professor. When the students go on vacation, we hole up with supplies, good reading light and a red pen, and get to it. This is a strange and exhausting process, as it demands intense focus for hour after hour. Luckily the students I grade these days submit computer written papers - when I also have to decode handwriting, this is headsplittingly hard. Anyway.

Going into a long period of grading is like zoning into a very special slice of life. It's just you, the reading list, the course list, the demands to the papers, and this large stack in front of you. It's easy to feel disconnected from reality and worry that you're being too nice or too harsh, or believe you are hallucinating. Sometimes we are. Grading large stacks takes endurance and practice to do well - at the end of day two, the words blur and you turn pages with only a vague idea of what is on them. Again and again you go back, read over, check the notes, worry you may have been in that empty trance state too long, and can't get it all back, but you have to press forwards to the end of the stack.

Then you call the other assessor. You communicate - carefully - your idea about how the grades are this year. Surprisingly often you and the other assessor agree! This is always a miraculous moment. There is something measurable, and you have both sensed and measured it in a manner that is at least comparable!

This year I got this sneaking feeling early; that this was a generally good stack. (Now, if you are a student, no, I am not giving you your grade until all the formalities are in place, and then you have to call the administration. The papers are in the mail between me, the censor and the student administration. Wait for it.) Not that they were all brilliant, some were brilliant, some bad, some failed. It's how it is. But they were on average much better than I have been used to! I thought this was odd, because it's been a rough semester. I had to bring in several external leturers (thanks folks, you did really good by my students!) and rely on the TAs a lot (and they were amazingly reliable). So I started to pay attention, to try to figure out what had happened.

One thing stood out: They papers were organised! They had a strong and clear structure, they had developed the research questions, they had all elements we ask for from a paper, and they used literature well and correctly. I almost wanted to cry. I have been drilling this into student heads all autumn, repeating the structure of a paper, the reference systems, the reason for both and the logic of scholarly argumentation. I have linked reference systems and written outlines, and then repeated it all one-on-one for those who came for supervision. I worried I had overdone it.

If I overdid it, at least the message got through. Even the weak papers were structured. In several cases this saved them from a total crash and burn, as the adherence to structure helped discipline the progress. And this I know is different from the terms before. This one thing was drilled more, and more systematically. The teaching made a difference.

So, now I have that working for me. Tomorrow I'll be at the library, considering what else I need to spend time on. I am thinking of how to tighten up the reading list and the progress of lectures, and couple it better with the exercizes. I can't promise that it will work as well, but I can guarantee that I will try.

The written woman

This blogpost is about a lot of things, but mainly it's about the two books written by a former student, "som i himmelen, så og på jorden" and "men fri oss fra det vonde." Both titles are taken from the Norwegian version of The Lord's Prayer, which in Norwegian is named for the first two words: "Our Father - Fader vår." They are definitely not religious pamphlets though. I am not naming the author, since she has chosen to be anonymous.

In her books I hear her voice. It flows, wicked and funny, through them both, as she gives us mystery and thrillers from an intelligent woman's perspective. They are early books and flawed, showing the lack of a large publishing firm around them. Still, they are special, not just because this is a person I like, but because they have taken the traditional gender hierarchy in action novels and written the world as it both of ten is, and as it should be. In these books, Norwegian - nah, Scandinavian writers have been given crime novels where women are powerful not because they are sexy, but because they are intelligent, brave and willing to take risks.

The stories are complicated and ambitious. In one, the women set out to save the black rhino, taking on a large group of illegal hunters, poachers, and they use both feminine viles and feminine power to get there. These are not man-hating books though. Making all the heroes female and all the villains male would be too easy. The main actors are female, though. Matriarchs, even, in some cases, wise to the point of cruelty in the decisions they have to make.

Both books are concerned with what is happening to the planet, right now, the loss of species due to human greed and stupidity, and the threath of the climate change. There are conspiracies, spies and murders, silent actors pulling the threads from behind the scene, and more than one point of comic relief. However, there are weaknesses too. Abrupt, sudden leaps in the story, bewildering decisions, jokes that are a little too "in" for me, at least, to get them. I don't see this as a big problem though. It's early works, with a very small publisher. The books are still worth a read.

How come I read them?

I received these books last summer, as I was leaving Volda. It was, perhaps, the best gift anybody could have sent me off with; as the author and former student let me know, quite clearly, how important I had been to her as a teacher. Such things always take me by surprise. I work, work some more, and then I work, I never stop to think if I please, if I change something. I give of what knowledge I have and can gather. And then, sometimes, it is enough.

This student though, she wasn't hard to remember. She was on the front row, she asked questions, made comments, took notes. She was the terror of some of the other teachers, but I liked her sharp, often wicked, wit. Perhaps I saw myself in her?

In her recent mail, she told me I taught her to see and analyse in ways she had never done before. Little did she know that she taught me to be an honest, straightforwards and humble teacher.

Things I never thought I'd miss

On the list of things I never thought I was going to miss, Fronter is way up there. Volda University College started using it when it was new, experimental, horribly buggy and impossible to manouver in. Frustration abounded, with access, with the interface, with the demands on us for detailed administration and a much more computer-bound work structure. I hated it. Then I started to tolerate it. Then I discovered the occasional useful factor, such as not having to keep track of a stack of papers - I could just have the students submit to fronter, the system kept track and I could accept, comment and refuse, and the students got immediate feedback. There would be a history for me and the student to go back into (finally, after a couple of years fighting for the students to keep access to their earlier work), and it generally made for a much tidier way to cooperate and share, particularly when the students had to submit work to me.

I didn't even notice how easy that was, until I suddenly stood without this option.

I am now at ITU-Copenhagen, where some of the really interesting researchers on Internet technology and culture work. Of course we don't have an easy, simple, accessible net-based platform for students and teachers to cooperate on. That would have been too easy, wouldn't it? (Touch of academic traditional irony there: Organisation studies are disorganised, communication studies never communicate, art studies are in ugly, badly decorated buildings.)

This semester, I have 80 students to keep track of. I have to do it by hand. I never thought this would happen: I miss Fronter.

Why we don't like cell-phones in class

Thanks to Pattie Belle Hastings for posting this link, to a study about cellphones, ringtones and recall by Jill Shelton.

"Many of us consider a cell phone ringing in a public place to be an annoying disruption, but this study confirms that these nuisance noises also have real-life impacts," Shelton said. "These seemingly innocuous events are not only a distraction, but they have a real influence on learning."

Titled "The distracting effects of a ringing cell phone: An investigation of the laboratory and the classroom setting," the study was conducted at Louisiana State University, where Shelton received her doctoral degree. Her co-authors in the LSU psychology department include Emily Elliott, Sharon Eaves and Amanda Exner.

Apart from being annoying, distracting and rude, ringing cellphones makes students forget what they learned before and during the ringing of the phone. If the ring tone is a popular, well-known piece of music, this is even worse.

So: That mute button? Use it!

First day of term

I met the students for the first time - again - today. I am growing old. You see, I was just about moved to tears. Students are so incredibly fresh, like the term, like our plans and hopes, and there's so much that can happen from here on and through the next three years...

Anyway, to get away from the sentimental stuff:
We have a large group, almost doubled the number since the sudden low two years ago. This is something we are extremely happy about, and a challenge we're ready to meet. This is not the largest group we have worked with, but it is still big for a lot of the teaching-intensive subjects we teach.

Due to the reform of education in Norway, we now have very homogenous groups, where most students are young and have little work experience before they start. This is a challenge, as it means they have a lot to learn about society, but at the same time we have them for three years, not two, which means that we have more time for this process than we used to.

The information study staff are at a long-time high, there are less leaves and more of the regular staff at work at the same time than we have had for a long time. YAY for that, as working with a good, experienced staff makes life so much easier. It leaves room for all of us, if not for anything else, then to breathe in and out slowly a few times, look around and figure out what we are doing and what we should do. Definitely the best way to start a term.

The study has been tweaked in the last year, and I returned to find that it was changed to eliminate several of the issues that have been hard to work around. Do you understand what a good feeling that is: to go somewhere else and think about something else for a year and then return to something that works better than it did before I left? OK, the conclusion is that perhaps I should leave more often... but right now I am just delighted at the work that's been done while I was away.

And now I have a new class of students. Young, fresh, unmarked students. And this is where I quit before I get started again!

Understanding Games

Want to understand games, but can't get to the lecture? Well, go play this educational game about games! I have to admit I haven't gone through the whole thing, after a while they wanted me to actually INTERACT and that got too exhausting. But it's a fun idea, and should be interesting for academic gamers.

The game is a thesis developed in Potsdam, Germany, copyright is 2006-2007, so most likely submitted in 2007.

Understanding Games is a diploma thesis project designed and developed by Andreas Zecher at the University of Applied Sciences in Potsdam, Germany. The project was advised by Prof. Boris Müller, Prof. Klaus Dufke and Mattias Ljungström.

The game has been around for a while, and if you want to read more about the responses to it, Cory Doctorow has a pretty good description in Boing Boing.