D or F?

Some time ago, I assessed a master's thesis at a Norwegian university (I do so regularly, at different universities). It was a half-assed piece of work, and I gave it a low mark, as I should. This didn't happen without discussion or resistance; the other assessor disagreed initially. We went through the thesis very thoroughly, and also read the descriptions of the use of the different grades in the Norwegian system. At the end we agreed that the grade was the only possible one, from the content and the descriptions. The internal assessor had hoped for a better grade for the student, but that's why we use external assessors in this system: in order to have qualified, unbiased readings.

In this process, the other assessor said something that stuck with me. "But when they apply abroad, they look at this grade, and our students don't have a chance, because in the US, the grades are used differently." The implication was that this paper would have gotten a much better grade in the US. Now, I can't grade according to a hypothetical standard in another country, so the argument didn't change anything. But it triggered a curiosity about the performance of American students vs Norwegian, because one of my impressions is that in the US, there is a much higher general failure rate. In Norway, if you start an education, you may not always excel, but you have a very high chance of finishing. My impression was that in the higher education in the US, the drop-out rate is much higher.

Today I read about the failure rate in MOOC - Massive Online Open Courses - in California, or more precisely, San José. Now, comparing regular teaching with MOOCs is unfair. Any teacher with a tad of experience knows the difference between seeing the students face to face and writing them. However, a sentence in the article caught my attention:

Gov. Jerry Brown had lauded the goals of the program to allow students to graduate faster and reduce their debt loads at a time when only 16 percent of California State students graduate in four years.

 16%

Almost all Norwegian students are state students. The education is free. How many graduate in four years? From SSB, the state-owned statistics bureau (which, by the way, has been noted as being more critical of the educational system than the private agency paid to assess it.):

Av de om lag 30 600 studentene som startet på bachelorutdanning for første gang i 2006, fullførte 45 prosent innen tre år. Etter ytterligere to år hadde 62 prosent fullført. Sammenlignet med studentene som startet i 2005, har andelen som fullførte innen fem år økt med 4 prosentpoeng. I løpet av denne femårsperioden var det 23 prosent som avbrøt studiet. 9 prosent av disse avbrøt etter første året. Bachelorutdanningen i økonomiske og administrative fag hadde høyest andel av studenter som avbrøt i løpet av fem år, hele 33 prosent.

45% have finished in three years. In five years, 62% have finished. 23% have quit, and didn't finish over five years. We don't know if they resumed/will resume their education later. Now, discussing which universities and colleges we should compare our universities with is a bit useless. Norway doesn't have the private Ivy-league schools, but we have a pretty solid state-run educational system. We can't really compare either way, but we do tend to compare our system with the Ivy-league schools, as these are the goal for many of our students who go abroad. But who are the Ivy-league students? They are the children of the educated and well-off, and what does statistics tell us about academic success and social background?

If we look at the figure in the article from SSB, it's pretty clear. Children of parents with higher education finish more higher education. Children of parents with no education finish less. (Considering that none of my parents finished high-school, I am an anomaly according to this statistic: a person from my background should not have a doctorate. Yes, I thank the Norwegian welfare state and the miracle of our educational system regularly.) We compare a system which encompasses students from all backgrounds, with a system which exclusively caters to those with the best chances. We expect our students to compete with the very top of the heap of an extremely competitive system, where failure and dropping out is more common than completing an education.

There's no real conclusion to this. I am not saying which of these two systems is the better (although, see above, I know where my loyalty lies.) What I am saying is: They can't really be compared. The Norwegian educational system needs some serious tweaking, I agree. But I don't feel bad about giving a low grade to a student who doesn't perform well. They are still miles ahead of the large numbers of US students who do not get a grade at all. If we are to compare our students with the American ones, we need to compare both ends. And when it comes to making sure the population is educated, we perform very well indeed.

Teacher evaluations - can we learn from them?

A quick Facebook comment from a fellow professor made a very good point in a permanently ongoing discussion among teachers: can we learn from student evaluations? His point was that he could very easily create a course that would make the students happy and ensure good evaluations, but he didn't feel it would be an ethical act. Why? Because he would not act according to his standards of good teaching, but according to the format for the evaluations used at his University.

One of the discussants in that thread pointed out that high scores in evaluations depend on managing expectations. If my friend had made a standard course, according to the expectations of the students, the score would quite likely have been very high. A standard course makes students feel safe, it helps them deal with the insecurity of being in a learning process. For good reasons students abhor insecurity. They know they will be graded according to their performance, so they need to minimize the risk for bad grades. And so they have learned strategies for this, by learning how to score well on certain standard tests, how to learn well in certain environments, and how to study by using certain tools. When they are asked to use new tools, they react very negatively, and try to control the situation by managing our expectations. And their main tool for managing our expectations is the evaluation: if we are bad teachers, them failing will be our fault, not theirs.

I find that students who are stressed and afraid give much worse evaluations than students who feel reasonably secure. Also, students who have a wide range of experiences, and know that they can learn in very different environments, give much better evaluations when they have to relate to something new, than students who have only experienced a narrow range of teaching/learning styles. Whether or not the students are used to doing well, if they can expect to do well, or if they eventually do well doesn't matter, it's their experience of stress at the moment which influences the evaluation they give the teacher.

One term I taught the same course in three different classes. That gave me three different evaluations to read. One class was extremely negative in their evaluation, and several of the comments were pure character assassination of me and the other teachers (there were three of us). One class was critical, but not personal, and while they didn't all love the course, they didn't set out to trash the professors either. One class loved the course, loved the topics, and loved the teachers. All three classes had received the exact same lectures by the exact same teachers. The difference between them was in the dynamic in the room, how the students related to each other.

In the negative class, when we discussed something there would be two responses: The loud, main one was a demand to change things, mainly the teaching, in a way that they liked better. My experience from several years of trying out what would work didn't matter, their experience with what made them feel comfortable was more important. In the same class, several people who never spoke in class would later come up to us and say: "I love the way this is done. You are all great teachers. Don't change things just because of this. It works for us." The group obviously had an internal dynamic that was killing the openly spoken diversity. Some strong voices made the others feel afraid of speaking up, and when I did not submit to their pressure, it made them increasingly aggressive. Now I made a few errors with them - rather than saying "no" to the changes they wanted, I should have let them know that if I did what they wanted, I'd have to stop doing something else, which I knew they also wanted. I didn't have the resources to do both. Instead I just said "no, not going to happen." That was bad management of their demands, from my hand. After that, they didn't see the changes they wanted that we did make: they were all convinced I would never listen. Interestingly, the same group decided among themselves that a letter did not contain what to them was vital information, and afterwards they were not able to read that information when they read the letter. I had several students make the claim that they had not been told A, and then, when we read the letter together, I could show them where the letter said A. This kind of selective perception is interesting, and quite daunting when such a large group suffers from it. It does say something of the power of self-deception inherent in group dynamics, and the power of the negative thinking I was unable to break up.

In the fairly neutral class, the students were outspoken, vigorous and active, they took charge of the tasks they were given and approached them with a positive attitude, and worked hard at making things work. They tested out new things, asked questions, participated in discussions. When they came up after class, it was to talk about examples, ask questions, be to-the-point. They didn't think we were god's gift to teaching, but it didn't matter, they managed, they responded and they helped each other figure things out. The main characterestic of that class was that they were not afraid. They were not afraid of laughing, being laughed at, being sanctioned by each other. Their group dynamic was positive and generous, and they also addressed each other while in class, and not just us.

In the extremely positive class, they were all very quiet, but it was a contemplative silence. This group took it all in. Their questions were reasonable. They read not just the readings, but also the emails and the material describing the course. They smiled when they talked to us, and they approached us with questions about what they felt was unclear, rather than with demands to change things the way they wanted it.

So, how can I get a better evaluation, based on this experience? Obviously, the main errors were made with the negative class. I should have managed their demands better, given them a feeling of influence rather than saying "no". But both cases of "no" were based on my previous experience. The first was due to several years of positive feedback to how I presented the readings. If I was to do it the way this class wanted it, I'd have to give up the way I was doing it, a way I knew worked from several previous evaluations and conversations. The second was due to experience with student results. When I had previously done things the way these students wanted it, certain errors were a lot more prevalent than they ended up being. Sticking to my experience helped the class as a whole to maintain their focus on the correct task at the time, rather than mixing up the tasks, as bad timing had caused earlier.

I was in a situation where I knew that not giving in would make the students annoyed, but giving in would cause their results to deteriorate. At that point I chose to get a lower score on their evaluation of me, in order to ensure that they performed better in the long run.

And this is the reason why my friend on Facebook doesn't want to design the "perfect score" course. While the students are experts on what makes them feel they control their individual learning process, we, the teachers, know a thing or two about how to manage courses. Sadly, some, like me, also have a temper, and will in a stressed situation answer directly and honestly, rather than pause and consider how to make students feel good about what I am about to tell them. And this is actually a bad thing for a teacher, because students are genuinely nervous, with good reason. They are in a tense situation, and I need to address that fear as much as I address my own truth. I need to make them feel that I am on their side, and my choices are made to help them, not to make their lives more difficult.

Sadly, that doesn't always work. But all I can do is to try again. Redesigning the course to get a good score on the evaluation of me is not going to happen. Everything I do is intended to make sure the students get a good grade, and that, in my experience, they do. If it means they have to be angry with me to do it, well, so be it.

When teaching works

One of the things about being a teacher is that it's pretty hard to know when teaching works. I tend to believe that the reason things work out is because I have such brilliant students. I am a pretty privileged teacher, I have a lot of brilliant students, they participate, they share, the listen and they learn. It's a wonderful place to be in.

Sometimes, however, I get this sneaking suspicion that I have influenced them. And I am not talking about the many, many students traumatised to the point of never being able to squeeze another ketchup bottle after advertising lectures with graphic images. No, you see, sometimes I do something with my teaching because I want students to get better at one particular skill, and they get better at it!

I have just come out of a marathon of paper grading, the curse of every professor. When the students go on vacation, we hole up with supplies, good reading light and a red pen, and get to it. This is a strange and exhausting process, as it demands intense focus for hour after hour. Luckily the students I grade these days submit computer written papers - when I also have to decode handwriting, this is headsplittingly hard. Anyway.

Going into a long period of grading is like zoning into a very special slice of life. It's just you, the reading list, the course list, the demands to the papers, and this large stack in front of you. It's easy to feel disconnected from reality and worry that you're being too nice or too harsh, or believe you are hallucinating. Sometimes we are. Grading large stacks takes endurance and practice to do well - at the end of day two, the words blur and you turn pages with only a vague idea of what is on them. Again and again you go back, read over, check the notes, worry you may have been in that empty trance state too long, and can't get it all back, but you have to press forwards to the end of the stack.

Then you call the other assessor. You communicate - carefully - your idea about how the grades are this year. Surprisingly often you and the other assessor agree! This is always a miraculous moment. There is something measurable, and you have both sensed and measured it in a manner that is at least comparable!

This year I got this sneaking feeling early; that this was a generally good stack. (Now, if you are a student, no, I am not giving you your grade until all the formalities are in place, and then you have to call the administration. The papers are in the mail between me, the censor and the student administration. Wait for it.) Not that they were all brilliant, some were brilliant, some bad, some failed. It's how it is. But they were on average much better than I have been used to! I thought this was odd, because it's been a rough semester. I had to bring in several external leturers (thanks folks, you did really good by my students!) and rely on the TAs a lot (and they were amazingly reliable). So I started to pay attention, to try to figure out what had happened.

One thing stood out: They papers were organised! They had a strong and clear structure, they had developed the research questions, they had all elements we ask for from a paper, and they used literature well and correctly. I almost wanted to cry. I have been drilling this into student heads all autumn, repeating the structure of a paper, the reference systems, the reason for both and the logic of scholarly argumentation. I have linked reference systems and written outlines, and then repeated it all one-on-one for those who came for supervision. I worried I had overdone it.

If I overdid it, at least the message got through. Even the weak papers were structured. In several cases this saved them from a total crash and burn, as the adherence to structure helped discipline the progress. And this I know is different from the terms before. This one thing was drilled more, and more systematically. The teaching made a difference.

So, now I have that working for me. Tomorrow I'll be at the library, considering what else I need to spend time on. I am thinking of how to tighten up the reading list and the progress of lectures, and couple it better with the exercizes. I can't promise that it will work as well, but I can guarantee that I will try.

Thinking in your first language

This was supposed to be a post about cheating, and ghost-writing academic articles. However, Lisbeth connected it to educational language politics, and that started this. Read at your own peril - it's all about language and education from somebody who only knows half of it.


When ever I spend a lot of time away from Norway, my language suffers. Three - four weeks into a long stay in an English language country, and I forget all language. I am unable to formulate thoughts and ideas, numbed by the constant translation that takes place in my head, even if I do think in the language I speak. I am suffering a version of this here in Denmark, not as harsh, but more subtle and exhausting, as the constant translations in my head are concerned with subtle details. The only cure is to immerse myself, for a while, in Norwegian, to soak up the words, the music of the dialects, the structure of the sentences. I need to find the voice of my original thoughts in order to find a voice at all.

When I had this experience the first time, I never connected it to the language conflicts which gave birth to nynorsk - new Norwegian - or the political struggle for first language classes for non-Norwegian-speakers in school. But then I discovered a pattern: How not hearing or speaking my own language exhausts me, numbs me and slows me down. It doesn't replace one language with another, it takes away all languages, and I struggle in my search for words.

I guess you are now not surprised the to learn that no, I don't think there should be a lingua franca which all academics should use, from school-children up to professors. I also don't think we should force all students to write all their work in for instance English, just because it will make it easier for an American scholar to read their papers.

This doesn't mean I think we should stop communicating. I am truely grateful that I can talk to Italians, Portuguese, Koreans, Turkish, French and Finns without me having to learn 6 more languages. I'd love to, but I'd need some serious state grants to support the learning period. What I mean is that in order to be able to use our energy on thinking, language needs to take the back seat. We shouldn't insist that our students constantly struggle with words when what we want them to struggle with to be ideas, concepts, theories, hypothesis and arguments. It is enough, really. After all, if they, like me, were born to a social strata where higher education was a far away dream, not an everyday chore, then learning to be an academic is a whole new language in itself.

I would like to see two directions for language in education. First, I would like to see the language education in general strenghtened, and with it culture and literature. People who speak more than one language fluently are more empathic, think quicker, assimilate knowledge faster and in general appear to have more flexible minds. So that's a resounding yes for more emphasis on languages. However, I'd like education to contain more than two languages. We should learn to speak and regularly be encouraged to practice 3 or four languages. Why? Because if we can listen to how another person speaks in their own language, we will meet them as very different persons.

When we study, we should be allowed to learn in our own language, learn to have discourse within our own culture. With a base in the mother tongue, we should then challenge others, stretching ourselves in order to learn not only mastery of another academic tradition of discourse, but also respect for it, for the difficulties others need to overcome in order to communicate with us. It is hard to be a foreign student. It is hard to be an academic from a small country, forced to publish and cooperate internationally. But we should focus on the benefits to be had by learning not just one or two languages, so we can get by, but several: opening the world to us in a very different way.

If internationalisation is to get anywhere, we need to teach how to communicate. Even today, that means in more than one language - but it also means honouring our own while we do it.

Game of the future

The Norwegian University og Science and Technology NTNU celebrates its 100 year anniversary, and to celebrate they have designed the game Spillet om framtiden or "The game of the future". In this you can be the headmaster of the University, fighting some mysterious female evil bandit (I am not sure what she has done, but she is dressed in black and has short, white hair. Is she perhaps a representative of the humanist- or the social sciences, directions which are not prioritised in Trondheim?), solving several puzzles as (you) he zoomes around the Trondheim campus in his red coat.

Despite the language in the cut scenes (did I mention that this is a technical university, not one with a strong language department?), it's a very funny little game, and I am quite delighted with how it combines playfulness, science fiction, game clichés and information about what goes on at the campus.

Less women than 25 years ago

According to New York Times there are less women in computing - or at least in the educations - than 25 years ago. And it's not just about building the right game to use to draw them in.

ORLY?

It's been a while since I complained here about the "quality reform" in Norwegian education. The Norwegian education system was altered: 3 year bachelor degree instead of 4 year cand mag, master degree after 5 years instead of hovedfag after 6-7 years, more teaching and supervision, less time for reflection and independent research for the student and also less time for research for the faculty.

Imagine the surprise when it turns out that less time to study, more pressure towards keeping the schedule, less qualified and updated teachers and less resources for testing and assessing the students led to less knowledgeable students.
*insert sharp tone of voice, desillusioned sarcasm shining through*

Thankfully we still have people who remember how to do research in this country, and at the University of Oslo some of them did exactly that, they checked if students learned the same in three years as they used to do in four. The answer was no, they don't.

The paper "Morgenbladet", one of the few papers where they still read research reports and 100 page evaluations (or at least the press releases about them), calls the syndrome "kunnskapsfallet" - the fall of knowledge. The evaluation of students at the University of Oslo confirms what people in colleges and universities have been saying from the beginning. There is very, very little in the "quality reform" that makes for better students.

It is a quality reform though. That much is true.

Thanks to Jon for the links and the reminder.

---
For non-MMORPG addicts: ORLY is short for "Oh, really?" Often seen when n00bs state the obvious.

Understanding criticism

One of the things we do is to teach our students to be critical, perform criticism. Particularly journalism students are constantly being told they should perform critical journalism.

But what does this mean? From Princeton.edu:
Noun
S: (n) criticism, unfavorable judgment (disapproval expressed by pointing out faults or shortcomings) "the senator received severe criticism from his opponent"
S: (n) criticism, critique (a serious examination and judgment of something) "constructive criticism is always appreciated"
S: (n) criticism, literary criticism (a written evaluation of a work of literature)

Most of the time "criticism" is taken to mean the first thing: unfavourable judgement, pointing out faults.

I think this is one of the greatest mistakes of modern journalism: The understanding of "critical" to mean you have to find something wrong with whatever you are writing about. To be a critical journalist should be to employ the second meaning of the noun: a serious examination.

Today negativity seems to be the main criteria for a good story (unless it's the obligatory feel-good story). If journalists can't warn us of disaster, corruption, lies, danger to our health or what ever, it's not a story. What does this do to our perception of what is important?

First of all it scrambles the big picture. As long as we are only presented with the errors and the flaws, the public has no way to learn about possible strategies for improvement and growth. Only being introduced to options we should not choose makes choice almost impossible, and the result is that rather than searching for the better choice we settle with the lesser of several evils. The better choice is not interesting, as it doesn't offer the option of unfavourable judgement.

At the same time the warnings are important. We should not go without them. But my other problem with modern journalism is the mechanism which makes all news media report on exactly the same issues. This way you don't only get nothing but warnings, you get warned about the exact same thing everywhere. So the options for the public are diminished by the flawed perception of good competitive strategies: beating others at the same game is more important than looking around for some other and more interesting game.

So, I just delivered an unfavourable judgement. A common piece of criticism. How to make this constructive?

I think critical journalism is extremely important, and there's some really good work done out there. We need good gatekeepers and professionals to sort through the masses of information and ask questions about it on our behalf. It's an expremely important profession. I just wish they would put a bit more emphasis on the serious examination part, and a bit less on the unfavourable judgement part. And that they would stop hunting in such huge packs, victims to the blood fog of the deadline and the scoop.

Boys will be boys...

In one of the large Norwegian newspapers there was recently an article about game addiction, and how Norwegian hospitals are now treating young people who can't deal with their game addiction. They describe gaming and the feeling of mastery that you get from games as a high, comparable with using drugs or alcohol.

Yes, games are designed to give a feeling of control and mastery. We all like it. It's what drives research, athletes, artists. It's what gives us reliable, competent people in all kind of positions. It's not a bad thing.

The problem appears when a game is the only place where a human being can feel like they do anything good, where they have any kind of self-worth and are important to others.

Recently (but long before WoW) the performance of boys has dropped dramatically in schools not only in Norway, but all over the western world. Girls outperform them in many ways, mastering the education system, taking control of their futures. The educational systems have been adjusted to foster this, and it has been successful. Girls get better grades, girls continue for higher education. So far the professors are still men, but if nothing happens the gender gap will soon go the other way, as girls master the system of education, and boys fall by the side.

Education is the most important thing a person does from they are 6 years old until they are 25. If they can't perform well in this field, they are failures. Failures in the eyes of family, friends and the society in general. They know they will never get the job they want, they know a million little decisions are taken away from them.

I think this is very, very wrong.

People are individuals. Not all are cut out for 19 years of school, and they shouldn't be. There are so many other types of tasks, tasks which ask for other skills than the ability to control your physical and mental urges for hours and hours, while paying attention and processing information you are being fed. The society needs to change, to aknowledge the importance of all tasks which keeps the world moving towards a balanced and sustainable future. While we need a big stack of researchers and we need them NOW, that doesn't mean we don't need the other potential in a population. Knowing things are going down the drain isn't enough, if nobody can actually do the work to get us out of that sewer.

As for the boys? They shouldn't need to play WoW to feel appreciated. They shouldn't need to be a brilliant warlock in order to feel they have something they can actually be good at. Being young is hard, and society doesn't make it easier through rewarding only some types of expertize, while severely punishing the pursuit of many different interests. When I was in high-school, these boys skipped class to work the boats (work farms, do other types of unskilled seasonal labour which still existed and was still available to youngsters), trumping their lack of success at school with the thick wad of cash in their pockets. What is there for the 16 year old who needs to feel he does something that matters? Robots stole his job. Then we designed mobs to keep him occupied and off the streets.

The irony is overwhelming.