April 27, 2009
Incentives for filling out teaching evaluations?
The University of Michigan switched to an online course evaluation system this year (Fall 2008 was the first full run).
One of the primary concerns during the study that went into the decision, and during the implementation, is what would happen to response rates. Selection bias for respondents is a concern for any survey, and there are obvious (and documented) reasons to expect that response is correlated with student satisfaction with a course and a teacher, which definitely would create a bias, the more so the lower is the response rate. With the traditional fill-in-a-form system, if administered during a class late in the semester when attendance is high (perhaps because students are getting concerned about what will be on the exam!), through convenience and peer pressure the response rate can be fairly high (as I recall, the norm at UM was above 70%). With an online system, evals are filled out at the student's convenience. This might catch students who would have missed the evaluation class day, but might lose many others.
The evidence UM collected from other campuses (and from two schools at UM that had already implemented their own online system), response rates do tend to be a bit lower, though not much.
My ICD colleague, Yan Chen, and I have chatted a bit about providing incentives to students to complete the online evaluations. One idea that she'd heard seemed effective in another department is to award a nominal amount of "extra credit" to students who submit an evaluation (in some systems, even though the evaluation content is anonymized, it is possible to track whether a student submitted something).
Here is a blog entry on the use of course evaluation incentives from the Center for Teaching, Learning and Technology at Washington State University. Their impression is that the incentives (including extra credit and randomized cash awards) have had little impact on participation (at least at the level at which the incentives were offered). Their conclusions are mostly based on anecdotes, but they provide a link to a discussion of a comparative study that was done in their College of Engineering which, it appears, was being submitted for publication.
April 25, 2009
Wits & Wagers
The other day I created a prediction market for School of Information students, faculty and staff to see what kind of forecast we could generate for the coming year's enrollment.
One of our staff told me that she had played a board game that was quite similar, and indeed it is: Wits & Wagers is more or less a prediction market board game. It doesn't aggregate the side bets into a market prediction, but it does allow players to take a position in various claims (actually, they are trivia questions, not predictions).
(Thanks to Kelly Kowatch.)
April 24, 2009
Beautiful, but Hollywoodized mind
All economists have followed the story of John Nash as he recovered from 35+ years of paranoid schizophrenia in the mid 1990s, and was awarded the Nobel Prize for his Ph.D. dissertation in which he invented the core concepts of non-cooperative game theory. After all, every last one of us learned Nash equilibrium in our first year of graduate school (or earlier), and an awful lot of us use the concept daily in our research and teaching.
And so, like many others, I read Sylvia Nasar's biography A Beautiful Mind as soon as it came out. Then rushed to see the Russell Crowe movie by the same name.
Oh no! In the key scene of the movie (well, to us economists), when Nash excitedly explains his equilibrium concept to his buddies while sitting in a bar...the screenwriters got it wrong. The "Nash equilibrium" Crowe describes in fact is not an equilibrium. Not even close: every one of the three players in the game described has an incentive to deviate and go for the blonde. Grave disappointment. How could Hollywood spend many millions on a biography of a game theorist, and get his most famous, and let's face it, rather straightforward concept wrong?
Over the years, when I would teach introductory game theory to students, I usually mentioned the popular movie to motivate them a bit, and quickly sketched the highlights of Nash's strange life. Then I would say "but the Nash equilibrium in the bar scene is wrong" and let them figure it out as an exercise. (Easy in recent years, since they can go find the clip on YouTube.)
Of course, most anyone who knows a bit of game theory and saw the movie surely spotted the mistake: it's a bonehead error. But just today I discovered that his mistake has really made it in pop culture: it's the punch line of an xkcd comic (http://xkcd.com/182/)!
(Thanks to Jon Kleinberg via Chuck Severance for leading me to this gem.)
April 12, 2009
Lady Chatterly banned again
I am here engaging in what I usually describe as the "manipulation" subspecies of "pollution". I am doign this to participate in the Amazon Rank project to Google-bomb Amazon.
Apparently on 12 April 2009, Amazon removed books that it deemed had "adult content" from its sales rankings. Because of the way their systems works, this now means that the books are not found in standard searches. Example: use the main search box to query "Lady Chatterly's Lover". I just did this and did not get a hit on the book by D. H. Lawrence until #8, and this is an edition available through Amazon's used bookseller partners, not a new copy available from Amazon itself. Search on D. H. Lawrence's name, and his most popular book (LCL) does not come up in the first 16 entries (an audio CD edition pops up at #15).
So, an angry blogger started a Google-bombing campaign to make a search on Amazon Rank turn up a a critical definition (follow the link). As a political matter, I support this particular manipulation.