Grade Inflation

(Okay, I'm really catching up on stuff tonight.)

Spent the last three years or so chairing a Faculty Task Force on Grading, focused mainly on the issues of Grade Inflation. I found it hard to get motivated since I basically don't believe that inflation is a significant issue for our faculty, since the vast majority of our instructors use criterion based grading where the whole issue largely does not apply. So, most of the work for the first two years was done by a colleague who was herself greatly concerned with the issue and so took care of the literature review. The Dean wanted the final report this year, so I did a preliminary presentation to Faculty April 16, then -- based on input from a survey of faculty that meeting -- did a final presentation at our Faculty's annual retreat last Thursday. The results of the committee's deliberations can be found at the Grade Inflation Website I threw together in about 10 hours as a briefing for faculty who missed the initial meeting. I think I managed to create a balanced report that incorporated the views of both myself (and the rest of the committee who basically shared my perspective) and my colleague, even though they were essentially opposite positions. Going through the website one direction gives you my argument; going through the other gives you hers. (My route assumes readers from the digital generation clicking through window-sized 'sound bites'; her route gives one blocks of text you have to scroll through to get the sustained argument. Although I wish to avoid stereotyping here, I think matching the traditionalist message with the traditionalist medium and vice versa was a nice touch, if I do say so myself.)

I'm happy to talk about these issues with anyone who is interested...if there is sufficient interest in academic circles I will probably spend sometime expanding the website to be more complete. The first planned addition would be the survey results that reveal that while there are significant number of faculty concerned about grade inflation, none of the think the problem is with their grading! Quite amusing, really.

====
Comment from Dr. Wasserman:

I take exception to your remark that the concept of grade inflation does not apply to criterion-based grading.  I have yet to see, and cannot imagine, a real-life application of criterion-referenced scoring that can be applied by everyone with absolute consistency over time to different assignments or other work.  In my years of working with standard setters, the description of the characteristics of borderline students were pretty much the same, whether the standard setters were describing students in Grade 3, 6, 9, or 12.  It is only when descriptors are matched with exemplars that the concept of a borderline student takes on any operational meaning, and since the exemplars change with the particular task, there is plenty of room for drift.  If the standards drift down, one could call it grade inflation.
 

At any rate, your statement that everyone sees grade inflation as something other people do, while he or she is perfectly reliable, is right on the money.

===
Robert Again:

(I don't know what has happened to my comment function...it seems now to have completely disappeared...no time at the moment to fix it, perhaps after I get back from Toronto conference next week. In the mean time email runte at uleth.ca with your comments, if any.)

Great Examples of Bad Test Questions

As an evaluation instructor, alumni often send me examples of atrocious examinations they encounter in their subsequent courses. (Most of our alumni go on to a second undergraduate degree or graduate work.) Graduates of my exam-making course generally do very well on other people's exams, because once one knows how to write good examination questions oneself, it is easy to spot the errors in poorly written questions, and so be able to work out the answers. (See http://www.edu.uleth.ca/courses/ed3604/take/mc/how.html#Tricks for examples of what I mean.) On the other hand, my graduates also become hypercritical of sloppy exam-writing, and are often offended by the poor evaluation technique of otherwise excellent instructors. And thus my collection of really bad examinations continues to grow as alumni mail me the poor examples they encounter in other programs.

This week a former student sent me a wonderfully awful test from which I have drawn the following examples. (I won't, for obvious reasons, identify the campus that that student is now on, but suffice to say, this is from an experienced instructor at a legitimate North American university of some little repute, and not in any way an exceptional or unusual case.)

Mummification is first mentioned in 2nd Dynasty texts.
A) True
B) False
C) Maybe

Okay, ignore for the moment the embarrassment of a university instructor using true and false questions, how can one have a "maybe" category in a true/false items?! The whole point of true/false is that they address absolutes. The maybe category is invalid because a case can always be made for 'maybe' -- in this case, that there may well be other texts that have yet to be discovered. Since some questions on this test are true/false and others true/false/maybes, I would suspect "maybe" as the correct alternative anyway, since the instructor probably used it in those cases where there is some existing debate in the field (say an ambiguous reference in some earlier text that may or may not refer to mumification) but it doesn't really matter -- given a "maybe" in a true false question, I can always justify it as the correct answer. It will always win any formal grade appeal.

Old Kingdom Kings did no trade with Asiatics.
a) Not True
b) Not False

The classic double negative question! I have been looking for one of these for years! All the test construction textbooks warn against the use of a double negative (negative in both stem and alternatives) but I have never actually seen a real example of one of these before. Evaluation nstructors have always had to make up our own examples, and students always say, "Oh, nobody would really do that, would they?" and now at last I have a real example.

"Despite his reputation as a tomb robber, Belzoni was nevertheless a fine archaeologist".
A) True
B) False

The archetypal "opinion" question, the ultimate taboo in true/false item writing. Again, I have been looking for an example like this for quite a while. True/false questions can only be used for testing absolutes, not opinions, since one can always make the case for the other side (however tenuously) and we do not score people on their opinions in a democracy. In a formal grade appeal, the student will always win.

The rest of the test is of similarly disappointing quality. Every campus has some kind of Teaching Development Center (or at least a Teaching Development Committee, if the campus is too small to afford dedicated staff) that sponsors 'how to' workshops on instruction and assessment techniques, but of course those that need the workshops are never the ones who attend.

The sad thing is that the former student who sent me this test had been absolutely raving to me about what a wonderful professor this was and what a great course and how much the student was enjoying the class, prior to the test. Afterwards, the student was so disappointed with being robbed of the opportunity to demonstrate the deep learning achieved in that class, that their enthusiasm was considerably eroded. The student still considers that professor an all time favorite, but then this is a student motivated by a thirst for knowledge rather than grades, and so perhaps more willing then most to forgive such tragic flaws.

(Not, I hasten to add, that any of us are flawless. My exams may be good, but I could no doubt learn a thing or two about lecturing and motivating students from this instructor, judging by my graduate's enthusiasm for an otherwise arcane subject. But wouldn't it be great if professors could learn from each other, and build on each other's strengths, rather than merely perpetuate the poor teaching and assessment techniques they themselves endured as undergraduates?)

=================================================================
A news bite from John Herbert (current Editor of Under the Ozone Hole):

Here's a little Columbia disaster news blip that came and went under the radar....

I just happened to be watching NASA Administrator Sean O'Keefe testifying in front of Congress last week. An American Congressman (whose name I didn't catch but it was no one you've heard of) began questioning O'Keefe by laying out a string of events.

He said (and I paraphrase) that the contract for the glue (or "urea") that held the heat protection tiles in place was originally awarded to a Canadian company in "Fort Sasketchwhichwan, Alberta, Canada."

No company in the United States could meet the specs that NASA required for this urea -- in fact, only one manufacturer in the world could, and it was located in Fort Saskatchewan. Up until five years, this was the only source of shuttle glue tile.
What changed five years ago? This company was bought out by a larger Canadian company that does business in Cuba.

American government agencies cannot deal with companies that do business in Cuba, so NASA could no longer buy tile glue from the only source in the world that could meet its specs.

He then asked O'Keefe to comment and O'Keefe offered none, saying that he was unfamiliar with these events (O'Keefe has only been with NASA a year.)

So it may turn out that the American policy of sanctions against foreign companies operating in Cuba might have played a large role in the death of seven American astronauts.

Bet you won't see this on CNN.
================================end quote============================