赤ひげ

The Dark Art of Racecraft

Some good stuff over there, but the best is probably the Philip Uri Treisman talk.

From Amos Barshad’s profile of Gunplay.

Wonderful.

Matt Yglesias compares high-stakes test cheating in education to PEDs in sports, with the conclusion that cheating is not telling us does not prove anything about the system, but instead about the cheaters.

But the cheating scandals aren’t about cheating by teaching to the test¹ or teaching test-taking skills at the expense of teaching for understanding. The cheating scandals in Atlanta (and Chicago, DC, etc.) have all been about teachers (and/or other staff) literally erasing incorrect answers and replacing them with correct answers.

So let’s get this sports metaphor straight.

• Let Type I cheating be breaking rules on game day.

Easy to catch, with clearly defined penalties. In sports, this is stuff like fouls, offsides, and too many players on the field. In education, this is plagiarism and copying.

• Let Type II cheating be preparatory cheating.

It is harder to catch. It can happen at any time before the game, and it often looks to an observer like good preparation. This is the PED-driven miraculous rehabilitation from injury, or video taping the opponent’s signals. This can be student-centered, like faking a sick day to learn an exam’s contents. It can also be more subtle: cramming, for instance is analogous to blood doping. More subtle: “test-taking skills,” beyond the structure of questions and answers are like scouting the referees instead of the other team, to instruct players what they can get away with.

• Let Type III cheating be breaking administrative rules or otherwise “undermining the integrity of the system or game.”

Type III cheating is extremely difficult to catch without a whistle-blower. This is “contract tampering,” setting bounties to hurt opponents, or fixing matches. In education, the scandals in Atlanta, Chicago, and DC are Type III scandals: teachers changing answers after the test. But there have been other reports of Type III cheating. Some teachers deliberately create noisy environments and promote an unserious attitude for high stakes baseline assessments at the beginning of a term to inflate their “value added” when the high stakes summative assessment is taken at the end. Some administrators encourage specific students to be absent on test day. Many charter schools strongly urge specific students to go back into the public school system to inflate graduation rates and assessment scores. Some Departments of Education (Rhode Island, Washington) are using assessments outside of their designed scope. Edit: another form of Type III cheating is focusing on “bubble students,” those expected to be very close to proficiency at the expense of differentiating instruction for those expected to fail or expected to perform well.

Cheating takes something that should inform us and turns it into a disinformation tool. Type I and Type II cheating increase our uncertainty about a single specific student, or a single specific classroom and its teacher. Type III cheating increases our uncertainty about every student’s achievement. After Type III cheating we know less about how much students knew, learned, or know, less about teacher, school, and district efficacy, and less about state policy efficacy. And the root of this antiknowledge is the incentive structure. And incentive structures are policy. They can be changed.

We caught these folks in Atlanta, but don’t kid yourself. This is not about a few bad apples. What would a coach’s win/loss record be if he or she could edit the final score after each game? In single-player simulation video games, we know that players use the ability to rewrite final scores frequently, by abusing the save feature to replay losses². I would be amazed if higher (in-game) stakes were not correlated with increased cheating in simulations. Sounds like a fun research project. Many players likely play these games without cheating in this way, because the stakes are objectively low.

Now imagine that you have the means to rewrite the final score of something with real life importance after the fact, and that winning yields a promotion or a bonus and losing means getting reprimanded or fired. How many times would you “bowl a wide in The Ashes³“ before you started cheating?

This is why I take the Steven Levitt view on the cheating scandals⁴, that a policy of teacher-supervised testing with high stakes for students, teachers, and schools establishes a means/incentives structure that makes some form of cheating inevitable. Edit: to clarify, not inevitable to every individual, but to the system as a whole.

There are solutions to this incentive problem, but that’s another post.

[1] Teaching to the test is abhorrent if the test is bad, so that swipe is really about poor assessment design. For the remainder of this piece, imagine our high stakes assessments are objectively perfect. I’ll grind that ax in some other post.

[2] A recent article on Rock Paper Shotgun KillScreenDaily discusses the deficiencies of the latest incarnation of Sim City with exactly the complaint that because there is only a single canonical save, earlier world-states cannot be resurrected.

[3] The Full Number of Overs that are Scheduled to be Bowled That Day. That Mitchell and Webb Look [sketch]. Chose this because I know nothing about cricket, but I think Mitchell and Webb are funny, anyway.

[4] Levitt and Dubner, 2005. Freakonomics. Chapter 1: What do school teachers and sumo wrestlers have in common.