Economics of Cheat Detection (In Defence of Niemann)

In this article I will outline why I think the chess.com report does not show evidence of Hans Niemann’s cheating. First I will explain how economists tackle cheat detection. Secondly I will discuss the Chess.com Report. Third I will give an example of my favourite economics cheat detection paper.

ECONOMICS OF CHEATING

Economics tries to understand human behaviour through causal inference methods. The human behaviour we are discussing today is extraordinary human game behaviour (i.e being super good at chess). This behaviour can be explained by two factors: skill and cheating. The formula looks a bit like this:

x*cheating+y*skill=great achievement.

We can Identify a cheater by identifying variables that assign a non-zero value to x. To do so we need to find variables that cannot speak to y. If we think we found such a variable we always need to ask ourselves about the counterfactual. Can this variable also be ascribed to y?

Only if we can confidently ascribe Variable A to only x we can be confident we detected a cheater.

CHESS.COM REPORT

The Chess.com report identifies two such Variables: Strength Score and ELO over time. Both of these variables are not sufficient to identify cheating and only cheating, with the exclusion of explaining skill.

Strength Score.

Chess.com show a table of GMs with high strength scores that have admitted to cheating in the past. This includes Nieman. This table is misleading. Strength score is not a valid causal determinant of cheating. There are players with higher strength scores which have not admitted to cheating. To support their strength score variable they have to show that this score is highly correlated with admitting to cheating. The report does not mention false positives, where strength skores indicate cheating but cheating was not admitted. I suspect that this variable is not indicative at all of cheating.

ELO over time.

Chess.com claims that a rapid rise in ELO (chess rating) suggests cheating. Again, there are a few counterfactual explanations which make this variable an explanation for skill not cheating:

1. Started from the bottom now we here: In order to determine the rate of change the two comparison points are important. It is notable that Hans was the worst 11 year old chess player compared to other Grand Masters. His ELO was 1950 (which I could beat on a good day), while Praggnanadhaa was a little Wunderkind at 2450 ELO when 11 y o. Does a lower skill at 11 y o indicate cheating? To ask about the counterfactual: does being less skilled at 11 mean that one cannot become a grandmaster without cheating? I think the answer is no.

2. Similar to the Strength Score argument I want to see rate of change in ELO as a causal variable. I want to see others, maybe with lower rankings but similar rate of change admitting disproportionally to cheating. Also this is not shown in the report. What is the false positive rate here?

In conclusion, Chess.com has provided no variable which only speaks to cheating and not to skill. Thus there is no evidence.

CHEATING TEACHERS

I want to give an example where I think cheating allegations were discussed with counterfactuals in mind. “Catching Cheating Teachers: The Results of an Unusual Experiment in Implementing Theory” (Jacob and Levitt, 2003) try to catch cheating teachers. The setting is as such: school students in the US have to do yearly standardised tests. In 2002 California decided to give an incentive to teachers, that if their students perform well teachers can get a reward of up to 25K USD. This should incentivize teachers to work harder, but it can also incentivize them to cheat. Teachers can cheat by changing students' answers after they have written the test. The task now is to identify which teachers were genuinely good teachers, and which doctored the results of their students after sitting the exam.

The authors identified three variables:

1. Exceptionally high rise in test scores in 2002

A steep rise in score reminds us of the Chess.com argument. It could effect skill or cheating.

2. Exceptionally low fall in test scores in 2003

Also the fall of test scores of a class after 2002 might just mean that the teacher lost motivation when no incentive was given- thus no clear indication of cheating

3.String Answers

Those are patterns of answering which are hard to explain by the teachers skill and are thus likely to explain cheating.

a. Block of answers

The same answers given for the same block of questions (i.e. question 7-12)

b. High correlation of correct answers

Here students have the same answers for a number of unrelated answers.

c. High variation of correlation

Here some students have highly correlated correct answers. This means only some students are way better than others (or the teacher just corrected some students answers, not all)

d. Easy wrong, hard right

This measure shows a high % of hard questions answered correctly while some easier answers answer incorrectly.

They identified the most suspicious schools of these three factors and retested 100 classes. They saw a drop in almost all classes with the highest drop of all variables present (21.5 points). The highest single factors was getting hard question right and easy ones wrong, while the high variation of correlation was not determinant of score change. They did not definitively find evidence of cheating, they just found dramatic drops of points when testing under different conditions (so cheating).

Conclusive remarks:

For cheating detection we have to find variables that speak to only cheating not skill. Once we identify such variables we want to put them to the test against controls and see whether we see a decline in performance under controlled testing.

In Niemann’s case we find neither. There are no variables (in the report) that speak only to cheating. Testing him under controlled environments (OTB) sees no decline in performance (beating Magnus). I thus conclude that there is no evidence to support cheating claims regarding dear boi Mocke.