Notes
Short pieces from the design end and the engineering end, with the numbers attached. Links open beside each other.
I ran over 12 language models and a set of rule-based systems on interfaces from the Hall of Shame repository. Human raters scored 147 of the models’ responses from 0 to 10, and each model rated its own answers too. The models won against the rules. Then I compared the two ratings.
| Model | Human rating | Model's own rating | Gap |
|---|---|---|---|
| GPT-4.1 | 0.783 | 0.923 | +0.140 |
| GPT-4o | 0.741 | 0.893 | +0.152 |
| o1 | 0.708 | 0.875 | +0.167 |
| Claude 3.7 Sonnet | 0.658 | 0.860 | +0.202 |
Every model in the study rated itself higher than the human raters did, and the gap grew as the answers got worse. Part of the reason is what the plugin shows the model: language models read the words and miss the picture.
