Skip to main content

Evidence

How close are Markwise's scores to a human's?

We graded 30 essays from a public research corpus and compared every score with the human raters'. Each essay has 7 criteria, so each run is 210 scores. The numbers below come straight from the committed results files.

Try it on a paperHow to rerun it

This is a small sample. With 30 essays, one essay more or less moves a rate by about 3 points. Read these numbers as a rough check, not a measured accuracy.

Claude (claude-haiku-5-5)

Not run yet

No live run has been recorded, so there are no AI numbers to show. A live run costs money, so it is started by hand: 30 API calls to claude-haiku-5-5, at most about $0.13. Its results will replace this card.

Until then, Markwise makes no accuracy claim for its AI grading.

Demo grader

Not AI: a keyword heuristic

The engine Markwise runs without an API key. Run October 27, 2026 on 30 essays, 210 of 210 scores given. No API calls.

Within 1 point
82%
Always 3: 88%
Exact match
18%
Always 3: 22%
Average error
0.73pts
Always 3: 0.68

Its average error is higher than always answering 3: on this sample it adds no information.

By criterion: demo grader

Each criterion is scored 1 to 5 in half points. The last column is a baseline that always answers 3, the most common human score. A grader with a higher error than it isn't adding information.

Swipe sideways to see every column.

Agreement with human scores by criterion, demo grader, 30 essays
CriterionWithin 1ExactAverage errorAlways 3: error
Overall proficiency83%23%0.630.77
Cohesion90%20%0.700.62
Syntax77%23%0.700.73
Vocabulary80%13%0.780.63
Phraseology87%10%0.730.68
Grammar77%23%0.870.70
Conventions83%13%0.720.62
All criteria82%18%0.730.68

Largest disagreements: demo grader

Essay ids are the corpus's own, so anyone can look an essay up in the sample file (opens in a new tab)⁠.

Swipe sideways to see every column.

The largest differences from the human score, demo grader
EssayCriterionHumanDemo graderDifference
57DFE525E50AVocabulary52−3
57DFE525E50AGrammar4.52−2.5
57DFE525E50ASyntax4.52−2.5
4EB4E8611272Conventions24+2
4EB4E8611272Grammar24+2
57DFE525E50AConventions42−2
57DFE525E50APhraseology4.52.5−2
9F9C42BD3B7AConventions4.52.5−2

What this does and doesn't show

Language, not argument
ELLIPSE scores language proficiency: grammar, vocabulary, cohesion and the like. It says how well a grader judges those, not how it would grade a teacher's own rubric on content or argument.
Typed essays only
Handwriting, answer-key photos and test questions are not part of this check.
Not the safety checks
How often the injection flag fires, or how often a quote is wrong or weak, is not measured here. Those checks are covered by tests, not by a rate.
Confidence is not checked
The grader's self-reported confidence is not compared with its errors. It stays uncalibrated and only helps order To review.

The data and how to rerun it

Source. The ELLIPSE corpus (opens in a new tab)⁠ of essays by English language learners in grades 8 to 12. The corpus authors anonymized it, and each essay is scored by two trained raters on six analytic qualities plus an overall score. Crossley et al. (2023), International Journal of Learner Corpus Research 9(2).

License. CC BY-NC-SA 4.0. The sample is used for non-commercial evaluation and is shared under the same license.

Sample. Five essays from each of six bands of the human overall score, picked by a fixed hash so anyone can rebuild it. Only the id, prompt, text and scores are kept. No demographics.

Free run. npm run eval grades the sample with the demo grader, offline.

Live run. Claude grades each essay with the app's own prompt and validation. With the default model, claude-haiku-5-5, that is 30 API calls and at most about $0.13. It is started by hand only, after a dry run prints the model, the call count and the worst-case cost, and it stops at hard caps.

Details: eval/README.md (opens in a new tab)⁠, the sample's README (opens in a new tab)⁠, and the results files (opens in a new tab)⁠.