The Exam Dataset That Grades on Rubrics, Not Vibes

Picture a physics exam that got scanned on someone's phone, tilted a few degrees, half-lit by an overhead fluorescent tube. The student crossed out a wrong derivation, squeezed a correction into the margin, and drew a lopsided free-body diagram to go with it. Now hand that page to an AI grader and ask it a very specific question: does the diagram earn 2 of the 3 points allotted for "correctly identifies force directions," or just 1?

That question has almost nothing to do with reading the handwriting. Reading it is the easy part, or at least the part OCR research has spent two decades getting good at. The hard part is mapping messy human work onto a rubric that some professor wrote months earlier, and deciding, criterion by criterion, how much credit it deserves. A new dataset out of four academic institutions in Bangladesh, built by eight faculty members and described in a paper posted to arXiv on August 23, is aimed squarely at that harder problem. It doesn't report a model's score on anything. It's a data descriptor, the academic equivalent of "here's a carefully built box of stuff, go use it." But the box is more interesting than that sounds.

Grading is a different problem than reading

Most document AI benchmarks train and test on things like receipts, invoices, and forms. Those documents are structured and predictable: a total is a total, a date is a date. Exam answers are neither. A single page might contain a paragraph of prose, a chemical equation, a snippet of pseudocode, a hand-drawn diagram, and a table, sometimes all in the same answer. Add handwriting, which varies wildly from one student to the next, and you get a document type that generic OCR pipelines were never built for.

Then there's the part that's specific to this dataset: Outcome-Based Education, or OBE, a grading philosophy that many engineering and science programs use for accreditation. Instead of a professor reading an answer and writing "8/10" based on gut feel, OBE requires an explicit rubric: a list of criteria (say, "states the correct formula," "applies correct units," "shows a logically ordered derivation"), a description of what counts as strong, adequate, or weak performance on each one, and a maximum mark per criterion. The total score is just the sum of the criterion scores. It's a more auditable, more transparent way to grade, and it's exactly the kind of structure that automated grading research needs if it wants to produce anything more useful than a single number.

That's the gap this paper tries to close. Automated essay and short-answer scoring has existed for years, but most of it predicts a holistic score from text. It doesn't tell a student which specific requirement they missed, and it rarely works from real scanned handwriting with real rubrics attached. This dataset gives researchers both halves of that problem at once, on the same documents.

What's actually in the collection

The numbers are modest, and that's worth sitting with before getting excited. The dataset holds 485 individual exam answer submissions from 415 consenting students across four institutions, covering nine subjects and 12 distinct question templates. The 12 rubrics between them define 47 criteria in total, an average of roughly four criteria per rubric.

Each answer in the collection is a small bundle, not just a PDF. Alongside the scanned answer, the authors attached:

• a randomized identifier, so the file can't be traced back to a specific student
• the subject label and the original question
• a model answer written by the instructor
• the rubric: criterion definitions and performance-level descriptions for each one
• the marks actually awarded per criterion, plus the total

That last item matters more than it might look. A lot of "grading" datasets only have the final score. This one has the paper trail: which criterion earned how many points, so a model trained on it can be checked not just on whether it predicted the right total, but on whether it got there for the right reasons.

The scans themselves are deliberately unglamorous. Some came from CamScanner, some from Adobe Scan, some from conventional flatbed scanners, and the authors kept the resulting variation in lighting, contrast, orientation, compression, and resolution rather than normalizing it away. They also kept the human mess: different handwriting styles, crossed-out work, revised calculations, corrections squeezed into margins. If you've worked with a "clean" academic OCR dataset before, that's the opposite of what you usually get, and it's the more honest choice. Real classrooms don't hand you a flatbed-scanned, perfectly lit answer sheet. They hand you whatever a stressed student photographed with whatever phone they had.

The unglamorous work of making a dataset trustworthy

Data articles like this one live or die on the boring parts, and the authors spent real effort on them. Before anything went into the final collection, they consolidated submissions from four separate institutions with different filing conventions, standardized labels and text formatting, validated scores against rubric maximums, and randomized both the student identifiers and the PDF filenames so a file's name can't leak who submitted it. They also ran a JSON-to-PDF integrity check to make sure the metadata for each answer actually points at the correct scanned file, which is the kind of bug that's invisible until someone trains a model on a dataset where question 14's metadata is silently attached to answer 9's scan.

Then they audited the finished collection: 485 unique identifiers, 485 unique PDF filenames, agreement between every total mark and the sum of its criterion marks, and every score falling within its rubric's maximum. None of this is exciting to read about, but it's the difference between a dataset a researcher can trust and one where the first six months of any project using it go to discovering its own data-entry errors. If a total mark didn't match the sum of its parts, or two students shared a supposedly unique identifier, that would undermine anything trained on it before a single experiment ran. Getting that basic hygiene right and stating it explicitly is worth more than another few hundred submissions would have been.

Why it matters for anyone building or choosing OCR

If you're building or evaluating document AI, the lesson from this paper is that "OCR that grades exams" is a fundamentally different product than "OCR that reads exams." A system can transcribe a handwritten derivation perfectly and still be useless for grading if it can't connect that transcription to a specific criterion in a rubric and decide how well it satisfies it. That's a reasoning and matching problem layered on top of a perception problem, and most OCR benchmarks (receipts, invoices, forms, ID documents) simply don't test for it.

Concretely, this means a few things for anyone evaluating a vendor or building an in-house pipeline for grading, compliance checking, or any task that scores a document against explicit criteria rather than just extracting fields from it:

• Test on the messiest input you'll actually see, not the cleanest. This dataset's deliberate mix of scanning apps, lighting, and handwriting styles is a better proxy for real-world documents than a curated, evenly-lit benchmark.
• Ask whether a system can explain a score, not just produce one. A rubric-aware evaluator that returns "3/5 on criterion 2 because the derivation skips the boundary condition" is worth more than one that returns "72/100."
• Watch for silent misalignment between structured metadata and the underlying document, the kind of bug the JSON-to-PDF check in this paper was built to catch. Any pipeline that pairs extracted content with external records (a rubric, an invoice line, a claim form) needs the same kind of integrity check, and most teams don't build one until something has already gone wrong.

It's also worth being honest that "rubric-aware automated evaluation," one of the paper's stated use cases, is still largely aspirational. This dataset makes it possible to work on, but nobody has yet published results showing a model that reliably grades a hand-drawn diagram against a rubric criterion. That's the actual frontier, and it's further out than a lot of OCR marketing implies.

What this dataset can't tell you yet

A data descriptor paper is a promise, not a result, and it's worth being clear-eyed about the gaps.

Size is the obvious one. 485 answers across 12 rubrics is enough to fine-tune or evaluate a model, not enough to train one from scratch. Nine subjects sounds broad, but with only 12 question templates in total, the coverage of question styles within each subject is likely thin. A model that does well here might just be overfitting to a small number of rubric shapes.

Access is a real limitation too. The data isn't published openly; it's available "upon reasonable request" to the corresponding author, for research use only. That's a defensible choice for student privacy, especially given how much handwritten, personally identifiable work is involved, but it means the dataset can't become a shared public benchmark the way something like FUNSD or DocVQA did. Every paper that uses it will be working from a slightly different point of access, and reproducing results across labs gets harder.

There's also no mention of inter-rater agreement among the eight faculty contributors. If two instructors would have scored the same handwritten derivation differently on the same rubric, that disagreement becomes invisible noise baked into the "ground truth" marks. The paper reports agreement between total marks and criterion sums, which is an important internal-consistency check, but it isn't the same as confirming the criterion marks themselves are consistent across graders.

And this is purely a data article: no baseline model, no accuracy numbers, no sense of how hard the task actually turns out to be once someone tries it. That's fine for what the paper claims to be, but readers hoping for evidence that rubric-aware grading works should treat this as raw material, not proof of concept.

None of that erases the value here. Getting 415 students to consent, coordinating eight faculty across four institutions, and building a dataset that ties real messy scans to real rubrics and real awarded marks is genuinely hard logistical work, and it's the kind of unglamorous groundwork that has to exist before anyone can build a model worth trusting to grade a stranger's exam. The interesting research still has to happen. This is just the first honest attempt at giving it something real to chew on.