Content audit: checking every question before learners see it

Why audit

In the site’s feedback form, 1 in 3 pieces of feedback is about the content: that is what the first project, UserVoice, found.

We put the effort into the content itself, question by question. I led this work with a learning designer. I designed the audit grid and the tooling that applies it. The learning designer validated the grid. She also set the writing rules for the content (line breaks, editorial rules) and ran acceptance testing on the samples.

The scope covers six courses that prepare learners for language certifications. We started with the oldest one.

What the grid checks

The grid has six criteria. Two concern the answer: the expected answer is the right one, and no other answer is as defensible. A third checks that the question includes an explanation, and that it is not circular. The last three cover faithfulness to the source, language and internal consistency.

Each question then gets one of four verdicts: blocking, major, minor, or nothing to fix. A question is blocking when learners cannot answer it. That happens when the expected answer is wrong, or when two answers can be defended equally well.

AI agents apply the grid, question by question. They are Claude Code subagents, running on a subscription with no calls to a paid API. The model does not judge alone. Automated checks also run over the whole corpus, with verifiable rules that do not depend on any model. One of them, for example, checks that the explanation refers to the same audio as the question. A report then gives each question’s status and the issues raised against it.

CourseIts questions, each with its answer and its explanation.In parallelSupervised AI agentsThey apply the six-criterion grid, question by question.Automated checksIs the expected answer the right one? Is there an explanation? Is it circular?ReportOne verdict per question: blocking, major, minor or nothing to fix.Human reviewJean and the learning designer run acceptance testing on samples.Fix batchesReviewed, never written straight to the database.Going liveTo staging, then to production.

The audit chain, step by step:

1. The course: its questions, each with its answer and its explanation.
2. Two checks in parallel. Supervised AI agents apply the six-criterion grid, question by question. Automated checks verify that the expected answer is the right one, that there is an explanation, and that it is not circular.
3. The report: one verdict per question, blocking, major, minor or nothing to fix.
4. Human review: Jean and the learning designer run acceptance testing on samples.
5. Fix batches, reviewed, never written straight to the database.
6. Going live: to staging, then to production.

What we found

In the first course audited, 1 question in 20 was rated blocking. Learners failed these questions through no fault of their own. Each fix gives them back the chance to answer correctly.

This course is the oldest one. Its result does not predict the quality of the rest of the content on the site.

Fixing without breaking

The tool never writes directly to the database. It produces batches of fixes, and a person reviews each one before staging. If an agent gets an answer wrong, the worst case is a batch rejected at review.

All the blocking questions in the first course have been fixed. The fixes went into production the week of September 28, 2026.

What I took from it

This project gave me 25 rules for running a project with AI. They are there to keep control over what an agent produces. I measured them on the project itself, in tokens used over four weeks and in decision points logged in a register.

They change how the work gets done. Every workstream is costed before it starts, then again once it ends. The gap shows where the estimate was wrong. A rule that can be checked becomes a test. From then on it is enforced automatically, without relying on anyone’s memory.

The rule “the reviewer is not the author” sums up the audit. What one agent produces, another agent reviews, and a person decides. This rule applies to content fixes as much as to the code I write with AI.

Some rules were expensive before they were written down. Without version control from the first file, two weeks of work went untracked.