← all field notes
by Turni Saha · 2026-07-09ai-agents · content · evals1 min read

the reviewer that approved every draft

the marketing platform’s first model-based review missed writing problems we could already name. i split the review into explicit checks and a more focused model assessment.

The marketing platform uses 16 writing configurations across four websites. Each site has its own audience and voice. I wanted a review step that would catch recurring writing problems before an editor had to correct them again.

The first version asked a model whether the draft matched the intended voice. It approved everything, including drafts with the formulaic openings and stock phrases we wanted to remove.

The writer and reviewer used the same model family. That may have contributed to the overlap in their preferences, but changing models alone would not have given us a useful definition of acceptable writing. The review needed criteria we could inspect and test.

separating the checks

I split the review into two parts.

Explicit checks handle patterns we can identify directly: banned phrases, repeated openings, and particular sentence structures. These checks can explain exactly what failed.

Model review handles questions that need interpretation. Instead of asking for a general quality score, the prompt asks the reviewer to find specific ways the draft misses the brief or writing rules.

When a draft fails, it goes back with the findings. Passing the checks gets it to an editor; it does not make it finished.

checking the reviewer

The reviewer needs examples with known problems. Otherwise, a change that makes it approve more drafts can look like an improvement even when it has become less useful.

I keep regression checks around the writing rules and use rejected drafts to make the criteria more specific. A request to be “confident but approachable” leaves too much room for interpretation. A real passage an editor rejected, with the reason beside it, gives us something to work with.

This does not automate editorial judgment. It reduces the number of times an editor has to explain the same correction.

based on build notes, edited with AI assistance.