Matt Shumer: Keep Evidence for Each AI Revision
How can I tell whether an AI revision improved the actual work, and keep a record that explains what changed?
Choose, practice, reflectA practical learning exercise
The Lesson
An assistant says it has made a draft “clearer and more useful.” You open the new version and notice that it removed an important detail. The revision report sounds good; the actual result needs closer inspection.
In How to Run a Gauntlet Loop, Matt Shumer argues for a concrete quality reference and a critic that inspects the actual output rather than the builder’s summary. He also describes keeping visible progress records. His own site links his writing and software experiments.
You can practice one modest lesson from that discussion without setting up a multi-agent system: keep the artifact and the evidence for each revision. A claim of improvement should be connected to an observable change.
For writing, the artifact is the actual draft. For an interface, it might be the page and what happens when a reader uses it. A revision note is helpful context, but it cannot substitute for checking the work itself.
Keep your original brief beside both versions. Otherwise the standard can drift: the first version might be judged on factual completeness while the second wins because it sounds more polished. If you change the standard, record why. That makes the comparison understandable later.
An evidence record does not have to be complicated. Save the version, name the requested change, inspect the relevant detail and decide what to do next. A result can improve in one respect and regress in another. Both observations belong in the record.
Reflection
Think about a recent AI-assisted draft or prototype.
- Did you save the version before asking for changes?
- What did “better” mean in that task?
- Did you inspect the changed artifact or accept its summary?
- Which detail could have been lost during revision?
- What evidence would let another person understand your decision?
Choose a task you can judge directly. Start with a small written result if checking software behavior is unfamiliar.
Practice
Original SelfGrowthVideos exercise: Make a two-version revision record. This bounded worksheet is our exercise, not Matt Shumer’s full Gauntlet Loop method or an endorsement.
Use this fictional brief: write a short notice for a free spreadsheet workshop. It takes place Saturday at 10 a.m. at a fictional library. Beginners should bring a laptop if they have one; people without a laptop can observe the demonstration. Ask interested readers to contact the organizer for availability.
Write version A yourself or ask an assistant for it. Save the exact draft. Check it against the brief, then request one change: make the invitation easier for a beginner to understand while preserving all supplied facts.
Save version B separately. Read both drafts yourself and create this record:
| Record item | What to save |
|---|---|
| Requested change | The one improvement you asked for |
| Artifact | Both actual drafts, with version labels |
| Observations | Specific wording or details you inspected |
| Regression | Any useful information lost or distorted |
| Decision | Keep A, keep B, combine carefully or revise again |
For example, “B uses shorter sentences” is observable. “B is better” needs explanation. If B implies that a laptop is required, record that it changed the meaning of the brief even if the prose feels smoother.
Choose a stopping point for this small exercise: inspect the two versions and make one decision. More revisions are not automatically more useful. If you cannot judge a claimed improvement, note that limitation and choose a task or reviewer that can help.
Review
Can a willing peer reconstruct your decision from the brief, drafts and observations? Ask them which evidence is missing, not simply which version they like.
Write one final sentence: “I kept this version because ___, and I still need to check ___.” The second part keeps unfinished questions visible. No version should be called verified just because an assistant reported success.
At your next check-in, reuse the record for a different bounded task. Notice which checks transfer and which need to change. The aim is to become better at explaining your judgment, not to prove that a particular tool always wins.
Go Deeper
Explore Matt Shumer’s writing and interview profile and AI Content & Marketing. Continue with Matthew Berman’s same-task tool comparison or Claire Vo’s saved-result interface check.
Use the worksheet with Side Hustles: Writing, Editing & Translation to demonstrate a review process. A clear revision record shows what you checked; it does not guarantee professional competence or paid work.