How to fact-check AI content: the fixes our sources check asked for
To fact-check AI content, test every sourced sentence against the page it rests on, in a separate pass from the writing, and write each fix down. That is how our own desk does it. The record is specific: across the 69 articles we finished between August and November 2026, all drafted by our own writing agent, our sources agent asked for 470 fixes on its first read of the finished draft. That is a median of 7 per article, with a range from 0 to 16.
Those are fixes asked for and made before anything was published, counted by us from our own files. They are not errors that reached a reader, and they are not a test of any tool or of anyone else's writing.
The home page of our website says it plainly: "We use AI to write your articles." We use AI to write our own articles too, and a person checks each of them before it is published. The order matters, because this page is about one step in it. Sources are found and opened before a draft exists. Our writing agent drafts. The draft is scored. Then our sources agent reads the draft against its sources, sentence by sentence. An editor agent reviews it after that, two outside AI models read it cold, and a person reads it and approves it last. The count on this page comes from the fourth step.
Who are the top-ranking guides written for?
The top-ranking guides for this question teach fact checking to journalists, students and the organisations that monitor public claims, and none of the five we read reports what its own work caught. We reviewed the top-ranking pages on 4 November 2026.
- Full Fact AI offers software to organisations that monitor public statements; it "monitors text, video, audio and social media for important statements to verify".
- The Google News Initiative's lesson on Google's Fact Check Tools shows journalists how to find verdicts that others have already published.
- Stanford CRAFT's lesson on AI search results, published 21 January 2025, teaches students lateral reading: testing a statement by comparing it with outside sources.
- contentpen.ai's seven steps, updated 28 September 2026, is written for people who publish AI-generated content; its second step, "Cross-verify claims with trusted sources", is the step closest to ours.
- iMEdD Lab's republished reporter's guide to detecting AI-generated content, by Henk van Ess and dated 14 November 2025, is for journalists working against a deadline.
Each is useful to the reader it was written for. None of the five publishes a count of what its own review caught on its own articles: 0 of 5. That count is what we can add.
What did our sources agent find in 69 AI-drafted articles?
Our sources agent asked for 470 fixes across 69 AI-drafted articles, a median of 7 per article. Most drafts drew fixes after they had been written, sourced and scored; a few drew none. Here are the counts, all from the same 69 files:
- Articles in the count: 69, each drafted by our writing agent.
- Fixes asked for on the first read: 470 across the 69.
- Median per article: 7.
- Range: 0 to 16.
- Articles that needed no fix: 3 of 69.
- Articles that drew 10 or more: 16 of 69.
We counted what the pass asked for. We did not grade the drafts, and a high count does not mean a bad piece: every one of those fixes was made before the piece was published.
Where does this step sit in our process?
The sources agent reads the finished draft after two earlier steps: source verification and the score. So the 470 are what the pass still asked to change once both had run. The first is the verification before drafting. For every statement the piece will rest on, the sources agent records the exact page, its date, the source's own wording quoted, and whether the page still says that when opened that day. A statement it cannot verify goes back to the editor agent before the writer starts, and the section changes or goes.
The second is the score. Each draft is scored by the scoring engine we build, against the pages that rank for its keyword, and revised. The score records are a separate count, and we do not repeat it here.
Only then does the sources agent read the draft against the verified sources. Four questions decide each fix:
- Does each link sit on the statement it supports, not a paragraph away?
- Does the sentence say what the page says, in the same strength, with the right scope, date and place?
- Is anything cited that was never verified at all?
- Is any figure of our own framed as ours, with its limits?
The pass hands the editor agent a numbered list: the sentence, what the page says and the change it proposes. The editor agent orders the fixes, the writing agent makes them, and the sources agent also reads the changed sentences again. Only the writing agent changes the text.
We think the score misses these problems because it mostly compares the words in a draft with the words on the ranking pages, not each sentence with its source. We have not tested that; it is a reading of how the two steps work, not a measurement.
What does a fix look like?
A fix names one sentence and the change the pass asks for, checked against what its source says. Two of the 69 files show the range.
Our article on whether original data gets cited more by AI drew the most: 16 fixes on the first read, seven of them marked as must-fix. One caught a wrong origin. The draft said "The term comes from a Google patent"; the pass found that information gain is an older term, and that the study it cited says the idea "entered search-industry discussion through a Google patent". Another found a quoted sentence credited to the wrong part of the study: the draft placed it in the discussion section, and the file shows it sits in section 6, headed "SEO Implications". In the same file, all 35 of the draft's quotations matched their sources word for word. The problems were in scope, origin and strength, not in the quotes.
At the other end, our article on whether backlinks matter for AI search needed none. Its file says "every link sits on the claim it supports; every quoted string matches the source's own wording".
We did not count the kinds of fix across all 69. The files record them in different shapes: some tally must, should and polish, some split blocking from precision, some only number them. No single method sorts all 69, so these two are examples, not a breakdown.
How did we count the fixes?
We counted the fixes each file states at its first verdict line. One file per article holds the sources agent's first read. Where that line gives a number, in digits or words, as a range or as a sum of must, should and polish, that number is the count. Where it gives none, the count is the length of the numbered list the verdict closes.
The earliest of the 69 files is dated 13 August 2026 and the latest 4 November 2026. A script read the number from 41 of them. We read the other 28 by hand, because their verdicts were written in words or in forms the script missed. Eleven files needed a judgement, such as a re-read file that names the fixes of an earlier pass, and each judgement is written down beside its file. In a few of the earliest files the order of the steps differed slightly. The first count, made when this page was planned, and our re-count, using the same rule, were both made on 4 November 2026, and they matched file for file.
This is one desk, 69 articles and about ten weeks. It is small and early: this is the whole of our record so far. We will keep counting as the desk writes more.
What this count does not tell you
The count does not tell you how often AI writing is wrong in general. It is what our sources agent asked to change in drafts our writing agent wrote, under our own rules. Both follow rules we wrote, so we think a desk with other rules would find other things.
- It is not a count of errors published. Each file closes with its fixes made before publication.
- It is not a measurement of any writing software, any detector or any other writer.
- It does not sort the fixes by kind.
- It holds no figure for what readers found after publication; we have none.
- The files are our own records and are not yet on our research page.
What to do on your own next draft
Five steps come from the process behind our count, and none of them needs particular software. The count shows what that process asked for, not that each of the five is needed.
- List every sentence that rests on a source before you verify anything: each figure, date, quotation and claim about what a study found. A list makes the pass finite, and it shows what the draft asks the reader to trust.
- Open each source and compare the sentence with what it says: the scope, the strength, the date and where on the page it sits. Those were the kinds of problem in our 16-fix example, rather than misquoted words.
- Keep the verification separate from the writing, as a different pass, by a different agent or by a different person from whoever drafted it. We think the separation matters; we have not measured a writer reviewing its own draft.
- Write each fix down, with its reason. An example of the kind of line our sources agent writes: the page says a feature was announced, and the draft says it is live. A line like that can be counted later; our count exists only because every pass wrote its fixes down.
- Have a person read it last, before it is published. We hold no count of what that last read changes, so we make no claim about it.
If you want to go further, a cold read by an outside model is another step; we wrote up what two outside models said about our drafts separately. How much verification a piece needs depends on what your readers will do with it, and that is your call, not ours.
Two questions people ask
Can AI do the fact checking on AI content?
An AI agent does the first pass on our desk: our sources agent is an AI model, and it asked for 470 fixes in 69 drafts that another agent wrote. We counted that. What we believe, and have not tested, is that the pass works because it is separate and has the sources open, not because of which model runs it. A person still reads every piece before it is published.
Does fact checking help AI content pass a detector?
No. Fact checking tests statements against their sources; it says nothing about who or what wrote the text, and we have not tested any detector.
Sources
- Full Fact AI — https://fullfact.ai/
- Google News Initiative — https://newsinitiative.withgoogle.com/resources/trainings/google-fact-check-tools/
- Stanford CRAFT — https://craft.stanford.edu/resource/how-do-i-fact-check-ai-search-results/
- contentpen.ai — https://contentpen.ai/blog/how-to-fact-check-ai-generated-content-in-7-steps
- iMEdD Lab — https://lab.imedd.org/en/reporters-guide-to-detecting-ai-generated-content/
- The top-ranking pages we reviewed for "how to fact check ai content" on 4 November 2026: Full Fact AI, Google News Initiative, Stanford CRAFT, contentpen.ai and iMEdD Lab.
- All pages checked 4 November 2026.
