# Retrieval-provenance audit: replication package **What provenance do you get when you ask an AI assistant for a widely-circulated commercial statistic *and explicitly ask for the original source*?** Everything needed to re-run this study is in this folder. It takes an afternoon and costs roughly the price of lunch. If you disagree with a coding decision, the untouched response is in the data file. > ### Re-sealed 2026-08-08: what changed, and why you are being told > > The package first published on 2026-07-27 was a **partial** snapshot, and it said so in a way > that stopped being true the next day. It served **11 of the 47 collected responses**, omitted the > coding entirely, and its contents note read *"The coding sheet is still to be built."* The coding > sheet was built on **2026-07-28**. That sentence was therefore wrong for the whole time the > package was public, and the omission ran the wrong way: it withheld the material that lets a > reader disagree with us. > > This version publishes everything: all 47 responses, the hand-coded values (`coding.json`) and > the sheet generated from them (`coding-sheet.md`), the second-coder packet, its codes, and the > second-coder pass itself. The reliability pass in `second-coder-pass.md` was run by a language > model rather than a human reader; the protocol's requirement for an independent human reader was > waived on 2026-07-28 and no inter-coder reliability figure is claimed. > > One correction to the protocol text travels with this re-seal: **bcg.com** is described as *"the > firm the experiment ran inside: its consultants were the subjects and it collected the data"*, > not as *"the study's commissioning sponsor"*. The earlier phrasing mischaracterised the > relationship, and it changes what the finding means. > > The pre-registered protocol itself is unchanged in substance: it is registered, and registered > documents are superseded rather than rewritten. `SHA256SUMS.txt` is regenerated for the new file > set; the previous manifest covered eight files, this one covers sixteen. > > ### Integrity re-seal 2026-08-19 > > The 2026-08-08 manifest was not refreshed after a later intentional route and typography sweep > changed package files. The current manifest is regenerated against all 16 files; no response > records or coding values changed in this integrity re-seal. ### Documentation correction 2026-08-28 The current counted result is the one in Appendix F, `coding.json`, and `coding-sheet.md`: 36 counted responses across Q1, Q4 and Q5, with three runs per system. Appendix C and the earlier "What still has to happen" passage are retained as historical protocol text and are superseded by Appendix F. The independent human second-coder requirement was waived and logged in the changelog; the published language-model pass covers 9 of 36 rows, with 43 of 54 field comparisons agreeing and all disagreements unreconciled. `interested_share` remains intentionally uncoded. No essay claim turns any of these fields into a rate. --- ## Contents | File | What it is | | --- | --- | | `protocol.md` | The pre-registered protocol. Question-selection criteria, the ten-field coding scheme, the run design, and: written before the data: what would make the finding wrong. The control-pilot appendix and Appendices B–F carry every cell run so far | | `questions.json` | The six questions, the instruction appended to each, the **ground truth established before each question was run**, and the pre-registered discriminator strings | | `run_audit.py` | The runner. Calls four assistants, writes every raw response to JSONL. Codes nothing | | `responses.jsonl` | The collected responses, with timestamps, model identifiers, citation lists, and per-call cost where it was captured (`money_spent` is null on 11 of 43 records, including the Q5 run-1 cells and the `gpt-5.5` call) | | `coding.json` → `coding-sheet.md` | The hand-coded values, and the human-readable sheet generated from them. Counts are computed from `responses.jsonl`, not typed | | `repeat-runs.md` | Run-by-run comparison across the three questions that have been run | | `link-graph.md` | Referring domains, preprint vs version of record, for both control papers | | `q4-runs.json` | **Superseded.** An early Q4 run table whose eight runs have no record in `responses.jsonl`. Marked at the top of the file; kept as a record of what was observed, never counted | **On `responses.jsonl`:** these runs were collected through DataForSEO's AI-optimization endpoints, not through `run_audit.py`'s direct vendor calls. Every record says so in `collected_via`, because a replication package that hides its collection path is not one. Forty-three records; thirty-nine carry verbatim answer text. Four: the Q4 pilot cells, run before this harness existed: carry `answer_text: null` and a note, and are **not** reconstructed from memory here. **On coverage, stated up front.** A **counted run** is an API-surface run, on one of the four registered model configurations, with verbatim answer text, citations and a timestamp in `responses.jsonl` : nothing is counted that a reader cannot open, and the consumer scrapes and the `gpt-5.5` run are excluded by that definition rather than by convention. Three of the six questions have been run under that definition: **Q1 (runs 1–3), Q4 (runs 3–5) and Q5 (runs 1, 3, 4)**, four systems each, **36 counted responses**, all hand-coded in `coding-sheet.md`. Q2, Q3 and Q6 have not been run at all. Earlier observations that survive only as prose in the appendices are kept as observations and never counted. `interested_share` is null on all 36 rows. No independent human second-coder pass was run; the published language-model pass covers 9 of 36 rows and is explicitly not an inter-coder reliability result. --- ## Re-running it ```bash export OPENAI_API_KEY=... ANTHROPIC_API_KEY=... GEMINI_API_KEY=... PERPLEXITY_API_KEY=... python3 run_audit.py --dry-run # see the exact payloads, call nothing python3 run_audit.py --only Q4,Q5 --runs 1 # the two control questions, once each python3 run_audit.py --runs 3 # the full 72-response dataset python3 make_coding_sheet.py # regenerate coding-sheet.md from coding.json ``` Responses append to `responses.jsonl`. The runner prints which pre-registered discriminator strings appeared in each answer: `[40% -> preprint (deleted from the version of record)]`: as a **prompt to a human coder, not a coding decision**. Substring matching cannot tell a figure stated as the study's result from the same figure in an unrelated aside. Every counted cell is coded by hand. **A live example of why.** On Q1 the registered string `"5 times"` fires inside `"5–25 times"`, where it matches the tail of the *25*, so every ChatGPT run is flagged for a 5× claim it never makes as a standalone figure. The strings were frozen before the counted runs and the matcher has not been touched since, so the artefact stands in the data rather than being quietly repaired. **API shapes drift.** These four vendors change web-search parameters more often than they change models. A 4xx means a builder function needs updating against current vendor documentation; nothing else in the harness depends on it. --- ## Three things to know before you read the results **1 · The API surface and the consumer apps behave differently, and the difference is the finding.** The runner calls API endpoints with web search enabled. `chatgpt.com`, `claude.ai` and the Gemini app are different stacks: different retrieval, different system prompts, sometimes different models. Most people asking these questions use the product, not the API. Both were tested on the controls. See *the surface split* below; it is the most important result here and the runner alone will not reproduce it. **2 · This is not a measurement of whether the models are good.** It is a measurement of **the public evidence surface for commercial questions**, sampled through the tool most people now use to reach it. Where the sources are poor, that is a fact about what has been published as much as about what was retrieved. "AI is unreliable" is an available reading of this data, and it is the wrong one: see the Crimson case below. **3 · Two of the six questions are controls, and they carried the finding.** Q4 and Q5 were included because a clean, reachable, peer-reviewed primary source exists for each, so that if the assistants cited them correctly, the honest conclusion would narrow to "retrieval works where the literature is sound, and the problem is the literature." The protocol says so in advance, in the section headed *what would make this finding wrong*. That is not what happened. --- ## What the control cells found Three counted runs per system, both controls, 2026-07-27. **Q4: Brynjolfsson, Li & Raymond.** Version of record: *QJE* 140(2), 889–942 (2025), 15% lift, 5,172 agents. **Claude cited it on all three runs** with an `academic.oup.com` link, giving full pagination on two of them. Run 4, verbatim: *"Erik Brynjolfsson, Danielle Li, Lindsey Raymond, 'Generative AI at Work,' The Quarterly Journal of Economics, Volume 140, Issue 2, May 2025, Pages 889–942."* ChatGPT and Perplexity returned the 2023 NBER preprint's 14% on every run, 0 for 3 each. That matters more than a uniform failure would: it proves **the version of record is reachable**. No stack can be excused by the paywall. One found it three times without being asked. **But Claude's passes are not clean, and one Gemini run breaks the frame.** Claude's run 4 cites the QJE article *and* reports the preprint's 5,179 agents in the same answer: right provenance, superseded number. And **Gemini's run 5 reports 15% and 5,172, both version-of-record figures, names the QJE: and cites no link to it**, filling its slots with `stanford.edu`, `arxiv.org`, `revartis.com`, `itif.org`, `policycommons.net` and `richmondfed.org` instead. Correct content, absent provenance: against a prompt that asks for the source, still a failure, but a different one. Gemini also produced a third value for the same quantity on run 4: *"Some sources specify the average increase as 13.8%"*: a figure in neither version, attributed to nobody. **Q5: Dell'Acqua et al., the BCG/GPT-4 study.** This one discriminates cleanly, because peer review changed the headline. The preprint claims *"more than 40% higher quality"*; the published *Organization Science* paper reports **33.9%** and **29.9%** and contains no 40% claim at all. The preprint's "43% / 17%" skill-split claim was removed entirely. Across **thirteen API runs: four systems, three counted runs each, plus one on the strongest available model: every single one carried a quality figure the published paper does not contain, and not one cited the version of record.** Between them the citation sets contain a student newspaper, LinkedIn, Facebook, Reddit, YouTube, a legal-industry trade site, a national newspaper, a career-quiz site, a training vendor, general tech press, and the commissioning firm's own marketing page, and zero links to *Organization Science*. **The fabricated `40.2%` is ChatGPT's, and it is stable: three runs, three times.** That decimal appears in neither version of the paper. It appears on `jobcannon.io`, a career-assessment quiz site, which states *"produced results rated 40.2% higher in quality"*: checked at source 2026-07-27. On run 4 ChatGPT's **only** citation is BCG's own PDF: a document that states 40%. The figure has come loose from any source that contains it. **It has not spread, and that correction is itself a result.** Gemini returned 40.2% on one run, with `jobcannon.io` in its top slot, which looked like the decimal crossing to a second stack. The next run returned to 40%, `jobcannon.io` still in the citation set. One occurrence in three is a fluctuation, not a transfer, and the stronger claim was live for about half an hour before the next run removed it. **Why Q4 is passable and Q5 is not.** Both papers have an open preprint and a paywalled version of record, so the paywall is not the variable. Age and path are. *Generative AI at Work* has been the version of record since May 2025, the NBER page links forward to it, and citations have accumulated both ways. The *Organization Science* article appeared in **March 2026**: four months before these runs: while its preprint had three years to gather links under a working-paper number that trade press, vendors and SEO pages all cite. Measured in `link-graph.md`: 3:1 referring domains on Q4, 20:1 on Q5. So the mechanism is narrower than "paywalled papers lose." It is that **the version of record has to out-age its own preprint before retrieval finds it**, and during that window, which runs to years, the superseded numbers are what everyone gets. That window is where every recently-published finding lives. Three distinct mechanisms, each verified against primary documents: - **Version lag.** The peer-reviewed correction never surfaces. Replicated across two papers in two disciplines. - **Sponsor substitution.** BCG's own page says *"23% worse"* where the paper says 19 percentage points. Perplexity named BCG's marketing publication as the original source on all three counted runs: *"The original **BCG** source is…"* (run 1), *"The original source is BCG's report"* (run 3), *"The original source is BCG's publication"* (run 4). - **False precision, acquired downstream.** `40.2%` appears in **neither** version of the paper. A rounded floor over two experimental conditions grew a decimal point on its way down the chain and came back as a fact with a citation attached: on one stack, three times out of three. - **Correct content, absent provenance.** Gemini's Q4 run 5 has the published numbers and no link to the published paper. The instruction asked for the source; the source is what is missing. ### The case that decides how to read all of this Claude returned *"40 percent of the trial group produced higher quality results"*: a magnitude silently converted into a headcount, which is a claim about a different quantity entirely. It cited the Harvard Crimson. **The Crimson says exactly that, verbatim.** The model reproduced its source faithfully and cited it correctly. The error is the newspaper's. This is not a story about models inventing things. It is a story about a retrieval layer that is accurate with respect to a corpus that is wrong: with the actual paper two clicks away, and nothing in the chain having any reason to open it. --- ## What Q1 added: the question with no answer to find Q1 is the unrecoverable class: *"How much more expensive is acquiring a customer than retaining one?"* No primary document reports this ratio. Reichheld & Sasser's *"Zero Defections"* (HBR 1990) does not contain it; HBR 2014, the article everyone cites, states it hedged and cites nobody. Twelve runs. **121 citation slots, 23 unique domain labels, zero primary sources.** That was expected and on its own it proves little: a question chosen for having no source produces no source. The 23 counts registrable domain labels from every citation object, including Gemini's labels where the URL is null. A URL-only recount is 20. The difference is exactly three Gemini-only labels: `2wtech.com`, `proceragroup.com` and `thefundworks.com`. Their destinations are not verified, so the package uses the label-inclusive count descriptively and does not register it as a study claim. What the cell is worth is the three different ways the systems handled the absence, and where those handlings came from: - **Claude asserted a Bain / Reichheld origin that does not exist**, three times out of three: *"The original source … is **Bain & Company research conducted by Frederick Reichheld**"* (runs 1 and 3; run 2 words it *"research by Frederick Reichheld at Bain & Company"*). - **Gemini named a real document that does not contain the claim**: the 1990 HBR article, on two of three runs, which is the harder failure to catch, because the citation checks out until someone opens it. - **ChatGPT named no origin at all** on two of three runs, and filled both citation slots from one vendor blog. Zero provenance errors, earned by making no provenance claim. - **Perplexity reported the attribution as an attribution** on all three runs: *"most often **traced for** this rule of thumb"*, *"later articles **trace it to**"*. Those sentences are true, and they are the only construction in the cell that survives contact with the ground truth. **Both wrong provenances exist pre-written in the corpus, checked at source, but only one of them is a route.** `stealthagents.com`: a virtual-assistant staffing company's SEO page: states: *"The foundational figure **here** comes from Bain and Company research by Frederick Reichheld: acquiring a new customer costs 5 to 25 times more than retaining an existing one."* Claude's answer is that sentence lightly reworded: *"The foundational figure comes from Bain and Company research by Frederick Reichheld, which established this 5-25x range"*, and Claude cites that page on all three runs. `releva.ai` states the figure is *"from Frederick Reichheld's research at Bain & Company, first published in the 1990 Harvard Business Review article 'Zero Defections'."* That is Gemini's answer, pre-written, but **`releva.ai` is not in Gemini's citation set on any run**, so how it got there is unknown. The first is the Crimson case, replicated on a different question with a different kind of document. The second is a match, and the package says so rather than rounding it up. **The debunkings are still being cited as support.** On all three Perplexity runs, three cited documents dispute the claim they are cited for: `hashtagpaid.com` (*"one source traces its statistics to another source, which quotes another source"*), `churnmiser.com` (*"rarely to a controlled study you could reproduce"*), `releva.ai` (*"dangerously outdated"*). The pilot found the same pattern on two different documents. Five documents, two independent runs of the design. **One domain is cited by all four systems**: a growth consultancy's blog, refreshed in July 2026, carrying the ratio in its headline and naming no source for it. The one place every stack reaches for on this question has no provenance to give. (Domain-level: Gemini's citation URLs are not retained, so a document-level "most cited" claim is not available.) --- ## The surface split Both controls were also put to the **consumer ChatGPT interface**, because a study about "the tool people use" that only tested API endpoints would be making an unstated generalisation. The consumer surface is markedly better at *citation*. On Q5 it cited the *Organization Science* DOI : `pubsonline.informs.org/doi/10.1287/orsc.2025.21838`, and named both versions correctly. On Q4 it volunteered, unprompted, that *"a later peer-reviewed version published in The Quarterly Journal of Economics reports a 15% average productivity increase"*. It is no better at *content*. It still reported "more than 40% higher in quality", and cited the *Organization Science* page for a claim that paper does not contain. **Is that just a stronger model?** No. Tested directly: the same question, the same API, `gpt-5.5` with reasoning enabled at roughly three times the cost per call. It cited the preprint PDF and BCG's marketing page, reported 40% and 23%, and **never mentioned the published version at all**. The strongest model on the API did worse on provenance than the consumer product. The retrieval stack is doing the work, not the model. `gpt-5.5`'s own reasoning summary, on the way there: > *"There's a PDF that likely contains relevant figures, like a 12.2% increase in tasks completed, > 25.1% faster performance, and over 40% improved quality. **I can cite search14 even if it's not fully > opened yet.**"* It attaches a citation to a document it has not opened, on the strength of what it expects the document to say, and what it expects is the figure peer review removed. So there are two independent failures, and only one of them is improving: | | Citation | Content | | --- | --- | --- | | API | poor: career-quiz sites, sponsor marketing | wrong figure | | Consumer | good: DOI, both versions named | **same wrong figure** | **Better citation practice made the error harder to catch, not less likely to occur.** A wrong number sourced to `jobcannon.io` can be smelled. The same number sourced to a DOI cannot. *Limits:* one system (only ChatGPT has a consumer scrape here), one run each, one day, US location, and **no consumer arm on Q1 at all**. Nothing here supports a claim about consumer Claude, Gemini or Perplexity. --- ## What this does not establish Stated here rather than left for a critic to find. - **Half the question set has not been run.** Q2, Q3 and Q6 are missing entirely: 36 counted responses against a planned 72. - **Some earlier observations are prose, not data.** The "run A / run B" tables in Appendix C describe runs whose verbatim text is not in `responses.jsonl`. They are excluded from every **counted** figure : the 36-row coding sheet, and every rate in Appendix F, `repeat-runs.md` and this README. Appendix C itself still computes its own superseded rates over them ("nine API runs", "2 of its 6 runs"); it is kept as a record of what was seen and read against Appendix F, which supersedes it. This was found by an adversarial check of the write-up, not by a reader, and closing it took twelve further runs. - **`interested_share` is uncoded everywhere.** The protocol requires each domain to be classified from its own homepage, and that pass has not been run on any cell. So nothing here may be stated as a vendor share. One classification is already settled and cuts against the thesis: `jobcannon.io` is a career-assessment site and does **not** count as interested, because it does not sell in the category the claim supports. The rule was written to bite against the coder's convenience, and it did. - **No independent human second coder**, on any cell. Section 6 requires a 20% sample with disagreements reported rather than reconciled away. The author waived that requirement on 2026-07-28; the language-model pass is published as a separate, unreconciled check and does not substitute for a human reliability pass. - **Gemini's citation URLs are not in the dataset.** Every one resolves through a `vertexaisearch.cloud.google.com/grounding-api-redirect/` URL exposing only a domain label; the opaque token was not retained. Gemini's domains are coded, its destinations are unverified, and a reader of the answer is in exactly the same position. - **Corpus tracing is a match, not a route.** Where a reproduced sentence appears in a document the answer cites, the link is strong. Where it appears in a document the answer does not cite: as with the pilot's "Bain Capital" error, which `hashtagpaid.com` also makes: the path is unknown and the essay may not treat it as fabrication *or* as propagation. - **Six questions is a small set**, chosen against stated criteria, not sampled from a population. --- ## Novelty search for the companion essay, 2026-08-28 The companion essay's former novelty sentence asked whether prior work had traced an assistant's attribution back to the retrieved page. On 28 August 2026, the following stated searches were run: ```text site:aclanthology.org AI assistant source attribution provenance retrieved webpage citation tracing study site:aclanthology.org language models citation provenance source attribution web retrieval study site:arxiv.org language model source attribution citation provenance retrieved web page study Google Scholar: "source attribution" language model web retrieval citation provenance ``` The search found adjacent primary work, including [Document Attribution](https://aclanthology.org/2025.sdp-1.12/), [Evaluation of Attribution Bias](https://aclanthology.org/2025.findings-acl.1087/), [Assessing Web Search Credibility and Response Groundedness in Chat Assistants](https://aclanthology.org/2026.eacl-long.115/), and [Cited but Not Verified](https://arxiv.org/abs/2605.06635). These works evaluate document attribution, citation faithfulness, source credibility, or groundedness. No exact match was identified in this bounded search for an audit of a widely circulated commercial statistic that traces an assistant's asserted origin back through its cited web page. That absence is a search result, not proof of priority. The Google Scholar route was not readable in this browser session. **Disposition:** the live English and German companion pieces do not claim that this is the first such study. The former novelty sentence is retired and must not be reinstated as a universal claim. The contribution is the bounded Q1 audit, its raw response record, and the documented distinction between a supplied wording path and an established source origin. --- ## Licence and citation Data and code released for re-use and disagreement. > Isoglu, S. (2026). *Retrieval-provenance audit: what four assistants return when asked where a > number comes from.* isoglu.com.