---
type: pre-registered protocol
piece: research cluster: retrieval audit
registered: 2026-07-27, before any run beyond the single pilot question
rule: P6 type 3 (primary data) · P49 (decide the analysis before seeing the data) · P60
status: FROZEN. Changes after the first run go in the changelog below, dated, with the reason.
---

# Protocol: what four assistants return when asked where a number comes from

Registered before the runs. Published with the data so a reader can re-run it and disagree.

---

## 1 · The question this answers

When someone with a commercial decision to make asks an AI assistant a question whose answer is a
widely-circulated statistic, **and asks explicitly for the original source, what provenance do they
get?**

Two things make this answerable rather than rhetorical. Asking for the source in the prompt turns an
omission into a failure against a stated instruction. And provenance is checkable: either a primary
document exists and can be produced, or its absence can be established by search.

---

## 2 · What is being measured, and what is not

**Measured:** what the retrieval layer returns for these questions, on these dates.

**Not measured:** whether the models are good, whether they are getting better, or whether they are
reliable in general. Every result here is a statement about **the public evidence surface for
commercial questions**, sampled through the tool most people now use to reach it. If the sources are
poor, that is a fact about what has been published, not only about what was retrieved.

The essay must not become "AI is unreliable." That framing is available, cheap, and wrong.

---

## 3 · Question selection: criteria fixed before choosing

A question qualifies only if **all four** hold:

1. It states or implies a **specific numeric claim** that circulates in commercial practice.
2. The provenance is **checkable**: a primary source either exists and is reachable, or its absence
   can be established.
3. A commercial decision-maker would **plausibly ask it**, in roughly these words.
4. It was chosen **before** any assistant was asked.

### The control requirement, which is the important one

**At least one third of questions must be ones where a clean, reachable primary source exists.**

Without controls the design manufactures its own finding: ask only about statistics known to be
unsourceable and of course the answers are unsourced. Controls make the result falsifiable: if the
assistants cite the primary correctly whenever one exists, then the finding is about the literature,
not about retrieval, and the essay says so.

### The set

| # | Question | Class | Expected provenance |
| --- | --- | --- | --- |
| Q1 | How much more expensive is acquiring a customer than retaining one? | Unrecoverable | None exists: established 2026-07-27 |
| Q2 | What share of the B2B buying journey is complete before a vendor is contacted? | Contested | A real origin exists (CEB/Gartner) and is routinely misstated |
| Q3 | How many touchpoints does it take to book a B2B sales meeting? | Vendor-origin | Trade figures only, no recoverable method |
| Q4 | What did Brynjolfsson, Li and Raymond find about AI and customer-support productivity? | **CONTROL** | *QJE* 140(2), 889–942. Clean, open, findable |
| Q5 | What did the BCG study find about consultants using GPT-4? | **CONTROL** | *Organization Science* 37(2), 403–423. Clean, and see the ground-truth block below, established 2026-07-27 before the run |
| Q6 | What proportion of A/B tests improve the metric they target? | Semi-recoverable | Kohavi's one-third is real but conference-only; the circulating 90% is not his |

Two of six are controls. Q6 is deliberately awkward: a real figure exists in a hard-to-reach place
while a false one circulates freely, which is the most common shape in commercial evidence.

---

## 4 · The prompt

Identical across every model and run. Only the question varies.

> `<QUESTION>` **Give the figure and cite the original source.**

The instruction to cite is not optional decoration. It converts a silent omission into a
non-compliance, which is a different and more defensible finding.

---

## 5 · Run design

- **Systems:** ChatGPT, Claude, Gemini, Perplexity. Web search **on**. Model identifiers recorded per
  run, because they change.
- **Surface: the limitation that matters most.** These runs go to **API endpoints with web search
  enabled**, not to the consumer chat products. `chatgpt.com` and `claude.ai` are different stacks:
  different retrieval, different system prompts, different ranking, sometimes different models. Most
  people asking these questions are using the consumer app. **The finding as run is therefore about
  the API surface, and the essay may not silently generalise it to the product surface.** Either the
  claim is narrowed to what was tested, or the consumer surface is tested too: see the changelog.
  Recorded here, before the counted dataset, because discovering this after publication would be the
  same failure the study is about.
- **Repeats:** **three runs per model per question.** These systems are stochastic and a single run is
  an anecdote. 6 questions × 4 systems × 3 runs = **72 responses**.
- **Temperature:** left at the endpoint default, recorded.
- **Dates:** every response timestamped. The corpus moves; the finding is dated or it is nothing.
- **Cost:** roughly €8 at observed per-call pricing. The whole study is re-runnable by a reader for
  the price of lunch, which is the point.

---

## 6 · Coding scheme: fixed before any run

Every response coded on ten fields. Where a judgement is needed the rule is written here, not
decided later.

| Field | Values | Rule |
| --- | --- | --- |
| `figure_given` | yes / no / range | The number, verbatim |
| `source_named` | yes / no | Any specific document, author or organisation named as the origin |
| `source_class` | primary / secondary / tertiary / none | **Primary** = the study or document that first reported it. **Secondary** = a peer-reviewed work citing it. **Tertiary** = trade press, vendor content, encyclopedia, listicle |
| `citations_n` | integer | Total citation slots returned |
| `domains_unique` | integer | Distinct domains among them. The gap between this and `citations_n` is repetition presented as breadth |
| `interested_share` | 0–1 | Share of unique domains that **sell in the category the claim supports**. Coded from the domain's own homepage, not from memory |
| `contra_as_support` | yes / no | Does any cited document **argue against** the claim it is cited for? |
| `provenance_error` | list | Factual errors *in the provenance itself*: wrong organisation, wrong year, wrong paper, wrong author |
| `search_used` | yes / no | Whether the system actually retrieved, or answered from parametric memory with search requested. **Added after the control pilot**: see changelog |
| `version_returned` | vor / preprint / preprint+ / n/a | Where a published version of record and an earlier preprint both exist, which one the answer is built on. **`preprint+`** = built on the preprint *and* carrying a figure that appears in neither version: added 2026-07-27, see changelog |

**`interested_share` is the field most open to motivated coding, so it gets the strictest rule.** A
domain counts as interested only if it sells a product or service in the category the claim supports
:  a retention-software vendor for a retention claim. General business media do not count. Consultancies
do not count unless the claim is about the service they sell. Every classification is recorded with
the URL so a reader can disagree with individual calls.

**Second coder:** a pass by an independent reader on a 20% sample, with disagreements reported rather
than reconciled away. If agreement is poor, that is published too.

---

## 7 · Stated in advance: what would make this finding wrong

- **Controls come back clean and everything else is noise.** If Q4 and Q5 are cited to primary
  reliably, the honest conclusion is narrower: retrieval works where the literature is sound, and the
  problem is the literature. That is a *better* essay and the protocol should be allowed to produce it.
- **Repeat runs disagree wildly.** If the same model gives a different `source_class` across three
  runs, the headline is variance, not provenance, and it must be reported as such.
- **The interested-share coding does not survive a second coder.** Then that column comes out.

---

## 8 · What gets published

The protocol, the full response set with timestamps and model identifiers, the coding sheet with
every URL, the second-coder disagreements, and the script. A reader with an API key re-runs it in an
afternoon.

That is the falsifiability the previous two briefs could not offer, and it is the only reason this
candidate is alive when seventeen others are not.

---

## Changelog

| Date | Change | Reason |
| --- | --- | --- |
| 2026-07-27 | Registered. Q1 already run once as a pilot, before the protocol existed, that run is reported separately as a pilot and **excluded from the counted dataset**. | Pilot preceded registration; counting it would be coding after seeing the data |
| 2026-07-27 | Two fields added: `search_used` and `version_returned`: after the Q4 control pilot, before any counted run. | The control produced a failure mode the scheme could not record. One system answered with `web_search: false` despite search being requested, and both systems built their answer on the 2023 preprint rather than the 2025 version of record. "No source" and "the superseded source" are different findings with different causes, and the original eight fields collapsed them |
| 2026-07-27 | Q4 and Q5 promoted from control to **primary interest**. | The controls were included to make the design falsifiable by showing retrieval works where the literature is sound. On the first control cell it did not, which makes them the most informative questions in the set, not the least |
| 2026-07-27 | Q5 ground truth built and written into the protocol **before** the run, including the preprint/version-of-record difference table. | Q4 was coded against a ground truth that already existed in the verified bank. Q5's had to be constructed, and constructing it after seeing the answers would be coding on the data |
| 2026-07-27 | Added to the analysis plan: **count inbound links to preprint vs version of record** for both control papers. | Eight of eight cells are explained by "the link graph rewards the free version." That is currently an inference. If the essay states the mechanism, the mechanism has to be measured |
| 2026-07-27 | **Surface limitation recorded, and a consumer-surface arm added.** All runs to date are against API endpoints, not `chatgpt.com` / `claude.ai` / the Gemini app. Q5 will also be run through a consumer-interface scrape, and if the two surfaces disagree the essay reports both rather than the convenient one. | The essay's premise is "the tool most people now use." Most people use the product, not the API. Running only the API and writing about the product would be an unstated generalisation: precisely the failure mode this study documents in others. Caught before the counted dataset, not after |
| 2026-07-27 | **Surface arm run and reported** (Appendix B). Both controls put to the consumer ChatGPT interface; the capability confound broken with a `gpt-5.5` API run. | The limitation recorded an hour earlier turned out to be load-bearing: the consumer surface cites the version of record and the API does not, at any model strength. Had the essay shipped on API runs alone it would have been wrong about the mechanism |
| 2026-07-27 | **Mechanism measured** (Appendix D). Referring domains counted for both papers' preprint and version-of-record URLs. | The protocol required this before the essay could assert a mechanism. The ratio predicts the retrieval outcome on both questions: 3:1 with one system finding the VoR, 20:1 with none, so "the link graph rewards the free version" moves from inference to measurement |
| 2026-07-27 | Claude's Q5 cell re-run with search confirmed active; the two `web_search: false` runs are excluded from counted cells. | Runs answered from parametric memory do not measure retrieval and cannot sit in the same column as ones that do |
| 2026-07-28, 00:03 | **German-language anchor arm added** (`language: "de"`, `counted: false`): Q1 put to all four systems in German, one run each, records `run: 101`. Not a counted cell and never mixed into a rate. | The essay ships with a German twin, and a twin that re-uses an English finding with German words on it is a translation, not a localisation. The arm records what the German-language corpus actually returns, and it differs: a fourth ratio (16x) the English cell never produces, one system answering a German question entirely out of English-language sources, and one system declining to name an origin at all where the same model asserted a false one on 3 of 3 English runs |
| 2026-07-27, 20:44 | **A counted run redefined as one with verbatim text in `responses.jsonl`**, and twelve further control runs collected so both controls meet it (Appendix F). Q4 counted = runs 3, 4, 5; Q5 counted = runs 1, 3, 4; Q1 = runs 1, 2, 3. Thirty-six coded rows. | An adversarial check of the write-up found that several claimed runs: including two thirds of the "Claude 3/3 on Q4" result: existed only as prose in Appendices A and C, with no inspectable record. A replication package whose headline a reader cannot check is the failure this study documents in others. The narrative runs are kept as observations and excluded from every rate |
| 2026-07-27, 20:44 | **Two readings corrected against the new runs.** (a) "The fabricated 40.2% has crossed retrieval stacks": Gemini returned 40% again on the next run, so one occurrence in three is a fluctuation, not a transfer. (b) `releva.ai` was described as the pre-written source of Gemini's misattribution; it is not in Gemini's citation set on any run, so it is a match with no established route. | Both were stronger claims than the data carried, and both were found by checking rather than by a reader |
| 2026-07-27, 20:44 | **`preprint+` registered** as a fourth value of `version_returned`: an answer built on the preprint that also carries a figure appearing in **neither** version. Applied to four rows (ChatGPT's three 40.2% runs and Gemini's one). | The field as registered could not distinguish "returned the superseded number" from "returned a number that exists in no version of the paper", and those are different findings with different causes. The value was in use before it was defined, which is the error this row closes |
| 2026-07-27, 20:44 | **`q4-runs.json` marked superseded** in the file itself, alongside Appendix C. | Appendix C was marked and its machine-readable twin was not, and a re-analyst loads the JSON, not the prose. Same defect, worse format |
| 2026-07-27, 20:16 | **Coding sheet built** (`coding.json` → `coding-sheet.md`) for all twenty counted responses. `citations_n` and `domains_unique` are computed from `responses.jsonl` rather than typed; `interested_share` is null on every row. | Section 6 requires the sheet with every URL. Computing the two count fields removes the transcription error the sheet would otherwise be exposed to; leaving `interested_share` null is the honest state, since the domain-by-domain homepage pass has not been run on any cell |
| 2026-07-27, 20:16 | **System labels normalised** across `responses.jsonl` to the four collection labels: `chat_gpt`, `claude`, `gemini`, `perplexity`: replacing the mixed `anthropic`/`claude` usage inherited from the runner's own key names. Two records changed. | The dataset was collected through DataForSEO, not through the runner's direct vendor calls, and carried both naming schemes. One dataset, one label per system |
| 2026-07-27, 20:15 | **Controls closed at three runs** (Appendix F). Q4 and Q5 each now have three runs per system. Nothing in the two-run reading reverses. | The protocol requires three and Appendix C reported two. Two is not three |
| 2026-07-27, 20:11 | **Q1 run as the first counted cell** (Appendix E): four systems × three runs, coded by hand. | First non-control question to enter the counted dataset |
| 2026-07-27, 17:06 | **Q1 discriminators registered** in `questions.json`, before the counted Q1 runs and after the 13:57 pilot. Thirteen strings: the three circulating ratios, the four attributions the trail shows to be wrong or empty, the two debunking sources, and the 25–95% drift. | The field was empty at freeze because Q1 had no version-of-record/preprint pair to discriminate. It has something better: a documented trail of attributions that are checkably not the source. Registering the strings **after the pilot** is recorded here rather than hidden: the pilot is excluded from the counted dataset, the strings derive from the independently-verified ground truth rather than from any model's answer, and a substring hit is a prompt to a coder, never a coded value. A reader who thinks the set was chosen to flatter the finding can compare it against the ground-truth note, which predates both |
| 2026-07-27 | **Theme and scope ruled** (SI): the piece opens a new **evidence/method** cluster rather than being bent into revops, growth or gtm; and it publishes as **essay + open replication package**: protocol, full responses with timestamps and model IDs, coding sheet with every URL, and the script. | The finding is about provenance itself, so an existing cluster would distort it. And a piece arguing "you cannot check the numbers you are handed" that ships unverifiable would refute itself |
| 2026-07-28 | **The section 6 second-coder requirement is waived** (SI). No independent human pass will be run on this study. The language-model pass of 2026-07-28: `second-coder-pass.md`, 9 of 36 rows, 43/54 field calls agreeing, disagreements unreconciled: stands as the only reliability check, labelled as what it is. Section 6's text is left exactly as registered. | No independent reader was available and none is being sought, so leaving the requirement open reports a check as pending that is not coming. Editing section 6 after the data would be rewriting the pre-registration, which is worse. The requirement therefore stands, the deviation is recorded here, and the cost is carried in the limitations: no inter-coder reliability figure is claimed for this study, and `source_class` still cannot carry a rate. |

---

# Appendix: the control pilot, 2026-07-27

Run before the counted dataset, to check the scheme was operable. **It was not**, which is what a
pilot is for. Two fields were added and the controls were reclassified.

## Q4 · "What did Brynjolfsson, Li and Raymond find about the effect of AI assistance on customer support agent productivity? Give the figure and cite the original source."

**Ground truth**, from the verified bank: the version of record is *The Quarterly Journal of
Economics* **140**(2), 889–942 (2025). **15%** average lift, **5,172** agents. The **34%** figure for
novice and low-skilled workers **exists only in the 2023 NBER working paper**, which reports 14% and
5,179 agents. This is guard `AI-X4`, written weeks ago: *"never cite BLR as '2023, 14%, QJE'."*

| System | Figure | Named as the original source | `version_returned` | `search_used` |
| --- | --- | --- | --- | --- |
| Perplexity · sonar | **14%**, plus **34%** for novices | "the NBER working paper *Generative AI at Work*" | **preprint** | yes |
| Claude · sonnet-4.5 | **14%**, plus "about **35%**" for the bottom quartile | "NBER Working Paper No. 31161, April 2023" | **preprint** | **no** |
| Gemini · 2.5-flash | **14%**, **35%**, **5,179** agents | "NBER Working Paper No. 31161": *while stating it was later published in the QJE in May 2025, and citing oup.com* | **preprint** | yes |
| ChatGPT · gpt-4.1-mini | **14%**, plus **34%** for novices | "the working paper … published by NBER in April 2023" | **preprint** | yes: 4 citation slots, **all nber.org** |

**Four systems out of four returned the preprint. Not one cited the version of record.**

Perplexity returned eight citations: NBER twice, MIT Sloan, arXiv, a personal reading-list page, the
US Bureau of Labor Statistics, a law firm's blog, and a LinkedIn post, and not one of them is the
*QJE* article. Claude reported `web_search: false` despite search being requested, answered from
parametric memory, and produced "about 35%", a figure that appears in neither version. ChatGPT
retrieved, filled four citation slots, and every one of them was `nber.org`: `domains_unique` = 1
against `citations_n` = 4, which is the repetition-as-breadth pattern the coding scheme was built to
catch.

**Gemini is the case that settles the mechanism.** It stated in its own answer that the paper *"was
later published in The Quarterly Journal of Economics in May 2025."* It cited `oup.com`. It knew the
version of record existed, named it, and linked to its publisher, and then reported 14%, 35% and
5,179 agents, which are the preprint's numbers. The published paper says 15% and 5,172. This is not a
model that failed to find the paper. It is a model that found both and built its answer on the one
the web points at.

## Why this changes the piece

The controls were included to make the design falsifiable. **The first control failed on all four
systems, in a way the myth questions cannot.** A clean, peer-reviewed, top-five-journal version of
record exists, is indexed, and has a DOI, and the retrieval layer returned the three-year-old
preprint with numbers the authors themselves revised.

The failure is also *uniform*, which matters more than it being total. Four systems, four different
retrieval stacks, one shared answer. That rules out an idiosyncratic index and points at the thing
they have in common: the public link graph.

The mechanism is not model failure and should not be written as one. The working paper is **free,
open, and has had three years to accumulate links**. The version of record sits behind Oxford
University Press. The web's link graph rewards the preprint, so the preprint is what comes back : 
and with it, the superseded numbers.

That generalises to every paper whose preprint is open and whose version of record is paywalled,
which in economics is most of them. **The number you get is the number that was free.**

That is a better, narrower and more mechanistic finding than "commercial statistics are unsourced",
and it arrived from the question included specifically to disprove the thesis.

## What the pilot does not establish

Stated plainly, because the next runs are designed to test it and not to confirm it.

- **One question, one date, one run each.** Four systems agreeing on one paper is four data points,
  not a rate. The counted dataset runs three times per system per question for exactly this reason.
- **This paper may be unusually preprint-heavy.** *Generative AI at Work* circulated for two years as
  a working paper in a field where that is normal, and was widely read before publication. Q5 (BCG ·
  *Organization Science*) is the immediate test: same shape, different discipline, shorter preprint
  life.
- **The mechanism is inferred, not measured.** "The link graph rewards the preprint" is the best
  available explanation for four-of-four, but the pilot did not count inbound links to either
  version. If the essay asserts the mechanism it has to measure it, which is cheap, and should be a
  named step rather than a rhetorical flourish.

---

## Q5 · ground truth, established 2026-07-27 **before** the run

Written down first, deliberately. The Q4 cell was coded against a ground truth that already existed
in the verified bank; this one had to be built, and building it after seeing four answers would be
coding on the data.

**Version of record.** Dell'Acqua, McFowland III, Mollick, Lifshitz, Kellogg, Rajendran, Krayer,
Candelon & Lakhani (2026). *Organization Science* **37**(2), 403–423.
DOI `10.1287/orsc.2025.21838`. Pagination confirmed against Crossref, not inferred.

**Preprint.** Harvard Business School Working Paper **24-013**, 22 September 2023. Also on SSRN as
`4573321`.

| Claim | Preprint (2023) | Version of record (2026) |
| --- | --- | --- |
| Subjects | 758 **consultants**, "about 7% of the individual contributor-level consultants" | 758 **knowledge workers** |
| Tasks completed | 12.2% more | 12.2% more: **unchanged** |
| Speed | 25.1% more quickly | 25.1% more quickly: **unchanged** |
| **Quality** | **"more than 40% higher quality compared to a control group"** | **The 40% figure does not appear anywhere in the paper.** Table 4 reports **33.9%** (GPT + overview) and **29.9%** (GPT only) |
| **Skill split** | **"those below the average performance threshold increasing by 43% and those above increasing by 17%"** | **Neither figure appears anywhere in the paper** |
| Outside the frontier | 19 **percentage points** less likely to be correct | Abstract says "19% less likely"; the body still says "an average decrease of 19 percentage points" |

**Why this is a stronger version test than Q4.** The productivity headlines are byte-identical across
versions, so no system can be caught out on those. What changed is the *quality* result, and "40%
higher quality" is the single most-quoted number this study produced. Peer review cut it to roughly
30–34% and removed it from the abstract, and deleted the 43%/17% skill-equalisation claim from the
paper entirely.

So Q5 discriminates cleanly. **Any answer containing "40% higher quality", "43%" or "17%" is built on
the preprint, whatever it cites.** An answer citing the *Organization Science* article while quoting
40% is citing a document that does not contain its own number, which is a `provenance_error`, not a
version difference, and is coded as both.

**New guard for the verified bank, `AI-X5`:** *never cite the BCG/Dell'Acqua study as "40% higher
quality" or repeat the 43%/17% skill split: all three are preprint-only and were removed in peer
review. The published quality effects are 33.9% and 29.9% (Table 4).*

## Q5 · the run, 2026-07-27, all four systems

One run each, same prompt, web search requested on all four.

| System | Quality figure returned | Outside-frontier figure | Named as the original source | `version` | `search` |
| --- | --- | --- | --- | --- | --- |
| Perplexity · sonar | **40%** | **23%** | *"The original BCG source is **How People Create and Destroy Value with Generative AI** from Boston Consulting Group"* | preprint | yes |
| ChatGPT · gpt-4.1-mini | **40.2%** | **23%** | Never names the paper. Credits "a study conducted by the Boston Consulting Group" | preprint+ | yes |
| Claude · sonnet-4-5 | **"40 percent of the trial group produced higher quality results"** | 19% | "a working paper titled *Navigating the Jagged Technological Frontier*, September 22, 2023" | preprint | yes |
| Gemini · 2.5-flash | **40%** | not given | "a working paper … published by Harvard Business School. The lead author … is Fabrizio Dell'Acqua" | preprint | yes |

**All four returned the deleted number. Not one of the four cited the paper, in any version.**

That second sentence is the finding. On Q4 the systems at least linked to NBER. Here the citation
sets contain a student newspaper, LinkedIn posts, a Facebook post, a YouTube video, a legal-industry
trade site, an Indian national newspaper, a career-assessment quiz site, a training vendor, general
tech press, and the sponsoring firm's own marketing. Zero links to *Organization Science*, zero to
the SSRN record, and one to an HBS landing page.

### Citation sets, recorded in full so the coding can be disagreed with

- **Perplexity**: 18 slots, **13 unique domains**: bcg.com (×4, incl. media-publications.bcg.com),
  jobcannon.io, thecrimson.com, facebook.com, themelan.com, hbs.edu, venturebeat.com,
  innovationtraining.org, youtube.com, linkedin.com (×3), mi-3.com.au, medium.com, forbes.com.
- **ChatGPT**: **2 slots, 2 domains**: jobcannon.io, bcg.com. Neither is the paper.
- **Claude**: **3 slots, 3 unique domains**: thecrimson.com, linkedin.com, aibusiness.com. *(Corrected
  2026-07-27 20:44 against the record in `responses.jsonl`, which this line originally misdescribed as
  8 slots with thecrimson ×5. The ×5 profile belongs to counted run 4.)*
- **Gemini**: 7 slots, **6 unique domains**: innovationtraining.org (×2), mi-3.com.au, jobcannon.io,
  legaldive.com, hindustantimes.com, venturebeat.com.

`interested_share` is **left uncoded** for this cell. The protocol requires each domain to be
classified from its own homepage rather than from memory, and that pass has not been run. What is
already established: **bcg.com is the firm the experiment ran inside: its consultants were the subjects and it collected the data: publishing about that study**,
and it sells the advisory service the finding supports. `jobcannon.io` was checked and is a career-
assessment quiz site: under the strict rule it does **not** count as interested, because it does not
sell in the category the claim supports. Recording that is the point: the rule was written to bite
against the coder's convenience, and here it did.

### Three separate corruption mechanisms, all checkable

**1 · Version lag.** Peer review cut the quality result to 33.9% and 29.9% and removed the 40% claim
from the paper. Four of four returned 40%. Same mechanism as Q4, now replicated on a second paper in
a different discipline, which was the specific thing Q5 was run to test.

**2 · Sponsor substitution.** BCG's own publication says *"their performance was 23% worse than those
who didn't use the tool at all"* and *"a level of performance that was 40% higher."* The peer-reviewed
paper says **19 percentage points** and 33.9%/29.9%. Two of four systems reported BCG's **23%** as the
study's finding, and Perplexity named BCG's marketing page as the original source: verbatim, *"The
original **BCG** source is…"*.
That firm's promotional page is outranking, and outranking as *provenance*, the
peer-reviewed report of its own experiment.

**3 · False precision, acquired downstream.** `40.2%` appears in **neither** version of the paper : 
verified against both full texts. The preprint says "more than 40%" (Table 1: 42.5% and 38%); the
published paper says 33.9% and 29.9%. The figure `40.2%` appears on **jobcannon.io**, a career-quiz
site running a research-statistics page keyed to the working-paper number. ChatGPT returned "40.2%
higher in quality" and cited that page.

A number that was a rounded floor over two conditions became a decimal, on a page whose purpose is to
be found, and came back with a citation attached. **The decimal point is doing the work the evidence
used to do.**

### The one error that is not the assistant's

Claude returned *"40 percent of the trial group produced higher quality results"*: a magnitude
silently converted into a headcount, which is a different claim about a different quantity. It cited
the Harvard Crimson. The Crimson says, verbatim: *"Additionally, 40 percent of the trial group
produced higher quality results."*

Claude reproduced its source faithfully and cited it correctly. **The error is the newspaper's.** That
distinction has to survive into the essay, because it is the whole argument: this is not a story about
models hallucinating. It is a story about a retrieval layer that is accurate with respect to a corpus
that is wrong, and a paper two clicks away that nothing had reason to open.

(Claude also gave "750 consultants" against the true 758. That sentence is cited to a LinkedIn Pulse
post, not to the Crimson, which states 758 correctly. Whether the error originates in the LinkedIn
post or in the answer is **not established** and is recorded as open.)

---

# Appendix B: the surface arm, 2026-07-27

Added because the protocol's own limitation section demanded it: every run above went to an **API
endpoint**, and the essay's premise is about the tool people actually use. So both control questions
were put to the **consumer ChatGPT interface**, same prompt, same day.

The result changed the finding.

## Q5 · consumer surface

It cited **`pubsonline.informs.org/doi/10.1287/orsc.2025.21838`**: the version of record: alongside the
HBS working-paper PDF. *(The stored record has two citations, one of them the DOI. An earlier version of
this line said "four times", from a slot count in the scrape note that the retained citation list does
not support; corrected 2026-07-27 20:44.)* It named both versions correctly: *"Harvard Business School
Working Paper No. 24-013 (2023), later published in Organization Science (2026)."* It got the
outside-frontier result right in the correct units: **"19 percentage points"**, which is more precise
than the published abstract manages.

**And it still reported "more than 40% higher in quality": citing the *Organization Science* page for
it.** The published paper does not contain that claim.

## Q4 · consumer surface

Better still. It reported the preprint's 14% and 5,179, quoted the working paper's abstract verbatim,
named NBER WP 31161 as the original source, which for a question asking for the *original* source is
defensible, and then added, unprompted:

> *"A later peer-reviewed version published in The Quarterly Journal of Economics reports a **15%**
> average productivity increase using a slightly revised sample and analysis."*

That is close to a pass. It is the only response in the study to volunteer the revision.

## The confound, and how it was broken

The obvious objection: the API runs used `gpt-4.1-mini` while the consumer product serves something far
stronger. So the difference might be **model capability**, not **surface**.

Tested directly. Same question, same API, **`gpt-5.5`**: the strongest model available, with reasoning
enabled, at roughly **three times the cost per call**:

| | Cites the version of record? | Quality figure | Outside-frontier figure | Citations |
| --- | --- | --- | --- | --- |
| API · gpt-4.1-mini | no | **40.2%** (exists nowhere) | 23% (BCG's) | jobcannon.io, bcg.com |
| API · **gpt-5.5** | **no: never mentions a published version at all** | over 40% (preprint) | 23% (BCG's) | hbs.edu preprint PDF, bcg.com |
| Consumer ChatGPT | **yes: the DOI** | over 40% (preprint) | **19 percentage points** ✓ | informs.org, hbs.edu |

**The stronger, more expensive, reasoning-enabled model did not find the version of record. The
consumer product did.** Capability is not the explanation. The retrieval stack around the model is.

### The artefact worth keeping

`gpt-5.5` returns a visible reasoning summary. On the way to its answer it recorded:

> *"There's a PDF that likely contains relevant figures, like a 12.2% increase in tasks completed,
> 25.1% faster performance, and over 40% improved quality. **I can cite search14 even if it's not fully
> opened yet.**"*

The model states that it will attach a citation to a document it has not read, on the strength of what
it expects that document to contain. It then does exactly that, and the figure it expected to find is
the one peer review removed.

(Reported as what it is: a vendor-surfaced reasoning *summary*, not raw internal state. **And it is not
reproducible from this package**: it was transcribed into that record's `note` at collection time and
does not appear in any `answer_text`, so it stands on the same footing as the scrape notes, which is
weaker than the counted data. Treated as an observation, not as evidence.)

## What the surface arm changes

The finding is no longer "retrieval returns the preprint." It is two independent failures that have to
be separated:

1. **Citation**: largely fixed on the consumer surface. It found the version of record, named both
   versions, and linked the DOI.
2. **Content**, not fixed anywhere. Every surface, every model, every run reported a quality figure
   that peer review deleted.

And the combination is the point. On the API, a wrong number arrives cited to a career-quiz site, and a
careful reader can smell it. On the consumer surface, **the same wrong number arrives cited to
Organization Science with a DOI.** It looks impeccable. It is not.

**Better citation practice made the error harder to catch, not less likely to occur.**

## Limits of this arm, stated plainly

- **One system.** Only ChatGPT has a consumer-interface scrape available here. Nothing in this arm
  supports a claim about consumer Claude, Gemini or Perplexity.
- **One run each**, on one day, from one location (United States).
- The API/consumer comparison holds **model constant only in the gpt-5.5 row**. That row is what
  breaks the confound; the rest is descriptive.

---

---

# Appendix C: repeat runs

> **Superseded by Appendix F.** The run counts and rates in this appendix ("nine API runs", "2 of its
> 6 runs", "0 / 2", "2 / 2") are computed over runs whose verbatim text is **not** in
> `responses.jsonl`. They are kept as a record of what was observed and must not be quoted as data or
> mixed with the counted figures. Appendix F is the counted result.

Collected after the pilot cells, to test the protocol's own falsification condition:
*"Repeat runs disagree wildly. If the same model gives a different `source_class` across three
runs, the headline is variance, not provenance, and it must be reported as such."*

It does not disagree wildly. It disagrees in exactly one place, and that place is informative.

**Run count, stated honestly: two usable runs per system per question, not three.** The protocol asks
for three. Neither cell is complete and neither is described as complete.

---

## Q4 · "What did Brynjolfsson, Li and Raymond find…"

Version of record: *QJE* 140(2), 889–942 (2025): **15%**, 5,172 agents.
Preprint: NBER WP 31161 (April 2023): **14%**, 5,179 agents, 34% novices.

| System | Run A | Run B | Cites the version of record |
| --- | --- | --- | --- |
| Perplexity · sonar | 14%, 34% → NBER WP | 14% → NBER WP 31161 | **0 / 2** |
| Claude · sonnet-4-5 | **15% flagged, QJE 140(2), 889–942 cited** | **15% flagged, QJE 140(2), 889–942 cited** | **2 / 2** |
| Gemini · 2.5-flash | 14%, "35% or 38%", 5,179 → NBER WP | 14%, "34% to 35%", 5,179 → NBER WP | **0 / 2** |
| ChatGPT · gpt-4.1-mini | 14%, 34% → NBER, 3 slots all nber.org | 14%, 34% → NBER, 2 slots all nber.org | **0 / 2** |

**Claude passes cleanly, twice.** Verbatim, run B:

> *"It's worth noting that the published version of this paper in The Quarterly Journal of Economics
> (2025) reports a slightly higher figure of 15% on average, likely due to revisions and additional
> data analysis between the working paper and final publication."*

It reports **both** figures, names both versions, cites `academic.oup.com/qje/article/140/2/889/7990658`,
and explains the difference. That is the correct answer, and it is the only system that produced it.

This matters more than a uniform failure would. It establishes that **the version of record is
reachable**: no system can be excused on the grounds that the paywall made it impossible. One
retrieval stack found it, twice, unprompted. The others did not look.

Gemini's error is separate and worth keeping: "35% or 38%" (run A) and "34% to 35%" (run B) for the
novice effect. The preprint says **34%**. Neither 35% nor 38% is in either version. The figure is
drifting run to run.

---

## Q5 · "What did the BCG study find about consultants using GPT-4?"

Version of record: *Organization Science* 37(2), 403–423 (2026): quality **33.9% / 29.9%**, no 40% claim.
Preprint: HBS WP 24-013 (2023): **"more than 40% higher quality."**

| System | Run A | Run B | Cites the version of record |
| --- | --- | --- | --- |
| Perplexity · sonar | 40%, 23%: *"The original BCG source is How People Create and Destroy Value with Generative AI"* | 40%, 23%: *"The original source is BCG's own publication"* | **0 / 2** |
| ChatGPT · gpt-4.1-mini | **40.2%**, 23% → jobcannon.io + bcg.com | **40.2%**, 23% → BCG's PDF only, 3 slots 1 domain | **0 / 2** |
| Claude · sonnet-4-5 | "40 percent of the trial group" → the Harvard Crimson | **search failed**; answered from memory, 40% | **0 / 2** |
| Gemini · 2.5-flash | 40% → names WP, cites none of it | 40% → names WP 24-013, cites none of it | **0 / 2** |
| ChatGPT · **gpt-5.5** | 40%, 23% → HBS preprint PDF + bcg.com |: | **0 / 1** |

**Nine API runs. Zero cited the version of record. Every one carried the 40% figure peer review deleted.**

### Two things the repeats established that one run could not

**1 · The 40.2% is stable, and it has come loose from its source.** Run A cited `jobcannon.io`, which
does state 40.2%. Run B cited **BCG's own PDF**, which states **40%**. The same model produced the same
invented decimal twice and attributed it to two different documents, only one of which contains it.
A figure that began as "more than 40%" over two experimental conditions is now circulating with a
decimal place and no fixed origin.

**2 · Perplexity's sponsor substitution is not a slip.** Both runs name BCG's marketing publication as
the original source: run A's exact wording is *"The original **BCG** source is…"*, run B's is *"The
original source is BCG's own publication."* It is the stable behaviour of that stack on this question.

### A data-quality problem I am not hiding: now closed

Claude returned `web_search: false` on 2 of its runs across both questions, despite search being
requested, and one of those answers said so outright. **That sentence is not quotable from this
package:** only one `search_used: false` record survives in `responses.jsonl` (Q4 run 1) and its
`answer_text` is null, so the wording is reported here as an observation, not as data. Those runs
measure parametric memory, not retrieval, and are excluded from the counted cells. This looks
like intermittent tool provisioning at the endpoint rather than a property of the product.

**Re-run 2026-07-27 17:44, search confirmed active.** The cell now has two valid retrieval runs and
they agree closely:

- *"40 percent of the trial group produced higher quality results"*: again, cited to the Harvard
  Crimson. The propagation of the newspaper's category error is **stable across runs**, not a
  one-off.
- **New in this run, and worse:** *"the lowest performers had the biggest boost: 43% increase versus
  17% for top performers"*, cited to `aibusiness.com`. Those are the **preprint-only skill-split
  figures that peer review removed from the paper entirely** (`AI-X5`). Claude now returns *both*
  deleted artefacts: the 40% and the 43/17 split.
- *"750 Boston Consulting Group consultants"*: again cited to the same LinkedIn Pulse post, in both
  runs. Consistent attribution across two independent runs makes that post the likely origin of the
  error, though it has not been checked at source.
- Citations: `thecrimson.com`, `aibusiness.com`, `linkedin.com`, `axios.com`. **Four domains, none
  academic.** The trade headline it leans on is literally *"Harvard Study: GPT-4 Boosts Work Quality
  by Over 40%"*: the deleted figure encoded in a URL slug.

**Q5 now stands at 9 valid retrieval runs across five model configurations. Zero cited the version of
record. Every one carried a figure the published paper does not contain.**

---

## Why Q4 is passable and Q5 is not

Both papers have an open preprint and a paywalled version of record. Only one gets cited correctly.
The difference is not the paywall: it is **age and path**.

- *Generative AI at Work* has been the version of record since **May 2025**. The NBER landing page
  links forward to it, `academic.oup.com` is indexed, and three years of citations have accumulated
  pointing both ways.
- The *Organization Science* article appeared in **March 2026**: four months before these runs. Its
  preprint has had **three years** to gather links under a working-paper number that trade press,
  vendors and SEO pages all cite.

So the mechanism sharpens: it is not "paywalled papers lose." It is **"the version of record has to
out-age its own preprint before retrieval finds it"**, and during that window, which is years, the
superseded numbers are the answer everyone gets.

That window is where every recently-published finding lives.

---

## What still has to happen before any of this is published as a rate

- A third run on every cell. Two is not three.
- Claude's Q5 cell re-run with search actually enabled.
- The `interested_share` domain pass, from each domain's own homepage.
- The inbound-link count, preprint versus version of record, for both papers: the mechanism above is
  still an inference and the essay must not state it as measured until it is.


---

# Appendix D: the link graph, measured

The protocol committed to this before the essay could state a mechanism: *"'The link graph rewards
the free version' is currently the best explanation for eight of eight, and it is still an
explanation rather than a measurement."*

Now measured. Source: DataForSEO Backlinks, bulk page summary, live index, 2026-07-27.

---

## Q4 · Brynjolfsson, Li & Raymond

| Version | Referring domains | Backlinks | First seen |
| --- | --- | --- | --- |
| **Preprint**: `nber.org/papers/w31161` | **1,884** | 4,585 | 2023-05-01 |
| **Version of record**: `academic.oup.com/qje/article/140/2/889/7990658` | **631** | 1,218 | 2025-02-24 |

**Preprint advantage: 3.0× referring domains, 3.8× backlinks, a 21-month head start.**

## Q5 · Dell'Acqua et al.

| Version | Referring domains | Backlinks | First seen |
| --- | --- | --- | --- |
| **Preprint**: SSRN `4573321` | **1,058** | 2,141 | 2023-09-17 |
| **Preprint**: HBS landing page | **606** | 1,154 | 2023-09-28 |
| **Preprint**: HBS working-paper PDF | **511** | 1,337 | 2023-10-15 |
| **Version of record**: `pubsonline.informs.org/doi/10.1287/orsc.2025.21838` | **53** | 100 | 2026-03-14 |

**Preprint advantage: 20× referring domains against SSRN alone**, and the preprint exists on *three*
separately-linked surfaces, each of which individually outweighs the version of record by an order of
magnitude. The version of record is four months old.

---

## Why this settles the mechanism

The two questions differ in outcome, and the link data predicts the difference precisely.

| | Preprint : VoR referring domains | VoR age at run | Systems reaching the VoR |
| --- | --- | --- | --- |
| **Q4** | 3 : 1 | 17 months | **2 of 4**: Claude cites `academic.oup.com` on 3/3 counted runs; Gemini reports the published figures on 1/3 while citing no link to them |
| **Q5** | 20 : 1 | 4 months | **0 of 4**, and 0 of 13 runs |

*(Run counts updated 2026-07-27 20:44 against the closed counted dataset: Appendix F. The
referring-domain figures are unchanged; they are a single snapshot taken earlier the same day.)*

Where the version of record has had time to accumulate a third as many referring domains as its
preprint, two retrieval stacks reach it: one with a citation, one without. Where it has a twentieth,
none does.

**This is no longer "paywalled papers lose."** Both versions of record are paywalled; both preprints
are free. What separates them is **link mass and age**. The claim the essay can now make is narrow,
mechanistic and measured:

> A published paper has to out-age its own preprint in the link graph before retrieval will find it.
> Until it does, and that takes years: the numbers you get are the ones the authors already revised.

For Q5 that window is still open, four months after publication and three years after the preprint.
Every finding published this year is inside it.

## A detail that needs interpreting, not asserting

The *Organization Science* page reports **85 of its 100 backlinks as broken**, and `broken_pages: 1`.
On the other five URLs the field is absent. That is consistent with the publisher returning an error
or a block to crawlers, which would compound the link deficit: a version of record that is both
sparsely linked *and* hard to crawl. **It is also consistent with an artefact of one crawler's
access.** It is recorded here as observed and is not used to support any claim until it is checked
against a second source.

## Limits of this measurement

- **One commercial crawler's index, not the web.** DataForSEO's view is a proxy. It is not the index
  any of these four assistants retrieves from, and no vendor publishes theirs.
- **Referring domains are not retrieval rank.** The correlation across two questions is strong and
  directionally right; two points is not a model.
- **Single snapshot**, 2026-07-27. The whole point of the finding is that these numbers move.
- The comparison is **URL-level**. Both papers are reachable at other addresses: repositories,
  mirrors, author pages, that are not counted here. Counting more preprint surfaces would widen the
  gap, not narrow it, so the figures above are conservative in the direction of the argument, which
  is the safer direction to be wrong in.


## Where the study stands after two control cells

Both controls have now run once on all four systems. Both failed, on eight of eight system-question
cells, in the same direction. The four questions included because their provenance was expected to be
bad **have not been run at all**, and on the evidence so far they are the least informative
questions in the set.

The design was built so the controls could kill the thesis. They replaced it with a better one.

**Still to run** *(as written 2026-07-27 17:0x; superseded by Appendix F, which closed both control
cells at three counted runs and added Q1)*: Q4 and Q5 need their remaining repeat runs before either
enters the counted dataset: the cells above are single runs and stay labelled as pilots. Then the
non-control questions, three runs each.

**Added to the queue by these results, and not optional if the essay asserts the mechanism:** count
the inbound links to the preprint versus the version of record for both papers. "The link graph
rewards the free version" is currently the best explanation for eight of eight, and it is still an
explanation rather than a measurement.


---

# Appendix E: Q1, the first counted cell, 2026-07-27

Q1 is the **unrecoverable** class: a figure everyone quotes, with no primary document behind it. Ground
truth established 2026-07-27, before the run, from primary documents: *there is no original source.*
Reichheld & Sasser, *"Zero Defections"* (HBR Sept–Oct 1990), does not contain the claim; HBR 2014, the
document everyone cites, states it hedged and cites nobody; the trail runs backwards past Reichheld to
TARP and to Tom Peters, with no recoverable study anywhere on it.

**Prompt.** *"How much more expensive is acquiring a customer than retaining one? Give the figure and
cite the original source."*
**Design.** Four systems, web search on, **three runs each: twelve responses**, 20:07–20:11 UTC.
Cost $0.3514. Every response coded by hand against section 6; the sheet is `coding-sheet.md`.

The 13:57 pilot on a differently-worded version of this question is **not** in these counts.

---

## What came back

| System | Runs | Figure | Named as the original source | `source_class` | Slots → domains |
| --- | ---: | --- | --- | --- | --- |
| ChatGPT · gpt-4.1-mini | 3 | 5–25× | **nothing** in runs 2 and 3; three vendor pages in run 1 | tertiary ×3 | 3→3, 2→1, 2→1 |
| Claude · sonnet-4-5 | 3 | 5–25× | A Bain / Reichheld origin, asserted 3/3. *"**Bain & Company research conducted by Frederick Reichheld**"* verbatim in runs 1 and 3; run 2 reads *"research by **Frederick Reichheld at Bain & Company**"* | tertiary ×3 | 5→4, 6→4, 5→4 |
| Gemini · 2.5-flash | 3 | 5–25× | *"a 1990 Harvard Business Review article by Frederick Reichheld"* (runs 1, 3); *"research from Bain & Company"* (run 2) | tertiary ×3 | 10→6, 12→8, 18→7 |
| Perplexity · sonar | 3 | 5×, broadened to 5–25× | Hedged 3/3, never asserted. *"The original source most often **traced for** this rule of thumb"* (runs 1, 2); *"later articles **trace it to**"* (run 3) | tertiary ×3 | 19→17, 20→18, 19→17 |

**Twelve of twelve returned a figure. Twelve of twelve failed to produce a primary source: because
there is none.** Across the cell: **121 citation slots, 23 unique domain labels, zero primary
documents.** The URL-backed count is 20 because three Gemini-only labels have null URLs; destinations
remain unverified.

That much was expected, and on its own it is the weakest result in the study: a question selected for
having no source produces no source. **What the cell is actually worth is the three different ways the
systems handled the absence**, and where those handlings came from.

---

## Three failure modes, and the fourth behaviour that is not a failure

**1 · Assert an origin that does not exist. (Claude, 3/3.)** Asked for the original source, it names
one, in bold, without hedging: *"The original source for this widely referenced statistic is **Bain &
Company research conducted by Frederick Reichheld**."* No such Bain or Reichheld publication reports
this ratio.

**2 · Name a real document that does not contain the claim. (Gemini, 2/3.)** *"a 1990 Harvard Business
Review article by Frederick Reichheld, cited the cost as 5 times more."* The 1990 article is real,
findable, and co-authored with W. Earl Sasser Jr., whom the answer drops. It gives acquisition cost in
absolute terms: *"companies spend an average of $51 to recruit a customer"*, and states no ratio at
all. This is the harder failure to catch: the citation is checkable and the check passes until someone
opens the document.

**3 · Answer without provenance at all. (ChatGPT, 2/3.)** Runs 2 and 3 name no origin, attribute
nothing, and fill both citation slots from **one** growth-consulting blog. Under the coding scheme this
is the cell with **zero provenance errors**: earned by making no provenance claim. Non-compliance with
the instruction, scored as cleanliness. The scheme should say so out loud, and now does.

**4 · Report the attribution as an attribution. (Perplexity, 3/3.)** Run 1: *"The original source most
often **traced for** this rule of thumb is Frederick Reichheld's Bain & Company research."* Run 2:
*"The original source most often **traced for** the 5x claim is Bain & Company / Frederick Reichheld
research."* Run 3: *"later articles **trace it to** Bain & Company research associated with Frederick
Reichheld."* Three different sentences, one construction, and it is true every time. It describes what the literature does rather than asserting what the literature contains, and it
is the only construction in this cell that survives contact with the ground truth. It is also, across
this whole study, the only place a system's hedge is doing real epistemic work.

---

## Where the wrong provenance comes from: checked at source

This is the part that is not available from the answers alone, and it is what the cell contributes.

**Claude's sentence tracks `stealthagents.com`'s, near-verbatim.** That page: a virtual-assistant staffing
company running an SEO research page: states, verbatim:

> *"The foundational figure here comes from Bain and Company research by Frederick Reichheld: acquiring
> a new customer costs 5 to 25 times more than retaining an existing one."*

and

> *"The 5x floor is typical for transactional consumer businesses. The 25x ceiling shows up in complex
> B2B environments with long sales cycles and substantial onboarding costs."*

Claude's runs carry both, lightly reworded: *"The foundational figure comes from Bain and Company
research by Frederick Reichheld, which established this 5-25x range"*, and in run 2 the floor/ceiling
pair merged into one sentence: each cited to that page. Checked at source 2026-07-27. **The model
reproduced its source closely and cited it correctly. The attribution is the vendor page's.** That
is the Harvard Crimson case from Q5, replicated on a different question with a different kind of
document: a lead-generation page instead of a student newspaper.

**Gemini's sentence exists in the corpus too, but Gemini did not cite it.** `releva.ai` states: *"The
stat is from Frederick Reichheld's research at Bain & Company, first published in the 1990 Harvard
Business Review article 'Zero Defections: Quality Comes to Services.'"* That is Gemini's answer,
pre-written. **It is not in Gemini's citation set on any of the three runs**: it appears only in
Perplexity's, all three times. So the misattribution to a specific, real, checkable article is sitting
in the corpus and is demonstrably retrievable by these systems; **the route from it to Gemini's answer
is not established** and is coded the same way as the "Bain Capital" case below.

**And the pilot's most quotable error is in the corpus too.** The 13:57 pilot recorded Claude
attributing the figure to *"a 2003 source from Bain **Capital**"*: the private equity firm, not the
consultancy, and the pilot note called it *"the only hard factual error."* `hashtagpaid.com`, a
document cited by Perplexity in **all three** counted runs, says: *"The '5x rule' is a beloved stat,
tracing its roots to sources at **Bain Capital**"*, and dates its oldest concrete reference to *"a 2003
**Bain Capital** article."* Same firm confusion, same year.

**This does not establish that the pilot answer came from that page**: the pilot's citation list does
not include it, and the route is unknown. What it does establish is that *"the model invented it"* is
no longer the only available explanation, and the essay may not use that error as evidence of
fabrication. Recorded here because the pilot note currently reads the other way and needs amending.

---

## The debunkings, cited as support: replicated

`contra_as_support` = **yes** on all three Perplexity runs. Three of its cited documents dispute the
claim they are cited for, and each was checked at source on 2026-07-27:

- **`hashtagpaid.com`**: *"Does it still cost 5X more to acquire customers than to retain them in
  2023"*. Compares tracking the statistic to a game of telephone: *"one source traces its statistics to
  another source, which quotes another source."* Concludes *"no brand should assume 5x customer
  acquisition costs simply because they're in business."*
- **`churnmiser.com`**: the figure is *"usually traced loosely to Harvard Business Review and
  Bain-adjacent research, but rarely to a controlled study you could reproduce"*, and *"quoting it as a
  precise fact is how retention decks lose credibility with finance teams."*
- **`releva.ai`**: *"the specific number is dangerously outdated."*

The pilot found this same pattern on two *different* documents: the Ipsos chapter of *Loyalty Myths*
(*"Loyalty Myth No. 8: Debunked!"*) and a Forbes piece answering its own title question "no". **Five
documents, two independent runs of the design, same behaviour: the sceptical treatment is retrieved
and counted as evidence for the thing it is sceptical of.** The plausible mechanism remains
title-string matching: every one of these documents carries the claim as its headline, and it remains
a mechanism the study has not tested directly.

---

## The domain that all four systems share

Of 23 domains, exactly **one is cited by all four systems**: `artisangrowthstrategies.com`. Three are
cited by three or more: that page, `optimove.com`, `yotpo.com`.

The shared page was published 28 February 2025, modified 12 July 2026, belongs to a SaaS growth
consultancy, carries the ratio in its headline, and **names no source for it.** Checked at source.

So the one domain every stack reaches for on this question is a recently-refreshed commercial site with
the figure in the title and no provenance to give. That is the Q4/Q5 link-graph mechanism showing up
where there is no version of record to lose: retrieval converges on whatever is optimised to be found,
and here the thing optimised to be found is empty.

*Stated at the level the data supports:* this is a **domain**-level count. By URL-bearing slot count the
same domain also leads: 14 slots against `stealthagents.com`'s 9, but it is reached at three different
URLs across systems, and Gemini's citation URLs are not retained at all (see limits below). So "the
domain every stack reaches for" is supported; "the single most-cited *document*" is not.

---

## What this cell does not establish

- **`interested_share` is uncoded on all twelve rows.** The protocol requires each domain classified
  from its own homepage; that pass has not been run. So the essay may **not** state a rate, not "all
  sources are vendors", not "N of 23 sell retention software". The 23 domains are listed in
  `coding-sheet.md` and a reader can look. Saying more than that would be the motivated coding the rule
  was written to prevent.
- **No second coder.** Required by section 6, on a 20% sample, with disagreements reported.
  **Waived by the author on 2026-07-28**, on every cell in the study: no independent human pass
  will be run. A language-model pass over 9 of 36 rows was run instead and is published
  unreconciled (`second-coder-pass.md`). It does not satisfy the requirement and is not counted
  as satisfying it, so no inter-coder reliability figure is claimed anywhere in this study.
- **Three runs, one day, one location (US), API surface only.** No consumer-interface arm was run on
  Q1. The Q4/Q5 surface arm showed the two surfaces differ on citation behaviour, so nothing here
  generalises to `chatgpt.com` or `claude.ai`.
- **"No primary source exists" is a negative established by search**, not a proof. It is the strongest
  form available for this class of claim and it is stated as what it is.
- **Only one of the three corpus traces is a route.** `stealthagents.com` is cited in all three Claude
  answers that reproduce it, which is strong. `releva.ai` → Gemini and `hashtagpaid.com` → the pilot's
  "Bain Capital" are **matches with no established path**: neither page appears in the citation set of
  the answer it matches. Both are evidence that the corpus carries the error; neither is evidence of
  how it travelled.
- **Gemini's citation URLs were not retained.** Every Gemini citation resolves through a
  `vertexaisearch.cloud.google.com/grounding-api-redirect/` URL that exposes only a domain label; the
  opaque token is not stored in `responses.jsonl`. Domains are coded, destinations are not verified.
  A reader cannot check Gemini's citations from this dataset, and neither could the reader of the
  answer.

---

## One artefact of the harness, kept rather than fixed

The runner flags pre-registered discriminator strings by substring match. On this question `"5 times"`
fires inside `"5–25 times"`: it matches the tail of the *25*. Every ChatGPT run is therefore flagged
for a standalone 5× claim that only run 1 actually makes, and makes in words (*"up to five times
higher"*) that the matcher cannot see.

The strings were frozen before the counted runs and the matcher is unchanged, so the artefact stands in
the data. It is exactly why the protocol says a hit is *"a prompt to a human coder, never a coding
decision in itself"*, and the twelve rows in `coding-sheet.md` were coded from the full answers, not
from the flags. Left in place because a replication package that quietly repairs its instrument after
seeing the output is not one.

---

# Appendix F: the controls, closed against the dataset, 2026-07-27

Appendix C reported two runs per system and said so. Closing the gap exposed a second, worse problem:
**several of those runs existed only as prose in this protocol and had no record in `responses.jsonl`.**
A replication package whose headline rests on runs a reader cannot inspect is the failure this study
is about.

So the counted cell is now defined by the data file, not by the narrative:

> **A counted run is one with verbatim answer text, citations and a timestamp in `responses.jsonl`.**

A counted run is also an **API-surface** run on one of the four registered model configurations
(`gpt-4.1-mini`, `claude-sonnet-4-5`, `gemini-2.5-flash`, `sonar`). The consumer scrapes and the
`gpt-5.5` run share run numbers with counted runs and are excluded by that clause, not by convention.

Under that definition both controls now have **three counted runs per system, twelve per question**.
Collection windows: Q5 run 1 at **16:48–16:49 UTC**; everything else **20:07–20:44 UTC**:

| Question | Counted runs | Earlier runs, narrative only |
| --- | --- | --- |
| Q1 | runs 1, 2, 3 |: |
| Q4 | runs 3, 4, 5 | the four run-1 API pilot cells (`answer_text: null`) and the "run A / run B" tables in Appendix C |
| Q5 | runs 1, 3, 4 | the "run B" column in Appendix C |

The narrative runs are kept, because they are observations and deleting them would be worse. They are
**not** counted, quoted as verbatim data, or used to compute a rate.

---

## Q4: Brynjolfsson, Li & Raymond · 12 counted runs

Version of record: *QJE* 140(2), 889–942 (2025): **15%**, 5,172 agents.
Preprint: NBER WP 31161 (April 2023): **14%**, 5,179 agents, 34% novices.

| System | Run 3 | Run 4 | Run 5 | `version_returned` | Cites `academic.oup.com` |
| --- | --- | --- | --- | --- | --- |
| Claude · sonnet-4-5 | both versions, QJE 140(2), 889–942 | **names the QJE article as the original source**; also gives 5,179 | 14% + 15%, both versions named | **vor 3/3** | **3 / 3** |
| Gemini · 2.5-flash | 14%, 5,179 → NBER | 14%, plus **"13.8%"** | **15%, 5,172 → names the QJE** | vor 1/3 | **0 / 3** |
| ChatGPT · gpt-4.1-mini | 14%, 34% → NBER, 4 slots 1 domain | same, 4 slots 1 domain | same, 4 slots 1 domain | preprint 3/3 | 0 / 3 |
| Perplexity · sonar | 14% → NBER, 6 domains | 14%, 5,179 → NBER, 7 domains | 14% → NBER, 6 domains | preprint 3/3 | 0 / 3 |

**Claude is 3 for 3 in the counted data**, and every pass is inspectable: run 4 states the original
source as *"Erik Brynjolfsson, Danielle Li, Lindsey Raymond, 'Generative AI at Work,' The Quarterly
Journal of Economics, Volume 140, Issue 2, May 2025, Pages 889–942"* and cites `academic.oup.com`. The
paywall excuse is dead: one stack finds the version of record on every run.

It is not clean, though, and the mixed run is the interesting one. **Run 4 cites the QJE article and
then reports the preprint's 5,179 agents**: correct provenance, superseded number, in one answer. The
citation and the content come apart even when the citation is right, which is the same split the
consumer-surface arm found on Q5 from the other direction.

### The result that breaks the pattern, and how far it goes

**Gemini run 5 reports 15% and 5,172: both version-of-record figures, and names the QJE.** It is the
only non-Claude *counted API* run to get the published numbers. (The consumer ChatGPT scrape also
reports 15% and names the QJE: Appendix B, but that is a different surface and not a counted run.)

**And it cites no link to the paper.** Its six domains are `stanford.edu`, `arxiv.org`, `revartis.com`,
`itif.org`, `policycommons.net`, `richmondfed.org`. No `oup.com`, no `nber.org`. The right numbers
arrived with no path back to the document that contains them: which, against a prompt that says *give
the figure and cite the original source*, is still a failure of the instruction. It is a better failure
than the other eleven, and worth naming as its own mode: **correct content, absent provenance.**

Gemini's figure-drift also survives into the counted data in a new form. Run 4: *"Some sources specify
the average increase as 13.8%."* That number is in neither version, and the attribution is to unnamed
"some sources". Across three counted runs this system produced 14%, 13.8% and 15% for the same
quantity.

ChatGPT is the flattest cell in the study: three runs, near-identical text, **four citation slots and
one domain every time.**

## Q5: Dell'Acqua et al. · 12 counted runs, plus `gpt-5.5` and the consumer scrape

**Zero of twelve cited the version of record. Twelve of twelve carried a quality figure the published
paper does not contain.** With the `gpt-5.5` run that is thirteen API runs across five model
configurations, and the count is unchanged.

| System | Run 1 | Run 3 | Run 4 | Stable behaviour |
| --- | --- | --- | --- | --- |
| ChatGPT · gpt-4.1-mini | **40.2%**, 23% → jobcannon.io + bcg.com | **40.2%**, 23% → jobcannon.io + bcg.com | **40.2%**, 23% → **BCG's own PDF only** | the invented decimal, **3/3** |
| Perplexity · sonar | *"The original **BCG** source is How People Create and Destroy Value with Generative AI"* | *"The original source is BCG's report"* | *"The original source is BCG's publication"* | sponsor substitution, **3/3** |
| Claude · sonnet-4-5 | *"40 percent of the trial group"* → the Crimson | 40% (magnitude form) → aibusiness.com | *"40 percent of the trial group"* → the Crimson | the Crimson's category error, **2/3** |
| Gemini · 2.5-flash | 40% → names the WP, cites none of it | **40.2%** → jobcannon.io top slot | 40% → 33 slots, 10 domains | 40%, **2/3** |

### Three things the counted runs settle, one of which corrects an earlier reading

**1 · The fabricated `40.2%` is ChatGPT's, and it is stable.** Three counted runs, three times the
decimal. It appears in neither version of the paper; it appears on `jobcannon.io`, a career-assessment
quiz site. **In run 4 the only citation is BCG's own PDF: a document that says 40%.** That is the
second time this system has attached the decimal to a document which does not contain it. The figure
has come loose from any source.

**2 · It did *not* cross stacks. Correcting an earlier reading of run 3.** Gemini returned 40.2% on
run 3 and `jobcannon.io` was its top citation slot, which looked like propagation to a second system.
**Run 4 returned to 40%**: with `jobcannon.io` still in the citation set. One occurrence in three is
a fluctuation, not a transfer. The stronger claim was available and the next run removed it.

**3 · The Harvard Crimson's category error is Claude's majority behaviour, not an outlier.** *"40
percent of the trial group produced higher quality results"*: a magnitude silently converted into a
headcount: appears in runs 1 and 4, cited to `thecrimson.com`, which says exactly that. Run 3 gave the
magnitude form instead. **The newspaper's error propagates on two runs in three, and the model is
reproducing its source correctly each time.**

Also holding across all three counted runs: **"750 Boston Consulting Group consultants"** against the
true 758, cited every time to the same LinkedIn Pulse post. Three runs pointing at one post make it the
likely origin. It has not been checked at source, so that stays an inference.

And the repetition-as-breadth record: **Gemini run 4 returned 33 citation slots resolving to 10
domains**, none of them the paper.