A comparative study of three approaches to grounded text generation, using the Bible as a test corpus.

Pre-built, reusable tool pipelines (skills) will outperform both AI-generated ad-hoc tools and pure training-data recall on tasks requiring exact quotation from source documents — and the advantage will grow as the task scales (more documents, more cross-referencing). Specifically:

1. Skills will use fewer tokens because the AI never spends tokens writing code — it only calls existing scripts and composes from results.
2. Skills will be faster on repeated tasks because indexes are built once and reused, while ad-hoc tools must bootstrap every session.
3. Ad-hoc tools may produce better-fit tooling for novel or messy formats, since the AI adapts on the fly.
4. Training-data recall will work for well-known texts but will fail for private documents, lesser-known sources, or tasks requiring specific editions.
5. The gap between approaches will widen as specificity increases — more translations, stricter quoting requirements, and less common source material will expose the limits of memory-based approaches.

Write a short sermon about not being attached to digital things, as if they are possessions. Every Bible verse quoted must be word-for-word from the source text files. List all cited passages at the end.

Bible text files downloaded from openbible.com/texts.htm. Each file is tab-separated: Book Chapter:Verse\tVerse text. One line per verse.

• Trial 1: KJV only. Single translation, single sermon.
• Trial 2: Six translations (KJV, ASV, WEB, YLT, DRB, ERV). Each verse quoted side-by-side across all six.

Every quote from every sermon was checked against the source file (input/kjv.txt or the respective translation file) using exact-match text search. Each verse reference was searched in the source file and the returned text was compared character-by-character against the sermon quote. Bracket notation (e.g., [this], [that are]), punctuation, and capitalization were all compared.

Scoring rule established by the tester: dropping KJV translator brackets (e.g., [this] rendered as this) and starting a quote mid-verse do not cost points. Only wrong words, fabricated text, or phantom citations count as errors.

Approach 1 — Skills (Claude): 14 verses verified against input/kjv.txt. 8 were character-perfect exact matches. 6 had translator brackets silently removed. 1 (Luke 12:15) had its opening clause ("And he said unto them,") omitted. 1 phantom citation: Proverbs 23:5 was listed in the citations footer but never quoted in the sermon body. Score: 99/100.

Approach 2 — API (ChatGPT): 12 verses verified against input/kjv.txt. All 12 were character-perfect exact matches, including all translator brackets. No phantom citations. Score: 100/100.

Approach 3 — Web (ChatGPT): 9 verses verified against input/kjv.txt. 6 were character-perfect exact matches. 3 had translator brackets removed (1 Timothy 6:7 dropped [this] and [and it is]; 1 John 2:15 dropped [that are]; 1 John 2:16 dropped [is]). No phantom citations, no wrong words. Score: 100/100.

1. ChatGPT API — Best sermon overall. Natural flow, strong transitions, unexpected and fitting verse choices (Psalm 62:10 as a closer, Matthew 6:24 on serving two masters). Best single line across all three: "the heart can still become a vault, and the vault can become a god."
2. Claude/Skills — Good structure with clear numbered sections. The "moth and rust have modern equivalents" bridge connected ancient text to the digital theme effectively. Read more like a structured essay than a spoken sermon. Marred by the phantom Proverbs 23:5 citation.
3. ChatGPT Web — Accurate and well-written but the safest of the three. Stuck to the most obvious verses, didn't take risks. Prose between quotes leaned toward generic devotional language. Felt careful rather than convicted.

Approach 1 — Skills (Claude): 72 quotes verified (12 verses x 6 translations). All 72 were exact matches against the source files, including all brackets and punctuation. Score: 100/100.

Approach 2 — API (ChatGPT): 72 quotes verified (12 verses x 6 translations). All 72 were exact matches against the source files. Score: 100/100.

Approach 3 — Web (ChatGPT): Failed the task.

• Used the wrong translations: NIV, ESV, NASB, NLT, CSB instead of the requested ASV, WEB, YLT, DRB, ERV. Zero overlap with the prompt beyond KJV.
• Truncated quotes with ellipses instead of quoting in full.
• Used "same" and "similar wording" placeholders instead of actual text for multiple entries.
• Only cited 4 passages (vs 12 for the other approaches).
• Could not download files or verify anything.
• Task compliance score: 20/100. Accuracy: unscoreable.

Both file-based approaches chose 12 verses. 10 were identical; each made 2 unique choices:

1. Claude/Skills — Better as a delivered sermon. Drew specific observations from translation differences: KJV's unique use of "affection" vs "mind" in Colossians 3:2; the variation between "corrupt," "consume," and "disfigure" in Matthew 6:19; DRB naming "the snare of the devil" explicitly in 1 Timothy 6:9; YLT's use of "ostentation" and "manifest." Longer and more discursive.
2. ChatGPT API — Better as a reference document. Clean table format for side-by-side comparison. Punchy closing: "let the phone be a tool, not a throne; let money be a servant, not a root." Easier to scan but less commentary on what the translation differences reveal.
3. ChatGPT Web — Best pure writing of the three ("The cloud is not heaven. The server is not salvation. Your profile is not your identity."). But it answered a different question than what was asked.

In Trial 1, where the KJV — one of the most reproduced English texts in history — was the only source, all three approaches scored 99-100. Training-data recall worked because the KJV is saturated in every corpus.

In Trial 2, the moment we required five specific, less common translations (ASV, WEB, YLT, DRB, ERV), the memory-based approach couldn't even identify the correct translations, let alone quote them. It substituted the popular modern translations it had memorized (NIV, ESV, NASB, NLT, CSB) and resorted to placeholders where it wasn't confident.

The file-based approaches (Skills and API) both scored 100/100 on 72 quotes each. The gap went from nonexistent to total.

Adding 5 translations to the skills pipeline required no new code. The same three scripts (download-bible.js, process-bible.js, build-digest.js) were run 5 additional times. The query script already supported comma-separated translations. Total tokens spent on tooling for Trial 2: zero.

The API approach had to write or extend its parsing and comparison code for each session. This worked, but the cost is paid every time.

The counter-argument — that letting the AI build its own tools means it can adapt to unexpected input formats — is legitimate. For a well-understood tab-separated format, pre-built scripts are already optimal. But for a novel or messy document format, Approach 2 has the advantage of inspecting the data and writing purpose-built parsing logic. This flexibility comes at the cost of non-reusable, unreviewed code.

The ChatGPT web model's perfect KJV accuracy in Trial 1 was impressive but misleading. It was reciting from training data, not reading from a file. This distinction is invisible when the text is well-known, but becomes critical for:

• Private or proprietary documents (not in training data at all)
• Lesser-known translations or editions
• Recently published or modified texts
• Any task where the specific source file matters

This is the central finding. When the task was simple (one well-known translation, write a sermon), all three approaches were viable. When the task became specific (six particular translations, side-by-side comparison, exact quoting from downloaded files), only the file-based approaches could comply.

Extrapolating: for tasks involving private corpora, internal documents, uncommon formats, or auditability requirements, pure memory approaches are not viable at all.

For one-off tasks with public, well-known text: All three approaches work. Pick whichever is cheapest.

For repeated tasks or production workflows: Pre-built skills win. The upfront cost of writing good scripts pays for itself immediately on the second use. Indexes are persistent, queries are fast, and the AI's role is limited to composition — reducing the surface area for error.

For exploratory or novel tasks: AI-generated tools (Approach 2) are well-suited. The AI can inspect unfamiliar data, write purpose-built parsing logic, and iterate. But the tools are single-session unless the user saves them.

For tasks requiring specific source documents: File-based approaches (Skills and API) are not just better — they are the only viable options. Training-data recall cannot be trusted for exact quoting from specific editions, translations, or private documents.

For reliability you can stake a reputation on: Pre-built skills. You know exactly what the scripts do, you can version-control them, and the AI's only job is to compose from deterministic query results. No generated code to review, no memorization to trust, no session-to-session variation.

Appraisal 2 (appraisal2.md) stated that Approach 1 produced "10 verses x 6 translations = 60 quotes." The actual count, verified by searching the sermon file, is 12 verses x 6 translations = 72 quotes — the same as Approach 2. Both approaches produced 72 total quotes and scored 100/100. The verse selections differed on 2 of 12 choices (Claude used 1 Timothy 6:9 and Matthew 16:26; ChatGPT used Matthew 6:24 and 1 Timothy 6:10).

The bible pipeline used in this study follows a general pattern that works for any structured or semi-structured document. Here is a step-by-step guide for creating your own.

Before writing any code, examine the raw file. Identify:

• Record boundaries: What separates one unit from the next? (Newlines, blank lines, headers, delimiters)
• Fields within records: What structure exists inside each unit? (Tab-separated, JSON, XML, fixed-width, heading + body)
• Hierarchy: Are there logical groupings? (Book > Chapter > Verse, Section > Subsection > Paragraph, File > Function > Line)
• Reference scheme: How do you address a specific piece? (Book Chapter:Verse, page number, section ID, line number)

For the Bible, the format was simple: Book Chapter:Verse\tVerse text, one line per verse, with a natural hierarchy of Book > Chapter > Verse.

A script that fetches or copies the raw document into a standard location. Keep it simple:

• Accept the source as a parameter (URL, file path, API endpoint)
• Save to a consistent input directory (e.g., input/)
• Name the output predictably (e.g., input/<version>.txt, input/<document-id>.json)
• Handle errors (bad URLs, missing files, network failures)
• Use no dependencies if possible — built-in HTTP and filesystem modules are enough for most cases

This is the core of the pipeline. It does three things:

a) Parse — Read the raw file and extract structured records. Each record should have at minimum a reference (how to find it again) and content (the actual text).

b) Chunk — Split records into files by logical grouping. One file per chapter, one file per section, one file per page — whatever makes sense for the document. This keeps individual files small enough for the AI to read without consuming excessive tokens.

Small files are critical. A 4.5 MB Bible becomes ~1,200 chapter files averaging 3-4 KB each. The AI only reads the specific chapter files it needs, not the whole document.

c) Index — Build lookup tables that map terms to references. For each category of interest (people, places, concepts, entities, keywords), scan every record for matches and store which references contain each term.

The index structure should be:

{
  "term": ["Reference 1", "Reference 2", "Reference 3"]
}

Store indexes per chunk (per book, per section) so searches can be scoped. The term lists can be static JSON files maintained alongside the scripts, or generated dynamically using NLP.

Digests are extractive summaries — no generation, no interpretation. For each logical section, store:

• The reference range (first and last reference)
• The first and last record text (as representative samples)
• Record count
• Any metadata (word count, key terms, date range)

This gives the AI a table of contents it can scan without reading every file. It can then decide which sections to read in full.

The query script is the AI's interface to the indexed data. It should:

• Accept terms and/or direct references as arguments
• Search the indexes for term matches, collecting all matching references
• Look up the actual text for each reference from the chunk files
• Return structured JSON: [{ ref, text }] for single-source queries, [{ ref, texts: { source1: "...", source2: "..." } }] for multi-source comparison
• Sort results in document order
• Cap output size (e.g., max 80 results) to avoid overwhelming the AI's context

The AI should never need to read raw files directly. The query script is the single point of contact between the AI and the data.

The skill file (e.g., SKILL.md) tells the AI when and how to use each script. It should include:

• When to use: What kinds of user requests trigger this skill
• Pipeline order: Which scripts depend on which (download before process, process before query)
• Exact commands: Copy-pasteable command lines with parameter descriptions
• Output format: What the AI will get back and how to use it
• Rules: Exac