Files

tegwick b9173b6569 IB-WP-0016-T02: chapter-aware chunking and stable IDs

Resolve chapter labels from EPUB nav entries (when present) and from the
first in-document h1/h2/h3 heading, parse roman-numeral and "Chapter N"
labels into numeric chapter indices, and generate stable IDs of the form
chapter-NN with -part-NNN suffix when a chapter exceeds max_words. The
chunker now operates on cleaned body text, distributes id="Page_*" page
anchors per part via inline markers extracted before splitting, and
supports a configurable overlap_words evidence window between adjacent
parts of the same chapter. Reclassify body sections whose chapter label
matches contents/transcriber-notes/license/colophon tokens so they leave
the body stream by default. Strip <head>...</head> from HTML body
extraction to stop the <title> tag from duplicating heading text in the
chunk markdown.

Real Lefevre EPUB now detects all 24 roman-numeral chapters with stable
chapter-NN IDs, distributes Page_N anchors across multi-part chapters,
and reclassifies Contents and Transcriber's Notes out of body
(role histogram body=67, cover=1, header=1, toc=1, notes=1, footer=2).
82 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-05-17 15:52:47 +02:00

4.5 KiB

Raw Blame History

Lefevre EPUB3 Validation

Date: 2026-05-14

Source

Local source file:

/mnt/c/Users/bernd.worsch/Downloads/LefevreEdwin-ReminiscencesOfAStockOperator.epub

The EPUB is Project Gutenberg edition 60979, EPUB package version 3.0. The OPF metadata identifies:

title: Reminiscences of a Stock Operator
creator: Edwin Lefevre
subjects: Speculation, New York Stock Exchange, Investments
rights: public domain in the USA

Current Infrastructure Result

The current generic generator can initialize a disposable infospace from the file and run non-provider metrics:

disposable root: /tmp/infospace-bench-lefevre-583mopy_/infospaces/reminiscences-stock-operator
source chunks: 155
entities: 0
relations: 0
evaluations: 0
stale status: false
metrics snapshot: 5978ece0

The source-only metrics were:

redundancy ratio: 0.9225806451612903
coverage ratio: 1.0
coherence components: 155.0
consistency cycles: 0.0
granularity entropy: -0.0

Findings

The EPUB intake works mechanically, but it is not ready for a serious full-book OpenRouter generation run.

EPUB spine order is visible in OEBPS/content.opf, but current intake reads XHTML files by archive-name sorting.
Current titles mostly collapse to the same long Gutenberg page title instead of chapter labels such as I, II, and III.
Current intake includes non-body material such as cover/header/footer/license candidates unless the caller manually filters after import.
generate plan is not yet a compact cost/risk plan for a long book; a full all-stage run would imply hundreds of provider calls.
Resume state is run-level enough for the small generic path, but a long ebook needs chunk-level retry, stale, and skip policy.
Cross-chunk entity deduplication and merge/review policy are needed before a full narrative book becomes a coherent infospace.

Desired Readiness Bar

Before building the real Lefevre infospace with OpenRouter, the CLI should be able to show:

book metadata and selected source sections
body-only chapter order
stable chapter/chunk IDs
estimated provider call count and token/cost budget
selected chapter or chunk filters for smoke runs
deterministic fixture acceptance on a small Lefevre-like subset
optional live one-chapter smoke run with explicit provider/model/cost caps

T01 Result (2026-05-17)

Spine-aware EPUB3 intake landed. Re-running the local Lefevre EPUB through normalize_source(...) now yields:

148 body chunks (default), down from the original 155 mixed chunks
Spine reading order: indices 0..27 in declared order, not archive-name sort
Full OPF metadata on every chunk's book_metadata: title, creator, language, subjects, rights, identifier, source_url, modified
Section roles classified across the 154 spine items: body=148, footer=4, cover=1, header=1
The four Gutenberg footer/license/notes sections and the *** START OF… header section are now excluded from generation input by default and available via include_non_body=True for inspection
The legacy zip-without-OPF fallback path is preserved for malformed EPUBs

The remaining gap is title collapse: all body sections still share the Project Gutenberg page title because chapter headings are not yet read from in-document <h1> content. That collapse is T02's scope (chapter-aware chunking and stable IDs from in-document headings).

T02 Result (2026-05-17)

Chapter-aware chunking and stable IDs landed. The same local Lefevre EPUB now produces:

67 body chunks (default max_words=800 collapses to 24 single-chunk chapters once max_words=2000)
All 24 roman-numeral chapters detected and assigned stable IDs chapter-01 .. chapter-24; multi-part chapters get chapter-NN-part-001, chapter-NN-part-002, ...
Chapter labels resolved from the EPUB nav doc (when present) and from the first in-document <h2>/<h1> heading
Project Gutenberg page-title collapse is gone: each chunk's title is the chapter label, not the shared book title
TOC body section ("Contents") reclassified to toc; transcriber's notes section reclassified to notes; section-role histogram is now body=67, cover=1, header=1, toc=1, notes=1, footer=2
Page anchors of the form id="Page_N" are preserved per chunk via the page_anchors provenance field (e.g. chapter-01 carries Page_1..Page_14 distributed across its three parts)
Optional overlap_words parameter supports evidence-window context between adjacent parts of the same chapter without duplicating headings

4.5 KiB Raw Blame History