generated from coulomb/repo-seed
tegwick
b9173b6569
Resolve chapter labels from EPUB nav entries (when present) and from the first in-document h1/h2/h3 heading, parse roman-numeral and "Chapter N" labels into numeric chapter indices, and generate stable IDs of the form chapter-NN with -part-NNN suffix when a chapter exceeds max_words. The chunker now operates on cleaned body text, distributes id="Page_*" page anchors per part via inline markers extracted before splitting, and supports a configurable overlap_words evidence window between adjacent parts of the same chapter. Reclassify body sections whose chapter label matches contents/transcriber-notes/license/colophon tokens so they leave the body stream by default. Strip <head>...</head> from HTML body extraction to stop the <title> tag from duplicating heading text in the chunk markdown. Real Lefevre EPUB now detects all 24 roman-numeral chapters with stable chapter-NN IDs, distributes Page_N anchors across multi-part chapters, and reclassifies Contents and Transcriber's Notes out of body (role histogram body=67, cover=1, header=1, toc=1, notes=1, footer=2). 82 tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
115 lines
4.5 KiB
Markdown
115 lines
4.5 KiB
Markdown
# Lefevre EPUB3 Validation
|
|
|
|
Date: 2026-05-14
|
|
|
|
## Source
|
|
|
|
Local source file:
|
|
|
|
`/mnt/c/Users/bernd.worsch/Downloads/LefevreEdwin-ReminiscencesOfAStockOperator.epub`
|
|
|
|
The EPUB is Project Gutenberg edition 60979, EPUB package version 3.0. The OPF
|
|
metadata identifies:
|
|
|
|
- title: `Reminiscences of a Stock Operator`
|
|
- creator: `Edwin Lefevre`
|
|
- subjects: `Speculation`, `New York Stock Exchange`, `Investments`
|
|
- rights: public domain in the USA
|
|
|
|
## Current Infrastructure Result
|
|
|
|
The current generic generator can initialize a disposable infospace from the
|
|
file and run non-provider metrics:
|
|
|
|
- disposable root:
|
|
`/tmp/infospace-bench-lefevre-583mopy_/infospaces/reminiscences-stock-operator`
|
|
- source chunks: 155
|
|
- entities: 0
|
|
- relations: 0
|
|
- evaluations: 0
|
|
- stale status: false
|
|
- metrics snapshot: `5978ece0`
|
|
|
|
The source-only metrics were:
|
|
|
|
- redundancy ratio: `0.9225806451612903`
|
|
- coverage ratio: `1.0`
|
|
- coherence components: `155.0`
|
|
- consistency cycles: `0.0`
|
|
- granularity entropy: `-0.0`
|
|
|
|
## Findings
|
|
|
|
The EPUB intake works mechanically, but it is not ready for a serious full-book
|
|
OpenRouter generation run.
|
|
|
|
- EPUB spine order is visible in `OEBPS/content.opf`, but current intake reads
|
|
XHTML files by archive-name sorting.
|
|
- Current titles mostly collapse to the same long Gutenberg page title instead
|
|
of chapter labels such as `I`, `II`, and `III`.
|
|
- Current intake includes non-body material such as cover/header/footer/license
|
|
candidates unless the caller manually filters after import.
|
|
- `generate plan` is not yet a compact cost/risk plan for a long book; a full
|
|
all-stage run would imply hundreds of provider calls.
|
|
- Resume state is run-level enough for the small generic path, but a long ebook
|
|
needs chunk-level retry, stale, and skip policy.
|
|
- Cross-chunk entity deduplication and merge/review policy are needed before a
|
|
full narrative book becomes a coherent infospace.
|
|
|
|
## Desired Readiness Bar
|
|
|
|
Before building the real Lefevre infospace with OpenRouter, the CLI should be
|
|
able to show:
|
|
|
|
- book metadata and selected source sections
|
|
- body-only chapter order
|
|
- stable chapter/chunk IDs
|
|
- estimated provider call count and token/cost budget
|
|
- selected chapter or chunk filters for smoke runs
|
|
- deterministic fixture acceptance on a small Lefevre-like subset
|
|
- optional live one-chapter smoke run with explicit provider/model/cost caps
|
|
|
|
## T01 Result (2026-05-17)
|
|
|
|
Spine-aware EPUB3 intake landed. Re-running the local Lefevre EPUB through
|
|
`normalize_source(...)` now yields:
|
|
|
|
- 148 body chunks (default), down from the original 155 mixed chunks
|
|
- Spine reading order: indices 0..27 in declared order, not archive-name sort
|
|
- Full OPF metadata on every chunk's `book_metadata`:
|
|
title, creator, language, subjects, rights, identifier, source_url, modified
|
|
- Section roles classified across the 154 spine items:
|
|
`body=148`, `footer=4`, `cover=1`, `header=1`
|
|
- The four Gutenberg footer/license/notes sections and the `*** START OF…`
|
|
header section are now excluded from generation input by default and
|
|
available via `include_non_body=True` for inspection
|
|
- The legacy zip-without-OPF fallback path is preserved for malformed EPUBs
|
|
|
|
The remaining gap is title collapse: all body sections still share the
|
|
Project Gutenberg page title because chapter headings are not yet read from
|
|
in-document `<h1>` content. That collapse is T02's scope (chapter-aware
|
|
chunking and stable IDs from in-document headings).
|
|
|
|
## T02 Result (2026-05-17)
|
|
|
|
Chapter-aware chunking and stable IDs landed. The same local Lefevre EPUB
|
|
now produces:
|
|
|
|
- 67 body chunks (default `max_words=800` collapses to 24 single-chunk
|
|
chapters once `max_words=2000`)
|
|
- All 24 roman-numeral chapters detected and assigned stable IDs
|
|
`chapter-01` .. `chapter-24`; multi-part chapters get
|
|
`chapter-NN-part-001`, `chapter-NN-part-002`, ...
|
|
- Chapter labels resolved from the EPUB nav doc (when present) and from
|
|
the first in-document `<h2>`/`<h1>` heading
|
|
- Project Gutenberg page-title collapse is gone: each chunk's title is the
|
|
chapter label, not the shared book title
|
|
- TOC body section ("Contents") reclassified to `toc`; transcriber's notes
|
|
section reclassified to `notes`; section-role histogram is now
|
|
`body=67, cover=1, header=1, toc=1, notes=1, footer=2`
|
|
- Page anchors of the form `id="Page_N"` are preserved per chunk via the
|
|
`page_anchors` provenance field (e.g. chapter-01 carries
|
|
`Page_1..Page_14` distributed across its three parts)
|
|
- Optional `overlap_words` parameter supports evidence-window context
|
|
between adjacent parts of the same chapter without duplicating headings
|