scope digest workplan

workplans/STATE-WP-0044-recently-on-scope-digest.md

@@ -0,0 +1,246 @@
+---
+id: STATE-WP-0044
+type: workplan
+title: "RecentlyOnScope Domain Digest"
+domain: custodian
+repo: state-hub
+status: ready
+owner: codex
+topic_slug: custodian
+created: "2026-05-22"
+updated: "2026-05-22"
+---
+
+# RecentlyOnScope Domain Digest
+
+## Summary
+
+Add a file-backed RecentlyOnScope digest that answers: "What happened in this
+domain recently?" The digest is generated as Markdown from a deterministic
+Markdown template, with a time range parameter defaulting to one hour. Generated
+reports are collected in a State Hub report directory and are browsable from a
+new Domains navigation subentry.
+
+This should reuse MarkiTect template-rendering capabilities instead of adding a
+new ad hoc templating language. The local tool surface currently provides
+`markitect template render` and the Python API
+`markitect_tool.template.render_template`.
+
+## Current Findings
+
+- Domains are first-class records with topics and repos. Domain-scoped work can
+  be resolved through `Domain -> Topic -> Workstream -> Task`.
+- `GET /progress/` already supports `topic_id`, `workstream_id`, `task_id`,
+  `event_type`, and `since` filters, making progress events the first digest
+  source.
+- Decisions, workstreams, and tasks have timestamp fields, but their existing
+  list endpoints do not expose a domain/time-window digest view.
+- Agent messages are not currently domain-linked. They should only enter the
+  first digest when they can be tied to a topic, workstream, task, or explicit
+  domain reference.
+- The dashboard currently exposes `Domains` as a top-level page in
+  `dashboard/observablehq.config.js`. A subentry will require turning it into a
+  collapsible navigation group or adding a nested Domains section.
+- No State Hub report directory exists yet. The work should create a clear
+  generated-artifact location, proposed as `reports/recently-on-scope/` inside
+  this repo or the configured runtime data directory.
+
+## Out of Scope
+
+- LLM summarization of the digest. This work is deterministic collection and
+  Markdown rendering.
+- Scheduled/background generation. This work provides on-demand generation and
+  browsing hooks only.
+- Replacing the existing progress, task, decision, or workstream APIs.
+- Backfilling historic reports before the feature exists.
+
+## T01 - Define Digest Contract and Report Layout
+
+```task
+id: STATE-WP-0044-T01
+status: todo
+priority: high
+```
+
+Define the input, output, and file layout for a RecentlyOnScope digest before
+implementation.
+
+Implementation notes:
+
+- Required input: `domain_slug`.
+- Optional input: `range`, defaulting to `1h`.
+- Optional exact-window inputs: `since` and `until`; when supplied, they should
+  override or constrain `range` predictably.
+- Accept simple duration values such as `15m`, `1h`, `6h`, and `1d`; reject
+  ambiguous values with a 422 response or CLI error.
+- Normalize all comparisons and generated filenames to UTC.
+- Store the tracked template at
+  `templates/recently-on-scope/domain-digest.md`.
+- Store generated reports under a configurable report root, defaulting to
+  `reports/recently-on-scope/<domain_slug>/`.
+- Name reports with enough information to sort and de-duplicate, for example
+  `20260522T120000Z--1h.md` or
+  `20260522T110000Z--20260522T120000Z.md`.
+- Include frontmatter in each generated Markdown report with `domain_slug`,
+  `range`, `since`, `until`, `generated_at`, `template_version`, and source
+  counts.
+
+Done when the report contract is documented and has tests for duration parsing,
+default range behavior, and output path generation.
+
+## T02 - Build Domain Activity Collector
+
+```task
+id: STATE-WP-0044-T02
+status: todo
+priority: high
+```
+
+Create a domain-scoped collector that turns recent State Hub records into a
+stable data object for template rendering.
+
+Implementation notes:
+
+- Add a focused service module, for example
+  `api/services/recently_on_scope.py`.
+- Resolve `domain_slug` to the domain id, associated topic ids, active repos,
+  workstreams, and tasks.
+- Collect `ProgressEvent` rows for the domain topic ids and workstream/task ids
+  in the requested time window.
+- Collect decisions created, updated, or resolved in the time window for domain
+  topics and workstreams.
+- Collect workstreams and tasks updated in the time window, including status
+  changes where that can be inferred from current records or progress events.
+- Include open human-intervention items in a separate "still needs attention"
+  section when they belong to the domain, even if they were created before the
+  requested window.
+- Keep output deterministic: sort by timestamp descending within sections, and
+  include stable ids/links where available.
+- Do not infer domain ownership for messages unless a message is explicitly
+  linked to a domain, topic, workstream, task, or digest-generation request.
+
+Done when the collector returns a deterministic plain-Python data structure
+covering progress, decisions, workstreams, tasks, repos, and attention items for
+one domain and time window.
+
+## T03 - Reuse MarkiTect for Markdown Rendering
+
+```task
+id: STATE-WP-0044-T03
+status: todo
+priority: high
+```
+
+Render the digest with MarkiTect rather than custom string substitution.
+
+Implementation notes:
+
+- Add a thin adapter, for example `api/services/markitect_templates.py`, that
+  imports `markitect_tool.template.render_template` when available.
+- Keep a subprocess fallback to `markitect template render` for environments
+  where the CLI is installed but the Python package is not importable.
+- Fail clearly if neither MarkiTect path is available; do not silently switch to
+  a new template syntax.
+- Validate the template before generation by checking required variables.
+- Define a template data contract with keys such as `domain`, `window`,
+  `generated_at`, `progress_events`, `decisions`, `workstreams`, `tasks`,
+  `repos`, and `attention_items`.
+- Keep the template Markdown readable on its own, with no dashboard-specific
+  markup.
+- Persist the rendered Markdown atomically so partial reports are not listed by
+  the dashboard.
+
+Done when a report can be rendered from the template using MarkiTect in tests
+and written to the configured report directory.
+
+## T04 - Provide Generation and Listing Mechanisms
+
+```task
+id: STATE-WP-0044-T04
+status: todo
+priority: high
+```
+
+Expose a practical mechanism to generate, list, and read RecentlyOnScope
+reports.
+
+Implementation notes:
+
+- Add an API route group under Domains, for example:
+  - `POST /domains/{slug}/recently-on-scope` to generate a report.
+  - `GET /domains/{slug}/recently-on-scope` to list generated reports.
+  - `GET /domains/{slug}/recently-on-scope/{report_id}` to return Markdown.
+- Return report metadata from generation and list calls: id, domain, range,
+  since, until, generated_at, path, and source counts.
+- Return Markdown with `text/markdown` where appropriate.
+- Add a CLI script or make target for operator use, for example
+  `python -m scripts.recently_on_scope --domain custodian --range 1h`.
+- Make the report root configurable with an environment variable such as
+  `STATE_HUB_REPORT_DIR`, while preserving the repo-local default.
+- Avoid requiring dashboard access for generation; the CLI/API should be usable
+  by agents and operators.
+
+Done when operators can generate a digest from the command line or API, list
+existing reports, and read the Markdown contents.
+
+## T05 - Add Domains Navigation and Dashboard View
+
+```task
+id: STATE-WP-0044-T05
+status: todo
+priority: medium
+```
+
+Make the reports accessible from a subentry under Domains in the State Hub
+dashboard navigation.
+
+Implementation notes:
+
+- Update `dashboard/observablehq.config.js` so `Domains` can expose a child page
+  without hiding the existing `/domains` overview.
+- Add a page such as `dashboard/src/domains/recently-on-scope.md`.
+- Provide domain selection, a time-range control defaulting to `1h`, and a
+  generate action backed by the new API.
+- Show the generated report list grouped by domain and sorted by `generated_at`
+  descending.
+- Render or preview Markdown safely in the dashboard, or link to the raw
+  `text/markdown` endpoint when full rendering is out of scope for the first
+  pass.
+- Keep the view operational and compact: no landing page, no marketing copy,
+  and no generated-output text that overlaps on small screens.
+- Add reference docs at `dashboard/src/docs/domains.md` or a dedicated docs page
+  explaining where reports are stored and what the range parameter means.
+
+Done when the dashboard navigation has a Domains subentry for
+RecentlyOnScope, reports can be generated/browsed from it, and raw Markdown is
+accessible for each entry.
+
+## T06 - Verification, Docs, and Operational Notes
+
+```task
+id: STATE-WP-0044-T06
+status: todo
+priority: medium
+```
+
+Cover the feature with focused tests and operator documentation.
+
+Implementation notes:
+
+- Add unit tests for duration parsing, collector filtering, MarkiTect adapter
+  behavior, report naming, and report listing.
+- Add router tests for generation, list, missing domain, invalid range, and
+  missing MarkiTect failure modes.
+- Add dashboard data/API smoke tests where existing dashboard test patterns make
+  this practical.
+- Document the report directory, retention expectations, and whether generated
+  reports are committed or treated as runtime artifacts.
+- If generated reports should stay out of Git, add a targeted `.gitignore`
+  entry while keeping the template tracked.
+- Verify with `python3 -m pytest` and dashboard build/test commands already used
+  by this repo.
+- After the workplan file is synced, run
+  `make fix-consistency REPO=state-hub` from `~/state-hub`.
+
+Done when tests pass, docs explain the operator workflow, and generated reports
+have a clear retention and Git-tracking policy.