generated from coulomb/repo-seed
277 lines
10 KiB
Markdown
277 lines
10 KiB
Markdown
---
|
|
id: STATE-WP-0056
|
|
type: workplan
|
|
title: "Dashboard Loading Robustness and Efficiency"
|
|
domain: custodian
|
|
repo: state-hub
|
|
status: finished
|
|
owner: codex
|
|
topic_slug: custodian
|
|
created: "2026-06-05"
|
|
updated: "2026-06-05"
|
|
state_hub_workstream_id: "28f9569c-937b-4b79-b46c-f6b1f83c09c3"
|
|
---
|
|
|
|
# Dashboard Loading Robustness and Efficiency
|
|
|
|
## Summary
|
|
|
|
Make the State Hub dashboard overview page faster and more resilient under
|
|
normal polling. The current overview performs a broad concurrent fan-out of
|
|
full-list API calls and treats most request failures as whole-page failures.
|
|
This can surface frequent `Dashboard data load failed: The operation was
|
|
aborted.` warnings when one call crosses the frontend timeout, even if the API
|
|
eventually returns successfully.
|
|
|
|
This work should reduce request count, payload size, and backend contention;
|
|
preserve useful last-known data during partial failures; and give operators
|
|
clearer diagnostics when a section is stale or unavailable.
|
|
|
|
## Current Findings
|
|
|
|
Inspection on 2026-06-05 found:
|
|
|
|
- `dashboard/src/index.md` loads overview data with one eight-request
|
|
`Promise.all` batch.
|
|
- `dashboard/src/components/config.js` aborts most `apiFetch` calls after
|
|
`12_000` ms.
|
|
- A dashboard-style concurrent timing run produced several calls at or above the
|
|
default timeout: `/sbom/snapshots/`, `/repos/`, and `/workplans/index`.
|
|
- The same endpoints can be much faster when called alone, which points to
|
|
contention and over-fetching rather than one permanently slow endpoint.
|
|
- The overview calls `/tasks/?limit=2000`, but the tasks API currently ignores
|
|
`limit` and returns every task. In the observed run that response was roughly
|
|
2.1 MB just to compute per-workplan task counts.
|
|
- `/state/summary` has a short in-process cache, but a cache miss still runs a
|
|
large amount of sequential database and Python-side aggregation work.
|
|
- `/workplans/index` scans active repository workplan files and parses
|
|
frontmatter. It is cached, but concurrent dashboard loads can still wait on
|
|
the same expensive rebuild pattern.
|
|
- Several API routes set cache headers, but the shared dashboard fetch helper
|
|
forces `cache: "no-store"` for every request.
|
|
|
|
## Out of Scope
|
|
|
|
- Replacing Observable Framework.
|
|
- Redesigning the dashboard information architecture.
|
|
- Adding authentication, authorization, or multi-user session handling.
|
|
- Changing workplan file conventions.
|
|
- Moving State Hub to a different database or deployment substrate.
|
|
|
|
## T01 — Add Focused Dashboard Load Instrumentation
|
|
|
|
```task
|
|
id: STATE-WP-0056-T01
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "e5208053-0db1-4842-a221-c5289422677a"
|
|
```
|
|
|
|
Add enough timing and error visibility to confirm which overview calls are slow,
|
|
aborted, or oversized during normal use.
|
|
|
|
Implementation notes:
|
|
|
|
- Add lightweight server-side timing logs or response headers for overview-hot
|
|
endpoints: `/state/summary`, `/workplans/`, `/tasks/`, `/topics/`, `/repos/`,
|
|
`/sbom/snapshots/`, `/progress/`, and `/workplans/index`.
|
|
- Include request path, status, elapsed time, response size when practical, and
|
|
whether a cached result was used.
|
|
- Keep instrumentation local and low-noise; avoid logging full payloads or
|
|
secrets.
|
|
- Add a small dashboard diagnostic surface or console logging that distinguishes
|
|
timeout aborts from HTTP errors and network failures.
|
|
- Capture before/after timing notes in this workplan or a progress event.
|
|
|
|
Done when a normal dashboard refresh can be diagnosed without manually timing
|
|
each endpoint from a shell.
|
|
|
|
## T02 — Make Overview Polling Partially Resilient
|
|
|
|
```task
|
|
id: STATE-WP-0056-T02
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "2cdd960d-ba86-48d1-a7c6-e83671cd0e69"
|
|
```
|
|
|
|
Change the overview data loader so one slow or failed secondary request does
|
|
not mark the whole dashboard as failed.
|
|
|
|
Implementation notes:
|
|
|
|
- Replace fail-fast `Promise.all` behavior in `dashboard/src/index.md` with a
|
|
per-resource result model, for example `Promise.allSettled`.
|
|
- Keep last-known-good data for each section while a refresh is degraded.
|
|
- Treat optional resources such as SBOM snapshots, registration milestones, and
|
|
workplan file metadata independently from core summary/workplan status data.
|
|
- Display section-level stale/error indicators instead of one global warning
|
|
whenever possible.
|
|
- Keep exponential backoff for repeated failures, but do not discard usable
|
|
data just because one request timed out.
|
|
- Make abort errors user-readable, for example "timed out after 12s" instead of
|
|
only "The operation was aborted."
|
|
|
|
Done when an SBOM, repo-list, or workplan-index timeout leaves the rest of the
|
|
overview usable and visibly stale rather than failed.
|
|
|
|
## T03 — Respect Pagination and Add Task Count Aggregates
|
|
|
|
```task
|
|
id: STATE-WP-0056-T03
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "78484226-9ccc-460c-a2b3-750b3204caa3"
|
|
```
|
|
|
|
Stop returning all tasks for overview count calculations.
|
|
|
|
Implementation notes:
|
|
|
|
- Add `limit` and `offset` support to `GET /tasks/`, preserving existing filter
|
|
behavior and sensible limits.
|
|
- Add a lightweight aggregate endpoint for task counts by workplan and status,
|
|
for example `GET /tasks/counts?group_by=workstream,status`, or add an
|
|
overview-specific aggregate route.
|
|
- Prefer SQL `GROUP BY` over transferring every task to the browser.
|
|
- Update `dashboard/src/index.md`, `dashboard/src/tasks.md`,
|
|
`dashboard/src/interventions.md`, and workplan detail pages as needed so list
|
|
views still receive the rows they need.
|
|
- Add tests for pagination compatibility and aggregate counts.
|
|
|
|
Done when the overview no longer fetches the full task table to draw the
|
|
workplan chart.
|
|
|
|
## T04 — Build a Lightweight Overview Read Endpoint
|
|
|
|
```task
|
|
id: STATE-WP-0056-T04
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "2cf47a12-e8aa-49ca-963c-1f0d2933c344"
|
|
```
|
|
|
|
Create a dashboard-specific read model that returns exactly the data needed by
|
|
the overview page in one bounded response.
|
|
|
|
Implementation notes:
|
|
|
|
- Add an endpoint such as `GET /state/overview` or
|
|
`GET /state/dashboard-overview`.
|
|
- Include summary totals, recent progress needed by the page, blocking decision
|
|
counts, waiting-task counts, SBOM snapshot totals, registration milestones,
|
|
and workplan chart rows with repo/domain labels and task counts.
|
|
- Keep response fields stable and documented in dashboard reference docs.
|
|
- Reuse existing summary helpers where they are efficient, but avoid serializing
|
|
large full-list payloads that the overview does not display directly.
|
|
- Add cache headers and a short in-process cache with explicit invalidation
|
|
rules where appropriate.
|
|
- Update `dashboard/src/index.md` to prefer this endpoint and remove redundant
|
|
overview-only fetches.
|
|
|
|
Done when the overview's steady-state refresh is one bounded API call plus only
|
|
truly interactive secondary calls.
|
|
|
|
## T05 — Add Stale-While-Refresh for File-Backed Workplan Index
|
|
|
|
```task
|
|
id: STATE-WP-0056-T05
|
|
status: done
|
|
priority: medium
|
|
state_hub_task_id: "0c88c1a2-588b-41f8-bc1c-f94c8b4b0d1a"
|
|
```
|
|
|
|
Make `/workplans/index` resilient when repository filesystem scans are slow.
|
|
|
|
Implementation notes:
|
|
|
|
- Add singleflight behavior so concurrent requests share one in-progress
|
|
rebuild instead of starting or waiting on redundant scans.
|
|
- Return stale cached data quickly while a background refresh runs when the
|
|
cache is expired but still available.
|
|
- Include metadata such as `generated_at`, `stale`, `cache_age_seconds`, and
|
|
optionally `refresh_in_progress`.
|
|
- Consider reading only frontmatter rather than whole markdown files if this
|
|
can be done cleanly.
|
|
- Keep `refresh=true` as an explicit operator escape hatch.
|
|
- Add tests for cache hit, stale return, and forced refresh behavior.
|
|
|
|
Done when a slow filesystem scan cannot block normal dashboard refreshes for
|
|
longer than the frontend timeout if cached data exists.
|
|
|
|
## T06 — Use Browser and HTTP Caching Selectively
|
|
|
|
```task
|
|
id: STATE-WP-0056-T06
|
|
status: done
|
|
priority: medium
|
|
state_hub_task_id: "811f02ff-2e92-4c82-8b8a-e3d39a450b02"
|
|
```
|
|
|
|
Let stable lookup requests benefit from cache headers instead of forcing every
|
|
dashboard request to bypass caches.
|
|
|
|
Implementation notes:
|
|
|
|
- Extend `apiFetch` so callers can choose cache mode.
|
|
- Keep `no-store` for volatile mutation-sensitive resources.
|
|
- Use default browser caching or `reload` only where route cache headers are
|
|
already intentional, such as repo/topic lookup data.
|
|
- Review current route cache headers and align them with dashboard polling
|
|
needs.
|
|
- Avoid stale cached data for controls that immediately follow a mutation.
|
|
|
|
Done when stable overview lookup data no longer bypasses useful cache headers
|
|
by default.
|
|
|
|
## T07 — Optimize `/state/summary` Cache Misses
|
|
|
|
```task
|
|
id: STATE-WP-0056-T07
|
|
status: done
|
|
priority: medium
|
|
state_hub_task_id: "633f4cc6-ffeb-4086-9858-d239f50a9686"
|
|
```
|
|
|
|
Reduce the cost of a cold or expired `/state/summary` request.
|
|
|
|
Implementation notes:
|
|
|
|
- Profile the current sequential query groups in `api/routers/state.py`.
|
|
- Move Python-side counts and scans into SQL where straightforward.
|
|
- Remove unused work from the summary path, such as dead intermediate query
|
|
results.
|
|
- Cache derived sections independently when their freshness requirements differ.
|
|
- Add indexes only after profiling shows a query plan needs them.
|
|
- Keep summary response compatibility for existing consumers and MCP smoke
|
|
tests.
|
|
|
|
Done when a summary cache miss stays comfortably below the frontend timeout
|
|
under the current local data volume.
|
|
|
|
## T08 — Verify Under Dashboard-Style Load
|
|
|
|
```task
|
|
id: STATE-WP-0056-T08
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "353fb25a-5306-416b-8d6d-9b201e6fac87"
|
|
```
|
|
|
|
Prove the dashboard no longer produces frequent abort warnings under realistic
|
|
refresh behavior.
|
|
|
|
Implementation notes:
|
|
|
|
- Add or document a repeatable script that performs dashboard-style concurrent
|
|
endpoint timing before and after the changes.
|
|
- Run API tests and dashboard component tests.
|
|
- Open the dashboard locally and verify that initial load, refresh, hidden-tab
|
|
pause/resume, and partial API failure states behave correctly.
|
|
- Confirm payload sizes are lower than the baseline for the overview page.
|
|
- Update `dashboard/src/docs/overview.md` and `dashboard/src/docs/live-data.md`
|
|
with the new data-loading model.
|
|
|
|
Done when repeated dashboard refreshes do not show the global aborted-operation
|
|
warning during normal local operation, and degraded sections recover cleanly.
|