docs: update SCOPE.md post-0004 + new gap analysis (2026-05-28) vs INTENT.md

SCOPE.md

@@ -6,23 +6,27 @@
 
 It allows users to express intent in natural language from the terminal and receive safe, explainable, context-aware assistance while keeping memory, history, preferences, and adaptation under explicit user control.
 
-## Current Status (Post CYA-WP-0002 Memory Slice)
+## Current Status (Post CYA-WP-0004 Packaging & Distribution Slice)
 
-Two implementation slices have been delivered:
+Four implementation slices have been delivered:
 
-- **CYA-WP-0001 (Console-Native MVP)**: Core CLI, context collection, rule-based safety with mandatory confirmation, LLMAdapter seam, and thin memory ports.
-- **CYA-WP-0002 (Memory Integration)**: Real user-controlled, persisting memory implementation behind the explicit ports, wired into the orchestrator and `--explain-context`, memory signals integrated into safety, dedicated tests, and documentation.
+- **CYA-WP-0001 (Console-Native MVP)**: Core CLI, bounded context collection, rule-based safety with mandatory confirmation, LLMAdapter Protocol seam, basic orchestrator.
+- **CYA-WP-0002 (Memory Integration)**: Real user-controlled, persisting memory (scoped JSON) behind explicit ports, wired into context and safety.
+- **CYA-WP-0003 (Contextual Activation & Retrospection)**: Directory/project-bound automatic memory activation, `cya retrospect` guided reflection sessions, retrospection outcomes feeding future behavior (continuous user-driven optimization loop).
+- **CYA-WP-0004 (Dev-Head Install & Release Packaging)**: Reliable installation from development head (`make dev-install`, direct `git+` installs), dynamic versioning via `setuptools_scm`, clean distribution package building (`python -m build` + verification), lightweight release process, and supporting documentation/Makefile.
 
 Core capabilities now include:
-- Natural language request handling via a clean Typer CLI.
-- Bounded, transparent, non-recursive local context collection.
-- Genuine rule-based risk classification (now memory-aware) with mandatory terminal confirmation.
-- Stable `LLMAdapter` Protocol boundary.
-- Real, user-controlled memory for preferences and workflow patterns (scoped JSON backing under `~/.config/cya/memory/`, fully inspectable and user-editable).
-- Memory surfaced in explanations and fed into safety decisions.
-- A small orchestrator coordinating the pipeline.
+- Natural language request handling via clean Typer CLI.
+- Bounded, transparent local context collection.
+- Genuine rule-based (memory-aware) risk classification with mandatory confirmation.
+- Stable `LLMAdapter` Protocol.
+- Real, user-controlled, contextually activated memory (directory/project scoped) with retrospection support.
+- Automatic memory activation based on working directory/git root.
+- `cya retrospect` for structured reflection and goal setting.
+- Full developer workflow: dev-head install, testing, building distribution packages, and a documented release process.
+- Transparent, inspectable behavior via `--explain-context`.
 
-All LLM interaction flows through the documented adapter seam. Memory flows through explicit ports. No production path bypasses these boundaries.
+All LLM interaction flows through the documented adapter seam. Memory flows through explicit ports. Packaging and distribution are now first-class concerns with a clear path forward. No production path bypasses the defined boundaries.
 
 ## Owns
 
@@ -44,7 +48,7 @@ All LLM interaction flows through the documented adapter seam. Memory flows thro
 - Autonomous or background execution of commands without explicit user confirmation.
 - Deep repository indexing, embeddings, or large-scale content analysis (explicit non-goal of the first slice).
 - Voice, speech, phone-bridge, or non-terminal interfaces (future work).
-- Packaging, distribution, or multi-platform installers beyond basic editable install.
+- Production PyPI publishing and automated release CI (documented process and local tooling exist; actual publication is future work).
 - Long-lived conversational REPL or session state (one-shot + very lightweight session only).
 
 ## Integrates With
@@ -52,56 +56,59 @@ All LLM interaction flows through the documented adapter seam. Memory flows thro
 | Project       | Responsibility                          | Integration Style                  |
 |---------------|-----------------------------------------|------------------------------------|
 | `llm-connect` | Provider access, config, token counting, structured responses | Stable `LLMAdapter` Protocol      |
-| `phase-memory`| User-controlled memory, preferences, history | Explicit ports with real (local JSON) implementation (T02); long-term target is deeper profile-driven integration |
+| `phase-memory`| User-controlled memory, preferences, history, profiles, and activation planning | Explicit ports with real (local JSON + contextual activation + retrospection) implementation; long-term target is deeper profile-driven integration |
 | State Hub     | Work tracking, decisions, coordination  | HTTP REST (non-runtime)           |
 
-## MVP Scope (CYA-WP-0001)
+## Current Delivered Scope (Post 0004)
 
-What was explicitly in scope for the first slice:
+Significant slices have been delivered beyond the original MVP:
 
-- A functional `cya` CLI that accepts natural language requests.
-- Safe, bounded context collection with full transparency (`--explain-context`).
-- Rule-based safety classification as the primary mechanism, with mandatory confirmation.
-- A clean, documented adapter boundary for future real LLM backends.
-- Real user-controlled memory implementation (scoped, persisting, explainable) behind the explicit ports, with a clean seam for future deeper `phase-memory` integration.
-- Basic orchestrator that ties the pieces together.
-- Test coverage focused on safety invariants and context rules.
-- Clear public boundaries and extension points.
+- Full console-native CLI with rich output.
+- Context-aware, directory/project-bound memory activation.
+- User-driven retrospection and continuous optimization loops (`cya retrospect`).
+- Real, inspectable, user-controlled memory with strong explainability and safety integration.
+- Complete developer workflow: installation from dev head, testing, building distribution packages, and a documented release process.
+- Packaging and distribution now treated as first-class concerns (with registered future work).
+
+See the individual workplans for detailed scope per slice.
 
 ## Explicitly Out of Scope (Current and Near-Term)
 
-- Full deep integration with the complete `phase-memory` profile/planner/graph system (current implementation uses a deliberate, user-visible local JSON store as the first real backing; deeper integration is planned future work).
-- Real `llm-connect` client (only the contract + fake exists).
-- Deep git/repository understanding beyond basic status + log + user-declared memory.
-- Automatic command execution (even "safe" suggestions).
-- Structured editing / patch generation.
-- Rich multi-turn conversational state (one-shot + lightweight scoped memory only).
+- Full deep integration with the complete `phase-memory` profile/planner/graph system (current implementation uses a deliberate, user-visible local JSON store with contextual activation; deeper integration is planned future work per MemoryVision.md).
+- Real `llm-connect` client implementation (only the stable `LLMAdapter` Protocol contract + FakeLLMAdapter exists).
+- Deep semantic repository understanding or large-scale content analysis.
+- Automatic command execution (even "safe" suggestions) — explicit user confirmation remains mandatory for anything non-safe.
+- Rich multi-turn conversational state beyond lightweight scoped memory + retrospection.
 - Cost tracking, token budgeting, or usage dashboards.
 - Team/shared memory or collaboration features.
 - Plugin system or domain-specific extensions.
+- Production PyPI publishing and fully automated release CI (a lightweight documented local process exists; automation is future work).
 
 ## Extension Points (Registered)
 
 - `cya/llm/adapter.py` — `LLMAdapter` Protocol (the primary seam).
-- `cya/memory/__init__.py` — the four explicit ports with real (persisting, user-controlled) implementation behind them.
-- `cya/safety/risk.py` — the `_RULES` table and `classify()` function.
+- `cya/memory/__init__.py` — the explicit ports (with real implementation, contextual activation, and retrospection support).
+- `cya/safety/risk.py` — the `_RULES` table and `classify()` function (memory-aware).
 - `cya/context/collector.py` — collection functions and ignore policy.
 - `cya/orchestrator.py` — the main coordination entry point.
+- Packaging & distribution: `Makefile`, `pyproject.toml`, `docs/release-process.md`, and `MANIFEST.in` (first-class concern with registered future work).
 
-## Success Criteria (Current Slice)
+## Success Criteria (Current State)
 
-A new user can:
-- Clone the repo, run `pip install -e .`, and successfully use `cya` for 2–3 realistic tasks after reading only the README.
-- Understand exactly what context is being sent via `--explain-context`.
+A new user or contributor can:
+- Install the latest development code easily (`make dev-install` or direct git+) and use `cya` for realistic tasks after reading the README.
+- Understand exactly what context and memory influenced a response via `--explain-context`.
 - Trust that dangerous actions will never execute without explicit confirmation.
-- See a clear path for how real `llm-connect` and `phase-memory` will plug in later.
+- Use `cya retrospect` to reflect on usage and set goals that influence future behavior.
+- Build and verify distribution packages locally.
+- See a clear path for how real `llm-connect`, deeper `phase-memory`, and future PyPI releases will integrate.
 
-Sibling project owners can read the workplan + boundary documentation and know precisely where their packages integrate.
+Sibling project owners (llm-connect, phase-memory, State Hub) can read the workplans + boundary documentation and know precisely where their packages integrate.
 
 ---
 
-**This SCOPE document reflects the state after CYA-WP-0001 (MVP) + CYA-WP-0002 (Memory Integration).**
+**This SCOPE document reflects the state after CYA-WP-0004 (Dev-Head Install & Release Packaging).**
 
-It is intentionally narrower than the long-term vision in INTENT.md and MemoryVision.md, but now includes a real first slice of user-controlled memory as delivered by 0002.
+It remains intentionally narrower than the long-term vision in INTENT.md and MemoryVision.md, but now incorporates significant advances in contextual memory activation, user-driven retrospection/optimization loops, and proper packaging & distribution capabilities.
 
-See `workplans/CYA-WP-0002-memory-integration-roadmap.md` and `MemoryVision.md` for the intended direction of deeper `phase-memory` integration.
+See the individual workplans (especially CYA-WP-0003 and CYA-WP-0004) and `MemoryVision.md` for the intended direction of deeper `phase-memory` integration and future evolution.

history/2026-05-28-CYA-Intent-Scope-Gap-Analysis-Post-0004.md

@@ -0,0 +1,131 @@
+# Gap Analysis: INTENT.md vs SCOPE.md (Post CYA-WP-0004)
+
+**Date:** 2026-05-28  
+**Repo:** can-you-assist  
+**Workplan:** CYA-WP-0004 (completed)  
+**Previous Analysis:** 2026-05-27 (Post 0002)  
+**Author:** Grok
+
+## Executive Summary
+
+Since the May 27 analysis (post-0002), two major follow-on workplans have been completed:
+
+- **CYA-WP-0003**: Contextual Memory Activation & Retrospection Loops — delivered directory/project-bound automatic memory activation and the `cya retrospect` guided reflection mechanism for continuous user-driven optimization.
+- **CYA-WP-0004**: Developer Installation from Git and Release Distribution Packaging — delivered reliable dev-head installation, dynamic versioning, clean distribution package building, a lightweight release process, and clear documentation for both dev and future released usage.
+
+The largest previous gap (memory & longitudinal adaptation) has been substantially addressed in practice (though deeper `phase-memory` integration remains future work). Packaging/distribution has moved from a notable gap to a documented, owned capability.
+
+**Overall Assessment**: Strong progress. The product is now significantly closer to the "Personalized Console Helper" vision in INTENT.md, with excellent safety, explainability, and user control. Remaining gaps are more about depth and future evolution than foundational missing pieces.
+
+## Strong Alignments (Updated)
+
+| Area                              | INTENT.md Position                              | Current Reality (Post-0004)                                      | Assessment          |
+|-----------------------------------|--------------------------------------------------|------------------------------------------------------------------|---------------------|
+| Console-native experience         | Foundational                                     | Excellent                                                        | Strong match        |
+| Safety & mandatory confirmation   | Important                                        | Core product behavior + memory-aware                             | Exceeded            |
+| Explainability & transparency     | Strong requirement                               | Very strong (provenance for context + memory + retrospection)    | Strong              |
+| Backend agnosticism               | Must use llm-connect seam                        | Clean LLMAdapter Protocol + Fake                                 | Excellent           |
+| User-controlled memory            | Central principle                                | Real, contextually activated, retrospection-supported (local)    | Major improvement   |
+| Longitudinal adaptation           | Personalized helper that improves over time      | `cya retrospect` + automatic activation now provides a real loop | Significantly better|
+| Packaging & distribution          | Not heavily emphasized                           | Dev-head install + release process now first-class               | Major positive shift|
+| Clear boundaries                  | cya / llm-connect / phase-memory separation      | Clearly documented                                               | Good                |
+
+## Key Gaps (Post-0004)
+
+### 1. Memory & Adaptation — Substantially Addressed, Deeper Integration Pending
+
+**Previous Status (Post-0002)**: Real but mostly passive local JSON memory.
+
+**Current Status (Post-0004)**:
+- Strong contextual activation based on directory/project (T03 of 0003).
+- `cya retrospect` provides a structured mechanism for reflection and goal setting (T04 of 0003).
+- Retrospection outcomes feed back into future behavior (continuous optimization loop).
+- Excellent explainability and user control.
+
+**Remaining Sub-Gaps** (intentional):
+- Still a local JSON implementation. Full integration with `phase-memory`'s profile/planner/graph system (as described in MemoryVision.md) is planned future work.
+- Richer memory kinds (beyond preference + interaction goals) and automatic pattern learning are not yet present.
+
+**Assessment**: Large positive movement. The "memory as passive store" problem has been meaningfully solved. The foundation for true longitudinal, user-steerable adaptation now exists.
+
+### 2. Depth of Local Context Understanding
+
+**INTENT.md** envisions rich assistance with code repositories, notes, project structures, and conventions.
+
+**Current State**:
+- Context collector remains intentionally shallow (top-level entries + basic git).
+- Memory helps significantly with *user-declared* project conventions and patterns, which is a practical improvement.
+- No deep semantic understanding or large-scale analysis.
+
+**Assessment**: Still a medium gap. 0003's activation + retrospection helps users bring their own context effectively, but the system itself does not deeply understand repositories or notes.
+
+### 3. One-Shot vs Rich Longitudinal Value
+
+**INTENT.md**: Strong vision of the assistant becoming more useful over time through memory of habits, conventions, and recurring workflows.
+
+**Current Reality (Post-0004)**:
+- Users can now teach the system via normal use + explicit `cya retrospect` sessions.
+- Directory/project-bound activation makes memory feel proactive.
+- The retrospection loop provides a deliberate mechanism for continuous improvement.
+
+**Assessment**: Significantly improved. The basic mechanism for longitudinal, user-controlled adaptation is now in place and usable. Rich automatic pattern learning remains future work.
+
+### 4. Packaging & Distribution — Major Positive Shift
+
+**Previous Status**: Effectively a gap (only editable install documented).
+
+**Current Status (Post-0004)**:
+- Reliable dev-head installation from git (local or direct).
+- Dynamic versioning.
+- Clean, verifiable distribution package building.
+- Documented lightweight release process.
+- Clear documentation distinguishing dev vs future released usage.
+
+**Remaining Sub-Gaps** (registered as debt):
+- Actual PyPI publishing workflow.
+- Automated releases via CI.
+- Package signing.
+- Multi-Python distribution testing as a CI requirement.
+
+**Assessment**: Excellent progress. What was previously a notable weakness is now a documented strength with a clear path forward.
+
+### 5. Safety & Explainability — Continued Strength
+
+Remains a core strength. Memory activation and retrospection outcomes are required to flow through the rule-based risk classifier, and all memory influence is visible via `--explain-context`.
+
+## Summary Table (Updated)
+
+| Gap Area                          | Severity (Post-0002) | Severity (Post-0004) | Trend              | Notes |
+|-----------------------------------|----------------------|----------------------|--------------------|-------|
+| Memory & Longitudinal Adaptation  | Medium               | Small-Medium         | Much better        | Strong activation + retrospection loop now exists |
+| Depth of Context Understanding    | Medium-Large         | Medium               | Improved (via memory) | Collector still shallow; user memory helps |
+| Packaging & Distribution          | Medium (gap)         | Small                | Major improvement  | Now first-class with process + docs |
+| Safety & Explainability           | Positive             | Positive             | Stable             | Remains a core strength |
+| One-shot vs Rich Longitudinal     | Medium               | Small                | Significantly better | Foundation for continuous optimization in place |
+
+## Recommendations
+
+1. **Continue evolving the memory story** toward deeper `phase-memory` integration (as outlined in MemoryVision.md and the 0003 contract). This is the natural next major deepening.
+
+2. **Consider a dedicated "Context Depth" exploration** if richer repository/note understanding becomes a priority (separate from the memory activation work).
+
+3. **Move packaging forward** when ready:
+   - Add `make check-dist` (or equivalent) as a required CI step.
+   - Define and implement the PyPI publishing workflow.
+   - Consider automated releases on annotated tags.
+
+4. **Keep the explicit seam discipline** — the ports, activation model, and retrospection flow from 0003 remain excellent boundaries.
+
+## Conclusion
+
+CYA-WP-0003 and 0004 have together delivered substantial progress against the original INTENT.md vision. The product now has credible, usable mechanisms for contextual memory, user-driven continuous optimization, and proper packaging/distribution — areas that were previously either missing or significantly weaker.
+
+The remaining gaps are largely about *depth and future evolution* rather than missing foundational capabilities. The project is in a much stronger position relative to its original intent than it was after 0002.
+
+---
+
+**Related Documents**
+- Previous analysis: `history/2026-05-27-CYA-Intent-Scope-Gap-Analysis-Post-0002.md`
+- Memory vision: `MemoryVision.md`
+- Recent workplans: `workplans/CYA-WP-0003-...md` and `CYA-WP-0004-...md`
+- Current SCOPE.md (updated as part of this assessment)