diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e3c30c..5b14dcb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - **sys-medic agent**: Linux/Kubernetes node health assessment agent integrated as a standard kaizen-agentic infrastructure agent (KAIZEN-WP-0002 Part 1) +- **Ecosystem integration (WP-0004)**: Helix Forge correlation (`metrics correlate`, `HELIX_SESSION_UID` env), artifact-store publish (`metrics publish`), activity-core ActivityDefinition references, integration patterns docs - **Project metrics convention (ADR-004)**: `.kaizen/metrics//` storage via `MetricsStore` and `OptimizerStore` - **Metrics CLI**: `kaizen-agentic metrics record|show|list|export|optimize` for per-execution records and optimizer analysis - **Optimizer integration**: `OptimizationLoop.from_metrics_store()` wired to project metrics; `memory brief` includes `## Performance Summary` diff --git a/docs/CLI_CHEAT_SHEET.md b/docs/CLI_CHEAT_SHEET.md index 504cb08..8287f83 100644 --- a/docs/CLI_CHEAT_SHEET.md +++ b/docs/CLI_CHEAT_SHEET.md @@ -64,6 +64,16 @@ kaizen-agentic metrics export tdd-workflow kaizen-agentic metrics optimize tdd-workflow # analyze one agent (≥10 records) kaizen-agentic metrics optimize # analyze all agents with metrics +# Helix Forge correlation (fleet layer — agentic-resources) +export HELIX_SESSION_UID="claude:" +kaizen-agentic metrics record tdd-workflow --success --time 120 --quality 0.9 +kaizen-agentic metrics correlate claude: # needs HELIX_STORE_DB + +# Publish optimizer evidence to artifact-store (optional) +export ARTIFACTSTORE_API_URL=http://127.0.0.1:8000 +export ARTIFACTSTORE_API_TOKEN= +kaizen-agentic metrics publish + # Scaffold memory + metrics together kaizen-agentic memory init tdd-workflow kaizen-agentic memory init tdd-workflow --no-metrics # memory only diff --git a/docs/INTEGRATION_PATTERNS.md b/docs/INTEGRATION_PATTERNS.md index ee52745..bf60110 100644 --- a/docs/INTEGRATION_PATTERNS.md +++ b/docs/INTEGRATION_PATTERNS.md @@ -1,401 +1,105 @@ -# Integration Patterns for Existing Projects +# Integration Patterns -This guide documents proven patterns for integrating Kaizen Agentic agents into existing projects that already have agent systems. +How kaizen-agentic composes with ecosystem repos **by contract** — no merged +codebases, no duplicated capabilities. -## Overview +Reference: [wiki/EcosystemIntegration.md](../wiki/EcosystemIntegration.md), +[KAIZEN-WP-0004](../workplans/kaizen-agentic-WP-0004-ecosystem-integration.md). -When introducing Kaizen agents to existing projects, you'll encounter various scenarios that require different integration approaches. This guide provides tested patterns and strategies. +--- -## Integration Scenarios +## Pattern 1 — Helix Forge correlation (agentic-resources) -### Scenario 1: Clean Integration (No Existing Agents) +**Problem:** Project metrics and fleet session metrics answer different questions. -**When to use**: Project has no existing agent systems. +**Contract:** Optional `helix_session_uid` on ADR-004 execution records. + +| kaizen-agentic | agentic-resources | +|----------------|-------------------| +| `metrics record` at session close | Helix capture → digest store | +| `metrics correlate ` read-only lookup | `Store.get_digest(session_uid)` | +| `HELIX_SESSION_UID` env auto-merge | `Session.session_uid` | + +**Docs:** [integrations/helix-forge-correlation.md](integrations/helix-forge-correlation.md) + +**Boundary:** kaizen-agentic does not ingest session JSONL. + +--- + +## Pattern 2 — activity-core triggers + +**Problem:** Recurring kaizen checks need scheduling without custom cron in this repo. + +**Contract:** ActivityDefinition markdown files declare triggers + actions that +invoke kaizen-agentic CLI commands. + +| Definition | Trigger | CLI command | +|------------|---------|-------------| +| [weekly-metrics-optimize](integrations/activity-definitions/weekly-metrics-optimize.md) | Cron Mon 08:00 | `metrics optimize` | +| [post-install-metrics-scaffold](integrations/activity-definitions/post-install-metrics-scaffold.md) | `kaizen.agent.installed` | `memory init` validation | +| [low-success-rate-review](integrations/activity-definitions/low-success-rate-review.md) | `kaizen.metrics.recorded` | `metrics show` + `optimize` | + +**Activation:** + +1. Copy or symlink definitions from `docs/integrations/activity-definitions/` into + activity-core's `activity-definitions/` tree (or register as external ConfigMap). +2. Run `make sync-activity-definitions` in activity-core. +3. Enable definitions (`enabled: true`) after resolver wiring is verified. + +**Smoke test (manual):** -**Pattern**: Direct installation ```bash -kaizen-agentic init . --agents keepaTodofile,keepaChangelog,tdd-workflow +# Against a repo with populated metrics +cd /path/to/project-with-kaizen +kaizen-agentic metrics list +kaizen-agentic metrics optimize +# Verify analysis.json written +test -f .kaizen/metrics/optimizer/analysis.json && echo OK ``` -**Benefits**: -- Straightforward setup -- No conflicts to resolve -- Full Kaizen agent functionality +**Boundary:** kaizen-agentic does not run Temporal schedules. -### Scenario 2: Claude Code Integration +--- -**When to use**: Project already uses Claude Code with CLAUDE.md. +## Pattern 3 — artifact-store evidence retention + +**Problem:** Optimizer outputs need durable, attributable retention beyond local disk. + +**Contract:** `metrics publish` registers `analysis.json` + `recommendations.jsonl` +as an artifact package with `retention_class: raw-evidence`. -**Pattern**: Respectful coexistence ```bash -# 1. Detect existing setup -kaizen-agentic detect - -# 2. Install compatible agents -kaizen-agentic install keepaTodofile keepaChangelog - -# 3. Update CLAUDE.md with new agent references +export ARTIFACTSTORE_API_URL=http://127.0.0.1:8000 +export ARTIFACTSTORE_API_TOKEN= +kaizen-agentic metrics optimize +kaizen-agentic metrics publish --target . ``` -**Considerations**: -- Preserve existing CLAUDE.md content -- Add Kaizen agent references to existing documentation -- Maintain Claude Code workflow compatibility +**Manifest:** [integrations/optimizer-artifact-manifest.md](integrations/optimizer-artifact-manifest.md) -### Scenario 3: Custom Agent Replacement +**Boundary:** Publish is optional; local `.kaizen/metrics/optimizer/` remains canonical. -**When to use**: Project has custom agents that overlap with Kaizen functionality. +--- -**Pattern**: Gradual migration with backup -```bash -# 1. Analyze existing agents -kaizen-agentic detect --detailed +## Pattern 4 — Canon and knowledge (stretch) -# 2. Create migration plan -kaizen-agentic migrate --dry-run +Design-only paths for info-tech-canon and kontextual-engine: -# 3. Execute migration with backup -kaizen-agentic migrate -``` +- [integrations/canon-template-mapping.md](integrations/canon-template-mapping.md) +- [integrations/briefs/tdd-workflow-canon-brief.md](integrations/briefs/tdd-workflow-canon-brief.md) +- [integrations/kontextual-wiki-ingestion-spike.md](integrations/kontextual-wiki-ingestion-spike.md) -**Steps**: -1. **Backup** existing agents -2. **Map** custom agents to Kaizen equivalents -3. **Migrate** functionality to extensions -4. **Test** new agent workflow -5. **Archive** old agents after verification +No runtime dependency in WP-0004. -### Scenario 4: Hybrid Coexistence +--- -**When to use**: Project has essential custom agents that cannot be replaced. +## Environment variables -**Pattern**: Namespace separation -```bash -# 1. Install Kaizen agents in parallel -kaizen-agentic install keepaTodofile --target agents/kaizen/ - -# 2. Keep custom agents in separate directory -# agents/custom/todo_manager.py -# agents/kaizen/agent-keepaTodofile.md - -# 3. Create integration extensions -kaizen-agentic extensions create custom-integration keepaTodofile -``` - -**Directory Structure**: -``` -project/ -├── agents/ -│ ├── custom/ # Existing custom agents -│ │ ├── todo_manager.py -│ │ └── code_reviewer.py -│ └── kaizen/ # Kaizen agents -│ ├── agent-keepaTodofile.md -│ └── agent-code-refactoring.md -├── .kaizen/ -│ └── extensions/ # Integration extensions -└── CLAUDE.md # Updated configuration -``` - -### Scenario 5: Extension-Based Integration - -**When to use**: Custom agents have unique functionality that should be preserved. - -**Pattern**: Extend Kaizen agents with custom functionality -```bash -# 1. Create project-specific extension -kaizen-agentic extensions create project-todo keepaTodofile \ - --description "TODO manager with custom workflow integration" - -# 2. Configure custom behavior -# Edit .kaizen/extensions/project-todo/extension.yml - -# 3. Migrate custom logic to extension -``` - -**Extension Configuration Example**: -```yaml -name: project-todo -base_agent: keepaTodofile -extension_type: functional_extension -description: "TODO manager with custom workflow integration" - -configuration: - custom_instructions: | - Follow our project-specific TODO format: - - Use JIRA ticket references - - Include priority levels (P0-P3) - - Auto-assign based on component - -custom_commands: - create-epic: "Create epic-level TODO items" - sync-jira: "Synchronize with JIRA tickets" - priority-report: "Generate priority-based reports" - -environment_overrides: - JIRA_URL: "https://company.atlassian.net" - TODO_FORMAT: "custom" -``` - -## Conflict Resolution Patterns - -### Name Conflicts - -**Problem**: Multiple agents with the same name. - -**Pattern**: Rename with suffix -```bash -# Automatic resolution -todo_manager -> todo_manager_custom -keepaTodofile -> keepaTodofile (Kaizen agent) -``` - -**Implementation**: -- Add `_custom` suffix to project-specific agents -- Update references in scripts and documentation -- Create aliases for backward compatibility - -### Functional Overlaps - -**Problem**: Multiple agents perform similar functions. - -**Pattern**: Choose primary, extend secondary -```bash -# Primary: Kaizen agent (standardized) -# Secondary: Custom agent -> extension - -# Example: Both have TODO management -# Decision: Use keepaTodofile as primary -# Convert custom logic to extension -``` - -**Decision Matrix**: -| Factor | Choose Kaizen | Choose Custom | Create Extension | -|--------|---------------|---------------|------------------| -| Standard functionality | ✅ | ❌ | ✅ | -| Custom business logic | ❌ | ✅ | ✅ | -| Maintenance burden | ✅ | ❌ | ⚠️ | -| Team familiarity | ⚠️ | ✅ | ✅ | - -### Integration Order - -**Pattern**: Infrastructure first, features last -1. **Infrastructure agents** (setupRepository, tooling-optimization) -2. **Core functionality** (keepaTodofile, keepaChangelog) -3. **Development process** (tdd-workflow, code-refactoring) -4. **Specialized features** (testing-efficiency, datamodel-optimization) - -## Project Structure Respect Patterns - -### Existing Directory Structures - -**Pattern**: Adaptive installation -```bash -# Respect existing structure -project/ -├── tools/agents/ # Existing agent directory -├── scripts/ # Existing automation -└── docs/ # Existing documentation - -# Kaizen adaptation -kaizen-agentic install --target tools/agents/ keepaTodofile -# Creates: tools/agents/agent-keepaTodofile.md -``` - -### Configuration File Integration - -**Pattern**: Merge, don't replace -```bash -# Before -CLAUDE.md # Existing Claude config -project-config.yml # Existing project config - -# After (merged) -CLAUDE.md # Updated with Kaizen agents -project-config.yml # Preserved -.kaizen/extensions.yml # New Kaizen-specific config -``` - -### Build System Integration - -**Pattern**: Extend existing targets -```makefile -# Existing Makefile -test: - pytest tests/ - -# After Kaizen integration (extended) -test: test-core test-agents - @echo "All tests completed" - -test-core: - pytest tests/ - -test-agents: - kaizen-agentic validate - -# New Kaizen targets -agents-status: - kaizen-agentic status - -agents-update: - kaizen-agentic update -``` - -## Safe Transition Strategies - -### Phased Rollout - -**Phase 1: Detection and Planning** -```bash -# Week 1: Analysis -kaizen-agentic detect --detailed -kaizen-agentic migrate --dry-run - -# Decision point: Continue or modify approach -``` - -**Phase 2: Infrastructure Agents** -```bash -# Week 2: Core infrastructure -kaizen-agentic install setupRepository -# Test and validate before proceeding -``` - -**Phase 3: Core Functionality** -```bash -# Week 3: Essential agents -kaizen-agentic install keepaTodofile keepaChangelog -# Create extensions for custom functionality -``` - -**Phase 4: Advanced Features** -```bash -# Week 4: Specialized agents -kaizen-agentic install tdd-workflow code-refactoring -# Full integration testing -``` - -### Rollback Strategy - -**Pattern**: Versioned backups with restore capability -```bash -# Before migration -.kaizen-migration-backup-timestamp/ -├── agents/ # Original agents -├── CLAUDE.md # Original configuration -└── restoration.md # Rollback instructions - -# Rollback command (if needed) -kaizen-agentic rollback --backup .kaizen-migration-backup-timestamp/ -``` - -### Validation Gates - -**Pattern**: Automated validation at each phase -```bash -# After each phase -kaizen-agentic validate -make test -make agents-status - -# Success criteria for proceeding: -# ✅ All agents load without errors -# ✅ All tests pass -# ✅ No functionality regressions -``` - -## Best Practices - -### Communication - -1. **Team Notification**: Inform team before starting migration -2. **Documentation**: Update project docs with new agent workflows -3. **Training**: Provide team training on Kaizen agents -4. **Gradual Adoption**: Allow team to adapt gradually - -### Technical - -1. **Backup Everything**: Create comprehensive backups -2. **Test Thoroughly**: Validate each integration step -3. **Monitor Impact**: Watch for performance or workflow impacts -4. **Version Control**: Commit changes in logical phases - -### Maintenance - -1. **Regular Updates**: Keep Kaizen agents updated -2. **Extension Maintenance**: Maintain custom extensions -3. **Documentation Sync**: Keep docs synchronized with agent changes -4. **Team Feedback**: Collect and act on team feedback - -## Troubleshooting Common Issues - -### Agent Conflicts - -**Issue**: Multiple agents trying to manage the same files. - -**Solution**: -```bash -# Identify conflicts -kaizen-agentic detect --detailed - -# Resolve with namespace separation -mkdir agents/legacy agents/kaizen -mv agents/todo_manager.py agents/legacy/ -kaizen-agentic install --target agents/kaizen/ keepaTodofile -``` - -### Configuration Conflicts - -**Issue**: Conflicting configuration files. - -**Solution**: -```bash -# Merge configurations -cp CLAUDE.md CLAUDE.md.backup -kaizen-agentic install keepaTodofile -# Manually merge CLAUDE.md.backup content -``` - -### Workflow Disruption - -**Issue**: New agents disrupt existing workflows. - -**Solution**: -```bash -# Create compatibility extensions -kaizen-agentic extensions create workflow-compat keepaTodofile -# Configure extension to match existing workflow -``` - -## Success Metrics - -### Technical Metrics -- ✅ Zero agent loading errors -- ✅ All tests passing -- ✅ No performance regressions -- ✅ Successful backup/restore capability - -### Team Metrics -- ✅ Team adoption of new agents -- ✅ Maintained productivity during transition -- ✅ Positive feedback on new capabilities -- ✅ Reduced maintenance overhead - -### Project Metrics -- ✅ Improved code quality metrics -- ✅ Better documentation coverage -- ✅ Enhanced development workflow efficiency -- ✅ Standardized agent ecosystem - -## Conclusion - -Successful integration of Kaizen agents into existing projects requires: - -1. **Careful analysis** of existing agent systems -2. **Respectful approach** to existing project structure -3. **Gradual migration** with proper backup strategies -4. **Extension mechanisms** for preserving custom functionality -5. **Team communication** and training throughout the process - -Follow these patterns and your integration will be smooth, reversible, and beneficial to your development workflow. \ No newline at end of file +| Variable | Used by | Purpose | +|----------|---------|---------| +| `HELIX_SESSION_UID` | `metrics record` | Fleet session correlation | +| `HELIX_REPO`, `HELIX_FLAVOR` | `metrics record` | Session context | +| `HELIX_TOKENS`, `HELIX_INFRA_OVERHEAD_SHARE` | `metrics record` | Fleet cost fields | +| `HELIX_STORE_DB` | `metrics correlate` | Digest lookup database | +| `ARTIFACTSTORE_API_URL` | `metrics publish` | Registry endpoint | +| `ARTIFACTSTORE_API_TOKEN` | `metrics publish` | Write auth bearer token | \ No newline at end of file diff --git a/docs/agency-framework.md b/docs/agency-framework.md index d63bf78..f9f28d2 100644 --- a/docs/agency-framework.md +++ b/docs/agency-framework.md @@ -260,6 +260,8 @@ kaizen-agentic metrics show # Summary + recent executions kaizen-agentic metrics list # Agents with metrics in project kaizen-agentic metrics export # Dump executions.jsonl kaizen-agentic metrics optimize [agent] # Run optimizer on project metrics (≥10 records) +kaizen-agentic metrics correlate # Helix Forge digest lookup (read-only) +kaizen-agentic metrics publish # Register optimizer output in artifact-store ``` `memory brief` includes a `## Performance Summary` when metrics exist (success @@ -272,12 +274,28 @@ skip). Record outcomes at session close per ### Fleet correlation Project metrics correlate with **Helix Forge** fleet session metrics in -`agentic-resources` via optional `helix_session_uid` (ADR-004). See -[wiki/EcosystemIntegration.md](../wiki/EcosystemIntegration.md). +`agentic-resources` via optional `helix_session_uid` (ADR-004). + +- `HELIX_SESSION_UID` (and related env vars) auto-merge on `metrics record` +- `metrics correlate ` looks up fleet digest when `HELIX_STORE_DB` is set + +See [integrations/helix-forge-correlation.md](integrations/helix-forge-correlation.md) +and [wiki/EcosystemIntegration.md](../wiki/EcosystemIntegration.md). ### Evidence retention -Optimizer outputs may be published to `artifact-store` (WP-0004 Part 3). +After `metrics optimize`, optionally publish optimizer outputs to **artifact-store**: + +```bash +export ARTIFACTSTORE_API_URL=http://127.0.0.1:8000 +export ARTIFACTSTORE_API_TOKEN= +kaizen-agentic metrics publish --target . +``` + +Package uses `retention_class: raw-evidence` (180d). Local +`.kaizen/metrics/optimizer/` remains authoritative when publish is skipped. + +Manifest: [integrations/optimizer-artifact-manifest.md](integrations/optimizer-artifact-manifest.md). --- diff --git a/docs/integrations/activity-definitions/low-success-rate-review.md b/docs/integrations/activity-definitions/low-success-rate-review.md new file mode 100644 index 0000000..c85e7a8 --- /dev/null +++ b/docs/integrations/activity-definitions/low-success-rate-review.md @@ -0,0 +1,43 @@ +--- +id: kaizen-low-success-rate-review +name: Low Agent Success Rate Review +enabled: false +owner: kaizen-agentic +governance: custodian +status: proposed +trigger: + type: event + event_type: kaizen.metrics.recorded +context_sources: + - type: event-payload + bind_to: context.metrics +--- + +# Low Agent Success Rate Review + +When a project agent's rolling success rate drops below 0.8, create a review +task in issue-core for human or optimizer-agent follow-up. + +```rule +id: flag-low-success-rate +condition: 'context.metrics.summary.success_rate < 0.8 && context.metrics.summary.execution_count >= 5' +action: + task_template: "Review {{context.metrics.agent}} success rate ({{context.metrics.summary.success_rate}})" + description: | + Agent {{context.metrics.agent}} in {{context.metrics.project}} has success_rate + below 0.8 over {{context.metrics.summary.execution_count}} executions. + Run: kaizen-agentic metrics show {{context.metrics.agent}} + Then: kaizen-agentic metrics optimize {{context.metrics.agent}} + target_repo: "{{context.metrics.project}}" + priority: high + labels: ["kaizen", "metrics", "review", "automated"] +``` + +**Threshold:** 0.8 success rate, minimum 5 executions (avoids noise on early pilots). + +**CLI mapping:** Event emitter is future work; manual check today: + +```bash +kaizen-agentic metrics show # inspect summary.success_rate +kaizen-agentic metrics optimize +``` \ No newline at end of file diff --git a/docs/integrations/activity-definitions/post-install-metrics-scaffold.md b/docs/integrations/activity-definitions/post-install-metrics-scaffold.md new file mode 100644 index 0000000..b2a64c0 --- /dev/null +++ b/docs/integrations/activity-definitions/post-install-metrics-scaffold.md @@ -0,0 +1,41 @@ +--- +id: kaizen-post-install-metrics-scaffold +name: Post-Install Metrics Scaffold Validation +enabled: false +owner: kaizen-agentic +governance: custodian +status: proposed +trigger: + type: event + event_type: kaizen.agent.installed +context_sources: + - type: event-payload + bind_to: context.install +--- + +# Post-Install Metrics Scaffold Validation + +Fires when an agent is installed into a project. Verifies that memory and metrics +scaffolds exist for the installed agent. + +```rule +id: validate-metrics-scaffold +condition: 'context.install.agent != ""' +action: + task_template: "Validate kaizen scaffold for {{context.install.agent}}" + description: | + In {{context.install.project_root}} verify: + - .kaizen/agents/{{context.install.agent}}/memory.md exists OR run: + kaizen-agentic memory init {{context.install.agent}} + - .kaizen/metrics/{{context.install.agent}}/ exists OR re-run init without --no-metrics + target_repo: "{{context.install.repo}}" + priority: low + labels: ["kaizen", "metrics", "scaffold", "automated"] +``` + +**CLI mapping:** + +```bash +kaizen-agentic memory init # scaffolds memory + metrics by default +kaizen-agentic metrics list # confirms metrics directory after first record +``` \ No newline at end of file diff --git a/docs/integrations/activity-definitions/weekly-metrics-optimize.md b/docs/integrations/activity-definitions/weekly-metrics-optimize.md new file mode 100644 index 0000000..f75e5b7 --- /dev/null +++ b/docs/integrations/activity-definitions/weekly-metrics-optimize.md @@ -0,0 +1,44 @@ +--- +id: kaizen-weekly-metrics-optimize +name: Weekly Kaizen Metrics Optimization +enabled: false +owner: kaizen-agentic +governance: custodian +status: proposed +trigger: + type: cron + cron_expression: "0 8 * * 1" + timezone: Europe/Berlin + misfire_policy: skip +context_sources: + - type: shell + query: discover_kaizen_projects + params: + marker: .kaizen/metrics + bind_to: context.projects +--- + +# Weekly Kaizen Metrics Optimization + +Runs every Monday 08:00 Berlin time on repos that contain `.kaizen/metrics/`. +Invokes the kaizen-agentic optimizer CLI per project. + +```rule +id: run-weekly-optimizer +for_each: context.projects +bind_as: p +condition: 'p.has_metrics == true' +action: + task_template: "Run kaizen metrics optimize on {{p.repo}}" + description: | + cd {{p.root}} && kaizen-agentic metrics optimize + Optional: kaizen-agentic metrics publish (when artifact-store configured) + target_repo: "{{p.repo}}" + priority: medium + labels: ["kaizen", "metrics", "optimizer", "automated"] +``` + +**Activation:** sync this definition into activity-core via `make sync-activity-definitions` +after enabling the shell resolver for `discover_kaizen_projects`. + +**CLI mapping:** `kaizen-agentic metrics optimize` (no agent filter = all agents with metrics). \ No newline at end of file diff --git a/docs/integrations/briefs/tdd-workflow-canon-brief.md b/docs/integrations/briefs/tdd-workflow-canon-brief.md new file mode 100644 index 0000000..2237b9c --- /dev/null +++ b/docs/integrations/briefs/tdd-workflow-canon-brief.md @@ -0,0 +1,44 @@ +# tdd-workflow — InfoTechCanon-style Brief + +Compact agent brief derived from `agents/agent-tdd-workflow.md` (metrics pilot). +Reference for fleet-wide brief rollout. + +```yaml +profile: + id: kaizen/tdd-workflow + version: "1.0" + domain: development-process + intent: + summary: Guide TDD8 ISSUE-TEST-RED-GREEN-REFACTOR-DOCUMENT-REFINE-PUBLISH cycles + outcomes: + - Acceptance criteria covered by tests before PUBLISH + - Sidequests tracked without blocking parent issues + - Workspace integrated cleanly via make tdd-finish + metrics: + primary: + name: test_pass_rate + target: 1.0 + measurement: passing_tests / total_tests at PUBLISH + secondary: + - name: cycle_time_s + measurement: session duration (execution_time_s) + collection: + storage: .kaizen/metrics/tdd-workflow/ + frequency: per_execution + idempotency: + signals: + - current_issue.json workspace state + - idempotency_key on metrics record + session_protocol: + start: read .kaizen/agents/tdd-workflow/memory.md + close: + - update memory.md sections + - kaizen-agentic metrics record tdd-workflow + ecosystem: + fleet_correlation: helix_session_uid (ADR-004) + optimizer: kaizen-agentic metrics optimize + evidence: kaizen-agentic metrics publish (optional) +``` + +Full specification: [agents/agent-tdd-workflow.md](../../../agents/agent-tdd-workflow.md). +Pilot documentation: [wiki/AboutKaizenAgents.md](../../../wiki/AboutKaizenAgents.md). \ No newline at end of file diff --git a/docs/integrations/canon-template-mapping.md b/docs/integrations/canon-template-mapping.md new file mode 100644 index 0000000..73d65ca --- /dev/null +++ b/docs/integrations/canon-template-mapping.md @@ -0,0 +1,32 @@ +# KaizenAgentTemplate → InfoTechCanon Profile Mapping + +Design note (WP-0004 Part 4). No runtime dependency on info-tech-canon. + +## Section mapping + +| `wiki/KaizenAgentTemplate.md` | InfoTechCanon profile outline | +|------------------------------|------------------------------| +| `specification.outcomes` | `profile.intent.outcomes[]` | +| `specification.constraints` | `profile.constraints.hard[]` / `soft[]` | +| `idempotency.detection` | `profile.idempotency.signals[]` | +| `idempotency.rollback` | `profile.safety.rollback` | +| `metrics.primary` | `profile.metrics.primary` | +| `metrics.secondary[]` | `profile.metrics.secondary[]` | +| `metrics.collection` | `profile.observability.collection` | +| `testing.unit_tests[]` | `profile.validation.unit[]` | +| `testing.integration_tests[]` | `profile.validation.integration[]` | +| `evolution.history` | `profile.evolution.changelog` | +| `evolution.optimization_hooks` | `profile.evolution.feedback_sources[]` | + +## Validation hooks (future) + +Extend `kaizen-agentic validate` to check: + +1. Frontmatter contains `metrics.primary.name` when `memory: enabled` +2. Session-close block references `metrics record` +3. Required template sections present in agent body (warn, not fail) + +## Reference pilot + +`tdd-workflow` brief in [briefs/tdd-workflow-canon-brief.md](briefs/tdd-workflow-canon-brief.md) +demonstrates a compact canon-style export derived from the full agent spec. \ No newline at end of file diff --git a/docs/integrations/helix-forge-correlation.md b/docs/integrations/helix-forge-correlation.md new file mode 100644 index 0000000..bc797ea --- /dev/null +++ b/docs/integrations/helix-forge-correlation.md @@ -0,0 +1,103 @@ +# Helix Forge Correlation Contract + +Cross-repo contract between **kaizen-agentic** (project metrics, ADR-004) and +**agentic-resources** (Helix Forge fleet session metrics). + +## Purpose + +Link a project-scoped agent execution record to the fleet session that produced +it, without duplicating session JSONL ingestion in kaizen-agentic. + +## Layers + +| Layer | Owner | Storage | +|-------|-------|---------| +| Project | kaizen-agentic | `.kaizen/metrics//executions.jsonl` | +| Fleet | agentic-resources | Helix Forge digest store (`digests` table) | + +## Correlation fields (ADR-004) + +Optional on each project execution record: + +```json +{ + "helix_session_uid": "claude:17092961-…", + "repo": "kaizen-agentic", + "flavor": "claude", + "tokens": 12500, + "infra_overhead_share": 0.12 +} +``` + +### Field mapping + +| Helix Forge (`session_memory`) | ADR-004 project record | +|-------------------------------|------------------------| +| `Session.session_uid` | `helix_session_uid` | +| `Session.repo` | `repo` | +| `Session.flavor` | `flavor` | +| `digest.cost.input_tokens + output_tokens` | `tokens` | +| MCP tool share of `tool_histogram` | `infra_overhead_share` | +| `digest.outcome == "success"` | informs `success` at record time | +| `digest.cost.wall_clock_s` | complements `execution_time_s` | + +## Population at session close + +### Automatic (environment) + +When Helix Forge capture is active in the same shell session: + +```bash +export HELIX_SESSION_UID="claude:17092961-…" +export HELIX_REPO="kaizen-agentic" +export HELIX_FLAVOR="claude" +export HELIX_TOKENS="12500" +export HELIX_INFRA_OVERHEAD_SHARE="0.12" + +kaizen-agentic metrics record tdd-workflow --success --time 4200 --quality 0.9 +``` + +`metrics record` merges env vars into the execution record before append. + +### Explicit (JSON) + +```bash +echo '{ + "success": true, + "execution_time_s": 4200, + "quality_score": 0.9, + "helix_session_uid": "claude:17092961-…", + "repo": "kaizen-agentic", + "flavor": "claude", + "tokens": 12500, + "infra_overhead_share": 0.12 +}' | kaizen-agentic metrics record tdd-workflow --json +``` + +## Fleet lookup (read-only) + +```bash +export HELIX_STORE_DB=~/.helix-forge/store.db # agentic-resources session store +kaizen-agentic metrics correlate claude:17092961-… +``` + +When `HELIX_STORE_DB` is unset, `metrics correlate` returns a **stub** response +documenting expected fields — no ingestion code runs in kaizen-agentic. + +## Bidirectional references + +| Document | Repo | +|----------|------| +| [ADR-004](../adr/ADR-004-project-metrics-convention.md) | kaizen-agentic | +| [wiki/EcosystemIntegration.md](../../wiki/EcosystemIntegration.md) | kaizen-agentic | +| [DESIGN-session-memory.md](https://github.com/coulomb/agentic-resources/blob/main/docs/DESIGN-session-memory.md) | agentic-resources | +| `session_memory/core/store.py` — `get_digest()` | agentic-resources | + +agentic-resources should link back to this document from its session-memory design +notes when documenting downstream consumers of `session_uid`. + +## Non-goals + +- No Claude/Codex/Grok JSONL ingestion in kaizen-agentic +- No write path to Helix Forge from kaizen-agentic CLI +- No merge of fleet baselines into project `summary.json` (Coach may cite both) \ No newline at end of file diff --git a/docs/integrations/kontextual-wiki-ingestion-spike.md b/docs/integrations/kontextual-wiki-ingestion-spike.md new file mode 100644 index 0000000..dce7c10 --- /dev/null +++ b/docs/integrations/kontextual-wiki-ingestion-spike.md @@ -0,0 +1,41 @@ +# kontextual-engine Wiki Ingestion Spike + +Design note (WP-0004 Part 4). No runtime dependency. + +## Proposed manifest + +```yaml +ingestion: + source_repo: kaizen-agentic + asset_class: strategic-knowledge + paths: + - wiki/**/*.md + - INTENT.md + - docs/adr/ADR-*.md + exclude: + - wiki/**/xxx + metadata: + domain: custodian + topic_id: cee7bedf-2b48-46ef-8601-006474f2ad7a + producer: kaizen-agentic + refresh: + trigger: git-push-main + retention_class: operational-knowledge +``` + +## Rationale + +- `wiki/` holds product narrative and integration contracts not suited for agent prompts alone +- ADRs are normative; kontextual-engine can index them for cross-repo retrieval +- Agent definitions (`agents/`) remain separate — executable personas vs strategic docs + +## Open questions + +1. Chunking strategy for `KaizenAgentTemplate.md` (section-aware vs whole-file) +2. Whether Coach synthesis outputs should be ingested as derived assets +3. Correlation with info-tech-canon profiles when both exist for one agent + +## Next step + +Dedicated workplan after WP-0004 baseline; evaluate kontextual-engine ingestion API +stability before hard dependency. \ No newline at end of file diff --git a/docs/integrations/optimizer-artifact-manifest.md b/docs/integrations/optimizer-artifact-manifest.md new file mode 100644 index 0000000..b44c143 --- /dev/null +++ b/docs/integrations/optimizer-artifact-manifest.md @@ -0,0 +1,60 @@ +# Optimizer Evidence Artifact Manifest + +Package schema for `kaizen-agentic metrics publish` → **artifact-store**. + +## Package identity + +| Field | Value | +|-------|-------| +| `producer` | `kaizen-agentic` | +| `retention_class` | `raw-evidence` (180d default, ADR-004 aligned) | +| `name` | `kaizen-optimizer-` | +| `subject` | project directory name (override with `--subject`) | + +## Files + +| Relative path | Source | Media type | +|---------------|--------|------------| +| `optimizer/analysis.json` | `.kaizen/metrics/optimizer/analysis.json` | `application/json` | +| `optimizer/recommendations.jsonl` | `.kaizen/metrics/optimizer/recommendations.jsonl` | `application/x-ndjson` | + +`recommendations.jsonl` is omitted from upload when absent (e.g. insufficient samples). + +## Metadata (`POST /packages`) + +```json +{ + "schema": "kaizen-agentic/optimizer-evidence/v1", + "project": "demo-app", + "project_root": "/path/to/demo-app", + "producer": "kaizen-agentic", + "retention_class": "raw-evidence", + "retention_days": 180, + "optimized_at": "2026-06-18", + "agents": ["tdd-workflow", "coach"], + "files": [ + "optimizer/analysis.json", + "optimizer/recommendations.jsonl" + ] +} +``` + +## Publish workflow + +```bash +# 1. Ensure optimizer has run +kaizen-agentic metrics optimize + +# 2. Publish (artifact-store must be reachable) +export ARTIFACTSTORE_API_URL=http://127.0.0.1:8000 +export ARTIFACTSTORE_API_TOKEN= +kaizen-agentic metrics publish --target . +``` + +Local-only workflows skip publish; `.kaizen/metrics/optimizer/` remains authoritative. + +## Related + +- [artifact-store ingestion API](https://github.com/coulomb/artifact-store) — `POST /packages`, `/files`, `/finalize` +- [ADR-004](../adr/ADR-004-project-metrics-convention.md) +- [INTEGRATION_PATTERNS.md](../INTEGRATION_PATTERNS.md) \ No newline at end of file diff --git a/history/2026-06-16-ecosystem-assessment.md b/history/2026-06-16-ecosystem-assessment.md index 99ed7f7..197b569 100644 --- a/history/2026-06-16-ecosystem-assessment.md +++ b/history/2026-06-16-ecosystem-assessment.md @@ -138,5 +138,35 @@ See `wiki/EcosystemIntegration.md` for integration contracts. ## Follow-Up Workplans -- **KAIZEN-WP-0003** — measurement loop (active, State Hub registered) -- **KAIZEN-WP-0004** — ecosystem integration (active, depends on WP-0003 Part 3) \ No newline at end of file +- **KAIZEN-WP-0003** — measurement loop (completed 2026-06-18) +- **KAIZEN-WP-0004** — ecosystem integration (completed 2026-06-18) + +--- + +## WP-0004 Outcomes (2026-06-18) + +### Part 1 — Helix Forge correlation + +- `HELIX_SESSION_UID` env auto-merge on `metrics record` +- `kaizen-agentic metrics correlate ` read-only adapter (sqlite or stub) +- Contract: `docs/integrations/helix-forge-correlation.md` +- Worked example in `wiki/EcosystemIntegration.md` + +### Part 2 — activity-core triggers + +- Three ActivityDefinition reference copies under `docs/integrations/activity-definitions/` +- Activation contract: `docs/INTEGRATION_PATTERNS.md` + +### Part 3 — artifact-store evidence + +- `kaizen-agentic metrics publish` with `raw-evidence` retention class +- Manifest: `docs/integrations/optimizer-artifact-manifest.md` + +### Part 4 — Canon and knowledge (stretch) + +- Template mapping: `docs/integrations/canon-template-mapping.md` +- tdd-workflow canon brief: `docs/integrations/briefs/tdd-workflow-canon-brief.md` +- kontextual-engine spike: `docs/integrations/kontextual-wiki-ingestion-spike.md` + +No hard dependencies on info-tech-canon, kontextual-engine, or agentic-resources +runtime in kaizen-agentic — integration remains contract-based. \ No newline at end of file diff --git a/src/kaizen_agentic/cli.py b/src/kaizen_agentic/cli.py index 11a2620..b710f76 100644 --- a/src/kaizen_agentic/cli.py +++ b/src/kaizen_agentic/cli.py @@ -11,6 +11,12 @@ from typing import List, Optional from .registry import AgentRegistry, AgentCategory from .installer import AgentInstaller, ProjectInitializer, InstallationConfig +from .integrations.artifact_store import ( + default_api_token, + default_api_url, + publish_optimizer_evidence, +) +from .integrations.helix import HelixCorrelationAdapter, enrich_helix_correlation from .metrics import MetricsStore, OptimizerStore, performance_summary_markdown from .optimization import OptimizationLoop, MIN_SAMPLES_FOR_RECOMMENDATIONS @@ -999,6 +1005,8 @@ def metrics_record( if session_id: payload["session_id"] = session_id + payload = enrich_helix_correlation(payload) + if store.append(payload, idempotency_key=idempotency_key): click.echo(f"Recorded metrics for '{agent_name}'") else: @@ -1106,6 +1114,84 @@ def metrics_optimize(agent_name: Optional[str], target: str, min_samples: int): click.echo(f"Wrote optimizer analysis: {analysis_path}") +@metrics.command("correlate") +@click.argument("session_uid") +@click.option( + "--store-db", + envvar="HELIX_STORE_DB", + help="Helix Forge session-memory SQLite database path", +) +def metrics_correlate(session_uid: str, store_db: Optional[str]): + """Look up Helix Forge digest summary for a session UID (read-only).""" + adapter = HelixCorrelationAdapter( + store_db=Path(store_db).resolve() if store_db else None + ) + if adapter.store_db is None: + adapter = HelixCorrelationAdapter.from_env() + summary = adapter.lookup(session_uid) + click.echo(json.dumps(summary, indent=2, sort_keys=True)) + + +@metrics.command("publish") +@click.option("--target", "-t", default=".", help="Project root (default: current)") +@click.option( + "--api-url", + default=default_api_url, + show_default=True, + help="artifact-store API base URL (ARTIFACTSTORE_API_URL)", +) +@click.option( + "--token", + default=default_api_token, + help="artifact-store bearer token (ARTIFACTSTORE_API_TOKEN)", +) +@click.option( + "--subject", + help="Package subject (default: project directory name)", +) +@click.option( + "--retention-class", + default="raw-evidence", + show_default=True, + help="artifact-store retention class", +) +def metrics_publish( + target: str, + api_url: str, + token: str, + subject: Optional[str], + retention_class: str, +): + """Publish optimizer evidence to artifact-store (optional integration).""" + project_root = _project_root(target) + if not token: + click.echo( + "Error: artifact-store token required. Set ARTIFACTSTORE_API_TOKEN or --token.", + err=True, + ) + sys.exit(1) + try: + result = publish_optimizer_evidence( + project_root, + api_url=api_url, + token=token, + subject=subject, + retention_class=retention_class, + ) + except FileNotFoundError as exc: + click.echo(f"Error: {exc}", err=True) + sys.exit(1) + except RuntimeError as exc: + click.echo(f"Error: {exc}", err=True) + sys.exit(1) + + click.echo(f"Published optimizer evidence package: {result.package_id}") + click.echo(f" Files uploaded: {result.files_uploaded}") + click.echo(f" Retention class: {result.retention_class}") + if result.manifest_digest: + click.echo(f" Manifest digest: {result.manifest_digest}") + + @metrics.command("export") @click.argument("agent_name") @click.option("--target", "-t", default=".", help="Project root (default: current)") diff --git a/src/kaizen_agentic/integrations/__init__.py b/src/kaizen_agentic/integrations/__init__.py new file mode 100644 index 0000000..dcc65aa --- /dev/null +++ b/src/kaizen_agentic/integrations/__init__.py @@ -0,0 +1,10 @@ +"""Ecosystem integration adapters (Helix Forge, artifact-store).""" + +from .artifact_store import publish_optimizer_evidence +from .helix import HelixCorrelationAdapter, enrich_helix_correlation + +__all__ = [ + "HelixCorrelationAdapter", + "enrich_helix_correlation", + "publish_optimizer_evidence", +] \ No newline at end of file diff --git a/src/kaizen_agentic/integrations/artifact_store.py b/src/kaizen_agentic/integrations/artifact_store.py new file mode 100644 index 0000000..5d16875 --- /dev/null +++ b/src/kaizen_agentic/integrations/artifact_store.py @@ -0,0 +1,233 @@ +"""artifact-store publish adapter for optimizer evidence (WP-0004 Part 3).""" + +from __future__ import annotations + +import json +import os +import uuid +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Dict, List, Optional +from urllib import error, parse, request + +from ..metrics import OptimizerStore + +ENV_API_URL = "ARTIFACTSTORE_API_URL" +ENV_API_TOKEN = "ARTIFACTSTORE_API_TOKEN" +DEFAULT_RETENTION_CLASS = "raw-evidence" +PRODUCER = "kaizen-agentic" + + +@dataclass +class PublishResult: + package_id: str + manifest_digest: Optional[str] + files_uploaded: int + retention_class: str + + +def build_optimizer_manifest( + project_root: Path, + *, + agents: Optional[List[str]] = None, +) -> Dict[str, Any]: + """Manifest metadata for an optimizer evidence package.""" + store = OptimizerStore(project_root) + analysis = {} + if store.analysis_path.exists(): + analysis = json.loads(store.analysis_path.read_text(encoding="utf-8")) + + return { + "schema": "kaizen-agentic/optimizer-evidence/v1", + "project": project_root.name, + "project_root": str(project_root.resolve()), + "producer": PRODUCER, + "retention_class": DEFAULT_RETENTION_CLASS, + "retention_days": 180, + "optimized_at": analysis.get("optimized_at"), + "agents": agents or [item.get("agent") for item in analysis.get("agents", [])], + "files": [ + "optimizer/analysis.json", + "optimizer/recommendations.jsonl", + ], + } + + +def publish_optimizer_evidence( + project_root: Path, + *, + api_url: str, + token: str, + subject: Optional[str] = None, + retention_class: str = DEFAULT_RETENTION_CLASS, +) -> PublishResult: + """Register optimizer outputs as an artifact-store package.""" + store = OptimizerStore(project_root) + if not store.analysis_path.exists(): + raise FileNotFoundError( + f"No optimizer analysis at {store.analysis_path}. " + "Run: kaizen-agentic metrics optimize" + ) + + manifest = build_optimizer_manifest(project_root) + package_name = f"kaizen-optimizer-{project_root.name}" + package_subject = subject or project_root.name + + created = _http_json( + "POST", + api_url, + "/packages", + token, + { + "name": package_name, + "producer": PRODUCER, + "subject": package_subject, + "retention_class": retention_class, + "metadata": manifest, + }, + ) + package_id = created["id"] + + uploads = [ + ( + store.analysis_path, + "optimizer/analysis.json", + "application/json", + ), + ] + if store.recommendations_path.exists(): + uploads.append( + ( + store.recommendations_path, + "optimizer/recommendations.jsonl", + "application/x-ndjson", + ) + ) + + for path, relative_path, media_type in uploads: + _http_multipart( + api_url, + f"/packages/{package_id}/files", + token, + fields={"relative_path": relative_path, "media_type": media_type}, + file_field="file", + file_name=path.name, + file_content_type=media_type, + file_bytes=path.read_bytes(), + ) + + finalized = _http_json( + "POST", + api_url, + f"/packages/{package_id}/finalize", + token, + {}, + ) + + return PublishResult( + package_id=package_id, + manifest_digest=finalized.get("manifest_digest"), + files_uploaded=len(uploads), + retention_class=retention_class, + ) + + +def default_api_url() -> str: + return os.environ.get(ENV_API_URL, "http://127.0.0.1:8000").rstrip("/") + + +def default_api_token() -> str: + return os.environ.get(ENV_API_TOKEN, "") + + +def _http_json( + method: str, + base_url: str, + path: str, + token: str, + payload: Dict[str, Any], +) -> Dict[str, Any]: + body = json.dumps(payload).encode("utf-8") if payload else None + headers = {"Accept": "application/json"} + if body is not None: + headers["Content-Type"] = "application/json" + response = _http_bytes(method, base_url, path, token, body=body, headers=headers) + decoded = json.loads(response) + if not isinstance(decoded, dict): + raise ValueError(f"expected JSON object from {path}") + return decoded + + +def _http_multipart( + base_url: str, + path: str, + token: str, + *, + fields: Dict[str, str], + file_field: str, + file_name: str, + file_content_type: str, + file_bytes: bytes, +) -> Dict[str, Any]: + boundary = f"kaizen-{uuid.uuid4().hex}" + body = bytearray() + for name, value in fields.items(): + body.extend(f"--{boundary}\r\n".encode("ascii")) + body.extend( + f'Content-Disposition: form-data; name="{_quote(name)}"\r\n\r\n'.encode() + ) + body.extend(value.encode()) + body.extend(b"\r\n") + body.extend(f"--{boundary}\r\n".encode("ascii")) + body.extend( + ( + f'Content-Disposition: form-data; name="{_quote(file_field)}"; ' + f'filename="{_quote(file_name)}"\r\n' + f"Content-Type: {file_content_type}\r\n\r\n" + ).encode() + ) + body.extend(file_bytes) + body.extend(b"\r\n") + body.extend(f"--{boundary}--\r\n".encode("ascii")) + + response = _http_bytes( + "POST", + base_url, + path, + token, + body=bytes(body), + headers={ + "Content-Type": f"multipart/form-data; boundary={boundary}", + "Accept": "application/json", + }, + ) + decoded = json.loads(response) + if not isinstance(decoded, dict): + raise ValueError(f"expected JSON object from {path}") + return decoded + + +def _http_bytes( + method: str, + base_url: str, + path: str, + token: str, + *, + body: Optional[bytes] = None, + headers: Optional[Dict[str, str]] = None, +) -> bytes: + url = f"{base_url.rstrip('/')}/{path.lstrip('/')}" + effective_headers = dict(headers or {}) + if token: + effective_headers["Authorization"] = f"Bearer {token}" + req = request.Request(url, data=body, headers=effective_headers, method=method) + try: + with request.urlopen(req, timeout=30) as resp: + return resp.read() + except error.HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace") + raise RuntimeError(f"HTTP {exc.code} from {path}: {detail}") from exc + + +def _quote(value: str) -> str: + return parse.quote(value, safe="") \ No newline at end of file diff --git a/src/kaizen_agentic/integrations/helix.py b/src/kaizen_agentic/integrations/helix.py new file mode 100644 index 0000000..9bdc931 --- /dev/null +++ b/src/kaizen_agentic/integrations/helix.py @@ -0,0 +1,171 @@ +"""Helix Forge correlation adapter (ADR-004, agentic-resources).""" + +from __future__ import annotations + +import json +import os +import sqlite3 +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Dict, Optional + + +ENV_SESSION_UID = "HELIX_SESSION_UID" +ENV_REPO = "HELIX_REPO" +ENV_FLAVOR = "HELIX_FLAVOR" +ENV_TOKENS = "HELIX_TOKENS" +ENV_INFRA_SHARE = "HELIX_INFRA_OVERHEAD_SHARE" +ENV_STORE_DB = "HELIX_STORE_DB" + + +def enrich_helix_correlation(record: Dict[str, Any]) -> Dict[str, Any]: + """Apply optional Helix correlation fields from env or existing record.""" + payload = dict(record) + + uid = payload.get("helix_session_uid") or os.environ.get(ENV_SESSION_UID) + if uid: + payload["helix_session_uid"] = uid + + repo = payload.get("repo") or os.environ.get(ENV_REPO) + if repo: + payload["repo"] = repo + + flavor = payload.get("flavor") or os.environ.get(ENV_FLAVOR) + if flavor: + payload["flavor"] = flavor + + tokens_raw = payload.get("tokens") + if tokens_raw is None and ENV_TOKENS in os.environ: + try: + tokens_raw = int(os.environ[ENV_TOKENS]) + except ValueError: + pass + if tokens_raw is not None: + payload["tokens"] = int(tokens_raw) + + infra = payload.get("infra_overhead_share") + if infra is None and ENV_INFRA_SHARE in os.environ: + try: + infra = float(os.environ[ENV_INFRA_SHARE]) + except ValueError: + pass + if infra is not None: + payload["infra_overhead_share"] = float(infra) + + return payload + + +def digest_to_correlation_summary( + session_uid: str, + digest: Dict[str, Any], + *, + adapter: str, +) -> Dict[str, Any]: + """Project a Helix digest into ADR-004 correlation summary fields.""" + cost = digest.get("cost") or {} + input_tokens = int(cost.get("input_tokens") or 0) + output_tokens = int(cost.get("output_tokens") or 0) + wall_clock_s = cost.get("wall_clock_s") + + summary: Dict[str, Any] = { + "helix_session_uid": session_uid, + "repo": digest.get("repo"), + "flavor": digest.get("flavor"), + "fleet_outcome": digest.get("outcome"), + "tokens": input_tokens + output_tokens, + "adapter": adapter, + } + if wall_clock_s is not None: + summary["wall_clock_s"] = float(wall_clock_s) + + markers = digest.get("markers") or {} + tool_histogram = digest.get("tool_histogram") or {} + mcp_calls = sum( + count for tool, count in tool_histogram.items() if str(tool).startswith("mcp__") + ) + total_calls = sum(tool_histogram.values()) or 0 + if total_calls: + summary["infra_overhead_share"] = round(mcp_calls / total_calls, 3) + elif "infra_overhead_share" in digest: + summary["infra_overhead_share"] = digest["infra_overhead_share"] + + if markers: + summary["markers"] = { + key: markers[key] + for key in ("errors", "retries", "test_runs") + if key in markers + } + + return summary + + +@dataclass +class HelixCorrelationAdapter: + """Read-only lookup of Helix Forge session digests.""" + + store_db: Optional[Path] = None + + @classmethod + def from_env(cls) -> "HelixCorrelationAdapter": + raw = os.environ.get(ENV_STORE_DB) + return cls(store_db=Path(raw).resolve() if raw else None) + + def lookup(self, session_uid: str) -> Dict[str, Any]: + if self.store_db and self.store_db.exists(): + digest = self._load_digest_sqlite(session_uid) + if digest is not None: + return digest_to_correlation_summary( + session_uid, + digest, + adapter="helix-sqlite", + ) + return { + "helix_session_uid": session_uid, + "adapter": "helix-sqlite", + "status": "not_found", + "message": f"No digest for session_uid in {self.store_db}", + } + + return { + "helix_session_uid": session_uid, + "adapter": "stub", + "status": "not_configured", + "message": ( + "Set HELIX_STORE_DB to an agentic-resources session-memory SQLite " + "database for live lookup. Correlation fields on project metrics " + "still work via HELIX_SESSION_UID at record time." + ), + "expected_fields": [ + "helix_session_uid", + "repo", + "flavor", + "tokens", + "infra_overhead_share", + "fleet_outcome", + "wall_clock_s", + ], + } + + def _load_digest_sqlite(self, session_uid: str) -> Optional[Dict[str, Any]]: + conn = sqlite3.connect(str(self.store_db)) + try: + row = conn.execute( + "SELECT json FROM digests WHERE session_uid = ?", + (session_uid,), + ).fetchone() + if not row: + return None + digest = json.loads(row[0]) + digest.setdefault("session_uid", session_uid) + + session_row = conn.execute( + "SELECT json FROM sessions WHERE session_uid = ?", + (session_uid,), + ).fetchone() + if session_row: + session = json.loads(session_row[0]) + digest.setdefault("repo", session.get("repo")) + digest.setdefault("flavor", session.get("flavor")) + return digest + finally: + conn.close() \ No newline at end of file diff --git a/tests/test_helix_correlation.py b/tests/test_helix_correlation.py new file mode 100644 index 0000000..ed15b36 --- /dev/null +++ b/tests/test_helix_correlation.py @@ -0,0 +1,161 @@ +"""Tests for Helix Forge correlation (WP-0004 Part 1).""" + +from __future__ import annotations + +import json +import os +import sqlite3 +from pathlib import Path + +import pytest +from click.testing import CliRunner + +from kaizen_agentic.cli import cli +from kaizen_agentic.integrations.helix import ( + HelixCorrelationAdapter, + enrich_helix_correlation, +) + + +def test_enrich_helix_correlation_from_env(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("HELIX_SESSION_UID", "claude:test-uid") + monkeypatch.setenv("HELIX_REPO", "kaizen-agentic") + monkeypatch.setenv("HELIX_FLAVOR", "claude") + monkeypatch.setenv("HELIX_TOKENS", "9900") + monkeypatch.setenv("HELIX_INFRA_OVERHEAD_SHARE", "0.15") + + result = enrich_helix_correlation({"success": True}) + + assert result["helix_session_uid"] == "claude:test-uid" + assert result["repo"] == "kaizen-agentic" + assert result["flavor"] == "claude" + assert result["tokens"] == 9900 + assert result["infra_overhead_share"] == 0.15 + + +def test_enrich_does_not_override_existing_fields(): + record = { + "success": True, + "helix_session_uid": "grok:existing", + "repo": "other-repo", + } + result = enrich_helix_correlation(record) + assert result["helix_session_uid"] == "grok:existing" + assert result["repo"] == "other-repo" + + +def test_adapter_stub_when_store_unconfigured(): + adapter = HelixCorrelationAdapter(store_db=None) + summary = adapter.lookup("claude:missing") + assert summary["adapter"] == "stub" + assert summary["status"] == "not_configured" + + +def test_adapter_sqlite_lookup(tmp_path: Path): + db_path = tmp_path / "store.db" + conn = sqlite3.connect(db_path) + conn.execute( + "CREATE TABLE digests (session_uid TEXT PRIMARY KEY, json TEXT NOT NULL)" + ) + conn.execute( + "CREATE TABLE sessions (session_uid TEXT PRIMARY KEY, json TEXT NOT NULL)" + ) + digest = { + "outcome": "success", + "cost": {"input_tokens": 800, "output_tokens": 200, "wall_clock_s": 3600}, + "tool_histogram": {"mcp__state-hub__x": 3, "Bash": 7}, + "markers": {"errors": 0, "retries": 1}, + } + session = {"repo": "demo-app", "flavor": "claude"} + conn.execute( + "INSERT INTO digests VALUES (?, ?)", + ("claude:abc", json.dumps(digest)), + ) + conn.execute( + "INSERT INTO sessions VALUES (?, ?)", + ("claude:abc", json.dumps(session)), + ) + conn.commit() + conn.close() + + adapter = HelixCorrelationAdapter(store_db=db_path) + summary = adapter.lookup("claude:abc") + + assert summary["adapter"] == "helix-sqlite" + assert summary["repo"] == "demo-app" + assert summary["flavor"] == "claude" + assert summary["fleet_outcome"] == "success" + assert summary["tokens"] == 1000 + assert summary["wall_clock_s"] == 3600 + assert summary["infra_overhead_share"] == 0.3 + + +class TestHelixCorrelationCli: + def test_record_populates_helix_uid_from_env( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ): + monkeypatch.setenv("HELIX_SESSION_UID", "claude:session-42") + monkeypatch.setenv("HELIX_REPO", "kaizen-agentic") + + runner = CliRunner() + result = runner.invoke( + cli, + [ + "metrics", + "record", + "tdd-workflow", + "--target", + str(tmp_path), + "--success", + "--time", + "10", + ], + ) + assert result.exit_code == 0 + + show = runner.invoke( + cli, + ["metrics", "show", "tdd-workflow", "--target", str(tmp_path)], + ) + assert "claude:session-42" in show.output + assert "kaizen-agentic" in show.output + + def test_correlate_stub_output(self): + runner = CliRunner() + result = runner.invoke(cli, ["metrics", "correlate", "claude:stub-uid"]) + assert result.exit_code == 0 + payload = json.loads(result.output) + assert payload["helix_session_uid"] == "claude:stub-uid" + assert payload["adapter"] == "stub" + + def test_brief_works_with_correlated_metrics( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ): + memory_dir = tmp_path / ".kaizen" / "agents" / "tdd-workflow" + memory_dir.mkdir(parents=True) + (memory_dir / "memory.md").write_text( + "---\nagent: tdd-workflow\nproject: demo\nsession_count: 1\n---\n\n## Session Log\n", + encoding="utf-8", + ) + monkeypatch.setenv("HELIX_SESSION_UID", "claude:brief-test") + + runner = CliRunner() + runner.invoke( + cli, + [ + "metrics", + "record", + "tdd-workflow", + "--target", + str(tmp_path), + "--success", + "--quality", + "0.9", + ], + ) + brief = runner.invoke( + cli, + ["memory", "brief", "tdd-workflow", "--target", str(tmp_path)], + ) + assert brief.exit_code == 0 + assert "## Performance Summary" in brief.output \ No newline at end of file diff --git a/tests/test_integration_patterns.py b/tests/test_integration_patterns.py new file mode 100644 index 0000000..5758fb4 --- /dev/null +++ b/tests/test_integration_patterns.py @@ -0,0 +1,33 @@ +"""Smoke tests for WP-0004 integration artifacts.""" + +from __future__ import annotations + +from pathlib import Path + +import yaml + + +DEFINITIONS_DIR = ( + Path(__file__).parent.parent / "docs" / "integrations" / "activity-definitions" +) + + +def test_activity_definitions_have_required_frontmatter(): + files = list(DEFINITIONS_DIR.glob("*.md")) + assert len(files) == 3 + + for path in files: + text = path.read_text(encoding="utf-8") + assert text.startswith("---\n") + end = text.index("\n---\n", 4) + frontmatter = yaml.safe_load(text[4:end]) + assert frontmatter["id"] + assert frontmatter["trigger"]["type"] in ("cron", "event") + assert frontmatter["owner"] == "kaizen-agentic" + + +def test_integration_docs_exist(): + root = Path(__file__).parent.parent / "docs" + assert (root / "INTEGRATION_PATTERNS.md").exists() + assert (root / "integrations" / "helix-forge-correlation.md").exists() + assert (root / "integrations" / "optimizer-artifact-manifest.md").exists() \ No newline at end of file diff --git a/tests/test_metrics_publish.py b/tests/test_metrics_publish.py new file mode 100644 index 0000000..db3fdb8 --- /dev/null +++ b/tests/test_metrics_publish.py @@ -0,0 +1,143 @@ +"""Tests for artifact-store publish integration (WP-0004 Part 3).""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import patch + +import pytest +from click.testing import CliRunner + +from kaizen_agentic.cli import cli +from kaizen_agentic.integrations.artifact_store import ( + PublishResult, + build_optimizer_manifest, + publish_optimizer_evidence, +) +from kaizen_agentic.metrics import OptimizerStore + + +@pytest.fixture +def project_with_optimizer(tmp_path: Path) -> Path: + store = OptimizerStore(tmp_path) + store.write_analysis( + { + "project": "demo", + "optimized_at": "2026-06-18", + "agents": [{"agent": "tdd-workflow"}], + } + ) + store.append_recommendations( + "tdd-workflow", + [{"type": "reliability", "message": "Improve test stability"}], + metrics_count=10, + ) + return tmp_path + + +def test_build_optimizer_manifest(project_with_optimizer: Path): + manifest = build_optimizer_manifest(project_with_optimizer) + assert manifest["schema"] == "kaizen-agentic/optimizer-evidence/v1" + assert manifest["retention_class"] == "raw-evidence" + assert manifest["retention_days"] == 180 + assert "tdd-workflow" in manifest["agents"] + + +def test_publish_optimizer_evidence_calls_api(project_with_optimizer: Path): + calls: list[tuple[str, str]] = [] + + def fake_json(method, base_url, path, token, payload): + calls.append((method, path)) + if path == "/packages": + return {"id": "pkg-123"} + if path.endswith("/finalize"): + return {"id": "pkg-123", "manifest_digest": "blake3:deadbeef"} + raise AssertionError(path) + + def fake_multipart(base_url, path, token, **kwargs): + calls.append(("POST", path)) + return {"id": "file-1"} + + with patch( + "kaizen_agentic.integrations.artifact_store._http_json", + side_effect=fake_json, + ), patch( + "kaizen_agentic.integrations.artifact_store._http_multipart", + side_effect=fake_multipart, + ): + result = publish_optimizer_evidence( + project_with_optimizer, + api_url="http://api.test", + token="secret", + ) + + assert result.package_id == "pkg-123" + assert result.files_uploaded == 2 + assert result.retention_class == "raw-evidence" + assert calls[0] == ("POST", "/packages") + assert any("/files" in path for _, path in calls) + assert calls[-1] == ("POST", "/packages/pkg-123/finalize") + + +class TestMetricsPublishCli: + def test_publish_requires_token(self, project_with_optimizer: Path): + runner = CliRunner() + result = runner.invoke( + cli, + ["metrics", "publish", "--target", str(project_with_optimizer)], + ) + assert result.exit_code != 0 + assert "token" in result.output.lower() + + def test_publish_success(self, project_with_optimizer: Path): + runner = CliRunner() + with patch( + "kaizen_agentic.cli.publish_optimizer_evidence", + return_value=PublishResult( + package_id="pkg-99", + manifest_digest="blake3:abc", + files_uploaded=2, + retention_class="raw-evidence", + ), + ): + result = runner.invoke( + cli, + [ + "metrics", + "publish", + "--target", + str(project_with_optimizer), + "--token", + "test-token", + "--api-url", + "http://127.0.0.1:8000", + ], + ) + assert result.exit_code == 0 + assert "pkg-99" in result.output + + +@pytest.mark.integration +def test_publish_against_live_artifact_store(project_with_optimizer: Path): + """Optional live test — skipped when artifact-store is unreachable.""" + import urllib.error + import urllib.request + + api_url = "http://127.0.0.1:8000" + try: + urllib.request.urlopen(f"{api_url}/health", timeout=2) + except (urllib.error.URLError, TimeoutError): + pytest.skip("artifact-store not reachable") + + token = __import__("os").environ.get("ARTIFACTSTORE_API_TOKEN") + if not token: + pytest.skip("ARTIFACTSTORE_API_TOKEN not set") + + result = publish_optimizer_evidence( + project_with_optimizer, + api_url=api_url, + token=token, + ) + assert result.package_id + assert result.files_uploaded >= 1 \ No newline at end of file diff --git a/wiki/EcosystemIntegration.md b/wiki/EcosystemIntegration.md index cda19b2..1881f6b 100644 --- a/wiki/EcosystemIntegration.md +++ b/wiki/EcosystemIntegration.md @@ -58,17 +58,45 @@ Helix session UIDs on project execution records for correlation. **Workplan:** KAIZEN-WP-0004 Part 1. +#### Worked example + +A TDD8 session captured by Helix Forge and closed with kaizen metrics: + +```bash +# Helix capture sets (or operator exports) session identity +export HELIX_SESSION_UID="claude:17092961-abc" +export HELIX_REPO="kaizen-agentic" +export HELIX_FLAVOR="claude" +export HELIX_TOKENS="12500" + +# Session close — project layer +kaizen-agentic metrics record tdd-workflow --success --time 4200 --quality 0.92 + +# Inspect project record (includes correlation fields) +kaizen-agentic metrics show tdd-workflow + +# Fleet lookup — read-only, no ingestion in kaizen-agentic +export HELIX_STORE_DB=~/.helix-forge/store.db +kaizen-agentic metrics correlate claude:17092961-abc +``` + +Project `executions.jsonl` carries `helix_session_uid` for audit; fleet analytics +remain in agentic-resources digest store. Coach `memory brief` surfaces project +`## Performance Summary`; correlate adds fleet context when needed. + +Contract: [docs/integrations/helix-forge-correlation.md](../docs/integrations/helix-forge-correlation.md). + ### activity-core (P1) **Event bridge** — scheduled and event-driven task creation. -Example ActivityDefinitions (after metrics CLI ships): +ActivityDefinition reference copies (sync into activity-core to activate): -- Weekly: run `kaizen-agentic metrics optimize` on repos with `.kaizen/` -- On low success_rate threshold: create review task in issue-core -- Post agent install: verify metrics scaffold exists +- [weekly-metrics-optimize](../docs/integrations/activity-definitions/weekly-metrics-optimize.md) +- [post-install-metrics-scaffold](../docs/integrations/activity-definitions/post-install-metrics-scaffold.md) +- [low-success-rate-review](../docs/integrations/activity-definitions/low-success-rate-review.md) -**Workplan:** KAIZEN-WP-0004 Part 2. +**Workplan:** KAIZEN-WP-0004 Part 2. Patterns: [docs/INTEGRATION_PATTERNS.md](../docs/INTEGRATION_PATTERNS.md). ### artifact-store (P1) @@ -82,6 +110,13 @@ Register after optimizer runs: Retention class: `raw-evidence` (180d default, aligned with ADR-004). +```bash +kaizen-agentic metrics optimize +kaizen-agentic metrics publish # requires ARTIFACTSTORE_API_URL + TOKEN +``` + +Manifest: [docs/integrations/optimizer-artifact-manifest.md](../docs/integrations/optimizer-artifact-manifest.md). + **Workplan:** KAIZEN-WP-0004 Part 3. ### info-tech-canon (P2) diff --git a/workplans/kaizen-agentic-WP-0004-ecosystem-integration.md b/workplans/kaizen-agentic-WP-0004-ecosystem-integration.md index 46ebe11..492b769 100644 --- a/workplans/kaizen-agentic-WP-0004-ecosystem-integration.md +++ b/workplans/kaizen-agentic-WP-0004-ecosystem-integration.md @@ -4,17 +4,17 @@ type: workplan title: "Ecosystem Integration: Helix Forge, activity-core, and artifact-store" domain: custodian repo: kaizen-agentic -status: active +status: completed owner: kaizen-agentic topic_slug: custodian state_hub_workstream_id: 76be7294-e201-4074-91c0-6421992470fe created: "2026-06-16" -updated: "2026-06-17" +updated: "2026-06-18" --- # KAIZEN-WP-0004 — Ecosystem Integration: Helix Forge, activity-core, and artifact-store -**Status:** active +**Status:** completed **Owner:** kaizen-agentic **Repo:** kaizen-agentic **Depends on:** KAIZEN-WP-0003 Part 3 (metrics CLI + `metrics optimize` operational) @@ -40,11 +40,11 @@ session ingestion. ### Tasks -- [ ] T01 — Document correlation contract in `agentic-resources` (cross-repo PR or shared doc link from both repos) -- [ ] T02 — Add optional `helix_session_uid` population to `metrics record` when env `HELIX_SESSION_UID` is set -- [ ] T03 — Add `kaizen-agentic metrics correlate` — lookup Helix digest summary by UID (read-only adapter stub if Helix API not ready) -- [ ] T04 — Integration test: synthetic project record with `helix_session_uid` round-trips through show/brief -- [ ] T05 — Update `wiki/EcosystemIntegration.md` with worked correlation example +- [x] T01 — Document correlation contract in `agentic-resources` (cross-repo PR or shared doc link from both repos) +- [x] T02 — Add optional `helix_session_uid` population to `metrics record` when env `HELIX_SESSION_UID` is set +- [x] T03 — Add `kaizen-agentic metrics correlate` — lookup Helix digest summary by UID (read-only adapter stub if Helix API not ready) +- [x] T04 — Integration test: synthetic project record with `helix_session_uid` round-trips through show/brief +- [x] T05 — Update `wiki/EcosystemIntegration.md` with worked correlation example ### Definition of done @@ -60,11 +60,11 @@ Define ActivityDefinitions for recurring kaizen operations. ### Tasks -- [ ] T06 — Draft ActivityDefinition: weekly `metrics optimize` on repos with `.kaizen/metrics/` -- [ ] T07 — Draft ActivityDefinition: post-install metrics scaffold validation (`memory init` check) -- [ ] T08 — Draft ActivityDefinition: success_rate below 0.8 → issue-core review task -- [ ] T09 — Document ActivityDefinition paths and activation contract in `docs/INTEGRATION_PATTERNS.md` -- [ ] T10 — Smoke test: manual activation against a test repo with populated metrics +- [x] T06 — Draft ActivityDefinition: weekly `metrics optimize` on repos with `.kaizen/metrics/` +- [x] T07 — Draft ActivityDefinition: post-install metrics scaffold validation (`memory init` check) +- [x] T08 — Draft ActivityDefinition: success_rate below 0.8 → issue-core review task +- [x] T09 — Document ActivityDefinition paths and activation contract in `docs/INTEGRATION_PATTERNS.md` +- [x] T10 — Smoke test: manual activation against a test repo with populated metrics ### Definition of done @@ -80,11 +80,11 @@ Persist optimizer outputs as registered artifact packages. ### Tasks -- [ ] T11 — Define artifact package manifest for optimizer run (`analysis.json` + `recommendations.jsonl`) -- [ ] T12 — Add `kaizen-agentic metrics publish` — register optimizer output with artifact-store API (configurable endpoint) -- [ ] T13 — Map retention class `raw-evidence` (180d) in publish manifest metadata -- [ ] T14 — Integration test with artifact-store local backend (skip if service unavailable; mark `@pytest.mark.integration`) -- [ ] T15 — Document publish workflow in `docs/agency-framework.md` metrics section +- [x] T11 — Define artifact package manifest for optimizer run (`analysis.json` + `recommendations.jsonl`) +- [x] T12 — Add `kaizen-agentic metrics publish` — register optimizer output with artifact-store API (configurable endpoint) +- [x] T13 — Map retention class `raw-evidence` (180d) in publish manifest metadata +- [x] T14 — Integration test with artifact-store local backend (skip if service unavailable; mark `@pytest.mark.integration`) +- [x] T15 — Document publish workflow in `docs/agency-framework.md` metrics section ### Definition of done @@ -100,10 +100,10 @@ Secondary integrations for template conformance and knowledge asset lifecycle. ### Tasks -- [ ] T16 — Map `wiki/KaizenAgentTemplate.md` sections to info-tech-canon profile outline (design doc only) -- [ ] T17 — Draft one InfoTechCanon-style agent brief for `tdd-workflow` pilot -- [ ] T18 — Spike: kontextual-engine ingestion manifest for `wiki/` directory (design note, no runtime dependency) -- [ ] T19 — Update `history/2026-06-16-ecosystem-assessment.md` with Part 4 outcomes +- [x] T16 — Map `wiki/KaizenAgentTemplate.md` sections to info-tech-canon profile outline (design doc only) +- [x] T17 — Draft one InfoTechCanon-style agent brief for `tdd-workflow` pilot +- [x] T18 — Spike: kontextual-engine ingestion manifest for `wiki/` directory (design note, no runtime dependency) +- [x] T19 — Update `history/2026-06-16-ecosystem-assessment.md` with Part 4 outcomes ### Definition of done