WP-0004: ecosystem integration complete
Add Helix Forge correlation (HELIX_SESSION_UID env, metrics correlate), artifact-store publish (metrics publish), activity-core ActivityDefinition references, integration patterns docs, and canon/knowledge design artifacts.
This commit is contained in:
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
- **sys-medic agent**: Linux/Kubernetes node health assessment agent integrated as a standard kaizen-agentic infrastructure agent (KAIZEN-WP-0002 Part 1)
|
||||
- **Ecosystem integration (WP-0004)**: Helix Forge correlation (`metrics correlate`, `HELIX_SESSION_UID` env), artifact-store publish (`metrics publish`), activity-core ActivityDefinition references, integration patterns docs
|
||||
- **Project metrics convention (ADR-004)**: `.kaizen/metrics/<agent>/` storage via `MetricsStore` and `OptimizerStore`
|
||||
- **Metrics CLI**: `kaizen-agentic metrics record|show|list|export|optimize` for per-execution records and optimizer analysis
|
||||
- **Optimizer integration**: `OptimizationLoop.from_metrics_store()` wired to project metrics; `memory brief` includes `## Performance Summary`
|
||||
|
||||
@@ -64,6 +64,16 @@ kaizen-agentic metrics export tdd-workflow
|
||||
kaizen-agentic metrics optimize tdd-workflow # analyze one agent (≥10 records)
|
||||
kaizen-agentic metrics optimize # analyze all agents with metrics
|
||||
|
||||
# Helix Forge correlation (fleet layer — agentic-resources)
|
||||
export HELIX_SESSION_UID="claude:<native-id>"
|
||||
kaizen-agentic metrics record tdd-workflow --success --time 120 --quality 0.9
|
||||
kaizen-agentic metrics correlate claude:<native-id> # needs HELIX_STORE_DB
|
||||
|
||||
# Publish optimizer evidence to artifact-store (optional)
|
||||
export ARTIFACTSTORE_API_URL=http://127.0.0.1:8000
|
||||
export ARTIFACTSTORE_API_TOKEN=<token>
|
||||
kaizen-agentic metrics publish
|
||||
|
||||
# Scaffold memory + metrics together
|
||||
kaizen-agentic memory init tdd-workflow
|
||||
kaizen-agentic memory init tdd-workflow --no-metrics # memory only
|
||||
|
||||
@@ -1,401 +1,105 @@
|
||||
# Integration Patterns for Existing Projects
|
||||
# Integration Patterns
|
||||
|
||||
This guide documents proven patterns for integrating Kaizen Agentic agents into existing projects that already have agent systems.
|
||||
How kaizen-agentic composes with ecosystem repos **by contract** — no merged
|
||||
codebases, no duplicated capabilities.
|
||||
|
||||
## Overview
|
||||
Reference: [wiki/EcosystemIntegration.md](../wiki/EcosystemIntegration.md),
|
||||
[KAIZEN-WP-0004](../workplans/kaizen-agentic-WP-0004-ecosystem-integration.md).
|
||||
|
||||
When introducing Kaizen agents to existing projects, you'll encounter various scenarios that require different integration approaches. This guide provides tested patterns and strategies.
|
||||
---
|
||||
|
||||
## Integration Scenarios
|
||||
## Pattern 1 — Helix Forge correlation (agentic-resources)
|
||||
|
||||
### Scenario 1: Clean Integration (No Existing Agents)
|
||||
**Problem:** Project metrics and fleet session metrics answer different questions.
|
||||
|
||||
**When to use**: Project has no existing agent systems.
|
||||
**Contract:** Optional `helix_session_uid` on ADR-004 execution records.
|
||||
|
||||
| kaizen-agentic | agentic-resources |
|
||||
|----------------|-------------------|
|
||||
| `metrics record` at session close | Helix capture → digest store |
|
||||
| `metrics correlate <uid>` read-only lookup | `Store.get_digest(session_uid)` |
|
||||
| `HELIX_SESSION_UID` env auto-merge | `Session.session_uid` |
|
||||
|
||||
**Docs:** [integrations/helix-forge-correlation.md](integrations/helix-forge-correlation.md)
|
||||
|
||||
**Boundary:** kaizen-agentic does not ingest session JSONL.
|
||||
|
||||
---
|
||||
|
||||
## Pattern 2 — activity-core triggers
|
||||
|
||||
**Problem:** Recurring kaizen checks need scheduling without custom cron in this repo.
|
||||
|
||||
**Contract:** ActivityDefinition markdown files declare triggers + actions that
|
||||
invoke kaizen-agentic CLI commands.
|
||||
|
||||
| Definition | Trigger | CLI command |
|
||||
|------------|---------|-------------|
|
||||
| [weekly-metrics-optimize](integrations/activity-definitions/weekly-metrics-optimize.md) | Cron Mon 08:00 | `metrics optimize` |
|
||||
| [post-install-metrics-scaffold](integrations/activity-definitions/post-install-metrics-scaffold.md) | `kaizen.agent.installed` | `memory init` validation |
|
||||
| [low-success-rate-review](integrations/activity-definitions/low-success-rate-review.md) | `kaizen.metrics.recorded` | `metrics show` + `optimize` |
|
||||
|
||||
**Activation:**
|
||||
|
||||
1. Copy or symlink definitions from `docs/integrations/activity-definitions/` into
|
||||
activity-core's `activity-definitions/` tree (or register as external ConfigMap).
|
||||
2. Run `make sync-activity-definitions` in activity-core.
|
||||
3. Enable definitions (`enabled: true`) after resolver wiring is verified.
|
||||
|
||||
**Smoke test (manual):**
|
||||
|
||||
**Pattern**: Direct installation
|
||||
```bash
|
||||
kaizen-agentic init . --agents keepaTodofile,keepaChangelog,tdd-workflow
|
||||
# Against a repo with populated metrics
|
||||
cd /path/to/project-with-kaizen
|
||||
kaizen-agentic metrics list
|
||||
kaizen-agentic metrics optimize
|
||||
# Verify analysis.json written
|
||||
test -f .kaizen/metrics/optimizer/analysis.json && echo OK
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Straightforward setup
|
||||
- No conflicts to resolve
|
||||
- Full Kaizen agent functionality
|
||||
**Boundary:** kaizen-agentic does not run Temporal schedules.
|
||||
|
||||
### Scenario 2: Claude Code Integration
|
||||
---
|
||||
|
||||
**When to use**: Project already uses Claude Code with CLAUDE.md.
|
||||
## Pattern 3 — artifact-store evidence retention
|
||||
|
||||
**Problem:** Optimizer outputs need durable, attributable retention beyond local disk.
|
||||
|
||||
**Contract:** `metrics publish` registers `analysis.json` + `recommendations.jsonl`
|
||||
as an artifact package with `retention_class: raw-evidence`.
|
||||
|
||||
**Pattern**: Respectful coexistence
|
||||
```bash
|
||||
# 1. Detect existing setup
|
||||
kaizen-agentic detect
|
||||
|
||||
# 2. Install compatible agents
|
||||
kaizen-agentic install keepaTodofile keepaChangelog
|
||||
|
||||
# 3. Update CLAUDE.md with new agent references
|
||||
export ARTIFACTSTORE_API_URL=http://127.0.0.1:8000
|
||||
export ARTIFACTSTORE_API_TOKEN=<token>
|
||||
kaizen-agentic metrics optimize
|
||||
kaizen-agentic metrics publish --target .
|
||||
```
|
||||
|
||||
**Considerations**:
|
||||
- Preserve existing CLAUDE.md content
|
||||
- Add Kaizen agent references to existing documentation
|
||||
- Maintain Claude Code workflow compatibility
|
||||
**Manifest:** [integrations/optimizer-artifact-manifest.md](integrations/optimizer-artifact-manifest.md)
|
||||
|
||||
### Scenario 3: Custom Agent Replacement
|
||||
**Boundary:** Publish is optional; local `.kaizen/metrics/optimizer/` remains canonical.
|
||||
|
||||
**When to use**: Project has custom agents that overlap with Kaizen functionality.
|
||||
---
|
||||
|
||||
**Pattern**: Gradual migration with backup
|
||||
```bash
|
||||
# 1. Analyze existing agents
|
||||
kaizen-agentic detect --detailed
|
||||
## Pattern 4 — Canon and knowledge (stretch)
|
||||
|
||||
# 2. Create migration plan
|
||||
kaizen-agentic migrate --dry-run
|
||||
Design-only paths for info-tech-canon and kontextual-engine:
|
||||
|
||||
# 3. Execute migration with backup
|
||||
kaizen-agentic migrate
|
||||
```
|
||||
- [integrations/canon-template-mapping.md](integrations/canon-template-mapping.md)
|
||||
- [integrations/briefs/tdd-workflow-canon-brief.md](integrations/briefs/tdd-workflow-canon-brief.md)
|
||||
- [integrations/kontextual-wiki-ingestion-spike.md](integrations/kontextual-wiki-ingestion-spike.md)
|
||||
|
||||
**Steps**:
|
||||
1. **Backup** existing agents
|
||||
2. **Map** custom agents to Kaizen equivalents
|
||||
3. **Migrate** functionality to extensions
|
||||
4. **Test** new agent workflow
|
||||
5. **Archive** old agents after verification
|
||||
No runtime dependency in WP-0004.
|
||||
|
||||
### Scenario 4: Hybrid Coexistence
|
||||
---
|
||||
|
||||
**When to use**: Project has essential custom agents that cannot be replaced.
|
||||
## Environment variables
|
||||
|
||||
**Pattern**: Namespace separation
|
||||
```bash
|
||||
# 1. Install Kaizen agents in parallel
|
||||
kaizen-agentic install keepaTodofile --target agents/kaizen/
|
||||
|
||||
# 2. Keep custom agents in separate directory
|
||||
# agents/custom/todo_manager.py
|
||||
# agents/kaizen/agent-keepaTodofile.md
|
||||
|
||||
# 3. Create integration extensions
|
||||
kaizen-agentic extensions create custom-integration keepaTodofile
|
||||
```
|
||||
|
||||
**Directory Structure**:
|
||||
```
|
||||
project/
|
||||
├── agents/
|
||||
│ ├── custom/ # Existing custom agents
|
||||
│ │ ├── todo_manager.py
|
||||
│ │ └── code_reviewer.py
|
||||
│ └── kaizen/ # Kaizen agents
|
||||
│ ├── agent-keepaTodofile.md
|
||||
│ └── agent-code-refactoring.md
|
||||
├── .kaizen/
|
||||
│ └── extensions/ # Integration extensions
|
||||
└── CLAUDE.md # Updated configuration
|
||||
```
|
||||
|
||||
### Scenario 5: Extension-Based Integration
|
||||
|
||||
**When to use**: Custom agents have unique functionality that should be preserved.
|
||||
|
||||
**Pattern**: Extend Kaizen agents with custom functionality
|
||||
```bash
|
||||
# 1. Create project-specific extension
|
||||
kaizen-agentic extensions create project-todo keepaTodofile \
|
||||
--description "TODO manager with custom workflow integration"
|
||||
|
||||
# 2. Configure custom behavior
|
||||
# Edit .kaizen/extensions/project-todo/extension.yml
|
||||
|
||||
# 3. Migrate custom logic to extension
|
||||
```
|
||||
|
||||
**Extension Configuration Example**:
|
||||
```yaml
|
||||
name: project-todo
|
||||
base_agent: keepaTodofile
|
||||
extension_type: functional_extension
|
||||
description: "TODO manager with custom workflow integration"
|
||||
|
||||
configuration:
|
||||
custom_instructions: |
|
||||
Follow our project-specific TODO format:
|
||||
- Use JIRA ticket references
|
||||
- Include priority levels (P0-P3)
|
||||
- Auto-assign based on component
|
||||
|
||||
custom_commands:
|
||||
create-epic: "Create epic-level TODO items"
|
||||
sync-jira: "Synchronize with JIRA tickets"
|
||||
priority-report: "Generate priority-based reports"
|
||||
|
||||
environment_overrides:
|
||||
JIRA_URL: "https://company.atlassian.net"
|
||||
TODO_FORMAT: "custom"
|
||||
```
|
||||
|
||||
## Conflict Resolution Patterns
|
||||
|
||||
### Name Conflicts
|
||||
|
||||
**Problem**: Multiple agents with the same name.
|
||||
|
||||
**Pattern**: Rename with suffix
|
||||
```bash
|
||||
# Automatic resolution
|
||||
todo_manager -> todo_manager_custom
|
||||
keepaTodofile -> keepaTodofile (Kaizen agent)
|
||||
```
|
||||
|
||||
**Implementation**:
|
||||
- Add `_custom` suffix to project-specific agents
|
||||
- Update references in scripts and documentation
|
||||
- Create aliases for backward compatibility
|
||||
|
||||
### Functional Overlaps
|
||||
|
||||
**Problem**: Multiple agents perform similar functions.
|
||||
|
||||
**Pattern**: Choose primary, extend secondary
|
||||
```bash
|
||||
# Primary: Kaizen agent (standardized)
|
||||
# Secondary: Custom agent -> extension
|
||||
|
||||
# Example: Both have TODO management
|
||||
# Decision: Use keepaTodofile as primary
|
||||
# Convert custom logic to extension
|
||||
```
|
||||
|
||||
**Decision Matrix**:
|
||||
| Factor | Choose Kaizen | Choose Custom | Create Extension |
|
||||
|--------|---------------|---------------|------------------|
|
||||
| Standard functionality | ✅ | ❌ | ✅ |
|
||||
| Custom business logic | ❌ | ✅ | ✅ |
|
||||
| Maintenance burden | ✅ | ❌ | ⚠️ |
|
||||
| Team familiarity | ⚠️ | ✅ | ✅ |
|
||||
|
||||
### Integration Order
|
||||
|
||||
**Pattern**: Infrastructure first, features last
|
||||
1. **Infrastructure agents** (setupRepository, tooling-optimization)
|
||||
2. **Core functionality** (keepaTodofile, keepaChangelog)
|
||||
3. **Development process** (tdd-workflow, code-refactoring)
|
||||
4. **Specialized features** (testing-efficiency, datamodel-optimization)
|
||||
|
||||
## Project Structure Respect Patterns
|
||||
|
||||
### Existing Directory Structures
|
||||
|
||||
**Pattern**: Adaptive installation
|
||||
```bash
|
||||
# Respect existing structure
|
||||
project/
|
||||
├── tools/agents/ # Existing agent directory
|
||||
├── scripts/ # Existing automation
|
||||
└── docs/ # Existing documentation
|
||||
|
||||
# Kaizen adaptation
|
||||
kaizen-agentic install --target tools/agents/ keepaTodofile
|
||||
# Creates: tools/agents/agent-keepaTodofile.md
|
||||
```
|
||||
|
||||
### Configuration File Integration
|
||||
|
||||
**Pattern**: Merge, don't replace
|
||||
```bash
|
||||
# Before
|
||||
CLAUDE.md # Existing Claude config
|
||||
project-config.yml # Existing project config
|
||||
|
||||
# After (merged)
|
||||
CLAUDE.md # Updated with Kaizen agents
|
||||
project-config.yml # Preserved
|
||||
.kaizen/extensions.yml # New Kaizen-specific config
|
||||
```
|
||||
|
||||
### Build System Integration
|
||||
|
||||
**Pattern**: Extend existing targets
|
||||
```makefile
|
||||
# Existing Makefile
|
||||
test:
|
||||
pytest tests/
|
||||
|
||||
# After Kaizen integration (extended)
|
||||
test: test-core test-agents
|
||||
@echo "All tests completed"
|
||||
|
||||
test-core:
|
||||
pytest tests/
|
||||
|
||||
test-agents:
|
||||
kaizen-agentic validate
|
||||
|
||||
# New Kaizen targets
|
||||
agents-status:
|
||||
kaizen-agentic status
|
||||
|
||||
agents-update:
|
||||
kaizen-agentic update
|
||||
```
|
||||
|
||||
## Safe Transition Strategies
|
||||
|
||||
### Phased Rollout
|
||||
|
||||
**Phase 1: Detection and Planning**
|
||||
```bash
|
||||
# Week 1: Analysis
|
||||
kaizen-agentic detect --detailed
|
||||
kaizen-agentic migrate --dry-run
|
||||
|
||||
# Decision point: Continue or modify approach
|
||||
```
|
||||
|
||||
**Phase 2: Infrastructure Agents**
|
||||
```bash
|
||||
# Week 2: Core infrastructure
|
||||
kaizen-agentic install setupRepository
|
||||
# Test and validate before proceeding
|
||||
```
|
||||
|
||||
**Phase 3: Core Functionality**
|
||||
```bash
|
||||
# Week 3: Essential agents
|
||||
kaizen-agentic install keepaTodofile keepaChangelog
|
||||
# Create extensions for custom functionality
|
||||
```
|
||||
|
||||
**Phase 4: Advanced Features**
|
||||
```bash
|
||||
# Week 4: Specialized agents
|
||||
kaizen-agentic install tdd-workflow code-refactoring
|
||||
# Full integration testing
|
||||
```
|
||||
|
||||
### Rollback Strategy
|
||||
|
||||
**Pattern**: Versioned backups with restore capability
|
||||
```bash
|
||||
# Before migration
|
||||
.kaizen-migration-backup-timestamp/
|
||||
├── agents/ # Original agents
|
||||
├── CLAUDE.md # Original configuration
|
||||
└── restoration.md # Rollback instructions
|
||||
|
||||
# Rollback command (if needed)
|
||||
kaizen-agentic rollback --backup .kaizen-migration-backup-timestamp/
|
||||
```
|
||||
|
||||
### Validation Gates
|
||||
|
||||
**Pattern**: Automated validation at each phase
|
||||
```bash
|
||||
# After each phase
|
||||
kaizen-agentic validate
|
||||
make test
|
||||
make agents-status
|
||||
|
||||
# Success criteria for proceeding:
|
||||
# ✅ All agents load without errors
|
||||
# ✅ All tests pass
|
||||
# ✅ No functionality regressions
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Communication
|
||||
|
||||
1. **Team Notification**: Inform team before starting migration
|
||||
2. **Documentation**: Update project docs with new agent workflows
|
||||
3. **Training**: Provide team training on Kaizen agents
|
||||
4. **Gradual Adoption**: Allow team to adapt gradually
|
||||
|
||||
### Technical
|
||||
|
||||
1. **Backup Everything**: Create comprehensive backups
|
||||
2. **Test Thoroughly**: Validate each integration step
|
||||
3. **Monitor Impact**: Watch for performance or workflow impacts
|
||||
4. **Version Control**: Commit changes in logical phases
|
||||
|
||||
### Maintenance
|
||||
|
||||
1. **Regular Updates**: Keep Kaizen agents updated
|
||||
2. **Extension Maintenance**: Maintain custom extensions
|
||||
3. **Documentation Sync**: Keep docs synchronized with agent changes
|
||||
4. **Team Feedback**: Collect and act on team feedback
|
||||
|
||||
## Troubleshooting Common Issues
|
||||
|
||||
### Agent Conflicts
|
||||
|
||||
**Issue**: Multiple agents trying to manage the same files.
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Identify conflicts
|
||||
kaizen-agentic detect --detailed
|
||||
|
||||
# Resolve with namespace separation
|
||||
mkdir agents/legacy agents/kaizen
|
||||
mv agents/todo_manager.py agents/legacy/
|
||||
kaizen-agentic install --target agents/kaizen/ keepaTodofile
|
||||
```
|
||||
|
||||
### Configuration Conflicts
|
||||
|
||||
**Issue**: Conflicting configuration files.
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Merge configurations
|
||||
cp CLAUDE.md CLAUDE.md.backup
|
||||
kaizen-agentic install keepaTodofile
|
||||
# Manually merge CLAUDE.md.backup content
|
||||
```
|
||||
|
||||
### Workflow Disruption
|
||||
|
||||
**Issue**: New agents disrupt existing workflows.
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Create compatibility extensions
|
||||
kaizen-agentic extensions create workflow-compat keepaTodofile
|
||||
# Configure extension to match existing workflow
|
||||
```
|
||||
|
||||
## Success Metrics
|
||||
|
||||
### Technical Metrics
|
||||
- ✅ Zero agent loading errors
|
||||
- ✅ All tests passing
|
||||
- ✅ No performance regressions
|
||||
- ✅ Successful backup/restore capability
|
||||
|
||||
### Team Metrics
|
||||
- ✅ Team adoption of new agents
|
||||
- ✅ Maintained productivity during transition
|
||||
- ✅ Positive feedback on new capabilities
|
||||
- ✅ Reduced maintenance overhead
|
||||
|
||||
### Project Metrics
|
||||
- ✅ Improved code quality metrics
|
||||
- ✅ Better documentation coverage
|
||||
- ✅ Enhanced development workflow efficiency
|
||||
- ✅ Standardized agent ecosystem
|
||||
|
||||
## Conclusion
|
||||
|
||||
Successful integration of Kaizen agents into existing projects requires:
|
||||
|
||||
1. **Careful analysis** of existing agent systems
|
||||
2. **Respectful approach** to existing project structure
|
||||
3. **Gradual migration** with proper backup strategies
|
||||
4. **Extension mechanisms** for preserving custom functionality
|
||||
5. **Team communication** and training throughout the process
|
||||
|
||||
Follow these patterns and your integration will be smooth, reversible, and beneficial to your development workflow.
|
||||
| Variable | Used by | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `HELIX_SESSION_UID` | `metrics record` | Fleet session correlation |
|
||||
| `HELIX_REPO`, `HELIX_FLAVOR` | `metrics record` | Session context |
|
||||
| `HELIX_TOKENS`, `HELIX_INFRA_OVERHEAD_SHARE` | `metrics record` | Fleet cost fields |
|
||||
| `HELIX_STORE_DB` | `metrics correlate` | Digest lookup database |
|
||||
| `ARTIFACTSTORE_API_URL` | `metrics publish` | Registry endpoint |
|
||||
| `ARTIFACTSTORE_API_TOKEN` | `metrics publish` | Write auth bearer token |
|
||||
@@ -260,6 +260,8 @@ kaizen-agentic metrics show <agent> # Summary + recent executions
|
||||
kaizen-agentic metrics list # Agents with metrics in project
|
||||
kaizen-agentic metrics export <agent> # Dump executions.jsonl
|
||||
kaizen-agentic metrics optimize [agent] # Run optimizer on project metrics (≥10 records)
|
||||
kaizen-agentic metrics correlate <uid> # Helix Forge digest lookup (read-only)
|
||||
kaizen-agentic metrics publish # Register optimizer output in artifact-store
|
||||
```
|
||||
|
||||
`memory brief` includes a `## Performance Summary` when metrics exist (success
|
||||
@@ -272,12 +274,28 @@ skip). Record outcomes at session close per
|
||||
### Fleet correlation
|
||||
|
||||
Project metrics correlate with **Helix Forge** fleet session metrics in
|
||||
`agentic-resources` via optional `helix_session_uid` (ADR-004). See
|
||||
[wiki/EcosystemIntegration.md](../wiki/EcosystemIntegration.md).
|
||||
`agentic-resources` via optional `helix_session_uid` (ADR-004).
|
||||
|
||||
- `HELIX_SESSION_UID` (and related env vars) auto-merge on `metrics record`
|
||||
- `metrics correlate <uid>` looks up fleet digest when `HELIX_STORE_DB` is set
|
||||
|
||||
See [integrations/helix-forge-correlation.md](integrations/helix-forge-correlation.md)
|
||||
and [wiki/EcosystemIntegration.md](../wiki/EcosystemIntegration.md).
|
||||
|
||||
### Evidence retention
|
||||
|
||||
Optimizer outputs may be published to `artifact-store` (WP-0004 Part 3).
|
||||
After `metrics optimize`, optionally publish optimizer outputs to **artifact-store**:
|
||||
|
||||
```bash
|
||||
export ARTIFACTSTORE_API_URL=http://127.0.0.1:8000
|
||||
export ARTIFACTSTORE_API_TOKEN=<write-token>
|
||||
kaizen-agentic metrics publish --target .
|
||||
```
|
||||
|
||||
Package uses `retention_class: raw-evidence` (180d). Local
|
||||
`.kaizen/metrics/optimizer/` remains authoritative when publish is skipped.
|
||||
|
||||
Manifest: [integrations/optimizer-artifact-manifest.md](integrations/optimizer-artifact-manifest.md).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
id: kaizen-low-success-rate-review
|
||||
name: Low Agent Success Rate Review
|
||||
enabled: false
|
||||
owner: kaizen-agentic
|
||||
governance: custodian
|
||||
status: proposed
|
||||
trigger:
|
||||
type: event
|
||||
event_type: kaizen.metrics.recorded
|
||||
context_sources:
|
||||
- type: event-payload
|
||||
bind_to: context.metrics
|
||||
---
|
||||
|
||||
# Low Agent Success Rate Review
|
||||
|
||||
When a project agent's rolling success rate drops below 0.8, create a review
|
||||
task in issue-core for human or optimizer-agent follow-up.
|
||||
|
||||
```rule
|
||||
id: flag-low-success-rate
|
||||
condition: 'context.metrics.summary.success_rate < 0.8 && context.metrics.summary.execution_count >= 5'
|
||||
action:
|
||||
task_template: "Review {{context.metrics.agent}} success rate ({{context.metrics.summary.success_rate}})"
|
||||
description: |
|
||||
Agent {{context.metrics.agent}} in {{context.metrics.project}} has success_rate
|
||||
below 0.8 over {{context.metrics.summary.execution_count}} executions.
|
||||
Run: kaizen-agentic metrics show {{context.metrics.agent}}
|
||||
Then: kaizen-agentic metrics optimize {{context.metrics.agent}}
|
||||
target_repo: "{{context.metrics.project}}"
|
||||
priority: high
|
||||
labels: ["kaizen", "metrics", "review", "automated"]
|
||||
```
|
||||
|
||||
**Threshold:** 0.8 success rate, minimum 5 executions (avoids noise on early pilots).
|
||||
|
||||
**CLI mapping:** Event emitter is future work; manual check today:
|
||||
|
||||
```bash
|
||||
kaizen-agentic metrics show <agent> # inspect summary.success_rate
|
||||
kaizen-agentic metrics optimize <agent>
|
||||
```
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
id: kaizen-post-install-metrics-scaffold
|
||||
name: Post-Install Metrics Scaffold Validation
|
||||
enabled: false
|
||||
owner: kaizen-agentic
|
||||
governance: custodian
|
||||
status: proposed
|
||||
trigger:
|
||||
type: event
|
||||
event_type: kaizen.agent.installed
|
||||
context_sources:
|
||||
- type: event-payload
|
||||
bind_to: context.install
|
||||
---
|
||||
|
||||
# Post-Install Metrics Scaffold Validation
|
||||
|
||||
Fires when an agent is installed into a project. Verifies that memory and metrics
|
||||
scaffolds exist for the installed agent.
|
||||
|
||||
```rule
|
||||
id: validate-metrics-scaffold
|
||||
condition: 'context.install.agent != ""'
|
||||
action:
|
||||
task_template: "Validate kaizen scaffold for {{context.install.agent}}"
|
||||
description: |
|
||||
In {{context.install.project_root}} verify:
|
||||
- .kaizen/agents/{{context.install.agent}}/memory.md exists OR run:
|
||||
kaizen-agentic memory init {{context.install.agent}}
|
||||
- .kaizen/metrics/{{context.install.agent}}/ exists OR re-run init without --no-metrics
|
||||
target_repo: "{{context.install.repo}}"
|
||||
priority: low
|
||||
labels: ["kaizen", "metrics", "scaffold", "automated"]
|
||||
```
|
||||
|
||||
**CLI mapping:**
|
||||
|
||||
```bash
|
||||
kaizen-agentic memory init <agent> # scaffolds memory + metrics by default
|
||||
kaizen-agentic metrics list # confirms metrics directory after first record
|
||||
```
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
id: kaizen-weekly-metrics-optimize
|
||||
name: Weekly Kaizen Metrics Optimization
|
||||
enabled: false
|
||||
owner: kaizen-agentic
|
||||
governance: custodian
|
||||
status: proposed
|
||||
trigger:
|
||||
type: cron
|
||||
cron_expression: "0 8 * * 1"
|
||||
timezone: Europe/Berlin
|
||||
misfire_policy: skip
|
||||
context_sources:
|
||||
- type: shell
|
||||
query: discover_kaizen_projects
|
||||
params:
|
||||
marker: .kaizen/metrics
|
||||
bind_to: context.projects
|
||||
---
|
||||
|
||||
# Weekly Kaizen Metrics Optimization
|
||||
|
||||
Runs every Monday 08:00 Berlin time on repos that contain `.kaizen/metrics/`.
|
||||
Invokes the kaizen-agentic optimizer CLI per project.
|
||||
|
||||
```rule
|
||||
id: run-weekly-optimizer
|
||||
for_each: context.projects
|
||||
bind_as: p
|
||||
condition: 'p.has_metrics == true'
|
||||
action:
|
||||
task_template: "Run kaizen metrics optimize on {{p.repo}}"
|
||||
description: |
|
||||
cd {{p.root}} && kaizen-agentic metrics optimize
|
||||
Optional: kaizen-agentic metrics publish (when artifact-store configured)
|
||||
target_repo: "{{p.repo}}"
|
||||
priority: medium
|
||||
labels: ["kaizen", "metrics", "optimizer", "automated"]
|
||||
```
|
||||
|
||||
**Activation:** sync this definition into activity-core via `make sync-activity-definitions`
|
||||
after enabling the shell resolver for `discover_kaizen_projects`.
|
||||
|
||||
**CLI mapping:** `kaizen-agentic metrics optimize` (no agent filter = all agents with metrics).
|
||||
44
docs/integrations/briefs/tdd-workflow-canon-brief.md
Normal file
44
docs/integrations/briefs/tdd-workflow-canon-brief.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# tdd-workflow — InfoTechCanon-style Brief
|
||||
|
||||
Compact agent brief derived from `agents/agent-tdd-workflow.md` (metrics pilot).
|
||||
Reference for fleet-wide brief rollout.
|
||||
|
||||
```yaml
|
||||
profile:
|
||||
id: kaizen/tdd-workflow
|
||||
version: "1.0"
|
||||
domain: development-process
|
||||
intent:
|
||||
summary: Guide TDD8 ISSUE-TEST-RED-GREEN-REFACTOR-DOCUMENT-REFINE-PUBLISH cycles
|
||||
outcomes:
|
||||
- Acceptance criteria covered by tests before PUBLISH
|
||||
- Sidequests tracked without blocking parent issues
|
||||
- Workspace integrated cleanly via make tdd-finish
|
||||
metrics:
|
||||
primary:
|
||||
name: test_pass_rate
|
||||
target: 1.0
|
||||
measurement: passing_tests / total_tests at PUBLISH
|
||||
secondary:
|
||||
- name: cycle_time_s
|
||||
measurement: session duration (execution_time_s)
|
||||
collection:
|
||||
storage: .kaizen/metrics/tdd-workflow/
|
||||
frequency: per_execution
|
||||
idempotency:
|
||||
signals:
|
||||
- current_issue.json workspace state
|
||||
- idempotency_key on metrics record
|
||||
session_protocol:
|
||||
start: read .kaizen/agents/tdd-workflow/memory.md
|
||||
close:
|
||||
- update memory.md sections
|
||||
- kaizen-agentic metrics record tdd-workflow
|
||||
ecosystem:
|
||||
fleet_correlation: helix_session_uid (ADR-004)
|
||||
optimizer: kaizen-agentic metrics optimize
|
||||
evidence: kaizen-agentic metrics publish (optional)
|
||||
```
|
||||
|
||||
Full specification: [agents/agent-tdd-workflow.md](../../../agents/agent-tdd-workflow.md).
|
||||
Pilot documentation: [wiki/AboutKaizenAgents.md](../../../wiki/AboutKaizenAgents.md).
|
||||
32
docs/integrations/canon-template-mapping.md
Normal file
32
docs/integrations/canon-template-mapping.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# KaizenAgentTemplate → InfoTechCanon Profile Mapping
|
||||
|
||||
Design note (WP-0004 Part 4). No runtime dependency on info-tech-canon.
|
||||
|
||||
## Section mapping
|
||||
|
||||
| `wiki/KaizenAgentTemplate.md` | InfoTechCanon profile outline |
|
||||
|------------------------------|------------------------------|
|
||||
| `specification.outcomes` | `profile.intent.outcomes[]` |
|
||||
| `specification.constraints` | `profile.constraints.hard[]` / `soft[]` |
|
||||
| `idempotency.detection` | `profile.idempotency.signals[]` |
|
||||
| `idempotency.rollback` | `profile.safety.rollback` |
|
||||
| `metrics.primary` | `profile.metrics.primary` |
|
||||
| `metrics.secondary[]` | `profile.metrics.secondary[]` |
|
||||
| `metrics.collection` | `profile.observability.collection` |
|
||||
| `testing.unit_tests[]` | `profile.validation.unit[]` |
|
||||
| `testing.integration_tests[]` | `profile.validation.integration[]` |
|
||||
| `evolution.history` | `profile.evolution.changelog` |
|
||||
| `evolution.optimization_hooks` | `profile.evolution.feedback_sources[]` |
|
||||
|
||||
## Validation hooks (future)
|
||||
|
||||
Extend `kaizen-agentic validate` to check:
|
||||
|
||||
1. Frontmatter contains `metrics.primary.name` when `memory: enabled`
|
||||
2. Session-close block references `metrics record`
|
||||
3. Required template sections present in agent body (warn, not fail)
|
||||
|
||||
## Reference pilot
|
||||
|
||||
`tdd-workflow` brief in [briefs/tdd-workflow-canon-brief.md](briefs/tdd-workflow-canon-brief.md)
|
||||
demonstrates a compact canon-style export derived from the full agent spec.
|
||||
103
docs/integrations/helix-forge-correlation.md
Normal file
103
docs/integrations/helix-forge-correlation.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Helix Forge Correlation Contract
|
||||
|
||||
Cross-repo contract between **kaizen-agentic** (project metrics, ADR-004) and
|
||||
**agentic-resources** (Helix Forge fleet session metrics).
|
||||
|
||||
## Purpose
|
||||
|
||||
Link a project-scoped agent execution record to the fleet session that produced
|
||||
it, without duplicating session JSONL ingestion in kaizen-agentic.
|
||||
|
||||
## Layers
|
||||
|
||||
| Layer | Owner | Storage |
|
||||
|-------|-------|---------|
|
||||
| Project | kaizen-agentic | `.kaizen/metrics/<agent>/executions.jsonl` |
|
||||
| Fleet | agentic-resources | Helix Forge digest store (`digests` table) |
|
||||
|
||||
## Correlation fields (ADR-004)
|
||||
|
||||
Optional on each project execution record:
|
||||
|
||||
```json
|
||||
{
|
||||
"helix_session_uid": "claude:17092961-…",
|
||||
"repo": "kaizen-agentic",
|
||||
"flavor": "claude",
|
||||
"tokens": 12500,
|
||||
"infra_overhead_share": 0.12
|
||||
}
|
||||
```
|
||||
|
||||
### Field mapping
|
||||
|
||||
| Helix Forge (`session_memory`) | ADR-004 project record |
|
||||
|-------------------------------|------------------------|
|
||||
| `Session.session_uid` | `helix_session_uid` |
|
||||
| `Session.repo` | `repo` |
|
||||
| `Session.flavor` | `flavor` |
|
||||
| `digest.cost.input_tokens + output_tokens` | `tokens` |
|
||||
| MCP tool share of `tool_histogram` | `infra_overhead_share` |
|
||||
| `digest.outcome == "success"` | informs `success` at record time |
|
||||
| `digest.cost.wall_clock_s` | complements `execution_time_s` |
|
||||
|
||||
## Population at session close
|
||||
|
||||
### Automatic (environment)
|
||||
|
||||
When Helix Forge capture is active in the same shell session:
|
||||
|
||||
```bash
|
||||
export HELIX_SESSION_UID="claude:17092961-…"
|
||||
export HELIX_REPO="kaizen-agentic"
|
||||
export HELIX_FLAVOR="claude"
|
||||
export HELIX_TOKENS="12500"
|
||||
export HELIX_INFRA_OVERHEAD_SHARE="0.12"
|
||||
|
||||
kaizen-agentic metrics record tdd-workflow --success --time 4200 --quality 0.9
|
||||
```
|
||||
|
||||
`metrics record` merges env vars into the execution record before append.
|
||||
|
||||
### Explicit (JSON)
|
||||
|
||||
```bash
|
||||
echo '{
|
||||
"success": true,
|
||||
"execution_time_s": 4200,
|
||||
"quality_score": 0.9,
|
||||
"helix_session_uid": "claude:17092961-…",
|
||||
"repo": "kaizen-agentic",
|
||||
"flavor": "claude",
|
||||
"tokens": 12500,
|
||||
"infra_overhead_share": 0.12
|
||||
}' | kaizen-agentic metrics record tdd-workflow --json
|
||||
```
|
||||
|
||||
## Fleet lookup (read-only)
|
||||
|
||||
```bash
|
||||
export HELIX_STORE_DB=~/.helix-forge/store.db # agentic-resources session store
|
||||
kaizen-agentic metrics correlate claude:17092961-…
|
||||
```
|
||||
|
||||
When `HELIX_STORE_DB` is unset, `metrics correlate` returns a **stub** response
|
||||
documenting expected fields — no ingestion code runs in kaizen-agentic.
|
||||
|
||||
## Bidirectional references
|
||||
|
||||
| Document | Repo |
|
||||
|----------|------|
|
||||
| [ADR-004](../adr/ADR-004-project-metrics-convention.md) | kaizen-agentic |
|
||||
| [wiki/EcosystemIntegration.md](../../wiki/EcosystemIntegration.md) | kaizen-agentic |
|
||||
| [DESIGN-session-memory.md](https://github.com/coulomb/agentic-resources/blob/main/docs/DESIGN-session-memory.md) | agentic-resources |
|
||||
| `session_memory/core/store.py` — `get_digest()` | agentic-resources |
|
||||
|
||||
agentic-resources should link back to this document from its session-memory design
|
||||
notes when documenting downstream consumers of `session_uid`.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No Claude/Codex/Grok JSONL ingestion in kaizen-agentic
|
||||
- No write path to Helix Forge from kaizen-agentic CLI
|
||||
- No merge of fleet baselines into project `summary.json` (Coach may cite both)
|
||||
41
docs/integrations/kontextual-wiki-ingestion-spike.md
Normal file
41
docs/integrations/kontextual-wiki-ingestion-spike.md
Normal file
@@ -0,0 +1,41 @@
|
||||
# kontextual-engine Wiki Ingestion Spike
|
||||
|
||||
Design note (WP-0004 Part 4). No runtime dependency.
|
||||
|
||||
## Proposed manifest
|
||||
|
||||
```yaml
|
||||
ingestion:
|
||||
source_repo: kaizen-agentic
|
||||
asset_class: strategic-knowledge
|
||||
paths:
|
||||
- wiki/**/*.md
|
||||
- INTENT.md
|
||||
- docs/adr/ADR-*.md
|
||||
exclude:
|
||||
- wiki/**/xxx
|
||||
metadata:
|
||||
domain: custodian
|
||||
topic_id: cee7bedf-2b48-46ef-8601-006474f2ad7a
|
||||
producer: kaizen-agentic
|
||||
refresh:
|
||||
trigger: git-push-main
|
||||
retention_class: operational-knowledge
|
||||
```
|
||||
|
||||
## Rationale
|
||||
|
||||
- `wiki/` holds product narrative and integration contracts not suited for agent prompts alone
|
||||
- ADRs are normative; kontextual-engine can index them for cross-repo retrieval
|
||||
- Agent definitions (`agents/`) remain separate — executable personas vs strategic docs
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Chunking strategy for `KaizenAgentTemplate.md` (section-aware vs whole-file)
|
||||
2. Whether Coach synthesis outputs should be ingested as derived assets
|
||||
3. Correlation with info-tech-canon profiles when both exist for one agent
|
||||
|
||||
## Next step
|
||||
|
||||
Dedicated workplan after WP-0004 baseline; evaluate kontextual-engine ingestion API
|
||||
stability before hard dependency.
|
||||
60
docs/integrations/optimizer-artifact-manifest.md
Normal file
60
docs/integrations/optimizer-artifact-manifest.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# Optimizer Evidence Artifact Manifest
|
||||
|
||||
Package schema for `kaizen-agentic metrics publish` → **artifact-store**.
|
||||
|
||||
## Package identity
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| `producer` | `kaizen-agentic` |
|
||||
| `retention_class` | `raw-evidence` (180d default, ADR-004 aligned) |
|
||||
| `name` | `kaizen-optimizer-<project-slug>` |
|
||||
| `subject` | project directory name (override with `--subject`) |
|
||||
|
||||
## Files
|
||||
|
||||
| Relative path | Source | Media type |
|
||||
|---------------|--------|------------|
|
||||
| `optimizer/analysis.json` | `.kaizen/metrics/optimizer/analysis.json` | `application/json` |
|
||||
| `optimizer/recommendations.jsonl` | `.kaizen/metrics/optimizer/recommendations.jsonl` | `application/x-ndjson` |
|
||||
|
||||
`recommendations.jsonl` is omitted from upload when absent (e.g. insufficient samples).
|
||||
|
||||
## Metadata (`POST /packages`)
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": "kaizen-agentic/optimizer-evidence/v1",
|
||||
"project": "demo-app",
|
||||
"project_root": "/path/to/demo-app",
|
||||
"producer": "kaizen-agentic",
|
||||
"retention_class": "raw-evidence",
|
||||
"retention_days": 180,
|
||||
"optimized_at": "2026-06-18",
|
||||
"agents": ["tdd-workflow", "coach"],
|
||||
"files": [
|
||||
"optimizer/analysis.json",
|
||||
"optimizer/recommendations.jsonl"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Publish workflow
|
||||
|
||||
```bash
|
||||
# 1. Ensure optimizer has run
|
||||
kaizen-agentic metrics optimize
|
||||
|
||||
# 2. Publish (artifact-store must be reachable)
|
||||
export ARTIFACTSTORE_API_URL=http://127.0.0.1:8000
|
||||
export ARTIFACTSTORE_API_TOKEN=<write-token>
|
||||
kaizen-agentic metrics publish --target .
|
||||
```
|
||||
|
||||
Local-only workflows skip publish; `.kaizen/metrics/optimizer/` remains authoritative.
|
||||
|
||||
## Related
|
||||
|
||||
- [artifact-store ingestion API](https://github.com/coulomb/artifact-store) — `POST /packages`, `/files`, `/finalize`
|
||||
- [ADR-004](../adr/ADR-004-project-metrics-convention.md)
|
||||
- [INTEGRATION_PATTERNS.md](../INTEGRATION_PATTERNS.md)
|
||||
@@ -138,5 +138,35 @@ See `wiki/EcosystemIntegration.md` for integration contracts.
|
||||
|
||||
## Follow-Up Workplans
|
||||
|
||||
- **KAIZEN-WP-0003** — measurement loop (active, State Hub registered)
|
||||
- **KAIZEN-WP-0004** — ecosystem integration (active, depends on WP-0003 Part 3)
|
||||
- **KAIZEN-WP-0003** — measurement loop (completed 2026-06-18)
|
||||
- **KAIZEN-WP-0004** — ecosystem integration (completed 2026-06-18)
|
||||
|
||||
---
|
||||
|
||||
## WP-0004 Outcomes (2026-06-18)
|
||||
|
||||
### Part 1 — Helix Forge correlation
|
||||
|
||||
- `HELIX_SESSION_UID` env auto-merge on `metrics record`
|
||||
- `kaizen-agentic metrics correlate <uid>` read-only adapter (sqlite or stub)
|
||||
- Contract: `docs/integrations/helix-forge-correlation.md`
|
||||
- Worked example in `wiki/EcosystemIntegration.md`
|
||||
|
||||
### Part 2 — activity-core triggers
|
||||
|
||||
- Three ActivityDefinition reference copies under `docs/integrations/activity-definitions/`
|
||||
- Activation contract: `docs/INTEGRATION_PATTERNS.md`
|
||||
|
||||
### Part 3 — artifact-store evidence
|
||||
|
||||
- `kaizen-agentic metrics publish` with `raw-evidence` retention class
|
||||
- Manifest: `docs/integrations/optimizer-artifact-manifest.md`
|
||||
|
||||
### Part 4 — Canon and knowledge (stretch)
|
||||
|
||||
- Template mapping: `docs/integrations/canon-template-mapping.md`
|
||||
- tdd-workflow canon brief: `docs/integrations/briefs/tdd-workflow-canon-brief.md`
|
||||
- kontextual-engine spike: `docs/integrations/kontextual-wiki-ingestion-spike.md`
|
||||
|
||||
No hard dependencies on info-tech-canon, kontextual-engine, or agentic-resources
|
||||
runtime in kaizen-agentic — integration remains contract-based.
|
||||
@@ -11,6 +11,12 @@ from typing import List, Optional
|
||||
|
||||
from .registry import AgentRegistry, AgentCategory
|
||||
from .installer import AgentInstaller, ProjectInitializer, InstallationConfig
|
||||
from .integrations.artifact_store import (
|
||||
default_api_token,
|
||||
default_api_url,
|
||||
publish_optimizer_evidence,
|
||||
)
|
||||
from .integrations.helix import HelixCorrelationAdapter, enrich_helix_correlation
|
||||
from .metrics import MetricsStore, OptimizerStore, performance_summary_markdown
|
||||
from .optimization import OptimizationLoop, MIN_SAMPLES_FOR_RECOMMENDATIONS
|
||||
|
||||
@@ -999,6 +1005,8 @@ def metrics_record(
|
||||
if session_id:
|
||||
payload["session_id"] = session_id
|
||||
|
||||
payload = enrich_helix_correlation(payload)
|
||||
|
||||
if store.append(payload, idempotency_key=idempotency_key):
|
||||
click.echo(f"Recorded metrics for '{agent_name}'")
|
||||
else:
|
||||
@@ -1106,6 +1114,84 @@ def metrics_optimize(agent_name: Optional[str], target: str, min_samples: int):
|
||||
click.echo(f"Wrote optimizer analysis: {analysis_path}")
|
||||
|
||||
|
||||
@metrics.command("correlate")
|
||||
@click.argument("session_uid")
|
||||
@click.option(
|
||||
"--store-db",
|
||||
envvar="HELIX_STORE_DB",
|
||||
help="Helix Forge session-memory SQLite database path",
|
||||
)
|
||||
def metrics_correlate(session_uid: str, store_db: Optional[str]):
|
||||
"""Look up Helix Forge digest summary for a session UID (read-only)."""
|
||||
adapter = HelixCorrelationAdapter(
|
||||
store_db=Path(store_db).resolve() if store_db else None
|
||||
)
|
||||
if adapter.store_db is None:
|
||||
adapter = HelixCorrelationAdapter.from_env()
|
||||
summary = adapter.lookup(session_uid)
|
||||
click.echo(json.dumps(summary, indent=2, sort_keys=True))
|
||||
|
||||
|
||||
@metrics.command("publish")
|
||||
@click.option("--target", "-t", default=".", help="Project root (default: current)")
|
||||
@click.option(
|
||||
"--api-url",
|
||||
default=default_api_url,
|
||||
show_default=True,
|
||||
help="artifact-store API base URL (ARTIFACTSTORE_API_URL)",
|
||||
)
|
||||
@click.option(
|
||||
"--token",
|
||||
default=default_api_token,
|
||||
help="artifact-store bearer token (ARTIFACTSTORE_API_TOKEN)",
|
||||
)
|
||||
@click.option(
|
||||
"--subject",
|
||||
help="Package subject (default: project directory name)",
|
||||
)
|
||||
@click.option(
|
||||
"--retention-class",
|
||||
default="raw-evidence",
|
||||
show_default=True,
|
||||
help="artifact-store retention class",
|
||||
)
|
||||
def metrics_publish(
|
||||
target: str,
|
||||
api_url: str,
|
||||
token: str,
|
||||
subject: Optional[str],
|
||||
retention_class: str,
|
||||
):
|
||||
"""Publish optimizer evidence to artifact-store (optional integration)."""
|
||||
project_root = _project_root(target)
|
||||
if not token:
|
||||
click.echo(
|
||||
"Error: artifact-store token required. Set ARTIFACTSTORE_API_TOKEN or --token.",
|
||||
err=True,
|
||||
)
|
||||
sys.exit(1)
|
||||
try:
|
||||
result = publish_optimizer_evidence(
|
||||
project_root,
|
||||
api_url=api_url,
|
||||
token=token,
|
||||
subject=subject,
|
||||
retention_class=retention_class,
|
||||
)
|
||||
except FileNotFoundError as exc:
|
||||
click.echo(f"Error: {exc}", err=True)
|
||||
sys.exit(1)
|
||||
except RuntimeError as exc:
|
||||
click.echo(f"Error: {exc}", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
click.echo(f"Published optimizer evidence package: {result.package_id}")
|
||||
click.echo(f" Files uploaded: {result.files_uploaded}")
|
||||
click.echo(f" Retention class: {result.retention_class}")
|
||||
if result.manifest_digest:
|
||||
click.echo(f" Manifest digest: {result.manifest_digest}")
|
||||
|
||||
|
||||
@metrics.command("export")
|
||||
@click.argument("agent_name")
|
||||
@click.option("--target", "-t", default=".", help="Project root (default: current)")
|
||||
|
||||
10
src/kaizen_agentic/integrations/__init__.py
Normal file
10
src/kaizen_agentic/integrations/__init__.py
Normal file
@@ -0,0 +1,10 @@
|
||||
"""Ecosystem integration adapters (Helix Forge, artifact-store)."""
|
||||
|
||||
from .artifact_store import publish_optimizer_evidence
|
||||
from .helix import HelixCorrelationAdapter, enrich_helix_correlation
|
||||
|
||||
__all__ = [
|
||||
"HelixCorrelationAdapter",
|
||||
"enrich_helix_correlation",
|
||||
"publish_optimizer_evidence",
|
||||
]
|
||||
233
src/kaizen_agentic/integrations/artifact_store.py
Normal file
233
src/kaizen_agentic/integrations/artifact_store.py
Normal file
@@ -0,0 +1,233 @@
|
||||
"""artifact-store publish adapter for optimizer evidence (WP-0004 Part 3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib import error, parse, request
|
||||
|
||||
from ..metrics import OptimizerStore
|
||||
|
||||
ENV_API_URL = "ARTIFACTSTORE_API_URL"
|
||||
ENV_API_TOKEN = "ARTIFACTSTORE_API_TOKEN"
|
||||
DEFAULT_RETENTION_CLASS = "raw-evidence"
|
||||
PRODUCER = "kaizen-agentic"
|
||||
|
||||
|
||||
@dataclass
|
||||
class PublishResult:
|
||||
package_id: str
|
||||
manifest_digest: Optional[str]
|
||||
files_uploaded: int
|
||||
retention_class: str
|
||||
|
||||
|
||||
def build_optimizer_manifest(
|
||||
project_root: Path,
|
||||
*,
|
||||
agents: Optional[List[str]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Manifest metadata for an optimizer evidence package."""
|
||||
store = OptimizerStore(project_root)
|
||||
analysis = {}
|
||||
if store.analysis_path.exists():
|
||||
analysis = json.loads(store.analysis_path.read_text(encoding="utf-8"))
|
||||
|
||||
return {
|
||||
"schema": "kaizen-agentic/optimizer-evidence/v1",
|
||||
"project": project_root.name,
|
||||
"project_root": str(project_root.resolve()),
|
||||
"producer": PRODUCER,
|
||||
"retention_class": DEFAULT_RETENTION_CLASS,
|
||||
"retention_days": 180,
|
||||
"optimized_at": analysis.get("optimized_at"),
|
||||
"agents": agents or [item.get("agent") for item in analysis.get("agents", [])],
|
||||
"files": [
|
||||
"optimizer/analysis.json",
|
||||
"optimizer/recommendations.jsonl",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def publish_optimizer_evidence(
|
||||
project_root: Path,
|
||||
*,
|
||||
api_url: str,
|
||||
token: str,
|
||||
subject: Optional[str] = None,
|
||||
retention_class: str = DEFAULT_RETENTION_CLASS,
|
||||
) -> PublishResult:
|
||||
"""Register optimizer outputs as an artifact-store package."""
|
||||
store = OptimizerStore(project_root)
|
||||
if not store.analysis_path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"No optimizer analysis at {store.analysis_path}. "
|
||||
"Run: kaizen-agentic metrics optimize"
|
||||
)
|
||||
|
||||
manifest = build_optimizer_manifest(project_root)
|
||||
package_name = f"kaizen-optimizer-{project_root.name}"
|
||||
package_subject = subject or project_root.name
|
||||
|
||||
created = _http_json(
|
||||
"POST",
|
||||
api_url,
|
||||
"/packages",
|
||||
token,
|
||||
{
|
||||
"name": package_name,
|
||||
"producer": PRODUCER,
|
||||
"subject": package_subject,
|
||||
"retention_class": retention_class,
|
||||
"metadata": manifest,
|
||||
},
|
||||
)
|
||||
package_id = created["id"]
|
||||
|
||||
uploads = [
|
||||
(
|
||||
store.analysis_path,
|
||||
"optimizer/analysis.json",
|
||||
"application/json",
|
||||
),
|
||||
]
|
||||
if store.recommendations_path.exists():
|
||||
uploads.append(
|
||||
(
|
||||
store.recommendations_path,
|
||||
"optimizer/recommendations.jsonl",
|
||||
"application/x-ndjson",
|
||||
)
|
||||
)
|
||||
|
||||
for path, relative_path, media_type in uploads:
|
||||
_http_multipart(
|
||||
api_url,
|
||||
f"/packages/{package_id}/files",
|
||||
token,
|
||||
fields={"relative_path": relative_path, "media_type": media_type},
|
||||
file_field="file",
|
||||
file_name=path.name,
|
||||
file_content_type=media_type,
|
||||
file_bytes=path.read_bytes(),
|
||||
)
|
||||
|
||||
finalized = _http_json(
|
||||
"POST",
|
||||
api_url,
|
||||
f"/packages/{package_id}/finalize",
|
||||
token,
|
||||
{},
|
||||
)
|
||||
|
||||
return PublishResult(
|
||||
package_id=package_id,
|
||||
manifest_digest=finalized.get("manifest_digest"),
|
||||
files_uploaded=len(uploads),
|
||||
retention_class=retention_class,
|
||||
)
|
||||
|
||||
|
||||
def default_api_url() -> str:
|
||||
return os.environ.get(ENV_API_URL, "http://127.0.0.1:8000").rstrip("/")
|
||||
|
||||
|
||||
def default_api_token() -> str:
|
||||
return os.environ.get(ENV_API_TOKEN, "")
|
||||
|
||||
|
||||
def _http_json(
|
||||
method: str,
|
||||
base_url: str,
|
||||
path: str,
|
||||
token: str,
|
||||
payload: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
body = json.dumps(payload).encode("utf-8") if payload else None
|
||||
headers = {"Accept": "application/json"}
|
||||
if body is not None:
|
||||
headers["Content-Type"] = "application/json"
|
||||
response = _http_bytes(method, base_url, path, token, body=body, headers=headers)
|
||||
decoded = json.loads(response)
|
||||
if not isinstance(decoded, dict):
|
||||
raise ValueError(f"expected JSON object from {path}")
|
||||
return decoded
|
||||
|
||||
|
||||
def _http_multipart(
|
||||
base_url: str,
|
||||
path: str,
|
||||
token: str,
|
||||
*,
|
||||
fields: Dict[str, str],
|
||||
file_field: str,
|
||||
file_name: str,
|
||||
file_content_type: str,
|
||||
file_bytes: bytes,
|
||||
) -> Dict[str, Any]:
|
||||
boundary = f"kaizen-{uuid.uuid4().hex}"
|
||||
body = bytearray()
|
||||
for name, value in fields.items():
|
||||
body.extend(f"--{boundary}\r\n".encode("ascii"))
|
||||
body.extend(
|
||||
f'Content-Disposition: form-data; name="{_quote(name)}"\r\n\r\n'.encode()
|
||||
)
|
||||
body.extend(value.encode())
|
||||
body.extend(b"\r\n")
|
||||
body.extend(f"--{boundary}\r\n".encode("ascii"))
|
||||
body.extend(
|
||||
(
|
||||
f'Content-Disposition: form-data; name="{_quote(file_field)}"; '
|
||||
f'filename="{_quote(file_name)}"\r\n'
|
||||
f"Content-Type: {file_content_type}\r\n\r\n"
|
||||
).encode()
|
||||
)
|
||||
body.extend(file_bytes)
|
||||
body.extend(b"\r\n")
|
||||
body.extend(f"--{boundary}--\r\n".encode("ascii"))
|
||||
|
||||
response = _http_bytes(
|
||||
"POST",
|
||||
base_url,
|
||||
path,
|
||||
token,
|
||||
body=bytes(body),
|
||||
headers={
|
||||
"Content-Type": f"multipart/form-data; boundary={boundary}",
|
||||
"Accept": "application/json",
|
||||
},
|
||||
)
|
||||
decoded = json.loads(response)
|
||||
if not isinstance(decoded, dict):
|
||||
raise ValueError(f"expected JSON object from {path}")
|
||||
return decoded
|
||||
|
||||
|
||||
def _http_bytes(
|
||||
method: str,
|
||||
base_url: str,
|
||||
path: str,
|
||||
token: str,
|
||||
*,
|
||||
body: Optional[bytes] = None,
|
||||
headers: Optional[Dict[str, str]] = None,
|
||||
) -> bytes:
|
||||
url = f"{base_url.rstrip('/')}/{path.lstrip('/')}"
|
||||
effective_headers = dict(headers or {})
|
||||
if token:
|
||||
effective_headers["Authorization"] = f"Bearer {token}"
|
||||
req = request.Request(url, data=body, headers=effective_headers, method=method)
|
||||
try:
|
||||
with request.urlopen(req, timeout=30) as resp:
|
||||
return resp.read()
|
||||
except error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")
|
||||
raise RuntimeError(f"HTTP {exc.code} from {path}: {detail}") from exc
|
||||
|
||||
|
||||
def _quote(value: str) -> str:
|
||||
return parse.quote(value, safe="")
|
||||
171
src/kaizen_agentic/integrations/helix.py
Normal file
171
src/kaizen_agentic/integrations/helix.py
Normal file
@@ -0,0 +1,171 @@
|
||||
"""Helix Forge correlation adapter (ADR-004, agentic-resources)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
ENV_SESSION_UID = "HELIX_SESSION_UID"
|
||||
ENV_REPO = "HELIX_REPO"
|
||||
ENV_FLAVOR = "HELIX_FLAVOR"
|
||||
ENV_TOKENS = "HELIX_TOKENS"
|
||||
ENV_INFRA_SHARE = "HELIX_INFRA_OVERHEAD_SHARE"
|
||||
ENV_STORE_DB = "HELIX_STORE_DB"
|
||||
|
||||
|
||||
def enrich_helix_correlation(record: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Apply optional Helix correlation fields from env or existing record."""
|
||||
payload = dict(record)
|
||||
|
||||
uid = payload.get("helix_session_uid") or os.environ.get(ENV_SESSION_UID)
|
||||
if uid:
|
||||
payload["helix_session_uid"] = uid
|
||||
|
||||
repo = payload.get("repo") or os.environ.get(ENV_REPO)
|
||||
if repo:
|
||||
payload["repo"] = repo
|
||||
|
||||
flavor = payload.get("flavor") or os.environ.get(ENV_FLAVOR)
|
||||
if flavor:
|
||||
payload["flavor"] = flavor
|
||||
|
||||
tokens_raw = payload.get("tokens")
|
||||
if tokens_raw is None and ENV_TOKENS in os.environ:
|
||||
try:
|
||||
tokens_raw = int(os.environ[ENV_TOKENS])
|
||||
except ValueError:
|
||||
pass
|
||||
if tokens_raw is not None:
|
||||
payload["tokens"] = int(tokens_raw)
|
||||
|
||||
infra = payload.get("infra_overhead_share")
|
||||
if infra is None and ENV_INFRA_SHARE in os.environ:
|
||||
try:
|
||||
infra = float(os.environ[ENV_INFRA_SHARE])
|
||||
except ValueError:
|
||||
pass
|
||||
if infra is not None:
|
||||
payload["infra_overhead_share"] = float(infra)
|
||||
|
||||
return payload
|
||||
|
||||
|
||||
def digest_to_correlation_summary(
|
||||
session_uid: str,
|
||||
digest: Dict[str, Any],
|
||||
*,
|
||||
adapter: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Project a Helix digest into ADR-004 correlation summary fields."""
|
||||
cost = digest.get("cost") or {}
|
||||
input_tokens = int(cost.get("input_tokens") or 0)
|
||||
output_tokens = int(cost.get("output_tokens") or 0)
|
||||
wall_clock_s = cost.get("wall_clock_s")
|
||||
|
||||
summary: Dict[str, Any] = {
|
||||
"helix_session_uid": session_uid,
|
||||
"repo": digest.get("repo"),
|
||||
"flavor": digest.get("flavor"),
|
||||
"fleet_outcome": digest.get("outcome"),
|
||||
"tokens": input_tokens + output_tokens,
|
||||
"adapter": adapter,
|
||||
}
|
||||
if wall_clock_s is not None:
|
||||
summary["wall_clock_s"] = float(wall_clock_s)
|
||||
|
||||
markers = digest.get("markers") or {}
|
||||
tool_histogram = digest.get("tool_histogram") or {}
|
||||
mcp_calls = sum(
|
||||
count for tool, count in tool_histogram.items() if str(tool).startswith("mcp__")
|
||||
)
|
||||
total_calls = sum(tool_histogram.values()) or 0
|
||||
if total_calls:
|
||||
summary["infra_overhead_share"] = round(mcp_calls / total_calls, 3)
|
||||
elif "infra_overhead_share" in digest:
|
||||
summary["infra_overhead_share"] = digest["infra_overhead_share"]
|
||||
|
||||
if markers:
|
||||
summary["markers"] = {
|
||||
key: markers[key]
|
||||
for key in ("errors", "retries", "test_runs")
|
||||
if key in markers
|
||||
}
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
@dataclass
|
||||
class HelixCorrelationAdapter:
|
||||
"""Read-only lookup of Helix Forge session digests."""
|
||||
|
||||
store_db: Optional[Path] = None
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "HelixCorrelationAdapter":
|
||||
raw = os.environ.get(ENV_STORE_DB)
|
||||
return cls(store_db=Path(raw).resolve() if raw else None)
|
||||
|
||||
def lookup(self, session_uid: str) -> Dict[str, Any]:
|
||||
if self.store_db and self.store_db.exists():
|
||||
digest = self._load_digest_sqlite(session_uid)
|
||||
if digest is not None:
|
||||
return digest_to_correlation_summary(
|
||||
session_uid,
|
||||
digest,
|
||||
adapter="helix-sqlite",
|
||||
)
|
||||
return {
|
||||
"helix_session_uid": session_uid,
|
||||
"adapter": "helix-sqlite",
|
||||
"status": "not_found",
|
||||
"message": f"No digest for session_uid in {self.store_db}",
|
||||
}
|
||||
|
||||
return {
|
||||
"helix_session_uid": session_uid,
|
||||
"adapter": "stub",
|
||||
"status": "not_configured",
|
||||
"message": (
|
||||
"Set HELIX_STORE_DB to an agentic-resources session-memory SQLite "
|
||||
"database for live lookup. Correlation fields on project metrics "
|
||||
"still work via HELIX_SESSION_UID at record time."
|
||||
),
|
||||
"expected_fields": [
|
||||
"helix_session_uid",
|
||||
"repo",
|
||||
"flavor",
|
||||
"tokens",
|
||||
"infra_overhead_share",
|
||||
"fleet_outcome",
|
||||
"wall_clock_s",
|
||||
],
|
||||
}
|
||||
|
||||
def _load_digest_sqlite(self, session_uid: str) -> Optional[Dict[str, Any]]:
|
||||
conn = sqlite3.connect(str(self.store_db))
|
||||
try:
|
||||
row = conn.execute(
|
||||
"SELECT json FROM digests WHERE session_uid = ?",
|
||||
(session_uid,),
|
||||
).fetchone()
|
||||
if not row:
|
||||
return None
|
||||
digest = json.loads(row[0])
|
||||
digest.setdefault("session_uid", session_uid)
|
||||
|
||||
session_row = conn.execute(
|
||||
"SELECT json FROM sessions WHERE session_uid = ?",
|
||||
(session_uid,),
|
||||
).fetchone()
|
||||
if session_row:
|
||||
session = json.loads(session_row[0])
|
||||
digest.setdefault("repo", session.get("repo"))
|
||||
digest.setdefault("flavor", session.get("flavor"))
|
||||
return digest
|
||||
finally:
|
||||
conn.close()
|
||||
161
tests/test_helix_correlation.py
Normal file
161
tests/test_helix_correlation.py
Normal file
@@ -0,0 +1,161 @@
|
||||
"""Tests for Helix Forge correlation (WP-0004 Part 1)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from kaizen_agentic.cli import cli
|
||||
from kaizen_agentic.integrations.helix import (
|
||||
HelixCorrelationAdapter,
|
||||
enrich_helix_correlation,
|
||||
)
|
||||
|
||||
|
||||
def test_enrich_helix_correlation_from_env(monkeypatch: pytest.MonkeyPatch):
|
||||
monkeypatch.setenv("HELIX_SESSION_UID", "claude:test-uid")
|
||||
monkeypatch.setenv("HELIX_REPO", "kaizen-agentic")
|
||||
monkeypatch.setenv("HELIX_FLAVOR", "claude")
|
||||
monkeypatch.setenv("HELIX_TOKENS", "9900")
|
||||
monkeypatch.setenv("HELIX_INFRA_OVERHEAD_SHARE", "0.15")
|
||||
|
||||
result = enrich_helix_correlation({"success": True})
|
||||
|
||||
assert result["helix_session_uid"] == "claude:test-uid"
|
||||
assert result["repo"] == "kaizen-agentic"
|
||||
assert result["flavor"] == "claude"
|
||||
assert result["tokens"] == 9900
|
||||
assert result["infra_overhead_share"] == 0.15
|
||||
|
||||
|
||||
def test_enrich_does_not_override_existing_fields():
|
||||
record = {
|
||||
"success": True,
|
||||
"helix_session_uid": "grok:existing",
|
||||
"repo": "other-repo",
|
||||
}
|
||||
result = enrich_helix_correlation(record)
|
||||
assert result["helix_session_uid"] == "grok:existing"
|
||||
assert result["repo"] == "other-repo"
|
||||
|
||||
|
||||
def test_adapter_stub_when_store_unconfigured():
|
||||
adapter = HelixCorrelationAdapter(store_db=None)
|
||||
summary = adapter.lookup("claude:missing")
|
||||
assert summary["adapter"] == "stub"
|
||||
assert summary["status"] == "not_configured"
|
||||
|
||||
|
||||
def test_adapter_sqlite_lookup(tmp_path: Path):
|
||||
db_path = tmp_path / "store.db"
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.execute(
|
||||
"CREATE TABLE digests (session_uid TEXT PRIMARY KEY, json TEXT NOT NULL)"
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE TABLE sessions (session_uid TEXT PRIMARY KEY, json TEXT NOT NULL)"
|
||||
)
|
||||
digest = {
|
||||
"outcome": "success",
|
||||
"cost": {"input_tokens": 800, "output_tokens": 200, "wall_clock_s": 3600},
|
||||
"tool_histogram": {"mcp__state-hub__x": 3, "Bash": 7},
|
||||
"markers": {"errors": 0, "retries": 1},
|
||||
}
|
||||
session = {"repo": "demo-app", "flavor": "claude"}
|
||||
conn.execute(
|
||||
"INSERT INTO digests VALUES (?, ?)",
|
||||
("claude:abc", json.dumps(digest)),
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO sessions VALUES (?, ?)",
|
||||
("claude:abc", json.dumps(session)),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
adapter = HelixCorrelationAdapter(store_db=db_path)
|
||||
summary = adapter.lookup("claude:abc")
|
||||
|
||||
assert summary["adapter"] == "helix-sqlite"
|
||||
assert summary["repo"] == "demo-app"
|
||||
assert summary["flavor"] == "claude"
|
||||
assert summary["fleet_outcome"] == "success"
|
||||
assert summary["tokens"] == 1000
|
||||
assert summary["wall_clock_s"] == 3600
|
||||
assert summary["infra_overhead_share"] == 0.3
|
||||
|
||||
|
||||
class TestHelixCorrelationCli:
|
||||
def test_record_populates_helix_uid_from_env(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
monkeypatch.setenv("HELIX_SESSION_UID", "claude:session-42")
|
||||
monkeypatch.setenv("HELIX_REPO", "kaizen-agentic")
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"metrics",
|
||||
"record",
|
||||
"tdd-workflow",
|
||||
"--target",
|
||||
str(tmp_path),
|
||||
"--success",
|
||||
"--time",
|
||||
"10",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
|
||||
show = runner.invoke(
|
||||
cli,
|
||||
["metrics", "show", "tdd-workflow", "--target", str(tmp_path)],
|
||||
)
|
||||
assert "claude:session-42" in show.output
|
||||
assert "kaizen-agentic" in show.output
|
||||
|
||||
def test_correlate_stub_output(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["metrics", "correlate", "claude:stub-uid"])
|
||||
assert result.exit_code == 0
|
||||
payload = json.loads(result.output)
|
||||
assert payload["helix_session_uid"] == "claude:stub-uid"
|
||||
assert payload["adapter"] == "stub"
|
||||
|
||||
def test_brief_works_with_correlated_metrics(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
memory_dir = tmp_path / ".kaizen" / "agents" / "tdd-workflow"
|
||||
memory_dir.mkdir(parents=True)
|
||||
(memory_dir / "memory.md").write_text(
|
||||
"---\nagent: tdd-workflow\nproject: demo\nsession_count: 1\n---\n\n## Session Log\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setenv("HELIX_SESSION_UID", "claude:brief-test")
|
||||
|
||||
runner = CliRunner()
|
||||
runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"metrics",
|
||||
"record",
|
||||
"tdd-workflow",
|
||||
"--target",
|
||||
str(tmp_path),
|
||||
"--success",
|
||||
"--quality",
|
||||
"0.9",
|
||||
],
|
||||
)
|
||||
brief = runner.invoke(
|
||||
cli,
|
||||
["memory", "brief", "tdd-workflow", "--target", str(tmp_path)],
|
||||
)
|
||||
assert brief.exit_code == 0
|
||||
assert "## Performance Summary" in brief.output
|
||||
33
tests/test_integration_patterns.py
Normal file
33
tests/test_integration_patterns.py
Normal file
@@ -0,0 +1,33 @@
|
||||
"""Smoke tests for WP-0004 integration artifacts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
DEFINITIONS_DIR = (
|
||||
Path(__file__).parent.parent / "docs" / "integrations" / "activity-definitions"
|
||||
)
|
||||
|
||||
|
||||
def test_activity_definitions_have_required_frontmatter():
|
||||
files = list(DEFINITIONS_DIR.glob("*.md"))
|
||||
assert len(files) == 3
|
||||
|
||||
for path in files:
|
||||
text = path.read_text(encoding="utf-8")
|
||||
assert text.startswith("---\n")
|
||||
end = text.index("\n---\n", 4)
|
||||
frontmatter = yaml.safe_load(text[4:end])
|
||||
assert frontmatter["id"]
|
||||
assert frontmatter["trigger"]["type"] in ("cron", "event")
|
||||
assert frontmatter["owner"] == "kaizen-agentic"
|
||||
|
||||
|
||||
def test_integration_docs_exist():
|
||||
root = Path(__file__).parent.parent / "docs"
|
||||
assert (root / "INTEGRATION_PATTERNS.md").exists()
|
||||
assert (root / "integrations" / "helix-forge-correlation.md").exists()
|
||||
assert (root / "integrations" / "optimizer-artifact-manifest.md").exists()
|
||||
143
tests/test_metrics_publish.py
Normal file
143
tests/test_metrics_publish.py
Normal file
@@ -0,0 +1,143 @@
|
||||
"""Tests for artifact-store publish integration (WP-0004 Part 3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from kaizen_agentic.cli import cli
|
||||
from kaizen_agentic.integrations.artifact_store import (
|
||||
PublishResult,
|
||||
build_optimizer_manifest,
|
||||
publish_optimizer_evidence,
|
||||
)
|
||||
from kaizen_agentic.metrics import OptimizerStore
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def project_with_optimizer(tmp_path: Path) -> Path:
|
||||
store = OptimizerStore(tmp_path)
|
||||
store.write_analysis(
|
||||
{
|
||||
"project": "demo",
|
||||
"optimized_at": "2026-06-18",
|
||||
"agents": [{"agent": "tdd-workflow"}],
|
||||
}
|
||||
)
|
||||
store.append_recommendations(
|
||||
"tdd-workflow",
|
||||
[{"type": "reliability", "message": "Improve test stability"}],
|
||||
metrics_count=10,
|
||||
)
|
||||
return tmp_path
|
||||
|
||||
|
||||
def test_build_optimizer_manifest(project_with_optimizer: Path):
|
||||
manifest = build_optimizer_manifest(project_with_optimizer)
|
||||
assert manifest["schema"] == "kaizen-agentic/optimizer-evidence/v1"
|
||||
assert manifest["retention_class"] == "raw-evidence"
|
||||
assert manifest["retention_days"] == 180
|
||||
assert "tdd-workflow" in manifest["agents"]
|
||||
|
||||
|
||||
def test_publish_optimizer_evidence_calls_api(project_with_optimizer: Path):
|
||||
calls: list[tuple[str, str]] = []
|
||||
|
||||
def fake_json(method, base_url, path, token, payload):
|
||||
calls.append((method, path))
|
||||
if path == "/packages":
|
||||
return {"id": "pkg-123"}
|
||||
if path.endswith("/finalize"):
|
||||
return {"id": "pkg-123", "manifest_digest": "blake3:deadbeef"}
|
||||
raise AssertionError(path)
|
||||
|
||||
def fake_multipart(base_url, path, token, **kwargs):
|
||||
calls.append(("POST", path))
|
||||
return {"id": "file-1"}
|
||||
|
||||
with patch(
|
||||
"kaizen_agentic.integrations.artifact_store._http_json",
|
||||
side_effect=fake_json,
|
||||
), patch(
|
||||
"kaizen_agentic.integrations.artifact_store._http_multipart",
|
||||
side_effect=fake_multipart,
|
||||
):
|
||||
result = publish_optimizer_evidence(
|
||||
project_with_optimizer,
|
||||
api_url="http://api.test",
|
||||
token="secret",
|
||||
)
|
||||
|
||||
assert result.package_id == "pkg-123"
|
||||
assert result.files_uploaded == 2
|
||||
assert result.retention_class == "raw-evidence"
|
||||
assert calls[0] == ("POST", "/packages")
|
||||
assert any("/files" in path for _, path in calls)
|
||||
assert calls[-1] == ("POST", "/packages/pkg-123/finalize")
|
||||
|
||||
|
||||
class TestMetricsPublishCli:
|
||||
def test_publish_requires_token(self, project_with_optimizer: Path):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["metrics", "publish", "--target", str(project_with_optimizer)],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "token" in result.output.lower()
|
||||
|
||||
def test_publish_success(self, project_with_optimizer: Path):
|
||||
runner = CliRunner()
|
||||
with patch(
|
||||
"kaizen_agentic.cli.publish_optimizer_evidence",
|
||||
return_value=PublishResult(
|
||||
package_id="pkg-99",
|
||||
manifest_digest="blake3:abc",
|
||||
files_uploaded=2,
|
||||
retention_class="raw-evidence",
|
||||
),
|
||||
):
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"metrics",
|
||||
"publish",
|
||||
"--target",
|
||||
str(project_with_optimizer),
|
||||
"--token",
|
||||
"test-token",
|
||||
"--api-url",
|
||||
"http://127.0.0.1:8000",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "pkg-99" in result.output
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_publish_against_live_artifact_store(project_with_optimizer: Path):
|
||||
"""Optional live test — skipped when artifact-store is unreachable."""
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
api_url = "http://127.0.0.1:8000"
|
||||
try:
|
||||
urllib.request.urlopen(f"{api_url}/health", timeout=2)
|
||||
except (urllib.error.URLError, TimeoutError):
|
||||
pytest.skip("artifact-store not reachable")
|
||||
|
||||
token = __import__("os").environ.get("ARTIFACTSTORE_API_TOKEN")
|
||||
if not token:
|
||||
pytest.skip("ARTIFACTSTORE_API_TOKEN not set")
|
||||
|
||||
result = publish_optimizer_evidence(
|
||||
project_with_optimizer,
|
||||
api_url=api_url,
|
||||
token=token,
|
||||
)
|
||||
assert result.package_id
|
||||
assert result.files_uploaded >= 1
|
||||
@@ -58,17 +58,45 @@ Helix session UIDs on project execution records for correlation.
|
||||
|
||||
**Workplan:** KAIZEN-WP-0004 Part 1.
|
||||
|
||||
#### Worked example
|
||||
|
||||
A TDD8 session captured by Helix Forge and closed with kaizen metrics:
|
||||
|
||||
```bash
|
||||
# Helix capture sets (or operator exports) session identity
|
||||
export HELIX_SESSION_UID="claude:17092961-abc"
|
||||
export HELIX_REPO="kaizen-agentic"
|
||||
export HELIX_FLAVOR="claude"
|
||||
export HELIX_TOKENS="12500"
|
||||
|
||||
# Session close — project layer
|
||||
kaizen-agentic metrics record tdd-workflow --success --time 4200 --quality 0.92
|
||||
|
||||
# Inspect project record (includes correlation fields)
|
||||
kaizen-agentic metrics show tdd-workflow
|
||||
|
||||
# Fleet lookup — read-only, no ingestion in kaizen-agentic
|
||||
export HELIX_STORE_DB=~/.helix-forge/store.db
|
||||
kaizen-agentic metrics correlate claude:17092961-abc
|
||||
```
|
||||
|
||||
Project `executions.jsonl` carries `helix_session_uid` for audit; fleet analytics
|
||||
remain in agentic-resources digest store. Coach `memory brief` surfaces project
|
||||
`## Performance Summary`; correlate adds fleet context when needed.
|
||||
|
||||
Contract: [docs/integrations/helix-forge-correlation.md](../docs/integrations/helix-forge-correlation.md).
|
||||
|
||||
### activity-core (P1)
|
||||
|
||||
**Event bridge** — scheduled and event-driven task creation.
|
||||
|
||||
Example ActivityDefinitions (after metrics CLI ships):
|
||||
ActivityDefinition reference copies (sync into activity-core to activate):
|
||||
|
||||
- Weekly: run `kaizen-agentic metrics optimize` on repos with `.kaizen/`
|
||||
- On low success_rate threshold: create review task in issue-core
|
||||
- Post agent install: verify metrics scaffold exists
|
||||
- [weekly-metrics-optimize](../docs/integrations/activity-definitions/weekly-metrics-optimize.md)
|
||||
- [post-install-metrics-scaffold](../docs/integrations/activity-definitions/post-install-metrics-scaffold.md)
|
||||
- [low-success-rate-review](../docs/integrations/activity-definitions/low-success-rate-review.md)
|
||||
|
||||
**Workplan:** KAIZEN-WP-0004 Part 2.
|
||||
**Workplan:** KAIZEN-WP-0004 Part 2. Patterns: [docs/INTEGRATION_PATTERNS.md](../docs/INTEGRATION_PATTERNS.md).
|
||||
|
||||
### artifact-store (P1)
|
||||
|
||||
@@ -82,6 +110,13 @@ Register after optimizer runs:
|
||||
|
||||
Retention class: `raw-evidence` (180d default, aligned with ADR-004).
|
||||
|
||||
```bash
|
||||
kaizen-agentic metrics optimize
|
||||
kaizen-agentic metrics publish # requires ARTIFACTSTORE_API_URL + TOKEN
|
||||
```
|
||||
|
||||
Manifest: [docs/integrations/optimizer-artifact-manifest.md](../docs/integrations/optimizer-artifact-manifest.md).
|
||||
|
||||
**Workplan:** KAIZEN-WP-0004 Part 3.
|
||||
|
||||
### info-tech-canon (P2)
|
||||
|
||||
@@ -4,17 +4,17 @@ type: workplan
|
||||
title: "Ecosystem Integration: Helix Forge, activity-core, and artifact-store"
|
||||
domain: custodian
|
||||
repo: kaizen-agentic
|
||||
status: active
|
||||
status: completed
|
||||
owner: kaizen-agentic
|
||||
topic_slug: custodian
|
||||
state_hub_workstream_id: 76be7294-e201-4074-91c0-6421992470fe
|
||||
created: "2026-06-16"
|
||||
updated: "2026-06-17"
|
||||
updated: "2026-06-18"
|
||||
---
|
||||
|
||||
# KAIZEN-WP-0004 — Ecosystem Integration: Helix Forge, activity-core, and artifact-store
|
||||
|
||||
**Status:** active
|
||||
**Status:** completed
|
||||
**Owner:** kaizen-agentic
|
||||
**Repo:** kaizen-agentic
|
||||
**Depends on:** KAIZEN-WP-0003 Part 3 (metrics CLI + `metrics optimize` operational)
|
||||
@@ -40,11 +40,11 @@ session ingestion.
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] T01 — Document correlation contract in `agentic-resources` (cross-repo PR or shared doc link from both repos)
|
||||
- [ ] T02 — Add optional `helix_session_uid` population to `metrics record` when env `HELIX_SESSION_UID` is set
|
||||
- [ ] T03 — Add `kaizen-agentic metrics correlate` — lookup Helix digest summary by UID (read-only adapter stub if Helix API not ready)
|
||||
- [ ] T04 — Integration test: synthetic project record with `helix_session_uid` round-trips through show/brief
|
||||
- [ ] T05 — Update `wiki/EcosystemIntegration.md` with worked correlation example
|
||||
- [x] T01 — Document correlation contract in `agentic-resources` (cross-repo PR or shared doc link from both repos)
|
||||
- [x] T02 — Add optional `helix_session_uid` population to `metrics record` when env `HELIX_SESSION_UID` is set
|
||||
- [x] T03 — Add `kaizen-agentic metrics correlate` — lookup Helix digest summary by UID (read-only adapter stub if Helix API not ready)
|
||||
- [x] T04 — Integration test: synthetic project record with `helix_session_uid` round-trips through show/brief
|
||||
- [x] T05 — Update `wiki/EcosystemIntegration.md` with worked correlation example
|
||||
|
||||
### Definition of done
|
||||
|
||||
@@ -60,11 +60,11 @@ Define ActivityDefinitions for recurring kaizen operations.
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] T06 — Draft ActivityDefinition: weekly `metrics optimize` on repos with `.kaizen/metrics/`
|
||||
- [ ] T07 — Draft ActivityDefinition: post-install metrics scaffold validation (`memory init` check)
|
||||
- [ ] T08 — Draft ActivityDefinition: success_rate below 0.8 → issue-core review task
|
||||
- [ ] T09 — Document ActivityDefinition paths and activation contract in `docs/INTEGRATION_PATTERNS.md`
|
||||
- [ ] T10 — Smoke test: manual activation against a test repo with populated metrics
|
||||
- [x] T06 — Draft ActivityDefinition: weekly `metrics optimize` on repos with `.kaizen/metrics/`
|
||||
- [x] T07 — Draft ActivityDefinition: post-install metrics scaffold validation (`memory init` check)
|
||||
- [x] T08 — Draft ActivityDefinition: success_rate below 0.8 → issue-core review task
|
||||
- [x] T09 — Document ActivityDefinition paths and activation contract in `docs/INTEGRATION_PATTERNS.md`
|
||||
- [x] T10 — Smoke test: manual activation against a test repo with populated metrics
|
||||
|
||||
### Definition of done
|
||||
|
||||
@@ -80,11 +80,11 @@ Persist optimizer outputs as registered artifact packages.
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] T11 — Define artifact package manifest for optimizer run (`analysis.json` + `recommendations.jsonl`)
|
||||
- [ ] T12 — Add `kaizen-agentic metrics publish` — register optimizer output with artifact-store API (configurable endpoint)
|
||||
- [ ] T13 — Map retention class `raw-evidence` (180d) in publish manifest metadata
|
||||
- [ ] T14 — Integration test with artifact-store local backend (skip if service unavailable; mark `@pytest.mark.integration`)
|
||||
- [ ] T15 — Document publish workflow in `docs/agency-framework.md` metrics section
|
||||
- [x] T11 — Define artifact package manifest for optimizer run (`analysis.json` + `recommendations.jsonl`)
|
||||
- [x] T12 — Add `kaizen-agentic metrics publish` — register optimizer output with artifact-store API (configurable endpoint)
|
||||
- [x] T13 — Map retention class `raw-evidence` (180d) in publish manifest metadata
|
||||
- [x] T14 — Integration test with artifact-store local backend (skip if service unavailable; mark `@pytest.mark.integration`)
|
||||
- [x] T15 — Document publish workflow in `docs/agency-framework.md` metrics section
|
||||
|
||||
### Definition of done
|
||||
|
||||
@@ -100,10 +100,10 @@ Secondary integrations for template conformance and knowledge asset lifecycle.
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] T16 — Map `wiki/KaizenAgentTemplate.md` sections to info-tech-canon profile outline (design doc only)
|
||||
- [ ] T17 — Draft one InfoTechCanon-style agent brief for `tdd-workflow` pilot
|
||||
- [ ] T18 — Spike: kontextual-engine ingestion manifest for `wiki/` directory (design note, no runtime dependency)
|
||||
- [ ] T19 — Update `history/2026-06-16-ecosystem-assessment.md` with Part 4 outcomes
|
||||
- [x] T16 — Map `wiki/KaizenAgentTemplate.md` sections to info-tech-canon profile outline (design doc only)
|
||||
- [x] T17 — Draft one InfoTechCanon-style agent brief for `tdd-workflow` pilot
|
||||
- [x] T18 — Spike: kontextual-engine ingestion manifest for `wiki/` directory (design note, no runtime dependency)
|
||||
- [x] T19 — Update `history/2026-06-16-ecosystem-assessment.md` with Part 4 outcomes
|
||||
|
||||
### Definition of done
|
||||
|
||||
|
||||
Reference in New Issue
Block a user