SAND-WP-0004: delegate make e2e to validate run
Replace e2e_framework monolith with wise-validator + sand-boxer shim. Makefile invokes validate run; legacy python -m e2e_framework delegates via shim.py with deprecation notice. Add verify-e2e-shim.sh.
This commit is contained in:
@@ -1,97 +1,82 @@
|
||||
# E2E Sandbox Framework — Runbook
|
||||
|
||||
> **Migrated (2026-06-23):** `make e2e REPO=` and `python -m e2e_framework` now
|
||||
> delegate to **wise-validator** (`validate run`) + **sand-boxer** (`sandboxer
|
||||
> create`). The modules in this directory are **deprecated** and will be removed
|
||||
> after one release cycle.
|
||||
>
|
||||
> **Canonical runbooks:**
|
||||
> - [wise-validator: validate-compose-e2e](~/wise-validator/docs/runbooks/validate-compose-e2e.md)
|
||||
> - [sand-boxer: profile-compose-e2e](~/sand-boxer/docs/runbooks/profile-compose-e2e.md)
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
**Workstation:**
|
||||
- `ssh` + `rsync` available
|
||||
- `python3` + `pyyaml` available (or `uv run`)
|
||||
- State-hub running on `:8000` (for result reporting)
|
||||
|
||||
**Sandbox host (railiance01):**
|
||||
- `validate` on PATH (`cd ~/wise-validator && make install`)
|
||||
- `sandboxer` on PATH (`cd ~/sand-boxer && make install`)
|
||||
- `ssh` available (BatchMode; respects `~/.ssh/config`)
|
||||
- State Hub on `:8000` (optional, for result reporting)
|
||||
|
||||
**Sandbox host (CoulombCore / sandboxer01):**
|
||||
|
||||
- SSH key access
|
||||
- Docker + docker compose plugin installed
|
||||
- `podman-compose` or `docker compose` (`SANDBOXER_COMPOSE_CMD` on CoulombCore)
|
||||
- Sufficient disk for images (~4 GB for activity-core stack)
|
||||
|
||||
## First run
|
||||
|
||||
```bash
|
||||
# Set sandbox host (once, or add to ~/.bashrc / .env)
|
||||
export RAILIANCE01_HOST=<ip-or-alias> # e.g. 92.205.130.254
|
||||
export RAILIANCE01_USER=root # optional, default=root
|
||||
export RAILIANCE01_KEY=~/.ssh/id_rsa # optional, uses ssh default otherwise
|
||||
export SANDBOXER_HOST=92.205.130.254 # CoulombCore; or RAILIANCE01_HOST (legacy)
|
||||
export SANDBOXER_COMPOSE_CMD=podman-compose
|
||||
|
||||
# From the-custodian:
|
||||
make e2e REPO=activity-core
|
||||
```
|
||||
|
||||
Output will show each step: rsync → compose up → health wait → tests → compose down.
|
||||
Exit code is 0 (all passed) or 1 (any failure).
|
||||
Output: sandbox create → health wait → tests → destroy. Exit 0 = pass, 1 = fail.
|
||||
|
||||
## Options
|
||||
|
||||
```bash
|
||||
# Keep sandbox alive after run (for debugging)
|
||||
make e2e REPO=activity-core KEEP=1
|
||||
|
||||
# Override host without env var
|
||||
make e2e REPO=activity-core HOST=192.168.1.50
|
||||
|
||||
# Attach result to a specific state-hub workstream
|
||||
make e2e REPO=activity-core HOST=92.205.130.254
|
||||
make e2e REPO=activity-core WORKSTREAM_ID=<uuid>
|
||||
make e2e REPO=activity-core NO_REPORT=1
|
||||
|
||||
# Skip posting to state-hub
|
||||
cd the-custodian && python3 -m e2e_framework ~/activity-core --no-report
|
||||
# Legacy entry (prints deprecation, delegates to validate run):
|
||||
python3 -m e2e_framework ~/activity-core --host $SANDBOXER_HOST
|
||||
```
|
||||
|
||||
## Adding a new repo
|
||||
|
||||
1. Create `<repo>/e2e/e2e.yml`:
|
||||
```yaml
|
||||
name: <repo-slug>
|
||||
compose_file: docker-compose.dev.yml # or e2e/compose.yml
|
||||
health_checks:
|
||||
- name: <service>
|
||||
url: http://localhost:<port>
|
||||
timeout: 120
|
||||
test_command: uv run python -m pytest e2e/tests/ -v
|
||||
timeout: 300
|
||||
cleanup: always
|
||||
```
|
||||
1. Create `<repo>/e2e/e2e.yml` (see wise-validator runbook for schema).
|
||||
2. Add tests under `<repo>/e2e/tests/` or inline `test_command`.
|
||||
3. Run: `make e2e REPO=<repo>` or `validate run ~/<repo>`.
|
||||
|
||||
2. Add `<repo>/e2e/tests/test_*.py` — test scripts that exit 0 on success.
|
||||
## Verification
|
||||
|
||||
3. Run: `make e2e REPO=<repo>`
|
||||
```bash
|
||||
./scripts/verify-e2e-shim.sh
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Sandbox not cleaned up:**
|
||||
```bash
|
||||
ssh root@$RAILIANCE01_HOST 'ls /tmp/custodian-e2e/'
|
||||
ssh root@$RAILIANCE01_HOST 'docker compose ls'
|
||||
# Manually clean:
|
||||
ssh root@$RAILIANCE01_HOST 'docker compose -p e2e-activity-core-<id> down -v; rm -rf /tmp/custodian-e2e/<id>'
|
||||
**`validate` / `sandboxer` not found:** Install wise-validator and sand-boxer CLIs.
|
||||
|
||||
**CoulombCore compose failures:** Set `SANDBOXER_COMPOSE_CMD=podman-compose`; use
|
||||
fully qualified image names in compose files.
|
||||
|
||||
**Stale sandboxes:** `sandboxer inspect stale` / `sandboxer reap-stale --apply`
|
||||
|
||||
## Architecture (current)
|
||||
|
||||
```
|
||||
make e2e REPO= → validate run → sandboxer create (sand-boxer)
|
||||
→ health + test (wise-validator)
|
||||
→ sandboxer destroy
|
||||
```
|
||||
|
||||
**Temporal startup slow (>2 min):**
|
||||
Elasticsearch takes 60–90 seconds. The health check waits up to 180s.
|
||||
If it times out, check:
|
||||
```bash
|
||||
ssh root@$RAILIANCE01_HOST 'docker logs temporal-elasticsearch | tail -20'
|
||||
```
|
||||
|
||||
**Worker fails to start:**
|
||||
Check that `uv` is installed on the sandbox host:
|
||||
```bash
|
||||
ssh root@$RAILIANCE01_HOST 'which uv || curl -LsSf https://astral.sh/uv/install.sh | sh'
|
||||
```
|
||||
|
||||
**rsync excluded paths:**
|
||||
`.git`, `__pycache__`, `*.pyc`, `.venv`, `node_modules` are excluded.
|
||||
This means `uv sync` runs on the remote after rsync (handled by `uv run`).
|
||||
|
||||
## Architecture notes
|
||||
|
||||
- Sandbox isolation: docker compose project name `e2e-{repo}-{sandbox_id}`
|
||||
- Sandbox dir: `/tmp/custodian-e2e/{sandbox_id}/`
|
||||
- No port conflicts: each sandbox uses its own docker network
|
||||
- Parallel runs of the same repo are safe (different sandbox_id)
|
||||
Legacy `e2e-framework/sandbox.py` provision path is **not** used by `make e2e`.
|
||||
Reference in New Issue
Block a user