generated from coulomb/repo-seed
108 lines
3.4 KiB
Markdown
108 lines
3.4 KiB
Markdown
# llm-connect Feature Requests
|
||
|
||
Raised by: IHF Phase 11 — Advanced AI Federation (IHUB-WP-0012)
|
||
Date: 2026-04-01
|
||
|
||
These gaps were identified during integration of llm-connect into the
|
||
Interaction Hub Framework (IHF) as a subprocess bridge for multi-agent
|
||
federation. None are blockers for Phase 11, but they affect performance
|
||
and architectural elegance.
|
||
|
||
---
|
||
|
||
## FR-1 — HTTP/JSON-RPC serve mode
|
||
|
||
**Problem:** The current architecture requires spawning a new `python3
|
||
scripts/llm_bridge.py` process for every agent invocation. This adds
|
||
significant overhead in production when collective proposals invoke 3–5
|
||
agents in sequence.
|
||
|
||
**Proposed API:**
|
||
```bash
|
||
python -m llm_connect.server --port 9999
|
||
```
|
||
IHP (Haskell) would call `POST localhost:9999/execute` with the same JSON
|
||
payload the bridge script currently reads from stdin.
|
||
|
||
**Impact:** Eliminates process spawn overhead. A single persistent server
|
||
process handles all requests in the session lifetime.
|
||
|
||
---
|
||
|
||
## FR-2 — `RoutingPolicy` class for declarative provider/model selection
|
||
|
||
**Problem:** `RunConfig.model_name` is the only selection mechanism. IHF
|
||
needs declarative routing rules — e.g. "for triage tasks, prefer
|
||
openrouter/claude-haiku-4-5; fall back to gemini if cost exceeds 0.5/1k
|
||
tokens; never use auto_apply trust agents for autonomous actions".
|
||
|
||
**Proposed API:**
|
||
```python
|
||
from llm_connect import RoutingPolicy
|
||
|
||
policy = RoutingPolicy(rules=[
|
||
{
|
||
"task_type": "triage",
|
||
"prefer": [{"provider": "openrouter", "model": "claude-haiku-4-5"}],
|
||
"max_cost_per_1k": 0.5,
|
||
"fallback": {"provider": "gemini", "model": "gemini-flash-1.5"},
|
||
}
|
||
])
|
||
adapter = policy.resolve(task_type="triage")
|
||
```
|
||
|
||
**Impact:** Moves routing logic into llm-connect instead of duplicating it
|
||
in every consumer (currently IHF implements this in `ModelRouter.hs`).
|
||
|
||
---
|
||
|
||
## FR-3 — `async_execute_prompt()` for concurrent execution
|
||
|
||
**Problem:** Collective proposals invoke agents sequentially because
|
||
`execute_prompt` is synchronous. With 3–5 agents this is 3–5× slower than
|
||
necessary.
|
||
|
||
**Proposed API:**
|
||
```python
|
||
import asyncio
|
||
from llm_connect import create_adapter
|
||
|
||
async def main():
|
||
adapters = [create_adapter(...) for _ in agents]
|
||
responses = await asyncio.gather(*[
|
||
a.async_execute_prompt(prompt, config) for a in adapters
|
||
])
|
||
```
|
||
|
||
Standard `asyncio` coroutine interface, same signature as `execute_prompt`.
|
||
|
||
**Impact:** Collective proposal latency scales with the slowest agent
|
||
rather than the sum of all agent latencies.
|
||
|
||
---
|
||
|
||
## FR-4 — `BudgetTracker` for delegation chains
|
||
|
||
**Problem:** IHF's inter-agent delegation model enforces token budgets at
|
||
the Haskell layer (`AgentDelegation.tokenBudget`), but the bridge itself
|
||
has no concept of a shared budget. A delegation chain (A → B → C) cannot
|
||
enforce that the total token spend stays below a cap set by A.
|
||
|
||
**Proposed API:**
|
||
```python
|
||
from llm_connect import BudgetTracker, RunConfig
|
||
|
||
tracker = BudgetTracker(total=4000)
|
||
config = RunConfig(model_name="...", budget_tracker=tracker)
|
||
# Subsequent calls on any adapter sharing this tracker will raise
|
||
# LLMBudgetExceededError if the cumulative spend exceeds 4000 tokens.
|
||
resp = adapter.execute_prompt(prompt, config)
|
||
```
|
||
|
||
`LLMBudgetExceededError` should be a subclass of `LLMError` so existing
|
||
error handling catches it.
|
||
|
||
**Impact:** Budget enforcement moves into the bridge layer where it can be
|
||
applied uniformly across all providers, rather than requiring each consumer
|
||
to track it manually.
|