generated from coulomb/repo-seed
47 lines
1.5 KiB
Markdown
47 lines
1.5 KiB
Markdown
# Problem Classes
|
|
|
|
`llm_connect.problem_classes` provides generic token estimators for recurring
|
|
LLM workflow shapes.
|
|
|
|
## Contract
|
|
|
|
Every problem class exposes:
|
|
|
|
- `name`: stable registry key.
|
|
- `base_dimensions`: required dimension names supplied by consumers.
|
|
- `tunable_params`: parameters that can be overridden or fitted.
|
|
- `estimate(dimensions, params=None) -> TokenEstimate`.
|
|
- `fit(observations, min_observations=3) -> ProblemClass`.
|
|
|
|
`TokenEstimate` contains `prompt_tokens`, `completion_tokens`, and a
|
|
`confidence` score from `0` to `1`.
|
|
|
|
## Built-Ins
|
|
|
|
| Name | Dimensions | Tunable params |
|
|
|---|---|---|
|
|
| `chunk-summarization` | `chunk_words`, `template_words` | `completion_ratio` |
|
|
| `entity-extraction` | `chunk_words`, `template_words`, `expected_entities` | `tokens_per_entity` |
|
|
| `relation-extraction` | `chunk_words`, `template_words`, `expected_relations` | `tokens_per_relation` |
|
|
| `judge-eval` | `artifact_words`, `template_words`, `n_criteria` | `tokens_per_criterion` |
|
|
| `report-synthesis` | `n_chunks`, `n_entities`, `n_relations`, `template_words` | `base_completion_tokens` |
|
|
|
|
## Observations
|
|
|
|
`fit()` accepts either `Observation` objects or `QualityObservation` rows whose
|
|
`tags` include:
|
|
|
|
```python
|
|
{
|
|
"problem_class": "entity-extraction",
|
|
"dimensions": {
|
|
"chunk_words": 900,
|
|
"template_words": 200,
|
|
"expected_entities": 4,
|
|
},
|
|
}
|
|
```
|
|
|
|
When fewer than `min_observations` usable rows are present, fitting falls back
|
|
to the current parameters.
|