Measured,
not promised.
Reproducible, offline measurements of every package — headline claims and runtime cost. No network and no API key anywhere in the suite: model calls use fake, provider-shaped clients.
$ uv run python benchmarks/run_all.py # all tables below $ uv run --with tiktoken python benchmarks/run_all.py # adds exact-token accuracy
Five numbers we stand behind.
Token counting, Decimal pricing, and the event bus every tool rides — measured for accuracy first, speed second.
| Metric | Result | Notes |
|---|---|---|
| Offline heuristic error vs tiktoken — prose | 35.8% | heuristic 163 vs exact 120 tokens |
| Offline heuristic error vs tiktoken — code | 8.4% | heuristic 103 vs exact 95 tokens |
| Offline heuristic error vs tiktoken — json | 18.4% | heuristic 62 vs exact 76 tokens |
| Exact mode error (default) | 0.0% | OpenAI counts are exact out of the box — tiktoken is a required dependency |
| Offline subword fallback vs o200k (Claude/Gemini) | 33.2% | the defensive no-tiktoken fallback; by default Claude/Gemini use o200k directly |
| tokens.count throughput — OpenAI heuristic | 1.24M ops/s | on a 1.4 KB string |
| tokens.count throughput — tiktoken exact | 9.7K ops/s | on a 1.4 KB string |
| instrument() overhead per call | 11.66 µs | bus emit + usage extraction + Decimal pricing; over a no-op client |
| bus dispatch (3 subscribers) | 1.96M emits/s | synchronous fan-out to subscribed tools |
Context assembly under a hard token budget — the claim isn't speed, it's that the budget is never exceeded and the output is deterministic.
| Metric | Result | Notes |
|---|---|---|
| Budget utilization | 100% | used 3500/3500 tokens (reserve 500); never overflows |
| Overflow safety | 0 over budget | 3/25 blocks kept/shrunk, rest dropped by priority |
| Determinism | exact ✓ | identical inputs → byte-identical messages |
| assemble() latency (25 blocks) | 23.80 ms | includes per-block token counting + eviction + ordering |
Reversible payload compression — ratios depend heavily on input shape, which is why the logs row appears twice.
| Metric | Result | Notes |
|---|---|---|
| JSON compression | 48.9% | 90.1 KB → 46.0 KB; 50.1% fewer tokens |
| Logs (repetitive) compression | 99.7% | 70.1 KB → 0.2 KB; 99.8% fewer tokens |
| Logs (mixed-entropy) compression | 30.1% | 80.9 KB → 56.5 KB; 35.9% fewer tokens |
| Code compression | 52.5% | 11.9 KB → 5.7 KB; 42.4% fewer tokens |
| Prose compression | 49.1% | 8.6 KB → 4.4 KB; 46.6% fewer tokens |
| Reversibility (expand() == original) | 5/5 exact | every kind restores byte-for-byte |
| compress() throughput (JSON) | 56 MB/s | 90 KB payload, 1.57 ms/call |
Budgets and spend tracking layered on instrument() — the question is what @budget adds to every call.
| Metric | Result | Notes |
|---|---|---|
| Added overhead per call (@budget + track) | 4.09 µs | records spend by tags + checks the active budget(s) |
| report() over 5000 spend rows | 7.64 ms | group-by aggregation into per-tag cost rows |
Record once, replay forever — the speedup is measured against a fake client sleeping 4 ms/call, a deliberate floor.
| Metric | Result | Notes |
|---|---|---|
| 25-call run: replayed vs live | 852.44 µs vs 116.03 ms | live = fake client sleeping 4 ms/call (real LLMs are far slower) |
| Replay speedup | 136× | at the modeled 4 ms/call; scales with real latency |
| Replay overhead per call | 34.10 µs | hash the request, look up the recorded response, reconstruct it |
A tamper-evident hash chain with no server — append/verify throughput, signing cost, and that a single edited byte is caught.
| Metric | Result | Notes |
|---|---|---|
| Append throughput (in-memory) | 16.4K entries/s | sha256 chain + default PII redaction per entry |
| HMAC signing overhead | +2% | per-entry HMAC-SHA256 on top of the chain hash |
| Append throughput (file-backed) | 2.5K entries/s | flush + fsync a JSONL line per entry on a kept-open handle |
| verify() throughput | 58.7K entries/s | re-walks a 2001-entry chain in 34.1 ms |
| Tamper detection | ✓ detected | one edited byte → chain hash mismatch → verify() returns False |
How the numbers were made — read before quoting.
Every caveat below is part of the result, not fine print. If a number above looks too good, its honest bound is here.
Every model call is a fake client matching the provider's shape.
The offline heuristic is compared to tiktoken's exact counts; the Claude/Gemini rows are a cross-tokenizer ballpark, not ground truth.
Production LLM calls are 100–1000× slower, so real-world speedup is far larger — the number shown is a floor of the mechanism, not a promise.
The log rows report both a repetition-heavy sample (~55% identical heartbeat lines) and a mixed-entropy one (~15%) — the mixed row is the honest lower bound.
Re-run locally for your own figures: uv run python benchmarks/run_all.py.