Benchmarks

Reproducible, offline measurements of every package in the stack — both the headline claims (compression ratios, token-count accuracy, tamper detection) and runtime cost (throughput, per-call overhead). There is no network and no API key anywhere in the suite: model calls use fake, provider-shaped clients, and timing is plain time.perf_counter.

How to reproduce

uv run python benchmarks/run_all.py            # all tables below
uv run --with tiktoken python benchmarks/run_all.py   # adds exact-token accuracy

Environment

cendor-core

One instrument() seam, provider-aware token counting, and offline pricing — measured for accuracy against tiktoken and for the per-call overhead the seam adds.

cendor-contextkit

Packing prioritized blocks into a token budget: how tightly it fills the budget, that it never overflows, and how fast it assembles.

cendor-squeeze

Content-aware, reversible compression: how much each kind shrinks (by characters and tokens), that every compression restores byte-for-byte, and throughput.

Code caveat (honest). The code path strips only comments, blank lines, and trailing whitespace, and it is string-literal-aware — string literals and docstrings are preserved verbatim. So the ratio tracks the input’s comment/whitespace density, not a fixed savings: ordinary comment-sparse code compresses ~10–17% (the number above, measured on a representative module), while a comment-heavy or auto-generated file can exceed 50%. Because docstrings are kept, fidelity="aggressive" ≈ "balanced" for normally-spaced code (aggressive only collapses runs of inner whitespace).

cendor-tokenguard

Budget enforcement + spend attribution as a bus subscriber: the cost it adds per call and how fast it aggregates spend.

cendor-guardrails

A deterministic gate at four intervention points: per-check latency for each built-in rule, the cost of a small pass-through gate, and the per-call overhead the interceptor adds.

cendor-cassette

Record once, replay forever: a full run replayed vs live, the per-call replay overhead, and meaning-based matching.

cendor-acttrace

A tamper-evident hash chain with no server: append/verify throughput, signing cost, and that a single edited byte is caught.

PII / secret detection — acttrace catalogue

Detection quality for the regex/validator catalogue the guardrails PII bridge (rules.pii / secrets / entropy) leans on: per-group precision/recall on a small synthetic corpus, the false-positive rate on look-alikes, and the regex-vs-NER split for free-text names/addresses. Read the corpus caveat below before quoting any number.

Method & caveats

• No network, no keys. Every model call is a fake client matching the provider’s shape; usage and responses are synthetic but realistic.
• Token accuracy compares the offline heuristic to tiktoken (the real OpenAI tokenizer). With the [tiktoken] extra installed, OpenAI counts are exact (0% error); the heuristic is the zero-dependency fallback. Claude/Gemini have no offline native tokenizer, so that row is a cross-tokenizer ballpark, not ground truth.
• Cassette speedup models a real call with a fake client that sleeps a few milliseconds; production LLM calls are 100×–1000× slower, so the real-world speedup is far larger than shown here.
• Compression ratios depend heavily on input shape and repetition (described per row). The log rows report both a repetition-heavy sample (~55% identical heartbeat lines, as much production log traffic is) and a mixed-entropy sample (~15% heartbeats, the rest distinct) — the mixed row is the honest lower bound. Inputs are typical verbose payloads, not adversarially chosen to flatter the compressors, but the headline log ratio is repetition-driven; read the mixed-entropy row for less repetitive logs.
• Throughput numbers are single-machine and relative; they vary with hardware. Re-run locally for your own figures.
• PII/secret detection is measured on a small, hand-labelled synthetic corpus (benchmarks/bench_pii_detectors.py) — every value is fabricated (test keys, Luhn-valid but non-issued cards, RFC-5737 documentation IPs), modelled on the formats used by public PII corpora (Presidio’s generator, Faker, the AWS Comprehend entity list) but scraping none of them, so the suite stays offline. The precision/recall figures establish the methodology and per-group behaviour of the shipped catalogue; they are not a headline ‘we catch X% of PII’ claim — that needs a larger corpus from a licensed public dataset. The regex catalogue targets structured PII (patterns + checksum validators), so its recall on free-text names/addresses is ~0 by design; the optional Presidio NER backend (cendor-acttrace[ner]) covers those and is measured only when installed.