cendor-squeeze — compress

Shrink verbose context — JSON, logs, code, prose — without throwing anything away. Compression returns a handle, and the original is always restorable byte-for-byte. It’s content-aware: each type is routed to a purpose-built, deterministic compressor. No LLM.

pip install cendor-squeeze
# Using uv? Same names, same extras: `uv add` instead of `pip install`.

npm i @cendor/squeeze

Quickstart

from cendor.squeeze import compress

small, handle = compress(huge_json, kind="auto")                  # detect + route
small, handle = compress(source_code, kind="code", fidelity="aggressive")
small, handle = compress(logs, kind="logs", target_tokens=400)    # compress to a budget
original = handle.expand()                                        # restore, byte-for-byte

import { compress } from '@cendor/squeeze';

let [small, handle] = compress(hugeJson, { kind: 'auto' });                      // detect + route
[small, handle] = compress(sourceCode, { kind: 'code', fidelity: 'aggressive' });
[small, handle] = compress(logs, { kind: 'logs', targetTokens: 400 });           // compress to a budget
const original = handle.expand();                                                // restore, byte-for-byte

See it in the stack. contextkit calls squeeze for you on any Block(evict="compress") — the connected example is in the Cookbook.

Core concepts

Four content-aware compressors

detect() routes by content (or pass kind= to force one). Each is deterministic and needs no model:

Compress to a budget

target_tokens is never exceeded for any kind. JSON shrinks to budget by dropping keys/elements structurally (staying valid JSON); every other kind ends with a hard truncate, so target_tokens is lossy in the emitted output. That loss is fully reversible — the original stays in the store, so handle.expand() returns it in full no matter how tight the budget was. fidelity (lossless / balanced / aggressive) separately trades structure for size.

Reversibility (the content-addressed store)

Every original is kept in a content-addressed store keyed by its sha256 hash (deduped across calls), so handle.expand() is always exact — no matter how hard you squeeze. The backend is pluggable via use_store(...).

Persisting across restarts

The default store is an in-memory one, so to restore after a process restart you persist the handle next to a durable store:

data = handle.to_dict()                 # {id, kind, original_ref, restore_map} — not the original
# ...next process, with the same SQLiteStore active...
from cendor.squeeze import Handle
original = Handle.from_dict(data).expand()

const data = handle.toDict();           // {id, kind, original_ref, restore_map} — not the original
// ...next process, with the same SQLiteStore active...
import { Handle } from '@cendor/squeeze';
const original = Handle.fromDict(data).expand();

Functions & classes

compress()

compress(content, kind="auto", target_tokens=None, model="gpt-4o", fidelity="balanced")  # -> (small, handle)

compress(content, { kind = 'auto', targetTokens = null,
                    model = 'gpt-4o', fidelity = 'balanced' })   // -> [small, handle]

Helpers & classes

Store backends

from cendor.squeeze import use_store
from cendor.squeeze.store import MemoryStore, SQLiteStore

use_store(SQLiteStore("ccr.db"))        # originals persist across processes
use_store(MemoryStore(max_items=1000))  # bounded in-memory (LRU eviction)

import { useStore, MemoryStore, SQLiteStore } from '@cendor/squeeze';

useStore(new SQLiteStore('ccr.db'));    // originals persist across processes (better-sqlite3)
useStore(new MemoryStore(1000));        // bounded in-memory (LRU eviction)

The default is an unbounded MemoryStore. SQLiteStore opens with check_same_thread=False, so one store can serve a threaded server (writes are idempotent). A bounded store can evict an original; expanding an evicted handle raises KeyError — the documented trade-off of a capped store.

TypeScript on Node 20: better-sqlite3 needs a build toolchain. SQLiteStore is backed by the optional native better-sqlite3, and on linux-x64 Node 20 the pinned 12.x line publishes no prebuilt binary — npm install runs prebuild-install || node-gyp rebuild and exits 1 unless python3, make and a C++ compiler are present. With them it compiles from source and works (measured on the node:20 image); without them the import fails with ERR_MODULE_NOT_FOUND (measured on node:20-slim). Node 22+ installs a prebuilt binary and needs nothing.

⚠️ Do not reach for better-sqlite3@13 as the workaround — the obvious fix, and it is worse. It installs cleanly on Node 20 and then segfaults on the first new Database() (measured, node:20-slim linux-x64), trading a loud install error for a silent process crash. That is why the optional dependency stays at ^12.11.1. On Node 22 both lines are fine.

npm silently skips an optionalDependency it cannot build, so npm install succeeds and the failure only appears at first import. MemoryStore needs no native module and is unaffected.

How it works

%%{init: {"flowchart": {"htmlLabels": false}} }%%
graph LR
    C["content<br/>(str or object)"]
    D{"detect kind"}
    J["JSON<br/>minify · drop nulls"]
    L["logs<br/>normalize · dedup ×N"]
    K["code<br/>strip comments"]
    P["prose<br/>extractive ranking"]
    SM["small text<br/>(within target_tokens)"]
    H["Handle"]
    CCR["content-addressed store<br/>sha256 → original"]
    EXP["expand → original<br/>byte-for-byte"]

    C --> D
    D -->|json| J --> SM
    D -->|logs| L --> SM
    D -->|code| K --> SM
    D -->|prose| P --> SM
    SM --> H
    C -->|"store original"| CCR
    H -->|"reads"| CCR --> EXP

    classDef sq fill:#22C55E,color:#0F172A,stroke:#16A34A;
    classDef co fill:#94A3BB,color:#0F172A,stroke:#64748B;
    class J,L,K,P,H sq;
    class CCR co;

Content is routed by kind to a deterministic compressor; the original is stashed in the content-addressed store, so expand() is always byte-exact regardless of how hard you squeezed.

Plugs into the stack

Inbound. Usually contextkit calls squeeze for you when a block is marked evict="compress" (pip install cendor-contextkit[squeeze]) — it satisfies core’s Compressor protocol by shape, so contextkit never imports it. Call it directly to shrink a single known-huge blob (e.g. a 50k-token tool response) before it enters the window. It operates purely on strings/objects — identical across any SDK, never touching the client.

See it live. squeeze emits a CompressionEvent on the bus (technique, before/after tokens, ratio — metadata only, never text); it renders on the squeeze proof page in Cendor Monitor, Cendor’s optional self-hosted monitor, so you can see the tokens compression saved. The event is computed only when the bus has a subscriber (cendor-squeeze ≥ 1.1.2 / @cendor/squeeze ≥ 3.1.0): filling it means token-counting the original and the compressed text, which dominates a large compress() — so with nothing attached the counting is skipped entirely, and with an audit log or a monitor attached it is the (paid-for) cost of visibility.

Honest limits

• Structural compressors are deterministic and need no LLM. Prose is extractive (deterministic); an LLM-summarization backend isn’t bundled, but the technique is pluggable.
• Token-reduction percentage depends on the tokenizer; reversibility is exact regardless.
• expand() is byte-for-byte for string input. When you pass an object (dict/list), compress() first serializes it with compact json.dumps and stores that — so expand() returns the canonical JSON string, not your original object (key order is preserved, but incidental whitespace and an int-valued float like 1.0 are normalized). Pass a string if you need the exact original bytes back.
• The benchmarks are the honest numbers. Headline ratios (JSON ~49% / prose ~49%, and logs anywhere from ~99% on repetition-heavy logs down to ~30% on high-entropy logs) come from the harness on realistic corpora — see Benchmarks, the source of truth. Eye-popping figures on synthetic, highly-repetitive data are not representative.
• Code compresses modestly (~10–17%), not dramatically. The code path strips only comments, blank lines, and trailing whitespace, and it is string-literal-aware — string literals and docstrings are preserved. So the ratio tracks the input’s comment density: comment-sparse code saves little; a comment-heavy or auto-generated file saves much more. For the same reason fidelity="aggressive" ≈ "balanced" on normally-spaced code.