cendor-squeeze — compress

Shrink verbose context — JSON, logs, code, prose — without throwing anything away. Compression returns a handle, and the original is always restorable byte-for-byte. It’s content-aware: each type is routed to a purpose-built, deterministic compressor. No LLM.

pip install cendor-squeeze

npm i @cendor/squeeze

Quickstart

from cendor.squeeze import compress

small, handle = compress(huge_json, kind="auto")                  # detect + route
small, handle = compress(source_code, kind="code", fidelity="aggressive")
small, handle = compress(logs, kind="logs", target_tokens=400)    # compress to a budget
original = handle.expand()                                        # restore, byte-for-byte

import { compress } from '@cendor/squeeze';

let [small, handle] = compress(hugeJson, { kind: 'auto' });                      // detect + route
[small, handle] = compress(sourceCode, { kind: 'code', fidelity: 'aggressive' });
[small, handle] = compress(logs, { kind: 'logs', targetTokens: 400 });           // compress to a budget
const original = handle.expand();                                                // restore, byte-for-byte

See it in the stack. contextkit calls squeeze for you on any Block(evict="compress") — the connected example is in the Cookbook.

Core concepts

Four content-aware compressors

detect() routes by content (or pass kind= to force one). Each is deterministic and needs no model:

Compress to a budget

target_tokens is never exceeded for any kind. JSON shrinks to budget by dropping keys/elements structurally (staying valid JSON); every other kind ends with a hard truncate, so target_tokens is lossy in the emitted output. That loss is fully reversible — the original stays in the store, so handle.expand() returns it in full no matter how tight the budget was. fidelity (lossless / balanced / aggressive) separately trades structure for size.

Reversibility (the content-addressed store)

Every original is kept in a content-addressed store keyed by its sha256 hash (deduped across calls), so handle.expand() is always exact — no matter how hard you squeeze. The backend is pluggable via use_store(...).

Persisting across restarts

The default store is an in-memory one, so to restore after a process restart you persist the handle next to a durable store:

data = handle.to_dict()                 # {id, kind, original_ref, restore_map} — not the original
# ...next process, with the same SQLiteStore active...
from cendor.squeeze import Handle
original = Handle.from_dict(data).expand()

const data = handle.toDict();           // {id, kind, original_ref, restore_map} — not the original
// ...next process, with the same SQLiteStore active...
import { Handle } from '@cendor/squeeze';
const original = Handle.fromDict(data).expand();

Functions & classes

compress()

compress(content, kind="auto", target_tokens=None, model="gpt-4o", fidelity="balanced")  # -> (small, handle)

compress(content, { kind = 'auto', targetTokens = null,
                    model = 'gpt-4o', fidelity = 'balanced' })   // -> [small, handle]

Helpers & classes

Store backends

from cendor.squeeze import use_store
from cendor.squeeze.store import MemoryStore, SQLiteStore

use_store(SQLiteStore("ccr.db"))        # originals persist across processes
use_store(MemoryStore(max_items=1000))  # bounded in-memory (LRU eviction)

import { useStore, MemoryStore, SQLiteStore } from '@cendor/squeeze';

useStore(new SQLiteStore('ccr.db'));    // originals persist across processes (better-sqlite3)
useStore(new MemoryStore(1000));        // bounded in-memory (LRU eviction)

The default is an unbounded MemoryStore. SQLiteStore opens with check_same_thread=False, so one store can serve a threaded server (writes are idempotent). A bounded store can evict an original; expanding an evicted handle raises KeyError — the documented trade-off of a capped store.

How it works

%%{init: {"flowchart": {"htmlLabels": false}} }%%
graph LR
    C["content<br/>(str or object)"]
    D{"detect kind"}
    J["JSON<br/>minify · drop nulls"]
    L["logs<br/>normalize · dedup ×N"]
    K["code<br/>strip comments"]
    P["prose<br/>extractive ranking"]
    SM["small text<br/>(within target_tokens)"]
    H["Handle"]
    CCR["content-addressed store<br/>sha256 → original"]
    EXP["expand → original<br/>byte-for-byte"]

    C --> D
    D -->|json| J --> SM
    D -->|logs| L --> SM
    D -->|code| K --> SM
    D -->|prose| P --> SM
    SM --> H
    C -->|"store original"| CCR
    H -->|"reads"| CCR --> EXP

    classDef sq fill:#22C55E,color:#0F172A,stroke:#16A34A;
    classDef co fill:#94A3BB,color:#0F172A,stroke:#64748B;
    class J,L,K,P,H sq;
    class CCR co;

Content is routed by kind to a deterministic compressor; the original is stashed in the content-addressed store, so expand() is always byte-exact regardless of how hard you squeezed.

Plugs into the stack

Inbound. Usually contextkit calls squeeze for you when a block is marked evict="compress" (pip install cendor-contextkit[squeeze]) — it satisfies core’s Compressor protocol by shape, so contextkit never imports it. Call it directly to shrink a single known-huge blob (e.g. a 50k-token tool response) before it enters the window. It operates purely on strings/objects — identical across any SDK, never touching the client.

Honest limits

• Structural compressors are deterministic and need no LLM. Prose is extractive (deterministic); an LLM-summarization backend isn’t bundled, but the technique is pluggable.
• Token-reduction percentage depends on the tokenizer; reversibility is exact regardless.
• The benchmarks are the honest numbers. Headline ratios (JSON ~49% / code ~53% / prose ~49%, and logs anywhere from ~99% on repetition-heavy logs down to ~30% on high-entropy logs) come from the harness on realistic corpora — see Benchmarks, the source of truth. Eye-popping figures on synthetic, highly-repetitive data are not representative.