Token-efficient serialization to reduce structured output overhead in production pipelines

[makroumi]

Is your feature request related to a problem?

Instructor's core value is structured outputs - getting LLMs to return valid, typed data reliably.

The irony: the JSON wire format used to achieve this structure is itself one of the largest token cost drivers in production pipelines.

Every Instructor call serializes:

• The JSON schema in the prompt
• The structured output response
• Retry attempts with validation errors
• Patch history across retries

~44% of tokens in typical Instructor payloads
are pure JSON syntax overhead.

At scale this compounds fast:

• Schema overhead on every single call
• Repeated field names across every retry
• Validation error payloads add more JSON
• Multi-step pipelines multiply the overhead

At 10M Instructor calls on GPT-4o:
~$59K spent on syntax noise. Not intelligence.

The frustration: Instructor already does the hard work of structured validation. The JSON wire format undermines that efficiency at scale.

Describe the solution you'd like

A pluggable serializer interface allowing token-efficient wire formats as opt-in replacement for JSON in Instructor pipelines.

I built ULMEN specifically for this problem.

Benchmarks on NVIDIA Tesla T4:

The natural fit with Instructor specifically:

Instructor validates structure on the Python side.
ULMEN's Semantic Firewall extends this to the wire:

• Validates structured output schemas
• Rejects malformed responses before retry
• Catches invalid enum states
• Raises structured errors vs silent failures

This aligns with Instructor's core philosophy:
never pass broken structure downstream.

How code might look:

import instructor
from openai import OpenAI

Current

client = instructor.from_openai(OpenAI())

Proposed

client = instructor.from_openai(
OpenAI(),
serializer="ulmen"
)

Per-call override

response = client.chat.completions.create(
model="gpt-4o",
response_model=MyModel,
serializer="ulmen",
messages=[...]
)

Pydantic model definitions unchanged.
ULMEN handles wire format transparently.
Pure Python fallback if Rust unavailable.
BSL license, free under $10M revenue.

Reproducible benchmark notebook:
github.com/makroumi/ulmen

Describe alternatives you've considered

1. orjson: Faster but identical token count. Doesn't address context window overhead.

2. Manual schema compressionLossy. Breaks with model changes.Not systematic across pipelines.

3. Reducing retry attempts: Trades reliability for cost. Wrong tradeoff for production systems.

4. Smaller models: Reduces capability not just cost.Wrong lever for this specific problem.

ULMEN addresses the root cause: JSON was designed for web APIs not LLM context windows.

Additional Context

Instructor users running structured extraction pipelines at scale are the most affected by this problem because:

1. Schema overhead appears on EVERY call
2. Retry payloads add compounding JSON overhead
3. Batch extraction pipelines multiply the cost
4. Multi-step pipelines chain the overhead

The teams most likely to benefit:

• Document extraction pipelines at scale
• Classification systems with high call volume
• Any production Instructor deployment over 1M calls per month

Reproducible benchmark notebook:
github.com/makroumi/ulmen

[rehan243]

Oh interesting, we've run into this too, especially with schema-heavy prompts bloating token counts. You're spot-on—JSON is brutal for token overhead, especially with retries piling up. We actually hacked around this in one of our pipelines by switching to a compact custom serialization format (CBOR, in our case) and only doing JSON <-> CBOR at the edges (input/output). Saved us ~30% on token costs.

Here’s a rough idea of how we wired it up:

import cbor2

def serialize_payload(payload):
    return cbor2.dumps(payload)

def deserialize_payload(payload):
    return cbor2.loads(payload)

# Example of swapping JSON for CBOR in Instructor pipeline
serialized = serialize_payload(my_schema)
deserialized = deserialize_payload(serialized)

Curious—how tight is ULMEN's integration with Instructor? Like, can it hook into retry handling directly, or are you wrapping calls outside of Instructor?

[MukundaKatta]

The 44% number aligns with what we measured on production extraction pipelines. JSON braces, quotes, and field name repetition dominate when you have many small typed fields per call.

Two practical wins without breaking the public API: drop the schema-in-prompt and rely on the model's tool-calling layer for shape constraint (works on any tools-capable model and saves the schema bytes), and for retry feedback, send only the diff against the failed value not the full schema again.

A more efficient wire format like TOON would be the long-term fix but it requires the model to be trained on it.

[makroumi]

@rehan243 The CBOR approach is a great optimization for wire size, but let me clarify the architectural boundary here between egress bytes and LLM tokens.

On CBOR vs ULMEN
CBOR is a binary format. It heavily reduces egress/storage (bytes over the wire), but LLMs cannot natively read raw binary CBOR in the context window. If you are passing it to the LLM, you are likely base64 encoding it or mapping it, which alters the token math.

ULMEN solves this by splitting the architecture into two distinct surfaces using the same underlying engine:

1. encode_binary() / encode_binary_zlib(): The binary surface for wire egress and database storage (outperforms JSON+zlib).
2. encode_ulmen_llm(): The text surface for the LLM context window. It uses a pipe-delimited, self-describing tabular format. This is what drops the token count by 44%.

On Instructor Integration
To answer your question honestly: ULMEN does not currently hook into Instructor's internals. That is exactly why I opened this RFC.

Wrapping calls outside of Instructor only saves tokens on the initial payload. To get the compounding savings you mentioned, a serializer protocol needs to hook directly into Instructor's Retrying logic. When a validation fails, the retry prompt sent back to the LLM must inject the error and schema using the ULMEN text format, not standard JSON.

That requires a defined hook in Instructor's core, which is the architectural discussion I'm hoping to start here.

(Also noting: we are moving ULMEN to Apache 2.0 this week, so the BSL licensing friction for enterprise pipelines will be removed).

[makroumi]

@MukundaKatta Spot on regarding the braces, quotes, and field repetition. The 44% bloat is entirely structural syntax.

Let me address the architectural trade-offs of the two practical wins you suggested, and clarify the model-training constraint.

1. Relying on the model's native tool-calling layer
This is a standard optimization, but it introduces a hard architectural lock-in: OpenAI's native tool calling and structured outputs (response_format={"type": "json_schema"}) mandate JSON at the API level. If you rely on the provider's backend to enforce the shape, you are forced to pay the JSON token tax on every call.

ULMEN takes the opposite trade-off: you bypass native JSON tool-calling, use standard text completions, and instruct the model to return ULMEN's pipe-delimited text format. You lose OpenAI's backend schema enforcement, but you claw back the 44% token budget and rely on ULMEN's client-side Semantic Firewall to validate the state transition before proceeding.

2. Diff-only retry feedback
Completely agree. Sending the delta instead of the full schema on a retry is a highly efficient pattern and orthogonal to the serialization format. Combining diff-only retries with a denser wire format yields the absolute minimum token overhead.

3. "Requires the model to be trained on it"
This is true for binary formats or highly abstracted ASTs, but not for ULMEN.

ULMEN does not require fine-tuning or model training. The LLM surface is a standard UTF-8 text format designed like a typed CSV (e.g., res|tc1|t1|3|web_search|Paris|done|142). Language models natively excel at parsing tabular/pipe-delimited data. The engine provides a generate_system_prompt() function that dynamically generates the rules and schema constraints for the LLM zero-shot. We benchmarked this against GPT-4o (o200k_base) with 100% semantic compliance on valid traces.

Different architectures for different scales, but at 1M+ calls/month, shifting validation client-side to escape the API-level JSON tax becomes a very real economic lever.