Skip to content

feat(embeddings): optional client-side char cap for OpenAI-compatible backends - #3091

Open
ai-ag2026 wants to merge 1 commit into
vectorize-io:mainfrom
ai-ag2026:pr/embed-truncate-cap
Open

feat(embeddings): optional client-side char cap for OpenAI-compatible backends#3091
ai-ag2026 wants to merge 1 commit into
vectorize-io:mainfrom
ai-ag2026:pr/embed-truncate-cap

Conversation

@ai-ag2026

Copy link
Copy Markdown
Contributor

Summary

OpenAI-compatible local embedding backends (llama.cpp /v1/embeddings
and friends) hard-reject inputs beyond the model context with an HTTP 4xx
instead of truncating server-side the way the SentenceTransformers provider
does. One oversized memory then fails the whole retain/recall batch.

This adds an optional client-side per-input character cap,
HINDSIGHT_API_EMBEDDINGS_OPENAI_TRUNCATE_CHARS. Default 0 = disabled —
current behaviour is unchanged unless the operator opts in. When set,
oversized inputs are cut to the cap before the request, with a warning
naming the affected count and the env var.

Why

We run Hindsight against a llama.cpp embedder (bge-m3, 8192-token context)
in production. Before the cap, a single long memory (pasted log, big
document chunk) poisoned entire retain batches with 400s; with
TRUNCATE_CHARS=3500 the batch survives and only the tail of the oversized
input is lost — for embedding purposes a far better trade than losing the
whole operation. Running this in production since 2026-07-16.

Changes

Wired exactly like the existing EMBEDDINGS_OPENAI_BATCH_SIZE knob:
config constant + HindsightConfig field + from_env parsing,
OpenAIEmbeddings constructor parameter, all construction sites in
create_embeddings_from_env() (incl. the CodexOAuthEmbeddings
subclass pass-through), .env.example.

Validation

tests/test_embeddings_openai_truncate_chars.py (new, in the pattern of
test_embeddings_openai_batch_size.py): default-is-disabled, env parsing,
and two behaviour tests against a fake client — oversized inputs are cut,
shorter ones untouched; 0 leaves everything untouched. Verified both
directions: the truncation test fails without the encode-side logic and
passes with it (15 passed together with the batch-size file).

… backends

OpenAI-compatible local backends (e.g. llama.cpp /v1/embeddings) hard-reject
inputs beyond the model context instead of truncating server-side the way
the SentenceTransformers provider does — one oversized memory then fails the
whole retain/recall batch with an HTTP 4xx.

Add HINDSIGHT_API_EMBEDDINGS_OPENAI_TRUNCATE_CHARS (default 0 = disabled,
current behaviour unchanged), wired through HindsightConfig exactly like
EMBEDDINGS_OPENAI_BATCH_SIZE: config constant + field + from_env parsing,
OpenAIEmbeddings constructor parameter, all construction sites incl. the
CodexOAuth subclass, and .env.example. When set, oversized inputs are cut
to the cap client-side with a warning naming the count and the env var.

Tests: config default + env parsing, and two behaviour tests against a fake
client (oversized input is cut, shorter inputs untouched; cap=0 leaves
everything untouched). The truncation test fails without the encode-side
logic and passes with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant