feat(embeddings): optional client-side char cap for OpenAI-compatible backends - #3091
Open
ai-ag2026 wants to merge 1 commit into
Open
feat(embeddings): optional client-side char cap for OpenAI-compatible backends#3091ai-ag2026 wants to merge 1 commit into
ai-ag2026 wants to merge 1 commit into
Conversation
… backends OpenAI-compatible local backends (e.g. llama.cpp /v1/embeddings) hard-reject inputs beyond the model context instead of truncating server-side the way the SentenceTransformers provider does — one oversized memory then fails the whole retain/recall batch with an HTTP 4xx. Add HINDSIGHT_API_EMBEDDINGS_OPENAI_TRUNCATE_CHARS (default 0 = disabled, current behaviour unchanged), wired through HindsightConfig exactly like EMBEDDINGS_OPENAI_BATCH_SIZE: config constant + field + from_env parsing, OpenAIEmbeddings constructor parameter, all construction sites incl. the CodexOAuth subclass, and .env.example. When set, oversized inputs are cut to the cap client-side with a warning naming the count and the env var. Tests: config default + env parsing, and two behaviour tests against a fake client (oversized input is cut, shorter inputs untouched; cap=0 leaves everything untouched). The truncation test fails without the encode-side logic and passes with it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OpenAI-compatible local embedding backends (llama.cpp
/v1/embeddingsand friends) hard-reject inputs beyond the model context with an HTTP 4xx
instead of truncating server-side the way the SentenceTransformers provider
does. One oversized memory then fails the whole retain/recall batch.
This adds an optional client-side per-input character cap,
HINDSIGHT_API_EMBEDDINGS_OPENAI_TRUNCATE_CHARS. Default0= disabled —current behaviour is unchanged unless the operator opts in. When set,
oversized inputs are cut to the cap before the request, with a warning
naming the affected count and the env var.
Why
We run Hindsight against a llama.cpp embedder (bge-m3, 8192-token context)
in production. Before the cap, a single long memory (pasted log, big
document chunk) poisoned entire retain batches with 400s; with
TRUNCATE_CHARS=3500the batch survives and only the tail of the oversizedinput is lost — for embedding purposes a far better trade than losing the
whole operation. Running this in production since 2026-07-16.
Changes
Wired exactly like the existing
EMBEDDINGS_OPENAI_BATCH_SIZEknob:config constant +
HindsightConfigfield +from_envparsing,OpenAIEmbeddingsconstructor parameter, all construction sites increate_embeddings_from_env()(incl. theCodexOAuthEmbeddingssubclass pass-through),
.env.example.Validation
tests/test_embeddings_openai_truncate_chars.py(new, in the pattern oftest_embeddings_openai_batch_size.py): default-is-disabled, env parsing,and two behaviour tests against a fake client — oversized inputs are cut,
shorter ones untouched;
0leaves everything untouched. Verified bothdirections: the truncation test fails without the encode-side logic and
passes with it (15 passed together with the batch-size file).