fix(flaggers): stop classifier structured-output truncation at max tokens

[geclos]

Summary

Fixes the escalating Structured-output JSON parse failure (flagger) signal in the Flaggers project (signal, ID gyloub76vor6ytuq12gkj77c).

Flagger classify traces on Bedrock Haiku were repeatedly ending with finishReason: length, exactly 2048 output tokens, and truncated assistant JSON like {"matched":true,"messageIndex":"1" — missing required feedback and a closing brace. Those malformed classify traces were then picked up by the deterministic output-schema-validation flagger, generating the signal.

Example traces: d16f58bc63c9ae4de462b0adeb8484d8, ca2bacf072a8395a18de7cfa3b9be507, 00e2d7778d81b2a663f7c7dd2f11e982.

Root cause

The classifier structured-output messageIndex was an open-ended field (first a JSON number, later a ^\d+$ string). Neither form is length-bounded in the tool schema Bedrock/Anthropic receives, so at temperature: 0 the model could get stuck emitting digits for messageIndex and never close the object — it exhausted the output budget (finishReason: length) and produced truncated, unparseable JSON. Prior work (#3369, #3377) changed the field's type but not its open-ended nature, so the same truncation kept recurring.

Fix

Keep messageIndex in the classify result, but make it structurally impossible to run away:

• Bound messageIndex to an enum. buildProviderFlaggerOutputSchema is rebuilt per classify call so messageIndex is a z.enum of the trace's real transcript indices as string literals (capped at 200 to stay within Bedrock's grammar-size limits; omitted when the trace has no messages). The field becomes a choice among a small finite set, which the constrained/tool decoder cannot run away on.
• Parsing stays lenient. A model that ignores the enum and emits any numeric string is still parsed; out-of-range values are dropped in parseFlaggerOutput rather than failing the whole classification.
• maxTokens lowered 2048 → 512 as defense-in-depth — the payload is a boolean plus one or two short sentences.
• Output contract tightened to require feedback under 300 characters and to instruct the model to pick one of the offered indices.

Existing no-output / schema-mismatch handling still maps any residual failure to matched=false.

Testing

• pnpm --filter @domain/flaggers test (240 pass, 2 skipped)
• pnpm --filter @domain/flaggers typecheck
• Added unit tests asserting the generation schema enum-bounds messageIndex to the trace's indices, omits it for empty traces, and is the schema passed to the classify call
• Added regression coverage for length-truncated output missing feedback (trace d16f58bc pattern)
• Updated the malformed-JSON regression harness to replay with the per-row enum schema and the new contract wording

Follow-up

Do not mute or resolve the signal until after deploy and a human verifies occurrence drop-off. A separate improvement worth considering: label each transcript line with its index in the classify prompt so the enum values map to something the model explicitly sees, improving anchor accuracy.

  

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
design-system	Ready	Preview, Comment	Jul 13, 2026 9:25am

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[claude]

PR #3910 — Diff Explanation

An interactive HTML walkthrough of this PR has been published:

Artifact (Cloudflare temporary): https://artifact-pr-3910-diff-review.modern-albertonykus.workers.dev

Claim within 60 minutes (permanent): https://dash.cloudflare.com/claim-preview?claimToken=2gv3j_qMiSvOXEoswWAH25z2xLu5T-srlnxvkuTIn6Y

The artifact covers:

1. Background — how the flagger classifier and structured generation work
2. Intuition — the root cause (open-ended digit grammar → token-cap runaway) and how bounding messageIndex to a finite enum of the trace's real indices fixes it
3. Code walkthrough — all 5 files, diff blocks, the two-schema design (strict generation / lenient parsing), and the new tests
4. Quiz — 5 interactive multiple-choice questions to verify understanding