feat(reflect,mental-models): surface structured output in the control plane - #3113
Merged
Merged
Conversation
Reflect's response_schema -> structured_output was already implemented and tested in the engine but never exposed in the UI. Surface it in the reflect (think) view, and extend the same structured-output extraction to mental models via a per-model response_schema stored in the trigger config. - engine: refresh_mental_model reads trigger.response_schema, forwards it to the internal reflect call, and persists the parsed structured_output onto the stored reflect_response payload; fix stale 'not yet supported' docstrings - api: add response_schema to MentalModelTrigger - control-plane: reflect route + api.ts forward response_schema; think-view gets a JSON-schema input and renders structured_output; create/update mental model dialogs get a schema editor; detail modal renders structured_output - tests: mental model structured-output plumbing (schema forwarded + persisted) - regenerate OpenAPI spec + client SDKs; add i18n keys for all locales
…config tab Adds a read-only JSON card for the mental model's trigger.response_schema in the detail modal's Configuration tab (mirrors the tag_groups card), plus the regenerated go openapi.yaml.
…intl
The response_schema placeholder was a t() message whose value is literal JSON.
next-intl parses messages as ICU, so the '{' in the example was read as an
argument placeholder, the parse failed, and the field rendered the raw message
key instead of the example. Inline the JSON example directly on the placeholder
prop (i18n:check skips JSON-shaped placeholders) and drop the now-unused
*Placeholder message keys. Caught by running the control plane.
…ema builder Validation (both reflect and mental models): a schema that is valid JSON but not a usable object-with-properties silently produced empty structured_output or blew up inside the LLM extraction call later. Now: - engine: validate_response_schema() enforces the usable-shape contract (object schema, non-empty properties, well-formed required); wired as Pydantic field_validators on ReflectRequest.response_schema and MentalModelTrigger.response_schema (invalid -> HTTP 422). - control-plane: the reflect and mental-model forms validate the schema shape on submit (not just JSON.parse) and surface the specific error. No-code schema builder: a 'Build schema' button on both the reflect view and the mental-model dialogs opens a dialog with Visual and Code modes. Visual mode edits a flat field list (name, type, array item-type, description, required); Code mode edits raw JSON. The two stay in sync and Apply is gated on a usable schema. Shared frontend lib (response-schema.ts) mirrors the backend contract. tests: test_response_schema_validation.py (16 cases: validator + model integration).
…status Removes the inline response_schema JSON textarea from the reflect view and the mental-model dialogs. Editing now happens exclusively in the schema builder; the page shows only whether a schema is set (field count + names, with Edit/Remove) or a Build schema button when none. Extracts the shared ResponseSchemaField component used identically by reflect and both mental-model dialogs.
…reflect's answer In delta mode reflect only sees facts created since the last refresh, so its answer (and any structured_output it derived) reflects just the delta — while the stored content is the delta-merged document. Persisting the reflect-derived value made structured_output inconsistent with the markdown. Now the mental-model refresh no longer passes response_schema to reflect; instead it extracts structured_output from the FINAL stored content (correct for both full and delta), and carries the previous value forward untouched when a delta refresh preserves content (no new facts). Adds a delta test asserting extraction runs against the merged document, not reflect's partial answer.
…o Visual
An empty schema serialises to properties:{}, which schemaToFields mapped to an
empty array — and the Code->Visual guard treated 'empty' the same as 'not
representable', blocking the switch. schemaToFields now returns [] (representable)
for a missing/empty properties map and null only for genuinely unrepresentable
schemas; the switch seeds a blank field when empty.
…builder Adds a Structured Output section to the reflect docs: how response_schema returns both text and a structured_output projection of the same answer, the schema rules, mental-model structured output (extracted from the final/merged document), and the no-code Build schema editor. Regenerates the docs-skill mirror.
…rays The visual editor was flat — object/array fields had no way to define their inner shape. Reworks the field model into a recursive tree (each field has a node; an object node nests fields, an array node nests an item node) so you can build nested objects and arrays-of-objects entirely in the visual editor. Code<->Visual round-trips losslessly; schemas using features the editor can't represent (enum, oneOf, $ref, tuple items, …) stay in code mode rather than being silently flattened.
…resh loudly
Two problems surfaced by nested schemas on Gemini:
1. _generate_structured_output mapped object/array properties to bare dict/list,
which serialize with additionalProperties — rejected by the Gemini API. So any
schema with a nested object/array silently failed extraction. Now it builds a
proper recursive Pydantic model (nested objects -> nested models, arrays ->
typed lists), matching how retain's structured output already works on Gemini.
2. On extraction failure the mental-model refresh silently persisted content with
no structured_output, clobbering the previously-stored value. Now, when a
response_schema is configured and extraction yields nothing, the refresh raises
MentalModelRefreshError — prior content and structured_output are preserved and
the refresh can be retried.
Verified live on Gemini: a nested {location:object, people:array} schema now
extracts (structured_output present) instead of failing on additionalProperties.
Adds a fail-loud regression test.
text-destructive resolves to a dark red (#C0183A) in dark mode, which is low-contrast on the dark dialog background. Use the codebase's standard readable pattern (text-red-600 dark:text-red-400) for the builder's validation error and the invalid-schema notice.
nicoloboschi
force-pushed
the
feat/structured-output-reflect-mental-model
branch
from
August 3, 2026 13:06
d1093dd to
b351d19
Compare
Adding response_schema to MentalModelTrigger regenerated the Rust MentalModelTriggerInput struct with a new field; the hand-written CLI struct literals must initialize it (E0063). Sets response_schema: None in the three construction sites (create/update mental model, knowledge-base pin).
…ext-empty claim; test schema lib - api/mental-models: document the trigger.response_schema flag + a Structured Output section (extraction from final content, fail-loud, validation). - api/reflect: correct the stale claim that text is empty with response_schema — reflect returns both text and structured_output. - control-plane: vitest unit tests for the response-schema lib (validation + recursive fields<->schema round-trip). - regenerate docs-skill mirror.
nicoloboschi
force-pushed
the
feat/structured-output-reflect-mental-model
branch
from
August 3, 2026 13:21
38d4dc1 to
3d6652e
Compare
…e_schema The bank template schema embeds MentalModelTrigger; adding response_schema to the trigger changed the generated schema. Regenerated so verify-generated-files passes.
Resolves conflicts with #3119 (mental-model dry-run refresh + keep_trace), which refactored refresh_mental_model into _execute_mental_model_refresh: - MentalModelTrigger/CLI/api.ts/UI: keep both response_schema and keep_trace fields. - memory_engine: re-apply structured-output extraction + fail-loud into the new refresh_mental_model persist path (dry-run/executor left untouched). - locales: deep-merge both key sets (+ the new mentalModelDiagnostics namespace). - regenerate openapi/clients/bank-template-schema/docs-skill for both new fields.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Reflect's structured output (
response_schema→structured_output) was fully implemented and tested in the engine but never surfaced in the control plane UI. This PR exposes it in the reflect (think) view, and extends the same structured-output extraction to mental models via a per-modelresponse_schemastored in the trigger config.The engine docstrings still claimed it was "not yet supported" — that was stale and is fixed here.
Changes
Engine / API
refresh_mental_modelnow readstrigger.response_schema, forwards it to the internal reflect call, and persists the parsedstructured_outputonto the storedreflect_responsepayload (alongside the markdown content).response_schematoMentalModelTrigger(lives in the existing trigger JSON blob — no migration).memory_engine.py.Control plane (the missing UI)
api/reflect/route.ts+lib/api.tsforwardresponse_schema; the think view gets a JSON-schema input (with invalid-JSON feedback) and rendersstructured_outputin a card.structured_output(content tab) and the configured schema (configuration tab).Generated / i18n / tests
test_mental_model_structured_output.py: verifies the trigger schema is forwarded to reflect and the parsed output is persisted (and that no schema ⇒ no structured output). Existing reflect structured-output tests unchanged.Testing
tests/test_mental_model_structured_output.py— 2 passing (deterministic,reflect_asyncmocked)tests/test_reflect_agent.py::TestReflectStructuredOutputstill green./scripts/hooks/lint.shpasses; control-planetscclean on changed filesNotes
response_schemais intentionally not added there (no parity regression — neither wrapper exposed these).