feat(mental-models): dry-run refresh and keep_trace for troubleshooting - #3119
Merged
Merged
Conversation
nicoloboschi
force-pushed
the
feat/mental-model-refresh-troubleshooting
branch
from
August 3, 2026 13:22
6cdfe84 to
3e2b9e1
Compare
When a refresh produced an unexpected document, nothing said why. The mode
decision, resolved scope, snapshot window, retrieved-versus-used fact counts
and dropped delta operations only ever reached a log line — and cron- or
consolidation-driven refreshes run with nobody watching.
Two ways to see that reasoning, from opposite directions.
POST /mental-models/{id}/dry-run-refresh runs the production refresh
pipeline and reports what it would do, skipping exactly two writes: the
content (with its structured document and history entry) and the watermark
that moves last_refreshed_at. It takes no parameters, on purpose — a dry run
you can configure stops predicting the refresh it exists to predict. Because
nothing is persisted, a delta dry run reads exactly the window the next real
refresh will.
trigger.keep_trace records the same reasoning on every refresh of a model,
scheduled ones included, under reflect_response.trace. It is written even
when a refresh fails, which is when it matters most. The trace is shaped
like reflect's — the calls the agent made plus the refresh decision — and
holds nothing derivable from elsewhere: evidence stays in based_on, and the
resolved scope and window are reported by the dry run. Each tool call
records the window bound it was given, named `updated_at` for what the
predicate actually filters; null means the tool applies no time bound at
all, which is what explains results older than the window would suggest.
refresh_mental_model is split into a shared _execute_mental_model_refresh
that computes a result and writes nothing, plus a thin persistence step, so
the preview and the real refresh run the same body. Existing refresh
behaviour is unchanged.
In the control plane the dry run is an action on the mental model, and its
result opens in a dialog built from the History tab's own diff components.
History shows each version's own trace: the history snapshot now carries
`trace` alongside `based_on` so it survives being superseded.
Surfaced but deliberately not fixed here: when delta operations fail, the
fallback writes a candidate built from a delta-scoped recall over the whole
document, dropping content grounded in older memories (#3112).
nicoloboschi
force-pushed
the
feat/mental-model-refresh-troubleshooting
branch
from
August 3, 2026 13:38
3e2b9e1 to
f747f6f
Compare
nicoloboschi
added a commit
that referenced
this pull request
Aug 3, 2026
Resolves conflicts with #3119 (mental-model dry-run refresh + keep_trace), which refactored refresh_mental_model into _execute_mental_model_refresh: - MentalModelTrigger/CLI/api.ts/UI: keep both response_schema and keep_trace fields. - memory_engine: re-apply structured-output extraction + fail-loud into the new refresh_mental_model persist path (dry-run/executor left untouched). - locales: deep-merge both key sets (+ the new mentalModelDiagnostics namespace). - regenerate openapi/clients/bank-template-schema/docs-skill for both new fields.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When a mental model refresh produces a document you didn't expect — nothing changed, the wrong things changed, delta edits didn't apply — the stored content doesn't say why. Every decision the refresh made (full vs delta and the reason, the resolved tag scope, the snapshot window, how much retrieval returned versus how much the agent used, which delta operations were dropped) existed only as a log line, and for cron- and consolidation-driven refreshes nobody is watching the logs.
This adds two ways to see that reasoning, from opposite directions.
Dry run
POST /v1/default/banks/{bank_id}/mental-models/{mental_model_id}/dry-run-refresh— synchronous, mirroring the existing/memories/dry-run-extractprecedent.It runs the real pipeline and reports what a refresh would do. Nothing is persisted: not content, structured document, watermark, nor
last_refreshed_at. Because nothing is persisted, a delta dry run reads exactly the window the next real refresh will, and repeating it reads that same window again.The response answers the questions the document can't:
requested_mode/effective_mode/mode_fallback_reasonscopewindowcreated_after/created_beforebounds and the watermark that would be writtenfacts.retrievedvsfacts.useddelta_operationsdiffoutcome/would_persist/warningsEvery refresh setting is overridable in the body (
mode,source_query,tags,tags_match,tag_groups,fact_types, …) to A/B a candidate configuration without editing the model.keep_trace
A dry run only explains a refresh you run yourself.
trigger.keep_tracerecords the same reasoning on every refresh of that model — including scheduled ones — underreflect_response.trace. It is written even when a refresh fails, which is when it matters most: a failed refresh otherwise leaves nothing behind to inspect.Tool outputs are reduced to result counts so the stored trace stays bounded; raw prompts and responses remain available through LLM request tracing.
Implementation note
refresh_mental_modelwas split into a shared_execute_mental_model_refreshpipeline that computes a result and writes nothing, plus a thin persistence step. The dry run and the real refresh run the same body — a preview that reasoned differently from the refresh it predicts would be worse than no preview. Behaviour of the existing refresh path is unchanged.Surfaces
dry_run_refresh_mental_model, trace recordingtrigger.keep_tracehindsight mental-model dry-run-refresh <bank> <id> [--mode] [--tags-match] [--source-query]Tests
19 new tests in
test_mental_model_dry_run_refresh.pycovering: the dry run persists nothing (including that it does not advancelast_refreshed_at), each mode-fallback reason, override plumbing reaching reflect without editing the model, retrieved-vs-used counts and their warnings, the empty-candidate outcome, scope resolution, trace presence/absence, trace on a failed refresh, and the HTTP surface (200, overrides, 404).Full mental-model suite: 96 passed.
lint.sh,cli-coverage-check, control-plane typecheck andi18n:checkall clean.Found along the way, deliberately not fixed here
#3112 — when a delta refresh's operations fail, the code logs "falling back to full synthesis" but the candidate was built from a delta-scoped recall, so writing it whole drops everything grounded in older memories. The dry run now warns about this condition; the behaviour itself is unchanged and tracked separately.