Document or mitigate prompt cache invalidation from dynamic tool changes

[vercel-gh-bot-2]

Summary

eve can register a newly available model-facing tool after a conversation has started. The primary case is a tool that is absent from one model request and present in the next, while every previously advertised tool definition remains unchanged.

connection_search follows this pattern. The first model request advertises connection_search. After the search returns matches, the next step registers qualified tools such as tfl__Journey_Meta and includes them in the next model request.

Anthropic builds cached prefixes in this order: tools, system, then messages. Adding a tool changes the prefix before the system and message blocks. The previous system and message cache entries therefore cannot match the expanded request. An unchanged leading tools prefix may still be reusable when an earlier tool breakpoint is within Anthropic's lookback window, so adding a tool should not be described as necessarily invalidating every cached tool block.

This differs from changing an existing dynamic tool's name, description, or parameters. Anthropic documents that modification as invalidating the tools cache and every later cache level. Recreating an identical definition does not change the serialized prefix.

Reproduction

Use the existing OpenAPI connection flow with a direct Anthropic model and a prompt long enough to meet the provider's cache minimum:

1. Start an agent with the tfl connection from e2e/fixtures/agent-openapi-swagger.
2. Ask it to call connection_search for the journey modes operation, then call tfl__Journey_Meta.
3. Capture the advertised tool names and cache usage for each step.completed event.
4. Compare the model requests around the search result.

The relevant tool membership changes from:

step 1: [...stable tools, connection_search]
step 2: [...stable tools, connection_search, tfl__Journey_Meta]

The existing connection eval already enforces this call order. The missing coverage is a combined live-provider run that records cacheReadTokens and cacheWriteTokens across the registration boundary.

A custom step.started dynamic resolver that adds a new tool after an earlier tool result exercises the same condition. The important variable is membership: the second request adds a definition that the first request did not advertise. Changing an existing definition is a separate reproduction with a stronger invalidation boundary.

Actual behavior

connection_search re-derives its tool set at every step.started event. It reads prior search results, keeps connection_search, and adds each discovered qualified tool. tool-loop.ts rebuilds and merges that dynamic set before every model call.

On the direct Anthropic path, eve places cache breakpoints on the final tool, final system message, and conversation history. Once a new tool appears before the system and messages, the cumulative system and message prefixes differ from the preceding request. Those prior larger entries cannot be exact matches. Depending on tool order, the number of added tools, and the provider's lookback window, an unchanged earlier tool prefix may still produce a shorter cache read.

The same boundary applies when any dynamic resolver adds or removes a tool. If a resolver instead changes an already advertised tool's name, description, or schema, Anthropic's documented invalidation table says the tools, system, and messages caches are invalidated.

Expected behavior

eve should make runtime tool-registration costs visible or avoid them where practical. Possible outcomes include:

• Document that adding or removing a tool changes all later system and message prefixes, even when existing definitions stay stable
• Document the stronger invalidation caused by changing an existing tool definition
• Add telemetry or a warning when the advertised tool signature changes during a cached session
• Offer a cache-preserving discovery pattern, such as a stable dispatcher or provider-supported deferred tool loading
• Add an eval that combines runtime tool discovery with prompt-cache usage accounting

Evidence

• Anthropic prompt caching defines the prefix hierarchy as tools, system, then messages, requires exact prefix matches, documents the 20-block lookback, and distinguishes tool-definition modification as full cache invalidation.
• connection-search-dynamic.ts extracts discovered tools from prior connection_search results and re-derives them on every step.
• connection-search-dynamic.ts returns connection_search plus the qualified tools discovered so far.
• tool-loop.ts rebuilds model tools, merges dynamic tools, and applies the final-tool cache breakpoint before each model call.
• prompt-cache.ts places direct Anthropic breakpoints on the final tool and final system message.
• step-hooks.ts applies conversation cache control on the direct Anthropic path.
• tfl-swagger.eval.ts verifies that connection_search runs before the newly available tfl__Journey_Meta tool.
• input-cache-rate.eval.ts shows how eve records and evaluates cache read and write usage.

Validation status

The source paths and provider documentation establish the prefix change and the built-in connection_search registration behavior. This issue does not yet include a combined live Anthropic run across that boundary, so it should not claim a measured cache-hit or token-cost delta.