Skip to content

Commit 95f9918

Browse files
zacclaude
zac
and
claude
authored
Macro-generate capability registrations; harden runtime timeouts and network egress; add platform + eval CI (#5)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent 258ec4e commit 95f9918

61 files changed

Lines changed: 11469 additions & 2792 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/codemode-evals.yml‎

Lines changed: 119 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,15 @@ on:
1515
description: "Delay before each model request"
1616
required: false
1717
default: "1000"
18-
schedule:
19-
- cron: "17 10 * * *"
2018

2119
concurrency:
2220
group: codemode-evals-${{ github.ref }}
2321
cancel-in-progress: false
2422

2523
jobs:
24+
# Deterministic evals + platform builds are continuous: they run on every PR
25+
# and on pushes to main (merges). No schedule — a daily cron re-running
26+
# unchanged main added no signal for these offline checks.
2627
deterministic:
2728
name: Deterministic evals
2829
runs-on: macos-latest
@@ -31,7 +32,19 @@ jobs:
3132
uses: actions/checkout@v4
3233

3334
- name: Toolchain
34-
run: swift --version
35+
run: |
36+
swift --version
37+
xcodebuild -version
38+
39+
- name: Cache SwiftPM build
40+
uses: actions/cache@v4
41+
with:
42+
path: |
43+
.build
44+
Tools/CodeModeEval/.build
45+
key: spm-${{ runner.os }}-${{ hashFiles('Package.swift', 'Tools/CodeModeEval/Package.swift') }}
46+
restore-keys: |
47+
spm-${{ runner.os }}-
3548
3649
- name: Test package
3750
run: swift test
@@ -42,42 +55,126 @@ jobs:
4255
- name: Run deterministic evals
4356
run: swift run --package-path Tools/CodeModeEval codemode-eval run
4457

45-
llm-plan:
46-
name: LLM eval planning
58+
platform-build:
59+
name: Build (${{ matrix.platform }})
60+
runs-on: macos-latest
61+
strategy:
62+
fail-fast: false
63+
matrix:
64+
include:
65+
- platform: iOS
66+
destination: generic/platform=iOS
67+
- platform: visionOS
68+
destination: generic/platform=visionOS
69+
steps:
70+
- name: Checkout
71+
uses: actions/checkout@v4
72+
73+
- name: Toolchain
74+
run: |
75+
swift --version
76+
xcodebuild -version
77+
78+
# The library-only CodeMode product must compile for every declared
79+
# platform. `swift test` on macOS never builds the UIKit presenters or the
80+
# CCodeModeJSC shim against the iOS/visionOS SDKs, so exercise them here.
81+
# CODE_SIGNING_ALLOWED=NO keeps the library build from requiring a profile.
82+
- name: List package schemes
83+
run: xcodebuild -list -json | tee schemes.json
84+
85+
# -skipMacroValidation: the package builds its own CodeModeMacros compiler
86+
# plugin, and xcodebuild refuses unvalidated macros in noninteractive runs.
87+
- name: Build CodeMode for ${{ matrix.platform }}
88+
run: |
89+
set -o pipefail
90+
xcodebuild build \
91+
-scheme CodeMode \
92+
-destination '${{ matrix.destination }}' \
93+
-skipMacroValidation \
94+
CODE_SIGNING_ALLOWED=NO
95+
96+
# Live LLM regression gate. Manual only (workflow_dispatch) — it spends real
97+
# provider calls, so it is never automatic.
98+
#
99+
# Requirements to actually execute the live run (all private, absent from this
100+
# public repo by design):
101+
# 1. Build the eval CLI with the private overlay so the real Wavelike-backed
102+
# `codemode-eval llm` command is compiled in place of the LLMUnavailable
103+
# stub (un-exclude LLM.swift + add the private Wavelike/CallableFunction
104+
# package dependencies).
105+
# 2. A WAVELIKE_API_KEY repository secret.
106+
# Until both are present the job runs the budget preview and then skips the
107+
# live+compare steps with a warning, instead of pretending or failing noisily.
108+
llm-live:
109+
name: LLM eval (live regression gate)
47110
runs-on: macos-latest
48111
needs: deterministic
49-
if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule'
112+
if: github.event_name == 'workflow_dispatch'
50113
env:
51114
INPUT_REPEAT_COUNT: ${{ github.event.inputs.repeat_count }}
52115
INPUT_REQUEST_DELAY_MS: ${{ github.event.inputs.request_delay_ms }}
116+
WAVELIKE_API_KEY: ${{ secrets.WAVELIKE_API_KEY }}
117+
WAVELIKE_MODEL_ID: ${{ vars.WAVELIKE_MODEL_ID }}
53118
steps:
54119
- name: Checkout
55120
uses: actions/checkout@v4
56121

57122
- name: Toolchain
58123
run: swift --version
59124

125+
- name: Determine whether live execution is configured
126+
id: gate
127+
run: |
128+
if [ -z "${WAVELIKE_API_KEY}" ]; then
129+
echo "::warning::WAVELIKE_API_KEY not set — live LLM evals need the private Wavelike runner and credentials. Running budget preview only."
130+
echo "live=false" >> "$GITHUB_OUTPUT"
131+
else
132+
echo "live=true" >> "$GITHUB_OUTPUT"
133+
fi
134+
60135
- name: Build eval CLI
61136
run: swift build --package-path Tools/CodeModeEval
62137

63-
- name: Preview LLM eval suites
138+
- name: Preview LLM eval budget
64139
run: |
65140
repeat_count="${INPUT_REPEAT_COUNT:-5}"
66141
request_delay_ms="${INPUT_REQUEST_DELAY_MS:-1000}"
142+
for suite in core failures catalog; do
143+
swift run --package-path Tools/CodeModeEval codemode-eval plan \
144+
--suite "$suite" \
145+
--repeat "$repeat_count" \
146+
--request-delay-ms "$request_delay_ms"
147+
done
67148
68-
swift run --package-path Tools/CodeModeEval codemode-eval plan \
69-
--suite core \
70-
--repeat "$repeat_count" \
71-
--request-delay-ms "$request_delay_ms"
72-
73-
swift run --package-path Tools/CodeModeEval codemode-eval plan \
74-
--suite failures \
75-
--repeat "$repeat_count" \
76-
--request-delay-ms "$request_delay_ms"
77-
78-
swift run --package-path Tools/CodeModeEval codemode-eval plan \
79-
--suite catalog \
80-
--repeat "$repeat_count" \
81-
--request-delay-ms "$request_delay_ms"
149+
# Real regression gate: run the live suites that have committed baselines,
150+
# then `compare` (default policy: no pass-rate or exact-capability
151+
# regression) which fails the job on regression. `catalog` is previewed
152+
# above but not compared until its baseline is generated and committed.
153+
- name: Run live LLM evals and compare against baselines
154+
if: steps.gate.outputs.live == 'true'
155+
run: |
156+
repeat_count="${INPUT_REPEAT_COUNT:-5}"
157+
request_delay_ms="${INPUT_REQUEST_DELAY_MS:-1000}"
158+
reports="Tools/CodeModeEval/.build/reports"
159+
mkdir -p "$reports"
160+
for suite in core failures; do
161+
candidate="${reports}/${suite}-r${repeat_count}.json"
162+
swift run --package-path Tools/CodeModeEval codemode-eval llm \
163+
--suite "$suite" \
164+
--repeat "$repeat_count" \
165+
--request-delay-ms "$request_delay_ms" \
166+
--output "$candidate"
167+
swift run --package-path Tools/CodeModeEval codemode-eval compare \
168+
"Tools/CodeModeEval/Baselines/${suite}-r5-summary.json" \
169+
"$candidate" \
170+
--retry-tolerance 0.5 \
171+
--turn-tolerance 0.5
172+
done
82173
83-
echo "Live Wavelike-backed LLM execution is disabled in this workflow while the default eval package avoids private dependencies."
174+
- name: Upload live eval reports
175+
if: steps.gate.outputs.live == 'true'
176+
uses: actions/upload-artifact@v4
177+
with:
178+
name: llm-eval-reports
179+
path: Tools/CodeModeEval/.build/reports/*.json
180+
if-no-files-found: warn

‎EVALS.md‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -61,13 +61,24 @@ swift run --package-path Tools/CodeModeEval codemode-eval summarize \
6161

6262
## CI Policy
6363

64-
The GitHub Actions workflow runs deterministic evals on PRs and pushes. Scheduled and manually dispatched runs build the same public eval CLI and preview `core`, `failures`, and `catalog` LLM suite budgets without making live provider calls or resolving private Wavelike dependencies.
65-
66-
Scheduled/manual planning runs:
67-
68-
- Preview `core`, `failures`, and `catalog` with repeat count 5 by default.
69-
- Honor the manual `repeat_count` and `request_delay_ms` inputs for budget estimates.
70-
- Leave baseline comparison to private live-report generation until a reviewed candidate report exists.
64+
The GitHub Actions workflow runs the deterministic evals and the iOS/visionOS
65+
platform builds continuously — on every pull request and on pushes to `main`
66+
(merges). There is no schedule; a daily cron over unchanged `main` added no
67+
signal for these offline checks.
68+
69+
The live LLM regression gate (`llm-live` job) is **manual only**
70+
(`workflow_dispatch`) because it spends real provider calls:
71+
72+
- It always previews `core`, `failures`, and `catalog` budgets (honoring the
73+
manual `repeat_count` / `request_delay_ms` inputs).
74+
- If a `WAVELIKE_API_KEY` secret is configured **and** the CLI is built with the
75+
private overlay (the real `codemode-eval llm` command rather than the
76+
`LLMUnavailable` stub), it then runs the `core` and `failures` suites live and
77+
`compare`s each against its committed baseline, failing the job on any
78+
pass-rate or exact-capability regression. `catalog` is previewed but not
79+
compared until its baseline is generated and committed.
80+
- Without those, the live+compare steps skip with a warning, so the manual run
81+
still succeeds and only reports the budget preview.
7182

7283
## Updating Baselines
7384

‎PHASE3-HANDOFF.md‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
# Phase 3 handoff: migrate remaining registrations to `@BuiltInCodeMode`
2+
3+
Self-contained work spec for converting the remaining hand-written capability
4+
registrations to the macro-authored tool idiom. Context: Phases 1–2 of
5+
`PLAN-registration-macros.md` are done; EventKit is the finished reference.
6+
This is mechanical work — the invariants below matter more than speed.
7+
8+
## The one invariant
9+
10+
**The advertised capability surface must not change except where this spec
11+
says it may.** `Tests/CodeModeTests/CapabilityMetadataGoldenTests.swift` pins
12+
every registration's full metadata (jsNames, title, summary, tags, example,
13+
permissions, argument lists/types/hints, constraints, resultSummary) against
14+
`Tests/CodeModeTests/capability-metadata-golden.json`.
15+
16+
Workflow per domain:
17+
1. Convert the domain (recipe below). Run `swift test`. The golden test fails.
18+
2. Regenerate: `CODEMODE_REGENERATE_GOLDEN=1 swift test --filter capabilityMetadata`
19+
3. `git diff Tests/CodeModeTests/capability-metadata-golden.json` — **this diff
20+
is the review artifact.** Every hunk must be one of the allowed diffs below;
21+
anything else is a bug in your conversion. Fix the conversion, not the spec.
22+
4. Full `swift test` green → one commit for the domain, golden diff included,
23+
and the commit message lists which allowed-diff categories appear.
24+
25+
Allowed golden diffs:
26+
- `argumentHints` gaining entries for arguments that previously had no hint
27+
(the tool idiom requires a hint per argument — write one consistent with the
28+
bridge's actual behavior, and call it out in the commit message).
29+
- `allowedStringValues` gaining alias spellings **only when the bridge already
30+
accepts them** (cite the bridge line in the commit message).
31+
- Nothing else. Not a reworded title, not a reordered jsNames array, not a
32+
type change, not a constraint that moved keys.
33+
34+
## Reference implementation (read these first)
35+
36+
- `Sources/CodeMode/Bridges/EventKitCodeModeTools.swift` — 9 macro-authored
37+
tools + 5 `CodeModeStringEnum`s. The pattern to replicate.
38+
- `Sources/CodeMode/Bridges/CapabilityRegistrations+EventKit.swift` — what a
39+
registration file looks like after conversion (a thin list).
40+
- `Sources/CodeMode/Bridges/BuiltInCodeModeTool.swift`,
41+
`Sources/CodeMode/Bridges/BuiltInCodeModeMacros.swift`,
42+
`Sources/CodeMode/API/CodeModeStringEnum.swift` — the infrastructure.
43+
- Commits `6f5e7e8` (Phase 1) and `c45fb10` (Phase 2) show the full shape of a
44+
domain conversion including bridge rewiring and tests.
45+
46+
## Recipe per registration
47+
48+
1. Create `Sources/CodeMode/Bridges/<Domain>CodeModeTools.swift`. One
49+
`@BuiltInCodeMode` struct per `CapabilityRegistration` in the old file.
50+
2. Copy **verbatim**: title, summary, tags, example, requiredPermissions,
51+
resultSummary. Do not improve the prose.
52+
3. `path` = the registration's first jsName; `aliases:` = the rest, in order.
53+
4. `Arguments` struct: one `@ToolParam("<exact existing hint>")` property per
54+
declared argument, in the old required-then-optional order. Required args
55+
are non-optional Swift types; optional args are optionals.
56+
5. **Property types must reproduce the old effective type.** If the old
57+
descriptor declared `argumentTypes`, match it. If it didn't, the effective
58+
type came from `CapabilityDescriptor.inferArgumentTypes`
59+
(`CapabilityRegistry.swift`) — look each name up in that table:
60+
`.string`→`String`, `.number`→`Int` or `Double` (pick what the bridge
61+
reads), `.bool`→`Bool`, `.array`→`[String]`/`[JSONValue]` (match bridge),
62+
`.object`→`[String: JSONValue]`, and **names absent from the table were
63+
`.any` — declare those as `JSONValue`**, never a tighter type. The golden
64+
test catches mistakes here; trust it.
65+
6. Constrained string arguments — any argument with a row in
66+
`CapabilityArgumentConstraints.defaults(for:)` (`CapabilityRegistry.swift`):
67+
- Define a `CodeModeStringEnum` whose **raw values are exactly the current
68+
advertised list** (case names = raw values). Add `codeModeAliases` only
69+
for spellings the bridge demonstrably accepts.
70+
- Use it as the property type; the macro derives the constraint from it.
71+
- Rewire the bridge's own parsing of that value to
72+
`EnumType.codeModeValue(matching:)` (see `EventKitBridge.eventSpan`,
73+
`SystemUIBridge.validateCalendarPickerArguments` for the pattern), so the
74+
enum is the single source of truth.
75+
- Delete the row from the `defaults(for:)` table in the same commit.
76+
- Exception: **dotted-path constraints** (`networkFetch`'s
77+
`options.responseEncoding`) stay in the central table — the tool argument
78+
model is flat. Leave them and note it.
79+
7. End every `Arguments` struct with `var raw: [String: JSONValue]` and call
80+
the same bridge method the old handler called, passing `arguments.raw` and
81+
the same `context`. Do not change bridge method signatures beyond the
82+
constrained-value parsing rewiring in step 6.
83+
8. Replace the old file's body with the thin builder-extension list (keep the
84+
function name the builder calls, e.g. `systemUIRegistrations()` — see
85+
`DefaultCapabilityLoader.loadAll()` for the roster).
86+
87+
## Domain order and notes
88+
89+
Work sequentially, one commit per domain, full suite green each time:
90+
91+
1. **SystemUI** (`CapabilityRegistrations+SystemUI.swift`, 12 registrations) —
92+
heaviest constraint user: `photosUIPick`/`contactsUIPick`/`cameraUICapture`/
93+
`cameraUIScanData` rows in the defaults table, and `SystemUIBridge` has
94+
matching `lowercased()` validations to rewire (mediaType, cameraDevice,
95+
flashMode, videoQuality, scan mode, preferredStyle…). Only enum-ify values
96+
that have a defaults-table row today; leave other `lowercased()` checks
97+
alone.
98+
2. **Core** (`+Core.swift`, 11) — includes `networkFetch` (dotted-path
99+
constraint stays in the table) and the filesystem/keychain area. Keychain is
100+
already converted; don't touch `SimpleBuiltInCodeModeProviders.swift`.
101+
3. **PeoplePhotosDocuments** (`+PeoplePhotosDocuments.swift`, 13) —
102+
`photosRead` mediaType row; `PhotosBridge` lowercases mediaType, rewire it.
103+
4. **SystemServices** (`+SystemServices.swift`, 19) — health/home/alarm/
104+
notifications. Do **not** add `.healthKit` to any `requiredPermissions`
105+
(see `noBuiltInRegistrationGatesOnHealthKitPermission` test). Preserve the
106+
existing permission declarations exactly.
107+
5. **CloudPushSpeech** (`+CloudPushSpeech.swift`, 15) — the four CloudKit
108+
capabilities share the `database` row; one enum, four tools.
109+
6. **IntentsModelsActivityMaps** (`+IntentsModelsActivityMaps.swift`, 18) —
110+
`activityEnd` dismissalPolicy and `mapsRouteEstimate`/`mapsOpen`
111+
transportType rows. transportType is consumed in
112+
`SystemAppleServiceClients.swift` (`SystemMapsMapping`) — rewire there.
113+
7. **Commerce** (`+Commerce.swift`, 18) — mostly pass-through to host-supplied
114+
clients (music/passKit/storeKit). Convert metadata + `musicPlaybackControl`
115+
action enum (replicate the existing list; host clients stay authoritative
116+
for semantics). Do not invent constraints for values the table doesn't
117+
constrain today.
118+
119+
## Hard rules
120+
121+
- Do not touch `Sources/CodeMode/Runtime/RuntimeJavaScript.swift` (the JS
122+
function table is a separate Phase-3 item, not this task).
123+
- Do not reword any advertised string. Copy-paste, don't retype.
124+
- Do not change permission ownership (some capabilities deliberately declare
125+
`requiredPermissions: []` and check in the bridge — e.g. calendarWrite).
126+
- Do not migrate `LocationWeather` or `EventKit` (done) and do not modify
127+
`Tools/CodeModeEval` or CI.
128+
- If a registration doesn't fit the recipe (unexpected handler shape, shared
129+
state, anything surprising), **stop and leave that registration on the old
130+
idiom in its file** with a `// PHASE3-SKIP: <reason>` comment rather than
131+
improvising. Mixed files are fine; wrong conversions are not.
132+
- When all domains are done: delete any now-empty rows from `defaults(for:)`,
133+
and update `TODO.md`'s structural-improvements section + the status block in
134+
`PLAN-registration-macros.md`.
135+
136+
## Definition of done (per domain)
137+
138+
- `swift test` fully green (230+ tests).
139+
- Golden diff contains only allowed categories, enumerated in the commit
140+
message.
141+
- The old registration file is a thin list; its metadata lives on tools.
142+
- Constraint rows for the domain are deleted from the central table (except
143+
dotted paths) and the owning bridge parses through the shared enum.

0 commit comments

Comments
 (0)