Skip to content

chore(ci): cut PR wall-clock ~9.5min → ~5-6min — dedupe dogfood, shard it, cache lint's build#3622

Merged
os-zhuang merged 2 commits into
mainfrom
claude/ci-performance-optimization-ms8ll6
Jul 27, 2026
Merged

chore(ci): cut PR wall-clock ~9.5min → ~5-6min — dedupe dogfood, shard it, cache lint's build#3622
os-zhuang merged 2 commits into
mainfrom
claude/ci-performance-optimization-ms8ll6

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Why

Job-level timing of recent PR runs shows where the time actually goes:

Job Wall time Long pole
CI / Test Core 9m21s turbo run test --affected — 8m35s
CI / Dogfood Regression Gate 8m41s dogfood suite — 7m34s
Lint / TypeScript Type Check 6m09s turbo run build — 4m38s (no cache)
CodeQL ~4m analysis (parallel, acceptable)

Two structural problems dominate: the dogfood suite ran twice per core PR (once in its dedicated job, once again inside Test Core's affected set, in parallel on two runners), and lint's typecheck job is the only build-running job with no turbo cache, so it rebuilt the entire workspace from scratch on every run — the identical build takes 51s in ci.yml's cached Build Core job.

What

  1. Exclude @objectstack/dogfood from Test Core (both the PR --affected run and the push run). The dedicated Dogfood job — gated on the same core filter — remains the sole, unconditional runner of the suite. Verified locally: turbo unions inclusive filters and then subtracts ! negations, so --affected --filter=!@objectstack/dogfood is exactly "affected minus dogfood" (140 → 137 tasks, nothing else dropped).

  2. Shard the Dogfood job 2-way (vitest run --shard=N/2, ~60 independent test files, deterministic file-level partition). The shard arg passes through turbo and is part of the turbo task hash, so each shard caches independently (verified: distinct hashes). The objectstack verify CLI step is shard-independent and runs on shard 1 only. Turbo cache keys are shard-scoped so the two matrix jobs don't race on one cache entry.

  3. Add the missing turbo cache step to lint's typecheck job — same key scheme as ci.yml, so the fallback prefix also hits caches saved by main pushes.

  4. Add concurrency cancel-in-progress to lint.yml and codeql.yml (ci.yml already has it). Both trigger on every PR sync; superseded runs were burning runners and delaying the queue.

Expected effect

  • CI workflow: ~9.5min → ~5–6min (Test Core sheds the duplicated 7.5-min suite; Dogfood halves to ~4–5min per shard).
  • Lint workflow: ~6min → ~2–2.5min on warm cache.
  • Meaningful runner-minute savings per PR sync from de-duplication + cancellation.

Verification

  • turbo run test --affected --filter=!@objectstack/dogfood --dry=json with a spec change: dogfood excluded, all other affected tasks intact.
  • vitest run --shard=1/30 in packages/qa/dogfood: 3 files / 13 tests, green — sharding partitions and runs correctly.
  • Full-chain smoke of the exact CI command shape: turbo run test --filter=@objectstack/dogfood -- --shard=1/30 → 61 tasks successful, exit 0.
  • --dry=json task-hash comparison between --shard=1/2 and --shard=2/2: hashes differ, so per-shard turbo caching is safe.
  • All three YAML files parse.

⚠️ Action needed at merge time

If branch protection lists "Dogfood Regression Gate" as a required status check, it must be updated to the two sharded check names (Dogfood Regression Gate (1/2) / Dogfood Regression Gate (2/2)).

🤖 Generated with Claude Code

https://claude.ai/code/session_01ECTCrcCdZpCHw5zFSgcmGt


Generated by Claude Code

…d it, cache lint's build

Three data-driven fixes from job-level timing of recent PR runs:

1. Test Core re-ran the whole ~7.5-minute dogfood suite that the dedicated
   Dogfood job was already running in parallel (both gate on the same `core`
   filter). Exclude @objectstack/dogfood from both the PR --affected run and
   the push run; the Dogfood job remains the sole (and unconditional) runner
   of the suite. Verified locally: turbo unions inclusive filters and then
   subtracts `!` negations, so `--affected --filter=!@objectstack/dogfood`
   is exactly "affected minus dogfood" (140 → 137 tasks, nothing else lost).

2. The Dogfood job itself was the workflow's longest pole (7m34s for the
   ~60-file suite on one 4-vCPU runner). Shard it 2-way with vitest's
   deterministic file-level --shard, passed through turbo (pass-through args
   are part of the turbo task hash, so each shard caches independently —
   verified: distinct hashes per shard). The `objectstack verify` CLI step is
   shard-independent and runs on shard 1 only. NOTE: if branch protection
   requires "Dogfood Regression Gate", the required check must be renamed to
   the two sharded names.

3. lint.yml's "TypeScript Type Check" job runs a full workspace build but was
   the only build-running job with NO turbo cache step — it rebuilt everything
   from scratch every run (4m38s observed, vs 51s for the same build in
   ci.yml's cached Build Core). Add the same .turbo/cache step.

Also: lint.yml and codeql.yml had no concurrency group, so superseded runs of
both (they trigger on every PR sync) kept burning runners and delaying the
queue. Add the same cancel-in-progress policy ci.yml already documents.

Full-chain smoke: `turbo run test --filter=@objectstack/dogfood -- --shard=1/30`
runs green end-to-end (turbo → pnpm → vitest, 61 tasks OK).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECTCrcCdZpCHw5zFSgcmGt
@vercel

vercel Bot commented Jul 27, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Jul 27, 2026 10:05am

Request Review

@github-actions github-actions Bot added documentation Improvements or additions to documentation tooling labels Jul 27, 2026
@os-zhuang
os-zhuang marked this pull request as ready for review July 27, 2026 10:31
@os-zhuang
os-zhuang merged commit 1676de0 into main Jul 27, 2026
16 checks passed
@os-zhuang
os-zhuang deleted the claude/ci-performance-optimization-ms8ll6 branch July 27, 2026 10:32
os-zhuang added a commit that referenced this pull request Jul 27, 2026
…name (#3622 follow-up) (#3644)

INCIDENT: every open PR in the repo is BLOCKED — mergeable, all checks
green, merge button dead.

Root cause: #3622 sharded the dogfood job 2-way, so its checks now
publish as "Dogfood Regression Gate (1/2)" / "(2/2)". Branch protection
still requires the bare context "Dogfood Regression Gate", which no job
emits anymore — the requirement can never be satisfied. #3622's own
comment called for updating branch protection; that step never happened.

Fix keeps the contract in CODE rather than in protection settings: a
tiny aggregator job carrying the stable name, gated on the matrix's
aggregate result. A future shard-count change therefore cannot deadlock
the repo again. `if: always()` + result inspection so a legitimately
skipped matrix (filter says no core paths changed) still satisfies the
gate.

Verified: workflow YAML parses; job graph has dogfood-gate needs:
dogfood with name "Dogfood Regression Gate".

Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd documentation Improvements or additions to documentation size/s tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants