test: add regression guards for api-proxy token tracking in smoke-claude

[Copilot]

The claude-token-usage-analyzer reported null token_usage, estimated_cost, and turns for all recent Smoke Claude runs. Investigation confirmed the api-proxy is already enabled and verify_token_usage job exists — the nulls were from an older lock file — but there was no test coverage preventing a silent regression.

Changes

• scripts/ci/smoke-claude-workflow.test.ts: Added assertions to both source and lock file tests to enforce the critical token tracking invariants:
  • Source must contain verify_token_usage, check-token-usage.js, and Token-usage sanity check
  • Compiled lock must contain "apiProxy":{"enabled":true, verify_token_usage, and the correct check-token-usage.js --artifact-root /tmp/gh-aw-agent --engine claude invocation

If either the api-proxy config or the token verification job is accidentally dropped during a workflow recompile, the CI test suite will now catch it immediately.

[Copilot]

Pull request overview

Adds regression tests protecting Claude token tracking in the smoke workflow.

Changes:

• Verifies token-usage validation exists in source and compiled workflows.
• Ensures the compiled workflow enables the API proxy and invokes the Claude usage checker correctly.

Show a summary per file

File	Description
scripts/ci/smoke-claude-workflow.test.ts	Adds source and lock-file assertions for token tracking invariants.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Medium

[github-actions]

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

[github-actions]

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	98.30%	98.33%	📈 +0.03%
Statements	98.23%	98.26%	📈 +0.03%
Functions	99.07%	99.07%	➡️ +0.00%
Branches	93.85%	93.85%	➡️ +0.00%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
src/log-directory-setup.ts	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

---

Coverage comparison generated by scripts/ci/compare-coverage.ts

[github-actions]

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

[github-actions]

✅ Contribution Check completed successfully!

[github-actions]

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

[github-actions]

📰 DEVELOPING STORY: Smoke Docker Sbx reports failed. Our correspondents are investigating the incident...

[github-actions]

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

[github-actions]

✅ Smoke Gemini completed. All facets verified. 💎

[github-actions]

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

[github-actions]

🚀 Security Guard has started processing this pull request

[github-actions]

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

[github-actions]

✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

[github-actions]

✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

[github-actions]

✅ Build Test Suite completed successfully!

[github-actions]

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

[github-actions]

✅ Smoke Claude passed

[github-actions]

🔌 Smoke Services — All services reachable! ✅

[github-actions]

Thanks for adding targeted regression coverage for the API-proxy token-tracking invariants. The changes are appropriately placed and include tests; documentation does not appear necessary for this test-only change. One contribution-guideline item remains: the PR description does not reference a related issue. If an issue tracks the null token-usage regression, please link it in the description (see CONTRIBUTING.md, Pull Request Requirements).

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Contribution Check for #6135 · 3.12 AIC · ⊞ 19.3K · ◷
_{Add label ready-for-aw to run again}

[github-actions]

Smoke Test: Services Connectivity

❌ Redis (PING): Network is unreachable
❌ PostgreSQL pg_isready: No response
❌ PostgreSQL SELECT 1: Network is unreachable

Overall: FAIL — host.docker.internal (172.17.0.1) is unreachable from this runner. Service containers may not be running or are not accessible from this environment.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

[github-actions]

@lpcox Smoke Test results:\n- MCP Testing: ✅\n- github.com Connectivity: ✅\n- File Write/Read: ✅\n- BYOK Inference: ✅\n\nRunning in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)\nOverall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
_{Add label ready-for-aw to run again}

[github-actions]

Smoke Test: Claude Engine Validation

Check	Result
API Status	✅ PASS
GH Check	✅ PASS
File Status	✅ PASS

Overall Result: PASS ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Smoke Claude for #6135 · 55.4 AIC · ⊞ 3.3K · ◷
_{Add label ready-for-aw to run again}

[github-actions]

Smoke Test: Copilot PAT Auth

Test	Result
GitHub MCP connectivity	✅ MCP reachable
GitHub.com HTTP	✅ HTTP 200
File write/read	⚠️ Pre-step template vars not expanded

Overall: PARTIAL — GitHub.com reachable and MCP works; pre-step data unavailable.

Auth mode: PAT (COPILOT_GITHUB_TOKEN) | PR author: @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 PAT report filed by Smoke Copilot PAT
_{Add label ready-for-aw to run again}

[github-actions]

Smoke Test: BYOK Direct Mode ✅ PASS

• ✅ GitHub MCP connectivity
• ✅ GitHub.com HTTP (200 OK)
• ✅ File write/read
• ✅ BYOK inference (agent → api-proxy → api.githubcopilot.com)

Running in direct BYOK mode with COPILOT_PROVIDER_API_KEY via api-proxy sidecar.

cc @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

[github-actions]

🔬 Smoke Test Results

Test	Status
GitHub MCP Connectivity	✅
GitHub.com HTTP	⚠️ pre-step outputs not interpolated
File Write/Read	⚠️ pre-step outputs not interpolated

Overall: PASS (engine reachable; workflow template variables were not expanded)

PR author: @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

[github-actions]

Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Result	Notes
Module Loading	✅	otel.js loads successfully; exports: startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled and internal helpers
Test Suite	✅	59 tests passed across 2 suites (otel.test.js, otel-fanout.test.js)
Env Var Forwarding	✅	api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID, OTEL_SERVICE_NAME
Token Tracker Integration	✅	onUsage callback exists in token-tracker-http.js as the OTEL hook point
OTEL Diagnostics	✅	No endpoint configured in sandbox; graceful degradation to file exporter (/var/log/api-proxy/otel.jsonl) as expected

All 5 scenarios passed. OTEL tracing integration is functional.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

[github-actions]

Smoke Test: Gemini Engine Validation

• PR titles: feat: add Docker sbx microVM runtime support (feat: add Docker sbx microVM runtime support #6101), [Unable to retrieve second PR]
• GitHub MCP Testing: ❌
• GitHub.com Connectivity: ❌
• File Writing Testing: ✅
• Bash Tool Testing: ✅

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
_{Add label ready-for-aw to run again}

[github-actions]

@lpcox and assignees

Smoke tests PASS for direct BYOK mode:

• GitHub MCP Testing ✅
• GitHub.com Connectivity ✅
• File I/O Test ✅
• BYOK Inference Test ✅

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
_{Add label ready-for-aw to run again}

[github-actions]

Smoke test PASS

• ✅ feat: add gVisor smoke tests for Claude, Codex, and build workloads
• ✅ chore: upgrade gh-aw to v0.82.8 pre-release and recompile workflows
• ✅ Browser title contains GitHub
• ✅ File write/read
• ✅ npm ci && npm run build
  Overall: PASS

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• awmgmcpg
• registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

[github-actions]

Chroot Version Comparison

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌ NO
Node.js	v24.18.0	v22.23.1	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Not all runtimes matched — ALL_TESTS_PASSED=false. Go versions match, but Python and Node.js differ between host and chroot environments.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

[github-actions]

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	all passed	✅ PASS
Node.js	execa	✅	all passed	✅ PASS
Node.js	p-limit	✅	all passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Build Test Suite for #6135 · 35.9 AIC · ⊞ 6.9K · ◷
_{Add label ready-for-aw to run again}