feat(billing): meter AI work per operation at cost-based credit prices - #3915
Conversation
…can charges Replace flagger-scan (30), live-eval-scan (30) and eval-generation (1000) with two primitive-level chargeable actions billed wherever they are produced: llm-call (250 credits) and semantic-query (30 credits). Prices are grounded in worst-case provider cost with a >=50% margin at the Pro overage rate ($0.002/credit): the worst LLM call is a Claude Sonnet 4.6 generation at high reasoning (~$0.30 for a 50k-in/10k-out envelope), so the estimated 100 credits was below the margin floor of 225 and is set to 250; the worst semantic query (voyage-4-large query embed + rerank pass, ~$0.015) keeps 30 credits with 4x headroom. The derivation is documented in dev-docs/billing.md. Metering happens at the AI layer: createAiLayer wraps every AI service with withAIMetering, which records one llm-call per generation and one semantic-query per query-time embedding against the ambient AIMeteringScope, sitting under the AI cache so cache hits are never charged. Expensive flows still authorize one llm-call at the boundary (free-cap and spend-cap gate) and then meter what actually runs, with sequence-numbered idempotency keys so deterministic retries dedupe. Scoped flows: flagger classification and annotation, live evaluations (judge scripts and semanticSimilarity comparisons), evaluation alignment judging, and GEPA optimization (proposer and candidate judges) — the last of which could previously burn up to 100M tokens against a flat 1000-credit charge. recordEvaluationGenerationUsage stays as a no-op activity for Temporal replay determinism. Web-interactive semantic search meters automatically once its server functions provide a scope; left as a documented follow-up because it needs 402/fallback UX decisions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Worst-case semantic query stays ~$0.015 (voyage-4-large query embed + rerank pass); price it at 2x rather than 4x: 15 credits = $0.03 at the Pro overage rate, still clearing the 50% margin floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
Settled margin targets: 1.5x over the worst LLM generation (Sonnet 4.6 high reasoning ~$0.30 → $0.45 = 225 credits) and 2x over the worst semantic query (unchanged at 15 credits). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
Most hosted AI work runs on MiniMax M2.5, so ground the llm-call price there instead of the Sonnet tier: worst case ~100k-in/4k-out ≈ $0.04, 1.5x margin → $0.06 = 30 credits at the overage rate. The rare Sonnet-tier calls (GEPA proposer, signal generation) are knowingly billed below cost per call; documented as a revisit trigger. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
geclos
changed the title
Each generation now bills creditsForLlmGenerationCost(estimatedCostUsd): provider-reported token usage priced through the @domain/models registry (the catalog that already prices customer spans), a 1.2x margin, converted at the overage credit value (2 mills/credit), rounded up to an integer with a 1-credit floor. Typical MiniMax judge call ~3 credits; a 100k-token-session judge ~24; a Sonnet GEPA proposal ~180. The flat ACTION_CREDITS llm-call price (30) remains the authorization estimate and the fallback when the registry has no pricing for the configured model or the provider reported no usage (including errored calls) — logged loudly so a stale registry doesn't silently eat margin. semantic-query stays flat at 15 (2x worst case): the embed adapter reports no usage. Ledger unchanged — billing_usage_events.credits was already per-event; record paths just accept an explicit credits value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
geclos
changed the title
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
📝 WalkthroughWalkthroughThis PR introduces AI metering scopes, cost-based ChangesAI metering and billing
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant WorkflowActivity
participant AIService
participant AIMeteringScope
participant Billing
WorkflowActivity->>Billing: authorize llm-call
WorkflowActivity->>AIMeteringScope: provide authorized scope
AIService->>AIMeteringScope: record AI primitive usage
AIMeteringScope->>Billing: record idempotent metered action
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Reconciles the cost-based billing model with development's session-level flagger screening and deterministic-eval-scan pricing: - Chargeable actions merge to trace (1), deterministic-eval-scan (1), semantic-query (cost-based, flat 15 fallback), llm-call (cost-based, flat 30 fallback). - Live evaluations keep development's capability split: llm()-capable scripts authorize one llm-call and meter each generation at cost through the AI metering scope; rules-only scripts record the flat 1-credit deterministic-eval-scan. - Development's new session flagger flow migrates from flat flagger-scan to per-call metering: the classifier activity is wrapped in withActivityAIMetering and the draft use-case authorizes llm-call and meters the fallback annotator, keeping anchor dedup before billing. - Semantic queries now bill at estimated embed cost with a 2x margin (voyage-4-large token rate; voyage is absent from the model registry), falling back to the flat price when the adapter reports no tokens. EmbedResult carries provider-reported tokens; the voyage adapter and AI cache round-trip them. - The per-comparison semantic-query record in semanticSimilarity() is removed: its query embeds are content-addressed document embeds and rule scans are covered by the deterministic-eval-scan credit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
|
📄 Generated a rich HTML walkthrough of this PR's diff against View it: https://artifact-pr-3915-diff-review.pine-music.workers.dev Claim within 60 minutes (otherwise the temporary link may expire): https://dash.cloudflare.com/claim-preview?claimToken=whuGcuFd5mSGa4cHQMWd47YRbtWmvQKBljxKVGFtbmY 🤖 Generated with Claude Code |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: faa70daf2a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| organizationId: input.organizationId, | ||
| projectId: input.projectId, | ||
| action: recordInput.action, | ||
| credits: recordInput.credits, |
There was a problem hiding this comment.
When this scope records a cost-based LLM call, recordInput.credits can be far above the 30-credit llm-call estimate that callers authorize up front. Because the append here reuses the authorization context but does not reserve/check recordInput.credits, a hard-capped free org or a pro org at a spending limit can pass the 30-credit gate and then have a GEPA/live-eval call append hundreds or thousands of credits, pushing consumedCredits and overage past the configured cap. Re-authorize/reserve the actual metered credits before appending, or otherwise bound the recorded amount.
Useful? React with 👍 / 👎.
| embed: (input: EmbedInput): Effect.Effect<EmbedResult, AIError> => | ||
| Effect.serviceOption(AIMeteringScope).pipe( | ||
| Effect.flatMap((scope) => { | ||
| if (Option.isNone(scope) || input.inputType !== "query") { |
There was a problem hiding this comment.
Live evals that call semanticSimilarity() embed the runtime query with inputType: "document" so it matches stored message vectors, but this wrapper only records semantic-query usage when inputType === "query". Those calls now run under an AIMeteringScope but never emit semantic-query usage, so semantic-only evals fall back to the 1-credit deterministic scan even when they make Voyage calls. Consider marking these runtime semantic embeds as billable independently of the Voyage inputType, or pass a billable query hint that does not change embedding semantics.
Useful? React with 👍 / 👎.
| traceId: input.traceId, | ||
| }), | ||
| }).pipe( | ||
| provideAIMeteringScope(meteringScope), |
There was a problem hiding this comment.
Adding this scope means withAIMetering can turn a billing persistence failure into an AIError, but the existing catchTags just below converts every AIError into a persisted errored evaluation. If the judge call succeeds and recording usage hits a transient DB/outbox failure, the worker will not retry; it will save a failed score and lose the billing event. Metering failures need a distinct error path or must bypass this provider-error catch so the job retries.
Useful? React with 👍 / 👎.
| input.contentHash, | ||
| ]) | ||
|
|
||
| const billing = yield* authorizeBillableAction({ |
There was a problem hiding this comment.
For matched flagger classifications that already include input.feedback, this draft path does not make another AI call and no longer records a usage event, but it still authorizes an llm-call before checking whether the annotator fallback is needed. On hard-capped or spending-limited orgs near the cap, a match whose classifier call was already metered can be blocked here and never saved, even though saving the provided feedback has no additional provider cost. Move this authorization into the feedback === undefined branch.
Useful? React with 👍 / 👎.
| ...(input.jobId !== undefined ? { jobId: input.jobId } : {}), | ||
| }, | ||
| }).pipe( | ||
| withActivityAIMetering({ |
There was a problem hiding this comment.
The incremental refresh use case returns a no-op result when there are zero new examples, which the workflow can hit because it collects with requirePositiveExamples: false; wrapping it in withActivityAIMetering authorizes an llm-call before that no-op branch runs. An out-of-credits org can therefore fail/retry an alignment refresh that would have done no AI work and only reported no changes. Check the example count before applying the metering gate, or make the no-op branch bypass authorization.
Useful? React with 👍 / 👎.
| evaluationHash: optimized.optimizedCandidate.hash, | ||
| } | ||
| }).pipe( | ||
| withActivityAIMetering({ |
There was a problem hiding this comment.
This single activity-wide metering scope is shared by GEPA evaluation callbacks, but LatitudeAdapter.evaluate batches up to 10 examples concurrently via server.batch(..., batch_size=10). Since the scope assigns idempotency keys from completion order, an activity retry can replay the same provider calls in a different order and attach existing sequence keys to different examples/candidates, skipping or duplicating charges despite the PR's retry-idempotency goal. Key GEPA records by stable candidate/example/propose identity instead of a shared sequence, or serialize scoped AI calls.
Useful? React with 👍 / 👎.
|
|
||
| return ai.generate(input).pipe( | ||
| Effect.tap((result) => recordGeneration(scope.value, input, result)), | ||
| Effect.tapError((error) => (error._tag === "AIError" ? recordFlat : Effect.void)), |
There was a problem hiding this comment.
If the provider call succeeds but recordGeneration fails, the failure is an AIError from metering and this tapError still runs the flat fallback recorder. That can insert a second flat llm-call under the next sequence key while the original effect still fails, so a retry can later record the cost-based charge as well. Limit the flat fallback to errors from ai.generate itself, or keep metering failures in a distinct error type.
Useful? React with 👍 / 👎.
| const authorization = yield* authorizeBillableAction({ | ||
| organizationId: OrganizationId(input.organizationId), | ||
| action: "eval-generation", | ||
| action: "llm-call", |
There was a problem hiding this comment.
optimizeEvaluationWorkflow still calls authorizeEvaluationGenerationBilling before the optimize/evaluate activities, but recordEvaluationGenerationUsage is now a no-op and those activities authorize/meter their own AI calls. This extra llm-call authorization reserves 30 credits with billingOperationId without any matching usage event, so hard-capped or spending-limited orgs lose headroom and can be blocked before the real per-call metering runs. Remove this reservation or make it a non-reserving availability check.
Useful? React with 👍 / 👎.
| result: GenerateResult<T>, | ||
| ): Effect.Effect<void, AIError> => { | ||
| const usage = result.tokenUsage | ||
| const costSpec = getCostSpec(input.provider, input.model) |
There was a problem hiding this comment.
Bedrock MiniMax generations can fall back to the configured openai.gpt-oss-120b-1:0 Bedrock model in @platform/ai-vercel; when that fallback succeeds, the returned token usage belongs to the fallback call, but metering looks up pricing and records metadata for the original input.provider/input.model here. That charges the wrong rate for fallback traffic and makes the billing event misleading. Have the adapter return the actual provider/model used, or fall back to flat billing when the executed model is unknown.
Useful? React with 👍 / 👎.
|
|
||
| if (!authorization.allowed) { | ||
| return yield* Effect.fail( | ||
| new NoCreditsRemainingError({ |
There was a problem hiding this comment.
withActivityAIMetering now fails activities with NoCreditsRemainingError, but the workflow default retry policy only marks BadRequestError as non-retryable, and the alignment/optimization activities wrap non-BadRequestError causes into 500-level activity errors. When an org is out of credits, refresh/optimization/flagger classify activities will therefore burn Temporal retries instead of failing fast. Throw a non-retryable Temporal failure or map this to the existing non-retryable billing error path.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/domain/flaggers/src/use-cases/draft-session-flagger-annotation.ts (1)
57-115: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winBilling gate blocks even the no-LLM-call path.
The
"llm-call"authorization check runs unconditionally before deciding whether an LLM call is even needed. Wheninput.feedbackis already provided (the documented "normally final" case), no LLM call occurs at all — yet an org out ofllm-callcredits will still getNoCreditsRemainingErrorfor this zero-cost path.billing.contextis only used inside theif (feedback === undefined)branch, so the gate should move there.🐛 Proposed fix: gate the authorize check on the LLM-needed branch
- const billing = yield* authorizeBillableAction({ - organizationId, - action: "llm-call", - skipIfBlocked: true, - idempotencyKey: buildBillingIdempotencyKey("llm-call", [ - input.organizationId, - "flagger", - input.flaggerSlug, - input.sessionId, - input.contentHash, - "authorize", - ]), - }) - - if (!billing.allowed) { - return yield* Effect.fail( - new NoCreditsRemainingError({ - organizationId, - planSlug: billing.context.planSlug, - action: "llm-call", - }), - ) - } - const scoreId = generateId<"ScoreId">() // The classifier's feedback is normally final; the annotator is the fallback // for a match that somehow arrived without feedback text. Its LLM calls bill // at cost through the metering scope, keyed by the flagged anchor so a // retried workflow replays the same idempotency keys. let feedback = input.feedback let messageIndex = input.messageIndex if (feedback === undefined) { + const billing = yield* authorizeBillableAction({ + organizationId, + action: "llm-call", + skipIfBlocked: true, + idempotencyKey: buildBillingIdempotencyKey("llm-call", [ + input.organizationId, + "flagger", + input.flaggerSlug, + input.sessionId, + input.contentHash, + "authorize", + ]), + }) + + if (!billing.allowed) { + return yield* Effect.fail( + new NoCreditsRemainingError({ + organizationId, + planSlug: billing.context.planSlug, + action: "llm-call", + }), + ) + } + const meteringScope = yield* makeAIMeteringScope({ organizationId, projectId, keyParts: ["flagger", input.flaggerSlug, input.sessionId, input.contentHash], context: billing.context, traceId: TraceId(input.latestTraceId), })🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/domain/flaggers/src/use-cases/draft-session-flagger-annotation.ts` around lines 57 - 115, Move the `"llm-call"` authorization and `NoCreditsRemainingError` handling into the `if (feedback === undefined)` branch before creating `met eringScope`, so the billing gate runs only when `annotateConversationForFlaggerUseCase` will make an LLM call. Keep the existing authorization parameters and use of `billing.context` unchanged for that fallback path; the pre-supplied feedback path must bypass billing entirely.
🧹 Nitpick comments (4)
packages/domain/billing/src/constants.ts (1)
17-22: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse concise single-line comments for the new billing documentation.
packages/domain/billing/src/constants.ts#L17-L22: condense the pricing explanation to one line; retain derivation indev-docs/billing.md.packages/domain/billing/src/constants.ts#L108-L137: condense the provider-pricing and conversion-helper documentation to single-line contract notes.apps/workers/src/workers/billing.test.ts#L836-L847: replace the scenario narrative with a concise test constraint.As per coding guidelines, “Keep comments rare and single-line.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/domain/billing/src/constants.ts` around lines 17 - 22, Condense the billing documentation comments to concise single-line contract notes: in packages/domain/billing/src/constants.ts lines 17-22, summarize the pricing behavior while retaining derivation in dev-docs/billing.md; in packages/domain/billing/src/constants.ts lines 108-137, reduce provider-pricing and conversion-helper documentation to single-line notes; and in apps/workers/src/workers/billing.test.ts lines 836-847, replace the scenario narrative with a concise statement of the test constraint.Source: Coding guidelines
packages/domain/billing/src/ai-metering.ts (1)
15-25: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCollapse these narrative block comments.
Keep only concise contract comments; the implementation details belong in the billing documentation. As per coding guidelines, “Keep comments rare and single-line.”
Also applies to: 38-44
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/domain/billing/src/ai-metering.ts` around lines 15 - 25, Replace the narrative block comments around the AI metering types and ambient billing scope with concise single-line contract comments. Retain only the public behavior needed to understand the exposed API, and remove implementation and usage details such as charge counts, billing examples, and internal tooling context.Source: Coding guidelines
apps/workflows/src/activities/evaluation-optimization-activities.ts (2)
78-79: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winConsider failing loudly instead of silently skipping metering when
meteringScopeisundefined.Today
withActivityAIMeteringalways provides the scope, so this branch shouldn't be hit in practice. But if it ever is,proposeOptimizationCandidate's LLM call would run completely unmetered with no signal — a silent billing-integrity gap.♻️ Proposed defensive log
}).pipe( - (effect) => (input.meteringScope ? provideAIMeteringScope(input.meteringScope)(effect) : effect), + (effect) => + input.meteringScope + ? provideAIMeteringScope(input.meteringScope)(effect) + : Effect.tap(effect, () => + Effect.sync(() => logger.warn("proposeOptimizationCandidate ran without an AI metering scope", { organizationId: input.organizationId })), + ),Also applies to: 122-122
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/workflows/src/activities/evaluation-optimization-activities.ts` around lines 78 - 79, Update proposeOptimizationCandidate and its AIMeteringScopeShape handling so an undefined meteringScope fails loudly before the LLM call instead of silently proceeding unmetered. Preserve the existing metered path when a scope is available, and apply the same validation at the other referenced meteringScope usage.
251-268: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the billing repository set into a shared layer instead of re-declaring it inline.
This is now the third place in the codebase composing the identical
BillingOverrideRepositoryLive, BillingUsageEventRepositoryLive, BillingUsagePeriodRepositoryLive, OutboxEventWriterLive, SettingsReaderLive, StripeSubscriptionLookupLiveset —evaluation-alignment-activities.tshasevaluationGenerationBillingRepositoriesLiveandflagger-session-activities.tshas a module-levelbillingLayersconst for the same six. Inlining a third copy here risks drift if the repo set changes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/workflows/src/activities/evaluation-optimization-activities.ts` around lines 251 - 268, Extract the repeated six-layer billing repository composition from the activity’s withPostgres call into a shared layer, reusing the existing evaluationGenerationBillingRepositoriesLive or billingLayers abstraction rather than declaring the set inline. Update the surrounding composition to consume that shared layer while preserving the current PostgreSQL client and organization configuration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/domain/billing/src/ai-metering.ts`:
- Around line 38-44: Replace the sequential-execution correctness requirement in
packages/domain/billing/src/ai-metering.ts lines 38-44 with a stable
per-logical-call identity requirement. Update the idempotency-key construction
at packages/domain/billing/src/ai-metering.ts lines 58-65 to use that stable
call identity instead of the scope-local execution sequence. In
dev-docs/billing.md lines 158-162, remove the cache-based retry guarantee until
parallel calls have deterministic billing identities.
In `@packages/domain/billing/src/constants.ts`:
- Around line 13-15: Update the billing flow for the ChargeableAction "llm-call"
so authorization reserves the request/model’s maximum possible cost before the
provider executes, rather than using the fixed 30-credit amount. After
completion, reconcile the reservation against actual usage, including cases
where actual cost exceeds the reservation, and add coverage for that overage
behavior while preserving existing handling for other chargeable actions.
In `@packages/domain/billing/src/use-cases/record-usage-event.ts`:
- Around line 15-16: Validate the optional credits override in the
usage-recording use case before applying it to creditsDelta: reject values that
are non-finite, non-integer, or less than 1 with the established domain error,
while preserving normal behavior when credits is omitted. Add boundary tests
covering zero, negative, fractional, NaN, and infinite values.
In `@packages/platform/ai/src/metering.ts`:
- Around line 52-55: Reorder the Effect pipeline in the metering flow so the
`Effect.tapError` handling for provider `AIError` runs before
`Effect.tap((result) => recordGeneration(...))`. Preserve the existing
`recordFlat` condition and generation recording behavior, ensuring failures from
`recordGeneration` do not trigger the flat-record path.
---
Outside diff comments:
In `@packages/domain/flaggers/src/use-cases/draft-session-flagger-annotation.ts`:
- Around line 57-115: Move the `"llm-call"` authorization and
`NoCreditsRemainingError` handling into the `if (feedback === undefined)` branch
before creating `met eringScope`, so the billing gate runs only when
`annotateConversationForFlaggerUseCase` will make an LLM call. Keep the existing
authorization parameters and use of `billing.context` unchanged for that
fallback path; the pre-supplied feedback path must bypass billing entirely.
---
Nitpick comments:
In `@apps/workflows/src/activities/evaluation-optimization-activities.ts`:
- Around line 78-79: Update proposeOptimizationCandidate and its
AIMeteringScopeShape handling so an undefined meteringScope fails loudly before
the LLM call instead of silently proceeding unmetered. Preserve the existing
metered path when a scope is available, and apply the same validation at the
other referenced meteringScope usage.
- Around line 251-268: Extract the repeated six-layer billing repository
composition from the activity’s withPostgres call into a shared layer, reusing
the existing evaluationGenerationBillingRepositoriesLive or billingLayers
abstraction rather than declaring the set inline. Update the surrounding
composition to consume that shared layer while preserving the current PostgreSQL
client and organization configuration.
In `@packages/domain/billing/src/ai-metering.ts`:
- Around line 15-25: Replace the narrative block comments around the AI metering
types and ambient billing scope with concise single-line contract comments.
Retain only the public behavior needed to understand the exposed API, and remove
implementation and usage details such as charge counts, billing examples, and
internal tooling context.
In `@packages/domain/billing/src/constants.ts`:
- Around line 17-22: Condense the billing documentation comments to concise
single-line contract notes: in packages/domain/billing/src/constants.ts lines
17-22, summarize the pricing behavior while retaining derivation in
dev-docs/billing.md; in packages/domain/billing/src/constants.ts lines 108-137,
reduce provider-pricing and conversion-helper documentation to single-line
notes; and in apps/workers/src/workers/billing.test.ts lines 836-847, replace
the scenario narrative with a concise statement of the test constraint.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 80e63708-1fd5-4a71-9275-1025c2e7e992
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (30)
apps/workers/src/workers/billing.test.tsapps/workers/src/workers/billing.tsapps/workflows/src/activities/ai-metering.tsapps/workflows/src/activities/evaluation-alignment-activities.test.tsapps/workflows/src/activities/evaluation-alignment-activities.tsapps/workflows/src/activities/evaluation-optimization-activities.tsapps/workflows/src/activities/flagger-session-activities.tsdev-docs/billing.mdpackages/domain/ai/src/index.tspackages/domain/billing/src/ai-metering.tspackages/domain/billing/src/constants.tspackages/domain/billing/src/errors.tspackages/domain/billing/src/index.tspackages/domain/billing/src/use-cases/authorize-billable-action.test.tspackages/domain/billing/src/use-cases/metering-and-overage.test.tspackages/domain/billing/src/use-cases/record-billable-action.tspackages/domain/billing/src/use-cases/record-usage-event.test.tspackages/domain/billing/src/use-cases/record-usage-event.tspackages/domain/evaluations/src/runtime/semantic-similarity.tspackages/domain/evaluations/src/use-cases/live/run-live-evaluation.test.tspackages/domain/evaluations/src/use-cases/live/run-live-evaluation.tspackages/domain/flaggers/src/use-cases/draft-session-flagger-annotation.tspackages/domain/queue/src/topic-registry.tspackages/platform/ai-voyage/src/ai.tspackages/platform/ai/package.jsonpackages/platform/ai/src/cache.tspackages/platform/ai/src/index.tspackages/platform/ai/src/metering.test.tspackages/platform/ai/src/metering.tspackages/platform/ai/src/with-ai.ts
| /** | ||
| * Identity of the logical operation, e.g. ["flagger", slug, traceId]. Keys are | ||
| * `{action}:{organizationId}:{...keyParts}:{sequence}` with the sequence assigned | ||
| * in call order, so retries of an operation whose calls replay deterministically | ||
| * re-produce the same keys and dedupe instead of double-charging. Parallel AI | ||
| * calls under one scope would break that guarantee — keep scoped calls sequential. | ||
| */ |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Use a deterministic logical-call key rather than execution order.
Ref.getAndUpdate assigns sequence when the record effect executes, so parallel calls can receive scheduler-dependent keys. A retry can therefore write a different usage event and double-charge when the cache does not satisfy the replay.
packages/domain/billing/src/ai-metering.ts#L38-L44: require a stable per-logical-call key part instead of documenting sequential execution as a correctness requirement.packages/domain/billing/src/ai-metering.ts#L58-L65: build the idempotency key from that stable call identity, not the scope-local execution sequence.dev-docs/billing.md#L158-L162: remove the cache-based retry guarantee until parallel calls have deterministic billing identities.
📍 Affects 2 files
packages/domain/billing/src/ai-metering.ts#L38-L44(this comment)packages/domain/billing/src/ai-metering.ts#L58-L65dev-docs/billing.md#L158-L162
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/domain/billing/src/ai-metering.ts` around lines 38 - 44, Replace the
sequential-execution correctness requirement in
packages/domain/billing/src/ai-metering.ts lines 38-44 with a stable
per-logical-call identity requirement. Update the idempotency-key construction
at packages/domain/billing/src/ai-metering.ts lines 58-65 to use that stable
call identity instead of the scope-local execution sequence. In
dev-docs/billing.md lines 158-162, remove the cache-based retry guarantee until
parallel calls have deterministic billing identities.
| export const CHARGEABLE_ACTIONS = ["trace", "deterministic-eval-scan", "semantic-query", "llm-call"] as const | ||
|
|
||
| export type ChargeableAction = (typeof CHARGEABLE_ACTIONS)[number] |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Reserve an upper-bound cost before executing llm-call.
llm-call is authorized at 30 credits, while post-call metering can record a larger actual cost. A generation exceeding 30 credits can therefore pass authorization and then push a hard-capped organization beyond its allowed spend. Reserve a request/model maximum before the provider call, then reconcile to actual usage; add coverage for actual cost above the reservation.
Also applies to: 23-28
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/domain/billing/src/constants.ts` around lines 13 - 15, Update the
billing flow for the ChargeableAction "llm-call" so authorization reserves the
request/model’s maximum possible cost before the provider executes, rather than
using the fixed 30-credit amount. After completion, reconcile the reservation
against actual usage, including cases where actual cost exceeds the reservation,
and add coverage for that overage behavior while preserving existing handling
for other chargeable actions.
…dback Every live evaluation scan now records the flat 1-credit eval-scan (renamed from deterministic-eval-scan) regardless of script capabilities; cost-based llm-call and semantic-query charges stack on top of it. Review fixes: - withAIMetering orders tapError before tap so a failed cost-based record cannot also commit the flat fallback record. - Live evaluations re-fail on metering-record failures instead of persisting a false errored score and losing the usage event. - The flagger draft use-case authorizes llm-call only on the annotator fallback branch; saving classifier-provided feedback costs nothing and succeeds even out of credits. - withActivityAIMetering fails blocked activities with BadRequestError so the workflow retry policy fails them fast instead of retrying. - recordUsageEvent dies on non-positive-integer credits overrides. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
20a1451 to
95c3006
Compare
|
Triage of the Codex and CodeRabbit findings, addressed in 95c3006: Fixed
Intentional tradeoffs, not changing in this PR
Also in 95c3006 per product direction: every live-eval scan now bills the flat 1-credit Generated by Claude Code |
A sweep for org-serving AI calls without a metering scope found seven gaps. This wires the standard-pattern ones: - Session analysis (analyzeSessionActivity) meters under a session-analysis activity scope — the highest-volume gap, one LLM pass per completed session. - Signal discovery (createSignalFromScore, assignOrCreateSignal), signal detail refresh (signals worker), taxonomy cluster naming, and annotation publication enrichment meter under per-activity scopes. The refresh worker keys its scope with a random suffix since the payload has no per-refresh identity; retried generates hit the 24h AI cache and are never re-charged. - withActivityAIMetering exports the shared billing repository layer set; the flagger and optimization activities reuse it instead of duplicating the six-layer composition. Still unbilled, documented in dev-docs/billing.md: web/API semantic search (needs out-of-credits UX), the signal-generation agent (AIAgent bypasses withAIMetering), and eval/signal previews (free by design). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk
3 participants
What does this PR do?
Changes how we bill AI work: instead of flat prices per feature scan, we now charge each LLM call and semantic query at what it actually costs us, plus a margin. Signal scans keep a flat baseline credit, with the AI charges stacked on top.
How we bill now
traceeval-scanllm-callsemantic-querySo a rules-only scan bills 1 credit; a judge scan bills 1 credit + its LLM calls at cost. Cost comes from the provider-reported token usage, priced through the same model registry we already use to price customer spans (Voyage embeds use a hardcoded rate since they're not in the registry).
Billing happens once, at the AI layer: every
generateand every query-timeembedcharges automatically when the calling flow provides a metering scope. Cache hits are never charged. Retries don't double-charge.Flows that bill after this PR: live evaluations, flagger classification and annotation, eval alignment, GEPA optimization — plus five flows a codebase sweep found running completely unbilled: session analysis (one LLM pass per completed session), signal discovery (name/description generation), signal detail refresh, taxonomy cluster naming, and annotation publication enrichment.
Revenue impact vs the old model
The old flat prices (
flagger-scan30,live-eval-scan30,eval-generation1,000) had margins that were random:Net effect: every AI operation carries a guaranteed 30% margin (100% on semantic queries), every scan contributes a baseline credit, typical users pay less, and heavy users stop being subsidized.
Not billed yet (documented in dev-docs/billing.md)
listTraces/listSessionsqueryparam and the web search boxes): the query embed costs us money but the endpoints don't provide a metering scope yet. Blocker is UX: deciding what an out-of-credits org sees (the planner already falls back to lexical search when the embedder is unavailable, so that's the likely answer).AIAgentservice, which bypasses the metering layer entirely — needs metering on the agent service itself.llm()free, treated as an authoring/testing surface by design for now.Related issue (if applicable)
N/A
How was this tested?
development:@domain/billing,@domain/evaluations,@domain/flaggers,@platform/ai, workers billing suite, workflows alignment suite. Full workspace typecheck passes.Checklist
🤖 Generated with Claude Code
https://claude.ai/code/session_01Pv3tfechUodwQspjfLv8Qk