refactor(accounting): remove company serialization from paid settlement (#18917)

Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work.

Co-Authored-By: Paperclip <[email protected]>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-04 07:29:28 -06:00
1 parent b891e69bd2
commit 0eaabe5c0b
76 files changed
+4372 -586

No files matched your search

+28
View File
@@ -1850,3 +1850,31 @@ the ordinary instance settings API. Seed inserts preserve an existing winner;
replacements lock the row and reject stale revisions. Database/transaction
handles permit atomic migration receipts. Connection URLs, API keys and unknown
fields are excluded. See [field ownership and startup order](cloud-runtime-configuration.md).
## Independent accounting and budget evaluation
`budget_evaluation_work` holds independent pending inputs for committed exposure
changes, with company and captured agent/project attribution. Producers append
in their financial transaction after registering transaction-aware notification.
The worker evaluates a consistent snapshot, fences policy publication through
`budget_policies.evaluation_version`, and deletes only its selected committed
input IDs in the publication transaction. A later commit with an older allocated
ID remains pending. Policy creation/edits also append inputs, so a previously
consumed no-policy batch cannot hide a newly applicable threshold.
No financial commit increments a company/agent spend counter or company dirty
version. Monthly API totals aggregate `cost_events`; runtime totals use existing
`accounting_runtime_baselines` plus acknowledged v2 run ledger. Retained counter
columns are non-authoritative. Invoice and verified late-receipt corrections
retain append-only `cost_adjustments` evidence and immutable reported amounts.
`cost_events.reported_usage_json` retains original placeholder usage before a
verified late receipt updates effective token/request fields; each adjustment
retains its previous usage in `previous_usage_json`.
Unknown late run receipts remain durable per-receipt debt; resolving one receipt
cannot clear another unresolved receipt.
Upgrade all controllers/workers to this accounting contract and drain incompatible
pre-upgrade database transactions before enabling independent writers. Do not run
pre-upgrade synchronous counter/evaluation writers beside unlocked writers.
Rollback must use a compatible version that reads the ledger and retains the
durable evaluation feed; a pre-contract binary is not a safe rollback target.
+11
View File
@@ -459,3 +459,14 @@ The agent performs authorized setup, reuses saved accounts and drafts, and hands
Authorized policy administration includes user-requested changes that loosen the acting agent’s own tool controls. These writes remain bounded by management grants, resource scope and the responsible user’s current authority; they do not need an additional approval solely because the agent is a target. Subsequent tool calls obey the resulting policy. Agents cannot approve their own governed action requests or edit their own principal permission grants.
The implementation, operation inventory, permission matrix and fixture qualification are recorded in [Agent API parity for connection configuration](plans/2026-10-13-connection-api-parity.md). Provider fixtures establish API behavior; they do not constitute live provider setup or an empirical agent-success claim.
## Budget control behavior
Paperclip records paid-operation charges durably and avoids company-wide
serialization during receipt ingestion and settlement. Budget checks and
threshold enforcement are best effort. Concurrent operations can start from
the same observed balance; estimates and provider caps do not replace the
actual bill. A background worker applies threshold holds and scoped Stop intent.
Manual Stop and authorization remain independently enforced. Unknown charges
remain visible, and stale unevaluated accounting blocks new limited-budget work
until reconciliation catches up.
+27 -11
View File
@@ -47,7 +47,7 @@ These decisions close open questions from `SPEC.md` for V1.
| Plugin framework | Local/self-hosted early plugin runtime is in scope; cloud marketplace and packaged public distribution remain out of scope |
| Auth | Mode-dependent human auth (`local_trusted` implicit board in current code; authenticated mode uses sessions), API keys for agents |
| Budget period | Monthly UTC calendar window |
| Budget enforcement | Soft alerts + hard limit auto-pause |
| Budget enforcement | Best-effort admission, soft alerts and asynchronous threshold auto-pause |
| Deployment modes | Canonical model is `local_trusted` + `authenticated` with `private/public` exposure policy (see `doc/DEPLOYMENT-MODES.md`) |
Low-trust agent presets are containment controls for hostile automated work, not
@@ -1736,15 +1736,31 @@ Scheduler must skip invocation when:
- an existing run is active
- hard budget limit has been hit
Authoritative paid-work admission and accounting mutations serialize under the
company accounting lock. Preliminary invocation checks use a read-only snapshot
and take that lock only when a hold, incident, or enforcement transition needs
reconciliation; reconciliation rereads current state under the lock.
The dispatch boundary rechecks current company, agent, and project policies and
reserves capacity before paid work starts. Accounting commits budget blocks and
durable stop intent without waiting for provider shutdown or reconciling unrelated
agents. An asynchronous worker cancels only work in the affected scope; committed
stop intent survives policy changes and restart until existing stop proof succeeds.
Paid-operation settlement and receipt ingestion use operation-scoped transactions.
They do not acquire the company accounting mutex or update shared company/agent
spend counters. The ledger, reservation transition and accounting acknowledgement
commit atomically. Duplicate evidence uses stable identities; a conflicting
receipt is not another charge. Monthly and lifetime totals read the canonical
ledger and retained historical baselines.
Admission reads current company, agent and project controls plus coherent budget
exposure. Reservations record an operation's estimated obligation, rather than
an exclusive claim on shared remaining capacity. Concurrent admissions may see
the same available balance and both start. Budget enforcement is best effort,
not a guaranteed bound on final billed spend. Manual Stop, lifecycle authority,
permissions and provider-dispatch ownership retain their final gates.
Financial changes append independent durable evaluation inputs. The existing
worker coalesces notifications over one second and publishes budget holds,
incidents and durable Stop intent from a consistent snapshot. Policy/control
publication is fenced separately from settlement. Inputs arriving or committing
later stay pending; no global sequence watermark consumes them. Limited-budget
admission refuses relevant unevaluated inputs older than fifteen seconds.
Unchanged idle decisions do not expire; unknown accounting retains its separate
conservative admission restriction. Provider reporting and cancellation delays
remain additional to evaluation lag. Cancellation delivery is asynchronous and
scoped; committed per-run Stop survives policy changes and restart until existing
stop proof succeeds. Budget resume cannot clear a manual hold.
Accepted cumulative per-run usage checkpoints contribute observed running spend
before the final ledger charge. Admission counts each unfinished operation's
@@ -1845,7 +1861,7 @@ for contracts, recovery behavior, Storybook, and acceptance workflows.
## 13.2 Enforcement Rules
- soft alert default threshold: 80%
- hard limit: at 100%, trigger:
- at 100%, the asynchronous evaluator triggers:
- set agent status to `paused`
- block new checkout/invocation for that agent
- emit high-priority activity event
+1 -1
View File
@@ -42,7 +42,7 @@ Credentials are resolved through the existing secret service immediately before
Cost events have `usageKind: decision` and may have no agent. Agentless reporting uses **Paperclip services**. The public agent cost-reporting API still requires an agent; only the internal service receipt path accepts agentless charges. Decision events snapshot responsible-user attribution; historical run costs continue to use their existing attribution rules. Decision events have no heartbeat receipt reference, so run receipt reconciliation cannot count them again.
Admission and settlement use the company accounting lock. Company and applicable agent/project budgets include held reservations and observed spend. Reservations use the largest configured applicable reservation amount. Settlement is idempotent. Explicit provider authentication/validation/rate-limit rejection releases the hold when no model work occurred. Uncertain dispatches, timeouts, and unknown prices remain unpriced with the reservation held. Startup/periodic accounting recovery marks stale invocations interrupted without resubmitting them. Use the existing audited cost adjustment workflow to resolve the charge and release its hold, including a verified zero charge when appropriate.
Paid admission and settlement use independent operation transactions. Admission keeps shared manual-control fences and best-effort budget observations; concurrent starts can exceed the same observed balance. Settlement does not acquire a company writer mutex. Company and applicable agent/project budgets include held reservations and observed spend. Reservations use the largest configured applicable reservation amount. Settlement is idempotent. Explicit provider authentication/validation/rate-limit rejection releases the hold when no model work occurred. Uncertain dispatches, timeouts, and unknown prices remain unpriced with the reservation held. Startup and deadline-driven accounting recovery marks stale invocations interrupted without resubmitting them. Late authoritative receipts can correct provisional interrupted accounting without another provider call. Use the existing audited cost adjustment workflow to resolve an otherwise unknown charge and release its hold, including a verified zero charge when appropriate.
OpenRouter-reported USD costs are converted to fractional cents. OpenAI uses a dedicated, versioned Decisions rate snapshot: $0.10 per million input tokens for the bounded short requests at the native default endpoint. No chat-model price is substituted. Missing usage or an unsupported rate stays unknown; it is never displayed as a free request. The ledger retains seven decimal places in cents. Pricing provenance accompanies the cost event and remains available to accounting corrections.
+1 -1
View File
@@ -26,7 +26,7 @@ External publication uses the existing sanitized outbox, exact originating endpo
## Accounting and endpoints
Every dispatched attempt uses the existing accounting lock, budget checks, reservation and cost ledger with `usageKind = fast_response`. Settlement is idempotent and keeps actual charges even when output is suppressed. OpenRouter supplies reported costs. Direct OpenAI/Anthropic models covered by the repository's reviewed rate cards receive explicit estimates. Unsupported pricing and ambiguous network outcomes remain unpriced, never fabricated as zero; reservations remain held until billing reconciliation.
Every dispatched attempt uses shared manual-control admission fences, best-effort budget checks, its own reservation and an independent cost-ledger transaction with `usageKind = fast_response`. Settlement is idempotent, avoids the company writer mutex, and keeps actual charges even when output is suppressed. Durable budget work is evaluated asynchronously. OpenRouter supplies reported costs. Direct OpenAI/Anthropic models covered by the repository's reviewed rate cards receive explicit estimates. Unsupported pricing and ambiguous network outcomes remain unpriced, never fabricated as zero; reservations remain held until billing reconciliation.
**Costs → Fast responses** shows connection, model, tokens, cost, timing, generation outcome and current publication outcome. Operational request rows contain identifiers and safe outcome codes, not prompts. Delivered comment text remains in conversation history.
+16 -13
View File
@@ -50,20 +50,23 @@ The operator-gated `paperclip.budgets` tracer records
`budget.accounting.transaction`, `budget.accounting.settlement`, `budget.accounting.lock_wait`,
`budget.exposure.evaluate`, `budget.enforcement.deliver`,
`budget.enforcement.scope`, `budget.cancellation.deliver`, and `budget.accounting.recover`.
Inspection spans cover the preliminary read-only snapshot. Reconciliation spans
exist only when that inspection finds a transition requiring the company lock;
compare their counts to measure how often preliminary checks bypass the lock.
Authoritative reservation and transactional evaluation retain their own spans.
Transaction spans include commit time. Lock-wait spans cover acquisition of the
company accounting lock. Settlement spans cover the ledger and reservation
transaction. Enforcement spans measure asynchronous delivery; cancellation
delivery spans start after acquiring a concurrency slot and cover the external
Stop attempt. Both remain separate from provider inference.
Inspection and admission spans cover read-only eligibility. Paid reservations,
receipts and all five settlement owners avoid the company accounting mutex.
`budget.admission.reconcile` remains a historical span name; eligibility no longer
publishes budget transitions. `budget.accounting.lock_wait` still describes the
legacy low-frequency configuration/control wrapper, not paid settlement.
Transaction spans include commit time. Settlement spans cover the atomic ledger,
reservation and owner acknowledgement. Acquisition duration is not a pure
PostgreSQL lock-wait measurement: it can include network and Node scheduling.
Enforcement spans cover asynchronous evaluation/delivery; cancellation delivery
covers the external Stop attempt, separate from provider inference.
The closed attribute list is `budget.policy_count` (number of applicable
policies) and `budget.enforcement.queue_delay_ms` (time since the pending policy's
last intent or retry timestamp). These spans contain no tool content, usage
receipt bodies, credentials, or provider error bodies. They do not change
The closed attribute list is `budget.policy_count` (applicable policy count),
`budget.enforcement.queue_delay_ms` (time since pending intent/retry), and
`budget.accounting.owner` (`run`, `decision`, `classifier`, `fast_response`,
`browser`). The owner tag identifies the shared settlement span's emitter.
These spans contain no prompts, usage receipt bodies, credentials or provider
error bodies. They remain gated by the operator OTLP endpoint and do not change
Paperclip's separate first-party Telemetry contract.
## Slack receipt reactions
@@ -0,0 +1,405 @@
# Remove company serialization from all five settlement paths
Date: 2026-11-03
Status: Implemented candidate; final verification/review in progress
Source reviewed: `a194166017597860a0e53c0c1314e95c597d75cc`
## Decision and finish line
Remove the company accounting mutex from run, decision-model, fast-response,
prompt-classifier and browser settlement. Independent paid operations in the
same company, including the same agent, must commit independently. Remove the
shared company/agent counter writes and synchronous budget evaluation from those
commits too; otherwise removing the explicit lock leaves the bottleneck intact.
Keep durable receipt/ledger accounting exact and replayable. Make budget
admission and threshold enforcement explicitly best effort: concurrent work can
start from the same available-budget observation, and cancellation follows a
coalesced decision. Reservations record each operation's obligation; they no
longer allocate an exclusive portion of a company balance.
This eliminates company-wide accounting serialization, not PostgreSQL's internal
locking. Keep the existing short operation-row arbitration and unique keys for
two workers settling the *same* operation. Resource ownership, lifecycle fencing,
foreign keys and manual Stop retain their domain owners. No company advisory
lock, company CAS counter or per-receipt policy-row update replaces the mutex.
This is the concrete next milestone after PR18830 and the earlier
[architecture proposal](2026-11-03-lock-free-accounting-design.md). It does not
require that proposal's full append-only journal/partition-reducer system.
Removing the mutex can improve settlement and dispatch capacity; it does not by
itself prove a CPU or event-loop improvement. Existing generic settlement spans
also do not identify which of the five callers owned a slow transaction.
## Current owners
All five call `withAccountingTransaction`, which awaits
`SELECT companies.id ... FOR NO KEY UPDATE` before running the callback.
| Path | Owning service and entry point | Existing local accounting identity |
| --- | --- | --- |
| Agent run | [run-cost-accounting.ts](../../server/src/services/run-cost-accounting.ts), `accountRunCost` | Run row; `heartbeat:<run>:final` / existing model-part keys |
| Decision model | [decision-models.ts](../../server/src/services/decision-models.ts), `settle` | Invocation row; `decision:<id>` |
| Fast response | [fast-responses.ts](../../server/src/services/fast-responses.ts), `settle` | Request row; `fast-response:<id>` |
| Prompt classifier | [prompt-injection-prevention.ts](../../server/src/services/prompt-injection-prevention.ts), `settle` | Invocation row; `prompt-injection:<id>` |
| Browser | [browser-use.ts](../../server/src/services/browser-use.ts), `account` | Browser-run row; provider run identity and cumulative exact total |
The common cost helper also updates `companies.spentMonthlyCents` and
`agents.spentMonthlyCents`, sometimes `agentRuntimeState` totals, then evaluates
budget exposure. Run settlement and billing correction have additional runtime
counter writes. These are prerequisites to remove, not optional follow-ups.
## Shared foundation before the first path is unlocked
### 1. A narrow ledger transaction contract
Split [costs.ts](../../server/src/services/costs.ts) so owners can call a
ledger-only insertion primitive. It validates tenant/attribution, normalizes exact
money, inserts the cost event, writes audit evidence and registers durable budget
work. It neither updates shared totals nor evaluates policies.
Preserve `(companyId, idempotencyKey)`, immutable receipt fingerprints and existing
legacy-hash replay compatibility. Replace read-then-insert reliance on the mutex
with conflict-safe insertion followed by comparison with the committed winner.
Equal evidence is replay; conflicting evidence is an explicit conflict. Only the
winner emits the charge/audit effects. Uncertain commit acknowledgement is safe
to retry with the same identity.
Each owner uses an ordinary transaction to:
1. Arbitrate its own operation row and recheck current accounting evidence.
2. Insert/reuse all qualified ledger parts through the common helper.
3. Replace that operation's live exposure/hold with its ledger contribution.
4. Record its acknowledgement and durable follow-up work, plus finance/audit
records required by that owner.
5. Commit, then publish existing post-commit activities and delete any acknowledged
receipt spool files.
All five steps roll back together on failure. Provider requests, provider polling,
filesystem reads and unrelated reconciliation stay outside the database transaction.
Keep provider-specific receipts, result handling and retries in their services;
do not create a generic helper that switches on arbitrary invocation tables.
Use the existing `cost_adjustments` mechanism for later authoritative valuation:
extract an event-scoped transactional correction core shared by invoice/manual
correction and verified late receipt handling. Preserve the original reported
amount/fingerprint, expected-value checks, correction provenance and idempotency.
Correction commits its owner's hold/evidence transition and budget work together.
No second charge is created for the same paid attempt.
Use one audited lock order across settlement and manual/late correction:
attribution issue/FK protection, owner row, existing ledger/correction targets,
then own reservation. Discover candidate attribution without locks and revalidate
it after acquiring the owner fence. Update the existing invoice correction's
ledger-first ordering when it also touches owner state; do not create an inversion.
### 2. Remove hidden company and agent counter writes
Stop monthly increments/reinitialization in `updateMonthlySpendProjections` for
every financial writer, including direct cost reporting and billing corrections.
Company/agent API hydration already aggregates the ledger; reuse bounded, batched
ledger reads. Monthly UTC windows and backdated adjustments then follow ledger
dates without a shared rollover initialization write.
Migrate direct raw-counter budget consumers in `email-channels.ts`,
`external-agents/dot.ts`, `muse.ts`, `native-broker.ts` and
`native-runtime/paperclip-runner-tool-authority.ts`. Budget authority comes from
the shared eligibility contract; display totals come from ledger aggregation.
Audit all other readers before retiring fields. Keep old columns temporarily
for migration compatibility, but make them non-authoritative.
Remove lifetime money/token increments from run settlement, late cost reporting
and billing reconciliation. Reuse the existing accounting-integrity formula:
`accounting_runtime_baselines + ledger for acknowledged v2/new-version runs`.
Hydrate public runtime-state reads, leaving frequent internal session/control
reads cheap. Preserve historical v2 contributions and existing baselines;
initialize missing baselines once under the old writer fence/drain with an
explicit reconciliation, never by adding the same historical ledger twice.
Do not use an ID/time cutoff as a commit-order boundary. Preserve exact arithmetic
until existing API serialization boundaries.
Update integrity inspection/repair to validate canonical receipts, acknowledgements,
baselines and ledger valuation. It must not flag intentionally retired counters
as corruption or write them back into authority. If read aggregation later needs
a materialized view, batch it off the settlement path; no per-charge company or
agent accumulator is part of this milestone.
### 3. Coalesced budget decisions with durable, independent inputs
Extend the existing budget worker/coordinator, not a new queue service. Add one
durable evaluation-work feed with independent rows for committed changes:
company, operation/source identity, input revision or event identity, affected
scope attribution, server enqueue time and pending/consumed state. Enforce logical
identity with a unique index and index pending work/oldest age by company/scope.
Settlements append their own rows; they never upsert one company dirty flag.
Cover *all* exposure producers: admissions/reservations, live usage, terminal
settlement, zero-charge releases, unknown outcomes, source handoffs/capture
failures, browser holds, direct charges, corrections, policy edits and UTC-window
transitions. A replay with no state change does not manufacture more work.
The evaluator coalesces committed inputs by affected scope. In one repeatable-read
snapshot, select explicit pending IDs and read policy configuration/decision and
manual-control versions plus ledger, observations, reservations and browser exposure using
`budget-exposure.ts`. A settlement is visible entirely before or after its atomic
hold-to-ledger handoff, so it cannot be omitted or counted twice. Initially reuse
the existing batched exposure queries; add partitioned totals only if measured
batch cost requires them. Use one fixed evaluation time/window for that snapshot.
Publish through a short transaction conditional on the captured policy,
decision and manual-control versions and the still-applicable window. A losing
evaluator recomputes from a new snapshot; it cannot
attach a fresh version to stale totals. Commit the winning decision, incident/
hold transition, idempotent durable stop intent and consumption of only the
selected inputs together. Consume an input only when every affected scope is
covered; if batches split scopes, retain per-scope acknowledgement. Include both
old and new attribution when a mutation changes scope. Policy edits invalidate
old decisions. New or late-committing inputs stay pending; never acknowledge
`MAX(id)`, sequence or time.
If no policy applies, consume that work; policy creation emits a fresh obligation.
Resume requires agreement of all applicable policies and the current manual-
control generation, not merely one policy dropping below its limit.
Admissions read current controls and applicable decisions from an authoritative
consistent snapshot, then write only their own invocation/reservation. They do
not reserve/decrement shared capacity or synchronously reconcile policy state.
Estimate-fit checks are advisory observations. Missing decisions or unresolved
unpriced work are not implicitly unlimited. Manual pause/Stop, permissions,
dispatch identity and lifecycle fences remain final admission gates. Preserve
provider caps where supported and agent-deletion fencing around new holds.
Proposed healthy-load settings, to qualify: about one-second coalescing, p99
priced-change commit to durable stop intent at most five seconds, and refusal
of new limited-budget paid work when relevant unevaluated changes exceed fifteen
seconds. Measure oldest applicable unevaluated-input age; do not refresh freshness
merely because a worker woke. Successful evaluation of unknown/capture-failed
evidence can consume its input while retaining a durable uncertainty/block state
and source recovery obligation. Unknown billing age is separate from evaluator
lag; unchanged uncertainty must not keep an empty evaluation loop running. An
unchanged idle decision does not expire. Policy edits and window changes create
real new obligations.
Provider reporting/cancellation delays and already in-flight spending are extra;
these targets do not bound billed-dollar overshoot. Receipt acceptance continues
while admission is blocked.
Use transaction-aware `notifyDeliveryWork` before the relevant write. Durable
rows are truth; notifications are hints. Startup/resume and reconnect reconstruct
pending work, and changes during a batch/drain remain discoverable. Use bounded
batches/retries and actual scheduler deadlines. Once settled, no recurring empty
SQL scan remains. Qualify cross-process delivery before supporting multiple
controllers; preserve existing sleep blockers until durable external wake is
proven. Cancellation delivery stays scoped, idempotent and recoverable; budget
resume cannot clear a manual hold or retract an already committed per-run Stop.
## The five owner refactors
### Run settlement: `accountRunCost`
- Replace the company wrapper with a run-scoped transaction; preserve issue/FK
lifecycle protection and the existing run-row arbitration order.
- Give usage receipt persistence a caller-transaction primitive so finalization
never enters a nested company wrapper. Migrate live receipts, source handoff
and capture-failure writers to the same run-local evidence contract.
- Capture a durable closed-source/final-receipt identity before reading receipt
files outside the transaction. Under the run fence, revalidate that closure,
source and accepted final identity; retry if they changed. A pre-read directory
scan is not proof the producer finished. Delete only acknowledged files after
commit. Genuinely late post-ack evidence must create a correction/uncertainty
obligation; the current journal-and-return behavior is insufficient.
- Preserve native closure, final-usage completeness, dispatch provenance and
capture-failure checks. A terminal runtime status alone is not accounting
closure. Accounting-only late receipt authority never authorizes execution.
- Atomically write all model parts with existing stable keys/order, settle or
release the run's reservation, replace running exposure and acknowledge the
run. Keep operator retry and background reconciliation on this same function.
- Remove agent runtime increments; mark the run for the baseline-plus-ledger read
contract instead. Test final receipt versus live checkpoint/recovery in both
orders, missing closure, multi-model replay and commit-before-file-delete crash.
### Decision-model settlement: `settle`
- Keep invocation-scoped arbitration and `decision:<id>` identity; atomically
commit result/accounting metadata, cost link and own reservation transition.
- Preserve registered-feature authorization, background sponsorship and archival
dispatch attribution; this independent charge has no heartbeat ledger link.
Do not store prompts/answers to obtain billing evidence.
- Persist normalized billing evidence promptly after provider completion and
retry persistence with the same invocation identity, without calling the
provider again. A crash with no recoverable receipt remains explicitly unknown.
- Make interrupted/unknown closure provisional for billing: later authoritative
evidence corrects the same event through the audited correction core. Racing
recovery cannot discard a real price because the result status is terminal.
- Test duplicate workers, priced failure, unpriced success, no-provider-work
release, late price after recovery and deletion of linked task/project.
### Prompt-classifier settlement: `settle`
- Use invocation-scoped arbitration and `prompt-injection:<id>` with the same
exact ledger/reservation contract and provisional-unknown correction behavior.
- Keep classifier verdict, attribution and billing certainty separate. A failed
or invalid classification can still cost money; an unpriced receipt does not
become a safe/unsafe verdict.
- Preserve the order that commits incurred charge before propagating caller
cancellation. Recovery and abort cannot release a hold as zero-work unless
provider work is proven absent.
- Test abort during provider completion, recovery racing a priced result, verdict
behavior when accounting retries, and idempotent hold resolution.
### Fast-response settlement: `settle`
- Use request-scoped arbitration and `fast-response:<id>`; explicitly include
`companyId` in claim/settlement predicates. Keep cost, terminal metadata and
reservation changes in one accounting transaction.
- Preserve the existing independent acknowledgement publication transaction:
publication precedes accounting and never waits for budget evaluation. Retain
source-current, expiry, disable and superseding-real-reply checks.
- An obsolete or suppressed reply can still have incurred a charge. Recover
unknown billing and late prices without republishing or rerunning the provider.
- Test expiry/disable/real-reply races, slow accounting after publication, two
recovery workers, publication success with lost accounting acknowledgement,
and late authoritative receipt after interrupted recovery.
### Browser settlement: `account`
- Keep provider terminal summary I/O outside the transaction. Arbitrate only the
browser-run row, not its whole company or session.
- Persist normalized observed summary evidence before settlement with a stable
receipt/occurrence time and captured attribution; retries must not move spend
across UTC windows by using a new `now`. The provider currently exposes no
billing revision/finality marker: a valid priced terminal summary settles the
known amount without claiming it can never change. Keep stream-drained status
separate from billing certainty; qualify late receipt/recovery ingress and
never keep polling completed work without an unresolved duty.
- Store exact accounted amount and normalized billing-evidence state in typed
accounting fields instead of relying on the integer display field/private
result JSON. Preserve backward reads while backfilling existing exact values.
- Validate the provider-run/invocation identity. Atomically advance its exact
cumulative amount, insert only the qualified positive delta, link the finance
event, replace its hold, record events-drained evidence and append budget work.
Replays/stale lower snapshots add no charge; a proven authoritative reduction
uses audited correction, never silently rewinds the total.
- Unknown POST outcomes/uncapped work remain unknown until proven. Keep provider
caps as dispatch controls; never clip the recorded bill to a configured cap.
- Fence stale session-sweeper/cursor writes by existing lease token/version;
financial correctness rests on run identity and unique evidence, not the session
lease. Test two reconcilers, cumulative growth, duplicate/lower summaries,
correction, lost commit acknowledgement, unknown POST and month rollover.
For all three AI services: proven no-provider-work releases its hold; a priced
outcome settles it; an unpriced outcome retains uncertainty until correction.
Result success/failure and billing certainty are independent dimensions.
## Delivery and compatibility
Prepare a reviewable prerequisite stack, keeping proof with each change:
1. Ledger-only append/correction primitives and owner-tagged observability,
initially behind the existing company wrappers. Compatibility callers retain
old projection/evaluation orchestration outside the new primitive. Test
conflict/replay and transaction atomicity without changing budget authority.
2. Retire monthly/runtime counter writes and migrate every reader/repair path.
Reconcile baselines and reports before enabling subsequent writers.
3. Install the durable input feed, fenced evaluator and read-only best-effort
admission contract across every producer/consumer. Existing settlement wrappers
can remain briefly, but no writer relies on them for aggregate consistency.
Keep synchronous evaluation authoritative until the new feed is caught up;
the shadow evaluator must not publish competing holds/Stops. Under the drain,
switch to one fenced evaluator/admission authority before unlocking a writer.
Update `doc/SPEC-implementation.md`, relevant product/API budget wording and
tests to remove the strict concurrent-capacity guarantee explicitly.
These together form the **compatibility floor**. Deploy it everywhere and drain
old controllers/workers and in-flight database transactions before unlocking a
path. External paid work may finish later through the compatible receipt path.
Old binaries cannot coexist with unlocked writers: their multi-query exposure
reads and unfenced budget transitions assumed the mutex isolated all writers.
Do not mix old counter increments with asynchronous absolute recomputation.
Then ship one owner change per PR: decision model first as the smallest exemplar,
followed by classifier, fast response, browser and run accounting. The latter
four may be developed independently on the shared floor. Include companion
admission/recheck/release/recovery entry points in the owning PR, not just the
visible settlement callback. In particular audit `reserveRunBudget`, browser
before-dispatch/rejected/unknown/stale-creating paths, usage receipts, direct
costs, billing reconciliation and budget startup/rollover/stop delivery.
Rollback after unlock returns to a compatible version that retains the new
ledger/evaluation contract, optionally re-enabling a wrapper. A pre-floor binary
is not a safe rollback. Keep additive fields and old read compatibility through
the rollout; remove obsolete helpers/columns only after all five migrate.
## Acceptance
- **Real PostgreSQL concurrency:** hold the company row `FOR NO KEY UPDATE` on
connection A. Every path must settle on B before A releases it. Repeat while
holding an agent row and a budget-policy row. This catches hidden shared writes.
Do not use company `FOR UPDATE` for this test: legitimate FK key-share checks
conflict with deletion/key-changing locks.
- **Operation isolation:** hold operation A's row; operation B in the same company
and agent still settles. Duplicate A produces one effective charge, finance
link, acknowledgement and reservation transition. Inject rollback at ledger /
owner / hold / work-feed boundaries and replay a lost commit acknowledgement.
- **Accounting correctness:** exercise the owner cases above, correction versus
settlement, exact fractional money, unknown evidence, multi-model identity,
invoice adjustment, UTC rollover and historical baselines. No lost money,
double-counted exposure or discarded authoritative late price.
- **Best-effort controls:** simultaneous admissions may oversubscribe an observed
balance and must retain all obligations. A visible block/manual Stop still
denies new work. Test policy edits racing evaluation, old/new evaluators,
stale-view refusal, lifecycle deletion and durable scoped cancellation.
- **Recovery/quiet:** committed late inputs survive a batch, rollback, restart,
reconnect and drain/resume; a lower allocated ID that commits later is found.
No recurring SQL/network after completion. Real cross-process wake remains a
separate acceptance claim from in-process coordinator tests.
- **Performance:** benchmark pinned baseline/treatment on real PostgreSQL with
concurrent same-company/same-agent paid operations plus foreground requests.
Match workload/policies/history/resources; report completed work, errors,
settlement/admission/request p95/p99, queries per operation, database wait/hold
evidence, evaluator backlog/stop lag, process CPU and event-loop delay.
Add closed owner tags (`run`, `decision`, `classifier`, `fast_response`,
`browser`) to existing operator-gated OpenTelemetry spans; include replay and
outcome categories without prompts/receipts/credentials. New spans or reduced
counts alone do not establish a speedup.
Use the smallest existing owner tests during development, real PostgreSQL for lock
proof, then required repository typecheck/tests/build and final-head PR checks.
Hosted acceptance confirms the serving revision and matched ordinary workload;
report lock removal, useful latency improvement and CPU/resource improvement as
separate findings. No live profiling or infrastructure change is required to
approve this plan.
## Implementation record
The candidate removes the company accounting wrapper from all five settlement
owners and their paid admission/recovery companions. Shared counter writes are
retired. Durable per-change budget inputs and coalesced fenced evaluation replace
synchronous policy evaluation. Exposure aggregates use one SQL statement, so
READ COMMITTED callers also observe the complete hold-to-ledger handoff.
Publication attempts use nonwaiting scope-row guards after the policy fence.
A conflicting existing control writer causes rollback/retry, rather than a
policy/resource lock inversion. New-policy edits leave durable inputs even when
a prior no-policy batch consumed their concurrent charge. Public report reads
create a fresh snapshot per call. UTC publication guards retain work on rollback.
Implementation delivery uses one coherent protocol-change PR, with separate
logical commits for the shared foundation and each owner. This replaces the
initial per-owner PR recommendation: ledger helper behavior, retired counters,
receipt recovery and budget authority must cut over together. Independent owner
commits remain inspectable. Deployment still requires draining incompatible
pre-upgrade controllers; this document does not authorize or certify deployment.
Targeted real PostgreSQL evidence includes all five shared-row blocker cases,
replay/rollback, late receipt recovery, scoped control publication and UTC
rollover. The browser source at the pinned baseline times out after three
seconds with the company row held; the candidate completes while the same locks
remain held (95 ms total test including fixture/assertions). This proves removal
of that serialization mechanism, not a hosted latency or CPU gain.
Late receipts preserve original monetary and usage evidence while audited
corrections update effective ledger tokens/provider request lookup as well as
price. Independent ambiguous run receipts retain independent debt. Browser
cumulative growth uses its own persisted occurrence time; retrying the same
receipt retains that time. Rare invoice import retains its old serialized
wrapper for cross-invoice finance deduplication; it is outside the five paid
settlement paths and does not block their ledger commits.
+6 -3
View File
@@ -77,9 +77,12 @@ connection vault, never process environment fallback.
`company_prompt_injection_prevention` retains selections after connection
removal. `prompt_injection_invocations` keeps archival attribution, policy/model,
verdict categories, bounded errors, token usage, provider request ID, and a link
to the cost event. It stores no scanned content. A provider request reserves
applicable budgets under the company accounting lock before dispatch. The lock
is released before network I/O.
to the cost event. It stores no scanned content. A provider request records its own reservation
from a best-effort budget observation. Shared manual-control admission fences
are released before network I/O; settlement uses an independent invocation
transaction without the company writer mutex. Financial changes leave durable
work for asynchronous threshold enforcement. Late receipts correct provisional
interrupted accounting without rerunning the classifier.
Charges use `usageKind: prompt_injection`, biller `openrouter`, and idempotency
key `prompt-injection:<invocation-id>`. Run correlation stays on the invocation;
+5 -4
View File
@@ -4,15 +4,16 @@ Product map: [Costs, budgets, and spending controls](../feature-map/budgets-cost
Feature ID: `budgets-costs`.
Budget admission commits under the company accounting lock. A separate ownership
transaction rechecks scope and capacity immediately before provider handoff;
provider work does not hold the company accounting lock. Company, applicable
Paid admission reads budget exposure without a company accounting writer mutex.
Reservations record best-effort obligations; concurrent admissions can exceed an
observed balance. Final handoff retains shared manual-control and scoped ownership
fences; settlement does not acquire those admission fences. Company, applicable
agent, and applicable project limits include unsettled
observed run usage and reservations without counting either twice. A committed
hard stop prevents fresh provider work immediately; scoped cancellation delivery
runs asynchronously and preserves existing Stop acknowledgement and recovery
requirements. Project enforcement does not pause an agent's unrelated projects.
Hosted browser dispatch reserves a separate scoped hold, rechecks authority at
Hosted browser dispatch records a separate scoped hold, rechecks authority at
the paid POST, and retains unknown charges until terminal accounting. The parent
run reservation does not authorize or replace this separate browser cost cap.
+1 -1
View File
@@ -63,7 +63,7 @@ Automated: [budget service](../../server/src/__tests__/budgets-service.test.ts)
Automated: [scoped budget accounting](../../server/src/__tests__/budget-enforcement-scoping.test.ts) exercises cumulative running usage, reservation capacity, final settlement, project isolation, and concurrent admission. [Budget cancellation](../../server/src/services/heartbeat/budget-cancellation.test.ts) exercises durable scoped cancellation, delayed providers, stale generations, and recovery. These controlled-provider tests establish enforcement semantics, not live provider termination latency.
Automated: [hosted browser accounting](../../server/src/__tests__/browser-use-connection.test.ts) covers separate browser holds, concurrent starts, final dispatch races, unknown POST outcomes, and exact terminal settlement. Browser caps reserve additional capacity alongside the parent run; internal holds are never returned as provider results.
Automated: [hosted browser accounting](../../server/src/__tests__/browser-use-connection.test.ts) covers separate browser holds, concurrent starts, final dispatch races, unknown POST outcomes, and exact terminal settlement. Browser caps record separate estimated obligations alongside the parent run; internal holds are never returned as provider results.
Manual: With disposable budgets and a controlled streaming adapter, admit a run, report cumulative usage across its hard limit, and verify that new work is denied immediately while the existing run receives Stop asynchronously. Delay the Stop acknowledgement and verify that unrelated requests remain responsive. Use separate projects to verify scope isolation. Raise the budget before an unfenced cancellation attempt and verify that newly admitted work survives; retain an already committed Stop until its own acknowledgement. Use the explicit budget benchmark to compare 0, 17, and 1,000 unrelated idle agents.
@@ -0,0 +1,13 @@
CREATE TABLE IF NOT EXISTS "budget_evaluation_work" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"company_id" uuid NOT NULL,
"agent_id" uuid,
"project_id" uuid,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
ALTER TABLE "budget_policies" ADD COLUMN IF NOT EXISTS "evaluation_version" integer DEFAULT 0 NOT NULL;--> statement-breakpoint
DO $$ BEGIN ALTER TABLE "budget_evaluation_work" ADD CONSTRAINT "budget_evaluation_work_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "budget_evaluation_work_company_pending_idx" ON "budget_evaluation_work" USING btree ("company_id","created_at","id");--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "budget_evaluation_work_agent_pending_idx" ON "budget_evaluation_work" USING btree ("company_id","agent_id","created_at");--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "budget_evaluation_work_project_pending_idx" ON "budget_evaluation_work" USING btree ("company_id","project_id","created_at");
@@ -0,0 +1,2 @@
ALTER TABLE "cost_adjustments" ADD COLUMN IF NOT EXISTS "previous_usage_json" jsonb;--> statement-breakpoint
ALTER TABLE "cost_events" ADD COLUMN IF NOT EXISTS "reported_usage_json" jsonb;
@@ -1,6 +1,6 @@
{
"id": "a1fcb4c6-40e7-47a6-b3f2-c43cd9bf332b",
"prevId": "8d9f4abf-57c7-443a-aa8b-3e3fd4f09dae",
"id": "b900b172-2cdc-45c6-a964-27e850bc4c74",
"prevId": "809726a1-762c-4c9d-a7fd-55670f1c856f",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -7171,6 +7171,147 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.budget_evaluation_work": {
"name": "budget_evaluation_work",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"agent_id": {
"name": "agent_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"project_id": {
"name": "project_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"budget_evaluation_work_company_pending_idx": {
"name": "budget_evaluation_work_company_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"budget_evaluation_work_agent_pending_idx": {
"name": "budget_evaluation_work_agent_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "agent_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"budget_evaluation_work_project_pending_idx": {
"name": "budget_evaluation_work_project_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "project_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"budget_evaluation_work_company_id_companies_id_fk": {
"name": "budget_evaluation_work_company_id_companies_id_fk",
"tableFrom": "budget_evaluation_work",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.budget_incidents": {
"name": "budget_incidents",
"schema": "",
@@ -7518,6 +7659,13 @@
"notNull": true,
"default": 0
},
"evaluation_version": {
"name": "evaluation_version",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"created_by_user_id": {
"name": "created_by_user_id",
"type": "text",
@@ -28957,6 +29105,12 @@
"primaryKey": false,
"notNull": true
},
"admitted_project_ids": {
"name": "admitted_project_ids",
"type": "uuid[]",
"primaryKey": false,
"notNull": false
},
"settlement_id": {
"name": "settlement_id",
"type": "uuid",
@@ -29076,6 +29230,10 @@
"execution_admissions_settlement_check": {
"name": "execution_admissions_settlement_check",
"value": "(\"execution_admissions\".\"settlement_id\" is null) = (\"execution_admissions\".\"settled_at\" is null)"
},
"execution_admissions_project_set_check": {
"name": "execution_admissions_project_set_check",
"value": "case when \"execution_admissions\".\"admitted_project_ids\" is null\n then \"execution_admissions\".\"settled_at\" is not null else array_position(\"execution_admissions\".\"admitted_project_ids\", null) is null and cardinality(\"execution_admissions\".\"admitted_project_ids\") <= 100 end"
}
},
"isRLSEnabled": false
@@ -30320,6 +30478,22 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_snapshots_imported_children_idx": {
"name": "execution_workspace_snapshots_imported_children_idx",
"columns": [
{
"expression": "(\"metadata\"->'importedWorkspaceChildren') jsonb_path_ops",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"execution_workspace_snapshots\".\"metadata\" ? 'importedWorkspaceChildren'",
"concurrently": false,
"method": "gin",
"with": {}
}
},
"foreignKeys": {},
@@ -44000,6 +44174,133 @@
},
"isRLSEnabled": false
},
"public.native_harness_backups": {
"name": "native_harness_backups",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"scope_digest": {
"name": "scope_digest",
"type": "text",
"primaryKey": false,
"notNull": true
},
"manifest_sha256": {
"name": "manifest_sha256",
"type": "text",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"reference": {
"name": "reference",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"native_harness_backups_current_idx": {
"name": "native_harness_backups_current_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "scope_digest",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"native_harness_backups\".\"status\" = 'current'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_harness_backups_scope_idx": {
"name": "native_harness_backups_scope_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "scope_digest",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "manifest_sha256",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.native_run_finalizations": {
"name": "native_run_finalizations",
"schema": "",
@@ -1,6 +1,6 @@
{
"id": "8d9f4abf-57c7-443a-aa8b-3e3fd4f09dae",
"prevId": "cb093c1d-3e54-4ab2-8009-9c243097224c",
"id": "9fcf29d1-a9a6-4ead-8d43-7e087a2b2b24",
"prevId": "b900b172-2cdc-45c6-a964-27e850bc4c74",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -713,6 +713,12 @@
"notNull": true,
"default": "'{}'::jsonb"
},
"previous_usage_json": {
"name": "previous_usage_json",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"reason": {
"name": "reason",
"type": "text",
@@ -7171,6 +7177,147 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.budget_evaluation_work": {
"name": "budget_evaluation_work",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"agent_id": {
"name": "agent_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"project_id": {
"name": "project_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"budget_evaluation_work_company_pending_idx": {
"name": "budget_evaluation_work_company_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"budget_evaluation_work_agent_pending_idx": {
"name": "budget_evaluation_work_agent_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "agent_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"budget_evaluation_work_project_pending_idx": {
"name": "budget_evaluation_work_project_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "project_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"budget_evaluation_work_company_id_companies_id_fk": {
"name": "budget_evaluation_work_company_id_companies_id_fk",
"tableFrom": "budget_evaluation_work",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.budget_incidents": {
"name": "budget_incidents",
"schema": "",
@@ -7518,6 +7665,13 @@
"notNull": true,
"default": 0
},
"evaluation_version": {
"name": "evaluation_version",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"created_by_user_id": {
"name": "created_by_user_id",
"type": "text",
@@ -21885,6 +22039,12 @@
"primaryKey": false,
"notNull": false
},
"reported_usage_json": {
"name": "reported_usage_json",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"pricing_provenance": {
"name": "pricing_provenance",
"type": "jsonb",
@@ -28825,6 +28985,337 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.execution_admission_project_charges": {
"name": "execution_admission_project_charges",
"schema": "",
"columns": {
"admission_id": {
"name": "admission_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"project_id": {
"name": "project_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
}
},
"indexes": {
"execution_admission_project_charges_project_idx": {
"name": "execution_admission_project_charges_project_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "project_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"execution_admission_project_charges_company_admission_fk": {
"name": "execution_admission_project_charges_company_admission_fk",
"tableFrom": "execution_admission_project_charges",
"tableTo": "execution_admissions",
"columnsFrom": [
"company_id",
"admission_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"execution_admission_project_charges_company_project_fk": {
"name": "execution_admission_project_charges_company_project_fk",
"tableFrom": "execution_admission_project_charges",
"tableTo": "projects",
"columnsFrom": [
"company_id",
"project_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "no action",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {
"execution_admission_project_charges_admission_id_project_id_pk": {
"name": "execution_admission_project_charges_admission_id_project_id_pk",
"columns": [
"admission_id",
"project_id"
]
}
},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.execution_admissions": {
"name": "execution_admissions",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"attempt": {
"name": "attempt",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"primary_project_id": {
"name": "primary_project_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"context_revision": {
"name": "context_revision",
"type": "text",
"primaryKey": false,
"notNull": true
},
"admitted_project_ids": {
"name": "admitted_project_ids",
"type": "uuid[]",
"primaryKey": false,
"notNull": false
},
"settlement_id": {
"name": "settlement_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"settled_at": {
"name": "settled_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"execution_admissions_company_id_uq": {
"name": "execution_admissions_company_id_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_admissions_run_attempt_uq": {
"name": "execution_admissions_run_attempt_uq",
"columns": [
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "attempt",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_admissions_live_run_uq": {
"name": "execution_admissions_live_run_uq",
"columns": [
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"execution_admissions\".\"settled_at\" is null",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"execution_admissions_company_id_companies_id_fk": {
"name": "execution_admissions_company_id_companies_id_fk",
"tableFrom": "execution_admissions",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "no action",
"onUpdate": "no action"
},
"execution_admissions_run_id_heartbeat_runs_id_fk": {
"name": "execution_admissions_run_id_heartbeat_runs_id_fk",
"tableFrom": "execution_admissions",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "no action",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"execution_admissions_attempt_check": {
"name": "execution_admissions_attempt_check",
"value": "\"execution_admissions\".\"attempt\" > 0"
},
"execution_admissions_settlement_check": {
"name": "execution_admissions_settlement_check",
"value": "(\"execution_admissions\".\"settlement_id\" is null) = (\"execution_admissions\".\"settled_at\" is null)"
},
"execution_admissions_project_set_check": {
"name": "execution_admissions_project_set_check",
"value": "case when \"execution_admissions\".\"admitted_project_ids\" is null\n then \"execution_admissions\".\"settled_at\" is not null else array_position(\"execution_admissions\".\"admitted_project_ids\", null) is null and cardinality(\"execution_admissions\".\"admitted_project_ids\") <= 100 end"
}
},
"isRLSEnabled": false
},
"public.project_task_execution_policies": {
"name": "project_task_execution_policies",
"schema": "",
"columns": {
"project_id": {
"name": "project_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"max_concurrent_tasks": {
"name": "max_concurrent_tasks",
"type": "integer",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {
"project_task_execution_policies_company_project_fk": {
"name": "project_task_execution_policies_company_project_fk",
"tableFrom": "project_task_execution_policies",
"tableTo": "projects",
"columnsFrom": [
"company_id",
"project_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"project_task_execution_policies_maximum_check": {
"name": "project_task_execution_policies_maximum_check",
"value": "\"project_task_execution_policies\".\"max_concurrent_tasks\" is null or \"project_task_execution_policies\".\"max_concurrent_tasks\" > 0"
}
},
"isRLSEnabled": false
},
"public.execution_workspace_repositories": {
"name": "execution_workspace_repositories",
"schema": "",
@@ -29999,6 +30490,22 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_snapshots_imported_children_idx": {
"name": "execution_workspace_snapshots_imported_children_idx",
"columns": [
{
"expression": "(\"metadata\"->'importedWorkspaceChildren') jsonb_path_ops",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"execution_workspace_snapshots\".\"metadata\" ? 'importedWorkspaceChildren'",
"concurrently": false,
"method": "gin",
"with": {}
}
},
"foreignKeys": {},
@@ -43679,6 +44186,133 @@
},
"isRLSEnabled": false
},
"public.native_harness_backups": {
"name": "native_harness_backups",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"scope_digest": {
"name": "scope_digest",
"type": "text",
"primaryKey": false,
"notNull": true
},
"manifest_sha256": {
"name": "manifest_sha256",
"type": "text",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"reference": {
"name": "reference",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"native_harness_backups_current_idx": {
"name": "native_harness_backups_current_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "scope_digest",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"native_harness_backups\".\"status\" = 'current'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_harness_backups_scope_idx": {
"name": "native_harness_backups_scope_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "scope_digest",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "manifest_sha256",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.native_run_finalizations": {
"name": "native_run_finalizations",
"schema": "",
+14
View File
@@ -2997,6 +2997,20 @@
"when": 1793760539457,
"tag": "0429_high_freak",
"breakpoints": true
},
{
"idx": 430,
"version": "7",
"when": 1793797974795,
"tag": "0430_regular_red_skull",
"breakpoints": true
},
{
"idx": 431,
"version": "7",
"when": 1793798006496,
"tag": "0431_magenta_vanisher",
"breakpoints": true
}
]
}
+1
View File
@@ -116,6 +116,7 @@ export const costAdjustments = pgTable("cost_adjustments", {
correctedCents: numeric("corrected_cents", { precision: 24, scale: 7 }).notNull(),
previousStatus: text("previous_status").notNull(),
previousPricing: jsonb("previous_pricing").$type<Record<string, unknown>>().notNull().default({}),
previousUsageJson: jsonb("previous_usage_json").$type<Record<string, unknown>>(),
reason: text("reason").notNull(),
actorId: text("actor_id").notNull(),
pricing: jsonb("pricing").$type<Record<string, unknown>>().notNull().default({}),
@@ -0,0 +1,15 @@
import { index, pgTable, timestamp, uuid } from "drizzle-orm/pg-core";
import { companies } from "./companies.js";
/** Independent durable inputs. No producer updates a shared company cursor. */
export const budgetEvaluationWork = pgTable("budget_evaluation_work", {
id: uuid("id").primaryKey().defaultRandom(),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
agentId: uuid("agent_id"),
projectId: uuid("project_id"),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
}, table => ({
companyPending: index("budget_evaluation_work_company_pending_idx").on(table.companyId, table.createdAt, table.id),
agentPending: index("budget_evaluation_work_agent_pending_idx").on(table.companyId, table.agentId, table.createdAt),
projectPending: index("budget_evaluation_work_project_pending_idx").on(table.companyId, table.projectId, table.createdAt),
}));
@@ -20,6 +20,7 @@ export const budgetPolicies = pgTable(
isActive: boolean("is_active").notNull().default(true),
enforcementVersion: integer("enforcement_version").notNull().default(0),
enforcementDeliveredVersion: integer("enforcement_delivered_version").notNull().default(0),
evaluationVersion: integer("evaluation_version").notNull().default(0),
createdByUserId: text("created_by_user_id"),
updatedByUserId: text("updated_by_user_id"),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
+1
View File
@@ -26,6 +26,7 @@ export const costEvents = pgTable(
receiptHash: text("receipt_hash"),
providerRequestId: text("provider_request_id"),
reportedCostCents: numeric("reported_cost_cents", { precision: 24, scale: 7 }),
reportedUsageJson: jsonb("reported_usage_json").$type<Record<string, unknown>>(),
pricingProvenance: jsonb("pricing_provenance").$type<Record<string, unknown>>(),
provider: text("provider").notNull(),
biller: text("biller").notNull().default("unknown"),
+1
View File
@@ -277,3 +277,4 @@ export { userWorkspacePreferences } from "./user_workspace_preferences.js";
export { issueThreadInteractionIdempotencyKeys } from "./issue_thread_interaction_idempotency_keys.js";
export { projectTaskExecutionPolicies, executionAdmissions, executionAdmissionProjectCharges } from "./execution_admissions.js";
export { budgetEvaluationWork } from "./budget_evaluation_work.js";
+1
View File
@@ -103,6 +103,7 @@ export interface CostAdjustment {
correctedCents: string;
previousStatus: string;
previousPricing: Record<string, unknown>;
previousUsageJson?: Record<string, unknown> | null;
pricing: Record<string, unknown>;
reason: string;
actorId: string;
+2
View File
@@ -24,6 +24,8 @@ export interface CostEvent {
outputTokens: number;
costCents: number;
costCentsExact?: string;
/** Original usage retained when a late authoritative receipt corrects a placeholder. */
reportedUsageJson?: Record<string, unknown> | null;
occurredAt: Date;
createdAt: Date;
}
@@ -8,7 +8,7 @@ import { subscribeDatabaseWork, authUsers, companyMemberships, agents, approvals
import { attentionRetentionService, registerAttentionRetentionWork } from "../services/attention-retention.js";
import { workspaceOperationService } from "../services/workspace-operations.js";
import * as workspaceOperationLogs from "../services/workspace-operation-log-store.js";
import { budgetService } from "../services/budgets.js";
import { budgetService, flushBudgetEvaluationWork } from "../services/budgets.js";
import { writeExecutionControl } from "../services/execution-control-notifications.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { attentionService } from "../services/attention.js";
@@ -912,6 +912,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000;
billingType: "metered_api", model: "fixture", inputTokens: 100, outputTokens: 20,
costCents: 90, occurredAt: new Date() }).returning();
await service.evaluateCostEvent(event);
await flushBudgetEvaluationWork(db, { companyId });
await settle(); expect(coordinator!.nextWakeAt()).not.toBeNull();
const [incident] = await db.select().from(budgetIncidents).where(eq(budgetIncidents.companyId, companyId));
expect(incident).toMatchObject({ status: "open", thresholdType: "soft" });
@@ -1,4 +1,5 @@
import { budgetService, computeObservedSpend } from "../services/budgets.js";
import { budgetService, computeObservedSpend, flushBudgetEvaluationWork } from "../services/budgets.js";
import * as costAccounting from "../services/costs.js";
import { buildPaperclipRuntimeMcpServers } from "../services/heartbeat.js";
import { resolveNativeRuntimeMcpSnapshot } from "../services/native-runtime/runtime-context.js";
import express from "express";
@@ -289,6 +290,7 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" };
.set({ nextPollAt: new Date(0) })
.where(eq(browserUseSessions.companyId, company.id));
await service.sweep(connection.connectionId);
await flushBudgetEvaluationWork(db, { companyId: company.id });
};
return {
company,
@@ -1399,7 +1401,7 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" };
)[0].status,
).toBe("closed");
});
it("serializes browser holds with parent reservations while provider work is delayed", async () => {
it("counts browser holds with parent reservations while provider work is delayed", async () => {
const f = await fixture();
const policy = await budgetService(db).upsertPolicy(f.company.id,
{ scopeType: "company", scopeId: f.company.id, amount: 100, hardStopEnabled: true }, actor.actorId);
@@ -1422,6 +1424,71 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" };
expect(await db.select({ id: companies.id }).from(companies).where(eq(companies.id, f.company.id))).toHaveLength(1);
} finally { release(); await first; }
});
it("keeps manual Stop authoritative when a paid create response arrives later", async () => {
const f = await fixture();
let release!: () => void, entered!: () => void;
const accepted = new Promise<void>(resolve => { entered = resolve; });
const barrier = new Promise<void>(resolve => { release = resolve; });
f.delayCreate(async () => { entered(); await barrier; });
const starting = f.service.execute(f.binding, f.grant, randomUUID(), "browser_start", { task: "Stop during dispatch" });
await accepted;
try {
const [session] = await db.select().from(browserUseSessions).where(eq(browserUseSessions.companyId, f.company.id));
await f.service.control(session, "end");
} finally { release(); }
await expect(starting).resolves.toMatchObject({ status: "stopping" });
await f.tick();
expect((await db.select().from(browserUseSessions).where(eq(browserUseSessions.companyId, f.company.id)))[0].status).toBe("closed");
expect(f.request.mock.calls.filter(([url]) => url.endsWith("/cancel"))).toHaveLength(1);
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.company.id))).toHaveLength(1);
});
it("replays simultaneous invocation retries without submitting another paid create", async () => {
const f = await fixture();
const invocation = randomUUID();
const another = browserUseService(db, f.request);
const results = await Promise.all([f.service, another].map(service =>
service.execute(f.binding, f.grant, invocation, "browser_start", { task: "One invocation" })));
expect((results[0] as { sessionId: string }).sessionId).toBe((results[1] as { sessionId: string }).sessionId);
expect(await db.select().from(browserUseRuns).where(eq(browserUseRuns.invocationId, invocation))).toHaveLength(1);
expect(await db.select().from(browserUseSessions).where(eq(browserUseSessions.companyId, f.company.id))).toHaveLength(1);
expect(f.request.mock.calls.filter(([url, init]) => url.endsWith("/runs") && init.method === "POST")).toHaveLength(1);
});
it.each([
{ scope: "company", phase: "initial" }, { scope: "project", phase: "initial" },
{ scope: "company", phase: "handoff" }, { scope: "project", phase: "handoff" },
])("honors a $scope pause committed after the $phase admission snapshot", async ({ scope, phase }) => {
const f = await fixture();
const [project] = await db.insert(projects).values({ companyId: f.company.id, name: "Browser authority" }).returning();
await db.update(issues).set({ projectId: project.id }).where(eq(issues.id, f.issue.id));
const invocation = randomUUID();
const transaction = db.transaction.bind(db);
let injected = false;
const race = vi.spyOn(db, "transaction").mockImplementation(async (callback, config) => {
const [run] = await db.select().from(browserUseRuns).where(eq(browserUseRuns.invocationId, invocation));
const intercept = !injected && config?.isolationLevel === "repeatable read" && config.accessMode !== "read only"
&& (phase === "initial" ? !run : run?.status === "creating");
return transaction(async tx => {
if (intercept) {
injected = true;
// Establish the real RR snapshot before the independent operator
// transaction commits. An unfenced read would keep seeing active.
await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, f.company.id));
if (scope === "company") await db.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() })
.where(eq(companies.id, f.company.id));
else await db.update(projects).set({ pauseReason: "manual", pausedAt: new Date() }).where(eq(projects.id, project.id));
}
return callback(tx);
}, config);
});
try {
await expect(f.service.execute(f.binding, f.grant, invocation, "browser_start", { task: "Stopped authority" })).rejects.toThrow();
} finally { race.mockRestore(); }
expect(injected).toBe(true);
expect(f.request.mock.calls.filter(([url, init]) => url.endsWith("/runs") && init.method === "POST")).toHaveLength(0);
const rows = await db.select().from(browserUseRuns).where(eq(browserUseRuns.invocationId, invocation));
if (phase === "initial") expect(rows).toHaveLength(0);
else expect(rows[0]).toMatchObject({ status: "failed", eventsDrained: 1, result: { paperclipBudgetHold: { state: "released" } } });
});
it.each(["reduction", "new_policy", "config", "charge", "cancel", "session_end", "policy_deny"])("rechecks %s at the actual provider handoff", async race => {
const f = await fixture();
if (race === "reduction" || race === "charge") await budgetService(db).upsertPolicy(f.company.id,
@@ -1598,6 +1665,65 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" };
expect(JSON.stringify(replay)).not.toContain("paperclipBudgetHold");
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.company.id))).toHaveLength(1);
});
it("settles while company, agent and policy rows are held by unrelated writers", async () => {
const f = await fixture();
const policy = await budgetService(db).upsertPolicy(f.company.id,
{ scopeType: "company", scopeId: f.company.id, amount: 100, hardStopEnabled: true }, actor.actorId);
await f.service.execute(f.binding, f.grant, randomUUID(), "browser_start", { task: "Independent settlement" });
f.complete();
await db.update(browserUseSessions).set({ nextPollAt: new Date(0) }).where(eq(browserUseSessions.companyId, f.company.id));
let release!: () => void, locked!: () => void;
const held = new Promise<void>(resolve => { locked = resolve; });
const barrier = new Promise<void>(resolve => { release = resolve; });
const blocker = db.transaction(async tx => {
await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, f.company.id)).for("no key update");
await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, f.agent.id)).for("no key update");
await tx.select({ id: budgetPolicies.id }).from(budgetPolicies).where(eq(budgetPolicies.id, policy.policyId)).for("no key update");
locked();
await barrier;
});
await held;
let timeout: ReturnType<typeof setTimeout> | undefined;
const settlement = f.service.sweep(f.connection.connectionId);
try {
await Promise.race([settlement, new Promise<never>((_resolve, reject) => {
timeout = setTimeout(() => reject(new Error("Browser settlement waited for a shared accounting row")), 3000);
})]);
const [event] = await db.select().from(costEvents).where(eq(costEvents.companyId, f.company.id));
expect(event).toMatchObject({ provider: "browser-use-cloud", costCents: 15 });
const [run] = await db.select().from(browserUseRuns).where(eq(browserUseRuns.companyId, f.company.id));
expect(run).toMatchObject({ eventsDrained: 1, result: { paperclipBudgetHold: { state: "settled" } } });
} finally {
if (timeout) clearTimeout(timeout);
release();
await blocker;
await settlement;
}
});
it("replays a captured terminal receipt after ledger failure without refetching or moving its attribution", async () => {
const f = await fixture();
const [project] = await db.insert(projects).values({ companyId: f.company.id, name: "Original browser project" }).returning();
const [laterProject] = await db.insert(projects).values({ companyId: f.company.id, name: "Later task project" }).returning();
await db.update(issues).set({ projectId: project.id }).where(eq(issues.id, f.issue.id));
const invocation = randomUUID();
await f.service.execute(f.binding, f.grant, invocation, "browser_start", { task: "Capture once" });
f.usage("0.150000001"); f.complete();
const failedLedger = vi.spyOn(costAccounting, "createCostEventInTransaction").mockRejectedValueOnce(new Error("Accounting unavailable"));
try { await f.tick(); } finally { failedLedger.mockRestore(); }
const [pending] = await db.select().from(browserUseRuns).where(eq(browserUseRuns.invocationId, invocation));
const receipt = (pending.result as { paperclipBudgetHold: { receipt: { occurredAt: string } } }).paperclipBudgetHold.receipt;
expect(pending.eventsDrained).toBe(0);
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.company.id))).toHaveLength(0);
await db.update(issues).set({ projectId: laterProject.id }).where(eq(issues.id, f.issue.id));
f.usage("0.99"); f.request.mockClear();
await f.tick();
const [event] = await db.select({ projectId: costEvents.projectId, occurredAt: costEvents.occurredAt,
cents: sql<string>`${costEvents.costCents}::text` }).from(costEvents).where(eq(costEvents.companyId, f.company.id));
expect(event).toMatchObject({ projectId: project.id, cents: "15.0000001" });
expect(event.occurredAt.toISOString()).toBe(receipt.occurredAt);
expect(f.request.mock.calls.some(([url]) => /\/runs\/[^/]+$/.test(new URL(url).pathname))).toBe(false);
expect(await db.select().from(financeEvents).where(eq(financeEvents.companyId, f.company.id))).toHaveLength(1);
});
it("retains exact cumulative accounting when an older browser row has no hold projection", async () => {
const f = await fixture();
const invocation = randomUUID();
@@ -1611,6 +1737,39 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" };
expect(total.cents).toBe("15.0000002");
expect((await db.select().from(browserUseRuns).where(eq(browserUseRuns.id, run.id)))[0].result)
.toMatchObject({ paperclipBudgetHold: { state: "settled", accountedCentsExact: "15.0000002" } });
await db.update(browserUseRuns).set({ eventsDrained: 0 }).where(eq(browserUseRuns.id, run.id));
f.usage("0.10"); await f.tick();
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.company.id))).toHaveLength(2);
expect((await db.select().from(browserUseRuns).where(eq(browserUseRuns.id, run.id)))[0].result)
.toMatchObject({ paperclipBudgetHold: { state: "settled", accountedCentsExact: "15.0000002" } });
});
it("dates a new cumulative delta in its capture month and keeps that receipt's retry time stable", async () => {
const f = await fixture();
const before = new Date("2026-11-30T23:59:59.900Z"), after = new Date("2026-12-01T00:00:00.100Z");
vi.useFakeTimers({ toFake: ["Date"] });
try {
vi.setSystemTime(before);
const invocation = randomUUID();
await f.service.execute(f.binding, f.grant, invocation, "browser_start", { task: "Cumulative usage across UTC rollover" });
f.usage("0.15"); f.complete(); await f.tick();
const [run] = await db.select().from(browserUseRuns).where(eq(browserUseRuns.invocationId, invocation));
await db.update(browserUseRuns).set({ eventsDrained: 0 }).where(eq(browserUseRuns.id, run.id));
vi.setSystemTime(after);
f.usage("0.20");
const failed = vi.spyOn(costAccounting, "createCostEventInTransaction").mockRejectedValueOnce(new Error("Retry this receipt"));
try { await f.tick(); } finally { failed.mockRestore(); }
const [captured] = await db.select().from(browserUseRuns).where(eq(browserUseRuns.id, run.id));
expect(captured).toMatchObject({ eventsDrained: 0, result: { paperclipBudgetHold: {
accountedCentsExact: "15.0000000", receipt: { totalCentsExact: "20.0000000", occurredAt: after.toISOString() },
} } });
vi.setSystemTime(new Date("2026-12-01T00:00:05.100Z"));
f.usage("0.99");
await f.tick(); await f.tick();
const events = await db.select({ cents: sql<string>`${costEvents.costCents}::text`, occurredAt: costEvents.occurredAt })
.from(costEvents).where(eq(costEvents.companyId, f.company.id)).orderBy(costEvents.occurredAt);
expect(events).toEqual([{ cents: "15.0000000", occurredAt: before }, { cents: "5.0000000", occurredAt: after }]);
expect(await db.select().from(financeEvents).where(eq(financeEvents.companyId, f.company.id))).toHaveLength(2);
} finally { vi.useRealTimers(); }
});
it("caps paid work at the remaining hard budget and enforces reported spend", async () => {
const f = await fixture();
@@ -1648,7 +1807,10 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" };
await f.tick();
expect(
(await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0],
).toMatchObject({ status: "paused", spentMonthlyCents: 15 });
).toMatchObject({ status: "paused" });
const [charged] = await db.select({ cents: sql<string>`sum(${costEvents.costCents})::text` }).from(costEvents)
.where(eq(costEvents.companyId, f.company.id));
expect(charged.cents).toBe("15.0000000");
expect(
(
await db
@@ -0,0 +1,126 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, budgetEvaluationWork, budgetIncidents, budgetPolicies, companies, createDb } from "@paperclipai/db";
import { budgetService, budgetServiceInTransaction, flushBudgetEvaluationWork } from "../services/budgets.js";
import { costService } from "../services/costs.js";
import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("budget publication versus resource controls", () => {
let temp: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
temp = await startEmbeddedPostgresTestDatabase("paperclip-budget-controls-");
db = createDb(temp.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterEach(() => { vi.useRealTimers(); vi.restoreAllMocks(); });
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await temp?.cleanup();
});
async function fixture() {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Budget controls", issuePrefix: `C${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", role: "engineer", status: "active", adapterType: "process" });
const policy = await budgetService(db).upsertPolicy(companyId, { scopeType: "company", scopeId: companyId, amount: 100 }, null);
await flushBudgetEvaluationWork(db, { companyId });
return { companyId, agentId, policyId: policy.policyId };
}
async function charge(f: { companyId: string; agentId: string }, costCents: number, occurredAt = new Date()) {
return costService(db).createEvent(f.companyId, { agentId: f.agentId, provider: "openai", model: "fixture", costCents,
occurredAt, idempotencyKey: randomUUID() });
}
const pending = (companyId: string) => db.select().from(budgetEvaluationWork).where(eq(budgetEvaluationWork.companyId, companyId));
async function bounded<T>(promise: Promise<T>): Promise<T> {
let timer: ReturnType<typeof setTimeout> | undefined;
try {
return await Promise.race([promise, new Promise<never>((_resolve, reject) => {
timer = setTimeout(() => reject(new Error("Budget evaluator waited for a resource while holding its policy")), 2000);
})]);
} finally { clearTimeout(timer); }
}
it("relinquishes its policy fence so a company-first control edit cannot deadlock or lose", async () => {
const f = await fixture();
await charge(f, 120);
let release!: () => void, locked!: () => void;
const ready = new Promise<void>(resolve => { locked = resolve; });
const gate = new Promise<void>(resolve => { release = resolve; });
// Retain the existing company/configuration writer's company -> policy
// order, while the evaluator starts from the opposite end of the race.
const control = db.transaction(async tx => {
await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, f.companyId)).for("no key update");
locked(); await gate;
await budgetServiceInTransaction(tx as unknown as Parameters<typeof budgetServiceInTransaction>[0]).upsertPolicy(f.companyId,
{ scopeType: "company", scopeId: f.companyId, amount: 1000 }, "operator");
await tx.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() }).where(eq(companies.id, f.companyId));
});
await ready;
const evaluation = flushBudgetEvaluationWork(db, f).then(value => ({ value }), error => ({ error }));
try {
expect(await bounded(evaluation)).toMatchObject({ value: 0 });
expect((await pending(f.companyId)).length).toBeGreaterThan(0);
} finally { release(); await control; await evaluation; }
await flushBudgetEvaluationWork(db, f);
expect((await db.select().from(budgetPolicies).where(eq(budgetPolicies.id, f.policyId)))[0].amount).toBe(1000);
expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0])
.toMatchObject({ status: "paused", pauseReason: "manual" });
expect(await db.select().from(budgetIncidents).where(eq(budgetIncidents.companyId, f.companyId))).toHaveLength(0);
expect(await pending(f.companyId)).toHaveLength(0);
});
it("rolls back old-window publication and retains its inputs when UTC rollover interrupts every attempt", async () => {
const before = new Date("2026-11-30T23:59:59.900Z"), after = new Date("2026-12-01T00:00:00.100Z");
vi.useFakeTimers({ toFake: ["Date"] });
vi.setSystemTime(before);
const f = await fixture();
await charge(f, 120);
const inputIds = (await pending(f.companyId)).map(row => row.id);
const [original] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.id, f.policyId));
const transaction = db.transaction.bind(db);
let crossed = 0;
const boundary = vi.spyOn(db, "transaction").mockImplementation((callback, options) => {
vi.setSystemTime(before);
return transaction(async tx => {
const execute = tx.execute.bind(tx);
// The exposure statement executes against real PostgreSQL; advance only
// the application clock after reading that old-window snapshot.
tx.execute = (async (...args: Parameters<typeof tx.execute>) => {
const result = await execute(...args);
if (Date.now() === before.getTime()) { crossed++; vi.setSystemTime(after); }
return result;
}) as typeof tx.execute;
return callback(tx);
}, options);
});
try {
expect(await flushBudgetEvaluationWork(db, f)).toBe(0);
} finally { boundary.mockRestore(); }
expect(crossed).toBeGreaterThan(0);
expect((await pending(f.companyId)).map(row => row.id)).toEqual(inputIds);
expect((await db.select().from(budgetPolicies).where(eq(budgetPolicies.id, f.policyId)))[0].evaluationVersion).toBe(original.evaluationVersion);
expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0].pauseReason).toBeNull();
expect(await db.select().from(budgetIncidents).where(eq(budgetIncidents.companyId, f.companyId))).toHaveLength(0);
vi.setSystemTime(after);
await flushBudgetEvaluationWork(db, f);
expect(await pending(f.companyId)).toHaveLength(0);
expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0].pauseReason).toBeNull();
});
it("reads the new UTC month through a budget service retained across rollover", async () => {
vi.useFakeTimers({ toFake: ["Date"] });
vi.setSystemTime(new Date("2026-11-30T12:00:00Z"));
const f = await fixture();
const service = budgetService(db);
await charge(f, 20, new Date("2026-11-15T12:00:00Z"));
await charge(f, 30, new Date("2026-12-15T12:00:00Z"));
const november = (await service.overview(f.companyId)).policies[0];
expect(november).toMatchObject({ observedAmountExact: "20.0000000" });
expect(november.windowStart.toISOString()).toBe("2026-11-01T00:00:00.000Z");
vi.setSystemTime(new Date("2026-12-20T12:00:00Z"));
const december = (await service.overview(f.companyId)).policies[0];
expect(december).toMatchObject({ observedAmountExact: "30.0000000" });
expect(december.windowStart.toISOString()).toBe("2026-12-01T00:00:00.000Z");
});
});
@@ -3,10 +3,10 @@ import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import { agents, budgetPolicies, budgetReservations, companies, costEvents, createDb, heartbeatRuns, projects, type Db } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { budgetService, computeObservedSpend } from "../services/budgets.js";
import { budgetService, computeObservedSpend, flushBudgetEvaluationWork } from "../services/budgets.js";
import { reserveRunBudget } from "../services/budget-reservations.js";
import { accountRunCost } from "../services/run-cost-accounting.js";
import { persistUsageReceipt, type UsageReceiptEnvelope } from "../services/usage-receipts.js";
import { accountRunCost as accountRun } from "../services/run-cost-accounting.js";
import { persistUsageReceipt as persistReceipt, type UsageReceiptEnvelope } from "../services/usage-receipts.js";
import { configureAgentLifecycle } from "../services/agent-lifecycle.js";
const support = await getEmbeddedPostgresTestSupport();
@@ -23,6 +23,17 @@ const support = await getEmbeddedPostgresTestSupport();
}); });
afterAll(async () => { await database?.cleanup(); });
// The explicit worker turn separates receipt commit from budget transitions.
async function persistUsageReceipt(...args: Parameters<typeof persistReceipt>) {
const result = await persistReceipt(...args);
await flushBudgetEvaluationWork(db);
return result;
}
async function accountRunCost(...args: Parameters<typeof accountRun>) {
const result = await accountRun(...args);
await flushBudgetEvaluationWork(db);
return result;
}
async function fixture(options: { scope?: "company" | "agent" | "project"; amount?: number; reservationCents?: string } = {}) {
const companyId = randomUUID(), agentId = randomUUID(), otherAgentId = randomUUID();
const projectId = randomUUID(), runId = randomUUID(), sourceId = randomUUID();
@@ -101,15 +112,15 @@ const support = await getEmbeddedPostgresTestSupport();
await expect(reserveRunBudget(db, f.companyId, await f.queuedRun(), f.projectId)).rejects.toThrow("reserved");
});
it("serializes competing dispatch reservations without overspending available capacity", async () => {
it("retains each best-effort competing dispatch reservation", async () => {
const f = await fixture({ reservationCents: "60" });
const first = await f.queuedRun(), second = await f.queuedRun();
const outcomes = await Promise.allSettled([
reserveRunBudget(db, f.companyId, first, f.projectId), reserveRunBudget(db, f.companyId, second, f.projectId),
]);
expect(outcomes.filter(result => result.status === "fulfilled")).toHaveLength(1);
expect(outcomes.filter(result => result.status === "rejected")).toHaveLength(1);
expect((await f.exposure()).effectiveHeldCents).toBe("60.0000000");
const admitted = outcomes.filter(result => result.status === "fulfilled").length;
expect(admitted).toBeGreaterThanOrEqual(1);
expect((await f.exposure()).effectiveHeldCents).toBe(`${admitted * 60}.0000000`);
});
it("commits a budget release while dispatch waits for asynchronous lifecycle readiness", async () => {
@@ -122,6 +133,8 @@ const support = await getEmbeddedPostgresTestSupport();
await worker.process(f.agentId);
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0].lifecycleState).toBe("paused");
await db.update(budgetPolicies).set({ amount: 200 }).where(eq(budgetPolicies.id, f.policy.id));
await budgetService(db).evaluateAffectedScopes(f.companyId, f.agentId, f.projectId);
await flushBudgetEvaluationWork(db);
const nextRunId = await f.queuedRun();
await expect(reserveRunBudget(db, f.companyId, nextRunId, f.projectId)).rejects.toThrow("lifecycle");
const [resuming] = await db.select().from(agents).where(eq(agents.id, f.agentId));
@@ -237,6 +250,8 @@ const support = await getEmbeddedPostgresTestSupport();
const f = await fixture();
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
expect(await budgetService(db).getInvocationBlock(f.companyId, f.agentId, { projectId: f.projectId })).not.toBeNull();
await budgetService(db).evaluateAffectedScopes(f.companyId, f.agentId, f.projectId);
await flushBudgetEvaluationWork(db);
const [held] = await db.select().from(agents).where(eq(agents.id, f.agentId));
expect(held.lifecycleHolds).toContain("budget");
await db.update(heartbeatRuns).set(kind === "pre-provider release"
@@ -7,14 +7,16 @@ import { publishLiveEvent } from "../services/live-events.js";
import { registerBudgetEnforcementWork, registerCostAccountingWork, wakeCostAccountingWork } from "../services/budget-enforcement-worker.js";
const work = vi.hoisted(() => ({
deliver: vi.fn(), pending: vi.fn(), next: vi.fn(), reconcile: vi.fn(), accounting: vi.fn(),
deliver: vi.fn(), pending: vi.fn(), next: vi.fn(), reconcile: vi.fn(), accounting: vi.fn(), evaluate: vi.fn(), evaluationPending: vi.fn(),
}));
vi.mock("../services/budgets.js", () => ({
deliverBudgetEnforcement: work.deliver,
hasPendingBudgetEnforcement: work.pending,
nextBudgetReconciliationAt: work.next,
reconcileDueBudgetPolicies: work.reconcile,
flushBudgetEvaluationWorkPage: work.evaluate,
}));
vi.mock("../services/budget-evaluation-work.js", () => ({ hasPendingBudgetEvaluation: work.evaluationPending }));
vi.mock("../services/run-cost-accounting.js", () => ({ readCostAccountingWork: work.accounting }));
const coordinators: ReturnType<typeof createDeliveryWorkCoordinator>[] = [];
@@ -27,6 +29,8 @@ beforeEach(() => {
work.next.mockResolvedValue(null);
work.reconcile.mockResolvedValue({ nextCursor: null });
work.accounting.mockResolvedValue({ pending: false, nextRunAt: null });
work.evaluate.mockResolvedValue({ consumed: 0, nextCursor: null });
work.evaluationPending.mockResolvedValue(false);
});
afterEach(async () => {
for (const stop of subscriptions.splice(0)) stop();
@@ -45,6 +49,28 @@ function setup() {
}
describe("budget accounting delivery", () => {
it("continues cancellation delivery and retries a failed startup reconciliation", async () => {
const s = setup();
work.reconcile.mockResolvedValueOnce({ nextCursor: null, hadFailures: true });
await registerBudgetEnforcementWork(s.db, s.coordinator, {}).ready;
expect(work.deliver).toHaveBeenCalledOnce();
await vi.advanceTimersByTimeAsync(1000);
expect(work.reconcile).toHaveBeenLastCalledWith(s.db, { startup: true });
expect(work.deliver).toHaveBeenCalledTimes(2);
expect(s.coordinator.nextWakeAt()).toBeNull();
});
it("rotates evaluation pages past failed companies before revisiting the first page", async () => {
const s = setup();
work.evaluate.mockResolvedValueOnce({ consumed: 0, nextCursor: "failed-company" });
await registerBudgetEnforcementWork(s.db, s.coordinator, {}).ready;
expect(work.deliver).toHaveBeenCalledOnce();
await vi.advanceTimersByTimeAsync(1000);
expect(work.evaluate).toHaveBeenLastCalledWith(s.db, { cursor: "failed-company" });
expect(work.deliver).toHaveBeenCalledTimes(2);
expect(s.coordinator.nextWakeAt()).toBeNull();
});
it("recovers at startup, then leaves unchanged budget history completely quiet", async () => {
const s = setup();
await registerBudgetEnforcementWork(s.db, s.coordinator, {}).ready;
@@ -55,7 +81,7 @@ describe("budget accounting delivery", () => {
expect(work.pending).toHaveBeenCalledOnce();
expect(work.reconcile).toHaveBeenCalledOnce();
await s.owner.transaction(tx => notifyDeliveryWork(tx as object, DELIVERY_QUEUES.budgetEnforcement));
await vi.advanceTimersByTimeAsync(1);
await vi.advanceTimersByTimeAsync(1000);
expect(work.reconcile).toHaveBeenCalledOnce();
expect(work.deliver).toHaveBeenCalledTimes(2);
expect(s.coordinator.nextWakeAt()).toBeNull();
@@ -95,7 +121,7 @@ describe("budget accounting delivery", () => {
await registerBudgetEnforcementWork(s.db, s.coordinator, {}).ready;
expect(work.reconcile).toHaveBeenLastCalledWith(s.db, { startup: true });
await s.owner.transaction(tx => notifyDeliveryWork(tx as object, DELIVERY_QUEUES.budgetEnforcement));
await vi.advanceTimersByTimeAsync(1);
await vi.advanceTimersByTimeAsync(1000);
expect(work.reconcile).toHaveBeenLastCalledWith(s.db, { startup: true, cursor: "company:agent:scope" });
expect(s.coordinator.nextWakeAt()).toBeNull();
});
@@ -0,0 +1,214 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, budgetEvaluationWork, budgetIncidents, budgetPolicies, companies, costEvents, createDb, projects } from "@paperclipai/db";
import { budgetService, budgetServiceInTransaction, flushBudgetEvaluationWork } from "../services/budgets.js";
import { costService, createCostEventInTransaction } from "../services/costs.js";
import { withAccountingTransaction, withOperationAccountingTransaction } from "../services/accounting-transaction.js";
import { computeObservedSpends, type SpendPolicy } from "../services/budget-exposure.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { registerBudgetEnforcementWork } from "../services/budget-enforcement-worker.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "./helpers/embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const postgres = support.supported ? describe : describe.skip;
postgres("independent accounting and deferred budget decisions", () => {
let temp: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
temp = await startEmbeddedPostgresTestDatabase("paperclip-budget-work-");
db = createDb(temp.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await temp?.cleanup();
});
async function fixture(limit = 100) {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Independent accounting", issuePrefix: `X${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", role: "engineer", status: "active", adapterType: "process" });
await budgetService(db).upsertPolicy(companyId, { scopeType: "company", scopeId: companyId, amount: limit }, null);
await flushBudgetEvaluationWork(db, { companyId });
return { companyId, agentId };
}
function charge(agentId: string, idempotencyKey = randomUUID(), costCents = 60) {
return { agentId, provider: "openai", biller: "openai", model: "gpt-4", billingType: "metered_api",
costCents, occurredAt: new Date(), idempotencyKey };
}
async function pending(companyId: string) {
return db.select().from(budgetEvaluationWork).where(eq(budgetEvaluationWork.companyId, companyId));
}
it("records every concurrent charge, then coalesces one recoverable hard-stop transition", async () => {
const f = await fixture();
await Promise.all([costService(db).createEvent(f.companyId, charge(f.agentId)), costService(db).createEvent(f.companyId, charge(f.agentId))]);
expect((await pending(f.companyId))).toHaveLength(2);
expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0].pauseReason).toBeNull();
await Promise.all([flushBudgetEvaluationWork(db, f), flushBudgetEvaluationWork(db, f)]);
expect(await pending(f.companyId)).toHaveLength(0);
expect((await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId)))).toHaveLength(2);
expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0].pauseReason).toBe("budget");
const incidents = await db.select().from(budgetIncidents).where(and(eq(budgetIncidents.companyId, f.companyId), eq(budgetIncidents.thresholdType, "hard")));
expect(incidents).toHaveLength(1);
expect((await budgetService(db).evaluateAdmission(f.companyId, f.agentId)).block).not.toBeNull();
});
it("keeps a lower allocated input that commits after a consumed batch", async () => {
const f = await fixture(1000);
let release!: () => void, inserted!: () => void;
const ready = new Promise<void>(resolve => { inserted = resolve; });
const gate = new Promise<void>(resolve => { release = resolve; });
const earlier = withOperationAccountingTransaction(db, f.companyId, async (tx, publications) => {
await createCostEventInTransaction(tx, f.companyId, charge(f.agentId), publications);
inserted();
await gate;
});
try {
await ready;
await costService(db).createEvent(f.companyId, charge(f.agentId));
expect(await flushBudgetEvaluationWork(db, f)).toBe(1);
release();
await earlier;
expect(await pending(f.companyId)).toHaveLength(1);
expect(await flushBudgetEvaluationWork(db, f)).toBe(1);
expect(await pending(f.companyId)).toHaveLength(0);
} finally { release(); await earlier; }
});
it("does not create more work or audit charges on identical receipt replay", async () => {
const f = await fixture();
const receipt = charge(f.agentId, randomUUID(), 1);
const events = await Promise.all([costService(db).createEvent(f.companyId, receipt), costService(db).createEvent(f.companyId, receipt)]);
expect(events[0].id).toBe(events[1].id);
expect(await pending(f.companyId)).toHaveLength(1);
await expect(costService(db).createEvent(f.companyId, { ...receipt, costCents: 2 })).rejects.toMatchObject({ status: 409 });
expect(await pending(f.companyId)).toHaveLength(1);
});
it("blocks stale limited admissions until committed work is evaluated, then stays fresh while idle", async () => {
const f = await fixture(1000);
await costService(db).createEvent(f.companyId, charge(f.agentId, randomUUID(), 1));
await db.update(budgetEvaluationWork).set({ createdAt: new Date(Date.now() - 16_000) }).where(eq(budgetEvaluationWork.companyId, f.companyId));
expect((await budgetService(db).evaluateAdmission(f.companyId, f.agentId)).block?.reason).toContain("awaiting reconciliation");
await flushBudgetEvaluationWork(db, f);
expect((await budgetService(db).evaluateAdmission(f.companyId, f.agentId)).block).toBeNull();
expect(await flushBudgetEvaluationWork(db, f)).toBe(0);
});
it("retains policy raise and manual pause when evaluation races them", async () => {
const f = await fixture();
await costService(db).createEvent(f.companyId, charge(f.agentId, randomUUID(), 120));
await Promise.all([flushBudgetEvaluationWork(db, f), budgetService(db).upsertPolicy(f.companyId,
{ scopeType: "company", scopeId: f.companyId, amount: 1000 }, null)]);
expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0].pauseReason).toBeNull();
await db.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() }).where(eq(companies.id, f.companyId));
await costService(db).createEvent(f.companyId, charge(f.agentId, randomUUID(), 1));
await flushBudgetEvaluationWork(db, f);
expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0].pauseReason).toBe("manual");
});
it("does not apply another agent's stale backlog to a separately limited agent", async () => {
const f = await fixture(0);
const otherAgentId = randomUUID();
await db.insert(agents).values({ id: otherAgentId, companyId: f.companyId, name: "Other worker", role: "engineer", status: "active", adapterType: "process" });
const service = budgetService(db);
for (const agentId of [f.agentId, otherAgentId]) await service.upsertPolicy(f.companyId,
{ scopeType: "agent", scopeId: agentId, amount: 1000 }, null);
await flushBudgetEvaluationWork(db, f);
await costService(db).createEvent(f.companyId, charge(f.agentId, randomUUID(), 1));
await db.update(budgetEvaluationWork).set({ createdAt: new Date(Date.now() - 16_000) }).where(eq(budgetEvaluationWork.companyId, f.companyId));
expect((await service.evaluateAdmission(f.companyId, f.agentId)).block).not.toBeNull();
expect((await service.evaluateAdmission(f.companyId, otherAgentId)).block).toBeNull();
});
it("evaluates many distinct scopes without exceeding PostgreSQL's aggregate column limit", async () => {
const f = await fixture(1000);
await costService(db).createEvent(f.companyId, charge(f.agentId, randomUUID(), 1));
const policies: SpendPolicy[] = [{ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
windowKind: "lifetime", metric: "billed_cents" }, ...Array.from({ length: 450 }, () => ({
companyId: f.companyId, scopeType: "project", scopeId: randomUUID(), windowKind: "lifetime", metric: "billed_cents",
}))];
const observed = await withOperationAccountingTransaction(db, f.companyId, tx => computeObservedSpends(tx, policies));
expect(observed).toHaveLength(451);
expect(observed[0].ledgerCents).toBe("1.0000000");
expect(observed.slice(1).every(row => row.totalExact === "0.0000000")).toBe(true);
});
it("proves the legacy mutex waits while the independent ledger transaction commits", async () => {
const f = await fixture(1000);
let release!: () => void, locked!: () => void;
const ready = new Promise<void>(resolve => { locked = resolve; });
const gate = new Promise<void>(resolve => { release = resolve; });
const holding = db.transaction(async tx => {
await tx.select().from(companies).where(eq(companies.id, f.companyId)).for("no key update");
locked(); await gate;
});
let legacyFinished = false;
let legacy: Promise<unknown> | undefined;
try {
await ready;
legacy = withAccountingTransaction(db, f.companyId, async () => { legacyFinished = true; });
await costService(db).createEvent(f.companyId, charge(f.agentId));
expect(legacyFinished).toBe(false);
expect(await pending(f.companyId)).toHaveLength(1);
} finally { release(); await holding; await legacy; }
expect(legacyFinished).toBe(true);
});
it("reevaluates a new policy committed after a no-policy batch consumed its charge", async () => {
const f = await fixture(1000);
const projectId = randomUUID();
await db.insert(projects).values({ id: projectId, companyId: f.companyId, name: "New policy scope" });
let release!: () => void, created!: () => void;
const ready = new Promise<void>(resolve => { created = resolve; });
const gate = new Promise<void>(resolve => { release = resolve; });
const policy = withOperationAccountingTransaction(db, f.companyId, async (tx, publications) => {
await budgetServiceInTransaction(tx, publications, { evaluate: true }).upsertPolicy(f.companyId,
{ scopeType: "project", scopeId: projectId, amount: 80 }, null);
created(); await gate;
});
async function bounded<T>(work: Promise<T>, phase: string): Promise<T> {
let timer: ReturnType<typeof setTimeout>;
try { return await Promise.race([work, new Promise<never>((_, reject) => {
timer = setTimeout(() => reject(new Error(`Policy race blocked at ${phase}`)), 2000);
})]); } finally { clearTimeout(timer!); }
}
try {
await bounded(Promise.race([ready, policy]), "policy prepared");
await bounded(costService(db).createEvent(f.companyId, { ...charge(f.agentId, randomUUID(), 120), projectId }), "charge commit");
expect(await bounded(flushBudgetEvaluationWork(db, f), "first evaluation")).toBe(1);
release(); await policy;
expect(await pending(f.companyId)).toHaveLength(1);
await flushBudgetEvaluationWork(db, f);
expect((await db.select().from(projects).where(eq(projects.id, projectId)))[0].pauseReason).toBe("budget");
} finally { release(); await policy; }
});
it("delivers another company's Stop when a busy company has more than a page of old inputs", async () => {
const busy = await fixture(), healthy = await fixture();
await costService(db).createEvent(busy.companyId, charge(busy.agentId, randomUUID(), 120));
await costService(db).createEvent(healthy.companyId, charge(healthy.agentId, randomUUID(), 120));
await db.insert(budgetEvaluationWork).values(Array.from({ length: 150 }, () => ({
companyId: busy.companyId, agentId: busy.agentId, createdAt: new Date(Date.now() - 20_000),
})));
let release!: () => void, locked!: () => void;
const ready = new Promise<void>(resolve => { locked = resolve; });
const gate = new Promise<void>(resolve => { release = resolve; });
const holding = db.transaction(async tx => {
await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, busy.companyId)).for("no key update");
locked(); await gate;
});
const errors: unknown[] = [], delivered: string[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
try {
await ready;
await registerBudgetEnforcementWork(db, coordinator, { cancelWorkForScope: async scope => { delivered.push(scope.companyId); } }).ready;
expect(errors).toHaveLength(0);
expect(delivered).toContain(healthy.companyId);
expect((await db.select().from(companies).where(eq(companies.id, healthy.companyId)))[0].pauseReason).toBe("budget");
expect(await pending(healthy.companyId)).toHaveLength(0);
expect((await pending(busy.companyId)).length).toBeGreaterThan(100);
} finally { release(); await holding; await coordinator.stop(); }
});
});
@@ -6,7 +6,7 @@ import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import { agents, budgetPolicies, budgetReservations, companies, costEvents, createDb, heartbeatRuns, type Db } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { computeObservedSpend } from "../services/budgets.js";
import { computeObservedSpend, flushBudgetEvaluationWork } from "../services/budgets.js";
import { accountRunCost } from "../services/run-cost-accounting.js";
import { createRunUsageRecorder, replayUsageReceipts, type UsageReceiptEnvelope } from "../services/usage-receipts.js";
@@ -61,6 +61,7 @@ const support = await getEmbeddedPostgresTestSupport();
expect(partial.usageJson).toMatchObject({ costUsdExact: "0.000000000", costStatus: "unpriced",
accountingReceiptReady: false, budgetExposure: { costCents: "100.2000000" } });
expect(await db.select().from(costEvents)).toHaveLength(0);
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0].lifecycleHolds).toContain("budget");
await f.recorder.capture({ ...f.partial, complete: true, costUsd: 0.8, costStatus: "reported" });
@@ -69,6 +70,8 @@ const support = await getEmbeddedPostgresTestSupport();
expect(await accountRunCost(db, f.runId)).toBe(true);
expect(await computeObservedSpend(db, f.policy)).toMatchObject({ liveObservedCents: "0.0000000", ledgerCents: "80.0000000" });
expect(await db.select().from(costEvents)).toHaveLength(1);
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0].lifecycleHolds).not.toContain("budget");
});
it("replays the fsynced per-attempt exposure after a failed database write", async () => {
@@ -83,6 +86,7 @@ const support = await getEmbeddedPostgresTestSupport();
f.offline(false);
await replayUsageReceipts(f.proxy, f.directory);
expect((await computeObservedSpend(db, f.policy)).liveObservedCents).toBe("100.2000000");
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0].lifecycleHolds).toContain("budget");
expect((await fs.readdir(f.directory)).filter(file => file.endsWith(".json"))).toEqual([]);
});
@@ -96,6 +100,7 @@ const support = await getEmbeddedPostgresTestSupport();
expect((await f.readRun()).usageJson).toMatchObject({ costUsdExact: "100.200000000",
usageSource: "per_run", budgetExposure: { costCents: "20.0000000" } });
expect((await computeObservedSpend(db, f.policy)).liveObservedCents).toBe("20.0000000");
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0].lifecycleHolds).not.toContain("budget");
});
+5 -1
View File
@@ -77,7 +77,11 @@ describe.skipIf(!support.supported)("budget history for deleted scopes", () => {
const [saved] = await db.select().from(budgetIncidents).where(eq(budgetIncidents.id, incident!.id));
expect(saved).toMatchObject({ ...preserved, status: "dismissed", resolvedAt: expect.any(Date) });
expect((await db.select().from(approvals).where(eq(approvals.id, approval!.id)))[0]).toMatchObject({ status: "rejected" });
expect(await db.select().from(budgetPolicies)).toEqual(before.policies);
const policiesAfter = await db.select().from(budgetPolicies);
// Incident mutation can advance the evaluator fence without changing
// the historical policy configuration or its spending controls.
expect(policiesAfter.map(({ evaluationVersion: _version, ...policy }) => policy))
.toEqual(before.policies.map(({ evaluationVersion: _version, ...policy }) => policy));
expect(await db.select().from(agents)).toEqual(before.agents);
expect(await db.select().from(costEvents)).toEqual(before.costs);
expect((await service.overview(companyId)).activeIncidents.some(row => row.id === incident!.id)).toBe(false);
+15 -3
View File
@@ -16,7 +16,7 @@ import {
nativeRunFinalizations,
projects,
} from "@paperclipai/db";
import { budgetService, deliverBudgetEnforcement, hasPendingBudgetEnforcement } from "../services/budgets.ts";
import { budgetService, deliverBudgetEnforcement, hasPendingBudgetEnforcement, flushBudgetEvaluationWork } from "../services/budgets.ts";
import {
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
getEmbeddedPostgresTestSupport,
@@ -170,7 +170,7 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
await reader.execute(sql`set lock_timeout = '500ms'`);
if (state === "recovering run") {
await expect(budgetService(reader).getInvocationBlock(companyId, agentId, { projectId }))
.rejects.toMatchObject({ cause: { code: "55P03" } });
.resolves.toMatchObject({ reason: "New work must wait while a native run recovers its unfinished accounting." });
} else {
const block = await budgetService(reader).getInvocationBlock(companyId, agentId, { projectId });
if (state === "manual company pause") {
@@ -189,7 +189,7 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
}
if (state === "no policy") {
await expect(budgetService(reader).evaluateAdmission(companyId, agentId, { projectId }))
.rejects.toMatchObject({ cause: { code: "55P03" } });
.resolves.toMatchObject({ block: null, policies: [], observations: [] });
}
}
} finally {
@@ -222,6 +222,9 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
await db.update(agents).set({ status: "paused", lifecycleState: "paused", pauseReason: "budget", pausedAt: new Date(), lifecycleHolds: ["budget"] })
.where(eq(agents.id, agentId));
expect(await budgetService(db).getInvocationBlock(companyId, agentId)).not.toBeNull();
await budgetService(db).evaluateAffectedScopes(companyId, agentId, null);
await flushBudgetEvaluationWork(db, { companyId });
expect(await budgetService(db).getInvocationBlock(companyId, agentId)).toBeNull();
expect((await db.select().from(agents).where(eq(agents.id, agentId)))[0]).toMatchObject({
lifecycleState: "resuming", pauseReason: null, lifecycleHolds: [],
@@ -237,6 +240,8 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
thresholdType, amountLimit: 100, amountObserved: 100 }).returning();
expect(await budgetService(db).getInvocationBlock(companyId, agentId)).toBeNull();
await budgetService(db).evaluateAffectedScopes(companyId, agentId, null);
await flushBudgetEvaluationWork(db, { companyId });
expect((await db.select().from(budgetIncidents).where(eq(budgetIncidents.id, incident.id)))[0])
.toMatchObject({ status: "resolved", resolvedAt: expect.any(Date) });
});
@@ -273,6 +278,7 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
const softEvent = await insertCostEvent({ companyId, agentId, costCents: 80 });
await service.evaluateCostEvent(softEvent);
await service.evaluateCostEvent(softEvent);
await flushBudgetEvaluationWork(db, { companyId });
let incidentRows = await db
.select()
@@ -298,6 +304,7 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
const hardEvent = await insertCostEvent({ companyId, agentId, costCents: 25 });
await service.evaluateCostEvent(hardEvent);
await service.evaluateCostEvent(hardEvent);
await flushBudgetEvaluationWork(db, { companyId });
incidentRows = await db
.select()
@@ -411,6 +418,8 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
const service = budgetService(db);
const blocks = await Promise.all(Array.from({ length: 4 }, () => service.getInvocationBlock(companyId, agentId, { projectId })));
expect(blocks.every(block => block?.scopeType === "project" && block.scopeId === projectId)).toBe(true);
await service.evaluateAffectedScopes(companyId, agentId, projectId);
await flushBudgetEvaluationWork(db, { companyId });
const [paused] = await db.select().from(projects).where(eq(projects.id, projectId));
expect(paused).toMatchObject({ pauseReason: alreadyPaused ? "manual" : "budget", pausedAt: alreadyPaused ? pausedAt : expect.any(Date) });
expect(dispatch).toHaveBeenCalledTimes(alreadyPaused ? 0 : 1);
@@ -442,6 +451,7 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
const event = await insertCostEvent({ companyId, agentId, projectId, costCents: 125 });
await service.evaluateCostEvent(event);
await service.evaluateCostEvent(event);
await flushBudgetEvaluationWork(db, { companyId });
const incidentRows = await db
.select()
@@ -539,6 +549,7 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
await service.upsertPolicy(companyId, { scopeType: "project", scopeId: projectId,
amount: 100, windowKind: "lifetime" }, "board");
await service.evaluateCostEvent(await insertCostEvent({ companyId, agentId, projectId, costCents: 125 }));
await flushBudgetEvaluationWork(db, { companyId });
expect(await hasPendingBudgetEnforcement(db)).toBe(true);
expect(cancelWorkForScope).not.toHaveBeenCalled();
await service.upsertPolicy(companyId, { scopeType: "project", scopeId: projectId,
@@ -556,6 +567,7 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => {
amount: 100, windowKind: "lifetime" }, "board");
const event = await insertCostEvent({ companyId, agentId, projectId, costCents: 125 });
await service.evaluateCostEvent(event);
await flushBudgetEvaluationWork(db, { companyId });
const cancelWorkForScope = vi.fn().mockRejectedValueOnce(new Error("Controlled stop delivery failure")).mockResolvedValue(undefined);
await deliverBudgetEnforcement(db, { cancelWorkForScope }, companyId);
expect(await hasPendingBudgetEnforcement(db)).toBe(true);
@@ -182,6 +182,30 @@ describe("actionable cost data quality", () => {
expect((await costDataQualityReport(db, owner.companyId, true, undefined, now)).runs[0].reason).toBe("retryable");
});
it("exposes acknowledged late receipt debt through existing run and retry metadata", async () => {
const owner = await company(), other = await company();
const pending = await run(owner, { costAccountedAt: now, accountingLastError: "late_usage_requires_reconciliation",
usageJson: { accountingReceiptReady: true, accountingLateReceiptPending: true,
accountingLateReceiptId: randomUUID(), privateProviderPayload: "PRIVATE LATE RECEIPT" } });
const report = await costDataQualityReport(db, owner.companyId, true, undefined, now);
expect(report.canRetry).toBe(true);
expect(report.runs).toEqual([expect.objectContaining({ id: pending.id, runId: pending.id, reason: "retryable" })]);
expect(JSON.stringify(report)).not.toContain("PRIVATE LATE RECEIPT");
expect((await costDataQualityReport(db, owner.companyId, false, undefined, now)).canRetry).toBe(false);
expect((await costDataQualityReport(db, other.companyId, true, undefined, now)).runs).toEqual([]);
const summary = await costService(db).summary(owner.companyId, { from: new Date("2026-10-01T00:00:00Z"), to: now });
expect(summary.pendingRunCount).toBe(1);
expect(summary.pricingComplete).toBe(false);
const health = await accountingIntegrityService(db).health(owner.companyId);
expect(health.pendingRunCount).toBe(1);
expect(health.items).toEqual([expect.objectContaining({ runId: pending.id, state: "retryable",
lastError: "late_usage_requires_reconciliation" })]);
const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, pending.id));
expect(saved.costAccountedAt?.toISOString()).toBe(now.toISOString());
expect(saved.costAccountingPending).toBe(true);
expect(saved.usageJson?.accountingLateReceiptPending).toBe(true);
});
it("bounds each category independently and reports truncation", async () => {
const owner = await company();
for (let i = 0; i < 26; i++) {
+5 -3
View File
@@ -472,13 +472,15 @@ describeEmbeddedPostgres("cost and finance aggregate overflow handling", () => {
cachedInputTokens: 2_632_998,
outputTokens: 32_644,
costCents: 0,
occurredAt: new Date("2026-07-13T14:22:54.000Z"),
occurredAt: new Date(),
});
expect(event.costStatus).toBe("unpriced");
expect(event.inputTokens).toBe(2_732_577);
const [agent] = await db.select().from(agents).where(eq(agents.id, agentId));
expect(agent?.spentMonthlyCents).toBe(0);
expect(await costs.summary(companyId)).toMatchObject({ eventCount: 1, spendCents: 0, spendCentsExact: "0.0000000" });
const [usage] = await costs.byAgent(companyId);
expect(usage).toMatchObject({ agentId, costCents: 0, inputTokens: 2_732_577,
cachedInputTokens: 2_632_998, outputTokens: 32_644 });
});
it("aggregates cost event sums above int32 without raising Postgres integer overflow", async () => {
+238 -6
View File
@@ -5,9 +5,9 @@ import os from "node:os";
import path from "node:path";
import { and, eq, sql } from "drizzle-orm";
import {
createDb, companies, agents, companyMemberships, companyDecisionModels, costEvents, decisionInvocations,
createDb, companies, agents, companyMemberships, companyDecisionModels, costEvents, costAdjustments, decisionInvocations,
connectionGrants, connectionGrantMembers, toolConnectionInstalls, budgetPolicies, budgetReservations,
heartbeatRuns, issues, projects, runIdentityContexts, secretAccessEvents, companySecrets,
heartbeatRuns, issues, projects, runIdentityContexts, secretAccessEvents, companySecrets, activityLog,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";
import { DECISION_TEST_REQUEST, updateDecisionModelSchema, createCostEventSchema } from "@paperclipai/shared";
@@ -18,11 +18,12 @@ import { decisionReceipt, type DecisionProviderOutcome } from "../services/decis
import { costService } from "../services/costs.js";
import { billingReconciliationService } from "../services/billing-reconciliation.js";
import { withAccountingTransaction } from "../services/accounting-transaction.js";
import { deleteAgentDependencies } from "../services/agent-deletion.js";
import express from "express";
import request from "supertest";
import { decisionModelRoutes } from "../routes/decision-models.js";
import { errorHandler } from "../middleware/index.js";
import { deliverBudgetEnforcement } from "../services/budgets.js";
import { deliverBudgetEnforcement, flushBudgetEvaluationWork } from "../services/budgets.js";
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
@@ -75,6 +76,7 @@ describe("company decision service", () => {
const events = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({ usageKind: "decision", costCents: 2 });
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
const [policy] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.companyId, f.companyId));
expect(policy.enforcementVersion).toBeGreaterThan(policy.enforcementDeliveredVersion);
delivery = deliverBudgetEnforcement(db, { cancelWorkForScope }, f.companyId);
@@ -84,6 +86,170 @@ describe("company decision service", () => {
expect((await db.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id)))[0].enforcementDeliveredVersion).toBe(policy.enforcementVersion);
} finally { release(); await pending.catch(() => undefined); await delivery?.catch(() => undefined); }
});
it.each(["company", "agent", "policy"] as const)("settles while an unrelated %s accounting row is locked", async locked => {
const f = await fixture(), runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
status: "running", invocationSource: "on_demand", responsibleUserId: "alice" });
const [policy] = await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
windowKind: "calendar_month_utc", amount: 100, notifyEnabled: false }).returning();
let providerEntered!: () => void, finishProvider!: () => void, unlock!: () => void, lockedRow!: () => void;
const entered = new Promise<void>(resolve => { providerEntered = resolve; });
const providerGate = new Promise<void>(resolve => { finishProvider = resolve; });
const lockGate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<void>(resolve => { lockedRow = resolve; });
f.provider.mockImplementationOnce(async () => { providerEntered(); await providerGate; return structuredClone(outcome); });
const context: DecisionContext = { ...f.context, actor: { type: "agent", companyId: f.companyId, agentId: f.agentId, runId, source: "agent_jwt" } };
const pending = f.service.decide(context, DECISION_TEST_REQUEST);
await entered;
const blocker = db.transaction(async tx => {
if (locked === "company") await tx.select().from(companies).where(eq(companies.id, f.companyId)).for("no key update");
if (locked === "agent") await tx.select().from(agents).where(eq(agents.id, f.agentId)).for("no key update");
if (locked === "policy") await tx.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id)).for("no key update");
lockedRow(); await lockGate;
});
let finished = false;
void pending.finally(() => { finished = true; }).catch(() => undefined);
try {
await acquired; finishProvider();
await vi.waitFor(() => expect(finished).toBe(true), { timeout: 3000 });
expect(await pending).toMatchObject({ status: "succeeded" });
expect((await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId)))[0].state).toBe("settled");
} finally { unlock(); finishProvider(); await blocker; await pending.catch(() => undefined); }
});
it.each(["company", "project"] as const)("denies paid admission when %s manual pause wins its control fence", async scope => {
const f = await fixture();
const [project] = await db.insert(projects).values({ companyId: f.companyId, name: "Control fence" }).returning();
let unlock!: () => void, locked!: (pid: number) => void;
const gate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
const holding = db.transaction(async tx => {
if (scope === "company") await tx.select().from(companies).where(eq(companies.id, f.companyId)).for("no key update");
else await tx.select().from(projects).where(eq(projects.id, project.id)).for("no key update");
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
locked(row.pid); await gate;
if (scope === "company") await tx.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() }).where(eq(companies.id, f.companyId));
else await tx.update(projects).set({ pauseReason: "manual", pausedAt: new Date() }).where(eq(projects.id, project.id));
});
const pid = await acquired;
const pending = f.service.decide({ ...f.context, projectId: project.id }, DECISION_TEST_REQUEST);
try {
await vi.waitFor(async () => {
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`);
expect(row.count).toBeGreaterThan(0);
}, { timeout: 3000 });
unlock(); await holding;
expect(await pending).toMatchObject({ status: "unavailable" });
expect(f.provider).not.toHaveBeenCalled();
expect(await db.select().from(decisionInvocations).where(eq(decisionInvocations.companyId, f.companyId))).toHaveLength(0);
} finally { unlock(); await holding; await pending.catch(() => undefined); }
});
it("settles an independent decision for the same agent while another invocation is locked", async () => {
const f = await fixture(), runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
status: "running", invocationSource: "on_demand", responsibleUserId: "alice" });
const context: DecisionContext = { ...f.context, actor: { type: "agent", companyId: f.companyId, agentId: f.agentId, runId, source: "agent_jwt" } };
let firstEntered!: () => void, finishFirst!: () => void, unlock!: () => void, lockedRow!: () => void;
const entered = new Promise<void>(resolve => { firstEntered = resolve; });
const providerGate = new Promise<void>(resolve => { finishFirst = resolve; });
const lockGate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<void>(resolve => { lockedRow = resolve; });
f.provider.mockImplementationOnce(async () => { firstEntered(); await providerGate; return structuredClone(outcome); });
const first = f.service.decide(context, DECISION_TEST_REQUEST);
await entered;
const [invocation] = await db.select().from(decisionInvocations).where(eq(decisionInvocations.companyId, f.companyId));
const blocker = db.transaction(async tx => {
await tx.select().from(decisionInvocations).where(eq(decisionInvocations.id, invocation.id)).for("update");
lockedRow(); await lockGate;
});
let second: ReturnType<typeof f.service.decide> | undefined;
try {
await acquired; finishFirst();
second = f.service.decide(context, DECISION_TEST_REQUEST);
let finished = false;
void second.finally(() => { finished = true; }).catch(() => undefined);
await vi.waitFor(() => expect(finished).toBe(true), { timeout: 3000 });
expect(await second).toMatchObject({ status: "succeeded" });
} finally { unlock(); finishFirst(); await blocker; await first; await second; }
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(2);
});
it("retries a rolled-back settlement without repeating the provider or charge", async () => {
const f = await fixture(), suffix = randomUUID().replaceAll("-", "");
const sequenceName = `decision_rollback_${suffix}`, functionName = `fail_decision_${suffix}`, triggerName = `decision_retry_${suffix}`;
await db.execute(sql.raw(`CREATE SEQUENCE ${sequenceName}`));
await db.execute(sql.raw(`CREATE FUNCTION ${functionName}() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF NEW.company_id = '${f.companyId}'::uuid AND NEW.status <> 'running' AND nextval('${sequenceName}') = 1 THEN
RAISE EXCEPTION 'retry injected after ledger insert' USING ERRCODE = '40001';
END IF;
RETURN NEW;
END $$`));
await db.execute(sql.raw(`CREATE TRIGGER ${triggerName} BEFORE UPDATE ON decision_invocations FOR EACH ROW EXECUTE FUNCTION ${functionName}()`));
try {
expect(await f.service.decide(f.context, DECISION_TEST_REQUEST)).toMatchObject({ status: "succeeded" });
expect(f.provider).toHaveBeenCalledOnce();
const events = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(events).toHaveLength(1);
expect((await db.select().from(decisionInvocations).where(eq(decisionInvocations.companyId, f.companyId)))[0].costEventId).toBe(events[0].id);
expect((await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId)))[0].state).toBe("settled");
} finally {
await db.execute(sql.raw(`DROP TRIGGER ${triggerName} ON decision_invocations`));
await db.execute(sql.raw(`DROP FUNCTION ${functionName}()`));
await db.execute(sql.raw(`DROP SEQUENCE ${sequenceName}`));
}
});
it.each([
{ recoveryFirst: true, priced: true, corrected: false }, { recoveryFirst: false, priced: true, corrected: false },
{ recoveryFirst: true, priced: false, corrected: false }, { recoveryFirst: false, priced: false, corrected: false },
{ recoveryFirst: true, priced: true, corrected: true },
])("keeps a real result with recovery first: $recoveryFirst, priced: $priced, corrected: $corrected", async ({ recoveryFirst, priced, corrected }) => {
const f = await fixture();
const realOutcome: DecisionProviderOutcome = priced ? structuredClone(outcome) : { ...structuredClone(outcome), receipt: {
...outcome.receipt, costCents: null, costStatus: "unpriced", pricingProvenance: { source: "unknown" } } };
let finishProvider!: () => void, providerEntered!: () => void;
const gate = new Promise<void>(resolve => { finishProvider = resolve; });
const entered = new Promise<void>(resolve => { providerEntered = resolve; });
f.provider.mockImplementationOnce(async () => { providerEntered(); await gate; return realOutcome; });
const pending = f.service.decide(f.context, DECISION_TEST_REQUEST);
await entered;
await db.update(decisionInvocations).set({ startedAt: new Date(Date.now() - 180_000) }).where(eq(decisionInvocations.companyId, f.companyId));
let originalHash: string | null = null;
try {
if (recoveryFirst) {
await Promise.all([f.service.recoverInterrupted(), f.service.recoverInterrupted()]);
const [event] = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(event.costStatus).toBe("unpriced"); originalHash = event.receiptHash;
if (corrected) await billingReconciliationService(db).adjust(f.companyId, event.id, { idempotencyKey: "invoice-before-receipt",
expectedCents: "0", correctedCents: "0.005", reason: "Verified invoice before provider completed", pricing: { source: "operator" } }, "alice");
}
finishProvider();
expect(await pending).toMatchObject({ status: "succeeded" });
await f.service.recoverInterrupted();
const events = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({ costStatus: corrected ? "reported" : priced ? "estimated" : "unpriced", costCents: corrected ? 0.005 : priced ? 0.001 : 0 });
expect(events[0]).toMatchObject({ inputTokens: 100, cachedInputTokens: 0, outputTokens: 0,
providerRequestId: outcome.receipt.providerRequestId });
const corrections = await db.select().from(costAdjustments).where(eq(costAdjustments.companyId, f.companyId));
expect(corrections).toHaveLength(recoveryFirst ? corrected ? 2 : 1 : 0);
if (recoveryFirst) {
expect(events[0]).toMatchObject({ receiptHash: originalHash, reportedCostCents: "0.0000000",
reportedUsageJson: { inputTokens: 0, cachedInputTokens: 0, outputTokens: 0, providerRequestId: null } });
expect(corrections.find(row => row.idempotencyKey.endsWith(":late-receipt") || row.idempotencyKey.endsWith(":late-usage"))?.previousUsageJson)
.toMatchObject({ inputTokens: 0, cachedInputTokens: 0, outputTokens: 0, providerRequestId: null });
if (priced && !corrected) expect(JSON.parse(corrections[0].pricing.evidence as string)).toMatchObject({ inputTokens: 100, outputTokens: 0 });
}
if (corrected) {
expect(events[0].pricingProvenance).toEqual({ source: "operator" });
const observed = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.companyId), eq(activityLog.action, "accounting.late_receipt_observed")));
expect(observed).toHaveLength(1);
expect(observed[0].details).toMatchObject({ effectiveCostCents: "0.0050000", costCents: "0.0010000", inputTokens: 100 });
}
expect((await db.select().from(decisionInvocations).where(eq(decisionInvocations.companyId, f.companyId)))[0]).toMatchObject({
status: "succeeded", costEventId: events[0].id, inputTokens: 100, outputTokens: 0 });
expect((await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId)))[0].state).toBe(priced ? "settled" : "held");
expect(f.provider).toHaveBeenCalledOnce();
} finally { finishProvider(); await pending.catch(() => undefined); }
});
it("starts unconfigured with background sponsorship on and makes availability free", async () => {
const f = await fixture();
await db.delete(companyDecisionModels).where(eq(companyDecisionModels.companyId, f.companyId));
@@ -146,13 +312,14 @@ describe("company decision service", () => {
await f.service.configure(f.companyId, "alice", { ...f.config, enabled: false });
expect(await f.service.availability(f.context)).toEqual({ available: false, reason: "disabled" }); expect(f.provider).not.toHaveBeenCalled();
});
it.each(["disabled", "rotated"])("rejects a secret %s while admission waits for accounting", async change => {
it.each(["disabled", "rotated"])("rejects a secret %s while final admission waits for configuration", async change => {
const f = await fixture();
const [secret] = await db.select().from(companySecrets).where(eq(companySecrets.companyId, f.companyId));
let release!: () => void, locked!: (pid: number) => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
const holding = withAccountingTransaction(db, f.companyId, async tx => {
const holding = db.transaction(async tx => {
await tx.select().from(companyDecisionModels).where(eq(companyDecisionModels.companyId, f.companyId)).for("no key update");
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
locked(row.pid); await gate;
});
@@ -201,7 +368,38 @@ describe("company decision service", () => {
await db.update(heartbeatRuns).set({ responsibleUserId: null }).where(eq(heartbeatRuns.id, runId));
expect(await f.service.availability(context)).toMatchObject({ available: false, reason: "responsible_user_missing" });
});
it("serializes budget admission against concurrent requests", async () => {
it("admits concurrent work from the same budget observation and records both obligations", async () => {
const f = await fixture();
await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
windowKind: "calendar_month_utc", amount: 1, reservationCents: "1", notifyEnabled: false });
let unlock!: () => void, locked!: (pid: number) => void, finishProviders!: () => void;
const lockGate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
const providerGate = new Promise<void>(resolve => { finishProviders = resolve; });
const holding = db.transaction(async tx => {
await tx.select().from(companyDecisionModels).where(eq(companyDecisionModels.companyId, f.companyId)).for("no key update");
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
locked(row.pid); await lockGate;
});
const pid = await acquired;
f.provider.mockImplementation(async () => { await providerGate; return structuredClone(outcome); });
const first = f.service.decide(f.context, DECISION_TEST_REQUEST), second = f.service.decide(f.context, DECISION_TEST_REQUEST);
try {
// Both admission snapshots exist before either independent reservation commits.
await vi.waitFor(async () => {
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`);
expect(row.count).toBe(2);
}, { timeout: 3000 });
unlock(); await holding;
await vi.waitFor(() => expect(f.provider).toHaveBeenCalledTimes(2));
const holds = await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId));
expect(holds).toHaveLength(2);
expect(holds.every(row => row.state === "held" && row.amountCents === "1.0000000")).toBe(true);
} finally { unlock(); finishProviders(); await holding; await Promise.allSettled([first, second]); }
expect((await first).status).toBe("succeeded"); expect((await second).status).toBe("succeeded");
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(2);
});
it("denies new work when an already committed reservation consumes the observed budget", async () => {
const f = await fixture();
await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId, windowKind: "calendar_month_utc", amount: 1, reservationCents: "1", notifyEnabled: false });
let release!: () => void; const gate = new Promise<void>(resolve => { release = resolve; });
@@ -213,6 +411,40 @@ describe("company decision service", () => {
finally { release(); }
expect((await first).status).toBe("succeeded"); expect(f.provider).toHaveBeenCalledTimes(1);
});
it("checks deletion holds after an admission waiting on its scoped identity fence commits", async () => {
const f = await fixture(), runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
status: "running", invocationSource: "on_demand", responsibleUserId: "alice" });
let unlock!: () => void, locked!: (pid: number) => void;
const gate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
const blocker = db.transaction(async tx => {
await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("no key update");
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
locked(row.pid); await gate;
});
const blockerPid = await acquired;
const context: DecisionContext = { ...f.context, actor: { type: "agent", companyId: f.companyId, agentId: f.agentId, runId, source: "agent_jwt" } };
const pending = f.service.decide(context, DECISION_TEST_REQUEST);
let deletion: Promise<unknown> | undefined;
try {
let admissionPid: number | undefined;
await vi.waitFor(async () => {
const [row] = await db.execute<{ pid: number }>(sql`select pid from pg_stat_activity where ${blockerPid} = any(pg_blocking_pids(pid))`);
expect(row).toBeDefined(); admissionPid = row?.pid;
}, { timeout: 3000 });
deletion = withAccountingTransaction(db, f.companyId, tx => deleteAgentDependencies(tx, f.companyId, f.agentId)).catch(error => error);
await vi.waitFor(async () => {
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${admissionPid} = any(pg_blocking_pids(pid))`);
expect(row.count).toBeGreaterThan(0);
}, { timeout: 3000 });
unlock(); await blocker;
expect(await deletion).toMatchObject({ status: 409, details: { code: "agent_decision_accounting_pending" } });
expect(await pending).toMatchObject({ status: "succeeded" });
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0]).toBeDefined();
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(1);
} finally { unlock(); await blocker; await deletion; await pending.catch(() => undefined); }
});
it("blocks agent deletion while an in-flight or unknown decision retains a budget hold", async () => {
const f = await fixture(), runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId, status: "running", invocationSource: "on_demand", responsibleUserId: "alice" });
@@ -4,7 +4,7 @@ import { setImmediate as immediate } from "node:timers/promises";
import { setTimeout as realTimeout, clearTimeout as clearRealTimeout } from "node:timers";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { budgetService } from "../services/budgets.js";
import { budgetService, flushBudgetEvaluationWork } from "../services/budgets.js";
import { configureAgentLifecycle, createAgentLifecycle } from "../services/agent-lifecycle.js";
import { accessService } from "../services/access.js";
import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../services/deferred-execution-wait.js";
@@ -275,6 +275,7 @@ describe("deferred receipt durable dispatch", () => {
await db.insert(costEvents).values({ companyId: f.companyId, agentId: f.agent.id, projectId,
provider: "test", model: "receipt-proof", costCents: 200, occurredAt: new Date() });
await budgets.upsertPolicy(f.companyId, policy, f.actorId);
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
} else await lifecycle.pauseAgent(f.agent.id);
await lifecycleWorker.process(f.agent.id);
await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready;
@@ -313,8 +314,10 @@ describe("deferred receipt durable dispatch", () => {
const { f, lifecycle, budgets, policy, lifecycleWorker, request, read } = await blockedRetry(gate);
// Keep the clock advancing with wall time through gate release and actual
// child-process exit. Accelerate only after that execution has settled.
if (gate !== "paused agent") await budgets.upsertPolicy(f.companyId, { ...policy, amount: 1000 }, f.actorId);
else await lifecycle.resumeAgent(f.agent.id);
if (gate !== "paused agent") {
await budgets.upsertPolicy(f.companyId, { ...policy, amount: 1000 }, f.actorId);
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
} else await lifecycle.resumeAgent(f.agent.id);
await lifecycleWorker.process(f.agent.id);
let runId: string | null = null;
await waitForReal(async () => {
+10 -5
View File
@@ -18,7 +18,7 @@ import { and, eq, inArray, sql } from "drizzle-orm";
import express, { type Request } from "express";
import request from "supertest";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { grokBotRoutines, grokBotDispatches, agents, authUsers, companies, companyMemberships, createDb, dotAgentBindings, dotMailboxItems, dotRunnerAssignments, dotRunnerOperations,
import { budgetPolicies, costEvents, grokBotRoutines, grokBotDispatches, agents, authUsers, companies, companyMemberships, createDb, dotAgentBindings, dotMailboxItems, dotRunnerAssignments, dotRunnerOperations,
heartbeatRuns, mcpOauthGrants, issueAttachments, activityLog, agentWakeupRequests, issueComments, issues, nativeRunResults, nativeRunFinalizations, completionContracts, mcpEventDeliveries, mcpEventSubscriptions, workspaceOperations, workAssessments, statusDecisions, issueThreadInteractions } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { createPublicMcpOAuth, DEVICE_GRANT } from "../services/public-mcp/oauth.js";
@@ -385,9 +385,11 @@ describe("durable Dot Runner integration", () => {
expect(intake[0]).toMatchObject({ assigneeAgentId: f.agent.id, createdByAgentId: f.agent.id, responsibleUserId: f.userId });
expect(await f.broker.capabilities(f.principal)).toMatchObject({ agentId: f.agent.id, responsibleUser: { id: f.userId }, idle: { start: provider === "grok_bot" ? "paperclip_grok_request_turn" : "paperclip_dot_request_turn" } });
await expect(f.broker.requestTurn(f.principal, "Different request", requestId)).rejects.toThrow("reused");
await db.update(companies).set({ budgetMonthlyCents: 1, spentMonthlyCents: 1 }).where(eq(companies.id, f.company.id));
const [budget] = await db.insert(budgetPolicies).values({ companyId: f.company.id, scopeType: "company", scopeId: f.company.id,
windowKind: "calendar_month_utc", amount: 1 }).returning();
await db.insert(costEvents).values({ companyId: f.company.id, agentId: f.agent.id, provider: "test", model: "test", costCents: 1, occurredAt: new Date() });
await expect(f.broker.requestTurn(f.principal, "Spend more", randomUUID())).rejects.toThrow("budget");
await db.update(companies).set({ budgetMonthlyCents: 0, spentMonthlyCents: 0 }).where(eq(companies.id, f.company.id));
await db.delete(budgetPolicies).where(eq(budgetPolicies.id, budget.id));
await f.broker.revoke(f.company.id, f.agent.id, f.userId);
expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0]!.adapterConfig).not.toHaveProperty(provider === "grok_bot" ? "externalBindingId" : "dotBindingId");
await expect(f.broker.requestTurn(f.principal, "After revoke", randomUUID())).rejects.toThrow();
@@ -1131,9 +1133,12 @@ describe("durable Dot Runner integration", () => {
expect(app).toBeUndefined();
expect((work.tools as Array<{ operationId: string }>).some(tool => ["get_workspace", "prepare_repository", "workspace_read"].includes(tool.operationId))).toBe(false);
}
await db.update(agents).set({ budgetMonthlyCents: 100, spentMonthlyCents: 100 }).where(eq(agents.id, f.agent.id));
const [budget] = await db.insert(budgetPolicies).values({ companyId: f.company.id, scopeType: "agent", scopeId: f.agent.id,
windowKind: "calendar_month_utc", amount: 100 }).returning();
await db.insert(costEvents).values({ companyId: f.company.id, agentId: f.agent.id, provider: "test", model: "test", costCents: 100, occurredAt: new Date() });
await expect(f.broker.read(f.principal, assignment!.id)).rejects.toThrow("execution authority");
await db.update(agents).set({ budgetMonthlyCents: 0, spentMonthlyCents: 0, status: "paused" }).where(eq(agents.id, f.agent.id));
await db.delete(budgetPolicies).where(eq(budgetPolicies.id, budget.id));
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agent.id));
await expect(f.broker.read(f.principal, assignment!.id)).rejects.toThrow("authority");
await db.update(agents).set({ status: "active" }).where(eq(agents.id, f.agent.id));
await db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, issue!.id));
@@ -38,6 +38,8 @@ import {
applyPendingMigrations,
createDb,
companies,
budgetPolicies,
costEvents,
agents,
chatEndpoints,
chatConversations,
@@ -1127,10 +1129,10 @@ describe("AgentMail durable email pipeline", () => {
chat.publishComment(f.endpointId, conversation.id, comment.id),
).rejects.toThrow(/explicit email/);
await chat.shutdown();
await db
.update(agents)
.set({ budgetMonthlyCents: 100, spentMonthlyCents: 100 })
.where(eq(agents.id, f.agentId));
await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "agent", scopeId: f.agentId,
windowKind: "calendar_month_utc", amount: 100 });
await db.insert(costEvents).values({ companyId: f.companyId, agentId: f.agentId,
provider: "test", model: "test", costCents: 100, occurredAt: new Date() });
await f.service.tick();
expect(f.sends).toHaveLength(0);
expect((await f.service.publication(queued.id, f.companyId)).outcome).toBe(
+123 -2
View File
@@ -3,14 +3,17 @@ import { randomUUID } from "node:crypto";
import { mkdtemp, realpath, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { and, eq } from "drizzle-orm";
import { and, eq, sql } from "drizzle-orm";
import {
createDb,
companies,
projects,
agents,
companyMemberships,
companyFastResponses,
costEvents,
costAdjustments,
activityLog,
fastResponseRequests,
connectionGrants,
connectionGrantMembers,
@@ -36,7 +39,10 @@ import {
import { instanceSettingsService } from "../services/instance-settings.js";
import { settleConversationTurn } from "../services/agent-conversations.js";
import { buildLowTrustSourceTrust, LOW_TRUST_QUARANTINED_BODY } from "../services/source-trust.js";
import { deliverBudgetEnforcement } from "../services/budgets.js";
import { deliverBudgetEnforcement, flushBudgetEvaluationWork } from "../services/budgets.js";
import { withAccountingTransaction } from "../services/accounting-transaction.js";
import { deleteAgentDependencies } from "../services/agent-deletion.js";
import { billingReconciliationService } from "../services/billing-reconciliation.js";
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined,
db: ReturnType<typeof createDb>,
home: string;
@@ -187,6 +193,7 @@ describe("fast response accepted turns", () => {
const events = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({ usageKind: "fast_response", costCents: 2 });
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
const [policy] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.companyId, f.companyId));
expect(policy.enforcementVersion).toBeGreaterThan(policy.enforcementDeliveredVersion);
delivery = deliverBudgetEnforcement(db, { cancelWorkForScope }, f.companyId);
@@ -196,6 +203,118 @@ describe("fast response accepted turns", () => {
expect((await db.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id)))[0].enforcementDeliveredVersion).toBe(policy.enforcementVersion);
} finally { release(); await pending.catch(() => undefined); await delivery?.catch(() => undefined); }
});
it.each(["company", "agent", "policy"] as const)("settles while an unrelated %s accounting row is locked", async locked => {
const f = await fixture(), request = await f.enqueue();
const [policy] = await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
windowKind: "calendar_month_utc", amount: 100, notifyEnabled: false }).returning();
let providerEntered!: () => void, finishProvider!: () => void, unlock!: () => void, lockedRow!: () => void;
const entered = new Promise<void>(resolve => { providerEntered = resolve; });
const providerGate = new Promise<void>(resolve => { finishProvider = resolve; });
const lockGate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<void>(resolve => { lockedRow = resolve; });
f.provider.mockImplementationOnce(async () => { providerEntered(); await providerGate; return structuredClone(outcome); });
const pending = f.service.process(request);
await entered;
const blocker = db.transaction(async tx => {
if (locked === "company") await tx.select().from(companies).where(eq(companies.id, f.companyId)).for("no key update");
if (locked === "agent") await tx.select().from(agents).where(eq(agents.id, f.agentId)).for("no key update");
if (locked === "policy") await tx.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id)).for("no key update");
lockedRow(); await lockGate;
});
let finished = false;
void pending.finally(() => { finished = true; }).catch(() => undefined);
try {
await acquired; finishProvider();
await vi.waitFor(() => expect(finished).toBe(true), { timeout: 3000 });
expect(await pending).toMatchObject({ status: "succeeded", text: outcome.text });
expect((await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId)))[0].state).toBe("settled");
} finally { unlock(); finishProvider(); await blocker; await pending.catch(() => undefined); }
});
it.each(["company", "project"] as const)("denies paid admission when %s manual pause wins its control fence", async scope => {
const f = await fixture();
const [project] = await db.insert(projects).values({ companyId: f.companyId, name: "Control fence" }).returning();
await db.update(issues).set({ projectId: project.id }).where(eq(issues.id, f.issueId));
const request = await f.enqueue();
let unlock!: () => void, locked!: (pid: number) => void;
const gate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
const holding = db.transaction(async tx => {
if (scope === "company") await tx.select().from(companies).where(eq(companies.id, f.companyId)).for("no key update");
else await tx.select().from(projects).where(eq(projects.id, project.id)).for("no key update");
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
locked(row.pid); await gate;
if (scope === "company") await tx.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() }).where(eq(companies.id, f.companyId));
else await tx.update(projects).set({ pauseReason: "manual", pausedAt: new Date() }).where(eq(projects.id, project.id));
});
const pid = await acquired;
const pending = f.service.process(request);
try {
await vi.waitFor(async () => {
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`);
expect(row.count).toBeGreaterThan(0);
}, { timeout: 3000 });
unlock(); await holding;
expect(await pending).toMatchObject({ status: "unavailable" });
expect(f.provider).not.toHaveBeenCalled();
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(0);
expect((await db.select().from(fastResponseRequests).where(eq(fastResponseRequests.id, request.id)))[0].status).toBe("skipped");
} finally { unlock(); await holding; await pending.catch(() => undefined); }
});
it.each([
{ recoveryFirst: true, priced: true, corrected: false }, { recoveryFirst: false, priced: true, corrected: false },
{ recoveryFirst: true, priced: false, corrected: false }, { recoveryFirst: false, priced: false, corrected: false },
{ recoveryFirst: true, priced: true, corrected: true },
])("keeps a real result with recovery first: $recoveryFirst, priced: $priced, corrected: $corrected", async ({ recoveryFirst, priced, corrected }) => {
const f = await fixture(), request = await f.enqueue();
const realOutcome: FastResponseOutcome = priced ? structuredClone(outcome) : { ...structuredClone(outcome), receipt: {
...outcome.receipt, costCents: null, costStatus: "unpriced", pricingProvenance: { source: "unknown" } } };
let finishProvider!: () => void, providerEntered!: () => void;
const gate = new Promise<void>(resolve => { finishProvider = resolve; });
const entered = new Promise<void>(resolve => { providerEntered = resolve; });
f.provider.mockImplementationOnce(async () => { providerEntered(); await gate; return realOutcome; });
const pending = f.service.process(request);
await entered;
await db.update(fastResponseRequests).set({ expiresAt: new Date(Date.now() - 60_000) }).where(eq(fastResponseRequests.id, request.id));
let originalHash: string | null = null;
try {
if (recoveryFirst) {
await Promise.all([f.service.sweepPending(), f.service.sweepPending()]);
const [event] = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(event.costStatus).toBe("unpriced"); originalHash = event.receiptHash;
if (corrected) await billingReconciliationService(db).adjust(f.companyId, event.id, { idempotencyKey: "invoice-before-receipt",
expectedCents: "0", correctedCents: "0.005", reason: "Verified invoice before provider completed", pricing: { source: "operator" } }, "alice");
}
finishProvider();
expect(await pending).toMatchObject({ status: "succeeded", text: outcome.text });
await f.service.sweepPending();
const events = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({ costStatus: priced ? "reported" : "unpriced", costCents: corrected ? 0.005 : priced ? 0.001 : 0 });
expect(events[0]).toMatchObject({ inputTokens: 80, cachedInputTokens: 0, outputTokens: 16,
providerRequestId: outcome.receipt.providerRequestId });
const corrections = await db.select().from(costAdjustments).where(eq(costAdjustments.companyId, f.companyId));
expect(corrections).toHaveLength(recoveryFirst ? corrected ? 2 : 1 : 0);
if (recoveryFirst) {
expect(events[0]).toMatchObject({ receiptHash: originalHash, reportedCostCents: "0.0000000",
reportedUsageJson: { inputTokens: 0, cachedInputTokens: 0, outputTokens: 0, providerRequestId: null } });
expect(corrections.find(row => row.idempotencyKey.endsWith(":late-receipt") || row.idempotencyKey.endsWith(":late-usage"))?.previousUsageJson)
.toMatchObject({ inputTokens: 0, cachedInputTokens: 0, outputTokens: 0, providerRequestId: null });
if (priced && !corrected) expect(JSON.parse(corrections[0].pricing.evidence as string)).toMatchObject({ inputTokens: 80, outputTokens: 16 });
}
if (corrected) {
expect(events[0].pricingProvenance).toEqual({ source: "operator" });
const observed = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.companyId), eq(activityLog.action, "accounting.late_receipt_observed")));
expect(observed).toHaveLength(1);
expect(observed[0].details).toMatchObject({ effectiveCostCents: "0.0050000", costCents: "0.0010000", inputTokens: 80, outputTokens: 16 });
}
expect((await db.select().from(fastResponseRequests).where(eq(fastResponseRequests.id, request.id)))[0]).toMatchObject({
status: "succeeded", costEventId: events[0].id, inputTokens: 80, outputTokens: 16 });
expect((await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId)))[0].state).toBe(priced ? "settled" : "held");
expect(await db.select().from(issueComments).where(and(eq(issueComments.companyId, f.companyId), eq(issueComments.origin, "fast_response"))))
.toHaveLength(recoveryFirst ? 0 : 1);
expect(f.provider).toHaveBeenCalledOnce();
} finally { finishProvider(); await pending.catch(() => undefined); }
});
it("does not enqueue or infer while the experimental setting is disabled", async () => {
const f = await fixture();
await instanceSettingsService(db).updateExperimental({ enableFastResponses: false });
@@ -511,6 +630,8 @@ describe("fast response accepted turns", () => {
.where(eq(budgetReservations.fastResponseRequestId, row.id))
)[0].state,
).toBe("held");
await expect(withAccountingTransaction(db, f.companyId, tx => deleteAgentDependencies(tx, f.companyId, f.agentId)))
.rejects.toMatchObject({ status: 409, details: { code: "agent_fast_response_accounting_pending" } });
});
it("sponsors accepted unlinked senders using only destination-authorized context", async () => {
const f = await fixture();
@@ -8,7 +8,7 @@ import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database
import { issueService } from "../services/issues.js";
import { materializeNativeProviderCapacityRetry } from "../services/native-runtime/native-provider-capacity-retry.js";
import { agentService } from "../services/agents.js";
import { budgetService } from "../services/budgets.js";
import { budgetService, flushBudgetEvaluationWork } from "../services/budgets.js";
import { heartbeatService } from "../services/heartbeat.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
@@ -191,6 +191,7 @@ describe.skipIf(!support.supported)("run dispatch production wiring (controlled
provider: "fixture", biller: "fixture", model: "controlled", inputTokens: 0, outputTokens: 0,
costCents: 125, occurredAt: new Date() }).returning();
await budgets.evaluateCostEvent(cost);
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0].status).toBe("paused");
const [queued] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
status: "queued", responsibleUserId: "dispatch-user", invocationSource: "automation", contextSnapshot: {} }).returning();
@@ -4,9 +4,9 @@ import { mkdtemp, realpath, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { createDb, companies, agents, companyMemberships, companyPromptInjectionPrevention, costEvents, promptInjectionInvocations,
import { createDb, companies, agents, companyMemberships, companyPromptInjectionPrevention, costEvents, costAdjustments, promptInjectionInvocations,
connectionGrants, connectionGrantMembers, toolConnectionInstalls, toolConnections, budgetPolicies, budgetReservations,
issues, projects, secretAccessEvents, companySecrets } from "@paperclipai/db";
issues, projects, secretAccessEvents, companySecrets, activityLog } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";
import { PROMPT_INJECTION_DEFAULT_MODEL, updatePromptInjectionPreventionSchema } from "@paperclipai/shared";
import { aiConnectionService } from "../services/ai-connections.js";
@@ -24,7 +24,7 @@ import { errorHandler } from "../middleware/index.js";
import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
import { SecretProviderClientError } from "../secrets/types.js";
import { validateToolContent } from "../services/tool-content-guards.js";
import { deliverBudgetEnforcement } from "../services/budgets.js";
import { deliverBudgetEnforcement, flushBudgetEvaluationWork } from "../services/budgets.js";
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>, db: ReturnType<typeof createDb>, home: string;
beforeAll(async () => {
home = await realpath(await mkdtemp(path.join(os.tmpdir(), "paperclip-injection-")));
@@ -71,6 +71,7 @@ describe("prompt injection service", () => {
const events = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({ usageKind: "prompt_injection", costCents: 2 });
await flushBudgetEvaluationWork(db, { companyId: f.companyId });
const [policy] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.companyId, f.companyId));
expect(policy.enforcementVersion).toBeGreaterThan(policy.enforcementDeliveredVersion);
delivery = deliverBudgetEnforcement(db, { cancelWorkForScope }, f.companyId);
@@ -80,6 +81,113 @@ describe("prompt injection service", () => {
expect((await db.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id)))[0].enforcementDeliveredVersion).toBe(policy.enforcementVersion);
} finally { release(); await pending.catch(() => undefined); await delivery?.catch(() => undefined); }
});
it.each(["company", "agent", "policy"] as const)("settles while an unrelated %s accounting row is locked", async locked => {
const f = await fixture();
const [policy] = await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
windowKind: "calendar_month_utc", amount: 100, notifyEnabled: false }).returning();
let providerEntered!: () => void, finishProvider!: () => void, unlock!: () => void, lockedRow!: () => void;
const entered = new Promise<void>(resolve => { providerEntered = resolve; });
const providerGate = new Promise<void>(resolve => { finishProvider = resolve; });
const lockGate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<void>(resolve => { lockedRow = resolve; });
f.provider.mockImplementationOnce(async () => { providerEntered(); await providerGate; return structuredClone(outcome); });
const pending = f.service.classify({ ...f.context, source: "gateway", agentId: f.agentId }, "sample");
await entered;
const blocker = db.transaction(async tx => {
if (locked === "company") await tx.select().from(companies).where(eq(companies.id, f.companyId)).for("no key update");
if (locked === "agent") await tx.select().from(agents).where(eq(agents.id, f.agentId)).for("no key update");
if (locked === "policy") await tx.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id)).for("no key update");
lockedRow(); await lockGate;
});
let finished = false;
void pending.finally(() => { finished = true; }).catch(() => undefined);
try {
await acquired; finishProvider();
await vi.waitFor(() => expect(finished).toBe(true), { timeout: 3000 });
expect(await pending).toMatchObject({ status: "classified", violation: false });
expect((await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId)))[0].state).toBe("settled");
} finally { unlock(); finishProvider(); await blocker; await pending.catch(() => undefined); }
});
it.each(["company", "project"] as const)("denies paid admission when %s manual pause wins its control fence", async scope => {
const f = await fixture();
const [project] = await db.insert(projects).values({ companyId: f.companyId, name: "Control fence" }).returning();
let unlock!: () => void, locked!: (pid: number) => void;
const gate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
const holding = db.transaction(async tx => {
if (scope === "company") await tx.select().from(companies).where(eq(companies.id, f.companyId)).for("no key update");
else await tx.select().from(projects).where(eq(projects.id, project.id)).for("no key update");
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
locked(row.pid); await gate;
if (scope === "company") await tx.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() }).where(eq(companies.id, f.companyId));
else await tx.update(projects).set({ pauseReason: "manual", pausedAt: new Date() }).where(eq(projects.id, project.id));
});
const pid = await acquired;
const pending = f.service.classify({ ...f.context, projectId: project.id }, "sample");
try {
await vi.waitFor(async () => {
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`);
expect(row.count).toBeGreaterThan(0);
}, { timeout: 3000 });
unlock(); await holding;
expect(await pending).toMatchObject({ status: "unavailable" });
expect(f.provider).not.toHaveBeenCalled();
expect(await db.select().from(promptInjectionInvocations).where(eq(promptInjectionInvocations.companyId, f.companyId))).toHaveLength(0);
} finally { unlock(); await holding; await pending.catch(() => undefined); }
});
it.each([
{ recoveryFirst: true, priced: true, corrected: false }, { recoveryFirst: false, priced: true, corrected: false },
{ recoveryFirst: true, priced: false, corrected: false }, { recoveryFirst: false, priced: false, corrected: false },
{ recoveryFirst: true, priced: true, corrected: true },
])("keeps a real result with recovery first: $recoveryFirst, priced: $priced, corrected: $corrected", async ({ recoveryFirst, priced, corrected }) => {
const f = await fixture();
const realOutcome: PromptInjectionProviderOutcome = priced ? structuredClone(outcome) : { ...structuredClone(outcome), receipt: {
...outcome.receipt, costCents: null, costStatus: "unpriced", pricingProvenance: { source: "unknown" } } };
let finishProvider!: () => void, providerEntered!: () => void;
const gate = new Promise<void>(resolve => { finishProvider = resolve; });
const entered = new Promise<void>(resolve => { providerEntered = resolve; });
f.provider.mockImplementationOnce(async () => { providerEntered(); await gate; return realOutcome; });
const pending = f.service.classify(f.context, "sample");
await entered;
await db.update(promptInjectionInvocations).set({ startedAt: new Date(Date.now() - 180_000) }).where(eq(promptInjectionInvocations.companyId, f.companyId));
let originalHash: string | null = null;
try {
if (recoveryFirst) {
await Promise.all([f.service.recoverInterrupted(), f.service.recoverInterrupted()]);
const [event] = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(event.costStatus).toBe("unpriced"); originalHash = event.receiptHash;
if (corrected) await billingReconciliationService(db).adjust(f.companyId, event.id, { idempotencyKey: "invoice-before-receipt",
expectedCents: "0", correctedCents: "0.005", reason: "Verified invoice before provider completed", pricing: { source: "operator" } }, "alice");
}
finishProvider();
expect(await pending).toMatchObject({ status: "classified", violation: false });
await f.service.recoverInterrupted();
const events = await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId));
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({ costStatus: priced ? "reported" : "unpriced", costCents: corrected ? 0.005 : priced ? 0.002 : 0 });
expect(events[0]).toMatchObject({ inputTokens: 100, cachedInputTokens: 0, outputTokens: 10,
providerRequestId: "fixture-request" });
const corrections = await db.select().from(costAdjustments).where(eq(costAdjustments.companyId, f.companyId));
expect(corrections).toHaveLength(recoveryFirst ? corrected ? 2 : 1 : 0);
if (recoveryFirst) {
expect(events[0]).toMatchObject({ receiptHash: originalHash, reportedCostCents: "0.0000000",
reportedUsageJson: { inputTokens: 0, cachedInputTokens: 0, outputTokens: 0, providerRequestId: null } });
expect(corrections.find(row => row.idempotencyKey.endsWith(":late-receipt") || row.idempotencyKey.endsWith(":late-usage"))?.previousUsageJson)
.toMatchObject({ inputTokens: 0, cachedInputTokens: 0, outputTokens: 0, providerRequestId: null });
if (priced && !corrected) expect(JSON.parse(corrections[0].pricing.evidence as string)).toMatchObject({ inputTokens: 100, outputTokens: 10, providerRequestId: "fixture-request" });
}
if (corrected) {
expect(events[0].pricingProvenance).toEqual({ source: "operator" });
const observed = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.companyId), eq(activityLog.action, "accounting.late_receipt_observed")));
expect(observed).toHaveLength(1);
expect(observed[0].details).toMatchObject({ effectiveCostCents: "0.0050000", costCents: "0.0020000", inputTokens: 100, providerRequestId: "fixture-request" });
}
expect((await db.select().from(promptInjectionInvocations).where(eq(promptInjectionInvocations.companyId, f.companyId)))[0]).toMatchObject({
status: "succeeded", violation: false, costEventId: events[0].id, inputTokens: 100, outputTokens: 10 });
expect((await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId)))[0].state).toBe(priced ? "settled" : "held");
expect(f.provider).toHaveBeenCalledOnce();
} finally { finishProvider(); await pending.catch(() => undefined); }
});
it.each(["missing_key", "wrong_key", "remote_vault_outage"])("uses full regex fallback for %s without dispatch or charge", async failure => {
const f = await fixture();
const previousKey = process.env.PAPERCLIP_SECRETS_MASTER_KEY;
@@ -187,15 +295,17 @@ describe("prompt injection service", () => {
await f.service.configure(f.companyId, "alice", { ...f.config, enabled: false });
expect(await f.service.classify(f.context, "sample")).toMatchObject({ reason: "disabled" }); expect(f.provider).not.toHaveBeenCalled();
});
it.each(["disabled", "rotated", "configuration"])("rejects %s changes while admission waits for accounting", async change => {
it.each(["disabled", "rotated", "configuration"])("rejects %s changes while final admission waits for configuration", async change => {
const f = await fixture();
const [secret] = await db.select().from(companySecrets).where(eq(companySecrets.companyId, f.companyId));
let release!: () => void, locked!: (pid: number) => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
const holding = withAccountingTransaction(db, f.companyId, async tx => {
const holding = db.transaction(async tx => {
await tx.select().from(companyPromptInjectionPrevention).where(eq(companyPromptInjectionPrevention.companyId, f.companyId)).for("no key update");
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
locked(row.pid); await gate;
if (change === "configuration") await tx.update(companyPromptInjectionPrevention).set({ model: "changed/model" }).where(eq(companyPromptInjectionPrevention.companyId, f.companyId));
});
const pid = await acquired;
const pending = f.service.classify(f.context, "sample");
@@ -204,10 +314,7 @@ describe("prompt injection service", () => {
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`);
expect(row.count).toBeGreaterThan(0);
}, { timeout: 5000 });
if (change === "configuration") {
// A timestamp alone is not a sufficient revision if two writes share a millisecond.
await db.update(companyPromptInjectionPrevention).set({ model: "changed/model" }).where(eq(companyPromptInjectionPrevention.companyId, f.companyId));
} else {
if (change !== "configuration") {
await db.update(companySecrets).set(change === "disabled" ? { status: "disabled" } : { latestVersion: secret.latestVersion + 1 }).where(eq(companySecrets.id, secret.id));
}
} finally { release(); await holding; }
@@ -8,7 +8,7 @@ import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi }
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { agentService } from "../services/agents.js";
import { approvalService } from "../services/approvals.js";
import { budgetService } from "../services/budgets.js";
import { budgetService, flushBudgetEvaluationWork } from "../services/budgets.js";
import { projectService } from "../services/projects.js";
import { recordResourceCreationEvent } from "../services/resource-lifecycle-events.js";
import { pluginLifecycleInbox } from "../services/plugin-lifecycle-inbox.js";
@@ -154,9 +154,12 @@ describePostgres("Resource lifecycle events", () => {
await db.insert(budgetPolicies).values({ companyId, scopeType: "agent", scopeId: agent.id, metric: "billed_cents", windowKind: "calendar_month_utc", amount: 100, notifyEnabled: false });
const [event] = await db.insert(costEvents).values({ companyId, agentId: agent.id, provider: "fixture", model: "fixture", costCents: 150, occurredAt: new Date() }).returning();
await service.evaluateCostEvent(event);
await flushBudgetEvaluationWork(db, { companyId });
await service.evaluateCostEvent(event);
await flushBudgetEvaluationWork(db, { companyId });
expect(await agentService(db).getById(agent.id)).toMatchObject({ status: "paused", pauseReason: "budget" });
await service.upsertPolicy(companyId, { scopeType: "agent", scopeId: agent.id, amount: 200 }, "fixture-board");
await flushBudgetEvaluationWork(db, { companyId });
await worker.process(agent.id);
expect(await agentService(db).getById(agent.id)).toMatchObject({ status: "idle", pauseReason: null });
expect((await events()).sort((a, b) => a.id - b.id).map(row => row.action)).toEqual(["create", "pause", "resume"]);
@@ -1,4 +1,4 @@
import { budgetService } from "../services/budgets.js";
import { budgetService, flushBudgetEvaluationWork } from "../services/budgets.js";
import { documentService } from "../services/documents.js";
import { projectService } from "../services/projects.js";
import * as companySearch from "../services/company-search.js";
@@ -870,6 +870,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
const [cost] = await db.insert(costEvents).values({ companyId, agentId, provider: "openai", biller: "openai", billingType: "metered_api", model: "test",
costCents: 200, occurredAt: new Date() }).returning();
await budgets.evaluateCostEvent(cost!);
await flushBudgetEvaluationWork(db, { companyId });
expect((await db.select().from(companies).where(eq(companies.id, companyId)))[0]?.status).toBe("paused");
await quiet();
await issueService(db).create(companyId, { title: "Paused blocker", status: "blocked", priority: "medium" }); await quiet();
+3 -4
View File
@@ -1,3 +1,4 @@
import { budgetServiceInTransaction } from "../../services/budgets.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../services/delivery-work-notifications.js";
import { withExternalAdmissionGuard, withExternalExecutionAdmission } from "./admission.js";
import { createNativeExternalBroker } from "./native-broker.js";
@@ -82,8 +83,7 @@ function createBroker(db: Db) {
|| !coordinator || coordinator.controllerGeneration !== a.controllerGeneration || !coordinator.leaseExpiresAt || coordinator.leaseExpiresAt <= new Date()
|| ["terminal_failure", "turn_stopping", "turn_stopped"].includes(run.nativePhase ?? "")
|| !company || company.status !== "active" || !agent || ["paused", "terminated", "pending_approval"].includes(agent.status)
|| (company.budgetMonthlyCents > 0 && company.spentMonthlyCents >= company.budgetMonthlyCents)
|| (agent.budgetMonthlyCents > 0 && agent.spentMonthlyCents >= agent.budgetMonthlyCents)) {
|| await budgetServiceInTransaction(queryDb).getInvocationBlock(a.companyId, a.agentId, { issueId: run.nativeIssueId })) {
throw fail("The run no longer owns this task or its execution authority.");
}
return { binding, assignment: a };
@@ -342,8 +342,7 @@ function createBroker(db: Db) {
const [company] = await tx.select().from(companies).where(eq(companies.id, b.companyId));
const [agent] = await tx.select().from(agents).where(eq(agents.id, b.agentId));
if (company?.status !== "active" || !agent || ["paused", "terminated", "pending_approval"].includes(agent.status)
|| (company.budgetMonthlyCents > 0 && company.spentMonthlyCents >= company.budgetMonthlyCents)
|| (agent.budgetMonthlyCents > 0 && agent.spentMonthlyCents >= agent.budgetMonthlyCents)) throw fail("Company, agent, or budget authority is unavailable.");
|| await budgetServiceInTransaction(tx as unknown as Db).getInvocationBlock(b.companyId, b.agentId, { projectId })) throw fail("Company, agent, or budget authority is unavailable.");
const decision = await authorizationService(tx as unknown as Db).decide({
actor: { type: "agent", source: "agent_jwt", companyId: b.companyId, agentId: b.agentId, onBehalfOfUserId: b.operatorId },
action: "tasks:assign", resource: { type: "issue", companyId: b.companyId, assigneeAgentId: b.agentId, assigneeUserId: null, projectId: projectId ?? null, parentIssueId: null },
+2 -1
View File
@@ -1,3 +1,4 @@
import { budgetServiceInTransaction } from "../../services/budgets.js";
import { pendingMuseTerminalAssignmentCondition } from "./muse-terminal-settlement.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../services/delivery-work-notifications.js";
import { createHash, randomBytes, randomUUID } from "node:crypto";
@@ -102,7 +103,7 @@ function createMuseBroker(db:Db, publicOrigin?:string) {
const owns=run&&issue&&(reviewContext?!!review&&(review.interaction.status!=="pending"||issue.executionRunId===run.id):issue.assigneeAgentId===a.agentId&&[issue.checkoutRunId,issue.executionRunId].includes(run.id));
if(!run||run.status!=="running"||run.runtimeMode!=="native"||run.nativeSessionId!==a.normalizedSessionId||!owns||!owner||owner.controllerGeneration!==a.controllerGeneration||!owner.leaseExpiresAt||owner.leaseExpiresAt<=new Date()
||["terminal_failure","turn_stopping","turn_stopped"].includes(run.nativePhase??"")||company?.status!=="active"||!agent||["paused","terminated","pending_approval"].includes(agent.status)
||(company.budgetMonthlyCents>0&&company.spentMonthlyCents>=company.budgetMonthlyCents)||(agent.budgetMonthlyCents>0&&agent.spentMonthlyCents>=agent.budgetMonthlyCents)) throw forbidden("Native run no longer owns execution authority.");
||await budgetServiceInTransaction(db).getInvocationBlock(a.companyId,a.agentId,{issueId:run.nativeIssueId})) throw forbidden("Native run no longer owns execution authority.");
return {b,a,run};
}
async function fence(tx:ExternalAdmissionTransaction,b:typeof bindings.$inferSelect,reason:string,assignmentId?:string) {
@@ -1,3 +1,4 @@
import { budgetServiceInTransaction } from "../../services/budgets.js";
import { notifyExternalAgentSettlement } from "./muse-terminal-settlement.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../services/delivery-work-notifications.js";
import { randomUUID } from "node:crypto";
@@ -76,7 +77,7 @@ export function createNativeExternalBroker<S>(db:Db,provider:Provider,identity:I
if(!grant||grant.companyId!==b.companyId||grant.agentId!==b.agentId||grant.userId!==b.operatorId)throw forbidden("External agent grant authority is unavailable.");
if(actualProvider==="grok_bot")await authorizeExternalBotOperator(database as Db,b.companyId,b.operatorId);
}
if((company.budgetMonthlyCents>0&&company.spentMonthlyCents>=company.budgetMonthlyCents)||(agent.budgetMonthlyCents>0&&agent.spentMonthlyCents>=agent.budgetMonthlyCents))throw forbidden("External execution budget is exhausted.");
if(await budgetServiceInTransaction(database as Db).getInvocationBlock(b.companyId,b.agentId))throw forbidden("External execution budget is exhausted.");
}
async function authorizeController(database:Database,a:Assignment,generation:number) {
const [owner]=await database.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.runId,a.runId),eq(nativeRunFinalizations.companyId,a.companyId),eq(nativeRunFinalizations.controllerGeneration,generation),gt(nativeRunFinalizations.leaseExpiresAt,new Date())));
@@ -10,6 +10,7 @@ import {
budgetIncidents,
budgetPolicies,
budgetReservations,
budgetEvaluationWork,
companies,
costEvents,
createDb,
@@ -36,7 +37,7 @@ import { createPostgresRunDispatchAdapter } from "./postgres.js";
import { settleUnrecoverableExecutions } from "../../../services/execution-recovery-resolution.js";
import { getExecutionBlocker } from "../../../services/execution-blocker.js";
import { reserveRunBudget } from "../../../services/budget-reservations.js";
import { budgetService, recheckAdmittedRunBudgetReadOnly } from "../../../services/budgets.js";
import { budgetService, flushBudgetEvaluationWork, recheckAdmittedRunBudgetReadOnly } from "../../../services/budgets.js";
import { costService } from "../../../services/costs.js";
// Proves the DB-to-facts mapping this adapter owns for each state the two
@@ -483,8 +484,10 @@ describeEmbeddedPostgres("run-dispatch postgres adapter", () => {
await Promise.all([arrived, handoffReady]);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(run.usageJson?.budgetDispatch).toMatchObject({ reservationId: reservation.id, recoveryLeaseOwner: null });
// This charge serializes after dispatch admission. It creates stop intent
// for already-admitted work without waiting for the provider's response.
// This charge commits after dispatch admission. It durably queues budget
// evaluation without waiting for the provider's response or ownership tx.
const beforeInputs = await db.select({ id: budgetEvaluationWork.id }).from(budgetEvaluationWork)
.where(eq(budgetEvaluationWork.companyId, companyId));
let charged = false;
charge = costService(db).createEvent(companyId, { agentId, provider: "fixture", model: "post-admission-charge",
costCents: 100, inputTokens: 1, outputTokens: 1, occurredAt: new Date() }).then(() => { charged = true; });
@@ -492,8 +495,9 @@ describeEmbeddedPostgres("run-dispatch postgres adapter", () => {
// Phase two still owns issue/run locks and the provider is waiting.
// Holding company accounting here would stall this independent charge.
await vi.waitFor(() => expect(charged).toBe(true), { timeout: 2_000 });
const [policy] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.companyId, companyId));
expect(policy.enforcementVersion).toBeGreaterThan(policy.enforcementDeliveredVersion);
const inputs = await db.select({ id: budgetEvaluationWork.id }).from(budgetEvaluationWork)
.where(eq(budgetEvaluationWork.companyId, companyId));
expect(inputs.length).toBeGreaterThan(beforeInputs.length);
expect(requests).toBe(1);
commitHandoff();
const gate = await admission;
@@ -501,6 +505,9 @@ describeEmbeddedPostgres("run-dispatch postgres adapter", () => {
expect(gate.dispatched).toBe(true);
if (!gate.dispatched) throw new Error("Expected committed provider admission");
result = gate.resultPromise;
await flushBudgetEvaluationWork(db, { companyId });
const [policy] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.companyId, companyId));
expect(policy.enforcementVersion).toBeGreaterThan(policy.enforcementDeliveredVersion);
finishProvider();
expect(await result).toBe("provider result");
} finally {
@@ -784,21 +791,24 @@ describeEmbeddedPostgres("run-dispatch postgres adapter", () => {
expect(dispatch).not.toHaveBeenCalled();
});
it("commits a discovered budget stop even when final handoff is denied", async () => {
it("denies an overrun at final handoff and leaves its budget stop to the worker", async () => {
const { companyId, agentId } = await seedCompanyAndAgent();
const runId = await seedRun({ companyId, agentId, status: "running" });
await budgetService(db).upsertPolicy(companyId, { scopeType: "company", scopeId: companyId,
metric: "billed_cents", windowKind: "lifetime", amount: 100, notifyEnabled: false }, null);
const reservation = await reserveRunBudget(db, companyId, runId, null);
// Seed committed historical spend whose lifecycle projection has not yet
// been repaired. The gate must persist that repair rather than roll it back.
await db.insert(costEvents).values({ companyId, agentId, provider: "test", model: "unprojected-charge",
// The committed charge queues enforcement. Admission reads the overrun
// immediately, while lifecycle transitions remain owned by the worker.
await costService(db).createEvent(companyId, { agentId, provider: "test", model: "unprojected-charge",
inputTokens: 1, outputTokens: 1, costCents: 110, occurredAt: new Date() });
const dispatch = vi.fn(async () => undefined);
expect(await createPostgresRunDispatchAdapter(db).dispatchResolvedInteractionIfCurrent({
companyId, runId, reservationId: reservation.id, expectedStatus: "running", now: new Date(), dispatch,
})).toMatchObject({ dispatched: false, admissionBlockReason: expect.any(String) });
expect(dispatch).not.toHaveBeenCalled();
expect((await db.select().from(companies).where(eq(companies.id, companyId)))[0])
.toMatchObject({ status: "active", pauseReason: null });
await flushBudgetEvaluationWork(db, { companyId });
expect((await db.select().from(companies).where(eq(companies.id, companyId)))[0])
.toMatchObject({ status: "paused", pauseReason: "budget" });
const [policy] = await db.select().from(budgetPolicies).where(eq(budgetPolicies.companyId, companyId));
@@ -31,7 +31,8 @@ import {
projects,
} from "@paperclipai/db";
import { ISSUE_DISPOSITION_REPAIR_RETRY_REASON, isUuidLike } from "@paperclipai/shared";
import { withAccountingTransaction } from "../../../services/accounting-transaction.js";
import { withOperationAccountingTransaction } from "../../../services/accounting-transaction.js";
import { lockBudgetAdmissionControls } from "../../../services/budget-admission-controls.js";
import type { ActivityPublication } from "../../../services/activity-log.js";
import { parseObject } from "../../../adapters/utils.js";
import { evaluateAgentInvokabilityFromDb } from "../../../services/agent-invokability.js";
@@ -224,8 +225,9 @@ export function createPostgresRunDispatchAdapter(
// that owns the issue lock and is waiting for the run.
for (let attempt = 0; attempt < 3; attempt += 1) {
const hint = await db
.select({ contextSnapshot: heartbeatRuns.contextSnapshot })
.select({ contextSnapshot: heartbeatRuns.contextSnapshot, agentId: heartbeatRuns.agentId, projectId: budgetReservations.projectId })
.from(heartbeatRuns)
.leftJoin(budgetReservations, and(eq(budgetReservations.runId, heartbeatRuns.id), eq(budgetReservations.companyId, heartbeatRuns.companyId)))
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)))
.then((rows) => rows[0] ?? null);
if (!hint) return onMissing();
@@ -236,8 +238,13 @@ export function createPostgresRunDispatchAdapter(
: readNonEmptyString(parseObject(hint.contextSnapshot).issueId);
const locked = async (typedTx: Db, publications: ActivityPublication[]) => {
// Recovery always takes coordinator before issue/run. Only the
// accounting phase acquires the company lock and evaluates budgets.
if (mode === "accounting") {
await lockBudgetAdmissionControls(typedTx, input.companyId, hint.projectId);
await typedTx.select({ id: agents.id }).from(agents)
.where(and(eq(agents.id, hint.agentId), eq(agents.companyId, input.companyId))).for("share");
}
// Recovery keeps coordinator -> issue -> run order after the shared
// dispatch-control fences. Settlement never uses these authority locks.
await lockRunBudgetRecoveryOwnerInTransaction(typedTx, input.companyId, input.runId, input.recoveryLeaseOwner);
if (hintedIssueId) {
await typedTx
@@ -266,7 +273,7 @@ export function createPostgresRunDispatchAdapter(
return { kind: "value" as const, value: await operation(typedTx, run, publications) };
};
const result: { kind: "missing" | "retry" } | { kind: "value"; value: T } = mode === "accounting"
? await withAccountingTransaction(db, input.companyId, locked)
? await withOperationAccountingTransaction(db, input.companyId, locked)
: await db.transaction(tx => locked(tx as unknown as Db, []));
if (result.kind === "missing") return onMissing();
@@ -0,0 +1,86 @@
import { and, eq, sql } from "drizzle-orm";
import { costAdjustments, costEvents, decisionInvocations, promptInjectionInvocations, fastResponseRequests, heartbeatRuns, issues, type Db } from "@paperclipai/db";
import { normalizeCents } from "@paperclipai/shared";
import { conflict, notFound } from "../errors.js";
import { receiptFingerprint } from "./receipt-fingerprint.js";
import { logActivity, type ActivityPublication } from "./activity-log.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export interface CostCorrectionInput {
companyId: string;
eventId: string;
idempotencyKey: string;
expectedCents: string;
correctedCents: string;
reason: string;
pricing: Record<string, unknown>;
actorId: string;
invoiceLineId?: string | null;
receiptHash?: string;
costStatus?: "reported" | "estimated" | "unpriced";
receiptUsage?: { inputTokens: number; cachedInputTokens: number; outputTokens: number; providerRequestId: string | null };
}
/** Manual correction follows the same owner-before-ledger order as settlement. */
export async function lockCostCorrectionOwner(tx: Db, companyId: string, eventId: string) {
const [event] = await tx.select().from(costEvents).where(and(eq(costEvents.id, eventId), eq(costEvents.companyId, companyId)));
if (!event) throw notFound("Cost event not found");
if (event.issueId) await tx.select({ id: issues.id }).from(issues)
.where(and(eq(issues.id, event.issueId), eq(issues.companyId, companyId))).for("key share");
if (event.usageKind === "decision") await tx.select({ id: decisionInvocations.id }).from(decisionInvocations)
.where(and(eq(decisionInvocations.costEventId, eventId), eq(decisionInvocations.companyId, companyId))).for("update");
else if (event.usageKind === "prompt_injection") await tx.select({ id: promptInjectionInvocations.id }).from(promptInjectionInvocations)
.where(and(eq(promptInjectionInvocations.costEventId, eventId), eq(promptInjectionInvocations.companyId, companyId))).for("update");
else if (event.usageKind === "fast_response") await tx.select({ id: fastResponseRequests.id }).from(fastResponseRequests)
.where(and(eq(fastResponseRequests.costEventId, eventId), eq(fastResponseRequests.companyId, companyId))).for("update");
else if (event.heartbeatRunId) await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, event.heartbeatRunId), eq(heartbeatRuns.companyId, companyId))).for("update");
}
/** Caller validates receipt/invoice authority and acquires its operation fence
* before this ledger row. Evidence and the original receipt hash are retained. */
export async function applyCostCorrectionInTransaction(tx: Db, publications: ActivityPublication[], input: CostCorrectionInput) {
const { companyId, eventId } = input;
const expectedCents = normalizeCents(input.expectedCents);
const correctedCents = normalizeCents(input.correctedCents);
const hash = input.receiptHash ?? receiptFingerprint({ eventId, expectedCents, correctedCents,
reason: input.reason, pricing: input.pricing, invoiceLineId: input.invoiceLineId ?? null, costStatus: input.costStatus,
receiptUsage: input.receiptUsage });
const [existing] = await tx.select().from(costAdjustments).where(and(
eq(costAdjustments.companyId, companyId), eq(costAdjustments.idempotencyKey, input.idempotencyKey)));
if (existing) {
if (existing.receiptHash !== hash) throw conflict("Correction key already has different contents");
return existing;
}
const [event] = await tx.select({ event: costEvents, exact: sql<string>`${costEvents.costCents}::text` }).from(costEvents)
.where(and(eq(costEvents.companyId, companyId), eq(costEvents.id, eventId))).for("update");
if (!event) throw notFound("Cost event not found");
if (normalizeCents(event.exact) !== expectedCents) throw conflict("Charge changed since review; review it again");
const [adjustment] = await tx.insert(costAdjustments).values({ companyId, costEventId: eventId,
invoiceLineId: input.invoiceLineId, idempotencyKey: input.idempotencyKey, receiptHash: hash,
previousCents: expectedCents, correctedCents, previousStatus: event.event.costStatus,
previousPricing: event.event.pricingProvenance ?? {},
previousUsageJson: input.receiptUsage ? { inputTokens: event.event.inputTokens, cachedInputTokens: event.event.cachedInputTokens,
outputTokens: event.event.outputTokens, providerRequestId: event.event.providerRequestId } : null,
reason: input.reason, actorId: input.actorId,
pricing: input.pricing }).onConflictDoNothing().returning();
if (!adjustment) throw Object.assign(new Error("Concurrent correction requires a fresh snapshot"), { code: "40001" });
// Daily-cost timer gates also depend on corrected ledger values when no
// formal budget policy exists. Replays above do not rearm this work.
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
const [updated] = await tx.update(costEvents).set({ costCents: sql`${correctedCents}::numeric`,
reportedCostCents: event.event.reportedCostCents ?? normalizeCents(event.exact),
...(input.receiptUsage ? { ...input.receiptUsage, reportedUsageJson: event.event.reportedUsageJson ?? {
inputTokens: event.event.inputTokens, cachedInputTokens: event.event.cachedInputTokens,
outputTokens: event.event.outputTokens, providerRequestId: event.event.providerRequestId,
} } : {}),
costStatus: input.costStatus ?? (input.pricing.source === "rate_card" ? "estimated" : "reported"), pricingProvenance: input.pricing,
}).where(and(eq(costEvents.id, eventId), eq(costEvents.companyId, companyId))).returning();
await queueBudgetEvaluation(tx, { companyId, agentId: updated.agentId, projectId: updated.projectId });
await logActivity(tx, { companyId, actorType: input.actorId === "cost_accounting" ? "system" : "user", actorId: input.actorId,
action: "accounting.charge_corrected", entityType: "cost_event", entityId: eventId,
details: { adjustmentId: adjustment.id, previousCents: expectedCents, correctedCents, reason: input.reason, pricing: input.pricing },
}, publications);
return adjustment;
}
+19 -38
View File
@@ -1,10 +1,9 @@
import { notifyAttentionRetentionWork } from "./retention-work-notifications.js";
import { and, eq, sql } from "drizzle-orm";
import { agents, agentRuntimeState, companies, heartbeatRuns, type Db } from "@paperclipai/db";
import { heartbeatRuns, type Db } from "@paperclipai/db";
import { normalizeCents, usdToCents, type AccountingFinding, type AccountingHealth, type AccountingInspection } from "@paperclipai/shared";
import { conflict, notFound } from "../errors.js";
import { receiptFingerprint } from "./receipt-fingerprint.js";
import { withAccountingReadSnapshot, withAccountingTransaction } from "./accounting-transaction.js";
import { withAccountingReadSnapshot, withOperationAccountingTransaction } from "./accounting-transaction.js";
import { logActivity } from "./activity-log.js";
import { accountRunCost } from "./run-cost-accounting.js";
import type { BudgetServiceHooks } from "./budgets.js";
@@ -12,31 +11,20 @@ import type { BudgetServiceHooks } from "./budgets.js";
/** Independent aggregation: deliberately does not reuse writer projections. */
async function inspect(tx: Db, companyId: string): Promise<AccountingInspection> {
const findings: AccountingFinding[] = [];
const monthly = await tx.execute<{ kind: "company_projection" | "agent_projection"; id: string; actual: string; expected: string }>(sql`
with month_events as (select agent_id, cost_cents from cost_events where company_id = ${companyId}
and occurred_at >= date_trunc('month', now() at time zone 'UTC') at time zone 'UTC'
and occurred_at < (date_trunc('month', now() at time zone 'UTC') + interval '1 month') at time zone 'UTC')
select 'company_projection' as kind, id, spent_monthly_cents::text as actual,
(select coalesce(sum(cost_cents),0)::text from month_events) as expected from companies where id = ${companyId}
union all select 'agent_projection', a.id, a.spent_monthly_cents::text,
coalesce((select sum(e.cost_cents) from month_events e where e.agent_id = a.id),0)::text from agents a where a.company_id = ${companyId}`);
for (const row of monthly) if (normalizeCents(row.actual) !== normalizeCents(row.expected)) findings.push({ kind: row.kind, entityId: row.id, repairable: true,
actual: { cents: normalizeCents(row.actual) }, expected: { cents: normalizeCents(row.expected) } });
const runtime = await tx.execute<{ id: string; baseline: string | null; actual: Record<string, string>; expected: Record<string, string> }>(sql`
with totals as (select r.agent_id, sum(e.cost_cents) as cents, sum(e.input_tokens) as input, sum(e.cached_input_tokens) as cached, sum(e.output_tokens) as output
from heartbeat_runs r join cost_events e on e.heartbeat_run_id = r.id and e.company_id = r.company_id
where r.company_id = ${companyId} and r.accounting_projection_version = 'v2' and r.cost_accounted_at is not null group by r.agent_id)
select a.agent_id as id, b.agent_id as baseline,
jsonb_build_object('cents', a.total_cost_cents::text, 'input', a.total_input_tokens::text, 'cached', a.total_cached_input_tokens::text, 'output', a.total_output_tokens::text) as actual,
jsonb_build_object('cents', (coalesce(b.cost_cents,0) + coalesce(t.cents,0))::text, 'input', (coalesce(b.input_tokens,0) + coalesce(t.input,0))::text,
'cached', (coalesce(b.cached_input_tokens,0) + coalesce(t.cached,0))::text, 'output', (coalesce(b.output_tokens,0) + coalesce(t.output,0))::text) as expected
from agent_runtime_state a left join accounting_runtime_baselines b on b.agent_id = a.agent_id and b.company_id = a.company_id
left join totals t on t.agent_id = a.agent_id where a.company_id = ${companyId} order by a.agent_id`);
for (const row of runtime) {
if (!row.baseline) { findings.push({ kind: "legacy_runtime", entityId: row.id, repairable: false, actual: row.actual, expected: {} }); continue; }
row.actual.cents = normalizeCents(row.actual.cents); row.expected.cents = normalizeCents(row.expected.cents);
if (Object.keys(row.expected).some(key => row.actual[key] !== row.expected[key])) findings.push({ kind: "runtime_projection", entityId: row.id, repairable: true, actual: row.actual, expected: row.expected });
}
// Monthly/runtime stored counters are retired compatibility fields. Inspect
// the evidence that remains authoritative, not differences from those fields.
const missingBaselines = await tx.execute<{ id: string }>(sql`
select distinct r.agent_id as id from heartbeat_runs r
left join accounting_runtime_baselines b on b.agent_id = r.agent_id and b.company_id = r.company_id
where r.company_id = ${companyId} and r.cost_accounted_at is not null
and r.accounting_projection_version = 'v2' and b.agent_id is null`);
for (const row of missingBaselines) findings.push({ kind: "legacy_runtime", entityId: row.id, repairable: false,
actual: { baseline: "missing" }, expected: { baseline: "reconciled" } });
const late = await tx.select({ id: heartbeatRuns.id, error: heartbeatRuns.accountingLastError })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
sql`${heartbeatRuns.usageJson}->>'accountingLateReceiptPending' = 'true'`));
for (const row of late) findings.push({ kind: "receipt_mismatch", entityId: row.id, repairable: false,
actual: { error: row.error ?? "late_usage_requires_reconciliation" }, expected: { lateReceipt: "reconciled" } });
const incomplete = await tx.execute<{ id: string; acknowledged: boolean; events: number }>(sql`
select r.id, r.cost_accounted_at is not null as acknowledged, count(e.id)::int as events from heartbeat_runs r
left join cost_events e on e.heartbeat_run_id = r.id and e.company_id = r.company_id and e.idempotency_key like ('heartbeat:' || r.id::text || ':%')
@@ -72,19 +60,12 @@ async function inspect(tx: Db, companyId: string): Promise<AccountingInspection>
export function accountingIntegrityService(db: Db, hooks: BudgetServiceHooks = {}) {
return {
inspect: (companyId: string) => withAccountingReadSnapshot(db, companyId, tx => inspect(tx, companyId)),
repair: (companyId: string, fingerprint: string, reason: string, actorId: string) => withAccountingTransaction(db, companyId, async (tx, publications) => {
repair: (companyId: string, fingerprint: string, reason: string, actorId: string) => withOperationAccountingTransaction(db, companyId, async (tx, publications) => {
if (!reason.trim()) throw conflict("A repair reason is required");
const before = await inspect(tx, companyId);
if (before.fingerprint !== fingerprint) throw conflict("Accounting changed since inspection; inspect again before repairing");
if (before.findings.some(f => f.repairable && f.kind === "agent_projection"))
await notifyAttentionRetentionWork(tx, { companyId, sourceKind: "agent_error_alert" });
for (const finding of before.findings.filter(f => f.repairable)) {
if (finding.kind === "company_projection") await tx.update(companies).set({ spentMonthlyCents: sql`${finding.expected.cents}::numeric`, updatedAt: new Date() }).where(eq(companies.id, companyId));
if (finding.kind === "agent_projection") await tx.update(agents).set({ spentMonthlyCents: sql`${finding.expected.cents}::numeric`, updatedAt: new Date() }).where(and(eq(agents.id, finding.entityId), eq(agents.companyId, companyId)));
if (finding.kind === "runtime_projection") await tx.update(agentRuntimeState).set({ totalCostCents: sql`${finding.expected.cents}::numeric`,
totalInputTokens: sql`${finding.expected.input}::bigint`, totalCachedInputTokens: sql`${finding.expected.cached}::bigint`, totalOutputTokens: sql`${finding.expected.output}::bigint`, updatedAt: new Date(),
}).where(and(eq(agentRuntimeState.agentId, finding.entityId), eq(agentRuntimeState.companyId, companyId)));
}
// Ledger valuation changes use audited cost adjustments. Retired stored
// counters must never be written back into accounting authority.
await logActivity(tx, { companyId, actorType: "user", actorId, action: "accounting.projections_repaired", entityType: "company", entityId: companyId,
details: { fingerprint, reason, repairs: before.findings.filter(f => f.repairable) } }, publications);
return inspect(tx, companyId);
@@ -21,6 +21,39 @@ export async function withAccountingTransaction<T>(db: Db, companyId: string, wo
return result;
}
/** Independent paid operations share a consistent snapshot, never a company
* writer mutex. Only retry transactions PostgreSQL proves were rolled back;
* an uncertain commit must be replayed by its durable operation identity. */
export async function withOperationAccountingTransaction<T>(db: Db, companyId: string,
work: (tx: Db, publications: ActivityPublication[]) => Promise<T>): Promise<T> {
for (let attempt = 0; ; attempt++) {
const publications: ActivityPublication[] = [];
try {
const result = await withBudgetSpan("budget.accounting.transaction", () => db.transaction(async transaction => {
const tx = transaction as unknown as Db;
const [company] = await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, companyId));
if (!company) throw notFound("Company not found");
return work(tx, publications);
}, { isolationLevel: "repeatable read" }));
publishAccountingActivities(companyId, publications);
return result;
} catch (error) {
if (attempt >= 3 || !isAccountingRetryable(error)) throw error;
await new Promise(resolve => setTimeout(resolve, 10 * (attempt + 1)));
}
}
}
export function isAccountingRetryable(error: unknown): boolean {
let current = error;
for (let depth = 0; current && typeof current === "object" && depth < 4; depth++) {
const row = current as { code?: string; cause?: unknown };
if (row.code === "40001" || row.code === "40P01") return true;
current = row.cause;
}
return false;
}
/** Consistent operator reports must not wait for or delay accounting writers.
* Reads see one committed snapshot; repairs still revalidate under the lock. */
export function withAccountingReadSnapshot<T>(db: Db, companyId: string, work: (tx: Db) => Promise<T>): Promise<T> {
+10
View File
@@ -17,6 +17,9 @@ import { issueThreadInteractionService } from "./issue-thread-interactions.js";
import { deleteAgentAdmissionHistory } from "../modules/execution-admission/index.js";
export async function deleteAgentDependencies(tx: Db, companyId: string, id: string) {
// Lifecycle callers use READ COMMITTED. Fence admissions before inspecting
// holds, so the next statement sees any admission that committed while waiting.
await tx.select({ id: agents.id }).from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, id))).for("update");
const [retainedHandoff] = await tx.select({ id: caseHandoffs.id }).from(caseHandoffs).where(and(
eq(caseHandoffs.companyId, companyId), or(eq(caseHandoffs.fromAgentId, id), eq(caseHandoffs.toAgentId, id)),
)).limit(1);
@@ -37,6 +40,13 @@ export async function deleteAgentDependencies(tx: Db, companyId: string, id: str
if (classifierHold) throw conflict("Wait for active prompt injection checks or resolve their unknown charges in Costs before deleting this agent", {
code: "agent_prompt_injection_accounting_pending",
});
const [fastResponseHold] = await tx.select({ id: budgetReservations.id }).from(budgetReservations).where(and(
eq(budgetReservations.companyId, companyId), eq(budgetReservations.agentId, id),
eq(budgetReservations.state, "held"), sql`${budgetReservations.fastResponseRequestId} is not null`,
)).limit(1);
if (fastResponseHold) throw conflict("Wait for active fast responses or resolve their unknown charges in Costs before deleting this agent", {
code: "agent_fast_response_accounting_pending",
});
await notifyAttentionRetentionWork(tx, { companyId });
await notifyExternalConversationSettlement(tx, { companyId });
await issueThreadInteractionService(tx).cancelPendingForDeletedAddressee(companyId, id);
+12 -28
View File
@@ -1,14 +1,14 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { createFinanceEventInTransaction } from "./finance.js";
import { and, desc, eq, sql } from "drizzle-orm";
import { fastResponseRequests, decisionInvocations, promptInjectionInvocations, budgetReservations, agentRuntimeState, costEvents, costAdjustments, billingInvoices, billingInvoiceLines, financeEvents, heartbeatRuns, type Db } from "@paperclipai/db";
import { fastResponseRequests, decisionInvocations, promptInjectionInvocations, budgetReservations, costEvents, costAdjustments, billingInvoices, billingInvoiceLines, financeEvents, heartbeatRuns, type Db } from "@paperclipai/db";
import { adjustCostSchema, importBillingInvoiceSchema, normalizeCents, subtractCents, type AdjustCost, type ImportBillingInvoice } from "@paperclipai/shared";
import { conflict, notFound, unprocessable } from "../errors.js";
import { withAccountingReadSnapshot, withAccountingTransaction } from "./accounting-transaction.js";
import { withAccountingReadSnapshot, withAccountingTransaction, withOperationAccountingTransaction } from "./accounting-transaction.js";
import { receiptFingerprint } from "./receipt-fingerprint.js";
import { logActivity } from "./activity-log.js";
import { updateMonthlySpendProjections } from "./costs.js";
import { budgetServiceInTransaction, type BudgetServiceHooks } from "./budgets.js";
import { applyCostCorrectionInTransaction, lockCostCorrectionOwner } from "./accounting-corrections.js";
import type { BudgetServiceHooks } from "./budgets.js";
async function invoiceCostCandidates(tx: Db, companyId: string, biller: string, line: {
costEventId?: string | null; runId?: string | null; providerRequestId?: string | null; model?: string | null;
@@ -46,7 +46,8 @@ export function billingReconciliationService(db: Db, hooks: BudgetServiceHooks =
if (!run) throw notFound("Invoice run not found");
}
}
const [invoice] = await tx.insert(billingInvoices).values({ companyId, biller: input.biller, externalId: input.externalId, currency: input.currency, receiptHash, importedBy: actorId }).returning();
const [invoice] = await tx.insert(billingInvoices).values({ companyId, biller: input.biller, externalId: input.externalId, currency: input.currency, receiptHash, importedBy: actorId }).onConflictDoNothing().returning();
if (!invoice) throw Object.assign(new Error("Concurrent invoice requires a fresh snapshot"), { code: "40001" });
await tx.insert(billingInvoiceLines).values(input.lines.map(line => ({ ...line, companyId, invoiceId: invoice.id, occurredAt: new Date(line.occurredAt), pricing: line.pricing ?? {} })));
const matches = [];
const matchCounts = new Map<string, number>();
@@ -109,13 +110,14 @@ export function billingReconciliationService(db: Db, hooks: BudgetServiceHooks =
adjustments: (companyId: string, eventId: string) => db.select().from(costAdjustments).where(and(eq(costAdjustments.companyId, companyId), eq(costAdjustments.costEventId, eventId))).orderBy(desc(costAdjustments.createdAt)),
adjust: async (companyId: string, eventId: string, raw: AdjustCost, actorId: string) => {
const input = adjustCostSchema.parse(raw);
const result = await withAccountingTransaction(db, companyId, async (tx, publications) => {
const result = await withOperationAccountingTransaction(db, companyId, async (tx, publications) => {
const hash = receiptFingerprint({ eventId, ...input });
const [existing] = await tx.select().from(costAdjustments).where(and(eq(costAdjustments.companyId, companyId), eq(costAdjustments.idempotencyKey, input.idempotencyKey)));
if (existing) {
if (existing.receiptHash !== hash) throw conflict("Correction key already has different contents");
return existing;
}
await lockCostCorrectionOwner(tx, companyId, eventId);
const [event] = await tx.select({ event: costEvents, exact: sql<string>`${costEvents.costCents}::text` }).from(costEvents)
.where(and(eq(costEvents.companyId, companyId), eq(costEvents.id, eventId))).for("update");
if (!event) throw notFound("Cost event not found");
@@ -138,23 +140,10 @@ export function billingReconciliationService(db: Db, hooks: BudgetServiceHooks =
const [used] = await tx.select().from(costAdjustments).where(and(eq(costAdjustments.companyId, companyId), eq(costAdjustments.invoiceLineId, input.invoiceLineId)));
if (used) throw conflict("Invoice line has already been applied");
}
const [adjustment] = await tx.insert(costAdjustments).values({ companyId, costEventId: eventId, invoiceLineId: input.invoiceLineId,
idempotencyKey: input.idempotencyKey, receiptHash: hash, previousCents: input.expectedCents, correctedCents: input.correctedCents,
previousStatus: event.event.costStatus, previousPricing: event.event.pricingProvenance ?? {},
reason: input.reason, actorId, pricing: input.pricing }).returning();
// Original provider value and fingerprint remain immutable. cost_cents
// is the effective valuation, backed by this append-only correction.
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
const [updated] = await tx.update(costEvents).set({ costCents: sql`${input.correctedCents}::numeric`,
reportedCostCents: event.event.reportedCostCents ?? normalizeCents(event.exact), costStatus: input.pricing.source === "rate_card" ? "estimated" : "reported", pricingProvenance: input.pricing,
}).where(eq(costEvents.id, eventId)).returning();
await updateMonthlySpendProjections(tx, companyId, updated.agentId, subtractCents(input.correctedCents, input.expectedCents), updated.occurredAt);
if (updated.heartbeatRunId && updated.agentId) {
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, updated.heartbeatRunId), eq(heartbeatRuns.companyId, companyId)));
if (run?.costAccountedAt && run.accountingProjectionVersion === "v2") await tx.update(agentRuntimeState)
.set({ totalCostCents: sql`${agentRuntimeState.totalCostCents} + ${subtractCents(input.correctedCents, input.expectedCents)}::numeric`, updatedAt: new Date() })
.where(and(eq(agentRuntimeState.agentId, updated.agentId), eq(agentRuntimeState.companyId, companyId)));
}
const adjustment = await applyCostCorrectionInTransaction(tx, publications, {
companyId, eventId, ...input, actorId, receiptHash: hash,
});
const [updated] = await tx.select().from(costEvents).where(and(eq(costEvents.id, eventId), eq(costEvents.companyId, companyId)));
if (updated.usageKind === "decision") {
const [invocation] = await tx.select({ id: decisionInvocations.id }).from(decisionInvocations).where(and(
eq(decisionInvocations.companyId, companyId), eq(decisionInvocations.costEventId, eventId),
@@ -175,11 +164,6 @@ export function billingReconciliationService(db: Db, hooks: BudgetServiceHooks =
const [invocation] = await tx.select({ id: fastResponseRequests.id }).from(fastResponseRequests).where(and(eq(fastResponseRequests.companyId, companyId), eq(fastResponseRequests.costEventId, eventId)));
if (invocation) await tx.update(budgetReservations).set({ state: "settled", settledAt: new Date() }).where(and(eq(budgetReservations.companyId, companyId), eq(budgetReservations.fastResponseRequestId, invocation.id), eq(budgetReservations.state, "held")));
}
const budgets = budgetServiceInTransaction(tx, publications);
await budgets.evaluateCostEvent(updated);
await budgets.getInvocationBlock(companyId, updated.agentId, { projectId: updated.projectId });
await logActivity(tx, { companyId, actorType: "user", actorId, action: "accounting.charge_corrected", entityType: "cost_event", entityId: eventId,
details: { adjustmentId: adjustment.id, previousCents: input.expectedCents, correctedCents: input.correctedCents, reason: input.reason, pricing: input.pricing } }, publications);
return adjustment;
});
return result;
+184 -75
View File
@@ -46,12 +46,14 @@ import { accessService } from "./access.js";
import { toolAccessPolicyService } from "./tool-access-policy.js";
import { createCostEventInTransaction } from "./costs.js";
import { createFinanceEventInTransaction } from "./finance.js";
import { withAccountingTransaction } from "./accounting-transaction.js";
import { withOperationAccountingTransaction } from "./accounting-transaction.js";
import { logActivity, type ActivityPublication } from "./activity-log.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { browserUseViewports } from "./browser-use-viewport.js";
import { withBudgetSpan } from "./budget-tracing.js";
import { forbidden, notFound, conflict } from "../errors.js";
import { isUniqueViolation } from "../db-errors.js";
import { lockBudgetAdmissionControls } from "./budget-admission-controls.js";
type Session = typeof sessions.$inferSelect;
type Binding = {
@@ -76,10 +78,29 @@ const liveSessions = ["starting", "running", "idle", "stopping"];
const requestMarker = (invocationId: string) => `\n\n[Paperclip request: ${invocationId}]`;
type BrowserBudgetHold = { invocationId: string; amountCents: string; projectId: string | null;
state: "held" | "released" | "settled"; uncapped?: boolean;
dispatchCommittedAt?: string; accountedCentsExact?: string; billingUnknown?: boolean };
dispatchCommittedAt?: string; accountedCentsExact?: string; billingUnknown?: boolean;
receipt?: BrowserUsageReceipt };
type BrowserUsageReceipt = { invocationId: string; providerRunId: string; totalCentsExact: string;
model: string; occurredAt: string };
const resultObject = (value: unknown): Record<string, unknown> =>
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const budgetHold = (value: unknown) => resultObject(value).paperclipBudgetHold as BrowserBudgetHold | undefined;
const exactNonnegativeCents = z.string().refine(value => {
try { return centsToUnits(value) >= 0n; } catch { return false; }
});
const browserReceiptSchema = z.object({ invocationId: z.string(), providerRunId: z.string(),
totalCentsExact: exactNonnegativeCents, model: z.string(),
occurredAt: z.string().refine(value => Number.isFinite(new Date(value).getTime())) });
const browserHoldSchema = z.object({ invocationId: z.string(), amountCents: exactNonnegativeCents.default("0.0000000"),
projectId: z.string().nullable().default(null), state: z.enum(["held", "released", "settled"]),
uncapped: z.boolean().optional(), dispatchCommittedAt: z.string().optional(), accountedCentsExact: exactNonnegativeCents.optional(),
billingUnknown: z.boolean().optional(), receipt: browserReceiptSchema.optional() });
function budgetHold(value: unknown): BrowserBudgetHold | undefined {
const raw = resultObject(value).paperclipBudgetHold;
if (raw === undefined) return undefined;
const parsed = browserHoldSchema.safeParse(raw);
if (!parsed.success) throw conflict("Browser accounting evidence is invalid.");
return parsed.data;
}
function publicResult(value: unknown) {
if (!value || typeof value !== "object" || Array.isArray(value)) return value;
const { paperclipBudgetHold: _internal, ...visible } = resultObject(value);
@@ -427,6 +448,17 @@ export function browserUseService(
.limit(1);
if (existing[0])
return snapshot(await owned(binding, grant, existing[0].sessionId));
const lockAdmissionScopes = async (tx: Db) => {
// Shared authority fences let concurrent paid claims proceed together.
// A company/project pause must precede the claim or see its durable run.
// Take these before browser/session/run rows, matching control writers.
const [scope] = await tx.select({ projectId: issues.projectId }).from(issues)
.where(and(eq(issues.id, binding.issueId!), eq(issues.companyId, binding.companyId)));
await lockBudgetAdmissionControls(tx, binding.companyId, scope?.projectId);
const [locked] = await tx.select({ projectId: issues.projectId }).from(issues)
.where(and(eq(issues.id, binding.issueId!), eq(issues.companyId, binding.companyId))).for("share");
if (locked?.projectId !== scope?.projectId) throw conflict("The browser task's project changed before dispatch.");
};
const admissionCheck = async (tx: Db, publications: ActivityPublication[], hold?: BrowserBudgetHold) => {
const [parent] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, binding.runId!), eq(heartbeatRuns.companyId, binding.companyId),
@@ -505,7 +537,8 @@ export function browserUseService(
return { amountCents: unitsToCents(amount ?? 0n), maxCostUsd: amount === undefined ? undefined : Number(centsToUsd(unitsToCents(amount))), projectId: issue.projectId };
};
const sid = args.sessionId;
const admitted = await withAccountingTransaction(db, binding.companyId, async (tx, publications) => {
const admitted = await withOperationAccountingTransaction(db, binding.companyId, async (tx, publications) => {
await lockAdmissionScopes(tx);
const [duplicate] = await tx.select().from(runs).where(and(eq(runs.companyId, binding.companyId), eq(runs.invocationId, invocationId)));
if (duplicate) return { existing: duplicate.sessionId };
const check = await admissionCheck(tx, publications);
@@ -561,7 +594,16 @@ export function browserUseService(
? new Date(Date.now() + BROWSER_USE_IDLE_MS)
: null,
});
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(binding.companyId, binding.agentId, check.projectId!);
return { session: row, hold, maxCostUsd: check.maxCostUsd };
}).catch(async error => {
// Parallel retries can both observe no invocation in their snapshots.
// Roll back the losing session/hold and replay the committed owner.
if (!isUniqueViolation(error, "browser_use_runs_invocation_uq")) throw error;
const [duplicate] = await db.select().from(runs).where(and(
eq(runs.companyId, binding.companyId), eq(runs.invocationId, invocationId)));
if (!duplicate) throw error;
return { existing: duplicate.sessionId, reason: undefined, session: undefined, hold: undefined, maxCostUsd: undefined };
});
if (admitted.reason) throw forbidden(admitted.reason);
if (admitted.existing) return snapshot(await owned(binding, grant, admitted.existing));
@@ -583,7 +625,8 @@ export function browserUseService(
},
...(maxCostUsd === undefined ? {} : { maxCostUsd }),
}, async () => {
const reason = await withAccountingTransaction(db, binding.companyId, async (tx, publications) => {
const reason = await withOperationAccountingTransaction(db, binding.companyId, async (tx, publications) => {
await lockAdmissionScopes(tx);
const [current] = await tx.select().from(runs).where(and(eq(runs.invocationId, invocationId), eq(runs.companyId, binding.companyId))).for("update");
const [currentSession] = await tx.select().from(sessions).where(and(eq(sessions.id, s.id), eq(sessions.companyId, binding.companyId))).for("update");
const hold = budgetHold(current?.result);
@@ -593,36 +636,48 @@ export function browserUseService(
const check = await admissionCheck(tx, publications, hold);
if (check.reason) return check.reason;
await tx.update(runs).set({ result: { ...resultObject(current.result), paperclipBudgetHold: { ...hold, dispatchCommittedAt: new Date().toISOString() } } }).where(eq(runs.id, current.id));
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(binding.companyId, binding.agentId, hold.projectId);
return null;
});
if (reason) throw new BrowserUseError(403, reason, 0, true);
providerAttempted = true;
});
await db.transaction(async (tx) => {
const persistedSession = await withOperationAccountingTransaction(db, binding.companyId, async (tx) => {
const [currentRun] = await tx.select().from(runs).where(and(eq(runs.invocationId, invocationId),
eq(runs.companyId, binding.companyId))).for("update");
const [currentSession] = await tx.select().from(sessions).where(and(eq(sessions.id, s.id),
eq(sessions.companyId, binding.companyId))).for("update");
if (!currentRun || !currentSession || (currentRun.providerRunId && currentRun.providerRunId !== result.id))
throw conflict("Browser provider response does not match its invocation.");
await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser);
await tx
.update(runs)
.set({ providerRunId: result.id, status: result.status })
.where(eq(runs.invocationId, invocationId));
await tx
.set({ providerRunId: result.id, status: browserUseTerminal(currentRun.status) ? currentRun.status : result.status })
.where(eq(runs.id, currentRun.id));
const [updated] = await tx
.update(sessions)
.set({
providerSessionId: result.sessionId,
status: "running",
error: null,
status: currentSession.stopRequested || currentSession.status === "stopping" ? "stopping" : currentSession.status === "closed" ? "closed" : "running",
error: currentSession.stopRequested ? currentSession.error : null,
nextPollAt: new Date(),
updatedAt: new Date(),
})
.where(eq(sessions.id, s.id));
.where(eq(sessions.id, s.id)).returning();
return updated;
});
Object.assign(s, persistedSession);
} catch (error) {
// A lost create response can mean paid work exists. Never submit it again.
const rejected = !providerAttempted || (error instanceof BrowserUseError && error.requestRejected);
const message = rejected ? (error instanceof Error ? error.message : "Browser dispatch was rejected.") :
"Browser run creation could not be confirmed. Paperclip is locating and stopping possible provider work. Do not start another run yet.";
await withAccountingTransaction(db, binding.companyId, async (tx, publications) => {
await withOperationAccountingTransaction(db, binding.companyId, async (tx, publications) => {
const [currentRun] = await tx.select().from(runs).where(and(eq(runs.invocationId, invocationId),
eq(runs.companyId, binding.companyId))).for("update");
const [current] = await tx.select().from(sessions).where(and(eq(sessions.id, s.id), eq(sessions.companyId, binding.companyId))).for("update");
if (!currentRun || !current || currentRun.providerRunId || currentRun.eventsDrained || budgetHold(currentRun.result)?.state === "settled") return;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser);
const [current] = await tx.select().from(sessions).where(eq(sessions.id, s.id)).for("update");
if (rejected) {
await tx.update(runs).set({ result: sql`jsonb_set(${runs.result}, '{paperclipBudgetHold,state}', '"released"'::jsonb)` }).where(eq(runs.invocationId, invocationId));
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(binding.companyId, binding.agentId, admitted.hold!.projectId);
@@ -654,7 +709,7 @@ export function browserUseService(
await signal(s, "browser.started");
return {
sessionId: s.id,
status: result.status,
status: s.stopRequested || s.status === "stopping" ? "stopping" : result.status,
instruction:
"Poll browser_status until terminal. The human's Browser tab opens when the browser is ready.",
};
@@ -1029,52 +1084,98 @@ export function browserUseService(
async function account(
s: Session,
run: typeof runs.$inferSelect,
summary: Awaited<
ReturnType<ReturnType<typeof browserUseClient>["summary"]>
>,
summary?: Awaited<ReturnType<ReturnType<typeof browserUseClient>["summary"]>>,
) {
let amount: bigint;
try { amount = usdToUnits(summary.totalCostUsd); }
catch { throw new BrowserUseError(502, "Browser Use returned invalid usage."); }
if (amount < 0n || !browserUseTerminal(summary.status))
throw new BrowserUseError(502, "Browser Use returned invalid terminal usage.");
const cents = unitsToCents(amount);
await withBudgetSpan("budget.accounting.settlement", () => withAccountingTransaction(db, s.companyId, async (tx, publications) => {
const [current] = await tx.select().from(runs).where(and(eq(runs.id, run.id), eq(runs.companyId, s.companyId))).for("update");
const hold = budgetHold(current.result);
const previous = centsToUnits(hold?.accountedCentsExact ?? current.accountedCents);
const [issue] = await tx.select({ projectId: issues.projectId }).from(issues)
.where(and(eq(issues.id, s.issueId), eq(issues.companyId, s.companyId)));
const projectId = hold ? hold.projectId : issue?.projectId ?? null;
// Replace the separate browser hold with its ledger charge atomically.
// Provider output never gets to overwrite the private accounting state.
const settled: BrowserBudgetHold = { ...(hold ?? { invocationId: current.invocationId, amountCents: "0.0000000", projectId }), state: "settled",
accountedCentsExact: unitsToCents(amount > previous ? amount : previous) };
await tx.update(runs).set({
result: { ...resultObject(sanitizeBrowserUse({ result: summary.result, output: summary.output })),
paperclipBudgetHold: settled },
eventsDrained: 1,
accountedCents: Math.min(2_147_483_647, Math.round(Number(unitsToCents(amount > previous ? amount : previous)))),
}).where(eq(runs.id, current.id));
if (amount <= previous) {
if (hold?.state === "held") await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(s.companyId, s.agentId, projectId);
return;
}
const e = await createCostEventInTransaction(tx, s.companyId, {
agentId: s.agentId, issueId: s.issueId, projectId,
heartbeatRunId: run.heartbeatRunId,
provider: "browser-use-cloud", biller: "browser-use-cloud", billingType: "metered_api", model: summary.model,
costCents: unitsToCents(amount - previous), occurredAt: new Date(),
billingCode: `browser-use-cloud:${run.id}`, idempotencyKey: `browser-use-cloud:${run.id}:${cents}`,
}, publications);
await createFinanceEventInTransaction(tx, publications, s.companyId, {
agentId: e.agentId, issueId: e.issueId, projectId: e.projectId, heartbeatRunId: e.heartbeatRunId,
costEventId: e.id, billingCode: e.billingCode, eventKind: "inference_charge", direction: "debit",
biller: "browser-use-cloud", provider: "browser-use-cloud", model: e.model,
amountCents: e.costCentsExact, currency: "USD", description: "Browser Use agent run", occurredAt: e.occurredAt,
let candidate: { amount: bigint; model: string; result: Record<string, unknown> } | undefined;
if (summary) {
let amount: bigint;
try { amount = usdToUnits(summary.totalCostUsd); }
catch { throw new BrowserUseError(502, "Browser Use returned invalid usage."); }
if (amount < 0n || !browserUseTerminal(summary.status))
throw new BrowserUseError(502, "Browser Use returned invalid terminal usage.");
candidate = { amount, model: summary.model,
result: resultObject(sanitizeBrowserUse({ result: summary.result, output: summary.output })) };
}
// Persist the provider observation before attempting the ledger transaction.
// A failed/uncertain commit can then recover its exact price, attribution and
// occurrence time without depending on another provider response.
for (;;) {
const receipt = await withOperationAccountingTransaction(db, s.companyId, async (tx, publications) => {
const [current] = await tx.select().from(runs).where(and(eq(runs.id, run.id), eq(runs.companyId, s.companyId))).for("update");
if (!current || current.sessionId !== s.id || !current.providerRunId || current.providerRunId !== run.providerRunId)
throw conflict("Browser usage no longer belongs to this run.");
const hold = budgetHold(current.result);
if (hold && hold.invocationId !== current.invocationId) throw conflict("Browser accounting identity does not match its invocation.");
const previous = centsToUnits(hold?.accountedCentsExact ?? current.accountedCents);
const saved = hold?.receipt;
if (saved && (saved.invocationId !== current.invocationId || saved.providerRunId !== current.providerRunId
|| !Number.isFinite(new Date(saved.occurredAt).getTime()) || centsToUnits(saved.totalCentsExact) < 0n))
throw conflict("Browser usage receipt identity is invalid.");
if (saved && (centsToUnits(saved.totalCentsExact) > previous || hold?.state === "held")) return saved;
if (!candidate) return null;
if (candidate.amount <= previous && hold?.state !== "held") {
if (!current.eventsDrained) await tx.update(runs).set({ eventsDrained: 1 }).where(eq(runs.id, current.id));
return null;
}
const [issue] = await tx.select({ projectId: issues.projectId }).from(issues)
.where(and(eq(issues.id, s.issueId), eq(issues.companyId, s.companyId)));
const projectId = hold ? hold.projectId : issue?.projectId ?? null;
const captured: BrowserUsageReceipt = { invocationId: current.invocationId, providerRunId: current.providerRunId,
totalCentsExact: unitsToCents(candidate.amount), model: candidate.model, occurredAt: new Date().toISOString() };
const remaining = candidate.amount > previous ? candidate.amount - previous : 0n;
const held = hold?.state === "held" ? centsToUnits(hold.amountCents) : 0n;
await tx.update(runs).set({ eventsDrained: 0, result: { ...candidate.result, paperclipBudgetHold: {
...(hold ?? { invocationId: current.invocationId, projectId }), state: "held",
amountCents: unitsToCents(held > remaining ? held : remaining), accountedCentsExact: unitsToCents(previous), receipt: captured,
} } }).where(eq(runs.id, current.id));
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(s.companyId, s.agentId, projectId);
return captured;
});
}));
if (!receipt) return;
const amount = centsToUnits(receipt.totalCentsExact);
await withBudgetSpan("budget.accounting.settlement", () => withOperationAccountingTransaction(db, s.companyId, async (tx, publications) => {
const [current] = await tx.select().from(runs).where(and(eq(runs.id, run.id), eq(runs.companyId, s.companyId))).for("update");
const hold = current && budgetHold(current.result);
if (!current || current.sessionId !== s.id || !hold || hold.invocationId !== receipt.invocationId
|| current.providerRunId !== receipt.providerRunId) throw conflict("Browser usage no longer belongs to this run.");
const previous = centsToUnits(hold.accountedCentsExact ?? current.accountedCents);
if (hold.receipt?.totalCentsExact !== receipt.totalCentsExact || hold.receipt.occurredAt !== receipt.occurredAt) return;
// Another reconciler may already have committed this receipt (or a later
// cumulative observation). Its local operation fence makes that replay.
if (amount <= previous && hold.state === "settled") return;
const total = amount > previous ? amount : previous;
const settled: BrowserBudgetHold = { ...hold, state: "settled", billingUnknown: false,
accountedCentsExact: unitsToCents(total) };
await tx.update(runs).set({
result: { ...resultObject(current.result), paperclipBudgetHold: settled }, eventsDrained: 1,
accountedCents: Math.min(2_147_483_647, Math.round(Number(unitsToCents(total)))),
}).where(eq(runs.id, current.id));
if (amount <= previous) {
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(s.companyId, s.agentId, hold.projectId);
return;
}
const e = await createCostEventInTransaction(tx, s.companyId, {
agentId: s.agentId, issueId: s.issueId, projectId: hold.projectId,
heartbeatRunId: current.heartbeatRunId,
provider: "browser-use-cloud", biller: "browser-use-cloud", billingType: "metered_api", model: receipt.model,
costCents: unitsToCents(amount - previous), occurredAt: new Date(receipt.occurredAt),
billingCode: `browser-use-cloud:${current.id}`, idempotencyKey: `browser-use-cloud:${current.id}:${receipt.totalCentsExact}`,
}, publications);
await createFinanceEventInTransaction(tx, publications, s.companyId, {
agentId: e.agentId, issueId: e.issueId, projectId: e.projectId, heartbeatRunId: e.heartbeatRunId,
costEventId: e.id, billingCode: e.billingCode, eventKind: "inference_charge", direction: "debit",
biller: "browser-use-cloud", provider: "browser-use-cloud", model: e.model,
amountCents: e.costCentsExact, currency: "USD", description: "Browser Use agent run", occurredAt: e.occurredAt,
idempotencyKey: `browser-use-cloud:${current.id}:${receipt.totalCentsExact}`,
});
}), { "budget.accounting.owner": "browser" });
if (!candidate || amount >= candidate.amount) return;
}
}
const sessionLeaseFence = (s: Session) => and(eq(sessions.id, s.id), eq(sessions.companyId, s.companyId),
s.leaseToken ? eq(sessions.leaseToken, s.leaseToken) : sql`false`, sql`${sessions.leaseUntil} > clock_timestamp()`);
const runLeaseFence = (s: Session, runId: string) => and(eq(runs.id, runId), eq(runs.companyId, s.companyId),
sql`exists (select 1 from ${sessions} where ${sessionLeaseFence(s)})`);
async function reconcile(s: Session) {
const { grant, connection, client } = await credentials(s, true);
let end =
@@ -1128,6 +1229,8 @@ export function browserUseService(
let pendingAccounting = false;
let discovered = false;
for (const run of allRuns) {
const [ownedLease] = await db.select({ id: sessions.id }).from(sessions).where(sessionLeaseFence(s));
if (!ownedLease) return;
if (!run.providerRunId) {
if (run.eventsDrained) continue; // A definitive rejection made no paid run.
if (run.status === "creating" && Date.now() - run.createdAt.getTime() <= 60_000) {
@@ -1135,21 +1238,22 @@ export function browserUseService(
continue;
}
if (budgetHold(run.result)?.state === "held" && !budgetHold(run.result)?.dispatchCommittedAt) {
const repaired = await withAccountingTransaction(db, s.companyId, async (tx, publications) => {
const repaired = await withOperationAccountingTransaction(db, s.companyId, async (tx, publications) => {
const [current] = await tx.select().from(runs).where(and(eq(runs.id, run.id), eq(runs.companyId, s.companyId))).for("update");
const hold = budgetHold(current?.result);
if (!current || current.providerRunId || current.status !== "creating" || hold?.state !== "held" || hold.dispatchCommittedAt
|| Date.now() - current.createdAt.getTime() <= 60_000) return null;
const [session] = await tx.select().from(sessions).where(sessionLeaseFence(s)).for("update");
if (!session) return null;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser);
await tx.update(runs).set({ status: "failed", eventsDrained: 1,
result: { ...resultObject(current.result), paperclipBudgetHold: { ...hold, state: "released" } } }).where(eq(runs.id, run.id));
const [session] = await tx.select().from(sessions).where(eq(sessions.id, s.id)).for("update");
result: { ...resultObject(current.result), paperclipBudgetHold: { ...hold, state: "released" } } }).where(runLeaseFence(s, run.id));
const stopping = session.stopRequested === "end";
const [updated] = await tx.update(sessions).set({
status: stopping ? "stopping" : session.providerSessionId ? "idle" : "failed",
idleDeadline: !stopping && session.providerSessionId ? new Date(Date.now() + BROWSER_USE_IDLE_MS) : null,
updatedAt: new Date(),
}).where(eq(sessions.id, s.id)).returning();
}).where(sessionLeaseFence(s)).returning();
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(s.companyId, s.agentId, hold.projectId);
return updated;
});
@@ -1163,7 +1267,7 @@ export function browserUseService(
}
end = true;
await db.update(sessions).set({ status: "stopping", stopRequested: "end" })
.where(eq(sessions.id, s.id));
.where(sessionLeaseFence(s));
let cursor = run.recoveryCursor;
for (let page = 0; page < 5 && !run.providerRunId; page++) {
const listed = await client.listRuns(s.providerSessionId, cursor);
@@ -1175,9 +1279,9 @@ export function browserUseService(
if (found) {
await db.transaction(async tx => {
await tx.update(runs).set({ providerRunId: found.id, status: found.status, recoveryCursor: null })
.where(eq(runs.id, run.id));
.where(and(runLeaseFence(s, run.id), isNull(runs.providerRunId)));
await tx.update(sessions).set({ providerSessionId: found.sessionId })
.where(eq(sessions.id, s.id));
.where(sessionLeaseFence(s));
});
run.providerRunId = found.id;
run.status = found.status;
@@ -1187,7 +1291,8 @@ export function browserUseService(
if (listed.hasMore && (!listed.nextCursor || listed.nextCursor === cursor))
throw new BrowserUseError(502, "Browser recovery cursor did not advance.");
cursor = listed.hasMore ? listed.nextCursor! : null;
await db.update(runs).set({ status: "unknown", recoveryCursor: cursor }).where(eq(runs.id, run.id));
await db.update(runs).set({ status: "unknown", recoveryCursor: cursor })
.where(and(runLeaseFence(s, run.id), isNull(runs.providerRunId)));
if (!listed.hasMore) break;
}
if (!run.providerRunId) {
@@ -1244,12 +1349,12 @@ export function browserUseService(
await db
.update(runs)
.set({
eventCursor: cursor,
eventCursor: sql`greatest(${runs.eventCursor}, ${cursor})`,
progress: progress
? String(sanitizeBrowserUse(progress)).slice(0, 200)
: null,
})
.where(eq(runs.id, run.id));
.where(runLeaseFence(s, run.id));
if (!events.hasMore) {
drained = true;
break;
@@ -1263,11 +1368,14 @@ export function browserUseService(
}
await db
.update(runs)
.set({ status: state.status })
.where(eq(runs.id, run.id));
.set({ status: sql`case when ${runs.status} in ('completed', 'failed', 'cancelled') then ${runs.status} else ${state.status} end` })
.where(runLeaseFence(s, run.id));
pendingAccounting ||= browserUseTerminal(state.status) && !drained;
if (browserUseTerminal(state.status) && drained) {
const summary = await client.summary(run.providerRunId);
const hold = budgetHold(run.result);
const saved = hold?.receipt;
const pendingReceipt = hold && saved && (centsToUnits(saved.totalCentsExact) > centsToUnits(hold.accountedCentsExact ?? run.accountedCents) || hold.state === "held");
const summary = pendingReceipt ? undefined : await client.summary(run.providerRunId);
await account(s, run, summary);
await signal(s, "browser.run_finished");
}
@@ -1277,7 +1385,7 @@ export function browserUseService(
await db
.update(sessions)
.set({ status: "stopping", stopRequested: "end" })
.where(eq(sessions.id, s.id));
.where(sessionLeaseFence(s));
const live = await db
.select()
.from(browsers)
@@ -1302,7 +1410,7 @@ export function browserUseService(
await db
.update(sessions)
.set({ status: "closed", error: null, stopRequested: null })
.where(eq(sessions.id, s.id));
.where(sessionLeaseFence(s));
await signal(s, "browser.closed");
}
} else if (
@@ -1321,7 +1429,7 @@ export function browserUseService(
})
.where(
and(
eq(sessions.id, s.id),
sessionLeaseFence(s),
ne(sessions.status, "starting"),
or(
isNull(sessions.stopRequested),
@@ -1378,6 +1486,7 @@ export function browserUseService(
)
.returning();
if (!lease.length) continue;
Object.assign(s, lease[0]);
// Renew while paginating provider events or stopping multiple browsers.
const renewal = setInterval(() => {
void db
@@ -1403,7 +1512,7 @@ export function browserUseService(
? "Browser shutdown is not confirmed. Paperclip will retry."
: "Browser synchronization failed. Paperclip will retry.",
})
.where(eq(sessions.id, s.id));
.where(sessionLeaseFence(s));
} finally {
clearInterval(renewal);
const release = (tx: Pick<Db, "update">) => tx.update(sessions).set({
@@ -0,0 +1,16 @@
import { and, eq } from "drizzle-orm";
import { companies, projects, type Db } from "@paperclipai/db";
import { notFound } from "../errors.js";
/** Shared execution-authority fences allow concurrent admissions. Acquire
* before feature/agent/run locks; they exclude manual pause/delete until this
* admission commits. Settlements and eligibility-only reads never call this. */
export async function lockBudgetAdmissionControls(tx: Db, companyId: string, projectId?: string | null) {
const [company] = await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, companyId)).for("share");
if (!company) throw notFound("Company not found");
if (projectId) {
const [project] = await tx.select({ id: projects.id }).from(projects)
.where(and(eq(projects.id, projectId), eq(projects.companyId, companyId))).for("share");
if (!project) throw notFound("Budget project not found");
}
}
@@ -7,8 +7,10 @@ import {
hasPendingBudgetEnforcement,
nextBudgetReconciliationAt,
reconcileDueBudgetPolicies,
flushBudgetEvaluationWorkPage,
type BudgetServiceHooks,
} from "./budgets.js";
import { hasPendingBudgetEvaluation } from "./budget-evaluation-work.js";
import { readCostAccountingWork } from "./run-cost-accounting.js";
import { subscribeAllCompanyLiveEvents } from "./live-events.js";
@@ -26,28 +28,35 @@ export function wakeCostAccountingWork(db: Db): void {
export function registerBudgetEnforcementWork(db: Db, coordinator: Coordinator, hooks: BudgetServiceHooks) {
let startup = true;
let reconciliationCursor: string | null = null;
let evaluationCursor: string | null = null;
let reconciling = false;
let reconciliationFailed = false;
let nextDeadline: number | null = null;
return coordinator.register(DELIVERY_QUEUES.budgetEnforcement, {
retryMs: 1_000,
coalesceMs: 1_000,
runDuringDrain: true,
async run(signal) {
if (signal.aborted) return;
if (startup || reconciling || (nextDeadline !== null && nextDeadline <= Date.now())) {
const page = await reconcileDueBudgetPolicies(db, { startup, ...(reconciliationCursor ? { cursor: reconciliationCursor } : {}) });
reconciliationCursor = page.nextCursor;
reconciliationFailed ||= Boolean(page.hadFailures);
reconciling = reconciliationCursor !== null;
if (!reconciling) startup = false;
if (!reconciling) { startup = reconciliationFailed; reconciliationFailed = false; }
}
if (signal.aborted) return;
const evaluation = await flushBudgetEvaluationWorkPage(db, { ...(evaluationCursor ? { cursor: evaluationCursor } : {}) });
evaluationCursor = evaluation.nextCursor;
if (signal.aborted) return;
await withBudgetSpan("budget.enforcement.deliver", () => deliverBudgetEnforcement(db, hooks));
},
async hasPending() {
const [pending, next] = await Promise.all([
hasPendingBudgetEnforcement(db), nextBudgetReconciliationAt(db),
const [pending, next, evaluation] = await Promise.all([
hasPendingBudgetEnforcement(db), nextBudgetReconciliationAt(db), hasPendingBudgetEvaluation(db),
]);
nextDeadline = next?.getTime() ?? null;
return pending || reconciling;
return pending || evaluation || reconciling || startup || evaluationCursor !== null;
},
nextRunAt: () => nextDeadline,
});
@@ -0,0 +1,26 @@
import { and, eq, or, sql } from "drizzle-orm";
import { budgetEvaluationWork, type Db } from "@paperclipai/db";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type BudgetEvaluationInput = { companyId: string; agentId?: string | null; projectId?: string | null };
/** Call only for an actual committed exposure change, on that change's tx. */
export async function queueBudgetEvaluation(tx: Db, input: BudgetEvaluationInput): Promise<void> {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.budgetEnforcement, input.companyId);
await tx.insert(budgetEvaluationWork).values(input);
}
export async function hasPendingBudgetEvaluation(db: Db): Promise<boolean> {
const [row] = await db.select({ id: budgetEvaluationWork.id }).from(budgetEvaluationWork).limit(1);
return Boolean(row);
}
/** A completed idle decision does not age. Only unconsumed relevant inputs do. */
export async function oldestBudgetInputAt(db: Db, input: BudgetEvaluationInput): Promise<Date | null> {
const [row] = await db.select({ oldest: sql<Date | null>`min(${budgetEvaluationWork.createdAt})` }).from(budgetEvaluationWork)
.where(and(eq(budgetEvaluationWork.companyId, input.companyId), !input.agentId && !input.projectId ? undefined : or(
input.agentId ? eq(budgetEvaluationWork.agentId, input.agentId) : undefined,
input.projectId ? eq(budgetEvaluationWork.projectId, input.projectId) : undefined,
)));
return row?.oldest ? new Date(row.oldest) : null;
}
+37 -15
View File
@@ -113,16 +113,17 @@ async function readObservedSpends(db: Db, policies: SpendPolicy[], now: Date): P
const inWindow = policy.windowKind === "calendar_month_utc"
? sql`coalesce(${heartbeatRuns.finishedAt}, ${now.toISOString()}::timestamptz) >= ${start.toISOString()}::timestamptz
and coalesce(${heartbeatRuns.finishedAt}, ${now.toISOString()}::timestamptz) < ${end.toISOString()}::timestamptz` : sql`true`;
const run = and(eq(heartbeatRuns.companyId, policy.companyId), eq(heartbeatRuns.costAccountingPending, true),
const lateReceiptDebt = sql`coalesce(${usage}->>'accountingLateReceiptPending', 'false') = 'true'`;
const run = and(eq(heartbeatRuns.companyId, policy.companyId), or(eq(heartbeatRuns.costAccountingPending, true), lateReceiptDebt),
policy.scopeType === "agent" ? eq(heartbeatRuns.agentId, policy.scopeId) : undefined,
policy.scopeType === "project" ? sql`${usage}->'ledgerScope'->>'projectId' = ${policy.scopeId}` : undefined)!;
const terminal = and(run, inArray(heartbeatRuns.status, ["succeeded", "failed", "timed_out", "cancelled", "interrupted"]), inWindow)!;
const terminal = and(run, inArray(heartbeatRuns.status, ["succeeded", "failed", "timed_out", "cancelled", "interrupted"]), or(inWindow, lateReceiptDebt))!;
// Keep correlated subqueries in nested SQL chunks. Drizzle strips column
// qualifiers from a single-table selection's direct column chunks.
const hasReservation = sql`exists (select 1 from ${budgetReservations}
where ${budgetReservations.companyId} = ${heartbeatRuns.companyId}
and ${budgetReservations.runId} = ${heartbeatRuns.id} and ${budgetReservations.state} = 'held')`;
const recovering = sql`exists (select 1 from ${nativeRunFinalizations}
const recovering = sql`not (${lateReceiptDebt}) and exists (select 1 from ${nativeRunFinalizations}
where ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
and ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId}
and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} <> 'terminal_failure')`;
@@ -137,15 +138,28 @@ async function readObservedSpends(db: Db, policies: SpendPolicy[], now: Date): P
runColumns[`pending${index}`] = sql`count(*) filter (where ${terminal})::int`;
runColumns[`recovering${index}`] = sql`count(*) filter (where ${terminal} and ${heartbeatRuns.runtimeMode} = 'native' and ${recovering})::int`;
}
const [ledger = {}] = billable.length ? await db.select(ledgerColumns).from(costEvents).where(or(...ledgerPredicates)) : [];
const [held = {}] = billable.length ? await db.select(heldColumns).from(budgetReservations)
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, budgetReservations.runId), eq(heartbeatRuns.companyId, budgetReservations.companyId)))
.where(or(...heldPredicates)) : [];
const [runs = {}] = billable.length ? await db.select(runColumns).from(heartbeatRuns).where(or(...runPredicates)) : [];
const [browser = {}] = billable.length ? await db.select(browserColumns).from(browserUseRuns)
.innerJoin(browserUseSessions, and(eq(browserUseSessions.id, browserUseRuns.sessionId), eq(browserUseSessions.companyId, browserUseRuns.companyId)))
.leftJoin(issues, and(eq(issues.id, browserUseSessions.issueId), eq(issues.companyId, browserUseSessions.companyId)))
.where(or(...browserPredicates)) : [];
// A single statement preserves hold-to-ledger atomicity even when a caller
// owns an otherwise READ COMMITTED transaction. Alias dynamic columns before
// JSON projection so the driver does not collapse repeated aggregate names.
const columns = (values: Record<string, SQL>) => Object.fromEntries(Object.entries(values).map(([key, value]) => [key, value.as(key)]));
let ledger: Record<string, string> = {}, held: Record<string, string> = {}, runs: Record<string, string> = {}, browser: Record<string, string> = {};
if (billable.length) {
const ledgerQuery = db.select(columns(ledgerColumns)).from(costEvents).where(or(...ledgerPredicates));
const heldQuery = db.select(columns(heldColumns)).from(budgetReservations)
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, budgetReservations.runId), eq(heartbeatRuns.companyId, budgetReservations.companyId)))
.where(or(...heldPredicates));
const runsQuery = db.select(columns(runColumns)).from(heartbeatRuns).where(or(...runPredicates));
const browserQuery = db.select(columns(browserColumns)).from(browserUseRuns)
.innerJoin(browserUseSessions, and(eq(browserUseSessions.id, browserUseRuns.sessionId), eq(browserUseSessions.companyId, browserUseSessions.companyId)))
.leftJoin(issues, and(eq(issues.id, browserUseSessions.issueId), eq(issues.companyId, browserUseSessions.companyId)))
.where(or(...browserPredicates));
const [row] = await db.execute<{ ledger: Record<string, string>; held: Record<string, string>; runs: Record<string, string>; browser: Record<string, string> }>(sql`
select row_to_json(ledger) as ledger, row_to_json(held) as held,
row_to_json(runs) as runs, row_to_json(browser) as browser
from (${ledgerQuery}) ledger cross join (${heldQuery}) held
cross join (${runsQuery}) runs cross join (${browserQuery}) browser`);
({ ledger, held, runs, browser } = row);
}
return policies.map((_, index) => {
const ledgerCents = normalizeCents(String(ledger[`total${index}`] ?? 0));
const liveObservedCents = normalizeCents(String(runs[`live${index}`] ?? 0));
@@ -158,10 +172,18 @@ async function readObservedSpends(db: Db, policies: SpendPolicy[], now: Date): P
});
}
export async function computeObservedSpend(db: Db, policy: SpendPolicy) {
return (await computeObservedSpends(db, [policy]))[0];
export async function computeObservedSpend(db: Db, policy: SpendPolicy, now = new Date()) {
return (await computeObservedSpends(db, [policy], now))[0];
}
export function computeObservedSpends(db: Db, policies: SpendPolicy[], now = new Date()): Promise<ObservedSpend[]> {
return withBudgetSpan("budget.exposure.evaluate", () => readObservedSpends(db, policies, now), { "budget.policy_count": policies.length });
return withBudgetSpan("budget.exposure.evaluate", async () => {
// Each policy contributes several aggregate columns. Bound statement width
// below PostgreSQL's target-list limit. Multi-scope evaluators/reports own
// a repeatable-read transaction; individual admission batches fit one SQL.
const result: ObservedSpend[] = [];
for (let offset = 0; offset < policies.length; offset += 128)
result.push(...await readObservedSpends(db, policies.slice(offset, offset + 128), now));
return result;
}, { "budget.policy_count": policies.length });
}
+10 -2
View File
@@ -2,7 +2,9 @@ import { and, eq, sql } from "drizzle-orm";
import { agents, budgetReservations, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { centsToUnits, unitsToCents } from "@paperclipai/shared";
import { conflict, notFound } from "../errors.js";
import { withAccountingTransaction } from "./accounting-transaction.js";
import { withOperationAccountingTransaction } from "./accounting-transaction.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
import { lockBudgetAdmissionControls } from "./budget-admission-controls.js";
import { budgetServiceInTransaction } from "./budgets.js";
import { withBudgetSpan } from "./budget-tracing.js";
import type { ActivityPublication } from "./activity-log.js";
@@ -74,7 +76,12 @@ export async function recheckRunBudgetReservationInTransaction(
* changes. Estimates constrain admission; they cannot cap a provider's bill.
* A reservation survives timeouts and restarts until accounting proves closure. */
export async function reserveRunBudget(db: Db, companyId: string, runId: string, projectId: string | null, ledgerScope: Record<string, unknown> = {}, recoveryLeaseOwner?: string) {
const result = await withBudgetSpan("budget.admission.reserve", () => withAccountingTransaction(db, companyId, async (tx, publications) => {
const result = await withBudgetSpan("budget.admission.reserve", () => withOperationAccountingTransaction(db, companyId, async (tx, publications) => {
await lockBudgetAdmissionControls(tx, companyId, projectId);
const [hint] = await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId)));
if (!hint) throw notFound("Run not found");
await tx.select({ id: agents.id }).from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, hint.agentId))).for("share");
await lockRunBudgetRecoveryOwnerInTransaction(tx, companyId, runId, recoveryLeaseOwner);
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))).for("update");
if (!run) throw notFound("Run not found");
@@ -129,6 +136,7 @@ export async function reserveRunBudget(db: Db, companyId: string, runId: string,
await tx.update(heartbeatRuns).set({ costAccountingPending: true,
usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || ${JSON.stringify({ accountingReceiptReady: false, ledgerScope: { ...(run.usageJson?.ledgerScope as object ?? {}), ...ledgerScope, projectId } })}::jsonb`,
}).where(eq(heartbeatRuns.id, runId));
await queueBudgetEvaluation(tx, { companyId, agentId: run.agentId, projectId });
return { ...reservation, reused: false };
}));
// Commit required block/resume transitions even when dispatch is denied.
+7 -1
View File
@@ -9,7 +9,13 @@ type BudgetOperation = "budget.accounting.lock_wait" | "budget.accounting.recove
/** Operator-gated OpenTelemetry only; these spans contain no prompt, receipt,
* credential, or provider error text. Diagnostics never change accounting. */
export function withBudgetSpan<T>(name: BudgetOperation, work: () => Promise<T>, attributes?: Record<string, number | boolean>) {
type BudgetAttributes = {
"budget.accounting.owner"?: "run" | "decision" | "classifier" | "fast_response" | "browser";
"budget.enforcement.queue_delay_ms"?: number;
"budget.policy_count"?: number;
};
export function withBudgetSpan<T>(name: BudgetOperation, work: () => Promise<T>, attributes?: BudgetAttributes) {
return withDatabaseOperation(name, async () => {
if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return work();
let operation: Promise<T> | undefined;
+196 -45
View File
@@ -11,6 +11,7 @@ import type { Db } from "@paperclipai/db";
import {
agents,
approvals,
budgetEvaluationWork,
budgetIncidents,
budgetPolicies,
budgetReservations,
@@ -34,12 +35,13 @@ import type {
} from "@paperclipai/shared";
import { HttpError, notFound, unprocessable } from "../errors.js";
import { logger } from "../middleware/logger.js";
import { withAccountingReadSnapshot, withAccountingTransaction } from "./accounting-transaction.js";
import { withAccountingReadSnapshot, withAccountingTransaction, withOperationAccountingTransaction } from "./accounting-transaction.js";
import { logActivity, type LogActivityInput, type ActivityPublication } from "./activity-log.js";
import { computeObservedSpend, computeObservedSpends, type ObservedSpend } from "./budget-exposure.js";
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { listPendingBudgetCancellationScopes } from "./heartbeat/budget-cancellation.js";
import { withBudgetSpan } from "./budget-tracing.js";
import { queueBudgetEvaluation, oldestBudgetInputAt } from "./budget-evaluation-work.js";
export { computeObservedSpend, computeObservedSpends } from "./budget-exposure.js";
export type { BudgetExposure } from "./budget-exposure.js";
@@ -347,8 +349,11 @@ async function markApprovalStatus(
.where(eq(approvals.id, approvalId));
}
/** Internal: caller must hold the company accounting transaction lock. */
export function budgetServiceInTransaction(db: Db, publications: ActivityPublication[] = []) {
/** Owner transactions read eligibility or append work. Only the fenced worker
* and explicit policy edits evaluate and publish aggregate transitions. */
export function budgetServiceInTransaction(db: Db, publications: ActivityPublication[] = [],
options: { evaluate?: boolean; now?: Date } = {}) {
const now = options.now ?? new Date();
const recordActivity = (input: LogActivityInput) => logActivity(db, input, publications);
async function pauseScopeForBudget(policy: PolicyRow) {
const now = new Date();
@@ -419,9 +424,8 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
eq(budgetPolicies.companyId, policy.companyId), eq(budgetPolicies.scopeType, policy.scopeType),
eq(budgetPolicies.scopeId, policy.scopeId), eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true),
));
const observations = await computeObservedSpends(db, policies);
const observations = await computeObservedSpends(db, policies, now);
if (policies.some((candidate, index) => observedBlocks(candidate, observations[index]))) return;
const now = new Date();
if (policy.scopeType === "agent") {
const { applyAgentBudgetHoldInTransaction } = await import("./agent-lifecycle.js");
await applyAgentBudgetHoldInTransaction(db, policy.companyId, policy.scopeId, false);
@@ -485,7 +489,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
async function buildPolicySummary(policy: PolicyRow): Promise<BudgetPolicySummary> {
const scope = await resolveSummaryScopeRecord(db, policy.companyId, policy.scopeType as BudgetScopeType, policy.scopeId);
const observed = await computeObservedSpend(db, policy);
const observed = await computeObservedSpend(db, policy, now);
const { total: observedAmount, totalExact: observedAmountExact, unpricedEventCount, pendingRunCount } = observed;
const { windowStart: start, windowEnd: end } = observed;
const amount = policy.isActive ? policy.amount : 0;
@@ -686,7 +690,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
eq(budgetPolicies.companyId, companyId), eq(budgetPolicies.scopeType, scopeType), eq(budgetPolicies.scopeId, scopeId),
));
let recoveringPolicy: PolicyRow | undefined;
const observations = prepared?.observations ?? await computeObservedSpends(db, policies);
const observations = prepared?.observations ?? await computeObservedSpends(db, policies, now);
for (const [index, policy] of policies.entries()) {
const observed = observations[index];
if (observedBlocks(policy, observed)) {
@@ -721,8 +725,9 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
return null;
}
async function evaluateAffectedScopes(companyId: string, agentId: string | null, projectId: string | null) {
const candidatePolicies = await db
async function evaluateAffectedScopes(companyId: string, agentId: string | null, projectId: string | null,
prepared?: { policies: PolicyRow[]; observations: ObservedSpend[] }) {
const candidatePolicies = prepared?.policies ?? await db
.select()
.from(budgetPolicies)
.where(
@@ -733,8 +738,9 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
),
);
const policies = candidatePolicies.filter(policy => policy.metric === "billed_cents" && policy.amount > 0);
const observations = await computeObservedSpends(db, policies);
const policies = candidatePolicies.filter(policy => policy.isActive && policy.metric === "billed_cents" && policy.amount > 0);
const observations = prepared ? policies.map(policy => prepared.observations[prepared.policies.findIndex(row => row.id === policy.id)])
: await computeObservedSpends(db, policies, now);
for (const [index, policy] of policies.entries()) {
const observed = observations[index];
const observedAmount = observed.total;
@@ -798,15 +804,40 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
async function evaluateAdmissionInTransaction(companyId: string, agentId: string | null,
context?: { issueId?: string | null; projectId?: string | null }) {
const { scopes, records, policies } = await readAdmissionScopes(db, companyId, agentId, context);
const observations = await computeObservedSpends(db, policies);
const observations = await computeObservedSpends(db, policies, now);
const reservationIndexes = policies.map((policy, i) => policy.isActive && policy.hardStopEnabled && policy.metric === "billed_cents" && policy.amount > 0 ? i : -1).filter(i => i >= 0);
const admission = (block: Awaited<ReturnType<typeof reconcileScope>>) => ({ block,
policies: reservationIndexes.map(i => policies[i]), observations: reservationIndexes.map(i => observations[i]) });
for (const [index, { scopeType, scopeId }] of scopes.entries()) {
const indexes = policies.map((policy, i) => policy.scopeType === scopeType && policy.scopeId === scopeId ? i : -1).filter(i => i >= 0);
const block = await reconcileScope(companyId, scopeType, scopeId, { scope: records[index],
policies: indexes.map(i => policies[i]), observations: indexes.map(i => observations[i]) });
if (block) return admission(block);
const scope = records[index];
const scoped = policies.map((policy, i) => ({ policy, observed: observations[i] }))
.filter(({ policy }) => policy.scopeType === scopeType && policy.scopeId === scopeId);
const pause = scopePauseBlock(scopeType, scopeId, scope, scoped.length > 0);
if (pause) return admission(pause);
for (const { policy, observed } of scoped) {
const title = scopeType === "company" ? "Company" : scopeType === "agent" ? "Agent" : "Project";
if (observedBlocks(policy, observed)) return admission({ scopeType, scopeId, scopeName: scope.name,
reason: observed.pendingRunCount > 0 ? `${title} cannot start work while completed runs await accounting.`
: unpricedUsageBlocks(policy, observed) ? `${title} cannot start work because recorded usage has no reliable price.`
: scopeType === "project" ? "Project cannot start work because its budget hard-stop is still exceeded."
: `${title} is paused because its budget hard-stop was reached.` });
if (policy.isActive && policy.hardStopEnabled && policy.amount > 0 && observed.recoveringRunCount > 0)
return admission({ scopeType, scopeId, scopeName: scope.name,
reason: "New work must wait while a native run recovers its unfinished accounting." });
}
if (scope.budgetHeld) return admission({ scopeType, scopeId, scopeName: scope.name,
reason: "Budget hold is awaiting reconciliation." });
}
if (reservationIndexes.length > 0) {
for (const index of reservationIndexes) {
const policy = policies[index];
const oldest = await oldestBudgetInputAt(db, { companyId,
agentId: policy.scopeType === "agent" ? policy.scopeId : null,
projectId: policy.scopeType === "project" ? policy.scopeId : null });
if (oldest && now.getTime() - oldest.getTime() > 15_000) return admission({ scopeType: policy.scopeType as BudgetScopeType,
scopeId: policy.scopeId, scopeName: records.find((_, i) => scopes[i].scopeType === policy.scopeType)?.name ?? "Budget scope",
reason: "Budget accounting changes are awaiting reconciliation." });
}
}
return admission(null);
}
@@ -900,6 +931,12 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
.returning()
.then((rows) => rows[0]);
// A newly created policy can race a no-policy batch that already consumed
// its charge. Its own committed input must trigger a fresh evaluation.
await queueBudgetEvaluation(db, { companyId,
agentId: input.scopeType === "agent" ? input.scopeId : null,
projectId: input.scopeType === "project" ? input.scopeId : null });
if (input.scopeType === "company" && windowKind === "calendar_month_utc") {
await db
.update(companies)
@@ -983,8 +1020,11 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
};
},
evaluateAffectedScopes,
evaluateCostEvent: (event: typeof costEvents.$inferSelect) => evaluateAffectedScopes(event.companyId, event.agentId, event.projectId),
evaluateAffectedScopes: options.evaluate ? evaluateAffectedScopes
: (companyId: string, agentId: string | null, projectId: string | null) => queueBudgetEvaluation(db, { companyId, agentId, projectId }),
evaluateCostEvent: (event: typeof costEvents.$inferSelect) => options.evaluate
? evaluateAffectedScopes(event.companyId, event.agentId, event.projectId)
: queueBudgetEvaluation(db, { companyId: event.companyId, agentId: event.agentId, projectId: event.projectId }),
evaluateAdmission,
getInvocationBlock: async (companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) =>
@@ -1013,8 +1053,12 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
: await resolveScopeRecord(db, policy.scopeType as BudgetScopeType, policy.scopeId);
if (scope.companyId !== companyId) throw notFound("Budget scope not found");
if (input.action === "raise_budget_and_resume") {
const now = new Date();
await queueBudgetEvaluation(db, { companyId,
agentId: policy.scopeType === "agent" ? policy.scopeId : null,
projectId: policy.scopeType === "project" ? policy.scopeId : null });
const nextAmount = Math.max(0, Math.floor(input.amount ?? 0));
const observed = await computeObservedSpend(db, policy);
const observed = await computeObservedSpend(db, policy, now);
if (unpricedUsageBlocks(policy, observed)) {
throw unprocessable("Resolve unpriced usage or explicitly allow it in the budget policy before resuming");
}
@@ -1023,7 +1067,6 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
throw unprocessable("New budget must exceed current observed spend");
}
const now = new Date();
await notifyAttentionRetentionWork(db, { companyId, sourceKind: "budget_alert" });
await notifyDeliveryWork(db, DELIVERY_QUEUES.budgetEnforcement, companyId);
await db
@@ -1104,11 +1147,11 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
* process shutdown must happen after this transaction, without holding locks
* that the executor needs to persist its final usage receipt. */
export async function withCurrentBudgetEnforcement<T>(db: Db, scope: BudgetEnforcementScope, work: (tx: Db) => Promise<T>): Promise<T | null> {
return withAccountingTransaction(db, scope.companyId, async (tx) => {
return withOperationAccountingTransaction(db, scope.companyId, async (tx) => {
if (scope.enforcement) {
const [policy] = await tx.select().from(budgetPolicies).where(and(
eq(budgetPolicies.id, scope.enforcement.policyId), eq(budgetPolicies.companyId, scope.companyId),
));
)).for("update");
if (!policy || policy.scopeType !== scope.scopeType || policy.scopeId !== scope.scopeId
|| policy.enforcementVersion !== scope.enforcement.version || !(await policyBlocks(tx, policy))) return null;
}
@@ -1135,7 +1178,7 @@ export async function deliverBudgetEnforcement(db: Db, hooks: BudgetServiceHooks
)).orderBy(budgetPolicies.updatedAt, budgetPolicies.id).limit(100);
await deliverScopes(pending, policy => withBudgetSpan("budget.enforcement.scope", async () => {
try {
const cancellation = await withAccountingTransaction(db, policy.companyId, async tx => {
const cancellation = await withOperationAccountingTransaction(db, policy.companyId, async tx => {
const [current] = await tx.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id));
if (!current || current.enforcementVersion !== policy.enforcementVersion || !(await policyBlocks(tx, current))) return null;
return { companyId: policy.companyId, scopeType: policy.scopeType as BudgetScopeType, scopeId: policy.scopeId,
@@ -1171,6 +1214,96 @@ export async function hasPendingBudgetEnforcement(db: Db): Promise<boolean> {
return Boolean(pending) || (await listPendingBudgetCancellationScopes(db, 1)).length > 0;
}
/** Policy-only coordination, off the financial path. RR detects a policy edit
* or newer evaluator after this snapshot; every pass advances its publication
* fence even when no threshold transition is needed. */
async function fenceBudgetPolicies(tx: Db, companyId: string, extra?: { scopeType: string; scopeId: string }) {
const policies = await tx.select().from(budgetPolicies).where(eq(budgetPolicies.companyId, companyId))
.orderBy(budgetPolicies.id).for("update");
// Existing control/configuration writers own their resource before changing
// policies. Never wait for those resources while holding the policy fence:
// relinquish this transaction so the control writer can finish. These locks
// are only for aggregate publication, never receipt or settlement commits.
const scopes = [...policies, ...(extra ? [extra] : [])];
try {
for (const [table, scopeType] of [[companies, "company"], [agents, "agent"], [projects, "project"]] as const) {
const ids = [...new Set(scopes.filter(scope => scope.scopeType === scopeType).map(scope => scope.scopeId))].sort();
if (ids.length) await tx.select({ id: table.id }).from(table).where(inArray(table.id, ids))
.orderBy(table.id).for("no key update", { noWait: true });
}
} catch (error) {
const code = (error as { cause?: { code?: string }; code?: string }).cause?.code ?? (error as { code?: string }).code;
if (code === "55P03") throw Object.assign(new Error("Budget control changed during evaluation", { cause: error }), { code: "40001" });
throw error;
}
if (policies.length) await tx.update(budgetPolicies).set({ evaluationVersion: sql`${budgetPolicies.evaluationVersion} + 1` })
.where(inArray(budgetPolicies.id, policies.map(row => row.id)));
return policies;
}
/** Consume selected committed IDs only. Earlier allocated IDs that commit
* later remain discoverable; no company high-water mark is written. */
export async function flushBudgetEvaluationWork(db: Db, options: { companyId?: string } = {}): Promise<number> {
return (await flushBudgetEvaluationWorkPage(db, options)).consumed;
}
/** The cursor rotates bounded company pages; it is a scheduling hint only.
* Pending inputs remain durable across retries, lower-ID arrivals and restart. */
export async function flushBudgetEvaluationWorkPage(db: Db, options: { companyId?: string; cursor?: string | null } = {}): Promise<{ consumed: number; nextCursor: string | null }> {
const pending = await db.selectDistinct({ companyId: budgetEvaluationWork.companyId }).from(budgetEvaluationWork)
.where(and(options.companyId ? eq(budgetEvaluationWork.companyId, options.companyId) : undefined,
options.cursor ? sql`${budgetEvaluationWork.companyId} > ${options.cursor}::uuid` : undefined))
.orderBy(budgetEvaluationWork.companyId).limit(100);
let consumed = 0;
for (const { companyId } of pending) {
try {
consumed += await withOperationAccountingTransaction(db, companyId, async (tx, publications) => {
const now = new Date();
const inputs = await tx.select().from(budgetEvaluationWork).where(eq(budgetEvaluationWork.companyId, companyId))
.orderBy(budgetEvaluationWork.createdAt, budgetEvaluationWork.id).limit(500);
if (!inputs.length) return 0;
const agents = new Set(inputs.map(input => input.agentId));
const projects = new Set(inputs.map(input => input.projectId));
const policies = (await fenceBudgetPolicies(tx, companyId)).filter(policy => policy.scopeType === "company"
|| (policy.scopeType === "agent" ? agents : projects).has(policy.scopeId));
const existing: PolicyRow[] = [];
const scopes = new Map<string, ScopeRecord>();
for (const policy of policies) {
const key = `${policy.scopeType}:${policy.scopeId}`;
try {
if (!scopes.has(key)) scopes.set(key, await resolveScopeRecord(tx, policy.scopeType as BudgetScopeType, policy.scopeId));
existing.push(policy);
} catch (error) {
if (!(error instanceof HttpError) || error.status !== 404) throw error;
// Archived charges outlive deleted scope resources; they still affect
// the company policy, without resurrecting the deleted scope.
}
}
const observations = await computeObservedSpends(tx, existing, now);
const service = budgetServiceInTransaction(tx, publications, { evaluate: true, now });
await service.evaluateAffectedScopes(companyId, null, null, { policies: existing, observations });
for (const [key, scope] of scopes) {
const scoped = existing.map((policy, index) => ({ policy, observed: observations[index] }))
.filter(({ policy }) => `${policy.scopeType}:${policy.scopeId}` === key);
if (!scoped.length) continue;
await service.reconcileScope(companyId, scoped[0].policy.scopeType as BudgetScopeType, scoped[0].policy.scopeId,
{ scope, policies: scoped.map(row => row.policy), observations: scoped.map(row => row.observed) });
}
if (existing.some(policy => policy.windowKind === "calendar_month_utc") && Date.now() >= currentUtcMonthWindow(now).end.getTime())
throw Object.assign(new Error("Budget window changed during evaluation"), { code: "40001" });
await tx.delete(budgetEvaluationWork).where(inArray(budgetEvaluationWork.id, inputs.map(input => input.id)));
return inputs.length;
});
} catch (error) {
// One company's control contention or malformed retained input must not
// suppress another company's evaluations or already-committed Stops.
// The rolled-back input IDs stay pending for the coordinator's retry.
logger.warn({ err: error, companyId }, "Budget evaluation remains pending");
}
}
return { consumed, nextCursor: pending.length === 100 ? pending[pending.length - 1].companyId : null };
}
const budgetOwnedScope = or(
and(eq(budgetPolicies.scopeType, "company"), sql`exists (select 1 from ${companies} where ${companies.id} = ${budgetPolicies.scopeId} and ${companies.pauseReason} = 'budget')`),
and(eq(budgetPolicies.scopeType, "agent"), sql`exists (select 1 from ${agents} where ${agents.id} = ${budgetPolicies.scopeId} and ${agents.lifecycleHolds} @> '["budget"]'::jsonb)`),
@@ -1193,7 +1326,7 @@ export async function nextBudgetReconciliationAt(db: Db): Promise<Date | null> {
/** Startup repairs configured policy scopes, not every agent. Afterwards only
* held monthly scopes owe a timed reevaluation; writes reconcile their own scopes. */
export async function reconcileDueBudgetPolicies(db: Db, options: { startup?: boolean; cursor?: string | null } = {}): Promise<{ nextCursor: string | null }> {
export async function reconcileDueBudgetPolicies(db: Db, options: { startup?: boolean; cursor?: string | null } = {}): Promise<{ nextCursor: string | null; hadFailures?: boolean }> {
const actionable = options.startup ? or(budgetOwnedScope, openIncidentScope,
sql`${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}`,
and(eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true), sql`${budgetPolicies.amount} > 0`, unfinishedRunScope))
@@ -1202,42 +1335,60 @@ export async function reconcileDueBudgetPolicies(db: Db, options: { startup?: bo
.from(budgetPolicies).where(and(actionable, options.cursor ? sql`${budgetPolicies.id} > ${options.cursor}::uuid` : undefined))
.orderBy(budgetPolicies.id).limit(100);
const scopes = new Map(rows.map(scope => [`${scope.companyId}:${scope.scopeType}:${scope.scopeId}`, scope]));
let hadFailures = false;
for (const scope of scopes.values()) {
try {
await withAccountingTransaction(db, scope.companyId, (tx, publications) => budgetServiceInTransaction(tx, publications)
.reconcileScope(scope.companyId, scope.scopeType as BudgetScopeType, scope.scopeId));
await withOperationAccountingTransaction(db, scope.companyId, async (tx, publications) => {
const now = new Date();
await fenceBudgetPolicies(tx, scope.companyId, scope);
const result = await budgetServiceInTransaction(tx, publications, { evaluate: true, now })
.reconcileScope(scope.companyId, scope.scopeType as BudgetScopeType, scope.scopeId);
if (Date.now() >= currentUtcMonthWindow(now).end.getTime())
throw Object.assign(new Error("Budget window changed during reconciliation"), { code: "40001" });
return result;
});
} catch (error) {
if (error instanceof HttpError && error.status === 404) continue;
throw error;
hadFailures = true;
logger.warn({ err: error, companyId: scope.companyId, policyId: scope.id }, "Budget scope reconciliation remains pending");
}
}
return { nextCursor: rows.length === 100 ? rows[rows.length - 1].id : null };
return { nextCursor: rows.length === 100 ? rows[rows.length - 1].id : null, ...(hadFailures ? { hadFailures: true } : {}) };
}
export function budgetService(db: Db, _hooks: BudgetServiceHooks = {}) {
const reads = budgetServiceInTransaction(db);
const mutate = <T>(companyId: string, work: (service: ReturnType<typeof budgetServiceInTransaction>) => Promise<T>) =>
withAccountingTransaction(db, companyId, (tx, publications) => work(budgetServiceInTransaction(tx, publications)));
const mutate = <T>(companyId: string, work: (service: ReturnType<typeof budgetServiceInTransaction>) => Promise<T>,
scope?: { scopeType: string; scopeId: string }) => withOperationAccountingTransaction(db, companyId, async (tx, publications) => {
const now = new Date();
const policies = await fenceBudgetPolicies(tx, companyId, scope);
const result = await work(budgetServiceInTransaction(tx, publications, { evaluate: true, now }));
if ((policies.some(policy => policy.windowKind === "calendar_month_utc") || scope) && Date.now() >= currentUtcMonthWindow(now).end.getTime())
throw Object.assign(new Error("Budget window changed during publication"), { code: "40001" });
return result;
});
return {
...reads,
listPolicies: (companyId: string) => withAccountingReadSnapshot(db, companyId,
tx => budgetServiceInTransaction(tx).listPolicies(companyId)),
overview: (companyId: string) => withAccountingReadSnapshot(db, companyId,
tx => budgetServiceInTransaction(tx).overview(companyId)),
evaluateAdmission: (companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) =>
mutate(companyId, service => service.evaluateAdmission(companyId, agentId, context)),
withAccountingReadSnapshot(db, companyId, tx => budgetServiceInTransaction(tx).evaluateAdmission(companyId, agentId, context)),
evaluateAffectedScopes: (companyId: string, agentId: string | null, projectId: string | null) =>
mutate(companyId, service => service.evaluateAffectedScopes(companyId, agentId, projectId)),
withOperationAccountingTransaction(db, companyId, tx => queueBudgetEvaluation(tx, { companyId, agentId, projectId })),
reconcileScope: (companyId: string, scopeType: BudgetScopeType, scopeId: string) =>
mutate(companyId, service => service.reconcileScope(companyId, scopeType, scopeId)),
mutate(companyId, service => service.reconcileScope(companyId, scopeType, scopeId), { scopeType, scopeId }),
reconcilePolicies: () => reconcileDueBudgetPolicies(db, { startup: true }),
upsertPolicy: (companyId: string, input: BudgetPolicyUpsertInput, actorUserId: string | null) =>
mutate(companyId, service => service.upsertPolicy(companyId, input, actorUserId)),
evaluateCostEvent: (event: typeof costEvents.$inferSelect) => mutate(event.companyId, service => service.evaluateCostEvent(event)),
getInvocationBlock: async (companyId: string, agentId: string | null, context?: AdmissionContext) => {
const eligibility = await withBudgetSpan("budget.admission.inspect", () =>
withAccountingReadSnapshot(db, companyId, tx => inspectInvocationEligibility(tx, companyId, agentId, context)));
if (!eligibility.needsReconciliation) return eligibility.block;
return withBudgetSpan("budget.admission.reconcile", () =>
mutate(companyId, service => service.getInvocationBlock(companyId, agentId, context)));
mutate(companyId, service => service.upsertPolicy(companyId, input, actorUserId), input),
evaluateCostEvent: (event: typeof costEvents.$inferSelect) => withOperationAccountingTransaction(db, event.companyId, tx => queueBudgetEvaluation(tx, { companyId: event.companyId, agentId: event.agentId, projectId: event.projectId })),
getInvocationBlock: (companyId: string, agentId: string | null, context?: AdmissionContext) =>
withBudgetSpan("budget.admission.inspect", () => withAccountingReadSnapshot(db, companyId,
tx => budgetServiceInTransaction(tx).getInvocationBlock(companyId, agentId, context))),
resolveIncident: async (companyId: string, incidentId: string, input: BudgetIncidentResolutionInput, actorUserId: string) => {
const [scope] = await db.select({ scopeType: budgetIncidents.scopeType, scopeId: budgetIncidents.scopeId }).from(budgetIncidents)
.where(and(eq(budgetIncidents.id, incidentId), eq(budgetIncidents.companyId, companyId)));
if (!scope) throw notFound("Budget incident not found");
return mutate(companyId, service => service.resolveIncident(companyId, incidentId, input, actorUserId), scope);
},
resolveIncident: (companyId: string, incidentId: string, input: BudgetIncidentResolutionInput, actorUserId: string) =>
mutate(companyId, service => service.resolveIncident(companyId, incidentId, input, actorUserId)),
};
}
+17 -43
View File
@@ -3,11 +3,12 @@ import { agentAvatarUrl, resolveAgentAppearance, createCostEventSchema, createSe
import { and, desc, eq, gte, inArray, isNotNull, isNull, lt, lte, ne, sql } from "drizzle-orm";
import { alias } from "drizzle-orm/pg-core";
import type { Db } from "@paperclipai/db";
import { activityLog, agents, agentRuntimeState, authUsers, companies, companyMemberships, costEvents, heartbeatRuns, issues, projects, goals, aiSubscriptions } from "@paperclipai/db";
import { activityLog, agents, authUsers, companies, companyMemberships, costEvents, heartbeatRuns, issues, projects, goals, aiSubscriptions } from "@paperclipai/db";
import { notFound, unprocessable, conflict } from "../errors.js";
import { budgetServiceInTransaction, type BudgetServiceHooks } from "./budgets.js";
import type { BudgetServiceHooks } from "./budgets.js";
import { logActivity, type LogActivityInput, type ActivityPublication } from "./activity-log.js";
import { withAccountingReadSnapshot, withAccountingTransaction } from "./accounting-transaction.js";
import { withAccountingReadSnapshot, withOperationAccountingTransaction } from "./accounting-transaction.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
import { visibleIssueCondition } from "./issue-visibility.js";
import { resolveCostDateRange, type CostDateRange } from "./cost-date-range.js";
@@ -39,27 +40,6 @@ function currentUtcMonthWindow(now = new Date()) {
};
}
/** Increment an initialized month under the company lock. On upgrade or UTC
* rollover, initialize from the ledger in the same transaction exactly once.
* Reports, budgets and the integrity checker continue reading the ledger. */
export async function updateMonthlySpendProjections(db: Db, companyId: string, agentId: string | null, amountCents: MoneyInput, occurredAt: Date) {
const { start, end } = currentUtcMonthWindow();
const month = start.toISOString().slice(0, 10);
const delta = occurredAt >= start && occurredAt < end ? normalizeCents(amountCents) : "0.0000000";
for (const table of [companies, agents] as const) {
const id = table === companies ? companyId : agentId;
if (!id) continue;
const companyCondition = table === companies ? eq(companies.id, companyId) : eq(agents.companyId, companyId);
await db.update(table).set({
spentMonthlyCents: sql`case when ${table.spendMonthUtc} = ${month} then ${table.spentMonthlyCents} + ${delta}::numeric else (
select coalesce(sum(${costEvents.costCents}),0) from ${costEvents} where ${costEvents.companyId} = ${companyId}
and ${costEvents.occurredAt} >= ${start.toISOString()}::timestamptz and ${costEvents.occurredAt} < ${end.toISOString()}::timestamptz
${table === agents ? sql`and ${costEvents.agentId} = ${agentId}` : sql``}) end`,
spendMonthUtc: month, updatedAt: new Date(),
}).where(and(eq(table.id, id), companyCondition));
}
}
export async function getMonthlySpendTotal(
db: Db,
scope: { companyId: string; agentId?: string | null },
@@ -137,25 +117,19 @@ export async function createCostEventInTransaction(db: Db, companyId: string, da
}
// Server-derived attribution stays outside the immutable receipt fingerprint;
// replaying a receipt must not depend on later account metadata changes.
const [event] = await db.insert(costEvents).values({ ...values, subscriptionId, costCents: sql`${values.costCents}::numeric`, reportedCostCents: values.costCents, id: data.id, companyId, receiptHash }).returning();
await updateMonthlySpendProjections(db, companyId, event.agentId, values.costCents, event.occurredAt);
// Separately reported charges linked to an already-accounted run contribute
// to lifetime totals too. Before acknowledgement, accountRunCost includes
// all of the run's events atomically when it initializes the projection.
if (event.heartbeatRunId && event.agentId) {
const [run] = await db.select({ acknowledged: heartbeatRuns.costAccountedAt, version: heartbeatRuns.accountingProjectionVersion })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, event.heartbeatRunId), eq(heartbeatRuns.companyId, companyId)));
if (run.acknowledged && run.version === "v2") await db.update(agentRuntimeState).set({
totalCostCents: sql`${agentRuntimeState.totalCostCents} + ${values.costCents}::numeric`,
totalInputTokens: sql`${agentRuntimeState.totalInputTokens} + ${event.inputTokens}`,
totalCachedInputTokens: sql`${agentRuntimeState.totalCachedInputTokens} + ${event.cachedInputTokens}`,
totalOutputTokens: sql`${agentRuntimeState.totalOutputTokens} + ${event.outputTokens}`,
updatedAt: new Date(),
}).where(and(eq(agentRuntimeState.agentId, event.agentId), eq(agentRuntimeState.companyId, companyId)));
const [inserted] = await db.insert(costEvents).values({ ...values, subscriptionId, costCents: sql`${values.costCents}::numeric`, reportedCostCents: values.costCents, id: data.id, companyId, receiptHash })
.onConflictDoNothing().returning();
if (!inserted) {
const [winner] = values.idempotencyKey ? await db.select().from(costEvents).where(and(
eq(costEvents.companyId, companyId), eq(costEvents.idempotencyKey, values.idempotencyKey))) : [];
if (!winner) throw Object.assign(new Error("Concurrent cost receipt requires a fresh snapshot"), { code: "40001" });
if (winner.receiptHash !== receiptHash && winner.receiptHash !== legacyHash)
throw conflict("Idempotency key already used for a different cost receipt");
const [exact] = await db.select({ costCents: sql<string>`${costEvents.costCents}::text` }).from(costEvents).where(eq(costEvents.id, winner.id));
return { ...winner, costCentsExact: normalizeCents(exact.costCents) };
}
// A multi-model run receipt inserts all parts before one scope evaluation.
// The owning accounting transaction must perform that deferred evaluation.
if (!options.deferBudgetEvaluation) await budgetServiceInTransaction(db, publications).evaluateCostEvent(event);
const event = inserted;
if (!options.deferBudgetEvaluation) await queueBudgetEvaluation(db, { companyId, agentId: event.agentId, projectId: event.projectId });
await logActivity(db, {
companyId, actorType: actor?.actorType ?? "system", actorId: actor?.actorId ?? "cost_accounting", agentId: actor?.agentId ?? event.agentId,
...((event.usageKind === "decision" || event.usageKind === "fast_response" || event.usageKind === "prompt_injection") ? { responsibleUserIdOverride: event.responsibleUserId } : {}),
@@ -169,7 +143,7 @@ export function costService(db: Db, _budgetHooks: BudgetServiceHooks = {}) {
return {
createEvent: async (companyId: string, data: Omit<typeof costEvents.$inferInsert, "companyId" | "receiptHash" | "costCents"> & { costCents: MoneyInput }, actor?: Pick<LogActivityInput, "actorType" | "actorId" | "agentId">) => {
if (data.idempotencyKey?.startsWith("heartbeat:")) throw unprocessable("The heartbeat idempotency namespace is reserved for run receipts");
const event = await withAccountingTransaction(db, companyId, (tx, publications) => createCostEventInTransaction(tx, companyId, data, publications, actor));
const event = await withOperationAccountingTransaction(db, companyId, (tx, publications) => createCostEventInTransaction(tx, companyId, data, publications, actor));
return event;
},
+81 -20
View File
@@ -14,10 +14,12 @@ import { HttpError, forbidden, unprocessable } from "../errors.js";
import { aiConnectionService } from "./ai-connections.js";
import { accessService } from "./access.js";
import { type AuthorizationActor } from "./authorization.js";
import { withAccountingTransaction } from "./accounting-transaction.js";
import { withAccountingTransaction, withOperationAccountingTransaction } from "./accounting-transaction.js";
import { withBudgetSpan } from "./budget-tracing.js";
import { budgetServiceInTransaction, type BudgetServiceHooks } from "./budgets.js";
import { createCostEventInTransaction } from "./costs.js";
import { applyCostCorrectionInTransaction } from "./accounting-corrections.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
import { logActivity } from "./activity-log.js";
import { decisionReceipt, runDecisionProvider, type DecisionProviderOutcome } from "./decision-model-provider.js";
@@ -83,19 +85,25 @@ export function decisionModelService(db: Db, options: { provider?: typeof runDec
return values;
});
}
async function resolve(context: DecisionContext, executor = db) {
async function resolve(context: DecisionContext, executor = db, lockAdmission = false) {
if (!registeredFeatures.has(context.feature)) throw forbidden("A registered decision feature is required");
const [config] = await executor.select().from(companyDecisionModels).where(eq(companyDecisionModels.companyId, context.companyId));
const configQuery = executor.select().from(companyDecisionModels).where(eq(companyDecisionModels.companyId, context.companyId));
const [config] = await (lockAdmission ? configQuery.for("share") : configQuery);
if (!config?.connectionId || !config.grantId) throw new DecisionUnavailable("not_configured");
if (!config.enabled) throw new DecisionUnavailable("disabled");
let userId: string | null = null, agentId: string | null = null, runId: string | null = null, identityContextId: string | null = null;
let issueId = context.issueId ?? null;
let actor = context.actor;
let admissionAgent: { status: string } | undefined;
if (actor.type === "system") {
if (!context.feature.background || !config.allowBackground) throw new DecisionUnavailable("background_disabled");
} else if (actor.type === "agent") {
if (!actor.agentId || !actor.runId || actor.companyId !== context.companyId) throw forbidden();
const [run] = await executor.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, context.companyId), eq(heartbeatRuns.id, actor.runId), eq(heartbeatRuns.agentId, actor.agentId)));
if (lockAdmission) [admissionAgent] = await executor.select({ status: agents.status }).from(agents).where(and(
eq(agents.id, actor.agentId), eq(agents.companyId, context.companyId),
)).for("share");
const runQuery = executor.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, context.companyId), eq(heartbeatRuns.id, actor.runId), eq(heartbeatRuns.agentId, actor.agentId)));
const [run] = await (lockAdmission ? runQuery.for("share") : runQuery);
if (!run || run.status !== "running") throw forbidden("An active run is required");
agentId = run.agentId; runId = run.id; userId = run.responsibleUserId; identityContextId = run.activeIdentityContextId;
const runIssue = run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId;
@@ -126,7 +134,7 @@ export function decisionModelService(db: Db, options: { provider?: typeof runDec
const connection = await aiConnectionService(executor).selectDecision({ companyId: context.companyId,
connectionId: config.connectionId, grantId: config.grantId, userId, agentId, sponsoredBackground: actor.type === "system" });
if (connection.provider !== config.provider || config.model !== DECISION_MODELS[connection.provider].id) throw new DecisionUnavailable("incompatible_connection");
return { config, connection, userId, agentId, runId, identityContextId, issueId, projectId,
return { config, connection, userId, agentId, runId, identityContextId, issueId, projectId, admissionAgent,
actorType: actor.type === "board" ? "user" as const : actor.type === "agent" ? "agent" as const : "system" as const,
actorId: actor.type === "system" ? "paperclip-decisions" : agentId ?? userId! };
}
@@ -140,9 +148,7 @@ export function decisionModelService(db: Db, options: { provider?: typeof runDec
return (context: DecisionContext) => { let result = cache.get(context); if (!result) { result = availability(context); cache.set(context, result); } return result; };
}
async function settle(invocation: typeof decisionInvocations.$inferSelect, outcome: DecisionProviderOutcome) {
await withBudgetSpan("budget.accounting.settlement", () => withAccountingTransaction(db, invocation.companyId, async (tx, publications) => {
const [current] = await tx.select().from(decisionInvocations).where(and(eq(decisionInvocations.companyId, invocation.companyId), eq(decisionInvocations.id, invocation.id))).for("update");
if (!current || current.status !== "running") return;
await withBudgetSpan("budget.accounting.settlement", () => withOperationAccountingTransaction(db, invocation.companyId, async (tx, publications) => {
const receipt = outcome.receipt;
// Keep dispatch identity on the invocation even if its task or agent was
// deleted while the provider worked. Ledger foreign keys use live rows.
@@ -153,7 +159,47 @@ export function decisionModelService(db: Db, options: { provider?: typeof runDec
const [row] = await tx.select({ id: table.id }).from(table).where(and(eq(table.id, id), eq(table.companyId, invocation.companyId))).for("key share");
if (row) links[key] = row.id;
}
const event = await createCostEventInTransaction(tx, invocation.companyId, {
const [current] = await tx.select().from(decisionInvocations).where(and(eq(decisionInvocations.companyId, invocation.companyId), eq(decisionInvocations.id, invocation.id))).for("update");
if (!current) return;
const [existing] = current.costEventId ? await tx.select({ id: costEvents.id, costStatus: costEvents.costStatus,
inputTokens: costEvents.inputTokens, cachedInputTokens: costEvents.cachedInputTokens, outputTokens: costEvents.outputTokens,
providerRequestId: costEvents.providerRequestId, pricingProvenance: costEvents.pricingProvenance,
costCents: sql<string>`${costEvents.costCents}::text` }).from(costEvents).where(and(
eq(costEvents.companyId, invocation.companyId), eq(costEvents.id, current.costEventId),
)).for("update") : [];
if (current.status !== "running") {
if (!existing || outcome.errorCode === "interrupted") return;
if (current.errorCode !== "interrupted" && (existing.costStatus !== "unpriced" || receipt.costStatus === "unpriced")) return;
}
let event: { id: string };
if (existing) {
const receiptUsage = { inputTokens: receipt.inputTokens ?? current.inputTokens ?? 0, cachedInputTokens: 0,
outputTokens: receipt.outputTokens ?? current.outputTokens ?? 0, providerRequestId: receipt.providerRequestId ?? current.providerRequestId };
// Recovery records uncertainty, not final billing evidence. A late receipt
// values the same attempt through the existing audited correction ledger.
if (existing.costStatus === "unpriced" && receipt.costStatus !== "unpriced") await applyCostCorrectionInTransaction(tx, publications, { companyId: invocation.companyId, eventId: existing.id,
idempotencyKey: `decision:${invocation.id}:late-receipt`, expectedCents: existing.costCents,
correctedCents: receipt.costCents ?? "0", reason: "Provider receipt arrived after interrupted accounting recovery",
costStatus: receipt.costStatus,
receiptUsage,
pricing: { ...receipt.pricingProvenance, evidence: JSON.stringify({ providerRequestId: receipt.providerRequestId,
inputTokens: receipt.inputTokens, outputTokens: receipt.outputTokens, evidence: receipt.pricingProvenance.evidence }) },
actorId: "cost_accounting" });
else {
if (Object.entries(receiptUsage).some(([key, value]) => existing[key as keyof typeof existing] !== value))
await applyCostCorrectionInTransaction(tx, publications, { companyId: invocation.companyId, eventId: existing.id,
idempotencyKey: `decision:${invocation.id}:late-usage`, expectedCents: existing.costCents, correctedCents: existing.costCents,
costStatus: existing.costStatus === "unpriced" ? "unpriced" : existing.costStatus === "estimated" ? "estimated" : "reported", receiptUsage,
reason: "Provider usage arrived after interrupted accounting recovery",
pricing: existing.pricingProvenance ?? { source: "unknown" }, actorId: "cost_accounting" });
if (existing.costStatus !== "unpriced") await logActivity(tx, { companyId: invocation.companyId,
actorType: "system", actorId: "cost_accounting", responsibleUserIdOverride: invocation.responsibleUserId,
action: "accounting.late_receipt_observed", entityType: "cost_event", entityId: existing.id,
details: { effectiveCostCents: existing.costCents, ...receipt },
}, publications);
}
event = existing;
} else event = await createCostEventInTransaction(tx, invocation.companyId, {
usageKind: "decision", responsibleUserId: invocation.responsibleUserId, ...links,
// Run attribution lives on the invocation. Run receipt reconciliation must not ingest this independent charge.
heartbeatRunId: null, idempotencyKey: `decision:${invocation.id}`,
@@ -165,13 +211,13 @@ export function decisionModelService(db: Db, options: { provider?: typeof runDec
const finishedAt = new Date();
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, invocation.companyId);
await tx.update(decisionInvocations).set({ status: outcome.answers ? "succeeded" : receipt.costStatus === "unpriced" ? "unknown" : "failed",
errorCode: outcome.errorCode ?? null, costEventId: event.id, providerRequestId: receipt.providerRequestId,
inputTokens: receipt.inputTokens, outputTokens: receipt.outputTokens, finishedAt, durationMs: Math.max(0, finishedAt.getTime() - invocation.startedAt.getTime()),
}).where(eq(decisionInvocations.id, invocation.id));
errorCode: outcome.errorCode ?? null, costEventId: event.id, providerRequestId: receipt.providerRequestId ?? current.providerRequestId,
inputTokens: receipt.inputTokens ?? current.inputTokens, outputTokens: receipt.outputTokens ?? current.outputTokens, finishedAt, durationMs: Math.max(0, finishedAt.getTime() - invocation.startedAt.getTime()),
}).where(and(eq(decisionInvocations.companyId, invocation.companyId), eq(decisionInvocations.id, invocation.id)));
if (receipt.costStatus !== "unpriced") await tx.update(budgetReservations).set({ state: outcome.noProviderWork ? "released" : "settled", settledAt: finishedAt }).where(and(
eq(budgetReservations.companyId, invocation.companyId), eq(budgetReservations.decisionInvocationId, invocation.id), eq(budgetReservations.state, "held"),
));
}));
}), { "budget.accounting.owner": "decision" });
}
async function decide(context: DecisionContext, request: DecisionRequest, signal?: AbortSignal): Promise<DecisionResult> {
const parsed = decisionRequestSchema.safeParse(request);
@@ -187,26 +233,40 @@ export function decisionModelService(db: Db, options: { provider?: typeof runDec
try { credential = await aiConnectionService(db).credential(selected.connection, 0, { responsibleUserId: selected.userId,
actorType: selected.actorType, actorId: selected.actorId, issueId: selected.issueId, heartbeatRunId: selected.runId }); }
catch (error) { return { status: "unavailable", reason: unavailableReason(error) ?? "connection_unavailable" }; }
const admission = await withAccountingTransaction(db, context.companyId, async (tx, publications) => {
const admission = await withOperationAccountingTransaction(db, context.companyId, async (tx, publications) => {
// Admission readers share these control fences. A manual pause either
// precedes this admission or sees its committed invocation when it wins.
const [company] = await tx.select({ status: companies.status }).from(companies)
.where(eq(companies.id, context.companyId)).for("share");
if (!company || company.status !== "active") return { reason: "budget_blocked" as const };
if (selected.projectId) {
const [project] = await tx.select({ id: projects.id }).from(projects).where(and(
eq(projects.companyId, context.companyId), eq(projects.id, selected.projectId),
)).for("share");
if (!project) return { reason: "budget_blocked" as const };
}
let current: Awaited<ReturnType<typeof resolve>>;
try { current = await resolve(context, tx); }
try { current = await resolve(context, tx, true); }
catch (error) { const reason = unavailableReason(error); if (reason) return { reason }; throw error; }
// A different setting or steering identity requires a fresh credential acquisition.
if (current.config.updatedAt.getTime() !== selected.config.updatedAt.getTime() || current.userId !== selected.userId || current.identityContextId !== selected.identityContextId
if (current.config.updatedAt.getTime() !== selected.config.updatedAt.getTime()
|| current.config.connectionId !== selected.config.connectionId || current.config.grantId !== selected.config.grantId
|| current.config.model !== selected.config.model || current.config.provider !== selected.config.provider
|| current.projectId !== selected.projectId
|| current.connection.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")?.secretId !== credentialSnapshot.id
|| current.userId !== selected.userId || current.identityContextId !== selected.identityContextId
|| current.connection.grant.updatedAt.getTime() !== selected.connection.grant.updatedAt.getTime()
|| current.connection.connection.updatedAt.getTime() !== selected.connection.connection.updatedAt.getTime())
return { reason: "connection_unavailable" as const };
// Secret changes do not update the connection revision. Serialize against
// rotation/revocation through admission, including time spent waiting for
// the accounting lock after credential resolution.
// rotation/revocation through this invocation's admission.
const [currentSecret] = await tx.select({ status: companySecrets.status, version: companySecrets.latestVersion }).from(companySecrets)
.where(and(eq(companySecrets.companyId, context.companyId), eq(companySecrets.id, credentialSnapshot.id))).for("share");
if (!currentSecret || currentSecret.status !== "active" || currentSecret.version !== credentialSnapshot.version)
return { reason: "connection_unavailable" as const };
const budgets = budgetServiceInTransaction(tx, publications);
const { block, policies, observations } = await budgets.evaluateAdmission(context.companyId, current.agentId, { projectId: current.projectId });
const [company] = await tx.select({ status: companies.status }).from(companies).where(eq(companies.id, context.companyId));
const [agent] = current.agentId ? await tx.select({ status: agents.status }).from(agents).where(and(eq(agents.id, current.agentId), eq(agents.companyId, context.companyId))) : [];
const agent = current.admissionAgent;
if (block || company.status !== "active" || (current.agentId && (!agent || agent.status === "paused" || agent.status === "terminated"))) return { reason: "budget_blocked" as const };
const amount = policies.reduce((max, p) => { const n = centsToUnits(p.reservationCents); return n > max ? n : max; }, 0n);
for (const [index, policy] of policies.entries()) {
@@ -223,6 +283,7 @@ export function decisionModelService(db: Db, options: { provider?: typeof runDec
}).returning();
await tx.insert(budgetReservations).values({ companyId: context.companyId, decisionInvocationId: invocation.id, agentId: current.agentId,
projectId: current.projectId, amountCents: unitsToCents(amount), providerStartedAt: invocation.startedAt });
await queueBudgetEvaluation(tx, { companyId: context.companyId, agentId: current.agentId, projectId: current.projectId });
return { invocation };
});
if (admission.reason) return { status: "unavailable", reason: admission.reason };
@@ -21,6 +21,9 @@ export function createDeliveryWorkCoordinator(input: {
nextWakeAt: scheduler.nextWakeAt,
register(queue: DeliveryQueue, task: {
retryMs: number;
// Bound bursts by the first notification's deadline; later notifications
// do not postpone the obligation indefinitely.
coalesceMs?: number;
run: (signal: AbortSignal) => Promise<unknown>;
hasPending: () => Promise<boolean>;
// Scoped hints belong to the outer transaction, including uncertainty recovery.
@@ -98,13 +101,13 @@ export function createDeliveryWorkCoordinator(input: {
running = null;
controller = null;
finish();
if (dirty) schedule(0);
if (dirty) schedule(task.coalesceMs ?? 0);
}));
}
function wake() {
if (workerStopped) return;
dirty = true;
if (!running) schedule(0);
if (!running) schedule(task.coalesceMs ?? 0);
}
const unsubscribe = subscribeDatabaseWork(input.owner, queue, (event, hints) => {
if (workerStopped) return;
+3 -4
View File
@@ -1,3 +1,4 @@
import { budgetServiceInTransaction } from "./budgets.js";
import { notifyToolConnectionHealth, writeToolConnectionHealthChange } from "./tool-connection-health-notifications.js";
import type { ConnectionManagementContext } from "./connection-management.js";
import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
@@ -33,7 +34,6 @@ import {
companyMemberships,
instanceUserRoles,
} from "@paperclipai/db";
import { compareCents } from "@paperclipai/shared";
import type {
AgentPermissions,
EmailEndpointSetupInput,
@@ -539,7 +539,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
);
await authorizeRead(endpoint.companyId, issueId, actor);
const [agent] = await db
.select({ ...getTableColumns(agents), spentMonthlyCentsExact: sql<string>`${agents.spentMonthlyCents}::text` })
.select(getTableColumns(agents))
.from(agents)
.where(
and(
@@ -550,8 +550,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
if (
!agent ||
["paused", "terminated", "pending_approval"].includes(agent.status) ||
(agent.budgetMonthlyCents > 0 &&
compareCents(agent.spentMonthlyCentsExact, agent.budgetMonthlyCents) >= 0)
await budgetServiceInTransaction(db).getInvocationBlock(endpoint.companyId, actor.agentId, { issueId })
)
throw forbidden("Agent is not available to send email");
if (!actor.runId)
+76 -24
View File
@@ -60,13 +60,15 @@ import {
publishActivity,
type ActivityPublication,
} from "./activity-log.js";
import { withAccountingTransaction } from "./accounting-transaction.js";
import { withAccountingTransaction, withOperationAccountingTransaction } from "./accounting-transaction.js";
import { withBudgetSpan } from "./budget-tracing.js";
import {
budgetServiceInTransaction,
type BudgetServiceHooks,
} from "./budgets.js";
import { createCostEventInTransaction } from "./costs.js";
import { applyCostCorrectionInTransaction } from "./accounting-corrections.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
import {
notifyDeliveryWork,
DELIVERY_QUEUES,
@@ -434,13 +436,15 @@ export function fastResponseService(
| "endpointId"
> &
Partial<Pick<Request, "connectionId" | "grantId" | "model">>,
lockConfiguration = false,
) {
if (!(await instanceSettingsService(tx).getExperimental()).enableFastResponses)
throw unprocessable("disabled");
const [config] = await tx
const configQuery = tx
.select()
.from(companyFastResponses)
.where(eq(companyFastResponses.companyId, request.companyId));
const [config] = await (lockConfiguration ? configQuery.for("share") : configQuery);
if (
!config?.enabled ||
!config.connectionId ||
@@ -564,22 +568,17 @@ export function fastResponseService(
.where(
and(
eq(fastResponseRequests.id, request.id),
eq(fastResponseRequests.companyId, request.companyId),
eq(fastResponseRequests.status, "pending"),
),
);
});
}
async function settle(request: Request, outcome: FastResponseOutcome) {
await withBudgetSpan("budget.accounting.settlement", () => withAccountingTransaction(
await withBudgetSpan("budget.accounting.settlement", () => withOperationAccountingTransaction(
db,
request.companyId,
async (tx, publications) => {
const [current] = await tx
.select()
.from(fastResponseRequests)
.where(eq(fastResponseRequests.id, request.id))
.for("update");
if (!current || current.status !== "running") return;
const receipt = outcome.receipt;
const links: {
agentId: string | null;
@@ -602,7 +601,47 @@ export function fastResponseService(
.for("key share");
if (row) links[key] = row.id;
}
const event = await createCostEventInTransaction(
const [current] = await tx.select().from(fastResponseRequests).where(and(
eq(fastResponseRequests.companyId, request.companyId), eq(fastResponseRequests.id, request.id),
)).for("update");
if (!current) return;
const [existing] = current.costEventId ? await tx.select({ id: costEvents.id, costStatus: costEvents.costStatus,
inputTokens: costEvents.inputTokens, cachedInputTokens: costEvents.cachedInputTokens, outputTokens: costEvents.outputTokens,
providerRequestId: costEvents.providerRequestId, pricingProvenance: costEvents.pricingProvenance,
costCents: sql<string>`${costEvents.costCents}::text` }).from(costEvents).where(and(
eq(costEvents.companyId, request.companyId), eq(costEvents.id, current.costEventId),
)).for("update") : [];
if (current.status !== "running") {
if (!existing || outcome.errorCode === "interrupted") return;
if (current.errorCode !== "interrupted" && (existing.costStatus !== "unpriced" || receipt.costStatus === "unpriced")) return;
}
let event: { id: string };
if (existing) {
const receiptUsage = { inputTokens: receipt.inputTokens ?? current.inputTokens ?? 0, cachedInputTokens: 0,
outputTokens: receipt.outputTokens ?? current.outputTokens ?? 0, providerRequestId: receipt.providerRequestId ?? current.providerRequestId };
if (existing.costStatus === "unpriced" && receipt.costStatus !== "unpriced") await applyCostCorrectionInTransaction(tx, publications, { companyId: request.companyId, eventId: existing.id,
idempotencyKey: `fast-response:${request.id}:late-receipt`, expectedCents: existing.costCents,
correctedCents: receipt.costCents ?? "0", reason: "Provider receipt arrived after interrupted accounting recovery",
costStatus: receipt.costStatus,
receiptUsage,
pricing: { ...receipt.pricingProvenance, evidence: JSON.stringify({ providerRequestId: receipt.providerRequestId,
inputTokens: receipt.inputTokens, outputTokens: receipt.outputTokens, evidence: receipt.pricingProvenance.evidence }) },
actorId: "cost_accounting" });
else {
if (Object.entries(receiptUsage).some(([key, value]) => existing[key as keyof typeof existing] !== value))
await applyCostCorrectionInTransaction(tx, publications, { companyId: request.companyId, eventId: existing.id,
idempotencyKey: `fast-response:${request.id}:late-usage`, expectedCents: existing.costCents, correctedCents: existing.costCents,
costStatus: existing.costStatus === "unpriced" ? "unpriced" : existing.costStatus === "estimated" ? "estimated" : "reported", receiptUsage,
reason: "Provider usage arrived after interrupted accounting recovery",
pricing: existing.pricingProvenance ?? { source: "unknown" }, actorId: "cost_accounting" });
if (existing.costStatus !== "unpriced") await logActivity(tx, { companyId: request.companyId,
actorType: "system", actorId: "cost_accounting", responsibleUserIdOverride: request.responsibleUserId,
action: "accounting.late_receipt_observed", entityType: "cost_event", entityId: existing.id,
details: { effectiveCostCents: existing.costCents, ...receipt },
}, publications);
}
event = existing;
} else event = await createCostEventInTransaction(
tx,
request.companyId,
{
@@ -641,14 +680,14 @@ export function fastResponseService(
: "failed",
errorCode: outcome.errorCode ?? null,
costEventId: event.id,
providerRequestId: receipt.providerRequestId,
inputTokens: receipt.inputTokens,
outputTokens: receipt.outputTokens,
providerRequestId: receipt.providerRequestId ?? current.providerRequestId,
inputTokens: receipt.inputTokens ?? current.inputTokens,
outputTokens: receipt.outputTokens ?? current.outputTokens,
finishedAt,
durationMs: finishedAt.getTime() - request.acceptedAt.getTime(),
...(outcome.text ? {} : { publicationStatus: "suppressed" }),
})
.where(eq(fastResponseRequests.id, request.id));
.where(and(eq(fastResponseRequests.companyId, request.companyId), eq(fastResponseRequests.id, request.id)));
if (receipt.costStatus !== "unpriced")
await tx
.update(budgetReservations)
@@ -664,7 +703,7 @@ export function fastResponseService(
),
);
},
));
), { "budget.accounting.owner": "fast_response" });
}
async function process(
@@ -799,16 +838,30 @@ export function fastResponseService(
);
let admitted: Request | null = null;
try {
admitted = await withAccountingTransaction(
admitted = await withOperationAccountingTransaction(
db,
request.companyId,
async (tx) => {
if (request.expiresAt.getTime() <= Date.now() || signal?.aborted)
return null;
const current = await resolve(tx, request);
const [company] = await tx.select({ status: companies.status }).from(companies)
.where(eq(companies.id, request.companyId)).for("share");
if (!company || company.status !== "active") return null;
if (selected.issue?.projectId) {
const [project] = await tx.select({ id: projects.id }).from(projects).where(and(
eq(projects.companyId, request.companyId), eq(projects.id, selected.issue.projectId),
)).for("share");
if (!project) return null;
}
const current = await resolve(tx, request, true);
if (
current.config.updatedAt.getTime() !==
selected.config.updatedAt.getTime() ||
current.config.connectionId !== selected.config.connectionId ||
current.config.grantId !== selected.config.grantId ||
current.config.model !== selected.config.model ||
current.issue?.projectId !== selected.issue?.projectId ||
current.connection.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")?.secretId !== secretId ||
current.connection.connection.updatedAt.getTime() !==
selected.connection.connection.updatedAt.getTime() ||
current.connection.grant.updatedAt.getTime() !==
@@ -834,10 +887,6 @@ export function fastResponseService(
)
return null;
if (request.endpointId) await options.authorizeExternal!(tx, request);
const [company] = await tx
.select()
.from(companies)
.where(eq(companies.id, request.companyId));
const [agent] = request.agentId
? await tx
.select()
@@ -847,7 +896,7 @@ export function fastResponseService(
eq(agents.id, request.agentId),
eq(agents.companyId, request.companyId),
),
)
).for("share")
: [];
if (
company?.status !== "active" ||
@@ -887,6 +936,7 @@ export function fastResponseService(
.where(
and(
eq(fastResponseRequests.id, request.id),
eq(fastResponseRequests.companyId, request.companyId),
eq(fastResponseRequests.status, "pending"),
),
)
@@ -900,6 +950,7 @@ export function fastResponseService(
amountCents: unitsToCents(amount),
providerStartedAt: claimed.startedAt,
});
await queueBudgetEvaluation(tx, { companyId: request.companyId, agentId: request.agentId, projectId: current.issue?.projectId });
return claimed;
},
);
@@ -942,7 +993,7 @@ export function fastResponseService(
const [locked] = await tx
.select()
.from(fastResponseRequests)
.where(eq(fastResponseRequests.id, request.id))
.where(and(eq(fastResponseRequests.companyId, request.companyId), eq(fastResponseRequests.id, request.id)))
.for("update");
if (
!locked ||
@@ -1006,7 +1057,7 @@ export function fastResponseService(
commentId: comment.id,
publicationStatus: request.endpointId ? "pending" : "published",
})
.where(eq(fastResponseRequests.id, request.id));
.where(and(eq(fastResponseRequests.companyId, request.companyId), eq(fastResponseRequests.id, request.id)));
await logActivity(
tx as unknown as Db,
{
@@ -1033,6 +1084,7 @@ export function fastResponseService(
.where(
and(
eq(fastResponseRequests.id, request.id),
eq(fastResponseRequests.companyId, request.companyId),
isNull(fastResponseRequests.commentId),
),
);
@@ -1,4 +1,5 @@
import { notifyExternalConversationSettlement } from "../external-conversation-settlement-notifications.js";
import { readRuntimeAccountingTotals } from "../runtime-accounting-totals.js";
import { conversationEventCondition, conversationEventColumns } from "./conversation-events.js";
import { deriveTaskKey } from "./run-preparation.js";
import { retryIdempotentDatabaseOperation } from "../../database-retry.js";
@@ -1907,6 +1908,7 @@ export function createHeartbeatRunState(db: Db) {
.then((rows) => rows[0] ?? null);
return {
...ensured,
...await readRuntimeAccountingTotals(db, agent.companyId, agentId),
sessionDisplayId:
latestTaskSession?.sessionDisplayId ?? ensured.sessionId,
sessionParamsJson: latestTaskSession?.sessionParamsJson ?? null,
@@ -1,6 +1,7 @@
import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js";
import { createNativeUsageDelivery } from "../native-runtime/native-usage-delivery.js";
import { flushBudgetEvaluationWork } from "../budgets.js";
import type { PrpEvent } from "../../vendor/paperclip-runner/index.js";
import { randomUUID } from "node:crypto";
import { mkdtemp, rm } from "node:fs/promises";
@@ -239,6 +240,7 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
exact = saved.costUsdExact as string;
expect(Number(exact)).toBe(1.002);
expect(saved.budgetExposure).toMatchObject({ costCents: "100.2000000" });
await flushBudgetEvaluationWork(db, { companyId: run.companyId });
expect((await db.select().from(agents).where(eq(agents.id, input.agent.id)))[0].lifecycleHolds).toContain("budget");
return { exitCode: 0, signal: null, timedOut: false, usageComplete: true, costUsd: null, model: "gpt-6.1-sol" };
});
@@ -11,6 +11,7 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, budgetPolicies, companies, costEvents, createDb, heartbeatRuns, runUsageReceipts, type Db } from "@paperclipai/db";
import { type PrpEvent } from "../../vendor/paperclip-runner/index.js";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { flushBudgetEvaluationWork } from "../budgets.js";
import { accountRunCost } from "../run-cost-accounting.js";
import { usageReceiptSpoolPath, type UsageReceiptEnvelope } from "../usage-receipts.js";
import { createNativeUsageContinuationDelivery, createNativeUsageDelivery } from "./native-usage-delivery.js";
@@ -89,7 +90,7 @@ describe("producer-owned native usage", () => {
});
it.each(["tokens", "cost", "incomplete", "unpriceable", "basis"])(
"holds a differing finalized source tail before acknowledgement: %s", async change => {
"retains a differing finalized source tail as unaccepted durable reconciliation debt: %s", async change => {
const { input, source } = await finalizedSource();
const continuation = await createNativeUsageContinuationDelivery(db, source);
const late = event(input, 3, "usage.reported");
@@ -101,7 +102,13 @@ describe("producer-owned native usage", () => {
};
await expect(continuation.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation");
await expect(continuation.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation");
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(2);
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(3);
const [pending] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
expect(pending.usageJson).toMatchObject({ accountingLateReceiptPending: true });
// Loading the durable journal on restart cannot turn rejected evidence
// into an acknowledged producer delivery.
const restarted = await createNativeUsageContinuationDelivery(db, source);
await expect(restarted.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation");
expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId)))[0].costCents).toBe(25);
});
@@ -214,7 +221,9 @@ describe("producer-owned native usage", () => {
const retainedLate = producerState().outbox.find(report => report.eventType === "usage.reported");
expect(retainedLate).toBeDefined();
expect(retainedLate!.envelope.payload).toMatchObject({ runId: input.runId, sourceInstanceId: input.runnerInstanceId });
expect((await receipts()).filter(row => (row.receiptJson.producerEvent as Record<string, unknown>)?.eventType === "usage.reported")).toHaveLength(1);
expect((await receipts()).filter(row => (row.receiptJson.producerEvent as Record<string, unknown>)?.eventType === "usage.reported")).toHaveLength(2);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)))[0].usageJson)
.toMatchObject({ accountingLateReceiptPending: true });
expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId))).toHaveLength(1);
expect(existsSync(usageReceiptSpoolPath())).toBe(false);
} finally {
@@ -233,6 +242,7 @@ describe("producer-owned native usage", () => {
await db.insert(budgetPolicies).values({ companyId: input.companyId, scopeType: "agent", scopeId: input.agentId,
metric: "billed_cents", windowKind: "lifetime", amount: 10, notifyEnabled: false });
await first.capture(event(input, 1, "usage.reported"));
await flushBudgetEvaluationWork(db, { companyId: input.companyId });
const [agent] = await db.select().from(agents).where(eq(agents.id, input.agentId));
expect(agent.lifecycleHolds).toContain("budget");
const replacement = { ...input, runnerInstanceId: randomUUID() };
@@ -27,7 +27,14 @@ export async function createNativeUsageDelivery(db: Db, input: NativeUsageDelive
const [bound] = await db.update(heartbeatRuns).set({ usageJson: sql`
coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || '{"accountingReceiptOwner":"native_producer"}'::jsonb ||
case when ${heartbeatRuns.usageJson}->>'accountingReceiptSourceId' = ${sourceId}
then '{}'::jsonb else ${JSON.stringify({ accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false })}::jsonb end`
then '{}'::jsonb else
case when jsonb_typeof(${heartbeatRuns.usageJson}->'accountingReceiptSourceId') = 'string'
then jsonb_build_object('accountingReceiptSourceFences',
coalesce(${heartbeatRuns.usageJson}->'accountingReceiptSourceFences', '{}'::jsonb)
|| jsonb_build_object(${heartbeatRuns.usageJson}->>'accountingReceiptSourceId',
coalesce(${heartbeatRuns.usageJson}->'accountingReceiptSequence', '0'::jsonb)))
else '{}'::jsonb end
|| ${JSON.stringify({ accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false })}::jsonb end`
}).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId),
isNull(heartbeatRuns.costAccountedAt))).returning({ id: heartbeatRuns.id });
if (!bound) throw new Error("native_usage_run_not_available");
@@ -59,7 +66,13 @@ function loadNativeUsageReceipts(db: Db, input: Pick<NativeUsageDeliveryInput, "
return db.select({ receipt: runUsageReceipts.receiptJson, hash: runUsageReceipts.receiptHash,
sequence: runUsageReceipts.sequence }).from(runUsageReceipts).where(and(
eq(runUsageReceipts.companyId, input.companyId), eq(runUsageReceipts.runId, input.runId),
eq(runUsageReceipts.sourceId, sourceId))).orderBy(asc(runUsageReceipts.sequence));
eq(runUsageReceipts.sourceId, sourceId),
// A journaled late tail is durable evidence, not an accepted delivery. Keep
// replaying it through the explicit reconciliation response after restart.
sql`not exists (select 1 from ${heartbeatRuns} r where r.id = ${runUsageReceipts.runId}
and r.company_id = ${runUsageReceipts.companyId} and r.usage_json->>'accountingLateReceiptPending' = 'true'
and (r.usage_json->>'accountingLateReceiptId' = ${runUsageReceipts.id}::text
or coalesce(r.usage_json->'accountingLateReceiptDebts', '{}'::jsonb) ? ${runUsageReceipts.id}::text))`)).orderBy(asc(runUsageReceipts.sequence));
}
function createNativeUsageReceiver(db: Db, input: NativeUsageDeliveryInput, sourceId: string,
@@ -1,3 +1,4 @@
import { budgetServiceInTransaction } from "../budgets.js";
import { DOWNLOAD_ARTIFACT, DOWNLOAD_ARTIFACT_DEFINITION, downloadArtifactSchema, createArtifactInstaller, type ArtifactInstaller } from "./artifact-transfer.js";
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
import { nativeRunKeyScope, nativeSkillTestToolAllowed, assertNativeSkillTestTool } from "./native-skill-test-scope.js";
@@ -1500,8 +1501,8 @@ export class PaperclipRunnerToolAuthority {
throw new Error("paperclip_runner_tool_binding_not_authorized");
}
const [company] = await this.db.select().from(companies).where(eq(companies.id, this.binding.companyId));
if (company?.status !== "active" || (company.budgetMonthlyCents > 0 && company.spentMonthlyCents >= company.budgetMonthlyCents)
|| (row.actor.budgetMonthlyCents > 0 && row.actor.spentMonthlyCents >= row.actor.budgetMonthlyCents)) throw forbidden("Company or budget authority unavailable");
if (company?.status !== "active"
|| await budgetServiceInTransaction(this.db).getInvocationBlock(this.binding.companyId, this.binding.agentId, { issueId: this.binding.issueId })) throw forbidden("Company or budget authority unavailable");
if (this.binding.nativeReview) {
const admitted = readNativeReviewAssignmentContext(row.run.contextSnapshot);
if (!admitted || admitted.nativeReviewInteractionId !== this.binding.nativeReview.nativeReviewInteractionId
@@ -2,17 +2,19 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio
import { randomUUID } from "node:crypto";
import { and, eq, lt, sql } from "drizzle-orm";
import { agents, budgetReservations, companies, companyPromptInjectionPrevention,
promptInjectionInvocations, companySecrets, heartbeatRuns, runIdentityContexts, issues, projects, type Db } from "@paperclipai/db";
promptInjectionInvocations, companySecrets, heartbeatRuns, runIdentityContexts, issues, projects, costEvents, type Db } from "@paperclipai/db";
import { PROMPT_INJECTION_DEFAULT_MODEL, PROMPT_INJECTION_POLICY_VERSION, decisionProviderForConnection,
updatePromptInjectionPreventionSchema, centsToUnits, unitsToCents,
type PromptInjectionPreventionSettings, type UpdatePromptInjectionPrevention, type PromptInjectionModelOutcome,
type PromptInjectionFallbackReason, type PromptInjectionTestResult, type DecisionUsage } from "@paperclipai/shared";
import { HttpError, forbidden } from "../errors.js";
import { aiConnectionService } from "./ai-connections.js";
import { withAccountingTransaction } from "./accounting-transaction.js";
import { withAccountingTransaction, withOperationAccountingTransaction } from "./accounting-transaction.js";
import { withBudgetSpan } from "./budget-tracing.js";
import { budgetServiceInTransaction, type BudgetServiceHooks } from "./budgets.js";
import { createCostEventInTransaction } from "./costs.js";
import { applyCostCorrectionInTransaction } from "./accounting-corrections.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
import { logActivity } from "./activity-log.js";
import { runPromptInjectionProvider, promptInjectionInput, promptInjectionRequestFits, type PromptInjectionProviderOutcome } from "./prompt-injection-provider.js";
import { providerUsageReceipt } from "./provider-usage.js";
@@ -74,16 +76,21 @@ export function promptInjectionPreventionService(db: Db, options: {
});
return settings(companyId);
}
async function resolve(context: PromptInjectionContext, executor = db) {
const [config] = await executor.select().from(companyPromptInjectionPrevention).where(eq(companyPromptInjectionPrevention.companyId, context.companyId));
async function resolve(context: PromptInjectionContext, executor = db, lockAdmission = false) {
const configQuery = executor.select().from(companyPromptInjectionPrevention).where(eq(companyPromptInjectionPrevention.companyId, context.companyId));
const [config] = await (lockAdmission ? configQuery.for("share") : configQuery);
if (!config?.connectionId || !config.grantId) throw new PromptInjectionUnavailable("not_configured");
if (!config.enabled) throw new PromptInjectionUnavailable("disabled");
const userId = context.responsibleUserId, agentId = context.agentId ?? null, runId = context.runId ?? null;
const identityContextId = context.identityContextId ?? null, issueId = context.issueId ?? null;
const [admissionAgent] = lockAdmission && agentId ? await executor.select({ status: agents.status }).from(agents).where(and(
eq(agents.id, agentId), eq(agents.companyId, context.companyId),
)).for("share") : [];
if (context.source === "settings_test" && (context.actorType !== "user" || !userId)) throw forbidden();
if (context.actorType === "user" && !userId) throw forbidden();
if (runId && !context.approvedAction) {
const [run] = await executor.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, context.companyId)));
const runQuery = executor.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, context.companyId)));
const [run] = await (lockAdmission ? runQuery.for("share") : runQuery);
if (!run || run.status !== "running" || run.agentId !== agentId || run.activeIdentityContextId !== identityContextId) throw forbidden("Execution identity changed");
const [identity] = identityContextId ? await executor.select().from(runIdentityContexts).where(and(
eq(runIdentityContexts.id, identityContextId), eq(runIdentityContexts.companyId, context.companyId))) : [];
@@ -101,12 +108,10 @@ export function promptInjectionPreventionService(db: Db, options: {
const connection = await aiConnectionService(executor).selectPromptInjectionPrevention({ companyId: context.companyId,
connectionId: config.connectionId, grantId: config.grantId, userId, agentId,
sponsoredBackground: context.source === "gateway" && !userId });
return { config, connection, userId, agentId, runId, identityContextId, issueId, projectId, actorType: context.actorType, actorId: context.actorId };
return { config, connection, userId, agentId, runId, identityContextId, issueId, projectId, admissionAgent, actorType: context.actorType, actorId: context.actorId };
}
async function settle(invocation: typeof promptInjectionInvocations.$inferSelect, outcome: PromptInjectionProviderOutcome) {
await withBudgetSpan("budget.accounting.settlement", () => withAccountingTransaction(db, invocation.companyId, async (tx, publications) => {
const [current] = await tx.select().from(promptInjectionInvocations).where(and(eq(promptInjectionInvocations.companyId, invocation.companyId), eq(promptInjectionInvocations.id, invocation.id))).for("update");
if (!current || current.status !== "running") return;
await withBudgetSpan("budget.accounting.settlement", () => withOperationAccountingTransaction(db, invocation.companyId, async (tx, publications) => {
const receipt = outcome.receipt;
// Keep dispatch identity on the invocation even if its task or agent was
// deleted while the provider worked. Ledger foreign keys use live rows.
@@ -117,7 +122,45 @@ export function promptInjectionPreventionService(db: Db, options: {
const [row] = await tx.select({ id: table.id }).from(table).where(and(eq(table.id, id), eq(table.companyId, invocation.companyId))).for("key share");
if (row) links[key] = row.id;
}
const event = await createCostEventInTransaction(tx, invocation.companyId, {
const [current] = await tx.select().from(promptInjectionInvocations).where(and(eq(promptInjectionInvocations.companyId, invocation.companyId), eq(promptInjectionInvocations.id, invocation.id))).for("update");
if (!current) return;
const [existing] = current.costEventId ? await tx.select({ id: costEvents.id, costStatus: costEvents.costStatus,
inputTokens: costEvents.inputTokens, cachedInputTokens: costEvents.cachedInputTokens, outputTokens: costEvents.outputTokens,
providerRequestId: costEvents.providerRequestId, pricingProvenance: costEvents.pricingProvenance,
costCents: sql<string>`${costEvents.costCents}::text` }).from(costEvents).where(and(
eq(costEvents.companyId, invocation.companyId), eq(costEvents.id, current.costEventId),
)).for("update") : [];
if (current.status !== "running") {
if (!existing || outcome.errorCode === "interrupted") return;
if (current.errorCode !== "interrupted" && (existing.costStatus !== "unpriced" || receipt.costStatus === "unpriced")) return;
}
let event: { id: string };
if (existing) {
const receiptUsage = { inputTokens: receipt.inputTokens ?? current.inputTokens ?? 0, cachedInputTokens: 0,
outputTokens: receipt.outputTokens ?? current.outputTokens ?? 0, providerRequestId: receipt.providerRequestId ?? current.providerRequestId };
if (existing.costStatus === "unpriced" && receipt.costStatus !== "unpriced") await applyCostCorrectionInTransaction(tx, publications, { companyId: invocation.companyId, eventId: existing.id,
idempotencyKey: `prompt-injection:${invocation.id}:late-receipt`, expectedCents: existing.costCents,
correctedCents: receipt.costCents ?? "0", reason: "Provider receipt arrived after interrupted accounting recovery",
costStatus: receipt.costStatus,
receiptUsage,
pricing: { ...receipt.pricingProvenance, evidence: JSON.stringify({ providerRequestId: receipt.providerRequestId,
inputTokens: receipt.inputTokens, outputTokens: receipt.outputTokens, evidence: receipt.pricingProvenance.evidence }) },
actorId: "cost_accounting" });
else {
if (Object.entries(receiptUsage).some(([key, value]) => existing[key as keyof typeof existing] !== value))
await applyCostCorrectionInTransaction(tx, publications, { companyId: invocation.companyId, eventId: existing.id,
idempotencyKey: `prompt-injection:${invocation.id}:late-usage`, expectedCents: existing.costCents, correctedCents: existing.costCents,
costStatus: existing.costStatus === "unpriced" ? "unpriced" : existing.costStatus === "estimated" ? "estimated" : "reported", receiptUsage,
reason: "Provider usage arrived after interrupted accounting recovery",
pricing: existing.pricingProvenance ?? { source: "unknown" }, actorId: "cost_accounting" });
if (existing.costStatus !== "unpriced") await logActivity(tx, { companyId: invocation.companyId,
actorType: "system", actorId: "cost_accounting", responsibleUserIdOverride: invocation.responsibleUserId,
action: "accounting.late_receipt_observed", entityType: "cost_event", entityId: existing.id,
details: { effectiveCostCents: existing.costCents, ...receipt },
}, publications);
}
event = existing;
} else event = await createCostEventInTransaction(tx, invocation.companyId, {
usageKind: "prompt_injection", responsibleUserId: invocation.responsibleUserId, ...links,
// Run attribution lives on the invocation. Run receipt reconciliation must not ingest this independent charge.
heartbeatRunId: null, idempotencyKey: `prompt-injection:${invocation.id}`,
@@ -129,13 +172,13 @@ export function promptInjectionPreventionService(db: Db, options: {
const finishedAt = new Date();
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, invocation.companyId);
await tx.update(promptInjectionInvocations).set({ status: outcome.verdict ? "succeeded" : receipt.costStatus === "unpriced" ? "unknown" : "failed",
violation: outcome.verdict?.violation ?? null, findings: outcome.verdict?.findings ?? null, errorCode: outcome.errorCode ?? null, costEventId: event.id, providerRequestId: receipt.providerRequestId,
inputTokens: receipt.inputTokens, outputTokens: receipt.outputTokens, finishedAt, durationMs: Math.max(0, finishedAt.getTime() - invocation.startedAt.getTime()),
}).where(eq(promptInjectionInvocations.id, invocation.id));
violation: outcome.verdict?.violation ?? null, findings: outcome.verdict?.findings ?? null, errorCode: outcome.errorCode ?? null, costEventId: event.id, providerRequestId: receipt.providerRequestId ?? current.providerRequestId,
inputTokens: receipt.inputTokens ?? current.inputTokens, outputTokens: receipt.outputTokens ?? current.outputTokens, finishedAt, durationMs: Math.max(0, finishedAt.getTime() - invocation.startedAt.getTime()),
}).where(and(eq(promptInjectionInvocations.companyId, invocation.companyId), eq(promptInjectionInvocations.id, invocation.id)));
if (receipt.costStatus !== "unpriced") await tx.update(budgetReservations).set({ state: outcome.noProviderWork ? "released" : "settled", settledAt: finishedAt }).where(and(
eq(budgetReservations.companyId, invocation.companyId), eq(budgetReservations.promptInjectionInvocationId, invocation.id), eq(budgetReservations.state, "held"),
));
}));
}), { "budget.accounting.owner": "classifier" });
}
async function classify(context: PromptInjectionContext, content: string, signal?: AbortSignal): Promise<PromptInjectionModelOutcome & { usage?: DecisionUsage }> {
signal?.throwIfAborted();
@@ -152,30 +195,38 @@ export function promptInjectionPreventionService(db: Db, options: {
try { credential = await aiConnectionService(db).credential(selected.connection, 0, { responsibleUserId: selected.userId,
actorType: selected.actorType, actorId: selected.actorId, issueId: selected.issueId, heartbeatRunId: selected.runId }); }
catch (error) { const reason = unavailableReason(error); if (reason) return { status: "unavailable", reason }; throw error; }
const admission = await withAccountingTransaction(db, context.companyId, async (tx, publications) => {
const admission = await withOperationAccountingTransaction(db, context.companyId, async (tx, publications) => {
const [company] = await tx.select({ status: companies.status }).from(companies)
.where(eq(companies.id, context.companyId)).for("share");
if (!company || company.status !== "active") return { reason: "budget_blocked" as const };
if (selected.projectId) {
const [project] = await tx.select({ id: projects.id }).from(projects).where(and(
eq(projects.companyId, context.companyId), eq(projects.id, selected.projectId),
)).for("share");
if (!project) return { reason: "budget_blocked" as const };
}
let current: Awaited<ReturnType<typeof resolve>>;
try { current = await resolve(context, tx); }
try { current = await resolve(context, tx, true); }
catch (error) { const reason = unavailableReason(error); if (reason) return { reason }; throw error; }
// A different setting or steering identity requires a fresh credential acquisition.
if (current.config.updatedAt.getTime() !== selected.config.updatedAt.getTime()
|| current.config.connectionId !== selected.config.connectionId || current.config.grantId !== selected.config.grantId
|| current.config.model !== selected.config.model
|| current.projectId !== selected.projectId
|| current.connection.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")?.secretId !== credentialSnapshot.id
|| current.userId !== selected.userId || current.identityContextId !== selected.identityContextId
|| current.connection.grant.updatedAt.getTime() !== selected.connection.grant.updatedAt.getTime()
|| current.connection.connection.updatedAt.getTime() !== selected.connection.connection.updatedAt.getTime())
return { reason: "connection_unavailable" as const };
// Secret changes do not update the connection revision. Serialize against
// rotation/revocation through admission, including time spent waiting for
// the accounting lock after credential resolution.
// rotation/revocation through this invocation's admission.
const [currentSecret] = await tx.select({ status: companySecrets.status, version: companySecrets.latestVersion }).from(companySecrets)
.where(and(eq(companySecrets.companyId, context.companyId), eq(companySecrets.id, credentialSnapshot.id))).for("share");
if (!currentSecret || currentSecret.status !== "active" || currentSecret.version !== credentialSnapshot.version)
return { reason: "connection_unavailable" as const };
const budgets = budgetServiceInTransaction(tx, publications);
const { block, policies, observations } = await budgets.evaluateAdmission(context.companyId, current.agentId, { projectId: current.projectId });
const [company] = await tx.select({ status: companies.status }).from(companies).where(eq(companies.id, context.companyId));
const [agent] = current.agentId ? await tx.select({ status: agents.status }).from(agents).where(and(eq(agents.id, current.agentId), eq(agents.companyId, context.companyId))) : [];
const agent = current.admissionAgent;
if (block || company.status !== "active" || (current.agentId && (!agent || agent.status === "paused" || agent.status === "terminated"))) return { reason: "budget_blocked" as const };
const amount = policies.reduce((max, p) => { const n = centsToUnits(p.reservationCents); return n > max ? n : max; }, 0n);
for (const [index, policy] of policies.entries()) {
@@ -191,6 +242,7 @@ export function promptInjectionPreventionService(db: Db, options: {
}).returning();
await tx.insert(budgetReservations).values({ companyId: context.companyId, promptInjectionInvocationId: invocation.id, agentId: current.agentId,
projectId: current.projectId, amountCents: unitsToCents(amount), providerStartedAt: invocation.startedAt });
await queueBudgetEvaluation(tx, { companyId: context.companyId, agentId: current.agentId, projectId: current.projectId });
return { invocation };
});
if (admission.reason) return { status: "unavailable", reason: admission.reason };
+304 -1
View File
@@ -4,9 +4,13 @@ import os from "node:os";
import path from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, costEvents, createDb, heartbeatRuns, promptInjectionInvocations, type Db } from "@paperclipai/db";
import { agents, budgetEvaluationWork, budgetPolicies, companies, costEvents, costAdjustments, createDb, heartbeatRuns, projects, promptInjectionInvocations, runUsageReceipts, type Db } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { accountRunCost, createCostAccountingReconciler, readCostAccountingWork } from "./run-cost-accounting.js";
import { createRunUsageRecorder, persistUsageReceipt, type UsageReceiptEnvelope } from "./usage-receipts.js";
import * as usageReceipts from "./usage-receipts.js";
import { billingReconciliationService } from "./billing-reconciliation.js";
import { accountingIntegrityService } from "./accounting-integrity.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("run cost accounting work", () => {
@@ -34,6 +38,14 @@ const support = await getEmbeddedPostgresTestSupport();
return { companyId, agentId, runId };
}
function receipt(owner: { companyId: string; runId: string }, sourceId: string, sequence: number, costUsd: number, inputTokens = 10): UsageReceiptEnvelope {
return { schema: "paperclip/accounting-receipt/v1", ...owner, id: randomUUID(), sourceId, sequence,
adapterType: "process", receivedAt: new Date().toISOString(),
receipt: { complete: true, usageBasis: "per_run", usage: { inputTokens, cachedInputTokens: 0, outputTokens: 2 },
provider: "test", biller: "test", model: "test", billingType: "metered_api", costUsd, costStatus: "reported",
pricingProvenance: { source: "provider_reported", version: "test" } } };
}
it("returns from settled or running calls without opening an accounting transaction", async () => {
const { runId } = await seed({ costAccountingPending: false });
const transaction = vi.fn(db.transaction.bind(db));
@@ -60,6 +72,297 @@ const support = await getEmbeddedPostgresTestSupport();
expect(await readCostAccountingWork(db, Date.now(), directory)).toEqual({ pending: false, nextRunAt: null });
});
it("settles while company and agent accounting rows are held by unrelated writers", async () => {
const owner = await seed();
const [policy] = await db.insert(budgetPolicies).values({ companyId: owner.companyId, scopeType: "company",
scopeId: owner.companyId, windowKind: "calendar_month_utc", amount: 100 }).returning();
let release!: () => void, locked!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const acquired = new Promise<void>(resolve => { locked = resolve; });
const holder = db.transaction(async tx => {
await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, owner.companyId)).for("no key update");
await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, owner.agentId)).for("no key update");
await tx.select({ id: budgetPolicies.id }).from(budgetPolicies).where(eq(budgetPolicies.id, policy.id)).for("update");
locked();
await gate;
});
await acquired;
const settle = accountRunCost(db, owner.runId);
let timer: ReturnType<typeof setTimeout> | undefined;
try {
expect(await Promise.race([settle, new Promise(resolve => { timer = setTimeout(() => resolve("blocked"), 1500); })])).toBe(true);
} finally { if (timer) clearTimeout(timer); release(); await holder; await settle; }
});
it("settles another run of the same company and agent while one run is locked", async () => {
const owner = await seed();
const [other] = await db.insert(heartbeatRuns).values({ companyId: owner.companyId, agentId: owner.agentId,
status: "succeeded", costAccountingPending: true, finishedAt: new Date(),
usageJson: { accountingReceiptReady: true, provider: "test", model: "test", billingType: "metered_api", costUsd: 0.02 } }).returning();
let release!: () => void, locked!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const acquired = new Promise<void>(resolve => { locked = resolve; });
const holder = db.transaction(async tx => {
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId)).for("update");
locked(); await gate;
});
await acquired;
const settle = accountRunCost(db, other.id);
let timer: ReturnType<typeof setTimeout> | undefined;
try {
expect(await Promise.race([settle, new Promise(resolve => { timer = setTimeout(() => resolve("blocked"), 1500); })])).toBe(true);
} finally { if (timer) clearTimeout(timer); release(); await holder; await settle; }
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId)))[0].costAccountedAt).toBeNull();
expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, other.id)))[0].costCents).toBe(2);
});
it("rejects a changed final receipt identity after the spool pre-read and retries from the new closure", async () => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 1,
accountingReceiptReady: true, costUsd: 0.01, provider: "test", model: "test", billingType: "metered_api" } });
let changed = false;
const observed = new Proxy(db, { get(target, property) {
if (property === "transaction") return async (...args: Parameters<Db["transaction"]>) => {
if (!changed) { changed = true; await persistUsageReceipt(db, receipt(owner, sourceId, 2, 0.02)); }
return target.transaction(...args);
};
const value = Reflect.get(target, property);
return typeof value === "function" ? value.bind(target) : value;
} }) as Db;
await expect(accountRunCost(observed, owner.runId)).rejects.toThrow("closure changed");
expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId))).toEqual([]);
expect(await accountRunCost(db, owner.runId)).toBe(true);
expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId)))[0].costCents).toBe(2);
});
it("retains spool evidence imported only by a transaction that rolled back before a retry skipped settlement", async () => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
const envelope = receipt(owner, sourceId, 1, 0.02);
const file = await usageReceipts.spoolUsageReceipt(envelope, directory);
const read = usageReceipts.readPendingRunUsageReceipts;
const spool = vi.spyOn(usageReceipts, "readPendingRunUsageReceipts").mockImplementation(input => read(input, directory));
let attempt = 0;
const observed = new Proxy(db, { get(target, property) {
if (property === "transaction") return async (callback: Parameters<Db["transaction"]>[0], options?: Parameters<Db["transaction"]>[1]) => {
attempt++;
if (attempt === 2) await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, owner.runId));
return target.transaction(async tx => {
const result = await callback(tx);
if (attempt === 1) throw Object.assign(new Error("forced serialization rollback"), { code: "40001" });
return result;
}, options);
};
const value = Reflect.get(target, property);
return typeof value === "function" ? value.bind(target) : value;
} }) as Db;
try {
expect(await accountRunCost(observed, owner.runId)).toBe(false);
expect(await fs.readFile(file, "utf8")).toContain(envelope.id);
expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId))).toEqual([]);
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, owner.runId))).toEqual([]);
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, owner.runId));
expect(await accountRunCost(db, owner.runId)).toBe(true);
await expect(fs.stat(file)).rejects.toMatchObject({ code: "ENOENT" });
} finally { spool.mockRestore(); }
});
it("races receipt acceptance and finalization without dropping or duplicating authoritative spend", async () => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
const results = await Promise.allSettled([accountRunCost(db, owner.runId), persistUsageReceipt(db, receipt(owner, sourceId, 2, 0.02))]);
for (const result of results) if (result.status === "rejected") expect(result.reason.message).toContain("closure changed");
await accountRunCost(db, owner.runId);
const events = await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId));
expect(events).toHaveLength(1);
expect(events[0].costCents).toBe(2);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId)))[0].costAccountingPending).toBe(false);
});
it("commits native late evidence before reporting reconciliation and keeps ambiguous debt quiet", async () => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
await accountRunCost(db, owner.runId);
const late = { ...receipt(owner, sourceId, 2, 0.02, 20), producerEvent: { sourceEventId: "late", eventType: "usage.reported" } };
await expect(persistUsageReceipt(db, late)).rejects.toThrow("requires_reconciliation");
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.id, late.id))).toHaveLength(1);
expect((await readCostAccountingWork(db, Date.now(), directory)).pending).toBe(true);
expect(await accountRunCost(db, owner.runId)).toBe(false);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId));
expect(run.costAccountingPending).toBe(true);
expect(run.usageJson).toMatchObject({ accountingLateReceiptPending: true, accountingLateReceiptAttemptedId: late.id });
expect((await readCostAccountingWork(db, Date.now(), directory)).pending).toBe(false);
expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId)))[0].costCents).toBe(1);
});
it("corrects authoritative same-token late money on the original charge", async () => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
await accountRunCost(db, owner.runId);
const [before] = await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId));
const late = receipt(owner, sourceId, 2, 0.025);
await persistUsageReceipt(db, late);
expect(await accountRunCost(db, owner.runId)).toBe(true);
expect(await accountRunCost(db, owner.runId)).toBe(false);
const events = await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId));
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({ id: before.id, receiptHash: before.receiptHash, costCents: 2.5, reportedCostCents: "1.0000000" });
expect(await db.select().from(costAdjustments).where(eq(costAdjustments.costEventId, before.id))).toHaveLength(1);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId)))[0].usageJson)
.toMatchObject({ accountingLateReceiptPending: false, accountingReconciledReceiptId: late.id });
});
it.each(["direct", "spool"])("does not revalue a corrected sequence3 from a late sequence2 via %s", async delivery => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
await accountRunCost(db, owner.runId);
const latest = receipt(owner, sourceId, 3, 0.03);
await persistUsageReceipt(db, latest);
expect(await accountRunCost(db, owner.runId)).toBe(true);
const older = receipt(owner, sourceId, 2, 0.02);
if (delivery === "spool") {
// Also exercise upgrade compatibility: the reconciled immutable receipt
// supplies the fence if explicit sequence metadata has not been written.
await db.update(heartbeatRuns).set({ usageJson: sql`${heartbeatRuns.usageJson}
- 'accountingReconciledReceiptSourceId' - 'accountingReconciledReceiptSequence'` }).where(eq(heartbeatRuns.id, owner.runId));
await usageReceipts.spoolUsageReceipt(older, directory);
expect(await usageReceipts.replayUsageReceipts(db, directory)).toEqual({ replayed: 1, failed: 0 });
} else await persistUsageReceipt(db, older);
expect(await accountRunCost(db, owner.runId)).toBe(false);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId));
expect(run.costAccountingPending).toBe(false);
expect(run.usageJson).toMatchObject({ accountingLateReceiptPending: false, accountingReconciledReceiptId: latest.id });
const [event] = await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId));
expect(event.costCents).toBe(3);
expect(await db.select().from(costAdjustments).where(eq(costAdjustments.costEventId, event.id))).toHaveLength(1);
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, owner.runId))).toHaveLength(3);
});
it("resolves an already queued older money-only debt without rewinding a newer correction", async () => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
await accountRunCost(db, owner.runId);
await persistUsageReceipt(db, receipt(owner, sourceId, 2, 0.02));
const newest = receipt(owner, sourceId, 3, 0.03);
await persistUsageReceipt(db, newest);
expect(await accountRunCost(db, owner.runId)).toBe(true);
expect(await accountRunCost(db, owner.runId)).toBe(true);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId));
expect(run.costAccountingPending).toBe(false);
expect(run.usageJson).toMatchObject({ accountingReconciledReceiptId: newest.id,
accountingReconciledReceiptSequence: 3, accountingLateReceiptDebts: {} });
const [event] = await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId));
expect(event.costCents).toBe(3);
expect(await db.select().from(costAdjustments).where(eq(costAdjustments.costEventId, event.id))).toHaveLength(1);
});
it("does not hide older ambiguous debt when a different late receipt is corrected", async () => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
await accountRunCost(db, owner.runId);
const ambiguous = receipt(owner, randomUUID(), 1, 0.09, 90);
await persistUsageReceipt(db, ambiguous);
const canonical = receipt(owner, sourceId, 2, 0.02);
await persistUsageReceipt(db, canonical);
expect(await accountRunCost(db, owner.runId)).toBe(true);
expect(await accountRunCost(db, owner.runId)).toBe(false);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId));
expect(run.costAccountingPending).toBe(true);
expect(run.usageJson).toMatchObject({ accountingLateReceiptPending: true,
accountingLateReceiptDebts: { [ambiguous.id]: true }, accountingLateReceiptAttemptedId: ambiguous.id });
expect((run.usageJson!.accountingLateReceiptDebts as Record<string, boolean>)[canonical.id]).toBeUndefined();
expect((await readCostAccountingWork(db, Date.now(), directory)).pending).toBe(false);
expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId)))[0].costCents).toBe(2);
});
it.each(["billingType", "biller"])("keeps a late %s classification change as debt instead of a money-only correction", async field => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
await accountRunCost(db, owner.runId);
const late = receipt(owner, sourceId, 2, 0.02);
late.receipt = { ...late.receipt, ...(field === "billingType" ? { billingType: "subscription_included" } : { biller: "different-biller" }) };
await persistUsageReceipt(db, late);
expect(await accountRunCost(db, owner.runId)).toBe(false);
const [event] = await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId));
expect(event).toMatchObject({ costCents: 1, billingType: "metered_api", biller: "test" });
expect(await db.select().from(costAdjustments).where(eq(costAdjustments.costEventId, event.id))).toEqual([]);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId));
expect(run.costAccountingPending).toBe(true);
expect(run.usageJson).toMatchObject({ accountingLateReceiptPending: true, accountingLateReceiptDebts: { [late.id]: true } });
expect((await readCostAccountingWork(db, Date.now(), directory)).pending).toBe(false);
});
it("retains reviewed operator valuation and resolves matching late debt through an explicit retry", async () => {
const sourceId = randomUUID();
const owner = await seed({ usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
await accountRunCost(db, owner.runId);
const [event] = await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, owner.runId));
await billingReconciliationService(db).adjust(owner.companyId, event.id, {
idempotencyKey: `operator:${owner.runId}`, expectedCents: "1", correctedCents: "4",
reason: "Reviewed authoritative invoice", pricing: { source: "operator", evidence: "Invoice review" },
}, "board");
const late = receipt(owner, sourceId, 2, 0.02);
await persistUsageReceipt(db, late);
expect(await accountRunCost(db, owner.runId)).toBe(false);
const [valued] = await db.select().from(costEvents).where(eq(costEvents.id, event.id));
expect(valued).toMatchObject({ costCents: 4, pricingProvenance: { source: "operator" } });
expect(await db.select().from(costAdjustments).where(eq(costAdjustments.costEventId, event.id))).toHaveLength(1);
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.id, late.id))).toHaveLength(1);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId)))[0].usageJson)
.toMatchObject({ accountingLateReceiptPending: true, accountingLateReceiptDebts: { [late.id]: true } });
expect((await readCostAccountingWork(db, Date.now(), directory)).pending).toBe(false);
await billingReconciliationService(db).adjust(owner.companyId, event.id, {
idempotencyKey: `operator-reviewed-late:${owner.runId}`, expectedCents: "4", correctedCents: "2",
reason: "Reviewed the retained late provider receipt", pricing: { source: "operator", evidence: "Reviewed late receipt" },
}, "board");
expect(await accountingIntegrityService(db).retry(owner.companyId, owner.runId, "board")).toEqual({ accounted: true });
const [resolved] = await db.select().from(costEvents).where(eq(costEvents.id, event.id));
expect(resolved).toMatchObject({ costCents: 2, pricingProvenance: { source: "operator", evidence: "Reviewed late receipt" } });
const adjustments = await db.select().from(costAdjustments).where(eq(costAdjustments.costEventId, event.id));
expect(adjustments).toHaveLength(2);
expect(adjustments.every(adjustment => adjustment.actorId === "board")).toBe(true);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId));
expect(run.costAccountingPending).toBe(false);
expect(run.usageJson).toMatchObject({ accountingLateReceiptPending: false, accountingLateReceiptDebts: {},
accountingReconciledReceiptId: late.id, accountingReconciledReceiptSequence: 2 });
});
it("fences a source handoff against a checkpoint committed during its spool pre-read", async () => {
const sourceId = randomUUID();
const owner = await seed({ status: "running", usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0 } });
const [project] = await db.insert(projects).values({ companyId: owner.companyId, name: "Archived accounting scope" }).returning();
await db.update(heartbeatRuns).set({ usageJson: sql`${heartbeatRuns.usageJson} || ${JSON.stringify({ ledgerScope: { projectId: project.id } })}::jsonb` })
.where(eq(heartbeatRuns.id, owner.runId));
await persistUsageReceipt(db, receipt(owner, sourceId, 1, 0.01));
let changed = false;
const observed = new Proxy(db, { get(target, property) {
if (property === "transaction") return async (...args: Parameters<Db["transaction"]>) => {
if (!changed) { changed = true; await persistUsageReceipt(db, receipt(owner, sourceId, 2, 0.02)); }
return target.transaction(...args);
};
const value = Reflect.get(target, property);
return typeof value === "function" ? value.bind(target) : value;
} }) as Db;
await expect(createRunUsageRecorder(observed, { ...owner, adapterType: "process" }, directory)).rejects.toThrow("source changed");
await createRunUsageRecorder(db, { ...owner, adapterType: "process" }, directory);
await db.delete(budgetEvaluationWork).where(eq(budgetEvaluationWork.companyId, owner.companyId));
await persistUsageReceipt(db, receipt(owner, sourceId, 3, 0.03));
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner.runId));
expect(run.usageJson).toMatchObject({ accountingCaptureFailed: true, accountingReceiptReady: false,
accountingReceiptSourceFences: { [sourceId]: 2 } });
expect(run.accountingLastError).toBe("usage_after_source_handoff_requires_reconciliation");
expect(await db.select({ projectId: budgetEvaluationWork.projectId }).from(budgetEvaluationWork)
.where(eq(budgetEvaluationWork.companyId, owner.companyId))).toEqual([{ projectId: project.id }]);
});
it("repairs a stale pending marker without creating another charge", async () => {
const { runId } = await seed({ costAccountedAt: new Date(), usageJson: { accountingReceiptReady: false } });
expect((await readCostAccountingWork(db, Date.now(), directory)).pending).toBe(true);
+55 -35
View File
@@ -1,23 +1,45 @@
import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js";
import { usdToUnits, unitsToCents } from "@paperclipai/shared";
import { and, asc, eq, inArray, or, sql } from "drizzle-orm";
import { accountingRuntimeBaselines, budgetReservations, agentRuntimeState, agents, costEvents, heartbeatRuns, nativeRunFinalizations, issues, projects, decisionInvocations, promptInjectionInvocations, type Db } from "@paperclipai/db";
import { withAccountingTransaction } from "./accounting-transaction.js";
import { budgetReservations, agents, heartbeatRuns, nativeRunFinalizations, issues, projects, decisionInvocations, promptInjectionInvocations, type Db } from "@paperclipai/db";
import { withOperationAccountingTransaction } from "./accounting-transaction.js";
import { withBudgetSpan } from "./budget-tracing.js";
import { conflict } from "../errors.js";
import { createCostEventInTransaction } from "./costs.js";
import { budgetServiceInTransaction, type BudgetServiceHooks } from "./budgets.js";
import type { BudgetServiceHooks } from "./budgets.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
import { ensureRuntimeAccountingBaseline } from "./runtime-accounting-totals.js";
import { logger } from "../middleware/logger.js";
import { promises as fs } from "node:fs";
import { replayUsageReceipts, indexPendingUsageReceipts, recoverPendingRunUsageReceipts, type UsageReceiptIndex } from "./usage-receipts.js";
import { replayUsageReceipts, indexPendingUsageReceipts, readPendingRunUsageReceipts, persistUsageReceiptInTransaction, reconcileLateRunUsageReceiptInTransaction, type UsageReceiptIndex } from "./usage-receipts.js";
const terminalStatuses = ["succeeded", "failed", "timed_out", "cancelled", "interrupted"];
const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const text = (value: unknown) => typeof value === "string" && value.length > 0 ? value : null;
const amount = (value: unknown) => typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : null;
/** The run's persisted usage is a durable receipt. Ledger, runtime totals, and
* acknowledgement commit together, including after a restart or failed run. */
/** The source/closed-provider identity is captured before filesystem reads and
* checked again after run arbitration. A newly closed/replaced source retries
* rather than acknowledging evidence from an earlier spool snapshot. */
async function readAccountingClosure(db: Db, runId: string) {
const [closure] = await db.select({
sourceId: sql<unknown>`${heartbeatRuns.usageJson}->'accountingReceiptSourceId'`,
sequence: sql<unknown>`${heartbeatRuns.usageJson}->'accountingReceiptSequence'`,
receiptId: sql<unknown>`${heartbeatRuns.usageJson}->'accountingReceiptId'`,
ready: sql<unknown>`${heartbeatRuns.usageJson}->'accountingReceiptReady'`,
failed: sql<unknown>`${heartbeatRuns.usageJson}->'accountingCaptureFailed'`,
runtimeMode: heartbeatRuns.runtimeMode,
preProviderFailure: sql<boolean>`(${heartbeatRuns.resultJson}->'executionRecovery'->>'providerWorkStarted' = 'false'
or ${heartbeatRuns.usageJson}->>'accountingProviderWorkStarted' = 'false') is true`,
coordinatorId: nativeRunFinalizations.runId, resultId: nativeRunFinalizations.resultId, phase: nativeRunFinalizations.phase,
}).from(heartbeatRuns).leftJoin(nativeRunFinalizations, and(
eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
)).where(eq(heartbeatRuns.id, runId));
return closure;
}
/** The run's persisted usage is a durable receipt. Ledger, reservation handoff
* and acknowledgement commit together, including after a restart or failed run. */
export async function accountRunCost(db: Db, runId: string, hooks: BudgetServiceHooks = {}, receiptIndex?: UsageReceiptIndex) {
const [identity] = await db.select({ companyId: heartbeatRuns.companyId, status: heartbeatRuns.status,
pending: heartbeatRuns.costAccountingPending, accountedAt: heartbeatRuns.costAccountedAt,
@@ -28,8 +50,17 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
// Pending acknowledgements still receive the locked stale-marker cleanup.
const index = identity.pending && !identity.accountedAt && terminalStatuses.includes(identity.status)
? receiptIndex ?? await indexPendingUsageReceipts() : undefined;
const closure = await readAccountingClosure(db, runId);
if (!closure) return false;
if (!identity.accountedAt && closure.runtimeMode === "native" && !closure.preProviderFailure
&& closure.coordinatorId && !closure.resultId && closure.phase !== "terminal_failure") return false;
const pendingReceipts = !identity.accountedAt
? await readPendingRunUsageReceipts({ companyId: identity.companyId, runId }, undefined, index) : [];
let recoveredFiles: string[] = [];
const accounted = await withBudgetSpan("budget.accounting.settlement", () => withAccountingTransaction(db, identity.companyId, async (tx, publications) => {
const accounted = await withBudgetSpan("budget.accounting.settlement", () => withOperationAccountingTransaction(db, identity.companyId, async (tx, publications) => {
// The operation wrapper can retry a transaction PostgreSQL rolled back.
// Never delete files imported only by an earlier, unsuccessful attempt.
recoveredFiles = [];
// Native recovery owns the issue before updating its run. Acquire the
// charge's issue FK lock first so inserting it cannot reverse that order.
const issueId = text(identity.issueId);
@@ -39,16 +70,18 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
let [run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
if (!run?.costAccountingPending || !terminalStatuses.includes(run.status)) return false;
if (run.costAccountedAt) {
if (object(run.usageJson).accountingLateReceiptPending === true)
return reconcileLateRunUsageReceiptInTransaction(tx, run, publications);
await notifyStrandedAgentStatus(tx, { companyId: run.companyId, agentId: run.agentId });
await tx.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, run.id));
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(run.companyId, run.agentId,
text(object(object(run.usageJson).ledgerScope).projectId));
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId,
projectId: text(object(object(run.usageJson).ledgerScope).projectId) });
return false;
}
// The bounded global replay can leave a newer receipt behind an older
// complete snapshot. Drain this run under the accounting lock before
// deciding completeness, and retain disk evidence until the outer commit.
recoveredFiles = await recoverPendingRunUsageReceipts(tx, { companyId: identity.companyId, runId }, undefined, { retainFiles: true, index });
if (JSON.stringify(await readAccountingClosure(tx, runId)) !== JSON.stringify(closure))
throw conflict("Run accounting closure changed during receipt recovery. Retry accounting.");
for (const item of pendingReceipts) await persistUsageReceiptInTransaction(tx, item.envelope);
recoveredFiles = pendingReceipts.map(item => item.file);
[run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
if (text(object(object(run.usageJson).ledgerScope).issueId) !== issueId) {
throw conflict("Run cost attribution changed. Retry accounting.");
@@ -69,6 +102,7 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
// arrives. Keep the debt pending instead of acknowledging an invented zero
// and permanently discarding that late receipt.
if (!preProviderFailure && object(run.usageJson).accountingReceiptReady === false) return false;
if (!preProviderFailure) await ensureRuntimeAccountingBaseline(tx, run.companyId, run.agentId);
// Acknowledgement is part of the same transaction. Budget checks inside
// this transaction must not mistake this receipt for missing accounting.
await notifyStrandedAgentStatus(tx, { companyId: run.companyId, agentId: run.agentId });
@@ -76,8 +110,8 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
await tx.update(budgetReservations).set({ state: preProviderFailure ? "released" : "settled", settledAt: new Date() })
.where(and(eq(budgetReservations.companyId, run.companyId), eq(budgetReservations.runId, run.id), eq(budgetReservations.state, "held")));
if (preProviderFailure) {
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(run.companyId, run.agentId,
text(object(object(run.usageJson).ledgerScope).projectId));
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId,
projectId: text(object(object(run.usageJson).ledgerScope).projectId) });
return true;
}
const usage = object(run.usageJson);
@@ -136,27 +170,10 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
} else {
await createCostEventInTransaction(tx, run.companyId, receipt, publications, undefined, { deferBudgetEvaluation: true });
}
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(run.companyId, run.agentId, text(scope.projectId));
const [event] = await tx.select({
inputTokens: sql<string>`coalesce(sum(${costEvents.inputTokens}),0)::text`,
cachedInputTokens: sql<string>`coalesce(sum(${costEvents.cachedInputTokens}),0)::text`,
outputTokens: sql<string>`coalesce(sum(${costEvents.outputTokens}),0)::text`,
costCents: sql<string>`coalesce(sum(${costEvents.costCents}),0)::text`,
}).from(costEvents).where(and(eq(costEvents.companyId, run.companyId), eq(costEvents.heartbeatRunId, run.id)));
await tx.insert(agentRuntimeState).values({ agentId: run.agentId, companyId: run.companyId, adapterType: agent.adapterType }).onConflictDoNothing();
await tx.execute(sql`insert into accounting_runtime_baselines (agent_id,company_id,cost_cents,input_tokens,cached_input_tokens,output_tokens)
select agent_id,company_id,total_cost_cents,total_input_tokens,total_cached_input_tokens,total_output_tokens
from agent_runtime_state where agent_id = ${run.agentId} on conflict do nothing`);
await tx.update(agentRuntimeState).set({
totalInputTokens: sql`${agentRuntimeState.totalInputTokens} + ${event.inputTokens}`,
totalCachedInputTokens: sql`${agentRuntimeState.totalCachedInputTokens} + ${event.cachedInputTokens}`,
totalOutputTokens: sql`${agentRuntimeState.totalOutputTokens} + ${event.outputTokens}`,
totalCostCents: sql`${agentRuntimeState.totalCostCents} + ${event.costCents}::numeric`,
updatedAt: new Date(),
}).where(eq(agentRuntimeState.agentId, run.agentId));
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId, projectId: text(scope.projectId) });
await tx.update(heartbeatRuns).set({ costAccountingPending: false, costAccountedAt: new Date(), accountingProjectionVersion: "v2", accountingLastError: null, accountingLastAttemptAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
return true;
}));
}), { "budget.accounting.owner": "run" });
for (const file of recoveredFiles) {
try { await fs.rm(file, { force: true }); }
catch (error) { logger.warn({ err: error, runId }, "Accounting committed; recovered receipt cleanup will retry"); }
@@ -167,7 +184,10 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
function actionableRunAccounting() {
return and(
eq(heartbeatRuns.costAccountingPending, true), inArray(heartbeatRuns.status, terminalStatuses),
or(sql`${heartbeatRuns.costAccountedAt} is not null`, and(
or(and(sql`${heartbeatRuns.costAccountedAt} is not null`,
or(sql`${heartbeatRuns.usageJson}->>'accountingLateReceiptPending' is distinct from 'true'`,
sql`${heartbeatRuns.usageJson}->>'accountingLateReceiptId' is distinct from ${heartbeatRuns.usageJson}->>'accountingLateReceiptAttemptedId'`)), and(
sql`${heartbeatRuns.costAccountedAt} is null`,
sql`${heartbeatRuns.usageJson}->>'accountingCaptureFailed' is distinct from 'true'`,
sql`(${heartbeatRuns.resultJson}->'executionRecovery'->>'providerWorkStarted' = 'false'
or ${heartbeatRuns.usageJson}->>'accountingProviderWorkStarted' = 'false'
@@ -0,0 +1,56 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { accountingRuntimeBaselines, agents, agentRuntimeState, companies, costEvents, createDb, heartbeatRuns, type Db } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { ensureRuntimeAccountingBaseline, readRuntimeAccountingTotals } from "./runtime-accounting-totals.js";
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("ledger-derived runtime totals", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("runtime-ledger-totals-");
db = createDb(database.connectionString);
}, 60_000);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
async function seed() {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Runtime totals", issuePrefix: companyId.slice(0, 8) });
await db.insert(agents).values({ id: agentId, companyId, name: "Runtime totals", adapterType: "process" });
await db.insert(agentRuntimeState).values({ agentId, companyId, adapterType: "process",
totalCostCents: 125.125, totalInputTokens: 100, totalCachedInputTokens: 20, totalOutputTokens: 30 });
return { companyId, agentId };
}
it("preserves the legacy baseline exactly and includes acknowledged ledger/valuation changes once", async () => {
const owner = await seed();
await db.transaction(tx => ensureRuntimeAccountingBaseline(tx as unknown as Db, owner.companyId, owner.agentId));
const [run] = await db.insert(heartbeatRuns).values({ ...owner, status: "succeeded", costAccountedAt: new Date(), accountingProjectionVersion: "v2" }).returning();
const [event] = await db.insert(costEvents).values({ ...owner, heartbeatRunId: run.id, provider: "test", model: "test",
costCents: 1.125, inputTokens: 10, cachedInputTokens: 2, outputTokens: 3, occurredAt: new Date("2020-01-01Z") }).returning();
const [pending] = await db.insert(heartbeatRuns).values({ ...owner, status: "succeeded", costAccountingPending: true, accountingProjectionVersion: "v2" }).returning();
await db.insert(costEvents).values({ ...owner, heartbeatRunId: pending.id, provider: "test", model: "test",
costCents: 99, inputTokens: 999, occurredAt: new Date() });
expect(await readRuntimeAccountingTotals(db, owner.companyId, owner.agentId)).toEqual({
totalCostCents: 126.25, totalInputTokens: 110, totalCachedInputTokens: 22, totalOutputTokens: 33,
});
// Effective valuation can be corrected without mutating an agent counter.
await db.update(costEvents).set({ costCents: 2.125 }).where(eq(costEvents.id, event.id));
expect((await readRuntimeAccountingTotals(db, owner.companyId, owner.agentId)).totalCostCents).toBe(127.25);
const [raw] = await db.select({ cents: sql<string>`${agentRuntimeState.totalCostCents}::text` }).from(agentRuntimeState)
.where(eq(agentRuntimeState.agentId, owner.agentId));
expect(raw.cents).toBe("125.1250000");
expect(await db.select().from(accountingRuntimeBaselines).where(eq(accountingRuntimeBaselines.agentId, owner.agentId))).toHaveLength(1);
});
it("does not double-count a historical acknowledged ledger when its baseline is missing", async () => {
const owner = await seed();
await db.insert(heartbeatRuns).values({ ...owner, status: "succeeded", costAccountedAt: new Date(), accountingProjectionVersion: "v2" });
expect((await readRuntimeAccountingTotals(db, owner.companyId, owner.agentId)).totalCostCents).toBe(125.125);
await expect(db.transaction(tx => ensureRuntimeAccountingBaseline(tx as unknown as Db, owner.companyId, owner.agentId)))
.rejects.toThrow("baseline is missing");
expect(await db.select().from(accountingRuntimeBaselines).where(eq(accountingRuntimeBaselines.agentId, owner.agentId))).toEqual([]);
});
});
@@ -0,0 +1,51 @@
import { and, eq, sql } from "drizzle-orm";
import { accountingRuntimeBaselines, heartbeatRuns, type Db } from "@paperclipai/db";
import { conflict } from "../errors.js";
/** A missing baseline is initialized once, before this owner's first ledger-only
* acknowledgement. The compatibility floor must have drained legacy writers. */
export async function ensureRuntimeAccountingBaseline(tx: Db, companyId: string, agentId: string) {
const [baseline] = await tx.select({ agentId: accountingRuntimeBaselines.agentId }).from(accountingRuntimeBaselines)
.where(and(eq(accountingRuntimeBaselines.companyId, companyId), eq(accountingRuntimeBaselines.agentId, agentId)));
if (baseline) return;
const [acknowledged] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId),
sql`${heartbeatRuns.costAccountedAt} is not null`,
sql`${heartbeatRuns.accountingProjectionVersion} in ('v2')`,
)).limit(1);
if (acknowledged) throw conflict("Runtime accounting baseline is missing; reconcile historical accounting before retrying");
// This row is independent from the mutable runtime/session row. Do not create
// or update agent_runtime_state from the settlement path.
await tx.execute(sql`insert into accounting_runtime_baselines
(agent_id, company_id, cost_cents, input_tokens, cached_input_tokens, output_tokens)
select ${agentId}::uuid, ${companyId}::uuid,
coalesce(a.total_cost_cents, 0), coalesce(a.total_input_tokens, 0),
coalesce(a.total_cached_input_tokens, 0), coalesce(a.total_output_tokens, 0)
from (select 1) seed left join agent_runtime_state a
on a.agent_id = ${agentId}::uuid and a.company_id = ${companyId}::uuid
on conflict (agent_id) do nothing`);
}
/** Public totals are a single committed ledger snapshot. Legacy rows without a
* baseline retain their frozen historical values until the first new receipt. */
export async function readRuntimeAccountingTotals(db: Db, companyId: string, agentId: string) {
const [totals] = await db.execute<{
totalCostCents: string; totalInputTokens: string; totalCachedInputTokens: string; totalOutputTokens: string;
}>(sql`with ledger as (
select coalesce(sum(e.cost_cents), 0) as cents, coalesce(sum(e.input_tokens), 0) as input,
coalesce(sum(e.cached_input_tokens), 0) as cached, coalesce(sum(e.output_tokens), 0) as output
from cost_events e join heartbeat_runs r on r.id = e.heartbeat_run_id and r.company_id = e.company_id
where r.company_id = ${companyId}::uuid and r.agent_id = ${agentId}::uuid
and r.cost_accounted_at is not null and r.accounting_projection_version in ('v2')
) select
(case when b.agent_id is null then coalesce(a.total_cost_cents, 0) else b.cost_cents + l.cents end)::text as "totalCostCents",
(case when b.agent_id is null then coalesce(a.total_input_tokens, 0) else b.input_tokens + l.input end)::text as "totalInputTokens",
(case when b.agent_id is null then coalesce(a.total_cached_input_tokens, 0) else b.cached_input_tokens + l.cached end)::text as "totalCachedInputTokens",
(case when b.agent_id is null then coalesce(a.total_output_tokens, 0) else b.output_tokens + l.output end)::text as "totalOutputTokens"
from ledger l left join accounting_runtime_baselines b on b.agent_id = ${agentId}::uuid and b.company_id = ${companyId}::uuid
left join agent_runtime_state a on a.agent_id = ${agentId}::uuid and a.company_id = ${companyId}::uuid`);
return {
totalCostCents: Number(totals.totalCostCents), totalInputTokens: Number(totals.totalInputTokens),
totalCachedInputTokens: Number(totals.totalCachedInputTokens), totalOutputTokens: Number(totals.totalOutputTokens),
};
}
+319 -110
View File
@@ -6,19 +6,21 @@ import { promises as fs } from "node:fs";
import path from "node:path";
import { z } from "zod";
import { and, eq, isNull, sql } from "drizzle-orm";
import { heartbeatRuns, runUsageReceipts, type Db } from "@paperclipai/db";
import { heartbeatRuns, runUsageReceipts, costEvents, costAdjustments, type Db } from "@paperclipai/db";
import type { AdapterExecutionResult, AdapterUsageCheckpoint } from "@paperclipai/adapter-utils";
import { compareCents, normalizeCents, pricingProvenanceSchema, usdToUnits } from "@paperclipai/shared";
import { compareCents, normalizeCents, pricingProvenanceSchema, usdToUnits, unitsToCents } from "@paperclipai/shared";
import { idleAccountingSpoolPath, registerIdleSpoolDirectory } from "./idle-local-work.js";
import { conflict, notFound } from "../errors.js";
import { logger } from "../middleware/logger.js";
import { receiptFingerprint } from "./receipt-fingerprint.js";
import { withAccountingTransaction } from "./accounting-transaction.js";
import { withOperationAccountingTransaction } from "./accounting-transaction.js";
import { ensureDurableDirectory, syncDirectory } from "../lib/durable-directory.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { knownRunCostCents } from "./budget-exposure.js";
import { createLiveRunUsageProjection, liveRunUsageCostCents } from "./budget-live-usage.js";
import { budgetServiceInTransaction } from "./budgets.js";
import { applyCostCorrectionInTransaction } from "./accounting-corrections.js";
import type { ActivityPublication } from "./activity-log.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
const token = z.number().int().nonnegative().max(2_147_483_647);
const price = z.number().finite().nonnegative().nullable().optional();
@@ -66,85 +68,256 @@ export async function spoolUsageReceipt(envelope: UsageReceiptEnvelope, director
return target;
}
export async function persistUsageReceipt(db: Db, raw: UsageReceiptEnvelope) {
function lateReceiptDebts(usage: Record<string, unknown>): Record<string, boolean> {
const raw = usage.accountingLateReceiptDebts;
const debts = Object.fromEntries(Object.entries(raw && typeof raw === "object" && !Array.isArray(raw) ? raw : {})
.filter((entry): entry is [string, boolean] => typeof entry[1] === "boolean"));
// Backward compatibility with the original single pending-receipt marker.
if (usage.accountingLateReceiptPending === true && typeof usage.accountingLateReceiptId === "string"
&& !(usage.accountingLateReceiptId in debts))
debts[usage.accountingLateReceiptId] = usage.accountingLateReceiptAttemptedId === usage.accountingLateReceiptId;
return debts;
}
function acceptedReceiptSequence(usage: Record<string, unknown>, reconciled?: UsageReceiptEnvelope) {
let sequence = Number(usage.accountingReceiptSequence ?? 0);
if (usage.accountingReconciledReceiptSourceId === usage.accountingReceiptSourceId)
sequence = Math.max(sequence, Number(usage.accountingReconciledReceiptSequence ?? 0));
// Rows corrected before the explicit sequence field was introduced still have
// their immutable reconciled receipt, which supplies the same source fence.
if (reconciled && reconciled.sourceId === usage.accountingReceiptSourceId) sequence = Math.max(sequence, reconciled.sequence);
return sequence;
}
/** Caller transaction may already own the run row. No company mutex or
* filesystem work belongs in journal acceptance. */
export async function persistUsageReceiptInTransaction(tx: Db, raw: UsageReceiptEnvelope) {
const envelope = envelopeSchema.parse(raw);
const hash = receiptFingerprint(envelope);
return withAccountingTransaction(db, envelope.companyId, async (tx, publications) => {
const [run] = await tx.select({
id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, status: heartbeatRuns.status,
usageJson: heartbeatRuns.usageJson, costAccountedAt: heartbeatRuns.costAccountedAt,
}).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, envelope.runId), eq(heartbeatRuns.companyId, envelope.companyId))).for("update");
if (!run) throw notFound("Accounting run not found");
const [existing] = await tx.select().from(runUsageReceipts).where(and(eq(runUsageReceipts.companyId, envelope.companyId), eq(runUsageReceipts.runId, envelope.runId), eq(runUsageReceipts.sourceId, envelope.sourceId), eq(runUsageReceipts.sequence, envelope.sequence)));
if (existing && existing.receiptHash !== hash) throw conflict("Conflicting usage checkpoint replay");
if (!existing && run.costAccountedAt && envelope.producerEvent) {
const previous = run.usageJson ?? {};
const [saved] = await tx.select({ receiptJson: runUsageReceipts.receiptJson }).from(runUsageReceipts).where(and(
eq(runUsageReceipts.companyId, envelope.companyId), eq(runUsageReceipts.runId, envelope.runId),
sql`${runUsageReceipts.id}::text = ${previous.accountingReceiptId ?? null}`));
const accepted = saved ? envelopeSchema.parse(saved.receiptJson).receipt : undefined;
const monetaryAmount = (receipt: UsageReceiptEnvelope["receipt"] | undefined) => {
const value = receipt?.cacheAdjustedCostUsd ?? receipt?.costUsdExact ?? receipt?.costUsd;
return value == null ? null : usdToUnits(value);
};
const incomingMoney = monetaryAmount(envelope.receipt);
const authoritativeMoney = incomingMoney !== null && envelope.receipt.pricingProvenance?.source !== "rate_card";
const authoritativeProvenance = envelope.receipt.pricingProvenance
&& !["rate_card", "unknown"].includes(envelope.receipt.pricingProvenance.source);
if ((envelope.producerEvent.eventType === "usage.reported" && (!envelope.receipt.usage || !accepted))
|| Object.entries(envelope.receipt.usage ?? {}).some(([key, value]) => previous[key] !== value)
|| (authoritativeMoney && incomingMoney !== monetaryAmount(accepted))
|| (authoritativeProvenance && receiptFingerprint(envelope.receipt.pricingProvenance) !== receiptFingerprint(accepted?.pricingProvenance ?? null))) {
throw conflict("native_usage_after_finalization_requires_reconciliation");
}
}
if (!existing) await tx.insert(runUsageReceipts).values({
id: envelope.id, companyId: envelope.companyId, runId: envelope.runId, sourceId: envelope.sourceId,
sequence: envelope.sequence, receiptHash: hash, receiptJson: envelope, complete: envelope.receipt.complete, receivedAt: new Date(envelope.receivedAt),
});
const previous = run.usageJson ?? {};
// A restarted/replaced controller owns a fresh source. Old spool files
// remain auditable but cannot overwrite its snapshots or acknowledged work.
if (run.costAccountedAt || previous.accountingReceiptSourceId !== envelope.sourceId || Number(previous.accountingReceiptSequence ?? 0) >= envelope.sequence) return;
const receipt = envelope.receipt;
const billingType = receipt.billingType === "api" ? "metered_api" : receipt.billingType === "subscription" ? "subscription_included" : receipt.billingType ?? "unknown";
const oldProjection = previous.budgetExposure && typeof previous.budgetExposure === "object"
? previous.budgetExposure as Record<string, unknown> : {};
const canonicalProjection = typeof oldProjection.sourceId === "string" && typeof oldProjection.sequence === "number";
let previousCost = canonicalProjection ? null : knownRunCostCents(previous);
if (typeof oldProjection.costCents === "string") {
try {
const projected = normalizeCents(oldProjection.costCents);
if (compareCents(projected, 0) >= 0 && (canonicalProjection || compareCents(projected, previousCost ?? "0") > 0)) previousCost = projected;
} catch { /* Older malformed projections are not billing evidence. */ }
}
const reportedCost = envelope.liveCostCents !== undefined ? envelope.liveCostCents
: liveRunUsageCostCents({ ...receipt, billingType } as AdapterUsageCheckpoint);
const observedCost = reportedCost === null ? previousCost
: !receipt.complete && previousCost !== null && compareCents(previousCost, reportedCost) > 0 ? previousCost : reportedCost;
// Running checkpoints cannot settle yet. Their terminal transition owns
// the accounting wake; only a late receipt needs to reactivate that worker.
if (!["queued", "running"].includes(run.status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, envelope.companyId);
await tx.update(heartbeatRuns).set({
costAccountingPending: true,
accountingLastError: previous.accountingCaptureFailed === true ? "usage_capture_failed" : null,
usageJson: { ...previous, ...receipt.usage,
provider: receipt.provider ?? "unknown", biller: receipt.biller ?? receipt.provider ?? "unknown", model: receipt.model ?? "unknown", billingType,
costUsd: receipt.costUsd ?? null, costUsdExact: receipt.costUsdExact ?? null, cacheAdjustedCostUsd: receipt.cacheAdjustedCostUsd ?? null,
usageByModel: receipt.usageByModel ?? null, usageSource: receipt.usageBasis ?? "per_run", providerRequestId: receipt.providerRequestId ?? null,
accountingReceiptReady: previous.accountingCaptureFailed !== true && receipt.complete && receipt.usageBasis !== "session_cumulative", accountingReceiptId: envelope.id,
accountingReceiptSequence: envelope.sequence, accountingReceiptReceivedAt: envelope.receivedAt,
costStatus: receipt.costStatus ?? null,
budgetExposure: { sourceId: envelope.sourceId, sequence: envelope.sequence, costCents: observedCost },
pricingProvenance: receipt.pricingProvenance ?? { source: receipt.costUsd != null || receipt.costUsdExact != null ? "provider_reported" : "unknown", version: "accounting-receipt/v1" },
},
}).where(and(eq(heartbeatRuns.id, run.id), isNull(heartbeatRuns.costAccountedAt)));
if (observedCost !== null && compareCents(observedCost, previousCost ?? "0") !== 0) {
const scope = previous.ledgerScope && typeof previous.ledgerScope === "object" ? previous.ledgerScope as Record<string, unknown> : {};
await budgetServiceInTransaction(tx, publications).evaluateAffectedScopes(envelope.companyId, run.agentId,
typeof scope.projectId === "string" ? scope.projectId : null);
}
const [run] = await tx.select({
id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, status: heartbeatRuns.status,
usageJson: heartbeatRuns.usageJson, costAccountedAt: heartbeatRuns.costAccountedAt,
}).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, envelope.runId), eq(heartbeatRuns.companyId, envelope.companyId))).for("update");
if (!run) throw notFound("Accounting run not found");
const [existing] = await tx.select().from(runUsageReceipts).where(and(eq(runUsageReceipts.companyId, envelope.companyId), eq(runUsageReceipts.runId, envelope.runId), eq(runUsageReceipts.sourceId, envelope.sourceId), eq(runUsageReceipts.sequence, envelope.sequence)));
if (existing && existing.receiptHash !== hash) throw conflict("Conflicting usage checkpoint replay");
if (!existing) await tx.insert(runUsageReceipts).values({
id: envelope.id, companyId: envelope.companyId, runId: envelope.runId, sourceId: envelope.sourceId,
sequence: envelope.sequence, receiptHash: hash, receiptJson: envelope, complete: envelope.receipt.complete, receivedAt: new Date(envelope.receivedAt),
});
const previous = run.usageJson ?? {};
if (run.costAccountedAt) {
const activeSource = previous.accountingReceiptSourceId === envelope.sourceId;
const closedSources = previous.accountingReceiptSourceFences as Record<string, number> | undefined;
if (!activeSource && typeof closedSources?.[envelope.sourceId] === "number" && envelope.sequence <= closedSources[envelope.sourceId])
return { reconciliationRequired: false };
const [reconciled] = typeof previous.accountingReconciledReceiptId === "string"
? await tx.select({ receiptJson: runUsageReceipts.receiptJson }).from(runUsageReceipts).where(and(
eq(runUsageReceipts.runId, run.id), eq(runUsageReceipts.companyId, envelope.companyId),
sql`${runUsageReceipts.id}::text = ${previous.accountingReconciledReceiptId}`)) : [];
const reconciledEnvelope = reconciled ? envelopeSchema.parse(reconciled.receiptJson) : undefined;
const newer = envelope.sequence > acceptedReceiptSequence(previous, reconciledEnvelope);
const comparison = reconciledEnvelope ? { ...previous, ...reconciledEnvelope.receipt } : previous;
const acceptedMoney = knownRunCostCents(comparison);
const incomingMoney = liveRunUsageCostCents(envelope.receipt as AdapterUsageCheckpoint);
const authoritative = envelope.receipt.pricingProvenance
&& !["rate_card", "unknown"].includes(envelope.receipt.pricingProvenance.source);
const rateCardOnly = envelope.producerEvent && envelope.receipt.pricingProvenance?.source === "rate_card";
const changed = (envelope.producerEvent?.eventType === "usage.reported" && !envelope.receipt.usage)
|| envelope.receipt.usageBasis === "session_cumulative"
|| ["provider", "biller", "model", "billingType"].some(key => {
const value = envelope.receipt[key as keyof typeof envelope.receipt];
return value !== undefined && value !== null && value !== previous[key];
}) || Object.entries(envelope.receipt.usage ?? {}).some(([key, value]) => previous[key] !== value)
|| (!rateCardOnly && incomingMoney !== null && (acceptedMoney === null || compareCents(incomingMoney, acceptedMoney) !== 0))
|| (!rateCardOnly && envelope.receipt.costStatus !== undefined && envelope.receipt.costStatus !== comparison.costStatus)
|| (authoritative
&& receiptFingerprint(envelope.receipt.pricingProvenance) !== receiptFingerprint(comparison.pricingProvenance ?? null));
const debts = lateReceiptDebts(previous);
if ((!existing && changed && (!activeSource || newer)) || envelope.id in debts) {
if (!existing) {
debts[envelope.id] = false;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, envelope.companyId);
await queueBudgetEvaluation(tx, { companyId: envelope.companyId, agentId: run.agentId,
projectId: typeof (previous.ledgerScope as Record<string, unknown> | undefined)?.projectId === "string"
? (previous.ledgerScope as Record<string, string>).projectId : null });
await tx.update(heartbeatRuns).set({ costAccountingPending: true,
accountingLastError: "late_usage_requires_reconciliation",
usageJson: { ...previous, accountingLateReceiptPending: true, accountingLateReceiptId: envelope.id,
accountingLateReceiptDebts: debts },
}).where(eq(heartbeatRuns.id, run.id));
}
return { reconciliationRequired: true };
}
return { reconciliationRequired: false };
}
// A replaced source cannot overwrite the accepted snapshot. Its immutable
// journal entry remains available to source-handoff/recovery diagnostics.
if (previous.accountingReceiptSourceId !== envelope.sourceId || Number(previous.accountingReceiptSequence ?? 0) >= envelope.sequence) {
const fences = previous.accountingReceiptSourceFences as Record<string, number> | undefined;
const [fenced] = !existing && typeof fences?.[envelope.sourceId] === "number" && envelope.sequence > fences[envelope.sourceId]
? await tx.select({ receiptJson: runUsageReceipts.receiptJson }).from(runUsageReceipts).where(and(
eq(runUsageReceipts.companyId, envelope.companyId), eq(runUsageReceipts.runId, run.id),
eq(runUsageReceipts.sourceId, envelope.sourceId), eq(runUsageReceipts.sequence, fences[envelope.sourceId]))) : [];
const fencedReceipt = fenced ? envelopeSchema.parse(fenced.receiptJson).receipt : undefined;
const sameFencedEvidence = fencedReceipt && receiptFingerprint({ usage: fencedReceipt.usage, money: liveRunUsageCostCents(fencedReceipt as AdapterUsageCheckpoint) })
=== receiptFingerprint({ usage: envelope.receipt.usage, money: liveRunUsageCostCents(envelope.receipt as AdapterUsageCheckpoint) });
if (!existing && typeof fences?.[envelope.sourceId] === "number" && envelope.sequence > fences[envelope.sourceId] && !sameFencedEvidence) {
await queueBudgetEvaluation(tx, { companyId: envelope.companyId, agentId: run.agentId,
projectId: typeof (previous.ledgerScope as Record<string, unknown> | undefined)?.projectId === "string"
? (previous.ledgerScope as Record<string, string>).projectId : null });
await tx.update(heartbeatRuns).set({ costAccountingPending: true, accountingLastError: "usage_after_source_handoff_requires_reconciliation",
usageJson: { ...previous, accountingCaptureFailed: true, accountingReceiptReady: false,
accountingSourceReceiptPending: envelope.id },
}).where(eq(heartbeatRuns.id, run.id));
}
return { reconciliationRequired: false };
}
const receipt = envelope.receipt;
const billingType = receipt.billingType === "api" ? "metered_api" : receipt.billingType === "subscription" ? "subscription_included" : receipt.billingType ?? "unknown";
const oldProjection = previous.budgetExposure && typeof previous.budgetExposure === "object"
? previous.budgetExposure as Record<string, unknown> : {};
const canonicalProjection = typeof oldProjection.sourceId === "string" && typeof oldProjection.sequence === "number";
let previousCost = canonicalProjection ? null : knownRunCostCents(previous);
if (typeof oldProjection.costCents === "string") {
try {
const projected = normalizeCents(oldProjection.costCents);
if (compareCents(projected, 0) >= 0 && (canonicalProjection || compareCents(projected, previousCost ?? "0") > 0)) previousCost = projected;
} catch { /* Older malformed projections are not billing evidence. */ }
}
const reportedCost = envelope.liveCostCents !== undefined ? envelope.liveCostCents
: liveRunUsageCostCents({ ...receipt, billingType } as AdapterUsageCheckpoint);
const observedCost = reportedCost === null ? previousCost
: !receipt.complete && previousCost !== null && compareCents(previousCost, reportedCost) > 0 ? previousCost : reportedCost;
// Running checkpoints cannot settle yet. Their terminal transition owns
// the accounting wake; only a late receipt needs to reactivate that worker.
if (!["queued", "running"].includes(run.status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, envelope.companyId);
await tx.update(heartbeatRuns).set({
costAccountingPending: true,
accountingLastError: previous.accountingCaptureFailed === true ? "usage_capture_failed" : null,
usageJson: { ...previous, ...receipt.usage,
provider: receipt.provider ?? "unknown", biller: receipt.biller ?? receipt.provider ?? "unknown", model: receipt.model ?? "unknown", billingType,
costUsd: receipt.costUsd ?? null, costUsdExact: receipt.costUsdExact ?? null, cacheAdjustedCostUsd: receipt.cacheAdjustedCostUsd ?? null,
usageByModel: receipt.usageByModel ?? null, usageSource: receipt.usageBasis ?? "per_run", providerRequestId: receipt.providerRequestId ?? null,
accountingReceiptReady: previous.accountingCaptureFailed !== true && receipt.complete && receipt.usageBasis !== "session_cumulative", accountingReceiptId: envelope.id,
accountingReceiptSequence: envelope.sequence, accountingReceiptReceivedAt: envelope.receivedAt,
costStatus: receipt.costStatus ?? null,
budgetExposure: { sourceId: envelope.sourceId, sequence: envelope.sequence, costCents: observedCost },
pricingProvenance: receipt.pricingProvenance ?? { source: receipt.costUsd != null || receipt.costUsdExact != null ? "provider_reported" : "unknown", version: "accounting-receipt/v1" },
},
}).where(and(eq(heartbeatRuns.id, run.id), isNull(heartbeatRuns.costAccountedAt)));
const scope = previous.ledgerScope && typeof previous.ledgerScope === "object" ? previous.ledgerScope as Record<string, unknown> : {};
await queueBudgetEvaluation(tx, { companyId: envelope.companyId, agentId: run.agentId,
projectId: typeof scope.projectId === "string" ? scope.projectId : null });
return { reconciliationRequired: false };
}
/** Caller owns the run row. A money-only authoritative revision can correct one
* existing final charge. Ambiguous usage/source/model evidence remains debt. */
export async function reconcileLateRunUsageReceiptInTransaction(tx: Db, run: typeof heartbeatRuns.$inferSelect, publications: ActivityPublication[]) {
const previous = run.usageJson ?? {};
if (previous.accountingLateReceiptPending !== true || typeof previous.accountingLateReceiptId !== "string") return false;
// Unchanged attempted debt is excluded from background scans. An explicit
// operator retry still rechecks it after a reviewed ledger correction.
const [saved] = await tx.select({ receiptJson: runUsageReceipts.receiptJson }).from(runUsageReceipts).where(and(
eq(runUsageReceipts.companyId, run.companyId), eq(runUsageReceipts.runId, run.id),
sql`${runUsageReceipts.id}::text = ${previous.accountingLateReceiptId}`));
const late = saved ? envelopeSchema.parse(saved.receiptJson) : undefined;
const receipt = late?.receipt;
const [reconciled] = typeof previous.accountingReconciledReceiptId === "string"
? await tx.select({ receiptJson: runUsageReceipts.receiptJson }).from(runUsageReceipts).where(and(
eq(runUsageReceipts.runId, run.id), eq(runUsageReceipts.companyId, run.companyId),
sql`${runUsageReceipts.id}::text = ${previous.accountingReconciledReceiptId}`)) : [];
const acceptedSequence = acceptedReceiptSequence(previous, reconciled ? envelopeSchema.parse(reconciled.receiptJson) : undefined);
const events = await tx.select({ event: costEvents, exact: sql<string>`${costEvents.costCents}::text` }).from(costEvents)
.where(and(eq(costEvents.companyId, run.companyId), eq(costEvents.heartbeatRunId, run.id),
eq(costEvents.idempotencyKey, `heartbeat:${run.id}:final`))).for("update");
const rawAmount = receipt?.cacheAdjustedCostUsd ?? receipt?.costUsdExact ?? receipt?.costUsd;
const automaticCorrectionPrefix = `heartbeat:${run.id}:late:`;
const [reviewedValuation] = events.length === 1 ? await tx.select({ id: costAdjustments.id }).from(costAdjustments)
.where(and(eq(costAdjustments.companyId, run.companyId), eq(costAdjustments.costEventId, events[0].event.id),
sql`not (${costAdjustments.actorId} = 'cost_accounting'
and left(${costAdjustments.idempotencyKey}, length(${automaticCorrectionPrefix})) = ${automaticCorrectionPrefix})`)).limit(1) : [];
const sameTokens = receipt?.usage && ["inputTokens", "cachedInputTokens", "outputTokens"].every(key =>
Number(previous[key] ?? 0) === Number(receipt.usage?.[key as keyof typeof receipt.usage] ?? 0));
const authoritative = receipt?.pricingProvenance && !["rate_card", "unknown"].includes(receipt.pricingProvenance.source);
const billingType = receipt?.billingType === "api" ? "metered_api"
: receipt?.billingType === "subscription" ? "subscription_included" : receipt?.billingType ?? "unknown";
const biller = receipt?.biller ?? receipt?.provider ?? "unknown";
const qualified = late && receipt?.complete && receipt.usageBasis !== "session_cumulative"
&& late.sourceId === previous.accountingReceiptSourceId && sameTokens && authoritative
&& rawAmount != null && receipt.costStatus !== "unpriced" && events.length === 1
&& receipt.model === events[0].event.model && receipt.provider === events[0].event.provider
&& biller === events[0].event.biller && billingType === events[0].event.billingType
&& (!receipt.providerRequestId || !events[0].event.providerRequestId || receipt.providerRequestId === events[0].event.providerRequestId)
&& (!receipt.usageByModel || receipt.usageByModel.length === 0);
const debts = lateReceiptDebts(previous);
const currentId = previous.accountingLateReceiptId;
async function updateDebtState(reconciled: boolean, advanceReceiptIdentity = reconciled) {
if (reconciled) delete debts[currentId];
else debts[currentId] = true;
const nextId = Object.keys(debts).find(id => !debts[id]) ?? Object.keys(debts)[0] ?? null;
await tx.update(heartbeatRuns).set({ costAccountingPending: nextId !== null,
accountingLastError: nextId ? "late_usage_requires_reconciliation" : null, accountingLastAttemptAt: new Date(),
usageJson: { ...previous, accountingLateReceiptPending: nextId !== null,
accountingLateReceiptDebts: debts, accountingLateReceiptId: nextId,
accountingLateReceiptAttemptedId: nextId && debts[nextId] ? nextId : null,
...(reconciled && advanceReceiptIdentity ? { accountingReconciledReceiptId: currentId,
accountingReconciledReceiptSourceId: late!.sourceId, accountingReconciledReceiptSequence: late!.sequence } : {}),
},
}).where(eq(heartbeatRuns.id, run.id));
}
if (!qualified) { await updateDebtState(false); return false; }
const corrected = receipt.billingType === "subscription_included" || receipt.billingType === "subscription"
? "0.0000000" : unitsToCents(usdToUnits(rawAmount));
if (reviewedValuation) {
if (compareCents(events[0].exact, corrected) !== 0) { await updateDebtState(false); return false; }
// The operator has already reconciled this exact billing evidence. Resolve
// the hold without creating another adjustment or replacing their pricing
// authority, and never rewind a newer accepted source sequence.
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId,
projectId: typeof (previous.ledgerScope as Record<string, unknown> | undefined)?.projectId === "string"
? (previous.ledgerScope as Record<string, string>).projectId : null });
await updateDebtState(true, late.sequence > acceptedSequence);
if (Object.keys(debts).length === 0) await notifyDeliveryWork(tx, DELIVERY_QUEUES.strandedAgentStatus,
JSON.stringify({ companyId: run.companyId, agentId: run.agentId }));
return true;
}
if (late.sequence <= acceptedSequence) {
// A higher complete receipt from this same cumulative source already
// supersedes this money-only debt. Resolve it without repricing or rewinding
// the reconciled receipt identity; unrelated source/usage debt remains.
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId,
projectId: typeof (previous.ledgerScope as Record<string, unknown> | undefined)?.projectId === "string"
? (previous.ledgerScope as Record<string, string>).projectId : null });
await updateDebtState(true, false);
if (Object.keys(debts).length === 0) await notifyDeliveryWork(tx, DELIVERY_QUEUES.strandedAgentStatus,
JSON.stringify({ companyId: run.companyId, agentId: run.agentId }));
return true;
}
if (compareCents(events[0].exact, corrected) !== 0 || events[0].event.costStatus !== "reported"
|| receiptFingerprint(events[0].event.pricingProvenance) !== receiptFingerprint(receipt.pricingProvenance)) {
await applyCostCorrectionInTransaction(tx, publications, { companyId: run.companyId, eventId: events[0].event.id,
idempotencyKey: `heartbeat:${run.id}:late:${late.id}`, expectedCents: events[0].exact, correctedCents: corrected,
reason: "Authoritative late run usage receipt", pricing: receipt.pricingProvenance!, actorId: "cost_accounting" });
}
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId,
projectId: typeof (previous.ledgerScope as Record<string, unknown> | undefined)?.projectId === "string"
? (previous.ledgerScope as Record<string, string>).projectId : null });
await updateDebtState(true);
if (Object.keys(debts).length === 0) await notifyDeliveryWork(tx, DELIVERY_QUEUES.strandedAgentStatus,
JSON.stringify({ companyId: run.companyId, agentId: run.agentId }));
return true;
}
export async function persistUsageReceipt(db: Db, raw: UsageReceiptEnvelope) {
const envelope = envelopeSchema.parse(raw);
const result = await withOperationAccountingTransaction(db, envelope.companyId,
tx => persistUsageReceiptInTransaction(tx, envelope));
// Native delivery keeps its existing explicit reconciliation response, after
// the journal/debt commits. Retrying cannot discard the paid evidence.
if (result.reconciliationRequired && envelope.producerEvent)
throw conflict("native_usage_after_finalization_requires_reconciliation");
}
export async function replayUsageReceipts(db: Db, directory = usageReceiptSpoolPath(), limit = 100) {
@@ -201,49 +374,50 @@ export async function indexPendingUsageReceipts(directory = usageReceiptSpoolPat
return index;
}
export async function recoverPendingRunUsageReceipts(db: Db, input: { companyId: string; runId: string }, directory = usageReceiptSpoolPath(), options: { retainFiles?: boolean; index?: UsageReceiptIndex } = {}) {
// The bounded sweep may not reach this run. Its durable evidence must
// reach the journal before source replacement or ledger acknowledgement.
// A failed save aborts the operation so recovery can retry.
const recovered = new Set<string>();
// A concurrent failed replay can rename a file after releasing its company
// lock. Rescan disappearing paths rather than certify an incomplete scan.
export type PendingRunUsageReceipt = { file: string; envelope: UsageReceiptEnvelope };
/** Capture immutable spool evidence before a database transaction. Finalizers
* revalidate their closed producer identity after taking the run-row fence. */
export async function readPendingRunUsageReceipts(input: { companyId: string; runId: string }, directory = usageReceiptSpoolPath(), index?: UsageReceiptIndex): Promise<PendingRunUsageReceipt[]> {
const pending = new Map<string, PendingRunUsageReceipt>();
for (let scan = 0; scan < 3; scan++) {
let names: string[];
try { names = await fs.readdir(directory); }
catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return [...recovered]; throw error; }
const names = await fs.readdir(directory).catch(error => {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return [];
throw error;
});
let moved = false;
for (const name of names.filter(name => name.endsWith(".json"))) {
const file = path.join(directory, name);
if (recovered.has(file)) continue;
const indexed = options.index?.get(file);
if (pending.has(file)) continue;
const indexed = index?.get(file);
if (indexed !== undefined && (indexed?.companyId !== input.companyId || indexed?.runId !== input.runId)) continue;
let raw: unknown;
try {
const info = await fs.lstat(file);
if (!info.isFile() || info.size > 1024 * 1024) continue;
raw = JSON.parse(await fs.readFile(file, "utf8"));
const raw: unknown = JSON.parse(await fs.readFile(file, "utf8"));
const identity = receiptIdentity(raw);
if (identity?.companyId !== input.companyId || identity?.runId !== input.runId) continue;
pending.set(file, { file, envelope: envelopeSchema.parse(raw) });
} catch (error) {
// The background replayer can move/remove a file while we scan. Invalid
// JSON remains on disk for its normal recovery/error reporting path.
if ((error as NodeJS.ErrnoException).code === "ENOENT") { moved = true; continue; }
if (error instanceof SyntaxError) continue;
throw error;
}
const identity = receiptIdentity(raw);
options.index?.set(file, identity);
if (identity?.companyId !== input.companyId || identity?.runId !== input.runId) continue;
await persistUsageReceipt(db, envelopeSchema.parse(raw));
// Settlement uses the caller's transaction: deleting before its commit
// would lose the only durable evidence if the later ledger write fails.
recovered.add(file);
if (!options.retainFiles) await fs.rm(file, { force: true });
}
if (!moved) return [...recovered];
if (!moved) return [...pending.values()];
}
throw new Error("Accounting spool changed during recovery; retry required");
}
export async function recoverPendingRunUsageReceipts(db: Db, input: { companyId: string; runId: string }, directory = usageReceiptSpoolPath(), options: { retainFiles?: boolean; index?: UsageReceiptIndex } = {}) {
const pending = await readPendingRunUsageReceipts(input, directory, options.index);
for (const item of pending) {
await persistUsageReceipt(db, item.envelope);
if (!options.retainFiles) await fs.rm(item.file, { force: true });
}
return pending.map(item => item.file);
}
export async function createRunUsageRecorder(db: Db, input: { companyId: string; runId: string; adapterType: string }, directory = usageReceiptSpoolPath()) {
registerIdleSpoolDirectory(directory);
const sourceId = randomUUID();
@@ -253,9 +427,33 @@ export async function createRunUsageRecorder(db: Db, input: { companyId: string;
const probe = await fs.open(path.join(directory, `${sourceId}.probe`), "wx", 0o600);
try { await probe.sync(); } finally { await probe.close(); await fs.rm(path.join(directory, `${sourceId}.probe`)); }
await syncDirectory(directory);
await recoverPendingRunUsageReceipts(db, input, directory);
const bound = await db.update(heartbeatRuns).set({ usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || ${JSON.stringify({ accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false })}::jsonb` })
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), isNull(heartbeatRuns.costAccountedAt))).returning({ id: heartbeatRuns.id, runtimeMode: heartbeatRuns.runtimeMode, usageJson: heartbeatRuns.usageJson });
const [beforeHandoff] = await db.select({ usageJson: heartbeatRuns.usageJson }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)));
const receiptIdentity = (state: Record<string, unknown> | null | undefined) => JSON.stringify([
state?.accountingReceiptSourceId, state?.accountingReceiptSequence, state?.accountingReceiptId,
]);
const pending = await readPendingRunUsageReceipts(input, directory);
const bound = await withOperationAccountingTransaction(db, input.companyId, async tx => {
const [run] = await tx.select({ id: heartbeatRuns.id, agentId: heartbeatRuns.agentId,
usageJson: heartbeatRuns.usageJson, costAccountedAt: heartbeatRuns.costAccountedAt }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))).for("update");
if (!run) throw notFound("Accounting run not found");
if (run.costAccountedAt) throw conflict("Run cannot accept a new usage recorder");
if (receiptIdentity(run.usageJson) !== receiptIdentity(beforeHandoff?.usageJson))
throw conflict("Usage source changed during receipt recovery; retry recorder handoff");
for (const item of pending) await persistUsageReceiptInTransaction(tx, item.envelope);
await queueBudgetEvaluation(tx, { companyId: input.companyId, agentId: run.agentId,
projectId: typeof (run.usageJson?.ledgerScope as Record<string, unknown> | undefined)?.projectId === "string"
? (run.usageJson!.ledgerScope as Record<string, string>).projectId : null });
const [current] = await tx.select({ usageJson: heartbeatRuns.usageJson }).from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
const old = current.usageJson ?? {};
const fences = { ...(old.accountingReceiptSourceFences as Record<string, number> | undefined) };
if (typeof old.accountingReceiptSourceId === "string") fences[old.accountingReceiptSourceId] = Number(old.accountingReceiptSequence ?? 0);
return tx.update(heartbeatRuns).set({ usageJson: { ...old, accountingReceiptSourceFences: fences,
accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false } })
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), isNull(heartbeatRuns.costAccountedAt))).returning({ id: heartbeatRuns.id, runtimeMode: heartbeatRuns.runtimeMode, usageJson: heartbeatRuns.usageJson });
});
for (const item of pending) await fs.rm(item.file, { force: true });
if (bound.length !== 1) throw conflict("Run cannot accept a new usage recorder");
let sequence = 0;
let currentAttempt = "default";
@@ -282,7 +480,15 @@ export async function createRunUsageRecorder(db: Db, input: { companyId: string;
let finishFailedCapture: (() => void) | undefined;
async function persistFailure() {
if (!captureFailed) return;
const persisted = await withAccountingTransaction(db, input.companyId, async tx => {
const persisted = await withOperationAccountingTransaction(db, input.companyId, async tx => {
const [run] = await tx.select({ agentId: heartbeatRuns.agentId, costAccountedAt: heartbeatRuns.costAccountedAt,
usageJson: heartbeatRuns.usageJson }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))).for("update");
if (!run || run.costAccountedAt || run.usageJson?.accountingReceiptSourceId !== sourceId) return false;
if (run.usageJson?.accountingCaptureFailed === true) return true;
await queueBudgetEvaluation(tx, { companyId: input.companyId, agentId: run.agentId,
projectId: typeof (run.usageJson?.ledgerScope as Record<string, unknown> | undefined)?.projectId === "string"
? (run.usageJson!.ledgerScope as Record<string, string>).projectId : null });
const rows = await tx.update(heartbeatRuns).set({
costAccountingPending: true, accountingLastError: "usage_capture_failed",
usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || '{"accountingCaptureFailed":true,"accountingReceiptReady":false}'::jsonb`,
@@ -407,7 +613,10 @@ export async function readAcceptedRunUsageCheckpoint(db: Db, input: { companyId:
const producerAccountingProjectionFields = [
"accountingReceiptOwner", "accountingReceiptSourceId", "accountingReceiptSequence", "accountingReceiptReady",
"accountingReceiptId", "accountingReceiptReceivedAt", "accountingCaptureFailed",
"accountingReceiptId", "accountingReceiptReceivedAt", "accountingCaptureFailed", "accountingReceiptSourceFences",
"accountingSourceReceiptPending", "accountingLateReceiptPending", "accountingLateReceiptId",
"accountingLateReceiptAttemptedId", "accountingReconciledReceiptId", "accountingLateReceiptDebts",
"accountingReconciledReceiptSourceId", "accountingReconciledReceiptSequence",
"inputTokens", "outputTokens", "cachedInputTokens", "cacheWriteTokens",
"provider", "biller", "model", "billingType", "usageSource", "providerRequestId", "usageByModel",
"costUsd", "costUsdExact", "cacheAdjustedCostUsd", "costStatus", "pricingProvenance", "budgetExposure",