mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
fix(native): settle a crashed controller's running workspace finalize once a later attempt finishes (#19807)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run saves its workspace in a `workspace_finalize` operation after the agent gives a result > - A crash loop can kill the server after it records this operation as `running` and before it finishes > - The next server resumes finalization and records a new attempt, which succeeds > - Nothing settles the first attempt, so it stays `running` forever and looks like live work > - This pull request settles the old attempt when a later attempt for the same run finishes, under the proof that the old controller is gone > - The benefit is that no workspace operation stays `running` after its owner is dead ## Linked Issues or Issue Description Refs #19666, #19759, #19721, #19717 **What happened?** A control-plane restart chaos test sent SIGKILL to the server while a native run finalized its workspace. The server crash-looped. One boot recorded a `workspace_finalize` operation as `running` and then died. A later boot resumed finalization. Its new operation (with `resumedFromOperationId` set to the old one) succeeded, and the run succeeded. The old operation stayed `running` at rest. **Expected behavior** When a later finalization attempt for the same run finishes, and the owner of the earlier attempt is proven gone, the earlier attempt gets a terminal status with a reason. A live owner keeps its attempt. **Steps to reproduce** 1. Start a native run that finalizes a workspace. 2. Kill the server with SIGKILL after it inserts the `workspace_finalize` row, and before it updates the row. 3. Restart the server and let it finalize the run again. 4. See that the first `workspace_finalize` row stays `running`. **Paperclip version or commit** `ef83d566ca` (also on `27170dca49`). ## What Changed - Root cause: `attemptNativeWorkspaceFinalization` in `server/src/services/native-runtime/native-workspace-finalizer.ts` reads the previous attempt only to set `resumedFromOperationId`. It records the new attempt but never settles the previous `running` attempt. The early "already succeeded" return path also left it alone. Only the recorder that started a row can settle it (`workspace-operations.ts` `recordOperation`), and a killed process never does. - Add `settleSupersededNativeWorkspaceFinalizations` in `native-workspace-finalization-supersession.ts`. It sets earlier `running` native finalize attempts for the run to `failed`. It writes `metadata.supersededFinalization` (reason `superseded_workspace_finalization`, superseding operation id and status), an stderr note, a terminal-workspace wake (registered before the write), and a `workspace_operation.reconciled` activity entry. - The proof is the run's workspace-finalization ownership, not age. The caller holds the advisory lock and the durable owner receipt. That claim requires the previous controller to have released it or to be proven stopped (same-host process proof, platform proof, or a completed copyback barrier). The function also skips an attempt that this process is still running, and attempts with independent obligations (`uncertain`, `saveWork`). It only touches rows that `native_workspace_finalizer` recorded and that are older than the superseding attempt. - The finalizer calls it after a new attempt reaches a terminal status, and on the "already succeeded" path. A settlement error is logged and does not change the finalization result. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-workspace-finalization-supersession.test.ts src/services/native-runtime/native-workspace-finalizer.test.ts` on embedded Postgres: 72 passed. - New tests in `native-workspace-finalization-supersession.test.ts` (a separate file, so `native-workspace-finalizer.test.ts` stays under 1,000 lines): - A `running` finalize from a dead controller plus a resumed attempt that succeeds: the old row becomes `failed` with the supersession reason, no `running` rows remain, and one activity entry exists. - The resumed attempt fails: the old row is still settled, and `supersedingStatus` is `failed`. - A `running` finalize older than an already succeeded attempt: the old row is settled on the next sweep. - While another controller holds ownership, the `running` row stays `running`. - A `running` finalize recorded by another service stays `running`. - Attempts with `saveWork` or `uncertain: true` stay `running`. - Red/green: with the settle call disabled, the three settle tests fail (row stays `running`). With the fix, all pass. - `npx tsc --noEmit -p server` passes. ## Risks - Low. The change only writes rows that are already proven abandoned, and only under the finalization ownership that already fences copyback. - A settled attempt now reads as `failed`. Readers pick the latest attempt by creation time, and the superseding attempt is always later, so the run's finalization outcome does not change. - Existing leaked rows on runs that never finalize again stay `running`. This PR does not add a backfill. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit) and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
e801be7aa9
commit
1674902b4f
3 files changed
+261
-2
No files matched your search
+148
@@ -0,0 +1,148 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import {
|
||||
activityLog,
|
||||
agents,
|
||||
companies,
|
||||
completionContracts,
|
||||
createDb,
|
||||
heartbeatRuns,
|
||||
issues,
|
||||
nativeRunFinalizations,
|
||||
nativeRunResults,
|
||||
workspaceOperations,
|
||||
} from "@paperclipai/db";
|
||||
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js";
|
||||
import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.js";
|
||||
|
||||
// RC-64: a crash loop killed the controller after it recorded a workspace
|
||||
// finalize attempt as `running`. A later owner finalized the same run, and the
|
||||
// orphaned attempt stayed `running` forever.
|
||||
describe("superseded native workspace finalization attempts", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let workspaceRoot: string;
|
||||
let priorLogRoot: string | undefined;
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
beforeAll(async () => {
|
||||
workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-finalize-supersession-"));
|
||||
priorLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = path.join(workspaceRoot, "operation-logs");
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_WORKSPACE_FINALIZER_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-finalize-supersession-");
|
||||
db = createDb(external ?? temporary!.connectionString, { maxConnections: 6 });
|
||||
await db.insert(companies).values({ id: companyId, name: "Finalize supersession", issuePrefix: "FSS" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Finalize agent",
|
||||
adapterType: "paperclip_runner", status: "running" });
|
||||
}, 30_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await temporary?.cleanup();
|
||||
await fs.rm(workspaceRoot, { recursive: true, force: true });
|
||||
if (priorLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorLogRoot;
|
||||
});
|
||||
|
||||
/** A succeeded native run with an accepted result and a dead controller's running finalize. */
|
||||
async function seedOrphanedFinalize(options: { cwdExists?: boolean; metadata?: Record<string, unknown> } = {}) {
|
||||
const issueId = randomUUID();
|
||||
const contractId = randomUUID();
|
||||
const runId = randomUUID();
|
||||
const resultId = randomUUID();
|
||||
const cwd = path.join(workspaceRoot, `orphan-${randomUUID()}`);
|
||||
if (options.cwdExists !== false) await fs.mkdir(cwd);
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Crash-loop finalize", status: "in_progress",
|
||||
workMode: "standard", assigneeAgentId: agentId });
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "finalize-supersession-test-v1",
|
||||
risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { objective: "finalize" }, canonicalSha256: `contract-${contractId}`,
|
||||
createdByActorType: "system", createdByActorId: "test" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded", runtimeMode: "native",
|
||||
nativeIssueId: issueId, completionContractId: contractId,
|
||||
runnerProfileJson: { nativeExecutionInput: { binding: { executionWorkspaceId: randomUUID() } } } });
|
||||
await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId,
|
||||
serverFingerprint: `result-${resultId}`, schemaStatus: "accepted", resultJson: {}, canonicalSha256: `result-${resultId}` });
|
||||
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "result_accepted", resultId });
|
||||
const startedAt = new Date(Date.now() - 60_000);
|
||||
const [orphan] = await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: runId, issueId,
|
||||
phase: "workspace_finalize", status: "running", cwd, startedAt, createdAt: startedAt, updatedAt: startedAt,
|
||||
metadata: { owningService: "native_workspace_finalizer", observation: "workspace_directory",
|
||||
resumedFromOperationId: null, ...options.metadata },
|
||||
}).returning();
|
||||
return { issueId, runId, orphan };
|
||||
}
|
||||
|
||||
const read = async (id: string) =>
|
||||
(await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, id)))[0]!;
|
||||
|
||||
it("settles the orphan once a resumed attempt succeeds", async () => {
|
||||
const seeded = await seedOrphanedFinalize();
|
||||
const operation = await resumeNativeWorkspaceFinalization({ db, runId: seeded.runId });
|
||||
expect(operation).toMatchObject({ status: "succeeded",
|
||||
metadata: expect.objectContaining({ resumedFromOperationId: seeded.orphan.id }) });
|
||||
const orphan = await read(seeded.orphan.id);
|
||||
expect(orphan).toMatchObject({ status: "failed", finishedAt: expect.any(Date) });
|
||||
expect(orphan.metadata).toMatchObject({ owningService: "native_workspace_finalizer",
|
||||
supersededFinalization: { reason: "superseded_workspace_finalization", supersedingOperationId: operation!.id,
|
||||
supersedingStatus: "succeeded", previousStatus: "running" } });
|
||||
expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, seeded.runId),
|
||||
eq(workspaceOperations.status, "running")))).toHaveLength(0);
|
||||
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, seeded.orphan.id),
|
||||
eq(activityLog.action, "workspace_operation.reconciled")))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("settles the orphan when the resumed attempt itself fails", async () => {
|
||||
const seeded = await seedOrphanedFinalize({ cwdExists: false });
|
||||
const operation = await resumeNativeWorkspaceFinalization({ db, runId: seeded.runId });
|
||||
expect(operation?.status).toBe("failed");
|
||||
const orphan = await read(seeded.orphan.id);
|
||||
expect(orphan.status).toBe("failed");
|
||||
expect(orphan.metadata).toMatchObject({ supersededFinalization: {
|
||||
supersedingOperationId: operation!.id, supersedingStatus: "failed" } });
|
||||
});
|
||||
|
||||
it("settles an orphan older than an already succeeded attempt", async () => {
|
||||
const seeded = await seedOrphanedFinalize();
|
||||
const [success] = await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: seeded.runId,
|
||||
issueId: seeded.issueId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(),
|
||||
metadata: { owningService: "native_workspace_finalizer", resumedFromOperationId: seeded.orphan.id } }).returning();
|
||||
const operation = await resumeNativeWorkspaceFinalization({ db, runId: seeded.runId });
|
||||
expect(operation?.id).toBe(success.id);
|
||||
const orphan = await read(seeded.orphan.id);
|
||||
expect(orphan.status).toBe("failed");
|
||||
expect(orphan.metadata).toMatchObject({ supersededFinalization: { supersedingOperationId: success.id } });
|
||||
});
|
||||
|
||||
it("leaves a running attempt alone while another controller owns the run", async () => {
|
||||
const seeded = await seedOrphanedFinalize();
|
||||
const result = await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId },
|
||||
() => resumeNativeWorkspaceFinalization({ db, runId: seeded.runId }));
|
||||
expect(result).toEqual({ acquired: true, value: null });
|
||||
expect((await read(seeded.orphan.id)).status).toBe("running");
|
||||
});
|
||||
|
||||
it("does not settle a running finalize recorded by another service", async () => {
|
||||
const seeded = await seedOrphanedFinalize({ metadata: { owningService: "heartbeat" } });
|
||||
expect((await resumeNativeWorkspaceFinalization({ db, runId: seeded.runId }))?.status).toBe("succeeded");
|
||||
expect((await read(seeded.orphan.id)).status).toBe("running");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["saveWork", { saveWork: { requested: true } }],
|
||||
["uncertain", { uncertain: true }],
|
||||
])("keeps an attempt with an independent %s obligation running", async (_name, metadata) => {
|
||||
const seeded = await seedOrphanedFinalize({ metadata });
|
||||
expect((await resumeNativeWorkspaceFinalization({ db, runId: seeded.runId }))?.status).toBe("succeeded");
|
||||
expect((await read(seeded.orphan.id)).status).toBe("running");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,91 @@
|
||||
import { and, asc, eq, ne, sql } from "drizzle-orm";
|
||||
import { workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { persistActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
|
||||
import { isNativeWorkspaceFinalizationOperationActive } from "../workspace-operations.js";
|
||||
import { notifyTerminalWorkspaceWork } from "../terminal-workspace-work.js";
|
||||
|
||||
export const SUPERSEDED_NATIVE_FINALIZATION_REASON = "superseded_workspace_finalization";
|
||||
|
||||
/**
|
||||
* Settle earlier `running` native workspace-finalization operations for one run
|
||||
* once a later attempt for that run has reached a terminal status (RC-64).
|
||||
*
|
||||
* The caller must hold the run's native workspace-finalization ownership. That
|
||||
* ownership is the proof that no other controller is executing an export for
|
||||
* this run: it holds the advisory lock and has claimed the durable owner receipt,
|
||||
* which requires the previous controller to have released it or to have been
|
||||
* proven stopped (same-host process proof, platform proof, or a completed
|
||||
* copyback barrier). The only remaining live executor could be this process,
|
||||
* which the in-process registry rules out. Age is never used.
|
||||
*/
|
||||
export async function settleSupersededNativeWorkspaceFinalizations(db: Db, input: {
|
||||
companyId: string;
|
||||
runId: string;
|
||||
issueId: string;
|
||||
superseding: { id: string; status: string };
|
||||
}): Promise<string[]> {
|
||||
if (input.superseding.status === "running") return [];
|
||||
const candidates = await db.select().from(workspaceOperations).where(and(
|
||||
eq(workspaceOperations.companyId, input.companyId),
|
||||
eq(workspaceOperations.heartbeatRunId, input.runId),
|
||||
eq(workspaceOperations.issueId, input.issueId),
|
||||
eq(workspaceOperations.phase, "workspace_finalize"),
|
||||
eq(workspaceOperations.status, "running"),
|
||||
ne(workspaceOperations.id, input.superseding.id),
|
||||
sql`${workspaceOperations.metadata}->>'owningService' = 'native_workspace_finalizer'`,
|
||||
// Keep PostgreSQL timestamp precision when proving which attempt is later.
|
||||
sql`${workspaceOperations.createdAt} < (select created_at from workspace_operations where id = ${input.superseding.id}::uuid)`,
|
||||
)).orderBy(asc(workspaceOperations.createdAt), asc(workspaceOperations.id));
|
||||
|
||||
const settled: string[] = [];
|
||||
for (const candidate of candidates) {
|
||||
if (isNativeWorkspaceFinalizationOperationActive({
|
||||
operationId: candidate.id, companyId: input.companyId, runId: input.runId, issueId: input.issueId,
|
||||
})) continue;
|
||||
const metadata = candidate.metadata ?? {};
|
||||
// Independent obligations are never retired by a later export.
|
||||
if (metadata.uncertain === true || metadata.uncertain === "true" || metadata.saveWork != null) continue;
|
||||
const now = new Date();
|
||||
const provenance = {
|
||||
schema: "paperclip.native-workspace-finalization-supersession.v1",
|
||||
reason: SUPERSEDED_NATIVE_FINALIZATION_REASON,
|
||||
proof: "native_workspace_finalization_ownership",
|
||||
previousStatus: candidate.status,
|
||||
supersedingOperationId: input.superseding.id,
|
||||
supersedingStatus: input.superseding.status,
|
||||
runId: input.runId,
|
||||
reconciledAt: now.toISOString(),
|
||||
};
|
||||
const message = "Workspace finalization was interrupted before it finished; "
|
||||
+ `a later finalization attempt (${input.superseding.id}) ${input.superseding.status}.\n`;
|
||||
let publication: ActivityPublication | undefined;
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
// Register the wake intent before the write so drain and sleep checks see it.
|
||||
await notifyTerminalWorkspaceWork(tx, candidate.executionWorkspaceId
|
||||
? `workspace:${candidate.executionWorkspaceId}` : `issue:${input.issueId}`);
|
||||
const [row] = await tx.update(workspaceOperations).set({
|
||||
status: "failed",
|
||||
finishedAt: now,
|
||||
updatedAt: now,
|
||||
stderrExcerpt: `${candidate.stderrExcerpt ?? ""}${message}`.slice(-4096),
|
||||
metadata: { ...metadata, supersededFinalization: provenance },
|
||||
}).where(and(
|
||||
eq(workspaceOperations.id, candidate.id),
|
||||
eq(workspaceOperations.companyId, input.companyId),
|
||||
eq(workspaceOperations.status, "running"),
|
||||
)).returning({ id: workspaceOperations.id });
|
||||
if (!row) return null;
|
||||
const activity = await persistActivity(tx as unknown as Db, {
|
||||
companyId: input.companyId, actorType: "system", actorId: "native-workspace-finalizer",
|
||||
action: "workspace_operation.reconciled", entityType: "workspace_operation", entityId: candidate.id,
|
||||
runId: input.runId, issueId: input.issueId, details: provenance,
|
||||
});
|
||||
publication = activity.publication;
|
||||
return row;
|
||||
});
|
||||
if (!updated) continue;
|
||||
if (publication) publishActivity(publication);
|
||||
settled.push(candidate.id);
|
||||
}
|
||||
return settled;
|
||||
}
|
||||
@@ -5,7 +5,9 @@ import { recordNativeFinalizationFailure } from "./native-run-finalizer.js";
|
||||
import { inspectPluginWorkspaceForFinalization } from "./native-plugin-workspace-finalization.js";
|
||||
import type { WorkspaceWorkers } from "../plugin-workspaces.js";
|
||||
import { restoreNativeWorkspaceBestEffort } from "./native-workspace-best-effort.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership, type NativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js";
|
||||
import { settleSupersededNativeWorkspaceFinalizations } from "./native-workspace-finalization-supersession.js";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import fs from "node:fs/promises";
|
||||
import { and, desc, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
@@ -83,6 +85,20 @@ export async function attemptNativeWorkspaceFinalization(input: {
|
||||
throw new Error("native_workspace_finalization_binding_missing");
|
||||
}
|
||||
|
||||
// An attempt interrupted by a controller crash stays `running`; once this
|
||||
// owner records a terminal attempt, settle the orphan under the same proof.
|
||||
const settleSuperseded = async (ownership: NativeWorkspaceFinalizationOwnership,
|
||||
superseding: { id: string; status: string }) => {
|
||||
try {
|
||||
await ownership.assertHeld();
|
||||
await settleSupersededNativeWorkspaceFinalizations(input.db, {
|
||||
companyId: bound.companyId, runId: input.runId, issueId: bound.issueId, superseding,
|
||||
});
|
||||
} catch (error) {
|
||||
// Settlement is bookkeeping; it must not change the finalization outcome.
|
||||
logger.warn({ runId: input.runId, error }, "Superseded workspace finalization could not be settled");
|
||||
}
|
||||
};
|
||||
const owned = await withNativeWorkspaceFinalizationOwnership({
|
||||
db: input.db, companyId: bound.companyId, runId: input.runId,
|
||||
}, async (ownership) => {
|
||||
@@ -113,7 +129,10 @@ export async function attemptNativeWorkspaceFinalization(input: {
|
||||
eq(workspaceOperations.status, "succeeded"),
|
||||
)).orderBy(desc(workspaceOperations.createdAt)).limit(1).then((rows) => rows[0] ?? null);
|
||||
// A stale failure from a pre-fencing controller cannot invalidate exported work.
|
||||
if (successful) return { kind: "completed", operation: successful } as const;
|
||||
if (successful) {
|
||||
await settleSuperseded(ownership, successful);
|
||||
return { kind: "completed", operation: successful } as const;
|
||||
}
|
||||
const previous = await input.db.select().from(workspaceOperations).where(and(
|
||||
eq(workspaceOperations.companyId, bound.companyId),
|
||||
eq(workspaceOperations.heartbeatRunId, input.runId),
|
||||
@@ -345,6 +364,7 @@ export async function attemptNativeWorkspaceFinalization(input: {
|
||||
};
|
||||
},
|
||||
});
|
||||
await settleSuperseded(ownership, operation);
|
||||
return { kind: "completed", operation } as const;
|
||||
});
|
||||
return owned.acquired ? owned.value : { kind: "waiting", reason: owned.reason } as const;
|
||||
|
||||
Reference in new issue
Block a user