fix(web): bound cron reads and make facts/release writes honest

Audit 0.10.1 js-ops, web (W02):

- W02-07 (Refs #6557): when facts_current has no head (revoked, expired
  or future-dated), resolveCloudFacts retires the KV last-good copy, so a
  later Supabase outage cannot resurrect a revoked envelope through the
  stale path. The last-good TTL drops from 30 days to 6 h, the same
  staleness bound every client already applies (payload ttl_secs).
- W02-10 (Refs #6559): facts-publish revoke flips only a `published`
  row, so a re-run keeps the first revocation's time and reason and says
  already_revoked; a version that matches nothing is an error, never a
  silent `revoked: 0`; the receipt says whether the head was revoked.
  publish proves the inserted row is exactly this envelope and treats a
  replay of the same bytes (lost reply) as already_published instead of
  a bare HTTP 409. Both are exported with an injectable request.
- W02-09 (Refs #6555): sync-latest-release --check fails when GitHub
  cannot be read ("could not look" is not "current"); a write validates
  all three files first, then replaces each through a unique temp file
  and rename, so an unreadable mirror no longer leaves the first file
  moved alone.
- W02-06 (Refs #6561): fetchBoundedText (bounded-body.ts) puts a
  deadline and byte cap on cron reads of other hosts; facts-drift and
  content-watch use it, and link probes carry a 10 s deadline and
  discard GET bodies.
- W02-05 (Refs #6556): logUsage writes one append-only record per model
  call instead of read-modify-writing a shared daily counter that
  overlapping cron tasks lost calls from (web/AGENT.md updated).
  runFactsDrift never replaces a snapshot from a newer source commit with
  an older one.

Validation (web, vitest 4.1.11):
- vitest run cloud-facts, facts-drift, content-watch, bounded-fetch,
  community-agent, sync-latest-release, bounded-form,
  security-boundary-behavior, public-api-security, check-facts, facts,
  community-agent-security: Test Files 12 passed, Tests 103 passed.
- tsc --noEmit -p web: exit 0; eslint on touched files: exit 0;
  node scripts/check-cloud-facts.mjs: OK.
- Negative controls (fix reverted, new test run, restored identical):
  cloud-facts.ts -> "never resurrects a revoked head": expected ok to
    match unavailable (1 failed)
  facts-publish revoke without the status filter -> "revokes once":
    revoked 1 where already_revoked expected (1 failed)
  sync-latest-release.mjs -> both new tests fail (check exits 0 on 503;
    target moved to v0.10.1 while the mirror was unusable)
  facts-drift.ts + content-watch.ts -> "every cron read": 15 reads
    without a signal (1 failed)
  community-agent.ts -> "keeps every call": expected 1 to be 3
  facts-drift guard disabled -> "never replaces a newer snapshot":
    changed true (1 failed)

Remaining (recorded, not changed): community-agent-tasks GitHub reads
and the paid agentChat call have no deadline yet; generator dedup
(hasFreshDraft -> model -> saveDraft) can double-spend when cron runs
overlap, and a posted draft is still guarded by the reviewed-body hash.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: CodeWhale Bot <[email protected]>
This commit is contained in:
CodeWhale BotandClaude Opus 5.5 committed 2026-09-30 00:44:42 -07:00
1 parent 332f60beef
commit ebab81f5ad
14 files changed
+377 -64

No files matched your search

+5 -3
View File
@@ -26,8 +26,8 @@ suffix over the finding's full identity, max 80 chars — so unchanged findings
dedup and changed findings land as new drafts. Semantic-drift model output is
validated and capped (10 drafts/run) before any KV writes.
Usage logged to:
usage:<YYYY-MM-DD>
Usage logged to (one record per model call; sum the day's prefix):
usage:<YYYY-MM-DD>:<timestamp>:<uuid>
```
## Cron schedule
@@ -57,7 +57,9 @@ All drafts follow these rules:
- Each cron invocation caps at ~30k input tokens and ~2k output tokens.
- Issue/PR bodies are truncated to 1000–4000 chars before sending to the model.
- Deduplication: `hasFreshDraft` checks if a draft already exists that's newer than the item's `updated_at`. Skips if so.
- Token usage is logged to `usage:<YYYY-MM-DD>` KV keys (retained 90 days).
- Token usage is logged as one `usage:<YYYY-MM-DD>:…` KV record per model call
(retained 90 days); list the day's prefix and sum `calls`/`inputTokens`/
`outputTokens`. Records are append-only because KV has no atomic increment.
- If `DEEPSEEK_API_KEY` is missing or the API errors, the cron returns 200 with `{ skipped: true, reason }` — never crashes, never retry-loops.
## Maintainer review surface
+32
View File
@@ -43,3 +43,35 @@ export async function readBoundedBody(request: { body: ReadableStream<Uint8Array
}
return bytes;
}
/** Budget for background (cron) reads of other hosts. */
export interface OutboundBudget {
/** Deadline for the whole exchange, body included. */
timeoutMs?: number;
/** Largest body read; a larger one is an error, never a silent truncation. */
maxBytes?: number;
}
export const OUTBOUND_TIMEOUT_MS = 15_000;
export const OUTBOUND_MAX_BYTES = 2 * 1024 * 1024;
/**
* One outbound GET-style read under a deadline and a byte cap. A slow or
* endless peer cannot hold a cron invocation until the platform kills it, and
* a huge body cannot exhaust isolate memory. Non-2xx answers return
* `{ ok: false }` with the body discarded; transport, timeout and size
* failures throw, so callers keep their own fallbacks.
*/
export async function fetchBoundedText(
url: string,
init: RequestInit = {},
{ timeoutMs = OUTBOUND_TIMEOUT_MS, maxBytes = OUTBOUND_MAX_BYTES }: OutboundBudget = {},
): Promise<{ ok: boolean; status: number; text: string }> {
const response = await fetch(url, { ...init, signal: AbortSignal.timeout(timeoutMs) });
if (!response.ok) {
try { await response.body?.cancel(); } catch { /* The status is the answer. */ }
return { ok: false, status: response.status, text: "" };
}
const bytes = await readBoundedBody(response, maxBytes);
return { ok: true, status: response.status, text: new TextDecoder().decode(bytes) };
}
+40
View File
@@ -0,0 +1,40 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { fetchBoundedText } from "./bounded-body";
import { runLinkCheck } from "./content-watch";
import { deriveFactsFromRemote } from "./facts-drift";
afterEach(() => vi.unstubAllGlobals());
/** A peer that accepts the request and never answers until aborted. */
function hangingFetch() {
return vi.fn((_: unknown, init?: RequestInit) => new Promise<Response>((_, reject) => {
init?.signal?.addEventListener("abort", () => reject(init.signal!.reason), { once: true });
}));
}
describe("bounded outbound reads", () => {
it("gives up on a silent peer at the deadline and refuses an oversized body", async () => {
vi.stubGlobal("fetch", hangingFetch());
const started = Date.now();
await expect(fetchBoundedText("https://slow.test", {}, { timeoutMs: 50 })).rejects.toThrow();
expect(Date.now() - started).toBeLessThan(2_000);
vi.stubGlobal("fetch", vi.fn(async () => new Response("x".repeat(64))));
await expect(fetchBoundedText("https://big.test", {}, { maxBytes: 16 })).rejects.toThrow("payload too large");
vi.stubGlobal("fetch", vi.fn(async () => new Response("gone", { status: 404 })));
expect(await fetchBoundedText("https://missing.test")).toEqual({ ok: false, status: 404, text: "" });
});
it("puts every cron read of other hosts under a deadline", async () => {
// Callers swallow transport errors, so record rather than assert inside.
const seen = vi.fn(async () => new Response("{}", { status: 500 }));
vi.stubGlobal("fetch", seen);
await deriveFactsFromRemote();
await runLinkCheck({ CURATED_KV: { get: async () => null, put: async () => undefined } } as never);
expect(seen.mock.calls.length).toBeGreaterThan(1);
const unbounded = seen.mock.calls
.filter((call) => !((call as unknown[])[1] as RequestInit | undefined)?.signal)
.map((call) => String((call as unknown[])[0]));
expect(unbounded).toEqual([]);
});
});
+57 -1
View File
@@ -11,7 +11,7 @@ import {
} from "./cloud-facts";
import { TRUSTED_KEYS, type TrustedKey } from "./cloud-facts/keys";
import { GET, HEAD } from "../app/api/facts/v1/[channel]/route";
import { activePublishingKey, emitSql, readBoundedFile, readBoundedResponse, verifyEnvelope as verifyForPublisher } from "../scripts/facts-publish.mjs";
import { activePublishingKey, emitSql, publishRelease, readBoundedFile, readBoundedResponse, revokeRelease, verifyEnvelope as verifyForPublisher } from "../scripts/facts-publish.mjs";
import { parseRustKeys, parseTsKeys } from "../scripts/check-cloud-facts.mjs";
const fixturePath = new URL("../../docs/cloud-facts/fixtures/envelope-stable-v7.json", import.meta.url);
@@ -225,6 +225,21 @@ describe("cloud facts transport", () => {
expect((await resolveCloudFacts("stable", { ...env, CURATED_KV: kv }, { keys: [EPHEMERAL_KEY], now: () => NOW, fetchImpl: failing })).kind).toBe("unavailable");
});
it("never resurrects a revoked head through the stale cache after an outage", async () => {
const kv = new MemKV();
const ttls: unknown[] = [];
const put = kv.put.bind(kv);
kv.put = async (key: string, value: string, options?: { expirationTtl?: number }) => { ttls.push(options?.expirationTtl); await put(key, value); };
expect((await resolveCloudFacts("stable", { ...env, CURATED_KV: kv }, { ...opts, fetchImpl: supabaseFetch([rowFrom(fixture)]) })).kind).toBe("ok");
// The operator revokes the head: facts_current returns no row.
expect((await resolveCloudFacts("stable", { ...env, CURATED_KV: kv }, { ...opts, fetchImpl: supabaseFetch([]) })).kind).toBe("none");
// Supabase then goes down. The revoked envelope must not come back.
expect(await resolveCloudFacts("stable", { ...env, CURATED_KV: kv }, { ...opts, fetchImpl: failing })).toMatchObject({ kind: "unavailable" });
// The bridge lasts no longer than a client's own 6 h staleness bound.
expect(ttls.length).toBeGreaterThan(0);
expect(ttls.every((ttl) => ttl === 6 * 60 * 60)).toBe(true);
});
it("caps and cancels PostgREST streaming bodies before JSON parsing", async () => {
const oversized = overflowingStream();
const fetchImpl = vi.fn(async () => new Response(oversized.stream));
@@ -301,6 +316,47 @@ describe("facts response protocol", () => {
describe("facts publisher boundaries", () => {
const script = fileURLToPath(new URL("../scripts/facts-publish.mjs", import.meta.url));
/** A PostgREST double over one channel and its releases. */
function fakePostgrest(head: number, releases: { facts_version: number; status: string; payload_sha256: string; revoked_at?: string; revoke_reason?: string }[]) {
const calls: string[] = [];
const request = async (path: string, init: { method?: string; body?: Record<string, unknown> } = {}) => {
calls.push(`${init.method ?? "GET"} ${path}`);
if (path.startsWith("facts_channel")) return [{ id: "c1", max_facts_version: head }];
if (path === "facts_key") return null;
if (path === "facts_release") {
if (Number(init.body?.facts_version) <= head) throw Object.assign(new Error("PostgREST request failed (HTTP 409)"), { status: 409 });
head = Number(init.body?.facts_version);
const stored = { id: "r-new", facts_version: head, status: "published", payload_sha256: fixture.sha256 };
releases.push(stored);
return [stored];
}
const version = Number(/facts_version=eq\.(\d+)/.exec(path)?.[1]);
const matches = releases.filter((r) => r.facts_version === version && (!path.includes("status=eq.published") || r.status === "published"));
if (init.method === "PATCH") for (const r of matches) Object.assign(r, init.body);
return matches.map((r) => ({ id: `r-${r.facts_version}`, ...r }));
};
return { request, calls, releases };
}
it("revokes once, keeps the first revocation on a re-run, and refuses an unknown version", async () => {
const pg = fakePostgrest(3, [{ facts_version: 3, status: "published", payload_sha256: "a" }]);
expect(await revokeRelease(pg.request, { channel: "stable", version: 3, reason: "bad price", at: "2026-09-30T00:00:00Z" }))
.toMatchObject({ revoked: 1, already_revoked: false, head: true });
const again = await revokeRelease(pg.request, { channel: "stable", version: 3, reason: "second run", at: "2026-10-01T00:00:00Z" });
expect(again).toMatchObject({ revoked: 0, already_revoked: true, revoked_at: "2026-09-30T00:00:00Z", revoke_reason: "bad price" });
expect(pg.releases[0]).toMatchObject({ revoked_at: "2026-09-30T00:00:00Z", revoke_reason: "bad price" });
await expect(revokeRelease(pg.request, { channel: "stable", version: 9, reason: "typo", at: "2026-10-01T00:00:00Z" })).rejects.toThrow("no published facts_version 9");
});
it("publishes exactly the envelope and treats a replay of the same bytes as already published", async () => {
const pg = fakePostgrest(fixture.facts_version - 1, []);
const row = { facts_version: fixture.facts_version };
expect(await publishRelease(pg.request, fixture, row, "pub")).toMatchObject({ published: true, payload_sha256: fixture.sha256 });
expect(await publishRelease(pg.request, fixture, row, "pub")).toMatchObject({ published: false, already_published: true, release_id: "r-new" });
const other = { ...fixture, sha256: "f".repeat(64) };
await expect(publishRelease(pg.request, other, row, "pub")).rejects.toThrow("high-water mark");
});
it("requires an active pinned primary key and cannot publish with an explicit fixture public key", () => {
expect(() => activePublishingKey(fixture, [])).toThrow("not pinned and active");
expect(() => activePublishingKey(fixture, [{ ...TEST_KEY, status: "retired" }])).toThrow("not pinned and active");
+18 -2
View File
@@ -8,7 +8,11 @@ export const CHANNEL_RE = /^[a-z0-9][a-z0-9-]{0,31}$/;
export const KV_PREFIX = "facts:cloud:";
export const SUPABASE_TIMEOUT_MS = 3000;
export const MAX_ENVELOPE_BYTES = 768 * 1024;
const KV_TTL_SECS = 60 * 60 * 24 * 30;
// The last-good copy only bridges a transport outage. It lives no longer than
// a client keeps applying its own cached envelope (payload ttl_secs, 6 h by
// default, docs/CLOUD_FACTS.md), so an outage cannot extend a withdrawn
// release beyond the staleness every client already accepts.
const KV_TTL_SECS = 60 * 60 * 6;
const KEY_ID_RE = /^cwf-[a-z0-9-]{1,32}$/;
const VERSION_REQ_RE = /^(\*|(?:>=|<=|>|<|=|\^|~)?\s*\d+(\.\d+){0,2}(-[0-9A-Za-z.-]+)?(\s*,\s*(?:>=|<=|>|<|=|\^|~)?\s*\d+(\.\d+){0,2}(-[0-9A-Za-z.-]+)?)*)$/;
const MAX_SIGNATURES = 7; // Plus the primary signature: eight candidates total.
@@ -286,6 +290,13 @@ async function kvGet(env: CloudFactsEnv, channel: string): Promise<unknown> {
} catch { return null; }
}
async function retireLastGood(env: CloudFactsEnv, channel: string): Promise<void> {
if (!env.CURATED_KV || !isEnvelope(await kvGet(env, channel))) return;
try {
await env.CURATED_KV.put(`${KV_PREFIX}${channel}`, JSON.stringify({ retired: "no-current-head" }), { expirationTtl: KV_TTL_SECS });
} catch { /* Best effort, like the write; the copy still expires on its TTL. */ }
}
export async function resolveCloudFacts(channel: string, env: CloudFactsEnv, opts: ResolveOptions = {}): Promise<CloudFactsResult> {
if (!isValidChannel(channel)) return { kind: "none" };
const keys = opts.keys ?? TRUSTED_KEYS;
@@ -295,7 +306,12 @@ export async function resolveCloudFacts(channel: string, env: CloudFactsEnv, opt
let source: "supabase" | "kv-stale" = "supabase";
try {
const row = await fetchCurrentRow(channel, env, opts);
if (!row) return { kind: "none" };
if (!row) {
// The head was revoked, expired or future-dated. Retire the last-good
// copy so a later outage cannot resurrect it through the stale path.
await retireLastGood(env, channel);
return { kind: "none" };
}
envelope = envelopeFromRow(row);
} catch {
source = "kv-stale";
+18 -1
View File
@@ -1,5 +1,5 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { validateSession } from "./community-agent";
import { logUsage, validateSession } from "./community-agent";
describe("validateSession", () => {
let mockKV: {
@@ -81,3 +81,20 @@ describe("validateSession", () => {
expect(mockKV.get).toHaveBeenCalledWith(SESSION_PREFIX + sid);
});
});
describe("logUsage", () => {
it("keeps every call when cron tasks log at the same time", async () => {
// A KV whose reads resolve before either write lands, as overlapping
// Worker invocations see it.
const store = new Map<string, string>();
const kv = {
get: async (key: string) => { await new Promise((resolve) => setTimeout(resolve, 5)); return store.get(key) ?? null; },
put: async (key: string, value: string) => { await new Promise((resolve) => setTimeout(resolve, 5)); store.set(key, value); },
} as unknown as Parameters<typeof logUsage>[0];
await Promise.all([logUsage(kv, 100, 10), logUsage(kv, 200, 20), logUsage(kv, 300, 30)]);
const records = [...store.values()].map((value) => JSON.parse(value) as { calls: number; inputTokens: number });
expect(records.reduce((sum, record) => sum + record.calls, 0)).toBe(3);
expect(records.reduce((sum, record) => sum + record.inputTokens, 0)).toBe(600);
expect([...store.keys()].every((key) => key.startsWith(`usage:${new Date().toISOString().slice(0, 10)}`))).toBe(true);
});
});
+9 -10
View File
@@ -810,16 +810,15 @@ export async function logUsage(
outputTokens: number
): Promise<void> {
if (!kv) return;
const date = new Date().toISOString().slice(0, 10);
const key = `usage:${date}`;
const raw = await kv.get(key);
const existing: UsageLog = raw
? JSON.parse(raw)
: { date, calls: 0, inputTokens: 0, outputTokens: 0 };
existing.calls += 1;
existing.inputTokens += inputTokens;
existing.outputTokens += outputTokens;
await kv.put(key, JSON.stringify(existing), { expirationTtl: 60 * 60 * 24 * 90 }); // 90 days
// One record per model call under the day's prefix. KV has no atomic
// increment, so a shared daily counter that overlapping cron tasks read,
// bumped and wrote back lost calls; an append-only record cannot.
const at = new Date().toISOString();
const date = at.slice(0, 10);
const record: UsageLog = { date, calls: 1, inputTokens, outputTokens };
await kv.put(`usage:${date}:${at}:${crypto.randomUUID()}`, JSON.stringify(record), {
expirationTtl: 60 * 60 * 24 * 90, // 90 days
});
}
/**
+12 -6
View File
@@ -13,6 +13,7 @@
* Both surface as drafts in CURATED_KV under `draft:linkcheck:<...>` and
* `draft:semantic-drift:<...>`, picked up by the existing /admin listing.
*/
import { fetchBoundedText } from "./bounded-body";
import { agentChat, draftStorageKey, getDraft, saveDraft, type AgentDraft, type DeepSeekEnv, VOICE_CONSTRAINTS } from "./community-agent";
interface KVNamespace {
@@ -95,15 +96,20 @@ export interface LinkCheckResult {
ms: number;
}
const PROBE_TIMEOUT_MS = 10_000;
async function probe(target: { url: string; label: string }): Promise<LinkCheckResult> {
const start = Date.now();
try {
// Use HEAD where possible; fall back to GET on 405/403 since some hosts
// (e.g. Cloudflare-protected) reject HEAD.
let r = await fetch(target.url, { method: "HEAD", redirect: "follow" });
// Each request has its own deadline; only the status matters, so a GET
// body is discarded unread.
let r = await fetch(target.url, { method: "HEAD", redirect: "follow", signal: AbortSignal.timeout(PROBE_TIMEOUT_MS) });
if (r.status === 405 || r.status === 403 || r.status === 404) {
// Some sites return 404 to HEAD but 200 to GET (e.g. NPM)
r = await fetch(target.url, { method: "GET", redirect: "follow" });
r = await fetch(target.url, { method: "GET", redirect: "follow", signal: AbortSignal.timeout(PROBE_TIMEOUT_MS) });
await r.body?.cancel().catch(() => undefined);
}
return { url: target.url, label: target.label, status: r.status, ok: r.ok, ms: Date.now() - start };
} catch {
@@ -291,10 +297,10 @@ export async function runSemanticDrift(env: WatchEnv): Promise<{ ok: boolean; dr
// Fetch CHANGELOG (truncated), recent commits, and live homepage HTML.
const [changelog, commits, homepageHtml, docsHtml] = await Promise.all([
fetch("https://raw.githubusercontent.com/Hmbown/CodeWhale/main/CHANGELOG.md", { headers: ghHeaders }).then((r) => r.ok ? r.text() : "").catch(() => ""),
fetch("https://api.github.com/repos/Hmbown/CodeWhale/commits?per_page=30", { headers: ghHeaders }).then((r) => r.ok ? r.json() as Promise<{ commit: { message: string }; sha: string }[]> : []).catch(() => []),
fetch("https://codewhale.net/en", { headers: { "User-Agent": "codewhale-watch" } }).then((r) => r.ok ? r.text() : "").catch(() => ""),
fetch("https://codewhale.net/en/docs", { headers: { "User-Agent": "codewhale-watch" } }).then((r) => r.ok ? r.text() : "").catch(() => ""),
fetchBoundedText("https://raw.githubusercontent.com/Hmbown/CodeWhale/main/CHANGELOG.md", { headers: ghHeaders }).then((r) => r.text).catch(() => ""),
fetchBoundedText("https://api.github.com/repos/Hmbown/CodeWhale/commits?per_page=30", { headers: ghHeaders }).then((r) => r.ok ? JSON.parse(r.text) as { commit: { message: string }; sha: string }[] : []).catch(() => []),
fetchBoundedText("https://codewhale.net/en", { headers: { "User-Agent": "codewhale-watch" } }).then((r) => r.text).catch(() => ""),
fetchBoundedText("https://codewhale.net/en/docs", { headers: { "User-Agent": "codewhale-watch" } }).then((r) => r.text).catch(() => ""),
]);
if (!changelog && (!commits || commits.length === 0)) {
+14
View File
@@ -151,6 +151,20 @@ describe("runFactsDrift", () => {
expect(isRepoFacts(JSON.parse(store.get("facts:current") ?? "null"))).toBe(true);
});
it("never replaces a snapshot from a newer source commit with an older one", async () => {
installGitHubFixture(VALID_GENERATED_FACTS);
const newer = JSON.stringify({ sourceCommittedAt: "2026-07-22T00:00:00Z", version: "9.9.9" });
const store = new Map<string, string>([["facts:current", newer]]);
const kv = {
get: async (key: string) => store.get(key) ?? null,
put: async (key: string, value: string) => { store.set(key, value); },
};
// The fixture's source commit is 2026-07-21T23:00:00Z: an overlapping,
// slower run finishing after a newer one.
expect(await runFactsDrift({ CURATED_KV: kv })).toEqual({ ok: true, changed: false });
expect(store.get("facts:current")).toBe(newer);
});
// The scheduled handler discards the result, so the log line is the only
// signal that the cron stopped refreshing KV.
it("warns and writes nothing when the derived facts fail validation", async () => {
+15 -7
View File
@@ -12,6 +12,7 @@
* fixes itself within one cron tick — no redeploy. Semantic drift (a new
* feature should be advertised on the homepage) is still left to humans.
*/
import { fetchBoundedText } from "./bounded-body";
import type {
PublishedReleaseFact,
RepoFacts,
@@ -46,9 +47,8 @@ async function fetchText(
};
if (ghToken) headers["Authorization"] = `Bearer ${ghToken}`;
try {
const r = await fetch(`${RAW_ROOT}/${revision}/${path}`, { headers });
if (!r.ok) return null;
return await r.text();
const r = await fetchBoundedText(`${RAW_ROOT}/${revision}/${path}`, { headers });
return r.ok ? r.text : null;
} catch {
return null;
}
@@ -62,12 +62,12 @@ async function fetchSourceMarker(ghToken?: string): Promise<SourceMarker | null>
};
if (ghToken) headers.Authorization = `Bearer ${ghToken}`;
try {
const response = await fetch(
const response = await fetchBoundedText(
"https://api.github.com/repos/Hmbown/CodeWhale/commits/main",
{ headers },
);
if (!response.ok) return null;
const json = (await response.json()) as {
const json = JSON.parse(response.text) as {
sha?: string;
commit?: { committer?: { date?: string } };
};
@@ -208,9 +208,9 @@ async function fetchLatestPublishedRelease(
};
if (ghToken) headers["Authorization"] = `Bearer ${ghToken}`;
try {
const r = await fetch("https://api.github.com/repos/Hmbown/CodeWhale/releases/latest", { headers });
const r = await fetchBoundedText("https://api.github.com/repos/Hmbown/CodeWhale/releases/latest", { headers });
if (!r.ok) return null;
const j = (await r.json()) as {
const j = JSON.parse(r.text) as {
tag_name?: string;
published_at?: string;
};
@@ -417,6 +417,14 @@ export async function runFactsDrift(env: { CURATED_KV?: KVNamespace; GITHUB_TOKE
}
}
// Overlapping runs can finish out of order; a snapshot from an older source
// commit never replaces a newer one (KV has no compare-and-set, so this
// narrows the race to the read-to-write window rather than closing it).
const cachedAt = Date.parse(String(cached.sourceCommittedAt ?? ""));
if (cachedRaw && Number.isFinite(cachedAt) && cachedAt > Date.parse(String(remote.sourceCommittedAt))) {
return { ok: true, changed: false };
}
const diffs = diff(cached, remote);
if (diffs.length === 0) {
return { ok: true, changed: false };
+52
View File
@@ -0,0 +1,52 @@
import { spawnSync } from "node:child_process";
import { copyFileSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { afterEach, describe, expect, it } from "vitest";
const script = fileURLToPath(new URL("../scripts/sync-latest-release.mjs", import.meta.url));
const roots: string[] = [];
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const OLD = { tag: "v0.10.0", version: "0.10.0", publishedAt: "2026-09-01T00:00:00Z", url: "https://github.com/Hmbown/CodeWhale/releases/tag/v0.10.0" };
/** The script in a throwaway web/ + docs/ tree, with GitHub stubbed. */
function run(args: string[], { status = 200, cloud = { release: { latest: "0.10.0" } } }: { status?: number; cloud?: unknown } = {}) {
const root = mkdtempSync(join(tmpdir(), "cw-sync-release-"));
roots.push(root);
mkdirSync(join(root, "web", "scripts"), { recursive: true });
mkdirSync(join(root, "web", "data"), { recursive: true });
mkdirSync(join(root, "docs", "cloud-facts"), { recursive: true });
copyFileSync(script, join(root, "web", "scripts", "sync-latest-release.mjs"));
writeFileSync(join(root, "web", "data", "latest-published-release.json"), `${JSON.stringify(OLD, null, 2)}\n`);
writeFileSync(join(root, "docs", "public-surface-facts.json"), `${JSON.stringify({ latestPublishedRelease: { ...OLD, sources: ["x"] } }, null, 2)}\n`);
writeFileSync(join(root, "docs", "cloud-facts", "stable.json"), `${JSON.stringify(cloud, null, 2)}\n`);
const release = JSON.stringify({ tag_name: "v0.10.1", published_at: "2026-09-02T00:00:00Z" });
const stub = `globalThis.fetch = async () => new Response(${JSON.stringify(release)}, { status: ${status} });`;
const result = spawnSync(process.execPath, ["--import", `data:text/javascript,${encodeURIComponent(stub)}`, join(root, "web", "scripts", "sync-latest-release.mjs"), ...args], { encoding: "utf8", env: { ...process.env, GITHUB_TOKEN: "" } });
const read = (...parts: string[]) => JSON.parse(readFileSync(join(root, ...parts), "utf8"));
return { result, root, target: () => read("web", "data", "latest-published-release.json"), mirror: () => read("docs", "public-surface-facts.json") };
}
describe("sync-latest-release", () => {
it("fails --check when GitHub cannot be read instead of reporting current", () => {
const { result } = run(["--check"], { status: 503 });
expect(result.status).toBe(1);
expect(result.stderr).toContain("could not read the latest release");
});
it("writes nothing when any mirror is unusable, then moves all three together", () => {
const broken = run([], { cloud: { notRelease: true } });
expect(broken.result.status).toBe(1);
expect(broken.target()).toEqual(OLD);
expect(broken.mirror().latestPublishedRelease.tag).toBe("v0.10.0");
expect(readdirSync(join(broken.root, "web", "data"))).toEqual(["latest-published-release.json"]);
const ok = run([]);
expect(ok.result.status, ok.result.stderr).toBe(0);
expect(ok.target().tag).toBe("v0.10.1");
expect(ok.mirror().latestPublishedRelease).toMatchObject({ tag: "v0.10.1", sources: ["x"] });
expect(JSON.parse(readFileSync(join(ok.root, "docs", "cloud-facts", "stable.json"), "utf8")).release).toMatchObject({ latest: "0.10.1" });
});
});
+3
View File
@@ -11,3 +11,6 @@ export function emitSql(envelope: unknown, options: { publishedBy?: string; publ
export function readBoundedFile(path: string, maxBytes?: number): Buffer;
export function readBoundedResponse(response: Response, maxBytes?: number): Promise<string>;
export function buildEnvelope(options: { privateKey: KeyObject; keyId: string; payload: Record<string, unknown> }): CloudFactsEnvelope;
export type PostgrestRequest = (path: string, init?: { method?: string; body?: Record<string, unknown>; prefer?: string }) => Promise<unknown>;
export function publishRelease(request: PostgrestRequest, envelope: CloudFactsEnvelope, row: Record<string, unknown>, publicKey: string): Promise<Record<string, unknown>>;
export function revokeRelease(request: PostgrestRequest, options: { channel: string; version: number; reason: string; at: string }): Promise<Record<string, unknown>>;
+56 -14
View File
@@ -463,11 +463,64 @@ async function postgrest(path, { method = "GET", body, prefer } = {}) {
},
body: body === undefined ? undefined : JSON.stringify(body),
});
if (!res.ok) { await res.body?.cancel(); throw new Error(`PostgREST request failed (HTTP ${res.status})`); }
if (!res.ok) { await res.body?.cancel(); throw Object.assign(new Error(`PostgREST request failed (HTTP ${res.status})`), { status: res.status }); }
const text = await readBoundedResponse(res);
return text ? JSON.parse(text) : null;
}
/**
* Insert one signed release and prove what landed. A re-run of the same
* envelope (the first run's reply was lost) reports the stored row instead of
* failing on the version high-water mark; different bytes at that version
* stay an error. `request` is postgrest() or a test double.
*/
export async function publishRelease(request, envelope, row, publicKey) {
const channels = await request(`facts_channel?scope=eq.global&slug=eq.${encodeURIComponent(envelope.channel)}&select=id`);
if (channels?.length !== 1) throw new Error(`channel ${envelope.channel} does not exist`);
const channelId = channels[0].id;
await request("facts_key", { method: "POST", body: { key_id: envelope.key_id, scope: "global", algorithm: "ed25519", public_key: publicKey, status: "active" }, prefer: "resolution=ignore-duplicates,return=minimal" });
let inserted;
try {
inserted = await request("facts_release", { method: "POST", body: { ...row, channel_id: channelId }, prefer: "return=representation" });
} catch (error) {
if (error?.status !== 409) throw error;
const existing = await request(`facts_release?channel_id=eq.${channelId}&facts_version=eq.${envelope.facts_version}&select=id,payload_sha256,status`);
if (existing?.length === 1 && existing[0].payload_sha256 === envelope.sha256) {
return { published: false, already_published: true, channel: envelope.channel, facts_version: envelope.facts_version, release_id: existing[0].id, payload_sha256: existing[0].payload_sha256, status: existing[0].status };
}
throw new Error(`facts_version ${envelope.facts_version} is at or below channel ${envelope.channel}'s high-water mark and is not this envelope; publish a higher facts_version`);
}
if (inserted?.length !== 1 || inserted[0].payload_sha256 !== envelope.sha256) {
throw new Error(`facts_release insert did not return exactly this envelope (rows: ${inserted?.length ?? 0}); inspect channel ${envelope.channel} before retrying`);
}
return { published: true, channel: envelope.channel, facts_version: envelope.facts_version, release_id: inserted[0].id, payload_sha256: inserted[0].payload_sha256 };
}
/**
* Revoke one published release. Only a `published` row flips, so a re-run
* keeps the first revocation's time and reason; a version that matches
* nothing is an error, never a silent `revoked: 0`. `head` says whether the
* channel now serves nothing (revoking an older version changes no delivery).
*/
export async function revokeRelease(request, { channel, version, reason, at }) {
const channels = await request(`facts_channel?scope=eq.global&slug=eq.${encodeURIComponent(channel)}&select=id,max_facts_version`);
if (channels?.length !== 1) throw new Error(`channel ${channel} does not exist`);
const head = Number(channels[0].max_facts_version) === version;
const release = `facts_release?channel_id=eq.${channels[0].id}&facts_version=eq.${version}`;
const updated = await request(`${release}&status=eq.published`, {
method: "PATCH",
body: { status: "revoked", revoked_at: at, revoke_reason: reason },
prefer: "return=representation",
});
if (updated?.length === 1) return { revoked: 1, already_revoked: false, head, channel, facts_version: version };
if ((updated?.length ?? 0) > 1) throw new Error(`revoke matched ${updated.length} releases for ${channel} v${version}; inspect the channel`);
const existing = await request(`${release}&select=status,revoked_at,revoke_reason`);
if (existing?.length === 1 && existing[0].status === "revoked") {
return { revoked: 0, already_revoked: true, head, channel, facts_version: version, revoked_at: existing[0].revoked_at, revoke_reason: existing[0].revoke_reason };
}
throw new Error(`channel ${channel} has no published facts_version ${version} to revoke`);
}
export async function readBoundedResponse(response, maxBytes = MAX_ENVELOPE_BYTES) {
const length = response.headers.get("content-length");
if (length !== null && (!/^\d+$/.test(length) || Number(length) > maxBytes)) {
@@ -591,11 +644,7 @@ async function main(argv) {
console.log(JSON.stringify({ dry_run: true, channel: envelope.channel, facts_key: { key_id: envelope.key_id, public_key: pub }, facts_release: { ...row, payload_b64: `<${envelope.payload_b64.length} chars>` } }, null, 2));
return 0;
}
const channels = await postgrest(`facts_channel?scope=eq.global&slug=eq.${encodeURIComponent(envelope.channel)}&select=id`);
if (!channels?.length) throw new Error(`channel ${envelope.channel} does not exist`);
await postgrest("facts_key", { method: "POST", body: { key_id: envelope.key_id, scope: "global", algorithm: "ed25519", public_key: pub, status: "active" }, prefer: "resolution=ignore-duplicates,return=minimal" });
const inserted = await postgrest("facts_release", { method: "POST", body: { ...row, channel_id: channels[0].id }, prefer: "return=representation" });
console.log(JSON.stringify({ published: true, channel: envelope.channel, facts_version: envelope.facts_version, release_id: inserted?.[0]?.id ?? null, payload_sha256: inserted?.[0]?.payload_sha256 ?? null }, null, 2));
console.log(JSON.stringify(await publishRelease(postgrest, envelope, row, pub), null, 2));
return 0;
}
if (cmd === "revoke") {
@@ -607,14 +656,7 @@ async function main(argv) {
console.log(JSON.stringify({ dry_run: true, channel, facts_version: version, status: "revoked", revoke_reason: reason }, null, 2));
return 0;
}
const channels = await postgrest(`facts_channel?scope=eq.global&slug=eq.${encodeURIComponent(channel)}&select=id`);
if (!channels?.length) throw new Error(`channel ${channel} does not exist`);
const updated = await postgrest(`facts_release?channel_id=eq.${channels[0].id}&facts_version=eq.${version}`, {
method: "PATCH",
body: { status: "revoked", revoked_at: nowIso(), revoke_reason: reason },
prefer: "return=representation",
});
console.log(JSON.stringify({ revoked: updated?.length ?? 0, channel, facts_version: version }, null, 2));
console.log(JSON.stringify(await revokeRelease(postgrest, { channel, version, reason, at: nowIso() }), null, 2));
return 0;
}
throw new Error(`unknown command ${cmd}`);
+46 -20
View File
@@ -31,9 +31,14 @@
// one release behind a release published under 24h ago: that is the window in
// which release.yml's sync-release-record PR is waiting to merge, and neither a
// PR author nor an unrelated push to main can fix it. Past 24h, or more than one
// release behind, it fails again.
// release behind, it fails again. An unreachable or failing GitHub API fails
// --check too: "could not look" is not "current".
//
// A write validates all three files first and then replaces each one through a
// unique temporary file and a rename, so a bad mirror never leaves the first
// file moved on its own and a crash never leaves a half-written JSON file.
import { readFileSync, writeFileSync } from "node:fs";
import { readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { dirname, resolve } from "node:path";
@@ -51,12 +56,18 @@ const headers = {
};
if (process.env.GITHUB_TOKEN) headers.authorization = `Bearer ${process.env.GITHUB_TOKEN}`;
const response = await fetch(`https://api.github.com/repos/${REPO}/releases/latest`, { headers });
if (!response.ok) {
console.error(`[sync-latest-release] GitHub returned ${response.status}; leaving the file alone.`);
process.exit(checkOnly ? 0 : 1);
let release;
try {
const response = await fetch(`https://api.github.com/repos/${REPO}/releases/latest`, {
headers,
signal: AbortSignal.timeout(30_000),
});
if (!response.ok) throw new Error(`GitHub returned ${response.status}`);
release = await response.json();
} catch (error) {
console.error(`[sync-latest-release] could not read the latest release (${error.message}); leaving the files alone.`);
process.exit(1);
}
const release = await response.json();
const tag = String(release.tag_name || "");
const version = tag.startsWith("v") ? tag.slice(1) : "";
@@ -95,7 +106,10 @@ async function isFreshlyOneBehind(recordedTag) {
const age = Date.now() - Date.parse(next.publishedAt);
if (!recordedTag || !(age >= 0 && age < GRACE_MS)) return false;
try {
const res = await fetch(`https://api.github.com/repos/${REPO}/releases?per_page=20`, { headers });
const res = await fetch(`https://api.github.com/repos/${REPO}/releases?per_page=20`, {
headers,
signal: AbortSignal.timeout(30_000),
});
if (!res.ok) return false;
const tags = (await res.json())
.filter((r) => !r.draft && !r.prerelease && Number.isFinite(Date.parse(r.published_at)))
@@ -138,26 +152,38 @@ if (checkOnly) {
process.exit(1);
}
writeFileSync(target, `${JSON.stringify(next, null, 2)}\n`);
// Every input must be usable before anything moves: the three files change
// together or not at all.
if (!matrix) {
console.error(`[sync-latest-release] could not read ${mirror}; the mirror is now stale.`);
console.error(`[sync-latest-release] could not read ${mirror}; nothing was written.`);
process.exit(1);
}
// stable.json is the unsigned cloud-facts authoring source; only the two
// release pointers move here. yanked/min_supported/notice stay human calls.
if (!cloud?.release || typeof cloud.release !== "object") {
console.error(`[sync-latest-release] could not read release in ${cloudFacts}; nothing was written.`);
process.exit(1);
}
// Preserve every key the matrix carries beyond the four synced fields (notably
// `sources`), so this stays a fact refresh and not a schema rewrite.
matrix.latestPublishedRelease = { ...currentMirror, ...next };
writeFileSync(mirror, `${JSON.stringify(matrix, null, 2)}\n`);
// stable.json is the unsigned cloud-facts authoring source; only the two
// release pointers move here. yanked/min_supported/notice stay human calls.
if (!cloud?.release) {
console.error(`[sync-latest-release] could not read release in ${cloudFacts}; it is now stale.`);
process.exit(1);
}
cloud.release.latest = next.version;
cloud.release.release_url = next.url;
writeFileSync(cloudFacts, `${JSON.stringify(cloud, null, 2)}\n`);
const staged = [
[target, next],
[mirror, matrix],
[cloudFacts, cloud],
].map(([path, value]) => {
const temporary = `${path}.${process.pid}.${Date.now()}.tmp`;
writeFileSync(temporary, `${JSON.stringify(value, null, 2)}\n`, { flag: "wx" });
return [temporary, path];
});
try {
for (const [temporary, path] of staged) renameSync(temporary, path);
} finally {
for (const [temporary] of staged) rmSync(temporary, { force: true });
}
console.log(`[sync-latest-release] wrote ${next.tag} (${next.publishedAt}) to all three facts`);