mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-09 09:59:45 -10:00
fix(web): bound cron reads and make facts/release writes honest
Audit 0.10.1 js-ops, web (W02): - W02-07 (Refs #6557): when facts_current has no head (revoked, expired or future-dated), resolveCloudFacts retires the KV last-good copy, so a later Supabase outage cannot resurrect a revoked envelope through the stale path. The last-good TTL drops from 30 days to 6 h, the same staleness bound every client already applies (payload ttl_secs). - W02-10 (Refs #6559): facts-publish revoke flips only a `published` row, so a re-run keeps the first revocation's time and reason and says already_revoked; a version that matches nothing is an error, never a silent `revoked: 0`; the receipt says whether the head was revoked. publish proves the inserted row is exactly this envelope and treats a replay of the same bytes (lost reply) as already_published instead of a bare HTTP 409. Both are exported with an injectable request. - W02-09 (Refs #6555): sync-latest-release --check fails when GitHub cannot be read ("could not look" is not "current"); a write validates all three files first, then replaces each through a unique temp file and rename, so an unreadable mirror no longer leaves the first file moved alone. - W02-06 (Refs #6561): fetchBoundedText (bounded-body.ts) puts a deadline and byte cap on cron reads of other hosts; facts-drift and content-watch use it, and link probes carry a 10 s deadline and discard GET bodies. - W02-05 (Refs #6556): logUsage writes one append-only record per model call instead of read-modify-writing a shared daily counter that overlapping cron tasks lost calls from (web/AGENT.md updated). runFactsDrift never replaces a snapshot from a newer source commit with an older one. Validation (web, vitest 4.1.11): - vitest run cloud-facts, facts-drift, content-watch, bounded-fetch, community-agent, sync-latest-release, bounded-form, security-boundary-behavior, public-api-security, check-facts, facts, community-agent-security: Test Files 12 passed, Tests 103 passed. - tsc --noEmit -p web: exit 0; eslint on touched files: exit 0; node scripts/check-cloud-facts.mjs: OK. - Negative controls (fix reverted, new test run, restored identical): cloud-facts.ts -> "never resurrects a revoked head": expected ok to match unavailable (1 failed) facts-publish revoke without the status filter -> "revokes once": revoked 1 where already_revoked expected (1 failed) sync-latest-release.mjs -> both new tests fail (check exits 0 on 503; target moved to v0.10.1 while the mirror was unusable) facts-drift.ts + content-watch.ts -> "every cron read": 15 reads without a signal (1 failed) community-agent.ts -> "keeps every call": expected 1 to be 3 facts-drift guard disabled -> "never replaces a newer snapshot": changed true (1 failed) Remaining (recorded, not changed): community-agent-tasks GitHub reads and the paid agentChat call have no deadline yet; generator dedup (hasFreshDraft -> model -> saveDraft) can double-spend when cron runs overlap, and a posted draft is still guarded by the reviewed-body hash. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: CodeWhale Bot <[email protected]>
This commit is contained in:
1 parent
332f60beef
commit
ebab81f5ad
14 files changed
+377
-64
No files matched your search
+5
-3
@@ -26,8 +26,8 @@ suffix over the finding's full identity, max 80 chars — so unchanged findings
|
||||
dedup and changed findings land as new drafts. Semantic-drift model output is
|
||||
validated and capped (10 drafts/run) before any KV writes.
|
||||
|
||||
Usage logged to:
|
||||
usage:<YYYY-MM-DD>
|
||||
Usage logged to (one record per model call; sum the day's prefix):
|
||||
usage:<YYYY-MM-DD>:<timestamp>:<uuid>
|
||||
```
|
||||
|
||||
## Cron schedule
|
||||
@@ -57,7 +57,9 @@ All drafts follow these rules:
|
||||
- Each cron invocation caps at ~30k input tokens and ~2k output tokens.
|
||||
- Issue/PR bodies are truncated to 1000–4000 chars before sending to the model.
|
||||
- Deduplication: `hasFreshDraft` checks if a draft already exists that's newer than the item's `updated_at`. Skips if so.
|
||||
- Token usage is logged to `usage:<YYYY-MM-DD>` KV keys (retained 90 days).
|
||||
- Token usage is logged as one `usage:<YYYY-MM-DD>:…` KV record per model call
|
||||
(retained 90 days); list the day's prefix and sum `calls`/`inputTokens`/
|
||||
`outputTokens`. Records are append-only because KV has no atomic increment.
|
||||
- If `DEEPSEEK_API_KEY` is missing or the API errors, the cron returns 200 with `{ skipped: true, reason }` — never crashes, never retry-loops.
|
||||
|
||||
## Maintainer review surface
|
||||
|
||||
@@ -43,3 +43,35 @@ export async function readBoundedBody(request: { body: ReadableStream<Uint8Array
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
|
||||
/** Budget for background (cron) reads of other hosts. */
|
||||
export interface OutboundBudget {
|
||||
/** Deadline for the whole exchange, body included. */
|
||||
timeoutMs?: number;
|
||||
/** Largest body read; a larger one is an error, never a silent truncation. */
|
||||
maxBytes?: number;
|
||||
}
|
||||
|
||||
export const OUTBOUND_TIMEOUT_MS = 15_000;
|
||||
export const OUTBOUND_MAX_BYTES = 2 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* One outbound GET-style read under a deadline and a byte cap. A slow or
|
||||
* endless peer cannot hold a cron invocation until the platform kills it, and
|
||||
* a huge body cannot exhaust isolate memory. Non-2xx answers return
|
||||
* `{ ok: false }` with the body discarded; transport, timeout and size
|
||||
* failures throw, so callers keep their own fallbacks.
|
||||
*/
|
||||
export async function fetchBoundedText(
|
||||
url: string,
|
||||
init: RequestInit = {},
|
||||
{ timeoutMs = OUTBOUND_TIMEOUT_MS, maxBytes = OUTBOUND_MAX_BYTES }: OutboundBudget = {},
|
||||
): Promise<{ ok: boolean; status: number; text: string }> {
|
||||
const response = await fetch(url, { ...init, signal: AbortSignal.timeout(timeoutMs) });
|
||||
if (!response.ok) {
|
||||
try { await response.body?.cancel(); } catch { /* The status is the answer. */ }
|
||||
return { ok: false, status: response.status, text: "" };
|
||||
}
|
||||
const bytes = await readBoundedBody(response, maxBytes);
|
||||
return { ok: true, status: response.status, text: new TextDecoder().decode(bytes) };
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { fetchBoundedText } from "./bounded-body";
|
||||
import { runLinkCheck } from "./content-watch";
|
||||
import { deriveFactsFromRemote } from "./facts-drift";
|
||||
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
/** A peer that accepts the request and never answers until aborted. */
|
||||
function hangingFetch() {
|
||||
return vi.fn((_: unknown, init?: RequestInit) => new Promise<Response>((_, reject) => {
|
||||
init?.signal?.addEventListener("abort", () => reject(init.signal!.reason), { once: true });
|
||||
}));
|
||||
}
|
||||
|
||||
describe("bounded outbound reads", () => {
|
||||
it("gives up on a silent peer at the deadline and refuses an oversized body", async () => {
|
||||
vi.stubGlobal("fetch", hangingFetch());
|
||||
const started = Date.now();
|
||||
await expect(fetchBoundedText("https://slow.test", {}, { timeoutMs: 50 })).rejects.toThrow();
|
||||
expect(Date.now() - started).toBeLessThan(2_000);
|
||||
|
||||
vi.stubGlobal("fetch", vi.fn(async () => new Response("x".repeat(64))));
|
||||
await expect(fetchBoundedText("https://big.test", {}, { maxBytes: 16 })).rejects.toThrow("payload too large");
|
||||
vi.stubGlobal("fetch", vi.fn(async () => new Response("gone", { status: 404 })));
|
||||
expect(await fetchBoundedText("https://missing.test")).toEqual({ ok: false, status: 404, text: "" });
|
||||
});
|
||||
|
||||
it("puts every cron read of other hosts under a deadline", async () => {
|
||||
// Callers swallow transport errors, so record rather than assert inside.
|
||||
const seen = vi.fn(async () => new Response("{}", { status: 500 }));
|
||||
vi.stubGlobal("fetch", seen);
|
||||
await deriveFactsFromRemote();
|
||||
await runLinkCheck({ CURATED_KV: { get: async () => null, put: async () => undefined } } as never);
|
||||
expect(seen.mock.calls.length).toBeGreaterThan(1);
|
||||
const unbounded = seen.mock.calls
|
||||
.filter((call) => !((call as unknown[])[1] as RequestInit | undefined)?.signal)
|
||||
.map((call) => String((call as unknown[])[0]));
|
||||
expect(unbounded).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -11,7 +11,7 @@ import {
|
||||
} from "./cloud-facts";
|
||||
import { TRUSTED_KEYS, type TrustedKey } from "./cloud-facts/keys";
|
||||
import { GET, HEAD } from "../app/api/facts/v1/[channel]/route";
|
||||
import { activePublishingKey, emitSql, readBoundedFile, readBoundedResponse, verifyEnvelope as verifyForPublisher } from "../scripts/facts-publish.mjs";
|
||||
import { activePublishingKey, emitSql, publishRelease, readBoundedFile, readBoundedResponse, revokeRelease, verifyEnvelope as verifyForPublisher } from "../scripts/facts-publish.mjs";
|
||||
import { parseRustKeys, parseTsKeys } from "../scripts/check-cloud-facts.mjs";
|
||||
|
||||
const fixturePath = new URL("../../docs/cloud-facts/fixtures/envelope-stable-v7.json", import.meta.url);
|
||||
@@ -225,6 +225,21 @@ describe("cloud facts transport", () => {
|
||||
expect((await resolveCloudFacts("stable", { ...env, CURATED_KV: kv }, { keys: [EPHEMERAL_KEY], now: () => NOW, fetchImpl: failing })).kind).toBe("unavailable");
|
||||
});
|
||||
|
||||
it("never resurrects a revoked head through the stale cache after an outage", async () => {
|
||||
const kv = new MemKV();
|
||||
const ttls: unknown[] = [];
|
||||
const put = kv.put.bind(kv);
|
||||
kv.put = async (key: string, value: string, options?: { expirationTtl?: number }) => { ttls.push(options?.expirationTtl); await put(key, value); };
|
||||
expect((await resolveCloudFacts("stable", { ...env, CURATED_KV: kv }, { ...opts, fetchImpl: supabaseFetch([rowFrom(fixture)]) })).kind).toBe("ok");
|
||||
// The operator revokes the head: facts_current returns no row.
|
||||
expect((await resolveCloudFacts("stable", { ...env, CURATED_KV: kv }, { ...opts, fetchImpl: supabaseFetch([]) })).kind).toBe("none");
|
||||
// Supabase then goes down. The revoked envelope must not come back.
|
||||
expect(await resolveCloudFacts("stable", { ...env, CURATED_KV: kv }, { ...opts, fetchImpl: failing })).toMatchObject({ kind: "unavailable" });
|
||||
// The bridge lasts no longer than a client's own 6 h staleness bound.
|
||||
expect(ttls.length).toBeGreaterThan(0);
|
||||
expect(ttls.every((ttl) => ttl === 6 * 60 * 60)).toBe(true);
|
||||
});
|
||||
|
||||
it("caps and cancels PostgREST streaming bodies before JSON parsing", async () => {
|
||||
const oversized = overflowingStream();
|
||||
const fetchImpl = vi.fn(async () => new Response(oversized.stream));
|
||||
@@ -301,6 +316,47 @@ describe("facts response protocol", () => {
|
||||
describe("facts publisher boundaries", () => {
|
||||
const script = fileURLToPath(new URL("../scripts/facts-publish.mjs", import.meta.url));
|
||||
|
||||
/** A PostgREST double over one channel and its releases. */
|
||||
function fakePostgrest(head: number, releases: { facts_version: number; status: string; payload_sha256: string; revoked_at?: string; revoke_reason?: string }[]) {
|
||||
const calls: string[] = [];
|
||||
const request = async (path: string, init: { method?: string; body?: Record<string, unknown> } = {}) => {
|
||||
calls.push(`${init.method ?? "GET"} ${path}`);
|
||||
if (path.startsWith("facts_channel")) return [{ id: "c1", max_facts_version: head }];
|
||||
if (path === "facts_key") return null;
|
||||
if (path === "facts_release") {
|
||||
if (Number(init.body?.facts_version) <= head) throw Object.assign(new Error("PostgREST request failed (HTTP 409)"), { status: 409 });
|
||||
head = Number(init.body?.facts_version);
|
||||
const stored = { id: "r-new", facts_version: head, status: "published", payload_sha256: fixture.sha256 };
|
||||
releases.push(stored);
|
||||
return [stored];
|
||||
}
|
||||
const version = Number(/facts_version=eq\.(\d+)/.exec(path)?.[1]);
|
||||
const matches = releases.filter((r) => r.facts_version === version && (!path.includes("status=eq.published") || r.status === "published"));
|
||||
if (init.method === "PATCH") for (const r of matches) Object.assign(r, init.body);
|
||||
return matches.map((r) => ({ id: `r-${r.facts_version}`, ...r }));
|
||||
};
|
||||
return { request, calls, releases };
|
||||
}
|
||||
|
||||
it("revokes once, keeps the first revocation on a re-run, and refuses an unknown version", async () => {
|
||||
const pg = fakePostgrest(3, [{ facts_version: 3, status: "published", payload_sha256: "a" }]);
|
||||
expect(await revokeRelease(pg.request, { channel: "stable", version: 3, reason: "bad price", at: "2026-09-30T00:00:00Z" }))
|
||||
.toMatchObject({ revoked: 1, already_revoked: false, head: true });
|
||||
const again = await revokeRelease(pg.request, { channel: "stable", version: 3, reason: "second run", at: "2026-10-01T00:00:00Z" });
|
||||
expect(again).toMatchObject({ revoked: 0, already_revoked: true, revoked_at: "2026-09-30T00:00:00Z", revoke_reason: "bad price" });
|
||||
expect(pg.releases[0]).toMatchObject({ revoked_at: "2026-09-30T00:00:00Z", revoke_reason: "bad price" });
|
||||
await expect(revokeRelease(pg.request, { channel: "stable", version: 9, reason: "typo", at: "2026-10-01T00:00:00Z" })).rejects.toThrow("no published facts_version 9");
|
||||
});
|
||||
|
||||
it("publishes exactly the envelope and treats a replay of the same bytes as already published", async () => {
|
||||
const pg = fakePostgrest(fixture.facts_version - 1, []);
|
||||
const row = { facts_version: fixture.facts_version };
|
||||
expect(await publishRelease(pg.request, fixture, row, "pub")).toMatchObject({ published: true, payload_sha256: fixture.sha256 });
|
||||
expect(await publishRelease(pg.request, fixture, row, "pub")).toMatchObject({ published: false, already_published: true, release_id: "r-new" });
|
||||
const other = { ...fixture, sha256: "f".repeat(64) };
|
||||
await expect(publishRelease(pg.request, other, row, "pub")).rejects.toThrow("high-water mark");
|
||||
});
|
||||
|
||||
it("requires an active pinned primary key and cannot publish with an explicit fixture public key", () => {
|
||||
expect(() => activePublishingKey(fixture, [])).toThrow("not pinned and active");
|
||||
expect(() => activePublishingKey(fixture, [{ ...TEST_KEY, status: "retired" }])).toThrow("not pinned and active");
|
||||
|
||||
+18
-2
@@ -8,7 +8,11 @@ export const CHANNEL_RE = /^[a-z0-9][a-z0-9-]{0,31}$/;
|
||||
export const KV_PREFIX = "facts:cloud:";
|
||||
export const SUPABASE_TIMEOUT_MS = 3000;
|
||||
export const MAX_ENVELOPE_BYTES = 768 * 1024;
|
||||
const KV_TTL_SECS = 60 * 60 * 24 * 30;
|
||||
// The last-good copy only bridges a transport outage. It lives no longer than
|
||||
// a client keeps applying its own cached envelope (payload ttl_secs, 6 h by
|
||||
// default, docs/CLOUD_FACTS.md), so an outage cannot extend a withdrawn
|
||||
// release beyond the staleness every client already accepts.
|
||||
const KV_TTL_SECS = 60 * 60 * 6;
|
||||
const KEY_ID_RE = /^cwf-[a-z0-9-]{1,32}$/;
|
||||
const VERSION_REQ_RE = /^(\*|(?:>=|<=|>|<|=|\^|~)?\s*\d+(\.\d+){0,2}(-[0-9A-Za-z.-]+)?(\s*,\s*(?:>=|<=|>|<|=|\^|~)?\s*\d+(\.\d+){0,2}(-[0-9A-Za-z.-]+)?)*)$/;
|
||||
const MAX_SIGNATURES = 7; // Plus the primary signature: eight candidates total.
|
||||
@@ -286,6 +290,13 @@ async function kvGet(env: CloudFactsEnv, channel: string): Promise<unknown> {
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
async function retireLastGood(env: CloudFactsEnv, channel: string): Promise<void> {
|
||||
if (!env.CURATED_KV || !isEnvelope(await kvGet(env, channel))) return;
|
||||
try {
|
||||
await env.CURATED_KV.put(`${KV_PREFIX}${channel}`, JSON.stringify({ retired: "no-current-head" }), { expirationTtl: KV_TTL_SECS });
|
||||
} catch { /* Best effort, like the write; the copy still expires on its TTL. */ }
|
||||
}
|
||||
|
||||
export async function resolveCloudFacts(channel: string, env: CloudFactsEnv, opts: ResolveOptions = {}): Promise<CloudFactsResult> {
|
||||
if (!isValidChannel(channel)) return { kind: "none" };
|
||||
const keys = opts.keys ?? TRUSTED_KEYS;
|
||||
@@ -295,7 +306,12 @@ export async function resolveCloudFacts(channel: string, env: CloudFactsEnv, opt
|
||||
let source: "supabase" | "kv-stale" = "supabase";
|
||||
try {
|
||||
const row = await fetchCurrentRow(channel, env, opts);
|
||||
if (!row) return { kind: "none" };
|
||||
if (!row) {
|
||||
// The head was revoked, expired or future-dated. Retire the last-good
|
||||
// copy so a later outage cannot resurrect it through the stale path.
|
||||
await retireLastGood(env, channel);
|
||||
return { kind: "none" };
|
||||
}
|
||||
envelope = envelopeFromRow(row);
|
||||
} catch {
|
||||
source = "kv-stale";
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import { validateSession } from "./community-agent";
|
||||
import { logUsage, validateSession } from "./community-agent";
|
||||
|
||||
describe("validateSession", () => {
|
||||
let mockKV: {
|
||||
@@ -81,3 +81,20 @@ describe("validateSession", () => {
|
||||
expect(mockKV.get).toHaveBeenCalledWith(SESSION_PREFIX + sid);
|
||||
});
|
||||
});
|
||||
|
||||
describe("logUsage", () => {
|
||||
it("keeps every call when cron tasks log at the same time", async () => {
|
||||
// A KV whose reads resolve before either write lands, as overlapping
|
||||
// Worker invocations see it.
|
||||
const store = new Map<string, string>();
|
||||
const kv = {
|
||||
get: async (key: string) => { await new Promise((resolve) => setTimeout(resolve, 5)); return store.get(key) ?? null; },
|
||||
put: async (key: string, value: string) => { await new Promise((resolve) => setTimeout(resolve, 5)); store.set(key, value); },
|
||||
} as unknown as Parameters<typeof logUsage>[0];
|
||||
await Promise.all([logUsage(kv, 100, 10), logUsage(kv, 200, 20), logUsage(kv, 300, 30)]);
|
||||
const records = [...store.values()].map((value) => JSON.parse(value) as { calls: number; inputTokens: number });
|
||||
expect(records.reduce((sum, record) => sum + record.calls, 0)).toBe(3);
|
||||
expect(records.reduce((sum, record) => sum + record.inputTokens, 0)).toBe(600);
|
||||
expect([...store.keys()].every((key) => key.startsWith(`usage:${new Date().toISOString().slice(0, 10)}`))).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -810,16 +810,15 @@ export async function logUsage(
|
||||
outputTokens: number
|
||||
): Promise<void> {
|
||||
if (!kv) return;
|
||||
const date = new Date().toISOString().slice(0, 10);
|
||||
const key = `usage:${date}`;
|
||||
const raw = await kv.get(key);
|
||||
const existing: UsageLog = raw
|
||||
? JSON.parse(raw)
|
||||
: { date, calls: 0, inputTokens: 0, outputTokens: 0 };
|
||||
existing.calls += 1;
|
||||
existing.inputTokens += inputTokens;
|
||||
existing.outputTokens += outputTokens;
|
||||
await kv.put(key, JSON.stringify(existing), { expirationTtl: 60 * 60 * 24 * 90 }); // 90 days
|
||||
// One record per model call under the day's prefix. KV has no atomic
|
||||
// increment, so a shared daily counter that overlapping cron tasks read,
|
||||
// bumped and wrote back lost calls; an append-only record cannot.
|
||||
const at = new Date().toISOString();
|
||||
const date = at.slice(0, 10);
|
||||
const record: UsageLog = { date, calls: 1, inputTokens, outputTokens };
|
||||
await kv.put(`usage:${date}:${at}:${crypto.randomUUID()}`, JSON.stringify(record), {
|
||||
expirationTtl: 60 * 60 * 24 * 90, // 90 days
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
* Both surface as drafts in CURATED_KV under `draft:linkcheck:<...>` and
|
||||
* `draft:semantic-drift:<...>`, picked up by the existing /admin listing.
|
||||
*/
|
||||
import { fetchBoundedText } from "./bounded-body";
|
||||
import { agentChat, draftStorageKey, getDraft, saveDraft, type AgentDraft, type DeepSeekEnv, VOICE_CONSTRAINTS } from "./community-agent";
|
||||
|
||||
interface KVNamespace {
|
||||
@@ -95,15 +96,20 @@ export interface LinkCheckResult {
|
||||
ms: number;
|
||||
}
|
||||
|
||||
const PROBE_TIMEOUT_MS = 10_000;
|
||||
|
||||
async function probe(target: { url: string; label: string }): Promise<LinkCheckResult> {
|
||||
const start = Date.now();
|
||||
try {
|
||||
// Use HEAD where possible; fall back to GET on 405/403 since some hosts
|
||||
// (e.g. Cloudflare-protected) reject HEAD.
|
||||
let r = await fetch(target.url, { method: "HEAD", redirect: "follow" });
|
||||
// Each request has its own deadline; only the status matters, so a GET
|
||||
// body is discarded unread.
|
||||
let r = await fetch(target.url, { method: "HEAD", redirect: "follow", signal: AbortSignal.timeout(PROBE_TIMEOUT_MS) });
|
||||
if (r.status === 405 || r.status === 403 || r.status === 404) {
|
||||
// Some sites return 404 to HEAD but 200 to GET (e.g. NPM)
|
||||
r = await fetch(target.url, { method: "GET", redirect: "follow" });
|
||||
r = await fetch(target.url, { method: "GET", redirect: "follow", signal: AbortSignal.timeout(PROBE_TIMEOUT_MS) });
|
||||
await r.body?.cancel().catch(() => undefined);
|
||||
}
|
||||
return { url: target.url, label: target.label, status: r.status, ok: r.ok, ms: Date.now() - start };
|
||||
} catch {
|
||||
@@ -291,10 +297,10 @@ export async function runSemanticDrift(env: WatchEnv): Promise<{ ok: boolean; dr
|
||||
|
||||
// Fetch CHANGELOG (truncated), recent commits, and live homepage HTML.
|
||||
const [changelog, commits, homepageHtml, docsHtml] = await Promise.all([
|
||||
fetch("https://raw.githubusercontent.com/Hmbown/CodeWhale/main/CHANGELOG.md", { headers: ghHeaders }).then((r) => r.ok ? r.text() : "").catch(() => ""),
|
||||
fetch("https://api.github.com/repos/Hmbown/CodeWhale/commits?per_page=30", { headers: ghHeaders }).then((r) => r.ok ? r.json() as Promise<{ commit: { message: string }; sha: string }[]> : []).catch(() => []),
|
||||
fetch("https://codewhale.net/en", { headers: { "User-Agent": "codewhale-watch" } }).then((r) => r.ok ? r.text() : "").catch(() => ""),
|
||||
fetch("https://codewhale.net/en/docs", { headers: { "User-Agent": "codewhale-watch" } }).then((r) => r.ok ? r.text() : "").catch(() => ""),
|
||||
fetchBoundedText("https://raw.githubusercontent.com/Hmbown/CodeWhale/main/CHANGELOG.md", { headers: ghHeaders }).then((r) => r.text).catch(() => ""),
|
||||
fetchBoundedText("https://api.github.com/repos/Hmbown/CodeWhale/commits?per_page=30", { headers: ghHeaders }).then((r) => r.ok ? JSON.parse(r.text) as { commit: { message: string }; sha: string }[] : []).catch(() => []),
|
||||
fetchBoundedText("https://codewhale.net/en", { headers: { "User-Agent": "codewhale-watch" } }).then((r) => r.text).catch(() => ""),
|
||||
fetchBoundedText("https://codewhale.net/en/docs", { headers: { "User-Agent": "codewhale-watch" } }).then((r) => r.text).catch(() => ""),
|
||||
]);
|
||||
|
||||
if (!changelog && (!commits || commits.length === 0)) {
|
||||
|
||||
@@ -151,6 +151,20 @@ describe("runFactsDrift", () => {
|
||||
expect(isRepoFacts(JSON.parse(store.get("facts:current") ?? "null"))).toBe(true);
|
||||
});
|
||||
|
||||
it("never replaces a snapshot from a newer source commit with an older one", async () => {
|
||||
installGitHubFixture(VALID_GENERATED_FACTS);
|
||||
const newer = JSON.stringify({ sourceCommittedAt: "2026-07-22T00:00:00Z", version: "9.9.9" });
|
||||
const store = new Map<string, string>([["facts:current", newer]]);
|
||||
const kv = {
|
||||
get: async (key: string) => store.get(key) ?? null,
|
||||
put: async (key: string, value: string) => { store.set(key, value); },
|
||||
};
|
||||
// The fixture's source commit is 2026-07-21T23:00:00Z: an overlapping,
|
||||
// slower run finishing after a newer one.
|
||||
expect(await runFactsDrift({ CURATED_KV: kv })).toEqual({ ok: true, changed: false });
|
||||
expect(store.get("facts:current")).toBe(newer);
|
||||
});
|
||||
|
||||
// The scheduled handler discards the result, so the log line is the only
|
||||
// signal that the cron stopped refreshing KV.
|
||||
it("warns and writes nothing when the derived facts fail validation", async () => {
|
||||
|
||||
+15
-7
@@ -12,6 +12,7 @@
|
||||
* fixes itself within one cron tick — no redeploy. Semantic drift (a new
|
||||
* feature should be advertised on the homepage) is still left to humans.
|
||||
*/
|
||||
import { fetchBoundedText } from "./bounded-body";
|
||||
import type {
|
||||
PublishedReleaseFact,
|
||||
RepoFacts,
|
||||
@@ -46,9 +47,8 @@ async function fetchText(
|
||||
};
|
||||
if (ghToken) headers["Authorization"] = `Bearer ${ghToken}`;
|
||||
try {
|
||||
const r = await fetch(`${RAW_ROOT}/${revision}/${path}`, { headers });
|
||||
if (!r.ok) return null;
|
||||
return await r.text();
|
||||
const r = await fetchBoundedText(`${RAW_ROOT}/${revision}/${path}`, { headers });
|
||||
return r.ok ? r.text : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
@@ -62,12 +62,12 @@ async function fetchSourceMarker(ghToken?: string): Promise<SourceMarker | null>
|
||||
};
|
||||
if (ghToken) headers.Authorization = `Bearer ${ghToken}`;
|
||||
try {
|
||||
const response = await fetch(
|
||||
const response = await fetchBoundedText(
|
||||
"https://api.github.com/repos/Hmbown/CodeWhale/commits/main",
|
||||
{ headers },
|
||||
);
|
||||
if (!response.ok) return null;
|
||||
const json = (await response.json()) as {
|
||||
const json = JSON.parse(response.text) as {
|
||||
sha?: string;
|
||||
commit?: { committer?: { date?: string } };
|
||||
};
|
||||
@@ -208,9 +208,9 @@ async function fetchLatestPublishedRelease(
|
||||
};
|
||||
if (ghToken) headers["Authorization"] = `Bearer ${ghToken}`;
|
||||
try {
|
||||
const r = await fetch("https://api.github.com/repos/Hmbown/CodeWhale/releases/latest", { headers });
|
||||
const r = await fetchBoundedText("https://api.github.com/repos/Hmbown/CodeWhale/releases/latest", { headers });
|
||||
if (!r.ok) return null;
|
||||
const j = (await r.json()) as {
|
||||
const j = JSON.parse(r.text) as {
|
||||
tag_name?: string;
|
||||
published_at?: string;
|
||||
};
|
||||
@@ -417,6 +417,14 @@ export async function runFactsDrift(env: { CURATED_KV?: KVNamespace; GITHUB_TOKE
|
||||
}
|
||||
}
|
||||
|
||||
// Overlapping runs can finish out of order; a snapshot from an older source
|
||||
// commit never replaces a newer one (KV has no compare-and-set, so this
|
||||
// narrows the race to the read-to-write window rather than closing it).
|
||||
const cachedAt = Date.parse(String(cached.sourceCommittedAt ?? ""));
|
||||
if (cachedRaw && Number.isFinite(cachedAt) && cachedAt > Date.parse(String(remote.sourceCommittedAt))) {
|
||||
return { ok: true, changed: false };
|
||||
}
|
||||
|
||||
const diffs = diff(cached, remote);
|
||||
if (diffs.length === 0) {
|
||||
return { ok: true, changed: false };
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { copyFileSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
|
||||
const script = fileURLToPath(new URL("../scripts/sync-latest-release.mjs", import.meta.url));
|
||||
const roots: string[] = [];
|
||||
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
|
||||
|
||||
const OLD = { tag: "v0.10.0", version: "0.10.0", publishedAt: "2026-09-01T00:00:00Z", url: "https://github.com/Hmbown/CodeWhale/releases/tag/v0.10.0" };
|
||||
|
||||
/** The script in a throwaway web/ + docs/ tree, with GitHub stubbed. */
|
||||
function run(args: string[], { status = 200, cloud = { release: { latest: "0.10.0" } } }: { status?: number; cloud?: unknown } = {}) {
|
||||
const root = mkdtempSync(join(tmpdir(), "cw-sync-release-"));
|
||||
roots.push(root);
|
||||
mkdirSync(join(root, "web", "scripts"), { recursive: true });
|
||||
mkdirSync(join(root, "web", "data"), { recursive: true });
|
||||
mkdirSync(join(root, "docs", "cloud-facts"), { recursive: true });
|
||||
copyFileSync(script, join(root, "web", "scripts", "sync-latest-release.mjs"));
|
||||
writeFileSync(join(root, "web", "data", "latest-published-release.json"), `${JSON.stringify(OLD, null, 2)}\n`);
|
||||
writeFileSync(join(root, "docs", "public-surface-facts.json"), `${JSON.stringify({ latestPublishedRelease: { ...OLD, sources: ["x"] } }, null, 2)}\n`);
|
||||
writeFileSync(join(root, "docs", "cloud-facts", "stable.json"), `${JSON.stringify(cloud, null, 2)}\n`);
|
||||
const release = JSON.stringify({ tag_name: "v0.10.1", published_at: "2026-09-02T00:00:00Z" });
|
||||
const stub = `globalThis.fetch = async () => new Response(${JSON.stringify(release)}, { status: ${status} });`;
|
||||
const result = spawnSync(process.execPath, ["--import", `data:text/javascript,${encodeURIComponent(stub)}`, join(root, "web", "scripts", "sync-latest-release.mjs"), ...args], { encoding: "utf8", env: { ...process.env, GITHUB_TOKEN: "" } });
|
||||
const read = (...parts: string[]) => JSON.parse(readFileSync(join(root, ...parts), "utf8"));
|
||||
return { result, root, target: () => read("web", "data", "latest-published-release.json"), mirror: () => read("docs", "public-surface-facts.json") };
|
||||
}
|
||||
|
||||
describe("sync-latest-release", () => {
|
||||
it("fails --check when GitHub cannot be read instead of reporting current", () => {
|
||||
const { result } = run(["--check"], { status: 503 });
|
||||
expect(result.status).toBe(1);
|
||||
expect(result.stderr).toContain("could not read the latest release");
|
||||
});
|
||||
|
||||
it("writes nothing when any mirror is unusable, then moves all three together", () => {
|
||||
const broken = run([], { cloud: { notRelease: true } });
|
||||
expect(broken.result.status).toBe(1);
|
||||
expect(broken.target()).toEqual(OLD);
|
||||
expect(broken.mirror().latestPublishedRelease.tag).toBe("v0.10.0");
|
||||
expect(readdirSync(join(broken.root, "web", "data"))).toEqual(["latest-published-release.json"]);
|
||||
|
||||
const ok = run([]);
|
||||
expect(ok.result.status, ok.result.stderr).toBe(0);
|
||||
expect(ok.target().tag).toBe("v0.10.1");
|
||||
expect(ok.mirror().latestPublishedRelease).toMatchObject({ tag: "v0.10.1", sources: ["x"] });
|
||||
expect(JSON.parse(readFileSync(join(ok.root, "docs", "cloud-facts", "stable.json"), "utf8")).release).toMatchObject({ latest: "0.10.1" });
|
||||
});
|
||||
});
|
||||
@@ -11,3 +11,6 @@ export function emitSql(envelope: unknown, options: { publishedBy?: string; publ
|
||||
export function readBoundedFile(path: string, maxBytes?: number): Buffer;
|
||||
export function readBoundedResponse(response: Response, maxBytes?: number): Promise<string>;
|
||||
export function buildEnvelope(options: { privateKey: KeyObject; keyId: string; payload: Record<string, unknown> }): CloudFactsEnvelope;
|
||||
export type PostgrestRequest = (path: string, init?: { method?: string; body?: Record<string, unknown>; prefer?: string }) => Promise<unknown>;
|
||||
export function publishRelease(request: PostgrestRequest, envelope: CloudFactsEnvelope, row: Record<string, unknown>, publicKey: string): Promise<Record<string, unknown>>;
|
||||
export function revokeRelease(request: PostgrestRequest, options: { channel: string; version: number; reason: string; at: string }): Promise<Record<string, unknown>>;
|
||||
@@ -463,11 +463,64 @@ async function postgrest(path, { method = "GET", body, prefer } = {}) {
|
||||
},
|
||||
body: body === undefined ? undefined : JSON.stringify(body),
|
||||
});
|
||||
if (!res.ok) { await res.body?.cancel(); throw new Error(`PostgREST request failed (HTTP ${res.status})`); }
|
||||
if (!res.ok) { await res.body?.cancel(); throw Object.assign(new Error(`PostgREST request failed (HTTP ${res.status})`), { status: res.status }); }
|
||||
const text = await readBoundedResponse(res);
|
||||
return text ? JSON.parse(text) : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert one signed release and prove what landed. A re-run of the same
|
||||
* envelope (the first run's reply was lost) reports the stored row instead of
|
||||
* failing on the version high-water mark; different bytes at that version
|
||||
* stay an error. `request` is postgrest() or a test double.
|
||||
*/
|
||||
export async function publishRelease(request, envelope, row, publicKey) {
|
||||
const channels = await request(`facts_channel?scope=eq.global&slug=eq.${encodeURIComponent(envelope.channel)}&select=id`);
|
||||
if (channels?.length !== 1) throw new Error(`channel ${envelope.channel} does not exist`);
|
||||
const channelId = channels[0].id;
|
||||
await request("facts_key", { method: "POST", body: { key_id: envelope.key_id, scope: "global", algorithm: "ed25519", public_key: publicKey, status: "active" }, prefer: "resolution=ignore-duplicates,return=minimal" });
|
||||
let inserted;
|
||||
try {
|
||||
inserted = await request("facts_release", { method: "POST", body: { ...row, channel_id: channelId }, prefer: "return=representation" });
|
||||
} catch (error) {
|
||||
if (error?.status !== 409) throw error;
|
||||
const existing = await request(`facts_release?channel_id=eq.${channelId}&facts_version=eq.${envelope.facts_version}&select=id,payload_sha256,status`);
|
||||
if (existing?.length === 1 && existing[0].payload_sha256 === envelope.sha256) {
|
||||
return { published: false, already_published: true, channel: envelope.channel, facts_version: envelope.facts_version, release_id: existing[0].id, payload_sha256: existing[0].payload_sha256, status: existing[0].status };
|
||||
}
|
||||
throw new Error(`facts_version ${envelope.facts_version} is at or below channel ${envelope.channel}'s high-water mark and is not this envelope; publish a higher facts_version`);
|
||||
}
|
||||
if (inserted?.length !== 1 || inserted[0].payload_sha256 !== envelope.sha256) {
|
||||
throw new Error(`facts_release insert did not return exactly this envelope (rows: ${inserted?.length ?? 0}); inspect channel ${envelope.channel} before retrying`);
|
||||
}
|
||||
return { published: true, channel: envelope.channel, facts_version: envelope.facts_version, release_id: inserted[0].id, payload_sha256: inserted[0].payload_sha256 };
|
||||
}
|
||||
|
||||
/**
|
||||
* Revoke one published release. Only a `published` row flips, so a re-run
|
||||
* keeps the first revocation's time and reason; a version that matches
|
||||
* nothing is an error, never a silent `revoked: 0`. `head` says whether the
|
||||
* channel now serves nothing (revoking an older version changes no delivery).
|
||||
*/
|
||||
export async function revokeRelease(request, { channel, version, reason, at }) {
|
||||
const channels = await request(`facts_channel?scope=eq.global&slug=eq.${encodeURIComponent(channel)}&select=id,max_facts_version`);
|
||||
if (channels?.length !== 1) throw new Error(`channel ${channel} does not exist`);
|
||||
const head = Number(channels[0].max_facts_version) === version;
|
||||
const release = `facts_release?channel_id=eq.${channels[0].id}&facts_version=eq.${version}`;
|
||||
const updated = await request(`${release}&status=eq.published`, {
|
||||
method: "PATCH",
|
||||
body: { status: "revoked", revoked_at: at, revoke_reason: reason },
|
||||
prefer: "return=representation",
|
||||
});
|
||||
if (updated?.length === 1) return { revoked: 1, already_revoked: false, head, channel, facts_version: version };
|
||||
if ((updated?.length ?? 0) > 1) throw new Error(`revoke matched ${updated.length} releases for ${channel} v${version}; inspect the channel`);
|
||||
const existing = await request(`${release}&select=status,revoked_at,revoke_reason`);
|
||||
if (existing?.length === 1 && existing[0].status === "revoked") {
|
||||
return { revoked: 0, already_revoked: true, head, channel, facts_version: version, revoked_at: existing[0].revoked_at, revoke_reason: existing[0].revoke_reason };
|
||||
}
|
||||
throw new Error(`channel ${channel} has no published facts_version ${version} to revoke`);
|
||||
}
|
||||
|
||||
export async function readBoundedResponse(response, maxBytes = MAX_ENVELOPE_BYTES) {
|
||||
const length = response.headers.get("content-length");
|
||||
if (length !== null && (!/^\d+$/.test(length) || Number(length) > maxBytes)) {
|
||||
@@ -591,11 +644,7 @@ async function main(argv) {
|
||||
console.log(JSON.stringify({ dry_run: true, channel: envelope.channel, facts_key: { key_id: envelope.key_id, public_key: pub }, facts_release: { ...row, payload_b64: `<${envelope.payload_b64.length} chars>` } }, null, 2));
|
||||
return 0;
|
||||
}
|
||||
const channels = await postgrest(`facts_channel?scope=eq.global&slug=eq.${encodeURIComponent(envelope.channel)}&select=id`);
|
||||
if (!channels?.length) throw new Error(`channel ${envelope.channel} does not exist`);
|
||||
await postgrest("facts_key", { method: "POST", body: { key_id: envelope.key_id, scope: "global", algorithm: "ed25519", public_key: pub, status: "active" }, prefer: "resolution=ignore-duplicates,return=minimal" });
|
||||
const inserted = await postgrest("facts_release", { method: "POST", body: { ...row, channel_id: channels[0].id }, prefer: "return=representation" });
|
||||
console.log(JSON.stringify({ published: true, channel: envelope.channel, facts_version: envelope.facts_version, release_id: inserted?.[0]?.id ?? null, payload_sha256: inserted?.[0]?.payload_sha256 ?? null }, null, 2));
|
||||
console.log(JSON.stringify(await publishRelease(postgrest, envelope, row, pub), null, 2));
|
||||
return 0;
|
||||
}
|
||||
if (cmd === "revoke") {
|
||||
@@ -607,14 +656,7 @@ async function main(argv) {
|
||||
console.log(JSON.stringify({ dry_run: true, channel, facts_version: version, status: "revoked", revoke_reason: reason }, null, 2));
|
||||
return 0;
|
||||
}
|
||||
const channels = await postgrest(`facts_channel?scope=eq.global&slug=eq.${encodeURIComponent(channel)}&select=id`);
|
||||
if (!channels?.length) throw new Error(`channel ${channel} does not exist`);
|
||||
const updated = await postgrest(`facts_release?channel_id=eq.${channels[0].id}&facts_version=eq.${version}`, {
|
||||
method: "PATCH",
|
||||
body: { status: "revoked", revoked_at: nowIso(), revoke_reason: reason },
|
||||
prefer: "return=representation",
|
||||
});
|
||||
console.log(JSON.stringify({ revoked: updated?.length ?? 0, channel, facts_version: version }, null, 2));
|
||||
console.log(JSON.stringify(await revokeRelease(postgrest, { channel, version, reason, at: nowIso() }), null, 2));
|
||||
return 0;
|
||||
}
|
||||
throw new Error(`unknown command ${cmd}`);
|
||||
|
||||
@@ -31,9 +31,14 @@
|
||||
// one release behind a release published under 24h ago: that is the window in
|
||||
// which release.yml's sync-release-record PR is waiting to merge, and neither a
|
||||
// PR author nor an unrelated push to main can fix it. Past 24h, or more than one
|
||||
// release behind, it fails again.
|
||||
// release behind, it fails again. An unreachable or failing GitHub API fails
|
||||
// --check too: "could not look" is not "current".
|
||||
//
|
||||
// A write validates all three files first and then replaces each one through a
|
||||
// unique temporary file and a rename, so a bad mirror never leaves the first
|
||||
// file moved on its own and a crash never leaves a half-written JSON file.
|
||||
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { dirname, resolve } from "node:path";
|
||||
|
||||
@@ -51,12 +56,18 @@ const headers = {
|
||||
};
|
||||
if (process.env.GITHUB_TOKEN) headers.authorization = `Bearer ${process.env.GITHUB_TOKEN}`;
|
||||
|
||||
const response = await fetch(`https://api.github.com/repos/${REPO}/releases/latest`, { headers });
|
||||
if (!response.ok) {
|
||||
console.error(`[sync-latest-release] GitHub returned ${response.status}; leaving the file alone.`);
|
||||
process.exit(checkOnly ? 0 : 1);
|
||||
let release;
|
||||
try {
|
||||
const response = await fetch(`https://api.github.com/repos/${REPO}/releases/latest`, {
|
||||
headers,
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
if (!response.ok) throw new Error(`GitHub returned ${response.status}`);
|
||||
release = await response.json();
|
||||
} catch (error) {
|
||||
console.error(`[sync-latest-release] could not read the latest release (${error.message}); leaving the files alone.`);
|
||||
process.exit(1);
|
||||
}
|
||||
const release = await response.json();
|
||||
|
||||
const tag = String(release.tag_name || "");
|
||||
const version = tag.startsWith("v") ? tag.slice(1) : "";
|
||||
@@ -95,7 +106,10 @@ async function isFreshlyOneBehind(recordedTag) {
|
||||
const age = Date.now() - Date.parse(next.publishedAt);
|
||||
if (!recordedTag || !(age >= 0 && age < GRACE_MS)) return false;
|
||||
try {
|
||||
const res = await fetch(`https://api.github.com/repos/${REPO}/releases?per_page=20`, { headers });
|
||||
const res = await fetch(`https://api.github.com/repos/${REPO}/releases?per_page=20`, {
|
||||
headers,
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
if (!res.ok) return false;
|
||||
const tags = (await res.json())
|
||||
.filter((r) => !r.draft && !r.prerelease && Number.isFinite(Date.parse(r.published_at)))
|
||||
@@ -138,26 +152,38 @@ if (checkOnly) {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
writeFileSync(target, `${JSON.stringify(next, null, 2)}\n`);
|
||||
|
||||
// Every input must be usable before anything moves: the three files change
|
||||
// together or not at all.
|
||||
if (!matrix) {
|
||||
console.error(`[sync-latest-release] could not read ${mirror}; the mirror is now stale.`);
|
||||
console.error(`[sync-latest-release] could not read ${mirror}; nothing was written.`);
|
||||
process.exit(1);
|
||||
}
|
||||
// stable.json is the unsigned cloud-facts authoring source; only the two
|
||||
// release pointers move here. yanked/min_supported/notice stay human calls.
|
||||
if (!cloud?.release || typeof cloud.release !== "object") {
|
||||
console.error(`[sync-latest-release] could not read release in ${cloudFacts}; nothing was written.`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Preserve every key the matrix carries beyond the four synced fields (notably
|
||||
// `sources`), so this stays a fact refresh and not a schema rewrite.
|
||||
matrix.latestPublishedRelease = { ...currentMirror, ...next };
|
||||
writeFileSync(mirror, `${JSON.stringify(matrix, null, 2)}\n`);
|
||||
|
||||
// stable.json is the unsigned cloud-facts authoring source; only the two
|
||||
// release pointers move here. yanked/min_supported/notice stay human calls.
|
||||
if (!cloud?.release) {
|
||||
console.error(`[sync-latest-release] could not read release in ${cloudFacts}; it is now stale.`);
|
||||
process.exit(1);
|
||||
}
|
||||
cloud.release.latest = next.version;
|
||||
cloud.release.release_url = next.url;
|
||||
writeFileSync(cloudFacts, `${JSON.stringify(cloud, null, 2)}\n`);
|
||||
|
||||
const staged = [
|
||||
[target, next],
|
||||
[mirror, matrix],
|
||||
[cloudFacts, cloud],
|
||||
].map(([path, value]) => {
|
||||
const temporary = `${path}.${process.pid}.${Date.now()}.tmp`;
|
||||
writeFileSync(temporary, `${JSON.stringify(value, null, 2)}\n`, { flag: "wx" });
|
||||
return [temporary, path];
|
||||
});
|
||||
try {
|
||||
for (const [temporary, path] of staged) renameSync(temporary, path);
|
||||
} finally {
|
||||
for (const [temporary] of staged) rmSync(temporary, { force: true });
|
||||
}
|
||||
|
||||
console.log(`[sync-latest-release] wrote ${next.tag} (${next.publishedAt}) to all three facts`);
|
||||
Reference in new issue
Block a user