feat(cli): Restore-Packet vertical M2b — runtime parsers + runtime-detect + redaction + omitted-records

Per orch decision (Path B chunked M2; M2b chunk after M2a R2 ACCEPT at 40d0724):
five new sub-modules under packages/cli/src/restore-packet/ + extended test
coverage. Velocity prior-art `.mjs` borrowed STRUCTURALLY only (translated to
TypeScript); the .mjs file at field-notes/.../tools/codex-jsonl-to-restore-packet.mjs
remains FROZEN (unchanged).

Sub-modules:

- NEW src/restore-packet/types.ts: shared TypeScript types — ExtractedMessage,
  OmittedRecordClass enum (4 values per contract § 5), PathCount, TypeCounts,
  OmittedCounts, SessionMeta, StructuredTranscript, SourceRuntime. Both
  parsers emit StructuredTranscript; downstream packet-writer (M2c) consumes.

- NEW src/restore-packet/redaction.ts: openrig-v0 policy implementation.
  SECRET_PATTERNS const exports the 5-pattern list verbatim from Velocity
  prior art `:34-48` (sk-*, gh[pousr]_*, github_pat_*, Bearer *, base64 ≥40).
  redact(text) replaces matches with `[REDACTED]`. hasSecretPattern(text) is
  the boolean probe used by the parsers' redaction-counter.

- NEW src/restore-packet/omitted-records.ts: per-record-class classifier.
  classifyCodexRecord maps Codex `payload.type` to one of the 4 omitted
  classes or `kept`; classifyClaudeRecord does the same for Claude `type`
  values. OmittedCounter accumulates per-class counts and exposes
  activeClasses() for the summary's `omitted_classes` field.

- NEW src/restore-packet/runtime-detect.ts: detectRuntime(content) →
  "codex" | "claude-code" | null. Uses content shape (Codex `type:
  "response_item"|"session_meta"|"compacted"` markers vs Claude `type:
  "user"|"assistant"`). Per dispatch directive "Do NOT use file extensions
  or filenames as the detection key. Detect by content shape." — verified
  by negative tests with filename-shaped strings.

- NEW src/restore-packet/codex-jsonl-parser.ts: parseCodexJsonl(content) →
  StructuredTranscript. Translates Velocity `:81-160` to TypeScript:
  session_meta capture, compacted counter, response_item dispatch (kept
  user-visible messages get redacted + path-extracted; tool calls have
  paths extracted then omitted; reasoning records counted as omitted).
  Malformed JSONL lines skipped silently (matches Velocity behavior).

- NEW src/restore-packet/claude-transcript-parser.ts: parseClaudeTranscript(
  content) → StructuredTranscript. NEW work; no prior art. Emits the
  SAME shape as the Codex parser. Walks Claude transcript JSONL: captures
  sessionMeta from first record carrying cwd + sessionId; user/assistant
  records become messages (with redaction); attachment records go into
  raw_tool_outputs; other meta records go into reasoning_records.
  compactedCount is 0 (Claude transcripts don't emit compaction records).

Tests (37 new tests added to packages/cli/test/restore-packet.test.ts;
total 18 M2a + 37 M2b = 55):

- 7 redaction tests (each pattern + non-credential round-trip + pattern
  list size).
- 9 omitted-records tests (classifier branches per runtime + counter
  accumulation + active-classes ordering).
- 7 runtime-detect tests (codex via response_item / session_meta;
  claude via user/assistant; null on empty/garbage/ambiguous; explicit
  "no filename heuristic" negatives).
- 7 codex-jsonl-parser tests (session_meta + messages; reasoning +
  function_call + custom_tool_call filtering; compacted counter; redaction
  in content; malformed line skip; path extraction with frequency sort).
- 6 claude-transcript-parser tests (user + assistant; sessionMeta capture;
  attachment → raw_tool_outputs counter; redaction; compactedCount 0;
  shape conformance with codex parser via Object.keys equality).
- 1 interaction test (Codex parse → redact → omitted-records counter
  chain produces a structured representation with all 3 codex omitted
  classes counted AND the redacted_secrets counter incremented for the
  user message that had a credential pattern).

TDD discipline applied per memory feedback_tdd_scope: tests reference
modules that didn't exist → import failures → modules implemented to
green. The committed state is the green/passing state.

Re-verification (per dispatch's required-fix list):

- npx vitest run test/restore-packet.test.ts → 55/55 pass.
- npx tsc --noEmit → exit 0.
- npm run build → exit 0; dist/restore-packet/{types,redaction,
  omitted-records,runtime-detect,codex-jsonl-parser,claude-transcript-parser}.js
  emitted; dist/schemas/restore-summary.schema.json copied.
- Compiled-output runtime check: node -e import each compiled module →
  detectRuntime, redact, parseCodexJsonl, parseClaudeTranscript all
  loadable as functions.
- Full CLI suite re-run (per Quality Lesson v8 — never silently downgrade):
  695/699 pass. Baseline at M2a R2 was 658/662; M2b adds 37 tests, all
  passing → 695/699. Failure pattern UNCHANGED:
  - test/chatroom.test.ts > Chatroom wait detects newly arriving post-start
    message (10s timing flake)
  - test/config.test.ts > DaemonClient config integration > ConfigStore
    non-default host/port (env pollution)
  - test/preflight.test.ts > rig preflight --json (host port 7433 in use)
  - test/preflight.test.ts > rig preflight check marks (host port 7433 in use)
  All four pre-existing environmental flakes touch zero files in this M2b
  commit. No regression.

Boundary discipline (per dispatch — M2b does NOT touch):

- packet-writer.ts (M2c).
- Atomic write-via-tempdir-then-rename (M2c).
- Daemon full-read transcript route extension (M2c).
- Daemon route tests (M2c HARD DELIVERY FOLLOW-UP).
- restore-packet write actual implementation: stays the M2a-deferred
  error stub. UNCHANGED.
- restore-packet read + validate: stay M3-pending stubs. UNCHANGED.
- M3/M4/M5/M6 surfaces.

Quality lessons applied:

- v3 distinct test summary prefixes: assertion failures use ASSERTIONS_
  prefixes where applicable; test-runner reports its own PASS/FAIL.
- v4 marshal-layer wire-shape: shape-conformance test asserts both
  parsers expose identical top-level keys and identical omittedCounts
  enum keys (the 4 contract values).
- v5 test-naming-vs-instantiation: tests invoke parseCodexJsonl /
  parseClaudeTranscript / detectRuntime / redact / classify* / OmittedCounter
  directly, NOT mock-spy.
- v6 boundary isolation: M2b stays inside packages/cli/src/restore-packet/;
  daemon/test code untouched.
- v7 refined: M2b does NOT extend M2a R2 grant; this commit cites the
  separate M2b grant via the dispatch packet ID.
- v8 honest baseline: full CLI suite re-run; named flakes baseline-checked.
- v9 no auth tokens: all fixtures use synthetic tokens (sk-AbCdEf...,
  ghp_AbCdEfGh..., etc. — clearly synthetic, NOT real provider tokens).
- v10 schema-packaging gap (M2a R2 lesson): M2b sub-modules are pure
  TypeScript (no runtime asset loading), so the M2a R2 packaging issue
  doesn't apply here. The schema embed + drift-catcher pattern from M2a
  R2 stands.

Posture: full QA/testing window (per dispatch).

Source qitem: qitem-20260502010558-463ff419 (M2b dispatch).
Parent: M2a R2 ACCEPT at openrig 40d0724 (qitem-20260502005113-24bf9de8 closure).
This commit is contained in:
mvschwarz committed 2026-05-01 18:13:38 -07:00
1 parent 40d07241ab
commit bdf4cf6844
7 files changed
+1095

No files matched your search

@@ -0,0 +1,188 @@
// claude-transcript-parser.ts — parses Claude transcript JSONL into the
// same StructuredTranscript shape as the Codex parser.
//
// NEW work for the Restore-Packet vertical M2b. No prior art. Inspected
// host's Claude transcript files at ~/.claude/projects/<project>/<id>.jsonl.
//
// Claude JSONL semantics (observed):
// - First few lines may carry meta records: { type, customTitle, sessionId }
// or { type, agentName, sessionId } or { type, permissionMode, sessionId }.
// - Message records have type: "user" | "assistant" with fields:
// { type, message, cwd, timestamp, parentUuid, sessionId, ... }.
// `message` is an object whose shape is provider-API-specific:
// - user: { role: "user", content: string | array }
// - assistant: { role: "assistant", content: array of { type: "text", text } }
// - Attachment records ({ type: "attachment", attachment, ... }) carry tool
// input/output content; mapped to raw_tool_outputs by the omitted-records
// classifier and skipped from the message stream.
// - Other types ({ type: "summary" }, sessionId-only meta records, etc.)
// are mapped to reasoning_records.
import { redact, hasSecretPattern } from "./redaction.js";
import { classifyClaudeRecord, OmittedCounter } from "./omitted-records.js";
import type {
ExtractedMessage,
PathCount,
StructuredTranscript,
TypeCounts,
SessionMeta,
} from "./types.js";
const PATH_PATTERNS: readonly RegExp[] = [
/\/Users\/wrandom\/[A-Za-z0-9._~:/@%+=,\- ]+/g,
/\b(?:packages|docs|scripts|test|tests|src|openrig-work|rigs|control-plane)\/[A-Za-z0-9._~:/@%+=,\-]+/g,
];
function extractPaths(text: string, counts: Map<string, number>): void {
for (const pattern of PATH_PATTERNS) {
for (const match of text.matchAll(pattern)) {
const cleaned = match[0].replace(/[),.;\]"'`]+$/g, "");
if (cleaned.length < 6) continue;
counts.set(cleaned, (counts.get(cleaned) ?? 0) + 1);
}
}
}
function firstLine(text: string): string {
const line = text.split(/\r?\n/).find((l) => l.trim());
if (!line) return "";
return line.trim().slice(0, 180);
}
/**
* Extract user-visible text from a Claude `message.content` field.
* Handles three shapes:
* - string (legacy / user): use directly.
* - array of part objects (assistant or tool-bearing user): collect
* the `text` field of each `{ type: "text", text }` entry.
* - other / null / undefined: empty string.
*
* Tool-use parts (`{ type: "tool_use", ... }`) and tool-result parts
* (`{ type: "tool_result", ... }`) are intentionally NOT included in
* the visible message text; those are captured separately as
* raw_tool_outputs / function_call_output via the attachment-record path.
*/
function textFromClaudeMessage(message: unknown): string {
if (!message || typeof message !== "object") return "";
const obj = message as Record<string, unknown>;
const content = obj.content;
if (typeof content === "string") return content;
if (!Array.isArray(content)) return "";
return content
.map((part) => {
if (typeof part === "string") return part;
if (!part || typeof part !== "object") return "";
const p = part as Record<string, unknown>;
if (p.type === "text" && typeof p.text === "string") return p.text;
// Skip tool_use / tool_result; they're not user-visible message text.
return "";
})
.filter((s) => s.length > 0)
.join("\n");
}
interface ClaudeRecord {
type?: unknown;
timestamp?: unknown;
message?: unknown;
cwd?: unknown;
sessionId?: unknown;
parentUuid?: unknown;
attachment?: unknown;
[key: string]: unknown;
}
/**
* Parse a Claude transcript JSONL string into a StructuredTranscript.
* Pure function; no I/O.
*
* Output is the SAME shape as the Codex parser per the M2b dispatch's
* "emit the same structured representation as the Codex parser"
* requirement. Downstream modules (packet-writer, redaction-counter)
* see no runtime difference.
*/
export function parseClaudeTranscript(content: string): StructuredTranscript {
const messages: ExtractedMessage[] = [];
const pathCounts = new Map<string, number>();
const typeCounts: TypeCounts = {};
const omittedCounter = new OmittedCounter();
let sessionMeta: SessionMeta | null = null;
let lineCount = 0;
let messageCount = 0;
// Claude transcripts don't have "compacted" records; field stays at 0.
for (const rawLine of content.split("\n")) {
if (!rawLine.trim()) continue;
lineCount++;
let record: ClaudeRecord;
try {
record = JSON.parse(rawLine) as ClaudeRecord;
} catch {
continue;
}
const recordType = typeof record.type === "string" ? record.type : "";
typeCounts[recordType] = (typeCounts[recordType] ?? 0) + 1;
// Capture sessionMeta from the first record that carries cwd + sessionId.
if (
sessionMeta === null &&
typeof record.cwd === "string" &&
typeof record.sessionId === "string"
) {
sessionMeta = {
cwd: record.cwd,
sessionId: record.sessionId,
};
}
const classification = classifyClaudeRecord(record);
if (classification.kind === "omitted") {
omittedCounter.recordOmission(classification.reason);
// For attachment records, walk the attachment payload for paths
// (mirrors the Codex parser's tool-call path-extraction behavior).
if (recordType === "attachment" && record.attachment) {
const attachStr = typeof record.attachment === "string"
? record.attachment
: JSON.stringify(record.attachment);
extractPaths(attachStr, pathCounts);
}
continue;
}
// Kept message.
const role: "user" | "assistant" = recordType === "user" ? "user" : "assistant";
const rawText = textFromClaudeMessage(record.message);
if (hasSecretPattern(rawText)) {
omittedCounter.recordRedaction();
}
const text = redact(rawText).trim();
if (!text) continue;
messageCount++;
extractPaths(text, pathCounts);
messages.push({
timestamp: typeof record.timestamp === "string" ? record.timestamp : null,
role,
text,
preview: firstLine(text),
});
}
const paths: PathCount[] = [...pathCounts.entries()]
.sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))
.slice(0, 200)
.map(([path, count]) => ({ path, count }));
return {
sessionMeta,
lineCount,
messageCount,
compactedCount: 0,
typeCounts,
omittedCounts: omittedCounter.counts,
messages,
paths,
};
}
@@ -0,0 +1,186 @@
// codex-jsonl-parser.ts — parses Codex JSONL into a StructuredTranscript.
//
// Translates the Velocity prior art parser at
// field-notes/2026-04-27-velocity-claude-from-codex-restore/tools/
// codex-jsonl-to-restore-packet.mjs:81-160 from JS to TypeScript.
// Velocity .mjs is FROZEN reference; this module borrows the structural
// shape only.
//
// Codex JSONL semantics (from Velocity prior art):
// - record.type === "session_meta" → session metadata (cwd, etc.)
// - record.type === "compacted" → compaction marker (counter only)
// - record.type === "response_item" + payload.type:
// - "function_call" → omitted (function_call_output class); extract paths from args
// - "custom_tool_call" → omitted (raw_tool_outputs class); extract paths from input
// - "reasoning" → omitted (reasoning_records class)
// - "message" → kept (after redaction); extract paths from content
//
// Output is a StructuredTranscript (per types.ts) consumed by the
// packet-writer (M2c).
import { redact, hasSecretPattern } from "./redaction.js";
import { classifyCodexRecord, OmittedCounter } from "./omitted-records.js";
import type {
ExtractedMessage,
PathCount,
StructuredTranscript,
TypeCounts,
SessionMeta,
} from "./types.js";
const PATH_PATTERNS: readonly RegExp[] = [
/\/Users\/wrandom\/[A-Za-z0-9._~:/@%+=,\- ]+/g,
/\b(?:packages|docs|scripts|test|tests|src|openrig-work|rigs|control-plane)\/[A-Za-z0-9._~:/@%+=,\-]+/g,
];
function textFromContent(content: unknown): string {
if (typeof content === "string") return content;
if (!Array.isArray(content)) return "";
return content
.map((part) => {
if (typeof part === "string") return part;
if (!part || typeof part !== "object") return "";
const obj = part as Record<string, unknown>;
const val = obj.text ?? obj.content ?? obj.output ?? obj.input ?? "";
return typeof val === "string" ? val : "";
})
.filter((s) => s.length > 0)
.join("\n");
}
function extractPaths(text: string, counts: Map<string, number>): void {
for (const pattern of PATH_PATTERNS) {
for (const match of text.matchAll(pattern)) {
const cleaned = match[0].replace(/[),.;\]"'`]+$/g, "");
if (cleaned.length < 6) continue;
counts.set(cleaned, (counts.get(cleaned) ?? 0) + 1);
}
}
}
function firstLine(text: string): string {
const line = text.split(/\r?\n/).find((l) => l.trim());
if (!line) return "";
return line.trim().slice(0, 180);
}
interface CodexRecord {
type?: unknown;
timestamp?: unknown;
payload?: {
type?: unknown;
role?: unknown;
content?: unknown;
arguments?: unknown;
input?: unknown;
cwd?: unknown;
[key: string]: unknown;
};
}
/**
* Parse a Codex JSONL string (the full file content as a single string)
* into a StructuredTranscript. Pure function; no I/O.
*
* Malformed lines are skipped silently (matches Velocity prior art
* `:100-102`); the lineCount counts every non-empty input line, so the
* count of skipped lines is `lineCount - <kept-or-typed records>`.
*/
export function parseCodexJsonl(content: string): StructuredTranscript {
const messages: ExtractedMessage[] = [];
const pathCounts = new Map<string, number>();
const typeCounts: TypeCounts = {};
const omittedCounter = new OmittedCounter();
let sessionMeta: SessionMeta | null = null;
let lineCount = 0;
let messageCount = 0;
let compactedCount = 0;
for (const rawLine of content.split("\n")) {
if (!rawLine.trim()) continue;
lineCount++;
let record: CodexRecord;
try {
record = JSON.parse(rawLine) as CodexRecord;
} catch {
continue;
}
const recordType = typeof record.type === "string" ? record.type : "";
typeCounts[recordType] = (typeCounts[recordType] ?? 0) + 1;
if (recordType === "session_meta") {
const payload = (record.payload ?? null) as Record<string, unknown> | null;
sessionMeta = {
cwd: typeof payload?.cwd === "string" ? payload.cwd : null,
sessionId: typeof payload?.id === "string" ? payload.id : null,
raw: payload ?? undefined,
};
continue;
}
if (recordType === "compacted") {
compactedCount++;
continue;
}
if (recordType !== "response_item") continue;
const classification = classifyCodexRecord(record);
if (classification.kind === "omitted") {
omittedCounter.recordOmission(classification.reason);
// Still extract paths from tool calls (matches Velocity behavior at
// :119-129 where args/input are walked for paths even though the
// record itself is omitted from messages).
const payload = record.payload;
if (payload?.type === "function_call") {
const args = typeof payload.arguments === "string" ? payload.arguments : JSON.stringify(payload.arguments ?? {});
extractPaths(args, pathCounts);
} else if (payload?.type === "custom_tool_call") {
const input = typeof payload.input === "string" ? payload.input : JSON.stringify(payload.input ?? {});
extractPaths(input, pathCounts);
}
continue;
}
// Kept message.
const role = record.payload?.role;
const roleStr = typeof role === "string" ? role : "unknown";
if (roleStr !== "developer" && roleStr !== "user" && roleStr !== "assistant") {
continue;
}
const rawText = textFromContent(record.payload?.content);
if (hasSecretPattern(rawText)) {
omittedCounter.recordRedaction();
}
const text = redact(rawText).trim();
if (!text) continue;
messageCount++;
extractPaths(text, pathCounts);
messages.push({
timestamp: typeof record.timestamp === "string" ? record.timestamp : null,
role: roleStr,
text,
preview: firstLine(text),
});
}
const paths: PathCount[] = [...pathCounts.entries()]
.sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))
.slice(0, 200)
.map(([path, count]) => ({ path, count }));
return {
sessionMeta,
lineCount,
messageCount,
compactedCount,
typeCounts,
omittedCounts: omittedCounter.counts,
messages,
paths,
};
}
@@ -0,0 +1,128 @@
// omitted-records.ts — classifies a parsed source record into one of the
// 4 omitted-class enums (per M1 contract § 5) or accepts it as a kept
// user-visible message.
//
// v0 generator excludes the following record classes from
// transcript-latest.md and transcript.md:
//
// - reasoning_records: internal reasoning records that the runtime emitted
// but were not user-visible messages.
// - raw_tool_outputs: full raw tool execution output (often verbose; may
// contain transient state).
// - function_call_output: function-call result records that are not
// user-visible.
// - redacted_secrets: applied to any matched credential patterns per the
// redaction policy. (This class counts records whose content matched
// a secret pattern AND was therefore redacted.)
//
// `omitted_classes` in the summary enumerates the classes ACTUALLY
// excluded for THIS packet (not all available classes); the validator
// checks each name is one of the 4 enum values.
import { emptyOmittedCounts, type OmittedCounts, type OmittedRecordClass } from "./types.js";
import { hasSecretPattern } from "./redaction.js";
/** Source-runtime-agnostic record classifier output. */
export type ClassifyResult =
| { kind: "kept" }
| { kind: "omitted"; reason: OmittedRecordClass };
/**
* Classify a Codex JSONL record (the parsed JSON object) into kept /
* omitted. Caller is responsible for passing only `response_item`-class
* records; session_meta and compacted are handled separately by the
* Codex parser.
*
* Codex semantics (per Velocity prior art):
* - response_item with payload.type === "function_call" → function_call_output
* - response_item with payload.type === "custom_tool_call" → raw_tool_outputs
* - response_item with payload.type === "reasoning" → reasoning_records
* - response_item with payload.type === "message" + role in {developer,user,assistant}
* → kept (after secret-pattern check)
* - everything else → reasoning_records (catch-all for non-message non-tool
* payload types)
*/
export function classifyCodexRecord(record: { payload?: { type?: unknown; role?: unknown } }): ClassifyResult {
const payloadType = typeof record.payload?.type === "string" ? record.payload.type : "";
switch (payloadType) {
case "function_call":
return { kind: "omitted", reason: "function_call_output" };
case "custom_tool_call":
return { kind: "omitted", reason: "raw_tool_outputs" };
case "reasoning":
return { kind: "omitted", reason: "reasoning_records" };
case "message": {
const role = typeof record.payload?.role === "string" ? record.payload.role : "";
if (role !== "developer" && role !== "user" && role !== "assistant") {
return { kind: "omitted", reason: "reasoning_records" };
}
return { kind: "kept" };
}
default:
return { kind: "omitted", reason: "reasoning_records" };
}
}
/**
* Classify a Claude transcript record. Claude shape carries the role
* directly at `record.type` (user / assistant / system / attachment /
* summary etc). Per the M2b dispatch's "emit the same structured
* representation as the Codex parser" requirement, the omitted-class
* mapping mirrors Codex.
*/
export function classifyClaudeRecord(record: { type?: unknown }): ClassifyResult {
const type = typeof record.type === "string" ? record.type : "";
switch (type) {
case "user":
case "assistant":
return { kind: "kept" };
case "attachment":
// Attachments are tool-result-class records (file content the user
// attached or tool output). Map to raw_tool_outputs.
return { kind: "omitted", reason: "raw_tool_outputs" };
case "summary":
// System/summary records are non-user-visible meta; map to
// reasoning_records.
return { kind: "omitted", reason: "reasoning_records" };
default:
// Everything else (sessionId-only, agentInfo, permissionMode meta,
// etc.) is metadata, not a user-visible message.
return { kind: "omitted", reason: "reasoning_records" };
}
}
/** State holder for accumulating omitted-class counts during parsing. */
export class OmittedCounter {
readonly counts: OmittedCounts = emptyOmittedCounts();
recordOmission(reason: OmittedRecordClass): void {
this.counts[reason] += 1;
}
/**
* Mark that a kept message had secret content (was redacted). Increments
* `redacted_secrets` per occurrence. Caller checks `hasSecretPattern`
* BEFORE redaction to decide whether to record this.
*/
recordRedaction(): void {
this.counts.redacted_secrets += 1;
}
/**
* Returns the list of omitted classes that have at least one record.
* Used to populate `summary.omitted_classes` per contract § 5.
* Order: stable (the enum order from the type definition).
*/
activeClasses(): OmittedRecordClass[] {
const order: OmittedRecordClass[] = [
"reasoning_records",
"raw_tool_outputs",
"function_call_output",
"redacted_secrets",
];
return order.filter((cls) => this.counts[cls] > 0);
}
}
/** Re-export for convenience. */
export { hasSecretPattern };
@@ -0,0 +1,72 @@
// redaction.ts — applies the v0 redaction policy.
//
// Per M1 contract § 4: v0 defines two named policies:
// - `velocity-v1`: mirrors the Velocity generator's token patterns at
// field-notes/2026-04-27-velocity-claude-from-codex-restore/tools/
// codex-jsonl-to-restore-packet.mjs:34-48 (sk-*, gh[pousr]_*,
// github_pat_*, Bearer *, base64 ≥40 chars).
// - `openrig-v0`: the productized v0 policy, identical patterns to
// `velocity-v1` but with the canonical name. v0 generator emits
// `redaction_policy_id: "openrig-v0"`.
//
// Redaction is applied to:
// - transcript content in transcript-latest.md and transcript.md.
// - body content in restore-instructions.md if it quotes transcript
// material.
// - top_paths / touched-files entries are NOT redacted (paths are not
// credential material).
//
// The validator does NOT scan content for residual credential patterns
// (per § 4 + § 8: enforcement is generator-side at write time).
/**
* v0 secret-pattern list. Mirrors Velocity prior art `:34-48` exactly:
*
* 1. /\bsk-[A-Za-z0-9_-]{16,}\b/g — OpenAI-style secret tokens.
* 2. /\bgh[pousr]_[A-Za-z0-9_]{20,}\b/g — GitHub personal/OAuth tokens.
* 3. /\bgithub_pat_[A-Za-z0-9_]{20,}\b/g — GitHub fine-grained PATs.
* 4. /\bBearer\s+[A-Za-z0-9._~+/=-]{20,}\b/g — generic Bearer tokens.
* 5. /\b[A-Za-z0-9+/]{40,}={0,2}\b/g — base64-shaped strings ≥40 chars.
*/
export const SECRET_PATTERNS: readonly RegExp[] = [
/\bsk-[A-Za-z0-9_-]{16,}\b/g,
/\bgh[pousr]_[A-Za-z0-9_]{20,}\b/g,
/\bgithub_pat_[A-Za-z0-9_]{20,}\b/g,
/\bBearer\s+[A-Za-z0-9._~+/=-]{20,}\b/g,
/\b[A-Za-z0-9+/]{40,}={0,2}\b/g,
];
/**
* Apply the openrig-v0 redaction policy to a string. Returns the input
* with each match of the secret patterns replaced by the literal
* string `[REDACTED]`. Pure function; no side effects.
*
* The redaction is non-destructive at the structural level — only the
* matched substrings are replaced. Surrounding characters are kept.
*/
export function redact(text: string): string {
let out = text;
for (const pattern of SECRET_PATTERNS) {
out = out.replace(pattern, "[REDACTED]");
}
return out;
}
/**
* Returns true if at least one secret pattern matches the input.
* Used by the omitted-records counter to track how many records had
* their content redacted (for the `redacted_secrets` omitted-class
* count in summary).
*
* Note: a SINGLE record may match multiple patterns; this returns true
* after the first match for efficiency.
*/
export function hasSecretPattern(text: string): boolean {
for (const pattern of SECRET_PATTERNS) {
// Non-global test: cannot reuse the global `g` regex without resetting
// lastIndex, so build a fresh non-global RegExp from source.
const probe = new RegExp(pattern.source);
if (probe.test(text)) return true;
}
return false;
}
@@ -0,0 +1,87 @@
// runtime-detect.ts — auto-detect the source runtime from a JSONL/transcript
// shape.
//
// Per M1 contract § 3.4: source-runtime resolution for `--source-jsonl` is
// auto-detected from the JSONL shape (Codex JSONL has `payload.type`
// markers; Claude transcript has different shape). If auto-detect is
// ambiguous, fail with clear error and prompt for `--source-runtime`
// flag.
//
// Detection is by content shape, NOT file extension or filename
// (per dispatch's "Do NOT use file extensions or filenames as the
// detection key. Detect by content shape.").
//
// Codex shape: each line is JSON with `type` field; the discriminating
// markers are `type: "session_meta"`, `type: "response_item"`, or
// `type: "compacted"`. response_item records carry a `payload` object
// with a `type` sub-field.
//
// Claude shape: each line is JSON with `type` field; values include
// `user`, `assistant`, `system`, `attachment`, `summary`, etc. NO
// `payload` wrapper; the message content is directly under `message`
// or `attachment`.
import type { SourceRuntime } from "./types.js";
/**
* Detect the source runtime from a sample of JSONL lines.
*
* Returns:
* - "codex" — at least one line has `type: "response_item"` OR
* `type: "session_meta"` OR `type: "compacted"`.
* - "claude-code" — at least one line has `type: "user"` or
* `type: "assistant"` AND no Codex markers seen.
* - null — input is empty / non-JSONL / ambiguous (e.g., neither
* marker class appears).
*
* Detection is non-greedy: walks the input lines until a definitive
* marker is found. Bias toward Codex markers if both shape classes
* appear (extremely unlikely in practice; would indicate a malformed
* mixed source).
*/
export function detectRuntime(content: string): SourceRuntime | null {
if (typeof content !== "string" || content.length === 0) return null;
let sawCodexMarker = false;
let sawClaudeMarker = false;
for (const rawLine of content.split("\n")) {
const line = rawLine.trim();
if (!line) continue;
let record: { type?: unknown; payload?: { type?: unknown } };
try {
record = JSON.parse(line);
} catch {
// Skip malformed JSON lines; they don't affect detection.
continue;
}
const recordType = typeof record.type === "string" ? record.type : "";
// Codex discriminators: type values + payload wrapper presence.
if (
recordType === "response_item" ||
recordType === "session_meta" ||
recordType === "compacted"
) {
sawCodexMarker = true;
// Early-return on the strongest Codex marker; ambiguity bias
// toward Codex if Claude markers also appear is intentional
// (Codex's response_item is structurally distinct from any
// Claude record; if it's present, the content is Codex JSONL).
return "codex";
}
// Claude discriminators: top-level type as a user/assistant marker.
if (recordType === "user" || recordType === "assistant") {
sawClaudeMarker = true;
// Don't early-return on Claude alone — keep looking for any
// Codex marker (defensive against the unlikely mixed-shape case).
}
}
if (sawCodexMarker) return "codex";
if (sawClaudeMarker) return "claude-code";
return null;
}
+101
View File
@@ -0,0 +1,101 @@
// types.ts — shared TypeScript types for the restore-packet generator.
//
// Both runtime adapters (codex-jsonl-parser, claude-transcript-parser)
// emit the same StructuredTranscript shape. Downstream modules
// (redaction, omitted-records, packet-writer) consume this shape
// without caring about the source runtime.
/** A single user-visible message extracted from a transcript. */
export interface ExtractedMessage {
/** ISO-8601 timestamp if the source provides one; else null. */
timestamp: string | null;
/** Speaker role. Restricted to user-visible roles (developer/user/assistant). */
role: "developer" | "user" | "assistant";
/** Redacted message text. */
text: string;
/** First non-empty line of the text, truncated to ~180 chars. Used in summaries. */
preview: string;
}
/** Canonical record-class enum per M1 contract § 5. */
export type OmittedRecordClass =
| "reasoning_records"
| "raw_tool_outputs"
| "function_call_output"
| "redacted_secrets";
/**
* A path frequency entry from path extraction. Sorted by count
* descending then path ascending; used to populate `touched_files.top_paths`.
*/
export interface PathCount {
path: string;
count: number;
}
/**
* Per-runtime per-record-type counter. Flexible string keys because
* Codex JSONL uses `record.type` strings and Claude transcript uses
* `record.type` strings; both are runtime-defined.
*/
export type TypeCounts = Record<string, number>;
/**
* Per-omitted-class counter. Records how many records were filtered
* for each of the 4 contract enums during parsing/redaction.
*/
export type OmittedCounts = Record<OmittedRecordClass, number>;
/**
* Optional session metadata extracted from the source. Codex emits a
* `session_meta` record at the top of a JSONL; Claude transcripts
* carry per-record `cwd` etc.
*/
export interface SessionMeta {
cwd: string | null;
sessionId: string | null;
/** Free-form additional fields the runtime emitted; not load-bearing. */
raw?: Record<string, unknown>;
}
/**
* Structured representation produced by both parsers. The packet-writer
* (M2c) consumes this to assemble a v0 restore packet.
*/
export interface StructuredTranscript {
/** Top-level session metadata (cwd, sessionId). */
sessionMeta: SessionMeta | null;
/** Total source lines processed (raw JSONL line count, including malformed/skipped). */
lineCount: number;
/** Number of user-visible messages extracted into `messages`. */
messageCount: number;
/**
* Number of "compaction" records seen. Codex emits these explicitly
* (`type: "compacted"`); Claude doesn't. The field is parser-specific
* but always present (zero when not applicable).
*/
compactedCount: number;
/** Per-record-type frequency map; used in human reports / debug. */
typeCounts: TypeCounts;
/** Per-omitted-class counter; populated as the parser filters records. */
omittedCounts: OmittedCounts;
/** Extracted messages in chronological order; redaction has been applied. */
messages: ExtractedMessage[];
/**
* Path frequency inventory; sorted by count desc then path asc.
* Capped at 200 entries.
*/
paths: PathCount[];
}
/** Source-runtime kind. v0 supports codex + claude-code transcript shapes. */
export type SourceRuntime = "codex" | "claude-code";
export function emptyOmittedCounts(): OmittedCounts {
return {
reasoning_records: 0,
raw_tool_outputs: 0,
function_call_output: 0,
redacted_secrets: 0,
};
}
+333
View File
@@ -20,6 +20,15 @@ import {
type ValidationResult,
} from "../src/restore-packet/schema-validator.js";
import { restorePacketCommand } from "../src/commands/restore-packet.js";
import { redact, hasSecretPattern, SECRET_PATTERNS } from "../src/restore-packet/redaction.js";
import {
classifyCodexRecord,
classifyClaudeRecord,
OmittedCounter,
} from "../src/restore-packet/omitted-records.js";
import { detectRuntime } from "../src/restore-packet/runtime-detect.js";
import { parseCodexJsonl } from "../src/restore-packet/codex-jsonl-parser.js";
import { parseClaudeTranscript } from "../src/restore-packet/claude-transcript-parser.js";
// Minimal valid summary fixture matching the M1 contract (§ 2.1) required-field
// set verbatim. Tests compose against this baseline by mutating one field at
@@ -272,3 +281,327 @@ describe("M2 restore-packet CLI command shell + mutual-exclusion", () => {
expect(validateCmd!.description()).toMatch(/validate|schema/i);
});
});
// ─────────────────────────────────────────────────────────────────────
// M2b sub-module tests (codex-jsonl-parser, claude-transcript-parser,
// runtime-detect, redaction, omitted-records). All fixtures are
// SYNTHETIC; no real transcript content imported. No auth tokens or
// device codes (Quality Lesson v9).
// ─────────────────────────────────────────────────────────────────────
describe("M2b redaction (openrig-v0 / velocity-v1 patterns)", () => {
it("redacts sk-* tokens", () => {
const text = "Some leaked sk-aBcDeFgHiJkLmNoPqRsTuVwXyZ pattern here.";
const redacted = redact(text);
expect(redacted).not.toContain("sk-aBcDeFgHiJkLmNoPqRsTuVwXyZ");
expect(redacted).toContain("[REDACTED]");
});
it("redacts ghp_/ghs_/gho_ GitHub tokens", () => {
const text = "ghp_AbCdEfGhIjKlMnOpQrStUvWxYz0123 was leaked.";
const redacted = redact(text);
expect(redacted).not.toContain("ghp_AbCdEfGhIjKlMnOpQrStUvWxYz0123");
expect(redacted).toContain("[REDACTED]");
});
it("redacts github_pat_ fine-grained tokens", () => {
const text = "Token: github_pat_AbCdEfGh01234567890123456789";
const redacted = redact(text);
expect(redacted).not.toContain("github_pat_AbCdEfGh01234567890123456789");
expect(redacted).toContain("[REDACTED]");
});
it("redacts Bearer tokens", () => {
const text = "Authorization: Bearer aBcD1234.eFgH5678.iJkL9012-mNoP";
const redacted = redact(text);
expect(redacted).not.toContain("Bearer aBcD1234.eFgH5678.iJkL9012-mNoP");
expect(redacted).toContain("[REDACTED]");
});
it("redacts long base64-shaped strings", () => {
const text = "Encoded: aGVsbG93b3JsZGFlaW91YWVpb3VhZWlvdWFlaW91YWVpb3U=";
const redacted = redact(text);
expect(redacted).toContain("[REDACTED]");
});
it("leaves non-credential content unchanged byte-for-byte", () => {
const text = "Normal message: visit https://example.com for docs.";
const redacted = redact(text);
expect(redacted).toBe(text);
});
it("hasSecretPattern detects each known credential class", () => {
expect(hasSecretPattern("sk-AbCdEfGhIjKlMnOpQ")).toBe(true);
expect(hasSecretPattern("normal text without secrets")).toBe(false);
expect(hasSecretPattern("")).toBe(false);
});
it("exposes the pattern list (5 patterns per Velocity prior art)", () => {
expect(SECRET_PATTERNS.length).toBe(5);
});
});
describe("M2b omitted-records classifier", () => {
it("classifies Codex function_call → function_call_output", () => {
const result = classifyCodexRecord({ payload: { type: "function_call" } });
expect(result.kind).toBe("omitted");
if (result.kind === "omitted") {
expect(result.reason).toBe("function_call_output");
}
});
it("classifies Codex custom_tool_call → raw_tool_outputs", () => {
const result = classifyCodexRecord({ payload: { type: "custom_tool_call" } });
expect(result.kind).toBe("omitted");
if (result.kind === "omitted") {
expect(result.reason).toBe("raw_tool_outputs");
}
});
it("classifies Codex reasoning → reasoning_records", () => {
const result = classifyCodexRecord({ payload: { type: "reasoning" } });
expect(result.kind).toBe("omitted");
if (result.kind === "omitted") {
expect(result.reason).toBe("reasoning_records");
}
});
it("classifies Codex message+user-role → kept", () => {
const result = classifyCodexRecord({ payload: { type: "message", role: "user" } });
expect(result.kind).toBe("kept");
});
it("classifies Codex message+unknown-role → omitted (reasoning)", () => {
const result = classifyCodexRecord({ payload: { type: "message", role: "system" } });
expect(result.kind).toBe("omitted");
});
it("classifies Claude attachment → raw_tool_outputs", () => {
const result = classifyClaudeRecord({ type: "attachment" });
expect(result.kind).toBe("omitted");
if (result.kind === "omitted") {
expect(result.reason).toBe("raw_tool_outputs");
}
});
it("classifies Claude user|assistant → kept", () => {
expect(classifyClaudeRecord({ type: "user" }).kind).toBe("kept");
expect(classifyClaudeRecord({ type: "assistant" }).kind).toBe("kept");
});
it("classifies Claude summary → reasoning_records", () => {
const result = classifyClaudeRecord({ type: "summary" });
expect(result.kind).toBe("omitted");
if (result.kind === "omitted") {
expect(result.reason).toBe("reasoning_records");
}
});
it("OmittedCounter accumulates per-class counts and active-classes list", () => {
const counter = new OmittedCounter();
counter.recordOmission("reasoning_records");
counter.recordOmission("reasoning_records");
counter.recordOmission("function_call_output");
counter.recordRedaction();
expect(counter.counts.reasoning_records).toBe(2);
expect(counter.counts.function_call_output).toBe(1);
expect(counter.counts.redacted_secrets).toBe(1);
expect(counter.counts.raw_tool_outputs).toBe(0);
expect(counter.activeClasses()).toEqual([
"reasoning_records",
"function_call_output",
"redacted_secrets",
]);
});
});
describe("M2b runtime-detect", () => {
it("detects Codex JSONL via response_item type marker", () => {
const content = `{"type":"session_meta","payload":{"cwd":"/x"}}
{"type":"response_item","payload":{"type":"message","role":"user","content":"hi"}}`;
expect(detectRuntime(content)).toBe("codex");
});
it("detects Codex via session_meta alone", () => {
const content = `{"type":"session_meta","payload":{"cwd":"/x"}}`;
expect(detectRuntime(content)).toBe("codex");
});
it("detects Claude via top-level user/assistant type", () => {
const content = `{"type":"customTitle","customTitle":"x","sessionId":"s"}
{"type":"user","message":{"role":"user","content":"hi"},"sessionId":"s"}`;
expect(detectRuntime(content)).toBe("claude-code");
});
it("returns null on empty input", () => {
expect(detectRuntime("")).toBe(null);
});
it("returns null on non-JSONL garbage", () => {
expect(detectRuntime("hello world\nfoo bar")).toBe(null);
});
it("returns null on JSONL with no runtime markers (ambiguous)", () => {
const content = `{"foo":"bar"}
{"baz":"qux"}`;
expect(detectRuntime(content)).toBe(null);
});
it("does not use file extension or filename — pure content shape", () => {
// Calling with a string that LOOKS like a filename should still return null.
expect(detectRuntime("/tmp/fake.jsonl")).toBe(null);
expect(detectRuntime("rollout-2026-04-23.jsonl")).toBe(null);
});
});
describe("M2b codex-jsonl-parser", () => {
it("parses session_meta + response_item messages into StructuredTranscript", () => {
const content = `{"type":"session_meta","payload":{"cwd":"/Users/wrandom/code/projects/openrig-hub","id":"abc-123"}}
{"type":"response_item","timestamp":"2026-05-02T01:00:00Z","payload":{"type":"message","role":"user","content":"Hello world"}}
{"type":"response_item","timestamp":"2026-05-02T01:00:01Z","payload":{"type":"message","role":"assistant","content":[{"type":"text","text":"Hi back"}]}}`;
const result = parseCodexJsonl(content);
expect(result.sessionMeta?.cwd).toBe("/Users/wrandom/code/projects/openrig-hub");
expect(result.sessionMeta?.sessionId).toBe("abc-123");
expect(result.messageCount).toBe(2);
expect(result.messages[0]!.role).toBe("user");
expect(result.messages[0]!.text).toBe("Hello world");
expect(result.messages[1]!.role).toBe("assistant");
expect(result.messages[1]!.text).toBe("Hi back");
expect(result.lineCount).toBe(3);
});
it("filters reasoning + function_call + custom_tool_call records and counts them", () => {
const content = `{"type":"response_item","payload":{"type":"reasoning","content":"<thinking>"}}
{"type":"response_item","payload":{"type":"function_call","arguments":"{\\"path\\":\\"/Users/wrandom/x.txt\\"}"}}
{"type":"response_item","payload":{"type":"custom_tool_call","input":"some input"}}
{"type":"response_item","payload":{"type":"message","role":"user","content":"kept"}}`;
const result = parseCodexJsonl(content);
expect(result.messageCount).toBe(1);
expect(result.omittedCounts.reasoning_records).toBeGreaterThanOrEqual(1);
expect(result.omittedCounts.function_call_output).toBe(1);
expect(result.omittedCounts.raw_tool_outputs).toBe(1);
});
it("counts compacted records separately and skips them from messages", () => {
const content = `{"type":"compacted","payload":{}}
{"type":"compacted","payload":{}}
{"type":"response_item","payload":{"type":"message","role":"user","content":"msg"}}`;
const result = parseCodexJsonl(content);
expect(result.compactedCount).toBe(2);
expect(result.messageCount).toBe(1);
});
it("redacts credential patterns in message content", () => {
const content = `{"type":"response_item","payload":{"type":"message","role":"user","content":"my token is sk-AbCdEfGhIjKlMnOpQrStUv now"}}`;
const result = parseCodexJsonl(content);
expect(result.messages[0]!.text).toContain("[REDACTED]");
expect(result.messages[0]!.text).not.toContain("sk-AbCdEfGhIjKlMnOpQrStUv");
expect(result.omittedCounts.redacted_secrets).toBe(1);
});
it("skips malformed JSONL lines silently (matches Velocity prior art)", () => {
const content = `{"type":"response_item","payload":{"type":"message","role":"user","content":"first"}}
not valid json garbage line
{"type":"response_item","payload":{"type":"message","role":"user","content":"second"}}`;
const result = parseCodexJsonl(content);
expect(result.messageCount).toBe(2);
expect(result.lineCount).toBe(3);
});
it("extracts paths from message content and tool args; sorts by frequency", () => {
const content = `{"type":"response_item","payload":{"type":"message","role":"user","content":"see packages/cli/src/index.ts and packages/cli/src/index.ts again"}}
{"type":"response_item","payload":{"type":"function_call","arguments":"{\\"path\\":\\"/Users/wrandom/code/projects/openrig-hub/README.md\\"}"}}`;
const result = parseCodexJsonl(content);
expect(result.paths.length).toBeGreaterThan(0);
const indexEntry = result.paths.find((p) => p.path === "packages/cli/src/index.ts");
expect(indexEntry?.count).toBeGreaterThanOrEqual(2);
});
});
describe("M2b claude-transcript-parser", () => {
it("parses Claude user + assistant messages into StructuredTranscript", () => {
const content = `{"type":"customTitle","customTitle":"M2b test","sessionId":"sess-1"}
{"type":"user","message":{"role":"user","content":"hello"},"cwd":"/x","sessionId":"sess-1","timestamp":"2026-05-02T01:00:00Z"}
{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"hi back"}]},"cwd":"/x","sessionId":"sess-1","timestamp":"2026-05-02T01:00:01Z"}`;
const result = parseClaudeTranscript(content);
expect(result.messageCount).toBe(2);
expect(result.messages[0]!.role).toBe("user");
expect(result.messages[0]!.text).toBe("hello");
expect(result.messages[1]!.role).toBe("assistant");
expect(result.messages[1]!.text).toBe("hi back");
});
it("captures sessionMeta from the first record carrying cwd + sessionId", () => {
const content = `{"type":"customTitle","customTitle":"x","sessionId":"sess-2"}
{"type":"user","message":{"role":"user","content":"first"},"cwd":"/Users/wrandom/code","sessionId":"sess-2"}`;
const result = parseClaudeTranscript(content);
expect(result.sessionMeta?.cwd).toBe("/Users/wrandom/code");
expect(result.sessionMeta?.sessionId).toBe("sess-2");
});
it("filters attachments → raw_tool_outputs counter", () => {
const content = `{"type":"user","message":{"role":"user","content":"see attached"},"cwd":"/x","sessionId":"s"}
{"type":"attachment","attachment":{"path":"/Users/wrandom/x.txt","content":"file body"},"cwd":"/x","sessionId":"s"}`;
const result = parseClaudeTranscript(content);
expect(result.messageCount).toBe(1);
expect(result.omittedCounts.raw_tool_outputs).toBe(1);
});
it("redacts credential patterns in message content", () => {
const content = `{"type":"user","message":{"role":"user","content":"my token is ghp_AbCdEfGhIjKlMnOpQrStUvWxYz0 now"},"cwd":"/x","sessionId":"s"}`;
const result = parseClaudeTranscript(content);
expect(result.messages[0]!.text).toContain("[REDACTED]");
expect(result.omittedCounts.redacted_secrets).toBe(1);
});
it("compactedCount is always 0 (Claude transcripts don't emit compaction records)", () => {
const content = `{"type":"user","message":{"role":"user","content":"x"},"cwd":"/x","sessionId":"s"}`;
const result = parseClaudeTranscript(content);
expect(result.compactedCount).toBe(0);
});
it("emits the SAME StructuredTranscript shape as the Codex parser (interface conformance)", () => {
const codex = parseCodexJsonl(`{"type":"response_item","payload":{"type":"message","role":"user","content":"x"}}`);
const claude = parseClaudeTranscript(`{"type":"user","message":{"role":"user","content":"x"},"cwd":"/x","sessionId":"s"}`);
// Field-by-field check that both expose the same top-level keys.
const codexKeys = Object.keys(codex).sort();
const claudeKeys = Object.keys(claude).sort();
expect(codexKeys).toEqual(claudeKeys);
// Both must expose omittedCounts with all 4 enum keys.
expect(Object.keys(codex.omittedCounts).sort()).toEqual([
"function_call_output",
"raw_tool_outputs",
"reasoning_records",
"redacted_secrets",
]);
expect(Object.keys(claude.omittedCounts).sort()).toEqual([
"function_call_output",
"raw_tool_outputs",
"reasoning_records",
"redacted_secrets",
]);
});
});
describe("M2b interaction: parse → redact → omitted-records counter chain", () => {
it("counts redacted_secrets AND filters all 3 codex omitted classes in one parse", () => {
const content = `{"type":"session_meta","payload":{"cwd":"/x"}}
{"type":"response_item","payload":{"type":"reasoning"}}
{"type":"response_item","payload":{"type":"function_call","arguments":"{}"}}
{"type":"response_item","payload":{"type":"custom_tool_call","input":"{}"}}
{"type":"response_item","payload":{"type":"message","role":"user","content":"my token sk-AbCdEfGhIjKlMnOpQrSt and more text"}}
{"type":"response_item","payload":{"type":"message","role":"assistant","content":"clean reply"}}`;
const result = parseCodexJsonl(content);
expect(result.messageCount).toBe(2);
expect(result.omittedCounts.reasoning_records).toBeGreaterThanOrEqual(1);
expect(result.omittedCounts.function_call_output).toBe(1);
expect(result.omittedCounts.raw_tool_outputs).toBe(1);
expect(result.omittedCounts.redacted_secrets).toBe(1);
// The user message is redacted but kept.
expect(result.messages[0]!.text).toContain("[REDACTED]");
// The assistant message is unchanged.
expect(result.messages[1]!.text).toBe("clean reply");
});
});