mirror of
https://github.com/mvschwarz/openrig.git
synced 2026-11-08 17:31:56 -10:00
feat(cli): Restore-Packet vertical M2b — runtime parsers + runtime-detect + redaction + omitted-records
Per orch decision (Path B chunked M2; M2b chunk after M2a R2 ACCEPT at40d0724): five new sub-modules under packages/cli/src/restore-packet/ + extended test coverage. Velocity prior-art `.mjs` borrowed STRUCTURALLY only (translated to TypeScript); the .mjs file at field-notes/.../tools/codex-jsonl-to-restore-packet.mjs remains FROZEN (unchanged). Sub-modules: - NEW src/restore-packet/types.ts: shared TypeScript types — ExtractedMessage, OmittedRecordClass enum (4 values per contract § 5), PathCount, TypeCounts, OmittedCounts, SessionMeta, StructuredTranscript, SourceRuntime. Both parsers emit StructuredTranscript; downstream packet-writer (M2c) consumes. - NEW src/restore-packet/redaction.ts: openrig-v0 policy implementation. SECRET_PATTERNS const exports the 5-pattern list verbatim from Velocity prior art `:34-48` (sk-*, gh[pousr]_*, github_pat_*, Bearer *, base64 ≥40). redact(text) replaces matches with `[REDACTED]`. hasSecretPattern(text) is the boolean probe used by the parsers' redaction-counter. - NEW src/restore-packet/omitted-records.ts: per-record-class classifier. classifyCodexRecord maps Codex `payload.type` to one of the 4 omitted classes or `kept`; classifyClaudeRecord does the same for Claude `type` values. OmittedCounter accumulates per-class counts and exposes activeClasses() for the summary's `omitted_classes` field. - NEW src/restore-packet/runtime-detect.ts: detectRuntime(content) → "codex" | "claude-code" | null. Uses content shape (Codex `type: "response_item"|"session_meta"|"compacted"` markers vs Claude `type: "user"|"assistant"`). Per dispatch directive "Do NOT use file extensions or filenames as the detection key. Detect by content shape." — verified by negative tests with filename-shaped strings. - NEW src/restore-packet/codex-jsonl-parser.ts: parseCodexJsonl(content) → StructuredTranscript. Translates Velocity `:81-160` to TypeScript: session_meta capture, compacted counter, response_item dispatch (kept user-visible messages get redacted + path-extracted; tool calls have paths extracted then omitted; reasoning records counted as omitted). Malformed JSONL lines skipped silently (matches Velocity behavior). - NEW src/restore-packet/claude-transcript-parser.ts: parseClaudeTranscript( content) → StructuredTranscript. NEW work; no prior art. Emits the SAME shape as the Codex parser. Walks Claude transcript JSONL: captures sessionMeta from first record carrying cwd + sessionId; user/assistant records become messages (with redaction); attachment records go into raw_tool_outputs; other meta records go into reasoning_records. compactedCount is 0 (Claude transcripts don't emit compaction records). Tests (37 new tests added to packages/cli/test/restore-packet.test.ts; total 18 M2a + 37 M2b = 55): - 7 redaction tests (each pattern + non-credential round-trip + pattern list size). - 9 omitted-records tests (classifier branches per runtime + counter accumulation + active-classes ordering). - 7 runtime-detect tests (codex via response_item / session_meta; claude via user/assistant; null on empty/garbage/ambiguous; explicit "no filename heuristic" negatives). - 7 codex-jsonl-parser tests (session_meta + messages; reasoning + function_call + custom_tool_call filtering; compacted counter; redaction in content; malformed line skip; path extraction with frequency sort). - 6 claude-transcript-parser tests (user + assistant; sessionMeta capture; attachment → raw_tool_outputs counter; redaction; compactedCount 0; shape conformance with codex parser via Object.keys equality). - 1 interaction test (Codex parse → redact → omitted-records counter chain produces a structured representation with all 3 codex omitted classes counted AND the redacted_secrets counter incremented for the user message that had a credential pattern). TDD discipline applied per memory feedback_tdd_scope: tests reference modules that didn't exist → import failures → modules implemented to green. The committed state is the green/passing state. Re-verification (per dispatch's required-fix list): - npx vitest run test/restore-packet.test.ts → 55/55 pass. - npx tsc --noEmit → exit 0. - npm run build → exit 0; dist/restore-packet/{types,redaction, omitted-records,runtime-detect,codex-jsonl-parser,claude-transcript-parser}.js emitted; dist/schemas/restore-summary.schema.json copied. - Compiled-output runtime check: node -e import each compiled module → detectRuntime, redact, parseCodexJsonl, parseClaudeTranscript all loadable as functions. - Full CLI suite re-run (per Quality Lesson v8 — never silently downgrade): 695/699 pass. Baseline at M2a R2 was 658/662; M2b adds 37 tests, all passing → 695/699. Failure pattern UNCHANGED: - test/chatroom.test.ts > Chatroom wait detects newly arriving post-start message (10s timing flake) - test/config.test.ts > DaemonClient config integration > ConfigStore non-default host/port (env pollution) - test/preflight.test.ts > rig preflight --json (host port 7433 in use) - test/preflight.test.ts > rig preflight check marks (host port 7433 in use) All four pre-existing environmental flakes touch zero files in this M2b commit. No regression. Boundary discipline (per dispatch — M2b does NOT touch): - packet-writer.ts (M2c). - Atomic write-via-tempdir-then-rename (M2c). - Daemon full-read transcript route extension (M2c). - Daemon route tests (M2c HARD DELIVERY FOLLOW-UP). - restore-packet write actual implementation: stays the M2a-deferred error stub. UNCHANGED. - restore-packet read + validate: stay M3-pending stubs. UNCHANGED. - M3/M4/M5/M6 surfaces. Quality lessons applied: - v3 distinct test summary prefixes: assertion failures use ASSERTIONS_ prefixes where applicable; test-runner reports its own PASS/FAIL. - v4 marshal-layer wire-shape: shape-conformance test asserts both parsers expose identical top-level keys and identical omittedCounts enum keys (the 4 contract values). - v5 test-naming-vs-instantiation: tests invoke parseCodexJsonl / parseClaudeTranscript / detectRuntime / redact / classify* / OmittedCounter directly, NOT mock-spy. - v6 boundary isolation: M2b stays inside packages/cli/src/restore-packet/; daemon/test code untouched. - v7 refined: M2b does NOT extend M2a R2 grant; this commit cites the separate M2b grant via the dispatch packet ID. - v8 honest baseline: full CLI suite re-run; named flakes baseline-checked. - v9 no auth tokens: all fixtures use synthetic tokens (sk-AbCdEf..., ghp_AbCdEfGh..., etc. — clearly synthetic, NOT real provider tokens). - v10 schema-packaging gap (M2a R2 lesson): M2b sub-modules are pure TypeScript (no runtime asset loading), so the M2a R2 packaging issue doesn't apply here. The schema embed + drift-catcher pattern from M2a R2 stands. Posture: full QA/testing window (per dispatch). Source qitem: qitem-20260502010558-463ff419 (M2b dispatch). Parent: M2a R2 ACCEPT at openrig40d0724(qitem-20260502005113-24bf9de8 closure).
This commit is contained in:
1 parent
40d07241ab
commit
bdf4cf6844
7 files changed
+1095
No files matched your search
@@ -0,0 +1,188 @@
|
||||
// claude-transcript-parser.ts — parses Claude transcript JSONL into the
|
||||
// same StructuredTranscript shape as the Codex parser.
|
||||
//
|
||||
// NEW work for the Restore-Packet vertical M2b. No prior art. Inspected
|
||||
// host's Claude transcript files at ~/.claude/projects/<project>/<id>.jsonl.
|
||||
//
|
||||
// Claude JSONL semantics (observed):
|
||||
// - First few lines may carry meta records: { type, customTitle, sessionId }
|
||||
// or { type, agentName, sessionId } or { type, permissionMode, sessionId }.
|
||||
// - Message records have type: "user" | "assistant" with fields:
|
||||
// { type, message, cwd, timestamp, parentUuid, sessionId, ... }.
|
||||
// `message` is an object whose shape is provider-API-specific:
|
||||
// - user: { role: "user", content: string | array }
|
||||
// - assistant: { role: "assistant", content: array of { type: "text", text } }
|
||||
// - Attachment records ({ type: "attachment", attachment, ... }) carry tool
|
||||
// input/output content; mapped to raw_tool_outputs by the omitted-records
|
||||
// classifier and skipped from the message stream.
|
||||
// - Other types ({ type: "summary" }, sessionId-only meta records, etc.)
|
||||
// are mapped to reasoning_records.
|
||||
|
||||
import { redact, hasSecretPattern } from "./redaction.js";
|
||||
import { classifyClaudeRecord, OmittedCounter } from "./omitted-records.js";
|
||||
import type {
|
||||
ExtractedMessage,
|
||||
PathCount,
|
||||
StructuredTranscript,
|
||||
TypeCounts,
|
||||
SessionMeta,
|
||||
} from "./types.js";
|
||||
|
||||
const PATH_PATTERNS: readonly RegExp[] = [
|
||||
/\/Users\/wrandom\/[A-Za-z0-9._~:/@%+=,\- ]+/g,
|
||||
/\b(?:packages|docs|scripts|test|tests|src|openrig-work|rigs|control-plane)\/[A-Za-z0-9._~:/@%+=,\-]+/g,
|
||||
];
|
||||
|
||||
function extractPaths(text: string, counts: Map<string, number>): void {
|
||||
for (const pattern of PATH_PATTERNS) {
|
||||
for (const match of text.matchAll(pattern)) {
|
||||
const cleaned = match[0].replace(/[),.;\]"'`]+$/g, "");
|
||||
if (cleaned.length < 6) continue;
|
||||
counts.set(cleaned, (counts.get(cleaned) ?? 0) + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function firstLine(text: string): string {
|
||||
const line = text.split(/\r?\n/).find((l) => l.trim());
|
||||
if (!line) return "";
|
||||
return line.trim().slice(0, 180);
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract user-visible text from a Claude `message.content` field.
|
||||
* Handles three shapes:
|
||||
* - string (legacy / user): use directly.
|
||||
* - array of part objects (assistant or tool-bearing user): collect
|
||||
* the `text` field of each `{ type: "text", text }` entry.
|
||||
* - other / null / undefined: empty string.
|
||||
*
|
||||
* Tool-use parts (`{ type: "tool_use", ... }`) and tool-result parts
|
||||
* (`{ type: "tool_result", ... }`) are intentionally NOT included in
|
||||
* the visible message text; those are captured separately as
|
||||
* raw_tool_outputs / function_call_output via the attachment-record path.
|
||||
*/
|
||||
function textFromClaudeMessage(message: unknown): string {
|
||||
if (!message || typeof message !== "object") return "";
|
||||
const obj = message as Record<string, unknown>;
|
||||
const content = obj.content;
|
||||
if (typeof content === "string") return content;
|
||||
if (!Array.isArray(content)) return "";
|
||||
return content
|
||||
.map((part) => {
|
||||
if (typeof part === "string") return part;
|
||||
if (!part || typeof part !== "object") return "";
|
||||
const p = part as Record<string, unknown>;
|
||||
if (p.type === "text" && typeof p.text === "string") return p.text;
|
||||
// Skip tool_use / tool_result; they're not user-visible message text.
|
||||
return "";
|
||||
})
|
||||
.filter((s) => s.length > 0)
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
interface ClaudeRecord {
|
||||
type?: unknown;
|
||||
timestamp?: unknown;
|
||||
message?: unknown;
|
||||
cwd?: unknown;
|
||||
sessionId?: unknown;
|
||||
parentUuid?: unknown;
|
||||
attachment?: unknown;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a Claude transcript JSONL string into a StructuredTranscript.
|
||||
* Pure function; no I/O.
|
||||
*
|
||||
* Output is the SAME shape as the Codex parser per the M2b dispatch's
|
||||
* "emit the same structured representation as the Codex parser"
|
||||
* requirement. Downstream modules (packet-writer, redaction-counter)
|
||||
* see no runtime difference.
|
||||
*/
|
||||
export function parseClaudeTranscript(content: string): StructuredTranscript {
|
||||
const messages: ExtractedMessage[] = [];
|
||||
const pathCounts = new Map<string, number>();
|
||||
const typeCounts: TypeCounts = {};
|
||||
const omittedCounter = new OmittedCounter();
|
||||
let sessionMeta: SessionMeta | null = null;
|
||||
let lineCount = 0;
|
||||
let messageCount = 0;
|
||||
// Claude transcripts don't have "compacted" records; field stays at 0.
|
||||
|
||||
for (const rawLine of content.split("\n")) {
|
||||
if (!rawLine.trim()) continue;
|
||||
lineCount++;
|
||||
|
||||
let record: ClaudeRecord;
|
||||
try {
|
||||
record = JSON.parse(rawLine) as ClaudeRecord;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
const recordType = typeof record.type === "string" ? record.type : "";
|
||||
typeCounts[recordType] = (typeCounts[recordType] ?? 0) + 1;
|
||||
|
||||
// Capture sessionMeta from the first record that carries cwd + sessionId.
|
||||
if (
|
||||
sessionMeta === null &&
|
||||
typeof record.cwd === "string" &&
|
||||
typeof record.sessionId === "string"
|
||||
) {
|
||||
sessionMeta = {
|
||||
cwd: record.cwd,
|
||||
sessionId: record.sessionId,
|
||||
};
|
||||
}
|
||||
|
||||
const classification = classifyClaudeRecord(record);
|
||||
if (classification.kind === "omitted") {
|
||||
omittedCounter.recordOmission(classification.reason);
|
||||
// For attachment records, walk the attachment payload for paths
|
||||
// (mirrors the Codex parser's tool-call path-extraction behavior).
|
||||
if (recordType === "attachment" && record.attachment) {
|
||||
const attachStr = typeof record.attachment === "string"
|
||||
? record.attachment
|
||||
: JSON.stringify(record.attachment);
|
||||
extractPaths(attachStr, pathCounts);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Kept message.
|
||||
const role: "user" | "assistant" = recordType === "user" ? "user" : "assistant";
|
||||
const rawText = textFromClaudeMessage(record.message);
|
||||
if (hasSecretPattern(rawText)) {
|
||||
omittedCounter.recordRedaction();
|
||||
}
|
||||
const text = redact(rawText).trim();
|
||||
if (!text) continue;
|
||||
|
||||
messageCount++;
|
||||
extractPaths(text, pathCounts);
|
||||
messages.push({
|
||||
timestamp: typeof record.timestamp === "string" ? record.timestamp : null,
|
||||
role,
|
||||
text,
|
||||
preview: firstLine(text),
|
||||
});
|
||||
}
|
||||
|
||||
const paths: PathCount[] = [...pathCounts.entries()]
|
||||
.sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))
|
||||
.slice(0, 200)
|
||||
.map(([path, count]) => ({ path, count }));
|
||||
|
||||
return {
|
||||
sessionMeta,
|
||||
lineCount,
|
||||
messageCount,
|
||||
compactedCount: 0,
|
||||
typeCounts,
|
||||
omittedCounts: omittedCounter.counts,
|
||||
messages,
|
||||
paths,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
// codex-jsonl-parser.ts — parses Codex JSONL into a StructuredTranscript.
|
||||
//
|
||||
// Translates the Velocity prior art parser at
|
||||
// field-notes/2026-04-27-velocity-claude-from-codex-restore/tools/
|
||||
// codex-jsonl-to-restore-packet.mjs:81-160 from JS to TypeScript.
|
||||
// Velocity .mjs is FROZEN reference; this module borrows the structural
|
||||
// shape only.
|
||||
//
|
||||
// Codex JSONL semantics (from Velocity prior art):
|
||||
// - record.type === "session_meta" → session metadata (cwd, etc.)
|
||||
// - record.type === "compacted" → compaction marker (counter only)
|
||||
// - record.type === "response_item" + payload.type:
|
||||
// - "function_call" → omitted (function_call_output class); extract paths from args
|
||||
// - "custom_tool_call" → omitted (raw_tool_outputs class); extract paths from input
|
||||
// - "reasoning" → omitted (reasoning_records class)
|
||||
// - "message" → kept (after redaction); extract paths from content
|
||||
//
|
||||
// Output is a StructuredTranscript (per types.ts) consumed by the
|
||||
// packet-writer (M2c).
|
||||
|
||||
import { redact, hasSecretPattern } from "./redaction.js";
|
||||
import { classifyCodexRecord, OmittedCounter } from "./omitted-records.js";
|
||||
import type {
|
||||
ExtractedMessage,
|
||||
PathCount,
|
||||
StructuredTranscript,
|
||||
TypeCounts,
|
||||
SessionMeta,
|
||||
} from "./types.js";
|
||||
|
||||
const PATH_PATTERNS: readonly RegExp[] = [
|
||||
/\/Users\/wrandom\/[A-Za-z0-9._~:/@%+=,\- ]+/g,
|
||||
/\b(?:packages|docs|scripts|test|tests|src|openrig-work|rigs|control-plane)\/[A-Za-z0-9._~:/@%+=,\-]+/g,
|
||||
];
|
||||
|
||||
function textFromContent(content: unknown): string {
|
||||
if (typeof content === "string") return content;
|
||||
if (!Array.isArray(content)) return "";
|
||||
return content
|
||||
.map((part) => {
|
||||
if (typeof part === "string") return part;
|
||||
if (!part || typeof part !== "object") return "";
|
||||
const obj = part as Record<string, unknown>;
|
||||
const val = obj.text ?? obj.content ?? obj.output ?? obj.input ?? "";
|
||||
return typeof val === "string" ? val : "";
|
||||
})
|
||||
.filter((s) => s.length > 0)
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
function extractPaths(text: string, counts: Map<string, number>): void {
|
||||
for (const pattern of PATH_PATTERNS) {
|
||||
for (const match of text.matchAll(pattern)) {
|
||||
const cleaned = match[0].replace(/[),.;\]"'`]+$/g, "");
|
||||
if (cleaned.length < 6) continue;
|
||||
counts.set(cleaned, (counts.get(cleaned) ?? 0) + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function firstLine(text: string): string {
|
||||
const line = text.split(/\r?\n/).find((l) => l.trim());
|
||||
if (!line) return "";
|
||||
return line.trim().slice(0, 180);
|
||||
}
|
||||
|
||||
interface CodexRecord {
|
||||
type?: unknown;
|
||||
timestamp?: unknown;
|
||||
payload?: {
|
||||
type?: unknown;
|
||||
role?: unknown;
|
||||
content?: unknown;
|
||||
arguments?: unknown;
|
||||
input?: unknown;
|
||||
cwd?: unknown;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a Codex JSONL string (the full file content as a single string)
|
||||
* into a StructuredTranscript. Pure function; no I/O.
|
||||
*
|
||||
* Malformed lines are skipped silently (matches Velocity prior art
|
||||
* `:100-102`); the lineCount counts every non-empty input line, so the
|
||||
* count of skipped lines is `lineCount - <kept-or-typed records>`.
|
||||
*/
|
||||
export function parseCodexJsonl(content: string): StructuredTranscript {
|
||||
const messages: ExtractedMessage[] = [];
|
||||
const pathCounts = new Map<string, number>();
|
||||
const typeCounts: TypeCounts = {};
|
||||
const omittedCounter = new OmittedCounter();
|
||||
let sessionMeta: SessionMeta | null = null;
|
||||
let lineCount = 0;
|
||||
let messageCount = 0;
|
||||
let compactedCount = 0;
|
||||
|
||||
for (const rawLine of content.split("\n")) {
|
||||
if (!rawLine.trim()) continue;
|
||||
lineCount++;
|
||||
|
||||
let record: CodexRecord;
|
||||
try {
|
||||
record = JSON.parse(rawLine) as CodexRecord;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
const recordType = typeof record.type === "string" ? record.type : "";
|
||||
typeCounts[recordType] = (typeCounts[recordType] ?? 0) + 1;
|
||||
|
||||
if (recordType === "session_meta") {
|
||||
const payload = (record.payload ?? null) as Record<string, unknown> | null;
|
||||
sessionMeta = {
|
||||
cwd: typeof payload?.cwd === "string" ? payload.cwd : null,
|
||||
sessionId: typeof payload?.id === "string" ? payload.id : null,
|
||||
raw: payload ?? undefined,
|
||||
};
|
||||
continue;
|
||||
}
|
||||
|
||||
if (recordType === "compacted") {
|
||||
compactedCount++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (recordType !== "response_item") continue;
|
||||
|
||||
const classification = classifyCodexRecord(record);
|
||||
if (classification.kind === "omitted") {
|
||||
omittedCounter.recordOmission(classification.reason);
|
||||
// Still extract paths from tool calls (matches Velocity behavior at
|
||||
// :119-129 where args/input are walked for paths even though the
|
||||
// record itself is omitted from messages).
|
||||
const payload = record.payload;
|
||||
if (payload?.type === "function_call") {
|
||||
const args = typeof payload.arguments === "string" ? payload.arguments : JSON.stringify(payload.arguments ?? {});
|
||||
extractPaths(args, pathCounts);
|
||||
} else if (payload?.type === "custom_tool_call") {
|
||||
const input = typeof payload.input === "string" ? payload.input : JSON.stringify(payload.input ?? {});
|
||||
extractPaths(input, pathCounts);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Kept message.
|
||||
const role = record.payload?.role;
|
||||
const roleStr = typeof role === "string" ? role : "unknown";
|
||||
if (roleStr !== "developer" && roleStr !== "user" && roleStr !== "assistant") {
|
||||
continue;
|
||||
}
|
||||
|
||||
const rawText = textFromContent(record.payload?.content);
|
||||
if (hasSecretPattern(rawText)) {
|
||||
omittedCounter.recordRedaction();
|
||||
}
|
||||
const text = redact(rawText).trim();
|
||||
if (!text) continue;
|
||||
|
||||
messageCount++;
|
||||
extractPaths(text, pathCounts);
|
||||
messages.push({
|
||||
timestamp: typeof record.timestamp === "string" ? record.timestamp : null,
|
||||
role: roleStr,
|
||||
text,
|
||||
preview: firstLine(text),
|
||||
});
|
||||
}
|
||||
|
||||
const paths: PathCount[] = [...pathCounts.entries()]
|
||||
.sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))
|
||||
.slice(0, 200)
|
||||
.map(([path, count]) => ({ path, count }));
|
||||
|
||||
return {
|
||||
sessionMeta,
|
||||
lineCount,
|
||||
messageCount,
|
||||
compactedCount,
|
||||
typeCounts,
|
||||
omittedCounts: omittedCounter.counts,
|
||||
messages,
|
||||
paths,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
// omitted-records.ts — classifies a parsed source record into one of the
|
||||
// 4 omitted-class enums (per M1 contract § 5) or accepts it as a kept
|
||||
// user-visible message.
|
||||
//
|
||||
// v0 generator excludes the following record classes from
|
||||
// transcript-latest.md and transcript.md:
|
||||
//
|
||||
// - reasoning_records: internal reasoning records that the runtime emitted
|
||||
// but were not user-visible messages.
|
||||
// - raw_tool_outputs: full raw tool execution output (often verbose; may
|
||||
// contain transient state).
|
||||
// - function_call_output: function-call result records that are not
|
||||
// user-visible.
|
||||
// - redacted_secrets: applied to any matched credential patterns per the
|
||||
// redaction policy. (This class counts records whose content matched
|
||||
// a secret pattern AND was therefore redacted.)
|
||||
//
|
||||
// `omitted_classes` in the summary enumerates the classes ACTUALLY
|
||||
// excluded for THIS packet (not all available classes); the validator
|
||||
// checks each name is one of the 4 enum values.
|
||||
|
||||
import { emptyOmittedCounts, type OmittedCounts, type OmittedRecordClass } from "./types.js";
|
||||
import { hasSecretPattern } from "./redaction.js";
|
||||
|
||||
/** Source-runtime-agnostic record classifier output. */
|
||||
export type ClassifyResult =
|
||||
| { kind: "kept" }
|
||||
| { kind: "omitted"; reason: OmittedRecordClass };
|
||||
|
||||
/**
|
||||
* Classify a Codex JSONL record (the parsed JSON object) into kept /
|
||||
* omitted. Caller is responsible for passing only `response_item`-class
|
||||
* records; session_meta and compacted are handled separately by the
|
||||
* Codex parser.
|
||||
*
|
||||
* Codex semantics (per Velocity prior art):
|
||||
* - response_item with payload.type === "function_call" → function_call_output
|
||||
* - response_item with payload.type === "custom_tool_call" → raw_tool_outputs
|
||||
* - response_item with payload.type === "reasoning" → reasoning_records
|
||||
* - response_item with payload.type === "message" + role in {developer,user,assistant}
|
||||
* → kept (after secret-pattern check)
|
||||
* - everything else → reasoning_records (catch-all for non-message non-tool
|
||||
* payload types)
|
||||
*/
|
||||
export function classifyCodexRecord(record: { payload?: { type?: unknown; role?: unknown } }): ClassifyResult {
|
||||
const payloadType = typeof record.payload?.type === "string" ? record.payload.type : "";
|
||||
switch (payloadType) {
|
||||
case "function_call":
|
||||
return { kind: "omitted", reason: "function_call_output" };
|
||||
case "custom_tool_call":
|
||||
return { kind: "omitted", reason: "raw_tool_outputs" };
|
||||
case "reasoning":
|
||||
return { kind: "omitted", reason: "reasoning_records" };
|
||||
case "message": {
|
||||
const role = typeof record.payload?.role === "string" ? record.payload.role : "";
|
||||
if (role !== "developer" && role !== "user" && role !== "assistant") {
|
||||
return { kind: "omitted", reason: "reasoning_records" };
|
||||
}
|
||||
return { kind: "kept" };
|
||||
}
|
||||
default:
|
||||
return { kind: "omitted", reason: "reasoning_records" };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a Claude transcript record. Claude shape carries the role
|
||||
* directly at `record.type` (user / assistant / system / attachment /
|
||||
* summary etc). Per the M2b dispatch's "emit the same structured
|
||||
* representation as the Codex parser" requirement, the omitted-class
|
||||
* mapping mirrors Codex.
|
||||
*/
|
||||
export function classifyClaudeRecord(record: { type?: unknown }): ClassifyResult {
|
||||
const type = typeof record.type === "string" ? record.type : "";
|
||||
switch (type) {
|
||||
case "user":
|
||||
case "assistant":
|
||||
return { kind: "kept" };
|
||||
case "attachment":
|
||||
// Attachments are tool-result-class records (file content the user
|
||||
// attached or tool output). Map to raw_tool_outputs.
|
||||
return { kind: "omitted", reason: "raw_tool_outputs" };
|
||||
case "summary":
|
||||
// System/summary records are non-user-visible meta; map to
|
||||
// reasoning_records.
|
||||
return { kind: "omitted", reason: "reasoning_records" };
|
||||
default:
|
||||
// Everything else (sessionId-only, agentInfo, permissionMode meta,
|
||||
// etc.) is metadata, not a user-visible message.
|
||||
return { kind: "omitted", reason: "reasoning_records" };
|
||||
}
|
||||
}
|
||||
|
||||
/** State holder for accumulating omitted-class counts during parsing. */
|
||||
export class OmittedCounter {
|
||||
readonly counts: OmittedCounts = emptyOmittedCounts();
|
||||
|
||||
recordOmission(reason: OmittedRecordClass): void {
|
||||
this.counts[reason] += 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mark that a kept message had secret content (was redacted). Increments
|
||||
* `redacted_secrets` per occurrence. Caller checks `hasSecretPattern`
|
||||
* BEFORE redaction to decide whether to record this.
|
||||
*/
|
||||
recordRedaction(): void {
|
||||
this.counts.redacted_secrets += 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the list of omitted classes that have at least one record.
|
||||
* Used to populate `summary.omitted_classes` per contract § 5.
|
||||
* Order: stable (the enum order from the type definition).
|
||||
*/
|
||||
activeClasses(): OmittedRecordClass[] {
|
||||
const order: OmittedRecordClass[] = [
|
||||
"reasoning_records",
|
||||
"raw_tool_outputs",
|
||||
"function_call_output",
|
||||
"redacted_secrets",
|
||||
];
|
||||
return order.filter((cls) => this.counts[cls] > 0);
|
||||
}
|
||||
}
|
||||
|
||||
/** Re-export for convenience. */
|
||||
export { hasSecretPattern };
|
||||
@@ -0,0 +1,72 @@
|
||||
// redaction.ts — applies the v0 redaction policy.
|
||||
//
|
||||
// Per M1 contract § 4: v0 defines two named policies:
|
||||
// - `velocity-v1`: mirrors the Velocity generator's token patterns at
|
||||
// field-notes/2026-04-27-velocity-claude-from-codex-restore/tools/
|
||||
// codex-jsonl-to-restore-packet.mjs:34-48 (sk-*, gh[pousr]_*,
|
||||
// github_pat_*, Bearer *, base64 ≥40 chars).
|
||||
// - `openrig-v0`: the productized v0 policy, identical patterns to
|
||||
// `velocity-v1` but with the canonical name. v0 generator emits
|
||||
// `redaction_policy_id: "openrig-v0"`.
|
||||
//
|
||||
// Redaction is applied to:
|
||||
// - transcript content in transcript-latest.md and transcript.md.
|
||||
// - body content in restore-instructions.md if it quotes transcript
|
||||
// material.
|
||||
// - top_paths / touched-files entries are NOT redacted (paths are not
|
||||
// credential material).
|
||||
//
|
||||
// The validator does NOT scan content for residual credential patterns
|
||||
// (per § 4 + § 8: enforcement is generator-side at write time).
|
||||
|
||||
/**
|
||||
* v0 secret-pattern list. Mirrors Velocity prior art `:34-48` exactly:
|
||||
*
|
||||
* 1. /\bsk-[A-Za-z0-9_-]{16,}\b/g — OpenAI-style secret tokens.
|
||||
* 2. /\bgh[pousr]_[A-Za-z0-9_]{20,}\b/g — GitHub personal/OAuth tokens.
|
||||
* 3. /\bgithub_pat_[A-Za-z0-9_]{20,}\b/g — GitHub fine-grained PATs.
|
||||
* 4. /\bBearer\s+[A-Za-z0-9._~+/=-]{20,}\b/g — generic Bearer tokens.
|
||||
* 5. /\b[A-Za-z0-9+/]{40,}={0,2}\b/g — base64-shaped strings ≥40 chars.
|
||||
*/
|
||||
export const SECRET_PATTERNS: readonly RegExp[] = [
|
||||
/\bsk-[A-Za-z0-9_-]{16,}\b/g,
|
||||
/\bgh[pousr]_[A-Za-z0-9_]{20,}\b/g,
|
||||
/\bgithub_pat_[A-Za-z0-9_]{20,}\b/g,
|
||||
/\bBearer\s+[A-Za-z0-9._~+/=-]{20,}\b/g,
|
||||
/\b[A-Za-z0-9+/]{40,}={0,2}\b/g,
|
||||
];
|
||||
|
||||
/**
|
||||
* Apply the openrig-v0 redaction policy to a string. Returns the input
|
||||
* with each match of the secret patterns replaced by the literal
|
||||
* string `[REDACTED]`. Pure function; no side effects.
|
||||
*
|
||||
* The redaction is non-destructive at the structural level — only the
|
||||
* matched substrings are replaced. Surrounding characters are kept.
|
||||
*/
|
||||
export function redact(text: string): string {
|
||||
let out = text;
|
||||
for (const pattern of SECRET_PATTERNS) {
|
||||
out = out.replace(pattern, "[REDACTED]");
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true if at least one secret pattern matches the input.
|
||||
* Used by the omitted-records counter to track how many records had
|
||||
* their content redacted (for the `redacted_secrets` omitted-class
|
||||
* count in summary).
|
||||
*
|
||||
* Note: a SINGLE record may match multiple patterns; this returns true
|
||||
* after the first match for efficiency.
|
||||
*/
|
||||
export function hasSecretPattern(text: string): boolean {
|
||||
for (const pattern of SECRET_PATTERNS) {
|
||||
// Non-global test: cannot reuse the global `g` regex without resetting
|
||||
// lastIndex, so build a fresh non-global RegExp from source.
|
||||
const probe = new RegExp(pattern.source);
|
||||
if (probe.test(text)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
// runtime-detect.ts — auto-detect the source runtime from a JSONL/transcript
|
||||
// shape.
|
||||
//
|
||||
// Per M1 contract § 3.4: source-runtime resolution for `--source-jsonl` is
|
||||
// auto-detected from the JSONL shape (Codex JSONL has `payload.type`
|
||||
// markers; Claude transcript has different shape). If auto-detect is
|
||||
// ambiguous, fail with clear error and prompt for `--source-runtime`
|
||||
// flag.
|
||||
//
|
||||
// Detection is by content shape, NOT file extension or filename
|
||||
// (per dispatch's "Do NOT use file extensions or filenames as the
|
||||
// detection key. Detect by content shape.").
|
||||
//
|
||||
// Codex shape: each line is JSON with `type` field; the discriminating
|
||||
// markers are `type: "session_meta"`, `type: "response_item"`, or
|
||||
// `type: "compacted"`. response_item records carry a `payload` object
|
||||
// with a `type` sub-field.
|
||||
//
|
||||
// Claude shape: each line is JSON with `type` field; values include
|
||||
// `user`, `assistant`, `system`, `attachment`, `summary`, etc. NO
|
||||
// `payload` wrapper; the message content is directly under `message`
|
||||
// or `attachment`.
|
||||
|
||||
import type { SourceRuntime } from "./types.js";
|
||||
|
||||
/**
|
||||
* Detect the source runtime from a sample of JSONL lines.
|
||||
*
|
||||
* Returns:
|
||||
* - "codex" — at least one line has `type: "response_item"` OR
|
||||
* `type: "session_meta"` OR `type: "compacted"`.
|
||||
* - "claude-code" — at least one line has `type: "user"` or
|
||||
* `type: "assistant"` AND no Codex markers seen.
|
||||
* - null — input is empty / non-JSONL / ambiguous (e.g., neither
|
||||
* marker class appears).
|
||||
*
|
||||
* Detection is non-greedy: walks the input lines until a definitive
|
||||
* marker is found. Bias toward Codex markers if both shape classes
|
||||
* appear (extremely unlikely in practice; would indicate a malformed
|
||||
* mixed source).
|
||||
*/
|
||||
export function detectRuntime(content: string): SourceRuntime | null {
|
||||
if (typeof content !== "string" || content.length === 0) return null;
|
||||
|
||||
let sawCodexMarker = false;
|
||||
let sawClaudeMarker = false;
|
||||
|
||||
for (const rawLine of content.split("\n")) {
|
||||
const line = rawLine.trim();
|
||||
if (!line) continue;
|
||||
|
||||
let record: { type?: unknown; payload?: { type?: unknown } };
|
||||
try {
|
||||
record = JSON.parse(line);
|
||||
} catch {
|
||||
// Skip malformed JSON lines; they don't affect detection.
|
||||
continue;
|
||||
}
|
||||
|
||||
const recordType = typeof record.type === "string" ? record.type : "";
|
||||
|
||||
// Codex discriminators: type values + payload wrapper presence.
|
||||
if (
|
||||
recordType === "response_item" ||
|
||||
recordType === "session_meta" ||
|
||||
recordType === "compacted"
|
||||
) {
|
||||
sawCodexMarker = true;
|
||||
// Early-return on the strongest Codex marker; ambiguity bias
|
||||
// toward Codex if Claude markers also appear is intentional
|
||||
// (Codex's response_item is structurally distinct from any
|
||||
// Claude record; if it's present, the content is Codex JSONL).
|
||||
return "codex";
|
||||
}
|
||||
|
||||
// Claude discriminators: top-level type as a user/assistant marker.
|
||||
if (recordType === "user" || recordType === "assistant") {
|
||||
sawClaudeMarker = true;
|
||||
// Don't early-return on Claude alone — keep looking for any
|
||||
// Codex marker (defensive against the unlikely mixed-shape case).
|
||||
}
|
||||
}
|
||||
|
||||
if (sawCodexMarker) return "codex";
|
||||
if (sawClaudeMarker) return "claude-code";
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
// types.ts — shared TypeScript types for the restore-packet generator.
|
||||
//
|
||||
// Both runtime adapters (codex-jsonl-parser, claude-transcript-parser)
|
||||
// emit the same StructuredTranscript shape. Downstream modules
|
||||
// (redaction, omitted-records, packet-writer) consume this shape
|
||||
// without caring about the source runtime.
|
||||
|
||||
/** A single user-visible message extracted from a transcript. */
|
||||
export interface ExtractedMessage {
|
||||
/** ISO-8601 timestamp if the source provides one; else null. */
|
||||
timestamp: string | null;
|
||||
/** Speaker role. Restricted to user-visible roles (developer/user/assistant). */
|
||||
role: "developer" | "user" | "assistant";
|
||||
/** Redacted message text. */
|
||||
text: string;
|
||||
/** First non-empty line of the text, truncated to ~180 chars. Used in summaries. */
|
||||
preview: string;
|
||||
}
|
||||
|
||||
/** Canonical record-class enum per M1 contract § 5. */
|
||||
export type OmittedRecordClass =
|
||||
| "reasoning_records"
|
||||
| "raw_tool_outputs"
|
||||
| "function_call_output"
|
||||
| "redacted_secrets";
|
||||
|
||||
/**
|
||||
* A path frequency entry from path extraction. Sorted by count
|
||||
* descending then path ascending; used to populate `touched_files.top_paths`.
|
||||
*/
|
||||
export interface PathCount {
|
||||
path: string;
|
||||
count: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-runtime per-record-type counter. Flexible string keys because
|
||||
* Codex JSONL uses `record.type` strings and Claude transcript uses
|
||||
* `record.type` strings; both are runtime-defined.
|
||||
*/
|
||||
export type TypeCounts = Record<string, number>;
|
||||
|
||||
/**
|
||||
* Per-omitted-class counter. Records how many records were filtered
|
||||
* for each of the 4 contract enums during parsing/redaction.
|
||||
*/
|
||||
export type OmittedCounts = Record<OmittedRecordClass, number>;
|
||||
|
||||
/**
|
||||
* Optional session metadata extracted from the source. Codex emits a
|
||||
* `session_meta` record at the top of a JSONL; Claude transcripts
|
||||
* carry per-record `cwd` etc.
|
||||
*/
|
||||
export interface SessionMeta {
|
||||
cwd: string | null;
|
||||
sessionId: string | null;
|
||||
/** Free-form additional fields the runtime emitted; not load-bearing. */
|
||||
raw?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Structured representation produced by both parsers. The packet-writer
|
||||
* (M2c) consumes this to assemble a v0 restore packet.
|
||||
*/
|
||||
export interface StructuredTranscript {
|
||||
/** Top-level session metadata (cwd, sessionId). */
|
||||
sessionMeta: SessionMeta | null;
|
||||
/** Total source lines processed (raw JSONL line count, including malformed/skipped). */
|
||||
lineCount: number;
|
||||
/** Number of user-visible messages extracted into `messages`. */
|
||||
messageCount: number;
|
||||
/**
|
||||
* Number of "compaction" records seen. Codex emits these explicitly
|
||||
* (`type: "compacted"`); Claude doesn't. The field is parser-specific
|
||||
* but always present (zero when not applicable).
|
||||
*/
|
||||
compactedCount: number;
|
||||
/** Per-record-type frequency map; used in human reports / debug. */
|
||||
typeCounts: TypeCounts;
|
||||
/** Per-omitted-class counter; populated as the parser filters records. */
|
||||
omittedCounts: OmittedCounts;
|
||||
/** Extracted messages in chronological order; redaction has been applied. */
|
||||
messages: ExtractedMessage[];
|
||||
/**
|
||||
* Path frequency inventory; sorted by count desc then path asc.
|
||||
* Capped at 200 entries.
|
||||
*/
|
||||
paths: PathCount[];
|
||||
}
|
||||
|
||||
/** Source-runtime kind. v0 supports codex + claude-code transcript shapes. */
|
||||
export type SourceRuntime = "codex" | "claude-code";
|
||||
|
||||
export function emptyOmittedCounts(): OmittedCounts {
|
||||
return {
|
||||
reasoning_records: 0,
|
||||
raw_tool_outputs: 0,
|
||||
function_call_output: 0,
|
||||
redacted_secrets: 0,
|
||||
};
|
||||
}
|
||||
@@ -20,6 +20,15 @@ import {
|
||||
type ValidationResult,
|
||||
} from "../src/restore-packet/schema-validator.js";
|
||||
import { restorePacketCommand } from "../src/commands/restore-packet.js";
|
||||
import { redact, hasSecretPattern, SECRET_PATTERNS } from "../src/restore-packet/redaction.js";
|
||||
import {
|
||||
classifyCodexRecord,
|
||||
classifyClaudeRecord,
|
||||
OmittedCounter,
|
||||
} from "../src/restore-packet/omitted-records.js";
|
||||
import { detectRuntime } from "../src/restore-packet/runtime-detect.js";
|
||||
import { parseCodexJsonl } from "../src/restore-packet/codex-jsonl-parser.js";
|
||||
import { parseClaudeTranscript } from "../src/restore-packet/claude-transcript-parser.js";
|
||||
|
||||
// Minimal valid summary fixture matching the M1 contract (§ 2.1) required-field
|
||||
// set verbatim. Tests compose against this baseline by mutating one field at
|
||||
@@ -272,3 +281,327 @@ describe("M2 restore-packet CLI command shell + mutual-exclusion", () => {
|
||||
expect(validateCmd!.description()).toMatch(/validate|schema/i);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
// M2b sub-module tests (codex-jsonl-parser, claude-transcript-parser,
|
||||
// runtime-detect, redaction, omitted-records). All fixtures are
|
||||
// SYNTHETIC; no real transcript content imported. No auth tokens or
|
||||
// device codes (Quality Lesson v9).
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
describe("M2b redaction (openrig-v0 / velocity-v1 patterns)", () => {
|
||||
it("redacts sk-* tokens", () => {
|
||||
const text = "Some leaked sk-aBcDeFgHiJkLmNoPqRsTuVwXyZ pattern here.";
|
||||
const redacted = redact(text);
|
||||
expect(redacted).not.toContain("sk-aBcDeFgHiJkLmNoPqRsTuVwXyZ");
|
||||
expect(redacted).toContain("[REDACTED]");
|
||||
});
|
||||
|
||||
it("redacts ghp_/ghs_/gho_ GitHub tokens", () => {
|
||||
const text = "ghp_AbCdEfGhIjKlMnOpQrStUvWxYz0123 was leaked.";
|
||||
const redacted = redact(text);
|
||||
expect(redacted).not.toContain("ghp_AbCdEfGhIjKlMnOpQrStUvWxYz0123");
|
||||
expect(redacted).toContain("[REDACTED]");
|
||||
});
|
||||
|
||||
it("redacts github_pat_ fine-grained tokens", () => {
|
||||
const text = "Token: github_pat_AbCdEfGh01234567890123456789";
|
||||
const redacted = redact(text);
|
||||
expect(redacted).not.toContain("github_pat_AbCdEfGh01234567890123456789");
|
||||
expect(redacted).toContain("[REDACTED]");
|
||||
});
|
||||
|
||||
it("redacts Bearer tokens", () => {
|
||||
const text = "Authorization: Bearer aBcD1234.eFgH5678.iJkL9012-mNoP";
|
||||
const redacted = redact(text);
|
||||
expect(redacted).not.toContain("Bearer aBcD1234.eFgH5678.iJkL9012-mNoP");
|
||||
expect(redacted).toContain("[REDACTED]");
|
||||
});
|
||||
|
||||
it("redacts long base64-shaped strings", () => {
|
||||
const text = "Encoded: aGVsbG93b3JsZGFlaW91YWVpb3VhZWlvdWFlaW91YWVpb3U=";
|
||||
const redacted = redact(text);
|
||||
expect(redacted).toContain("[REDACTED]");
|
||||
});
|
||||
|
||||
it("leaves non-credential content unchanged byte-for-byte", () => {
|
||||
const text = "Normal message: visit https://example.com for docs.";
|
||||
const redacted = redact(text);
|
||||
expect(redacted).toBe(text);
|
||||
});
|
||||
|
||||
it("hasSecretPattern detects each known credential class", () => {
|
||||
expect(hasSecretPattern("sk-AbCdEfGhIjKlMnOpQ")).toBe(true);
|
||||
expect(hasSecretPattern("normal text without secrets")).toBe(false);
|
||||
expect(hasSecretPattern("")).toBe(false);
|
||||
});
|
||||
|
||||
it("exposes the pattern list (5 patterns per Velocity prior art)", () => {
|
||||
expect(SECRET_PATTERNS.length).toBe(5);
|
||||
});
|
||||
});
|
||||
|
||||
describe("M2b omitted-records classifier", () => {
|
||||
it("classifies Codex function_call → function_call_output", () => {
|
||||
const result = classifyCodexRecord({ payload: { type: "function_call" } });
|
||||
expect(result.kind).toBe("omitted");
|
||||
if (result.kind === "omitted") {
|
||||
expect(result.reason).toBe("function_call_output");
|
||||
}
|
||||
});
|
||||
|
||||
it("classifies Codex custom_tool_call → raw_tool_outputs", () => {
|
||||
const result = classifyCodexRecord({ payload: { type: "custom_tool_call" } });
|
||||
expect(result.kind).toBe("omitted");
|
||||
if (result.kind === "omitted") {
|
||||
expect(result.reason).toBe("raw_tool_outputs");
|
||||
}
|
||||
});
|
||||
|
||||
it("classifies Codex reasoning → reasoning_records", () => {
|
||||
const result = classifyCodexRecord({ payload: { type: "reasoning" } });
|
||||
expect(result.kind).toBe("omitted");
|
||||
if (result.kind === "omitted") {
|
||||
expect(result.reason).toBe("reasoning_records");
|
||||
}
|
||||
});
|
||||
|
||||
it("classifies Codex message+user-role → kept", () => {
|
||||
const result = classifyCodexRecord({ payload: { type: "message", role: "user" } });
|
||||
expect(result.kind).toBe("kept");
|
||||
});
|
||||
|
||||
it("classifies Codex message+unknown-role → omitted (reasoning)", () => {
|
||||
const result = classifyCodexRecord({ payload: { type: "message", role: "system" } });
|
||||
expect(result.kind).toBe("omitted");
|
||||
});
|
||||
|
||||
it("classifies Claude attachment → raw_tool_outputs", () => {
|
||||
const result = classifyClaudeRecord({ type: "attachment" });
|
||||
expect(result.kind).toBe("omitted");
|
||||
if (result.kind === "omitted") {
|
||||
expect(result.reason).toBe("raw_tool_outputs");
|
||||
}
|
||||
});
|
||||
|
||||
it("classifies Claude user|assistant → kept", () => {
|
||||
expect(classifyClaudeRecord({ type: "user" }).kind).toBe("kept");
|
||||
expect(classifyClaudeRecord({ type: "assistant" }).kind).toBe("kept");
|
||||
});
|
||||
|
||||
it("classifies Claude summary → reasoning_records", () => {
|
||||
const result = classifyClaudeRecord({ type: "summary" });
|
||||
expect(result.kind).toBe("omitted");
|
||||
if (result.kind === "omitted") {
|
||||
expect(result.reason).toBe("reasoning_records");
|
||||
}
|
||||
});
|
||||
|
||||
it("OmittedCounter accumulates per-class counts and active-classes list", () => {
|
||||
const counter = new OmittedCounter();
|
||||
counter.recordOmission("reasoning_records");
|
||||
counter.recordOmission("reasoning_records");
|
||||
counter.recordOmission("function_call_output");
|
||||
counter.recordRedaction();
|
||||
|
||||
expect(counter.counts.reasoning_records).toBe(2);
|
||||
expect(counter.counts.function_call_output).toBe(1);
|
||||
expect(counter.counts.redacted_secrets).toBe(1);
|
||||
expect(counter.counts.raw_tool_outputs).toBe(0);
|
||||
|
||||
expect(counter.activeClasses()).toEqual([
|
||||
"reasoning_records",
|
||||
"function_call_output",
|
||||
"redacted_secrets",
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("M2b runtime-detect", () => {
|
||||
it("detects Codex JSONL via response_item type marker", () => {
|
||||
const content = `{"type":"session_meta","payload":{"cwd":"/x"}}
|
||||
{"type":"response_item","payload":{"type":"message","role":"user","content":"hi"}}`;
|
||||
expect(detectRuntime(content)).toBe("codex");
|
||||
});
|
||||
|
||||
it("detects Codex via session_meta alone", () => {
|
||||
const content = `{"type":"session_meta","payload":{"cwd":"/x"}}`;
|
||||
expect(detectRuntime(content)).toBe("codex");
|
||||
});
|
||||
|
||||
it("detects Claude via top-level user/assistant type", () => {
|
||||
const content = `{"type":"customTitle","customTitle":"x","sessionId":"s"}
|
||||
{"type":"user","message":{"role":"user","content":"hi"},"sessionId":"s"}`;
|
||||
expect(detectRuntime(content)).toBe("claude-code");
|
||||
});
|
||||
|
||||
it("returns null on empty input", () => {
|
||||
expect(detectRuntime("")).toBe(null);
|
||||
});
|
||||
|
||||
it("returns null on non-JSONL garbage", () => {
|
||||
expect(detectRuntime("hello world\nfoo bar")).toBe(null);
|
||||
});
|
||||
|
||||
it("returns null on JSONL with no runtime markers (ambiguous)", () => {
|
||||
const content = `{"foo":"bar"}
|
||||
{"baz":"qux"}`;
|
||||
expect(detectRuntime(content)).toBe(null);
|
||||
});
|
||||
|
||||
it("does not use file extension or filename — pure content shape", () => {
|
||||
// Calling with a string that LOOKS like a filename should still return null.
|
||||
expect(detectRuntime("/tmp/fake.jsonl")).toBe(null);
|
||||
expect(detectRuntime("rollout-2026-04-23.jsonl")).toBe(null);
|
||||
});
|
||||
});
|
||||
|
||||
describe("M2b codex-jsonl-parser", () => {
|
||||
it("parses session_meta + response_item messages into StructuredTranscript", () => {
|
||||
const content = `{"type":"session_meta","payload":{"cwd":"/Users/wrandom/code/projects/openrig-hub","id":"abc-123"}}
|
||||
{"type":"response_item","timestamp":"2026-05-02T01:00:00Z","payload":{"type":"message","role":"user","content":"Hello world"}}
|
||||
{"type":"response_item","timestamp":"2026-05-02T01:00:01Z","payload":{"type":"message","role":"assistant","content":[{"type":"text","text":"Hi back"}]}}`;
|
||||
const result = parseCodexJsonl(content);
|
||||
expect(result.sessionMeta?.cwd).toBe("/Users/wrandom/code/projects/openrig-hub");
|
||||
expect(result.sessionMeta?.sessionId).toBe("abc-123");
|
||||
expect(result.messageCount).toBe(2);
|
||||
expect(result.messages[0]!.role).toBe("user");
|
||||
expect(result.messages[0]!.text).toBe("Hello world");
|
||||
expect(result.messages[1]!.role).toBe("assistant");
|
||||
expect(result.messages[1]!.text).toBe("Hi back");
|
||||
expect(result.lineCount).toBe(3);
|
||||
});
|
||||
|
||||
it("filters reasoning + function_call + custom_tool_call records and counts them", () => {
|
||||
const content = `{"type":"response_item","payload":{"type":"reasoning","content":"<thinking>"}}
|
||||
{"type":"response_item","payload":{"type":"function_call","arguments":"{\\"path\\":\\"/Users/wrandom/x.txt\\"}"}}
|
||||
{"type":"response_item","payload":{"type":"custom_tool_call","input":"some input"}}
|
||||
{"type":"response_item","payload":{"type":"message","role":"user","content":"kept"}}`;
|
||||
const result = parseCodexJsonl(content);
|
||||
expect(result.messageCount).toBe(1);
|
||||
expect(result.omittedCounts.reasoning_records).toBeGreaterThanOrEqual(1);
|
||||
expect(result.omittedCounts.function_call_output).toBe(1);
|
||||
expect(result.omittedCounts.raw_tool_outputs).toBe(1);
|
||||
});
|
||||
|
||||
it("counts compacted records separately and skips them from messages", () => {
|
||||
const content = `{"type":"compacted","payload":{}}
|
||||
{"type":"compacted","payload":{}}
|
||||
{"type":"response_item","payload":{"type":"message","role":"user","content":"msg"}}`;
|
||||
const result = parseCodexJsonl(content);
|
||||
expect(result.compactedCount).toBe(2);
|
||||
expect(result.messageCount).toBe(1);
|
||||
});
|
||||
|
||||
it("redacts credential patterns in message content", () => {
|
||||
const content = `{"type":"response_item","payload":{"type":"message","role":"user","content":"my token is sk-AbCdEfGhIjKlMnOpQrStUv now"}}`;
|
||||
const result = parseCodexJsonl(content);
|
||||
expect(result.messages[0]!.text).toContain("[REDACTED]");
|
||||
expect(result.messages[0]!.text).not.toContain("sk-AbCdEfGhIjKlMnOpQrStUv");
|
||||
expect(result.omittedCounts.redacted_secrets).toBe(1);
|
||||
});
|
||||
|
||||
it("skips malformed JSONL lines silently (matches Velocity prior art)", () => {
|
||||
const content = `{"type":"response_item","payload":{"type":"message","role":"user","content":"first"}}
|
||||
not valid json garbage line
|
||||
{"type":"response_item","payload":{"type":"message","role":"user","content":"second"}}`;
|
||||
const result = parseCodexJsonl(content);
|
||||
expect(result.messageCount).toBe(2);
|
||||
expect(result.lineCount).toBe(3);
|
||||
});
|
||||
|
||||
it("extracts paths from message content and tool args; sorts by frequency", () => {
|
||||
const content = `{"type":"response_item","payload":{"type":"message","role":"user","content":"see packages/cli/src/index.ts and packages/cli/src/index.ts again"}}
|
||||
{"type":"response_item","payload":{"type":"function_call","arguments":"{\\"path\\":\\"/Users/wrandom/code/projects/openrig-hub/README.md\\"}"}}`;
|
||||
const result = parseCodexJsonl(content);
|
||||
expect(result.paths.length).toBeGreaterThan(0);
|
||||
const indexEntry = result.paths.find((p) => p.path === "packages/cli/src/index.ts");
|
||||
expect(indexEntry?.count).toBeGreaterThanOrEqual(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("M2b claude-transcript-parser", () => {
|
||||
it("parses Claude user + assistant messages into StructuredTranscript", () => {
|
||||
const content = `{"type":"customTitle","customTitle":"M2b test","sessionId":"sess-1"}
|
||||
{"type":"user","message":{"role":"user","content":"hello"},"cwd":"/x","sessionId":"sess-1","timestamp":"2026-05-02T01:00:00Z"}
|
||||
{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"hi back"}]},"cwd":"/x","sessionId":"sess-1","timestamp":"2026-05-02T01:00:01Z"}`;
|
||||
const result = parseClaudeTranscript(content);
|
||||
expect(result.messageCount).toBe(2);
|
||||
expect(result.messages[0]!.role).toBe("user");
|
||||
expect(result.messages[0]!.text).toBe("hello");
|
||||
expect(result.messages[1]!.role).toBe("assistant");
|
||||
expect(result.messages[1]!.text).toBe("hi back");
|
||||
});
|
||||
|
||||
it("captures sessionMeta from the first record carrying cwd + sessionId", () => {
|
||||
const content = `{"type":"customTitle","customTitle":"x","sessionId":"sess-2"}
|
||||
{"type":"user","message":{"role":"user","content":"first"},"cwd":"/Users/wrandom/code","sessionId":"sess-2"}`;
|
||||
const result = parseClaudeTranscript(content);
|
||||
expect(result.sessionMeta?.cwd).toBe("/Users/wrandom/code");
|
||||
expect(result.sessionMeta?.sessionId).toBe("sess-2");
|
||||
});
|
||||
|
||||
it("filters attachments → raw_tool_outputs counter", () => {
|
||||
const content = `{"type":"user","message":{"role":"user","content":"see attached"},"cwd":"/x","sessionId":"s"}
|
||||
{"type":"attachment","attachment":{"path":"/Users/wrandom/x.txt","content":"file body"},"cwd":"/x","sessionId":"s"}`;
|
||||
const result = parseClaudeTranscript(content);
|
||||
expect(result.messageCount).toBe(1);
|
||||
expect(result.omittedCounts.raw_tool_outputs).toBe(1);
|
||||
});
|
||||
|
||||
it("redacts credential patterns in message content", () => {
|
||||
const content = `{"type":"user","message":{"role":"user","content":"my token is ghp_AbCdEfGhIjKlMnOpQrStUvWxYz0 now"},"cwd":"/x","sessionId":"s"}`;
|
||||
const result = parseClaudeTranscript(content);
|
||||
expect(result.messages[0]!.text).toContain("[REDACTED]");
|
||||
expect(result.omittedCounts.redacted_secrets).toBe(1);
|
||||
});
|
||||
|
||||
it("compactedCount is always 0 (Claude transcripts don't emit compaction records)", () => {
|
||||
const content = `{"type":"user","message":{"role":"user","content":"x"},"cwd":"/x","sessionId":"s"}`;
|
||||
const result = parseClaudeTranscript(content);
|
||||
expect(result.compactedCount).toBe(0);
|
||||
});
|
||||
|
||||
it("emits the SAME StructuredTranscript shape as the Codex parser (interface conformance)", () => {
|
||||
const codex = parseCodexJsonl(`{"type":"response_item","payload":{"type":"message","role":"user","content":"x"}}`);
|
||||
const claude = parseClaudeTranscript(`{"type":"user","message":{"role":"user","content":"x"},"cwd":"/x","sessionId":"s"}`);
|
||||
// Field-by-field check that both expose the same top-level keys.
|
||||
const codexKeys = Object.keys(codex).sort();
|
||||
const claudeKeys = Object.keys(claude).sort();
|
||||
expect(codexKeys).toEqual(claudeKeys);
|
||||
// Both must expose omittedCounts with all 4 enum keys.
|
||||
expect(Object.keys(codex.omittedCounts).sort()).toEqual([
|
||||
"function_call_output",
|
||||
"raw_tool_outputs",
|
||||
"reasoning_records",
|
||||
"redacted_secrets",
|
||||
]);
|
||||
expect(Object.keys(claude.omittedCounts).sort()).toEqual([
|
||||
"function_call_output",
|
||||
"raw_tool_outputs",
|
||||
"reasoning_records",
|
||||
"redacted_secrets",
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("M2b interaction: parse → redact → omitted-records counter chain", () => {
|
||||
it("counts redacted_secrets AND filters all 3 codex omitted classes in one parse", () => {
|
||||
const content = `{"type":"session_meta","payload":{"cwd":"/x"}}
|
||||
{"type":"response_item","payload":{"type":"reasoning"}}
|
||||
{"type":"response_item","payload":{"type":"function_call","arguments":"{}"}}
|
||||
{"type":"response_item","payload":{"type":"custom_tool_call","input":"{}"}}
|
||||
{"type":"response_item","payload":{"type":"message","role":"user","content":"my token sk-AbCdEfGhIjKlMnOpQrSt and more text"}}
|
||||
{"type":"response_item","payload":{"type":"message","role":"assistant","content":"clean reply"}}`;
|
||||
const result = parseCodexJsonl(content);
|
||||
expect(result.messageCount).toBe(2);
|
||||
expect(result.omittedCounts.reasoning_records).toBeGreaterThanOrEqual(1);
|
||||
expect(result.omittedCounts.function_call_output).toBe(1);
|
||||
expect(result.omittedCounts.raw_tool_outputs).toBe(1);
|
||||
expect(result.omittedCounts.redacted_secrets).toBe(1);
|
||||
// The user message is redacted but kept.
|
||||
expect(result.messages[0]!.text).toContain("[REDACTED]");
|
||||
// The assistant message is unchanged.
|
||||
expect(result.messages[1]!.text).toBe("clean reply");
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user