diff --git a/experiments/codemode/README.md b/experiments/codemode/README.md index eb30d4d12..6af5497f1 100644 --- a/experiments/codemode/README.md +++ b/experiments/codemode/README.md @@ -51,11 +51,14 @@ Calls and executions sharing one active target are serialized; `Promise.all` currently provides composition rather than parallel provider execution. `npm test` runs actual upstream and downstream stdio MCP against this checkout's -compiled runtime. The standalone experiment's eleven tests cover discovery parity, +compiled runtime. The standalone experiment's twelve tests cover discovery parity, original results/errors, bulk promise settlement, host capability absence, CPU deadline recovery, retained Evidence after an error or provider timeout, queued-call cancellation, unresolved promises and the -two-tool downstream boundary. These tests do not establish real native/Android/ +two-tool downstream boundary, guest heap exhaustion/reuse, and SDK-converted +schemas validated as [JSON Schema 2020-12](https://modelcontextprotocol.io/specification/2025-11-25/basic/index#schema-dialect). +Execution rejects delays above Node's signed 32-bit timer range instead of +allowing Node to coerce them to 1 ms. These tests do not establish real native/Android/ firmware provider behavior. A lifecycle regression revealed that the QuickJS job pump can allocate an additional context: the execution scope now owns every context at creation and releases them before the runtime. Increasing memory did @@ -134,6 +137,9 @@ independent decompiler correctness. Unparseable result text is reported explicit IDs in a truncated JSON fragment cannot satisfy the automated reference check. The Android projection includes the provider's overload index, so a supported signature with that field omitted fails this stronger assertion. +An omitted DEX descriptor differs from an explicit null, and booleans differ from +numeric counts. Native containing-procedure reports compare resolved identity; +the full analyzed procedure is verified separately. Firmware's run-local region UUIDs remain in answers but are excluded from parity against an independently acquired report. Classification evidence, explicit unknowns and coverage remain part of the requested fact checks. @@ -145,6 +151,11 @@ review is still needed for limitations and claims beyond these assertions. Grades record the grader source digest, and earlier strict grades are retained when the comparison changes. These are report parsers, not REA API aliases. +The completed serial cohort is in [agent-results.json](agent-results.json), with +source/configuration fingerprints, measured API usage and sampled cgroup stats. +The recommendation and migration boundaries are in [decision.md](decision.md). +Raw transcripts and original replay replies remain outside Git. + The supplied feedback is a research lead. Its Ghidra timing claim also changed auto-analysis options, so it does not isolate Code Mode's effect. Relevant primary sources: [Cloudflare Code Mode](https://blog.cloudflare.com/code-mode/), diff --git a/experiments/codemode/agent-results.json b/experiments/codemode/agent-results.json new file mode 100644 index 000000000..75f2ed230 --- /dev/null +++ b/experiments/codemode/agent-results.json @@ -0,0 +1,995 @@ +{ + "baseline_sha": "fb94d9dde0ccda83ae520ac6fb22b1d283562cf3", + "scope": "real_cli_requested_fact_parity_and_observed_references_not_independent_runtime_truth", + "runtime_source_commit": "22326e8199975f31d8b7c99aaf5fd64140fbe00e", + "artifact_ids": "corpus.json", + "resource_limits": { + "cpu_affinity": [15], + "cpu_quota_percent": 100, + "hard_memory_bytes": 4294967296, + "early_stop_bytes": 3221225472, + "tasks_max": 192 + }, + "notes": [ + "Profiles differ in required mode-specific instructions; original model-selected arguments may differ.", + "Two Android/managed repetitions are descriptive samples, without confidence intervals or a causal speed claim.", + "Managed classification container omissions fail a stronger complete-object assertion; remaining requested facts match.", + "Normal-MCP managed JSONL truncation leaves one ID unavailable to the complete-envelope reference parser.", + "Uncached input is input minus cached input, not a currency cost estimate." + ], + "trials": [ + { + "trial": "android-direct-5", + "task": "android", + "mode": "direct", + "artifact": "newpipe.apk", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 67.329, + "input_tokens": 107007, + "cached_input_tokens": 83712, + "uncached_input_tokens": 23295, + "output_tokens": 1670, + "completed_mcp_calls": 3, + "original_operations": [ + "inspect_android_package", + "search_android_classes", + "inspect_android_class" + ], + "fact_assertions": 6, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "7129418f2d4428d86cc737124b53636ba4a46de522e702e8861180fc175acb23", + "developer_instructions_sha256": "eaddc85aa5b141538ec280d640c62db799231b394679a17f2e6cc6bc2146a95c", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "3424dd6ba375f96d88fd0ac38102998948fa0fc46e08af8fae414572ba97a62f", + "peak_memory_bytes_sampled": 1336168448, + "peak_tasks_sampled": 108, + "cpu_stat_last_observed": { + "usage_usec": 23974584, + "user_usec": 21485565, + "system_usec": 2489019, + "nr_periods": 636, + "nr_throttled": 45, + "throttled_usec": 67840, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "android-bridge-7", + "task": "android", + "mode": "bridge", + "artifact": "newpipe.apk", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 48.231, + "input_tokens": 113256, + "cached_input_tokens": 91648, + "uncached_input_tokens": 21608, + "output_tokens": 1456, + "completed_mcp_calls": 3, + "original_operations": [ + "inspect_android_package", + "search_android_classes", + "inspect_android_class" + ], + "fact_assertions": 6, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "7129418f2d4428d86cc737124b53636ba4a46de522e702e8861180fc175acb23", + "developer_instructions_sha256": "247ca3314340f748d1479cc1a0a1f33a234231f17b5768044034bdfe517299c6", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "27e234285e493b50929c0f98938284771d0d4a3787045872f2f517b2297aefeb", + "peak_memory_bytes_sampled": 1659629568, + "peak_tasks_sampled": 126, + "cpu_stat_last_observed": { + "usage_usec": 27717810, + "user_usec": 25042151, + "system_usec": 2675658, + "nr_periods": 458, + "nr_throttled": 50, + "throttled_usec": 73614, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "managed-bridge-2", + "task": "managed", + "mode": "bridge", + "artifact": "sharex.dll", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 73.145, + "input_tokens": 116974, + "cached_input_tokens": 92672, + "uncached_input_tokens": 24302, + "output_tokens": 2065, + "completed_mcp_calls": 3, + "original_operations": [ + "inspect_managed_artifact", + "inspect_managed_native_boundaries" + ], + "fact_assertions": 7, + "fact_mismatches": ["classification"], + "strict_grade_passed": false, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "fe5a74687126fca7aee78b3ba51f03401983ffeae593924b7c738d50d0f04e61", + "developer_instructions_sha256": "247ca3314340f748d1479cc1a0a1f33a234231f17b5768044034bdfe517299c6", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "06cbdda60ac930d00ade7e115cd4ba234527e297e6cfd931eedc175e00efa1c7", + "peak_memory_bytes_sampled": 939290624, + "peak_tasks_sampled": 103, + "cpu_stat_last_observed": { + "usage_usec": 12114716, + "user_usec": 9701516, + "system_usec": 2413200, + "nr_periods": 630, + "nr_throttled": 16, + "throttled_usec": 15944, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "managed-direct-2", + "task": "managed", + "mode": "direct", + "artifact": "sharex.dll", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 33.15, + "input_tokens": 108096, + "cached_input_tokens": 75776, + "uncached_input_tokens": 32320, + "output_tokens": 2315, + "completed_mcp_calls": 2, + "original_operations": [ + "inspect_managed_artifact", + "inspect_managed_native_boundaries" + ], + "fact_assertions": 7, + "fact_mismatches": [], + "strict_grade_passed": false, + "cited_ids_observed": false, + "unreadable_result_texts": 1, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "fe5a74687126fca7aee78b3ba51f03401983ffeae593924b7c738d50d0f04e61", + "developer_instructions_sha256": "eaddc85aa5b141538ec280d640c62db799231b394679a17f2e6cc6bc2146a95c", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "5d6957541bd979252d019817e631f5d889ff2dc3efb7f03b9de1033fe34d8430", + "peak_memory_bytes_sampled": 688373760, + "peak_tasks_sampled": 108, + "cpu_stat_last_observed": { + "usage_usec": 5863642, + "user_usec": 4359666, + "system_usec": 1503976, + "nr_periods": 282, + "nr_throttled": 4, + "throttled_usec": 6216, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "electron-inventory-direct-1", + "task": "electron-inventory", + "mode": "direct", + "artifact": "obsidian.asar", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 89.494, + "input_tokens": 266121, + "cached_input_tokens": 238848, + "uncached_input_tokens": 27273, + "output_tokens": 3215, + "completed_mcp_calls": 3, + "original_operations": [ + "open_binary", + "inspect_artifact", + "close_binary" + ], + "fact_assertions": 7, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "22941dc5d77ce06824e40b01abd3f725aa773d131648d7bba2e607b410ccbef9", + "developer_instructions_sha256": "eaddc85aa5b141538ec280d640c62db799231b394679a17f2e6cc6bc2146a95c", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "fcd0df1a3ae0b5103143addaa4922323354ddee7ba3d0e3020de64f05b6fe3d3", + "peak_memory_bytes_sampled": 703275008, + "peak_tasks_sampled": 115, + "cpu_stat_last_observed": { + "usage_usec": 7720494, + "user_usec": 5691854, + "system_usec": 2028640, + "nr_periods": 782, + "nr_throttled": 7, + "throttled_usec": 15453, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "electron-inventory-bridge-1", + "task": "electron-inventory", + "mode": "bridge", + "artifact": "obsidian.asar", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 145.577, + "input_tokens": 462809, + "cached_input_tokens": 407040, + "uncached_input_tokens": 55769, + "output_tokens": 5624, + "completed_mcp_calls": 8, + "original_operations": [ + "open_binary", + "inspect_artifact", + "close_binary", + "open_binary", + "inspect_artifact", + "close_binary" + ], + "fact_assertions": 7, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "22941dc5d77ce06824e40b01abd3f725aa773d131648d7bba2e607b410ccbef9", + "developer_instructions_sha256": "247ca3314340f748d1479cc1a0a1f33a234231f17b5768044034bdfe517299c6", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "ab7e25e4345a0e9bba6a06bb0a79aa39e1bf1308b6a66f8173a62bbc9e2d9fc9", + "peak_memory_bytes_sampled": 1029238784, + "peak_tasks_sampled": 112, + "cpu_stat_last_observed": { + "usage_usec": 14650718, + "user_usec": 12106416, + "system_usec": 2544302, + "nr_periods": 1224, + "nr_throttled": 24, + "throttled_usec": 30318, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "firmware-bridge-1", + "task": "firmware", + "mode": "bridge", + "artifact": "openwrt.bin", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 81.944, + "input_tokens": 152886, + "cached_input_tokens": 126976, + "uncached_input_tokens": 25910, + "output_tokens": 2192, + "completed_mcp_calls": 3, + "original_operations": ["inspect_firmware_regions"], + "fact_assertions": 5, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "93c94e26dc4a4b1985b3f8c4246ca5aff886f75dc2349a717ee940243dc73b80", + "developer_instructions_sha256": "247ca3314340f748d1479cc1a0a1f33a234231f17b5768044034bdfe517299c6", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "a5604d0d3140f6232851a81a952e51daf43c8e7f12ba062fac68a8fae0bef708", + "peak_memory_bytes_sampled": 927666176, + "peak_tasks_sampled": 104, + "cpu_stat_last_observed": { + "usage_usec": 11575373, + "user_usec": 9366624, + "system_usec": 2208749, + "nr_periods": 744, + "nr_throttled": 18, + "throttled_usec": 26000, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "firmware-direct-1", + "task": "firmware", + "mode": "direct", + "artifact": "openwrt.bin", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 30.161, + "input_tokens": 103224, + "cached_input_tokens": 74752, + "uncached_input_tokens": 28472, + "output_tokens": 1761, + "completed_mcp_calls": 1, + "original_operations": ["inspect_firmware_regions"], + "fact_assertions": 5, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "93c94e26dc4a4b1985b3f8c4246ca5aff886f75dc2349a717ee940243dc73b80", + "developer_instructions_sha256": "eaddc85aa5b141538ec280d640c62db799231b394679a17f2e6cc6bc2146a95c", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "c0bbd1aec08c9c30061f92064355c3213f680fab545a050271096e85db771a71", + "peak_memory_bytes_sampled": 686739456, + "peak_tasks_sampled": 109, + "cpu_stat_last_observed": { + "usage_usec": 5700866, + "user_usec": 4167696, + "system_usec": 1533169, + "nr_periods": 266, + "nr_throttled": 6, + "throttled_usec": 3978, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "native-direct-1", + "task": "native", + "mode": "direct", + "artifact": "jq", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 327.15, + "input_tokens": 151343, + "cached_input_tokens": 122368, + "uncached_input_tokens": 28975, + "output_tokens": 2458, + "completed_mcp_calls": 4, + "original_operations": [ + "open_binary", + "resolve_containing_procedure", + "analyze_function", + "close_binary" + ], + "fact_assertions": 7, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "234bfe93d09f9c76c1a64383c11cf7c174eafa4e4ad563124f705a7d5c8920aa", + "developer_instructions_sha256": "eaddc85aa5b141538ec280d640c62db799231b394679a17f2e6cc6bc2146a95c", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "681f38e4ba9b441e7816e868b30e347005fe074907349cd0022fc44302c5b14a", + "peak_memory_bytes_sampled": 1328496640, + "peak_tasks_sampled": 109, + "cpu_stat_last_observed": { + "usage_usec": 263737951, + "user_usec": 256864795, + "system_usec": 6873156, + "nr_periods": 3192, + "nr_throttled": 456, + "throttled_usec": 756678, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "native-bridge-1", + "task": "native", + "mode": "bridge", + "artifact": "jq", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 334.642, + "input_tokens": 285961, + "cached_input_tokens": 245504, + "uncached_input_tokens": 40457, + "output_tokens": 2129, + "completed_mcp_calls": 4, + "original_operations": [ + "open_binary", + "analyze_function", + "close_binary" + ], + "fact_assertions": 7, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "234bfe93d09f9c76c1a64383c11cf7c174eafa4e4ad563124f705a7d5c8920aa", + "developer_instructions_sha256": "247ca3314340f748d1479cc1a0a1f33a234231f17b5768044034bdfe517299c6", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "6c64953708df4a06b69fb57e5d70141147e4495adabd7340e87e727fcd340742", + "peak_memory_bytes_sampled": 1441775616, + "peak_tasks_sampled": 123, + "cpu_stat_last_observed": { + "usage_usec": 271095966, + "user_usec": 263888347, + "system_usec": 7207618, + "nr_periods": 3267, + "nr_throttled": 450, + "throttled_usec": 784719, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "android-bridge-8", + "task": "android", + "mode": "bridge", + "artifact": "newpipe.apk", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 134.555, + "input_tokens": 159370, + "cached_input_tokens": 134400, + "uncached_input_tokens": 24970, + "output_tokens": 3250, + "completed_mcp_calls": 7, + "original_operations": [ + "inspect_android_package", + "search_android_classes", + "inspect_android_class", + "inspect_android_package", + "inspect_android_package", + "search_android_classes", + "inspect_android_class" + ], + "fact_assertions": 6, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "7129418f2d4428d86cc737124b53636ba4a46de522e702e8861180fc175acb23", + "developer_instructions_sha256": "247ca3314340f748d1479cc1a0a1f33a234231f17b5768044034bdfe517299c6", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "f1743a341d334bd41760b54980aa70d5f3c8c1aed57c3ae9e470be63449c594d", + "peak_memory_bytes_sampled": 1659744256, + "peak_tasks_sampled": 115, + "cpu_stat_last_observed": { + "usage_usec": 29619173, + "user_usec": 26495167, + "system_usec": 3124005, + "nr_periods": 1229, + "nr_throttled": 60, + "throttled_usec": 84762, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "android-direct-6", + "task": "android", + "mode": "direct", + "artifact": "newpipe.apk", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 71.624, + "input_tokens": 153031, + "cached_input_tokens": 117504, + "uncached_input_tokens": 35527, + "output_tokens": 1501, + "completed_mcp_calls": 3, + "original_operations": [ + "inspect_android_package", + "search_android_classes", + "inspect_android_class" + ], + "fact_assertions": 6, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "7129418f2d4428d86cc737124b53636ba4a46de522e702e8861180fc175acb23", + "developer_instructions_sha256": "eaddc85aa5b141538ec280d640c62db799231b394679a17f2e6cc6bc2146a95c", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "b8d0422eec6bfd31a9a6ed79ac29e9487c2e68c47ff200904ec239daee9720c3", + "peak_memory_bytes_sampled": 1348767744, + "peak_tasks_sampled": 115, + "cpu_stat_last_observed": { + "usage_usec": 23033331, + "user_usec": 20566262, + "system_usec": 2467069, + "nr_periods": 674, + "nr_throttled": 40, + "throttled_usec": 61573, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "managed-direct-3", + "task": "managed", + "mode": "direct", + "artifact": "sharex.dll", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 75.229, + "input_tokens": 108593, + "cached_input_tokens": 75776, + "uncached_input_tokens": 32817, + "output_tokens": 2774, + "completed_mcp_calls": 2, + "original_operations": [ + "inspect_managed_artifact", + "inspect_managed_native_boundaries" + ], + "fact_assertions": 7, + "fact_mismatches": [], + "strict_grade_passed": false, + "cited_ids_observed": false, + "unreadable_result_texts": 1, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "fe5a74687126fca7aee78b3ba51f03401983ffeae593924b7c738d50d0f04e61", + "developer_instructions_sha256": "eaddc85aa5b141538ec280d640c62db799231b394679a17f2e6cc6bc2146a95c", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "64f0769b7ad83f8d86c005a48de7a0cf397043d77378983fabfaa226dcb70f87", + "peak_memory_bytes_sampled": 651026432, + "peak_tasks_sampled": 110, + "cpu_stat_last_observed": { + "usage_usec": 6252846, + "user_usec": 4688766, + "system_usec": 1564079, + "nr_periods": 635, + "nr_throttled": 7, + "throttled_usec": 11469, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "managed-bridge-3", + "task": "managed", + "mode": "bridge", + "artifact": "sharex.dll", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 40.609, + "input_tokens": 125519, + "cached_input_tokens": 96000, + "uncached_input_tokens": 29519, + "output_tokens": 2274, + "completed_mcp_calls": 3, + "original_operations": [ + "inspect_managed_artifact", + "inspect_managed_native_boundaries" + ], + "fact_assertions": 7, + "fact_mismatches": ["classification"], + "strict_grade_passed": false, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "fe5a74687126fca7aee78b3ba51f03401983ffeae593924b7c738d50d0f04e61", + "developer_instructions_sha256": "247ca3314340f748d1479cc1a0a1f33a234231f17b5768044034bdfe517299c6", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "5fbc3cf0b41224f2ef44ab996b174c8a48fe07a818ac2ad334ef59e5ddf26de6", + "peak_memory_bytes_sampled": 1005625344, + "peak_tasks_sampled": 113, + "cpu_stat_last_observed": { + "usage_usec": 11185145, + "user_usec": 9300437, + "system_usec": 1884707, + "nr_periods": 366, + "nr_throttled": 21, + "throttled_usec": 24147, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "electron-inventory-guided-bridge-1", + "task": "electron-inventory", + "mode": "bridge", + "artifact": "obsidian.asar", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 85.909, + "input_tokens": 101994, + "cached_input_tokens": 61440, + "uncached_input_tokens": 40554, + "output_tokens": 2563, + "completed_mcp_calls": 3, + "original_operations": [ + "open_binary", + "inspect_artifact", + "close_binary" + ], + "fact_assertions": 7, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "f070cdf721805d590c1bd5255ecf1f647ea165885cd5052af35c30ba531da815", + "developer_instructions_sha256": "247ca3314340f748d1479cc1a0a1f33a234231f17b5768044034bdfe517299c6", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "566391b8a6db73228fec00dc7cbf0eaf56ed899178a405d5d8f0c8002fb05d89", + "peak_memory_bytes_sampled": 1007996928, + "peak_tasks_sampled": 109, + "cpu_stat_last_observed": { + "usage_usec": 13304542, + "user_usec": 10842042, + "system_usec": 2462499, + "nr_periods": 787, + "nr_throttled": 19, + "throttled_usec": 27407, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + }, + { + "trial": "electron-inventory-guided-direct-1", + "task": "electron-inventory", + "mode": "direct", + "artifact": "obsidian.asar", + "requested_model": "gpt-5.6-terra", + "reasoning": "high", + "codex_version": "codex-cli 0.161.0", + "wall_seconds": 67.54, + "input_tokens": 263484, + "cached_input_tokens": 221952, + "uncached_input_tokens": 41532, + "output_tokens": 2499, + "completed_mcp_calls": 4, + "original_operations": [ + "open_binary", + "inspect_artifact", + "inspect_artifact", + "close_binary" + ], + "fact_assertions": 7, + "fact_mismatches": [], + "strict_grade_passed": true, + "cited_ids_observed": true, + "unreadable_result_texts": 0, + "grader_sha256": "707e4eef05634db5075ac64382500b9eea03b38f828358315e54381f9d4a8979", + "prompt_sha256": "f070cdf721805d590c1bd5255ecf1f647ea165885cd5052af35c30ba531da815", + "developer_instructions_sha256": "eaddc85aa5b141538ec280d640c62db799231b394679a17f2e6cc6bc2146a95c", + "provider_settings_sha256": "1db4e19adcb32bb64a3baad58a1b4ef04b52ba648bab29678693f9ab8dbdabb4", + "experiment_sources_sha256": { + "agent-trial.py": "3c96d0f7b4b3dd02d168258d8aedd79049bb8647f1ebb174b056bd8eaa582502", + "bridge.mjs": "c1afb88a3045f233006826ffd466590b33027cc7b95f9c6c551ba1aa93fa4727", + "package-lock.json": "6dab2b2dba5101992e9a881ce1820475bd304511dc44362bb1614b08e9b37a87" + }, + "events_sha256": "564711cf28211bb7059a04b919f1b6925a816ded3e2dbeb9678382568f700e58", + "peak_memory_bytes_sampled": 728817664, + "peak_tasks_sampled": 114, + "cpu_stat_last_observed": { + "usage_usec": 8708686, + "user_usec": 6609914, + "system_usec": 2098771, + "nr_periods": 597, + "nr_throttled": 12, + "throttled_usec": 19564, + "nr_bursts": 0, + "burst_usec": 0 + }, + "resource_abort": null, + "resource_events_last_observed": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } + } + ] +} diff --git a/experiments/codemode/bridge.mjs b/experiments/codemode/bridge.mjs index 1304302eb..e58a77435 100644 --- a/experiments/codemode/bridge.mjs +++ b/experiments/codemode/bridge.mjs @@ -8,6 +8,18 @@ import { compile } from "json-schema-to-typescript"; import { newQuickJSWASMModule, Scope } from "quickjs-emscripten"; import { z } from "zod"; +// Node maps larger delays to 1 ms. Reject them before creating a timer. +const executeInputSchema = z.strictObject({ + code: z.string(), + timeout_ms: z.number().int().positive().max(2_147_483_647).optional(), + memory_bytes: z + .number() + .int() + .positive() + .max(128 * 1024 * 1024) + .optional(), +}); + const digest = (value) => createHash("sha256").update(JSON.stringify(value)).digest("hex"); @@ -143,7 +155,8 @@ export class CodeModeBridge { /** Serialize scripts sharing the active target; no implicit batch races. */ execute(input, options = {}) { - const work = this.queue.then(() => this.run(input, options)); + const parsed = executeInputSchema.parse(input); + const work = this.queue.then(() => this.run(parsed, options)); this.queue = work.catch(() => undefined); return work; } @@ -358,16 +371,7 @@ export function createBridgeServer(bridge) { { description: "Run a JavaScript async function body with tools.(original_arguments), json(reply) and return. Await every call. json(reply) returns the original REA value: Evidence has normalized_result, evidence_id and limitations; session values have result. Select the needed normalized_result fields inside code before returning them with Evidence IDs and limitations. One upstream session persists. Calls run in order, including Promise.all. All original REA effects, validation and retention apply; projections are caller selections, not new Evidence. No host IO or imports. Call audit and Evidence IDs survive script errors. Default deadline 120s and guest memory 128 MiB; use longer deadline for native analysis.", - inputSchema: z.strictObject({ - code: z.string(), - timeout_ms: z.number().int().positive().optional(), - memory_bytes: z - .number() - .int() - .positive() - .max(128 * 1024 * 1024) - .optional(), - }), + inputSchema: executeInputSchema, annotations: { readOnlyHint: false, destructiveHint: true, diff --git a/experiments/codemode/bridge.test.mjs b/experiments/codemode/bridge.test.mjs index 0716a4bff..c436232e3 100644 --- a/experiments/codemode/bridge.test.mjs +++ b/experiments/codemode/bridge.test.mjs @@ -3,6 +3,7 @@ import { after, before, test } from "node:test"; import { fileURLToPath } from "node:url"; import { Client } from "@modelcontextprotocol/client"; import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import Ajv2020 from "ajv/dist/2020.js"; import { CodeModeBridge, connectUpstream, json } from "./bridge.mjs"; const entrypoint = fileURLToPath( @@ -131,6 +132,19 @@ test("CPU deadline interrupts runaway code and the next execution works", async assert.equal((await bridge.execute({ code: "return 42;" })).value, 42); }); +test("guest allocation pressure fails within its heap budget and permits reuse", async () => { + const result = await bridge.execute({ + code: "return new Array(1_000_000).fill('x');", + memory_bytes: 2 * 1024 * 1024, + }); + assert.match(result.bridge_error.message, /memory/i); + assert.deepEqual(result.calls, []); + assert.equal((await bridge.execute({ code: "return 42;" })).value, 42); + assert.throws(() => + bridge.execute({ code: "return 1;", timeout_ms: 2_147_483_648 }), + ); +}); + test("error after successful call preserves original Evidence reference", async () => { const result = await bridge.execute({ code: `await tools.analyze_javascript_application(${JSON.stringify(analysisArgs)}); throw new Error('after-call');`, @@ -235,6 +249,33 @@ test("bridge's two tools are usable through actual downstream stdio MCP", async catalog.tools.map((tool) => tool.name), ["rea_search", "rea_execute"], ); + const ajv = new Ajv2020({ strict: true }); + const byName = new Map(catalog.tools.map((tool) => [tool.name, tool])); + for (const tool of catalog.tools) { + assert.equal( + tool.inputSchema.$schema ?? + "https://json-schema.org/draft/2020-12/schema", + "https://json-schema.org/draft/2020-12/schema", + ); + assert.equal( + ajv.validateSchema(tool.inputSchema), + true, + ajv.errorsText(), + ); + } + const validate = ajv.compile(byName.get("rea_execute").inputSchema); + assert.equal( + validate({ code: "return 1;", timeout_ms: 2_147_483_647 }), + true, + ); + assert.equal( + validate({ code: "return 1;", timeout_ms: 2_147_483_648 }), + false, + ); + assert.equal( + validate({ code: "return 1;", memory_bytes: 128 * 1024 * 1024 + 1 }), + false, + ); const found = json( await client.callTool({ name: "rea_search", @@ -256,6 +297,16 @@ test("bridge's two tools are usable through actual downstream stdio MCP", async arguments: { code: 7 }, }); assert.equal(invalid.isError, true); + const oversized = await client.callTool({ + name: "rea_execute", + arguments: { code: "return 1;", timeout_ms: 2_147_483_648 }, + }); + assert.equal(oversized.isError, true); + const oversizedMemory = await client.callTool({ + name: "rea_execute", + arguments: { code: "return 1;", memory_bytes: 128 * 1024 * 1024 + 1 }, + }); + assert.equal(oversizedMemory.isError, true); } finally { await client.close(); } diff --git a/experiments/codemode/decision.md b/experiments/codemode/decision.md new file mode 100644 index 000000000..8d1dc8652 --- /dev/null +++ b/experiments/codemode/decision.md @@ -0,0 +1,87 @@ +# Decision from the frozen Code Mode experiment + +Keep REA's existing CLI/MCP contracts as the default. Improve Evidence-aware +workflow guidance and retain this two-tool bridge as an optional experiment. +The measured Codex profiles do not justify a broad API refactor or a default +switch to the external executor. + +## What supports that choice + +The [serial cohort](agent-results.json) contains sixteen real Codex CLI trials, +configured with `gpt-5.6-terra` high, across Android, managed PE/CLI, Electron ASAR +inventory, firmware and native Ghidra analysis. Twelve pass the strict requested +fact/reference checks. Two bridge managed answers omit the classification's +container field; two normal-MCP managed transcripts truncate a large producer +envelope, preventing complete-envelope verification of one cited ID. Their +remaining requested facts match the real-provider oracle. The complete-container +assertion is stronger than the question's general classification wording. + +The bridge's Android repetitions take 48.231 and 134.555 seconds, compared with +67.329 and 71.624 for normal MCP. The faster bridge trial composes three original +operations once; the slower one performs seven original calls across five scripts. +The faster trial uses typed discovery; this does not make the model reliably +choose that workflow. Uncached input is lower in both bridge repetitions; +output and latency vary. This is not a stable overall efficiency win. + +Managed bridge latency also varies, and its two answers omit the same field. +Firmware's one-operation question takes 81.944 seconds with the bridge versus +30.161 directly. Ghidra takes 334.642 versus 327.150 seconds, with sampled aggregate +CPU times of 271.096 and 263.738 seconds. The bridge performs three original +operations versus four directly, deriving containing identity from the analyzed +procedure. Native analysis cost remains dominant with unchanged default settings. + +Obsidian inventory exposes a useful contract boundary: nested Evidence types leave +their normalized payload generic. The model initially returns the whole inventory +and repeats analysis. Giving both profiles the concrete manifest/occurrences +layout yields a single focused bridge workflow: 101994 total input tokens versus +263484 directly, but 40554 versus 41532 uncached tokens and 85.909 versus 67.540 +seconds. Clear producer/layout guidance matters; total context compression alone +does not establish lower cost or latency. Cache pricing is not estimated here. + +Deterministic replays demonstrate large projection savings across real providers. +They measure a known script, not the model's ability to discover and write it. +SQLite WASM fails the frozen WABT parser on an unpadded DataCount row; Obsidian's +full AST analysis times out with a retained partial reference. Inventory does not +replace AST analysis. Neither failure is counted as a Code Mode success. + +## Migration scope + +Current users retain original operation names, arguments, provider choices, +Evidence and session behavior. A client with equivalent execution support can use +the existing registration and focused workflows. The installed Codex transcript +does not expose its executor source, so its exact internal composition is unknown. + +For clients that need the bridge, use a separate opt-in profile for an investigation. +The bridge owns one original REA connection and looks up live contracts by name. +Its session and Evidence ledger are separate from another REA registration; do +not mix target lifecycle calls or same-session IDs across those registrations. +Derived TypeScript assists code authoring; original JSON schemas remain available. +Original producer IDs, links, errors and partial-reference IDs survive auditing; +caller projections do not become new Evidence records. + +The outer execution tool combines original effects and client requirements. +This prototype has not forwarded roots/elicitation/sampling or preserved a client's +per-original-operation approval flow. That capability boundary must be addressed +before offering it as a production replacement. The guest's lack of host IO does +not remove effects reachable through original REA tools. + +An additive producer-description improvement explaining nested payload layout is +a smaller next candidate than changing names or inventing another workflow API. +Original inputs and output defaults should remain intact. Provider analysis +settings must be evaluated separately from discovery and projection improvements. + +## Verification limits + +These are descriptive samples with matching questions/provider settings per pair, +mode-specific instructions and model-selected arguments. They do not provide +confidence intervals, independent decompiler truth, Android/.NET runtime behavior, +full native-function coverage, complete firmware reconstruction or Windows/IDA +provider verification. Citations are checked against observed producer values and +links, not resolved individually after session exit. + +Sixteen trials finish without sampled memory/task-limit events or resource aborts. +Their maximum sampled memory is 1659744256 bytes (about 1.55 GiB), with at most +126 tasks, under one CPU and a 4 GiB hard ceiling/3 GiB early stop. Copied profiles +are removed. Twelve actual stdio tests additionally cover cancellation, partial +Evidence readback, guest allocation failure/recovery, and advertised schema +validation. The earlier 2 GiB build-scope kill remains a documented failed build. diff --git a/experiments/codemode/grade-trial.py b/experiments/codemode/grade-trial.py index faa6027bf..f18a9ccc9 100644 --- a/experiments/codemode/grade-trial.py +++ b/experiments/codemode/grade-trial.py @@ -6,6 +6,22 @@ import json from pathlib import Path +def required_field(row, key): + return row[key] if key in row else {'missing_field': key} + + +def same_fact(actual, expected): + if isinstance(actual, bool) or isinstance(expected, bool): + return type(actual) is type(expected) and actual == expected + if isinstance(expected, dict): + return isinstance(actual, dict) and actual.keys() == expected.keys() and all( + same_fact(actual[key], value) for key, value in expected.items()) + if isinstance(expected, list): + return isinstance(actual, list) and len(actual) == len(expected) and all( + same_fact(a, b) for a, b in zip(actual, expected)) + return actual == expected + + def evidence_ids(value): if isinstance(value, str): return {value} if value.startswith('ev_') and len(value) == 67 else set() @@ -33,9 +49,16 @@ def read_answer(path): def requested_facts(value, task): if task == 'native': - return {field: value.get(field) for field in [ + facts = {field: value.get(field) for field in [ 'artifact_sha256', 'entry_address', 'containing_procedure', 'procedure', 'pseudocode_prefix', 'caller_count', 'callee_count']} + containing = facts['containing_procedure'] + if isinstance(containing, dict) and containing.get('found', True) is True and \ + containing.get('query_address', facts['entry_address']) == facts['entry_address']: + identity = containing.get('procedure', containing) + if isinstance(identity, dict): + facts['containing_procedure'] = {key: identity.get(key) for key in ['address', 'name']} + return facts if task == 'electron-inventory': facts = {field: value.get(field) for field in [ 'artifact_sha256', 'manifest', 'entry_kind_counts', 'requested_paths', @@ -100,9 +123,12 @@ def requested_facts(value, task): if task == 'managed': def selected(field, keys): row = value.get(field) - return {key: row.get(key) for key in keys} if isinstance(row, dict) else row + return {key: required_field(row, key) for key in keys} if isinstance(row, dict) else row coverage = value.get('coverage') if isinstance(coverage, dict): + coverage = {'managed_artifact': coverage.get('managed_artifact'), + 'native_boundaries': coverage.get('native_boundaries', + coverage.get('managed_native_boundaries'))} coverage = {key: {field: row.get(field) for field in ['state', 'issues']} if isinstance(row, dict) else row for key, row in coverage.items() if key in ['managed_artifact', 'native_boundaries']} @@ -119,7 +145,7 @@ def requested_facts(value, task): 'artifact_sha256', 'package_name', 'version_name', 'version_code', 'matches', 'onCreate']} methods = facts['onCreate'] if isinstance(methods, list) and all(isinstance(row, dict) for row in methods): - facts['onCreate'] = [{key: row.get(key) for key in [ + facts['onCreate'] = [{key: required_field(row, key) for key in [ 'reported_signature', 'overload_index', 'dex_descriptor']} for row in methods] return facts @@ -184,7 +210,7 @@ def main(): actual_facts = requested_facts(answer, args.task) for field, wanted in expected.items(): actual = actual_facts[field] - if actual != wanted: + if not same_fact(actual, wanted): failures.append({'kind': 'fact_mismatch', 'field': field, 'expected': wanted, 'actual': actual}) observed = set() diff --git a/experiments/codemode/package-lock.json b/experiments/codemode/package-lock.json index 02a54401d..d940aa241 100644 --- a/experiments/codemode/package-lock.json +++ b/experiments/codemode/package-lock.json @@ -15,6 +15,9 @@ "quickjs-emscripten": "0.32.0", "zod": "4.3.6" }, + "devDependencies": { + "ajv": "8.20.0" + }, "engines": { "node": "24.18.0" } @@ -161,6 +164,23 @@ "integrity": "sha512-48NWfiY6oNcKOh47Z4h+sIvwcQR91h6KPhaYKgeMIrkT2jV82Gy90SErj60vW8jQbeMSOSbhIv3bZ7JoqPPVhA==", "license": "MIT" }, + "node_modules/ajv": { + "version": "8.20.0", + "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", + "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-deep-equal": "^3.1.3", + "fast-uri": "^3.0.1", + "json-schema-traverse": "^1.0.0", + "require-from-string": "^2.0.2" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/epoberezkin" + } + }, "node_modules/argparse": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz", @@ -222,6 +242,30 @@ "node": ">=18.0.0" } }, + "node_modules/fast-deep-equal": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz", + "integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==", + "dev": true, + "license": "MIT" + }, + "node_modules/fast-uri": { + "version": "3.1.8", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.8.tgz", + "integrity": "sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "license": "BSD-3-Clause" + }, "node_modules/fdir": { "version": "6.5.0", "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", @@ -307,6 +351,13 @@ "node": ">=16.0.0" } }, + "node_modules/json-schema-traverse": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", + "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", + "dev": true, + "license": "MIT" + }, "node_modules/lodash": { "version": "4.18.1", "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.18.1.tgz", @@ -392,6 +443,16 @@ "@jitl/quickjs-ffi-types": "0.32.0" } }, + "node_modules/require-from-string": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", + "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/shebang-command": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", diff --git a/experiments/codemode/package.json b/experiments/codemode/package.json index d1edfd3f7..705b69c14 100644 --- a/experiments/codemode/package.json +++ b/experiments/codemode/package.json @@ -19,5 +19,8 @@ "json-schema-to-typescript": "16.0.0", "quickjs-emscripten": "0.32.0", "zod": "4.3.6" + }, + "devDependencies": { + "ajv": "8.20.0" } } diff --git a/experiments/codemode/questions/electron-inventory-guided.txt b/experiments/codemode/questions/electron-inventory-guided.txt new file mode 100644 index 000000000..319340690 --- /dev/null +++ b/experiments/codemode/questions/electron-inventory-guided.txt @@ -0,0 +1 @@ +Statically inventory {{artifact_path}} using only the registered REA tools. Open this ASAR as an archive, inspect its artifact inventory, and close the session. Report exact target SHA-256, the complete inventory manifest, counts of every entry_kind across all occurrences, the original occurrence records for root paths package.json, main.js and preload.js if present, all integrity contradictions, inspection coverage, unexplored branches, real Evidence IDs and limitations. This question asks about package bytes and inventory, not AST interpretation or runtime behavior. Do not execute target code, analyze all JavaScript, extract files, launch a GUI, use shell/file tools/network or other agents. When a client JavaScript executor is available, compose original calls in dependency order and return only requested fields from normalized_result; the full inventory is the Evidence nested in the inspect_artifact inventory_artifact substep. Use Evidence.subject.digest.sha256 for the target identity and retain both inspection and inventory Evidence IDs. Final JSON fields: artifact_sha256, manifest, entry_kind_counts, requested_paths, integrity_contradictions, coverage, unexplored_branches, evidence_ids, limitations. Stop when these facts and cleanup are supported. Use the original open_binary, inspect_artifact and close_binary operations, with integrity_policy=fail. The inventory Evidence has normalized_result.manifest and normalized_result.occurrences; each occurrence has entry_kind and logical_path. Count the complete occurrences array and filter those three exact root logical_path values. Keep the complete inventory inside the executor when one is available; return the requested projection. diff --git a/experiments/codemode/results.md b/experiments/codemode/results.md index b96138f5d..cd2c967df 100644 --- a/experiments/codemode/results.md +++ b/experiments/codemode/results.md @@ -295,3 +295,73 @@ recursive record walk happens to match this corpus, but format-specific field selection is the stronger recipe. Both model profiles choose record-and-continue for integrity; deterministic inventory replays used fail. No contradictions were observed, and the exact caller-selected arguments are retained. + +## Completed serial cohort + +The final sixteen-row cohort, raw-event digests, exact runner/bridge/lockfile +fingerprints, prompt/instruction/provider digests and sampled resource stats are +in [agent-results.json](agent-results.json). The requested model configuration is +Terra high. Two managed bridge answers omit the classification container field +under the stronger full-object check; two normal-MCP managed logs have a truncated +producer envelope. The twelve other rows pass requested fact and observed-reference +checks. This is producer parity, not independent runtime truth or a per-fact +citation-association proof. Earlier stricter report-form grades remain retained. + +| Trial | Wall seconds | Input | Cached | Uncached | Output | MCP calls | Check | +| ---------------------------------- | -----------: | -----: | -----: | -------: | -----: | --------: | --------------------------------- | +| android-direct-5 | 67.329 | 107007 | 83712 | 23295 | 1670 | 3 | pass | +| android-bridge-7 | 48.231 | 113256 | 91648 | 21608 | 1456 | 3 | pass | +| managed-bridge-2 | 73.145 | 116974 | 92672 | 24302 | 2065 | 3 | classification container omitted | +| managed-direct-2 | 33.150 | 108096 | 75776 | 32320 | 2315 | 2 | facts match; reference unverified | +| electron-inventory-direct-1 | 89.494 | 266121 | 238848 | 27273 | 3215 | 3 | pass | +| electron-inventory-bridge-1 | 145.577 | 462809 | 407040 | 55769 | 5624 | 8 | pass | +| firmware-bridge-1 | 81.944 | 152886 | 126976 | 25910 | 2192 | 3 | pass | +| firmware-direct-1 | 30.161 | 103224 | 74752 | 28472 | 1761 | 1 | pass | +| native-direct-1 | 327.150 | 151343 | 122368 | 28975 | 2458 | 4 | pass | +| native-bridge-1 | 334.642 | 285961 | 245504 | 40457 | 2129 | 4 | pass | +| android-bridge-8 | 134.555 | 159370 | 134400 | 24970 | 3250 | 7 | pass | +| android-direct-6 | 71.624 | 153031 | 117504 | 35527 | 1501 | 3 | pass | +| managed-direct-3 | 75.229 | 108593 | 75776 | 32817 | 2774 | 2 | facts match; reference unverified | +| managed-bridge-3 | 40.609 | 125519 | 96000 | 29519 | 2274 | 3 | classification container omitted | +| electron-inventory-guided-bridge-1 | 85.909 | 101994 | 61440 | 40554 | 2563 | 3 | pass | +| electron-inventory-guided-direct-1 | 67.540 | 263484 | 221952 | 41532 | 2499 | 4 | pass | + +Ghidra's aggregate sampled CPU time is 263.738 seconds directly and 271.096 with +the bridge. The bridge uses three original operations versus four directly, +obtaining containing identity from the analyzed procedure. Both close the session +successfully. The native comparison uses the same default analysis settings; +these data do not establish a meaningful native speedup. + +The second Android bridge repetition completes seven original calls across five +scripts, without tool errors, before its final composed answer. Its 134.555-second +result reverses the first repetition's latency advantage. The model did not +reliably choose one composed execution even with explicit guidance. Uncached +input is lower in both bridge repetitions, while output and latency vary. + +The guided Electron question explicitly names the existing operations and the +nested inventory's manifest/occurrences layout for both profiles. Both pass, with +integrity_policy=fail. The bridge composes open/inspect/close once; the normal-MCP +profile repeats inspect_artifact. The total-input reduction from 263484 to 101994 +is large, but uncached input differs by only 978 tokens and bridge latency is +higher. Earlier non-guided profiles are a different question and cannot support +a causal prompt-improvement ratio. Unknown nested payload types remain an observed +contract-discovery problem; caller-selected values and identities are preserved. + +All sixteen runs stay below memory/task ceilings, with maximum sampled memory +1659744256 bytes and 126 tasks; no observed memory/task limit event or resource +abort occurs. Copied profile count is zero at completion. Runtime measurements +belong to the recorded source fingerprint at 22326e819; subsequent timer-range +validation and test-only Ajv installation are not new model measurements. + +Final boundary verification passes twelve actual SDK/stdio tests. A 2 MiB guest +budget rejects allocation pressure and the next execution succeeds. This is a +guest-budget failure, not an OS OOM or provider behavior claim. A local pinned-Node +probe shows a 2147483648-ms timeout becomes 1 ms; the owning named execution +contract now rejects that range before allocation. After SDK conversion, both +advertised bridge input schemas validate under JSON Schema 2020-12, and the actual +stdio client rejects invalid code, oversized memory and timer values. Sampled +suite memory peaks at 1387433984 bytes, with zero observed memory/task limit events. + +See [decision.md](decision.md) for the recommendation: keep current REA contracts, +improve focused Evidence-aware workflow guidance, and retain the bridge as an +opt-in candidate. A broad default switch is not justified by these samples.