anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 02:05:26 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

[BUG/PERF] Severe Memory Leak and Disk Swell leading to System Kernel Panic (macOS) #8831

New Issue
Open
opened 2026-02-16 18:10:57 -05:00 by yindo · 9 comments
yindo commented 2026-02-16 18:10:57 -05:00
Owner
Copy Link
Copy Source

Originally created by @dzianisv on GitHub (Feb 8, 2026).

Originally assigned to: @rekram1-node on GitHub.

Description

I am encountering a critical performance issue when using OpenCode at scale. While the TypeScript implementation is impressive, the current resource management leads to total system instability on macOS. After a few hours of intensive use, the system undergoes a forced restart due to memory exhaustion.

Observed Behavior
RAM/Swap Exhaustion: OpenCode consumes all available physical RAM and then expands into Swap until macOS triggers a kernel panic or forced restart.

Disk Swell: Approximately 50GB of disk space is consumed within a few hours. This space is reclaimed immediately after a restart, suggesting it is either a ballooning swapfile or unmanaged session storage.

System Impact: Daily mandatory restarts of the host machine (MacBook).

Investigation Plan
I am treating this as a research-based issue rather than a standard bug report. My next steps include:

Process Monitoring: Identifying if the leak is in the main process, a specific worker, or the underlying TUI/Language Server.

Storage Audit: Locating the specific directory where the 50GB swell occurs (e.g., ~/Library/Application Support/OpenCode or /private/var/vm).

Heap Snapshots: Attempting to capture V8 heap snapshots before the system freezes.

Plugins

No response

OpenCode version

No response

Steps to reproduce

No response

Screenshot and/or share link

No response

Operating System

No response

Terminal

No response

Originally created by @dzianisv on GitHub (Feb 8, 2026). Originally assigned to: @rekram1-node on GitHub. ### Description I am encountering a critical performance issue when using OpenCode at scale. While the TypeScript implementation is impressive, the current resource management leads to total system instability on macOS. After a few hours of intensive use, the system undergoes a forced restart due to memory exhaustion. Observed Behavior RAM/Swap Exhaustion: OpenCode consumes all available physical RAM and then expands into Swap until macOS triggers a kernel panic or forced restart. Disk Swell: Approximately 50GB of disk space is consumed within a few hours. This space is reclaimed immediately after a restart, suggesting it is either a ballooning swapfile or unmanaged session storage. System Impact: Daily mandatory restarts of the host machine (MacBook). Investigation Plan I am treating this as a research-based issue rather than a standard bug report. My next steps include: Process Monitoring: Identifying if the leak is in the main process, a specific worker, or the underlying TUI/Language Server. Storage Audit: Locating the specific directory where the 50GB swell occurs (e.g., ~/Library/Application Support/OpenCode or /private/var/vm). Heap Snapshots: Attempting to capture V8 heap snapshots before the system freezes. ### Plugins _No response_ ### OpenCode version _No response_ ### Steps to reproduce _No response_ ### Screenshot and/or share link _No response_ ### Operating System _No response_ ### Terminal _No response_
yindo added the bugperf labels 2026-02-16 18:10:57 -05:00
yindo commented 2026-02-16 18:10:57 -05:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Feb 8, 2026):

This issue might be a duplicate of existing issues. Please check:

  • #12513: [Performance] Critical CPU usage and memory leak even when starting a new session (similar resource exhaustion and memory leak symptoms)
  • #9239: High Memory Usage on macOS 15.7 (Apple M4, 16GB, arm64) (macOS-specific high memory usage)
  • #12499: Slow down the PC after hours usage and occupy large memory (memory and performance degradation after extended use)
  • #9743: [Bug] Memory Leak: OpenCode Process Killed by OOM Killer During Extended Runtime (similar OOM symptoms and memory exhaustion)

Feel free to ignore if none of these address your specific case.

@github-actions[bot] commented on GitHub (Feb 8, 2026): This issue might be a duplicate of existing issues. Please check: - #12513: [Performance] Critical CPU usage and memory leak even when starting a new session (similar resource exhaustion and memory leak symptoms) - #9239: High Memory Usage on macOS 15.7 (Apple M4, 16GB, arm64) (macOS-specific high memory usage) - #12499: Slow down the PC after hours usage and occupy large memory (memory and performance degradation after extended use) - #9743: [Bug] Memory Leak: OpenCode Process Killed by OOM Killer During Extended Runtime (similar OOM symptoms and memory exhaustion) Feel free to ignore if none of these address your specific case.
yindo commented 2026-02-16 18:10:57 -05:00
Author
Owner
Copy Link
Copy Source

@dzianisv commented on GitHub (Feb 11, 2026):

Forensic Analysis: Kernel Panic Root Cause (Feb 11, 2026)

Machine: MacBook Pro M1 (MacBookPro17,1), 16 GB RAM, macOS 15.5 (24F74), ~30 GB free disk

Kernel Panic Details

Two kernel panics in 3 days, both identical cause:

Feb 11, 01:47 AM:

panic(cpu 5): watchdog timeout: no checkins from watchdogd in 91 seconds
Compressor Info: 54% of compressed pages limit (OK) and 70% of segments limit (OK)
                 with 18 swapfiles and LOW swap space

Feb 9, 00:28 AM:

panic(cpu 0): watchdog timeout: no checkins from watchdogd in 91 seconds
Compressor Info: 52% of compressed pages limit (OK) and 95% of segments limit (OK)
                 with 23 swapfiles and LOW swap space

Both times the system was so memory-starved that the kernel watchdog daemon couldn't check in within 91 seconds, triggering a hard panic and forced reboot. Boot faults recorded as wdog,reset_in_1 timeout.

Jetsam (OOM) Events Confirm opencode as Primary Consumer

The macOS Jetsam event from Feb 10 explicitly identifies:

"largestProcess": "opencode"

Memory snapshot at time of Jetsam kill:

Process Resident Memory PID
opencode 1,525.9 MB 10748
opencode 1,350.3 MB 38850
opencode 1,325.4 MB 78124
opencode 1,211.9 MB 8579
opencode 1,103.5 MB 46921
opencode 789.1 MB 11105
opencode 731.6 MB 60304
opencode 465.1 MB 35103
opencode 447.3 MB 26047
opencode 422.6 MB 59824
Total (opencode only) ~9.4 GB 10 instances
node (2 instances) ~1.1 GB —

10 opencode processes consuming ~9.4 GB on a 16 GB machine. That's 59% of total physical RAM from opencode alone.

System memory state at time of Jetsam:

  • Free pages: 13,781 (~215 MB) — critically low
  • Active: 221,716 pages (~3.4 GB)
  • Inactive: 210,697 pages (~3.3 GB)
  • Wired: 129,341 pages (~2.0 GB)
  • Compressor size: 437,326 pages (~6.8 GB)

Evidence of Memory Leak

The opencode diagnostic report from Feb 9 (opencode_2026-02-09-134500) shows clear unbounded growth:

Footprint: 243.58 MB -> 1493.97 MB (+1250.39 MB) in ~35 minutes
Writes: 2147.49 MB of file backed memory dirtied over 7760 seconds
         (276.75 KB/s average, exceeding limit of 24.86 KB/s)

A single opencode instance grew from 244 MB to 1.5 GB in 35 minutes while also writing 2.1 GB to disk. This is characteristic of a memory leak — possibly conversation context accumulation, uncollected tool output buffers, or session state that is never freed.

Reboot Frequency

From last reboot, the majority of reboots are not preceded by a graceful shutdown, indicating forced restarts:

Date Type
Feb 11 01:47 Kernel panic (confirmed)
Feb 9 00:28 Kernel panic (confirmed)
Feb 5 20:51 Graceful shutdown/reboot
Feb 3 17:50 Graceful shutdown/reboot
Feb 3 10:15 Forced reboot (no shutdown)
Feb 3 01:31 Forced reboot (no shutdown)
Feb 2 00:47 Forced reboot (no shutdown)
Jan 31 22:41 Forced reboot (no shutdown)
Jan 29 17:26 Forced reboot (no shutdown)
Jan 28 14:36 Forced reboot (no shutdown)
Jan 28 01:19 Forced reboot (no shutdown)

~10 forced reboots in 2 weeks. The overnight pattern (01:47, 00:28, 01:31, 00:47, 01:19) suggests opencode sessions left running overnight accumulate memory until the system panics.

Additional Jetsam Events (Feb 7)

Two Jetsam OOM events on Feb 7 show 5+ opencode instances ranging from 533 MB to 1,057 MB each, alongside a Python process at 6.4 GB. opencode was consistently in the top consumers.

Summary

  1. Root cause: opencode has a memory leak — individual instances grow unboundedly (~1.25 GB in 35 min observed)
  2. Compounding factor: Stale/idle opencode sessions are not cleaned up, leading to 10+ zombie-like instances accumulating
  3. System impact: On 16 GB machines, 3–4 forgotten sessions are sufficient to exhaust RAM + swap and trigger a kernel panic
  4. Disk swell: Consistent with the 18–23 swapfiles observed; macOS creates ~2 GB swap files, so 23 × 2 GB = ~46 GB matches the ~50 GB disk swell reported in this issue
  5. Frequency: ~10 forced reboots in 2 weeks, primarily overnight when sessions are idle but still consuming memory
@dzianisv commented on GitHub (Feb 11, 2026): ## Forensic Analysis: Kernel Panic Root Cause (Feb 11, 2026) **Machine:** MacBook Pro M1 (MacBookPro17,1), 16 GB RAM, macOS 15.5 (24F74), ~30 GB free disk --- ### Kernel Panic Details Two kernel panics in 3 days, both identical cause: **Feb 11, 01:47 AM:** ``` panic(cpu 5): watchdog timeout: no checkins from watchdogd in 91 seconds Compressor Info: 54% of compressed pages limit (OK) and 70% of segments limit (OK) with 18 swapfiles and LOW swap space ``` **Feb 9, 00:28 AM:** ``` panic(cpu 0): watchdog timeout: no checkins from watchdogd in 91 seconds Compressor Info: 52% of compressed pages limit (OK) and 95% of segments limit (OK) with 23 swapfiles and LOW swap space ``` Both times the system was so memory-starved that the kernel watchdog daemon couldn't check in within 91 seconds, triggering a hard panic and forced reboot. Boot faults recorded as `wdog,reset_in_1 timeout`. ### Jetsam (OOM) Events Confirm `opencode` as Primary Consumer The macOS Jetsam event from **Feb 10** explicitly identifies: > **`"largestProcess": "opencode"`** Memory snapshot at time of Jetsam kill: | Process | Resident Memory | PID | |---------|----------------|-----| | opencode | **1,525.9 MB** | 10748 | | opencode | **1,350.3 MB** | 38850 | | opencode | **1,325.4 MB** | 78124 | | opencode | **1,211.9 MB** | 8579 | | opencode | **1,103.5 MB** | 46921 | | opencode | **789.1 MB** | 11105 | | opencode | **731.6 MB** | 60304 | | opencode | **465.1 MB** | 35103 | | opencode | **447.3 MB** | 26047 | | opencode | **422.6 MB** | 59824 | | **Total (opencode only)** | **~9.4 GB** | 10 instances | | node (2 instances) | **~1.1 GB** | — | **10 opencode processes consuming ~9.4 GB on a 16 GB machine.** That's 59% of total physical RAM from opencode alone. System memory state at time of Jetsam: - Free pages: **13,781** (~215 MB) — critically low - Active: 221,716 pages (~3.4 GB) - Inactive: 210,697 pages (~3.3 GB) - Wired: 129,341 pages (~2.0 GB) - Compressor size: 437,326 pages (~6.8 GB) ### Evidence of Memory Leak The opencode diagnostic report from **Feb 9** (`opencode_2026-02-09-134500`) shows clear unbounded growth: ``` Footprint: 243.58 MB -> 1493.97 MB (+1250.39 MB) in ~35 minutes Writes: 2147.49 MB of file backed memory dirtied over 7760 seconds (276.75 KB/s average, exceeding limit of 24.86 KB/s) ``` A single opencode instance grew from **244 MB to 1.5 GB in 35 minutes** while also writing 2.1 GB to disk. This is characteristic of a memory leak — possibly conversation context accumulation, uncollected tool output buffers, or session state that is never freed. ### Reboot Frequency From `last reboot`, the majority of reboots are **not preceded by a graceful shutdown**, indicating forced restarts: | Date | Type | |------|------| | Feb 11 01:47 | **Kernel panic** (confirmed) | | Feb 9 00:28 | **Kernel panic** (confirmed) | | Feb 5 20:51 | Graceful shutdown/reboot | | Feb 3 17:50 | Graceful shutdown/reboot | | Feb 3 10:15 | Forced reboot (no shutdown) | | Feb 3 01:31 | Forced reboot (no shutdown) | | Feb 2 00:47 | Forced reboot (no shutdown) | | Jan 31 22:41 | Forced reboot (no shutdown) | | Jan 29 17:26 | Forced reboot (no shutdown) | | Jan 28 14:36 | Forced reboot (no shutdown) | | Jan 28 01:19 | Forced reboot (no shutdown) | **~10 forced reboots in 2 weeks.** The overnight pattern (01:47, 00:28, 01:31, 00:47, 01:19) suggests opencode sessions left running overnight accumulate memory until the system panics. ### Additional Jetsam Events (Feb 7) Two Jetsam OOM events on Feb 7 show 5+ opencode instances ranging from 533 MB to 1,057 MB each, alongside a Python process at 6.4 GB. opencode was consistently in the top consumers. ### Summary 1. **Root cause:** opencode has a memory leak — individual instances grow unboundedly (~1.25 GB in 35 min observed) 2. **Compounding factor:** Stale/idle opencode sessions are not cleaned up, leading to 10+ zombie-like instances accumulating 3. **System impact:** On 16 GB machines, 3–4 forgotten sessions are sufficient to exhaust RAM + swap and trigger a kernel panic 4. **Disk swell:** Consistent with the 18–23 swapfiles observed; macOS creates ~2 GB swap files, so 23 × 2 GB = ~46 GB matches the ~50 GB disk swell reported in this issue 5. **Frequency:** ~10 forced reboots in 2 weeks, primarily overnight when sessions are idle but still consuming memory
yindo commented 2026-02-16 18:10:57 -05:00
Author
Owner
Copy Link
Copy Source

@dzianisv commented on GitHub (Feb 11, 2026):

Analysis of Related Open PRs

There are multiple open PRs addressing the same underlying memory leak problems described in this issue, though none directly reference #12687.

Most Relevant Open PRs

PR Author Approach Key Fixes
#12053 @sgInnora Targeted fixes for 6 specific leaks AsyncQueue resolver cleanup, bash output ring buffer (10MB cap), LSP diagnostics FIFO eviction (5K files), bus subscription cleanup, PTY buffer ring buffer, process exit disposal
#10392 @ztripez Broad session handling fixes (8 fixes) Provider LRU caches, remove deep clone in prompt loop, session/FileTime/ACP cleanup, compaction output deletion, pending request timeouts, processor map cleanup, string concat optimization
#12259 @Stranmor TTL + LRU eviction framework ACP session TTL (1h) + LRU (50 limit), Instance cache TTL (30min) + LRU (10), LSP client limit (20), PTY RingBuffer
#9693 @kov O(n²) bash output fix Streams large output to disk (>50KB threshold) instead of output += chunk string concatenation
#9146 @sauerdaniel ACP session lifecycle Proper session cleanup on process exit, event listener removal
#9147 @sauerdaniel TUI/Slack listener leaks Event listener cleanup
#7914 @hendem Subscription disposal Dispose functions for subscription memory leaks

Key Observations

  1. None of these PRs have been merged — all are still open with no reviews from maintainers.
  2. Significant overlap between PRs #12053, #10392, and #12259. They all target the same core issues (bash output accumulation, PTY buffers, session cleanup) with slightly different implementations.
  3. PR #10392 is the most comprehensive (15 files, 8 fixes) and has been tested by a user (@Thejuampi) who reported it didn't fully solve the issue for Java/jdtls LSP projects.
  4. PR #9693 takes the most practical approach for the bash tool specifically — streaming to disk instead of memory accumulation.

Root Causes Identified Across PRs

The community has converged on these root causes:

  • Unbounded output += chunk string concatenation in bash/PTY (O(n²) memory)
  • No eviction on LSP diagnostics, provider caches, ACP sessions, or instance caches
  • Subscription/event listener leaks not disposed on session end
  • Prompt loop deep clones copying entire message history every turn
  • Compaction not actually deleting tool output from storage

A coordinated effort to review and merge the best parts of these PRs would likely address the kernel panic symptoms reported here.

@dzianisv commented on GitHub (Feb 11, 2026): ## Analysis of Related Open PRs There are **multiple open PRs** addressing the same underlying memory leak problems described in this issue, though none directly reference #12687. ### Most Relevant Open PRs | PR | Author | Approach | Key Fixes | |----|--------|----------|-----------| | **#12053** | @sgInnora | Targeted fixes for 6 specific leaks | AsyncQueue resolver cleanup, bash output ring buffer (10MB cap), LSP diagnostics FIFO eviction (5K files), bus subscription cleanup, PTY buffer ring buffer, process exit disposal | | **#10392** | @ztripez | Broad session handling fixes (8 fixes) | Provider LRU caches, remove deep clone in prompt loop, session/FileTime/ACP cleanup, compaction output deletion, pending request timeouts, processor map cleanup, string concat optimization | | **#12259** | @Stranmor | TTL + LRU eviction framework | ACP session TTL (1h) + LRU (50 limit), Instance cache TTL (30min) + LRU (10), LSP client limit (20), PTY RingBuffer | | **#9693** | @kov | O(n²) bash output fix | Streams large output to disk (>50KB threshold) instead of `output += chunk` string concatenation | | **#9146** | @sauerdaniel | ACP session lifecycle | Proper session cleanup on process exit, event listener removal | | **#9147** | @sauerdaniel | TUI/Slack listener leaks | Event listener cleanup | | **#7914** | @hendem | Subscription disposal | Dispose functions for subscription memory leaks | ### Key Observations 1. **None of these PRs have been merged** — all are still open with no reviews from maintainers. 2. **Significant overlap** between PRs #12053, #10392, and #12259. They all target the same core issues (bash output accumulation, PTY buffers, session cleanup) with slightly different implementations. 3. **PR #10392** is the most comprehensive (15 files, 8 fixes) and has been tested by a user (@Thejuampi) who reported it didn't fully solve the issue for Java/jdtls LSP projects. 4. **PR #9693** takes the most practical approach for the bash tool specifically — streaming to disk instead of memory accumulation. ### Root Causes Identified Across PRs The community has converged on these root causes: - **Unbounded `output += chunk` string concatenation** in bash/PTY (O(n²) memory) - **No eviction** on LSP diagnostics, provider caches, ACP sessions, or instance caches - **Subscription/event listener leaks** not disposed on session end - **Prompt loop deep clones** copying entire message history every turn - **Compaction not actually deleting** tool output from storage A coordinated effort to review and merge the best parts of these PRs would likely address the kernel panic symptoms reported here.
yindo commented 2026-02-16 18:10:57 -05:00
Author
Owner
Copy Link
Copy Source

@dzianisv commented on GitHub (Feb 11, 2026):

Based on the forensic analysis in the issue and the PRs I reviewed:

What the PRs would fix
The output += chunk string concatenation issue (PRs #12053, #9693, #10392) is almost certainly a real and significant leak. This O(n²) pattern in bash/PTY output is a well-known pathological case in JavaScript/Bun — it creates massive GC pressure and intermediate string allocations that can easily explain the 244 MB to 1.5 GB growth in 35 minutes that @dzianisv observed. Multiple PRs converge on this fix independently, which gives confidence it's a real problem.
The subscription/listener leaks (PRs #7914, #9146, #9147) are also legitimate — unremoved listeners accumulate closures that retain references to large objects.
What the PRs likely don't fully solve
2. @Thejuampi already tested PR #10392 (the most comprehensive one) and reported it didn't help for their Java/jdtls project. This suggests the LSP layer itself may be a major contributor that isn't adequately addressed — jdtls is notoriously memory-hungry, and simply capping diagnostics at 5K files may not be enough if the LSP client holds references to response objects, document buffers, or AST data.
3. Overnight idle accumulation — The kernel panics happen at 01:47, 00:28, 01:31 etc. The PRs add eviction on access (LRU) and creation (FIFO), but they don't add periodic idle cleanup that would reclaim memory when opencode is sitting unused overnight. PR #12259 has a TTL-based cleanup interval for ACP sessions, but not for the other caches.
4. No actual memory profiling — None of the PRs include heap snapshots or profiling data showing before/after. They're based on code review and reasonable assumptions, not measurement. The real dominant leak source could be something none of them address.
Bottom line
Merging the bash/PTY output fix (especially #9693's disk-streaming approach) and the subscription cleanup would meaningfully reduce memory growth. But I'd estimate they address maybe 40-60% of the problem. The multi-instance accumulation, LSP memory, and idle session cleanup gaps remain unaddressed. A proper fix likely requires heap profiling to identify what's actually holding memory in a real long-running session.

@dzianisv commented on GitHub (Feb 11, 2026): Based on the forensic analysis in the issue and the PRs I reviewed: What the PRs would fix The output += chunk string concatenation issue (PRs #12053, #9693, #10392) is almost certainly a real and significant leak. This O(n²) pattern in bash/PTY output is a well-known pathological case in JavaScript/Bun — it creates massive GC pressure and intermediate string allocations that can easily explain the 244 MB to 1.5 GB growth in 35 minutes that @dzianisv observed. Multiple PRs converge on this fix independently, which gives confidence it's a real problem. The subscription/listener leaks (PRs #7914, #9146, #9147) are also legitimate — unremoved listeners accumulate closures that retain references to large objects. What the PRs likely don't fully solve 2. @Thejuampi already tested PR #10392 (the most comprehensive one) and reported it didn't help for their Java/jdtls project. This suggests the LSP layer itself may be a major contributor that isn't adequately addressed — jdtls is notoriously memory-hungry, and simply capping diagnostics at 5K files may not be enough if the LSP client holds references to response objects, document buffers, or AST data. 3. Overnight idle accumulation — The kernel panics happen at 01:47, 00:28, 01:31 etc. The PRs add eviction on access (LRU) and creation (FIFO), but they don't add periodic idle cleanup that would reclaim memory when opencode is sitting unused overnight. PR #12259 has a TTL-based cleanup interval for ACP sessions, but not for the other caches. 4. No actual memory profiling — None of the PRs include heap snapshots or profiling data showing before/after. They're based on code review and reasonable assumptions, not measurement. The real dominant leak source could be something none of them address. Bottom line Merging the bash/PTY output fix (especially #9693's disk-streaming approach) and the subscription cleanup would meaningfully reduce memory growth. But I'd estimate they address maybe 40-60% of the problem. The multi-instance accumulation, LSP memory, and idle session cleanup gaps remain unaddressed. A proper fix likely requires heap profiling to identify what's actually holding memory in a real long-running session.
yindo commented 2026-02-16 18:10:57 -05:00
Author
Owner
Copy Link
Copy Source

@dzianisv commented on GitHub (Feb 11, 2026):

Root Cause Analysis and Proposed Fix

After a deep dive into the codebase, here are the actual root causes ranked by impact, and a concrete fix plan.

1. No graceful shutdown on signals (THE #1 cause of zombie processes)

This is the biggest problem and explains the 10 zombie processes in the forensic analysis. index.ts:153 has a finally block that calls process.exit() but never calls Instance.disposeAll(). There are zero SIGTERM/SIGINT/SIGHUP handlers anywhere in the codebase. When you close a terminal:

  • The shell sends SIGHUP to opencode
  • Bun's default handler exits immediately
  • No cleanup runs — no LSP shutdown, no MCP close, no bash kill
  • Bash tool processes are spawned with detached: true (bash.ts:168), so they survive the parent dying
  • LSP servers (up to 25 per instance) may also linger since cleanup never fires

This single gap explains both the zombie processes AND the multiplying memory. Each time you open a terminal, run opencode, and close the terminal without a clean exit, you leave behind orphaned LSP/MCP/bash processes.

Fix: Add SIGTERM/SIGINT/SIGHUP handlers that call Instance.disposeAll() before exiting. Call disposeAll() in the finally block.

2. Unbounded output += chunk in bash tool (bash.ts:171-182)

This is the #1 per-process memory growth cause. Every bash command accumulates its entire output as a string via output += chunk.toString(). For verbose builds or large find outputs, this creates O(n²) memory pressure from string concatenation. The output is only truncated after the command finishes — by then, memory has already spiked.

Fix: Stream to disk above a threshold (like PR #9693 proposes), or use a Buffer[] ring buffer with a cap.

3. Deep clone of entire message history every prompt loop iteration (prompt.ts:577)

clone(msgs) deep-copies every message including all tool outputs on every loop turn. For a long conversation with large tool results, this temporarily doubles memory. Combined with MessageV2.stream() reloading all messages from disk each iteration (line 276), each turn has massive transient allocations.

Fix: Replace the deep clone with shallow copies of only the parts that get mutated (the <system-reminder> wrapping), or mutate in place and undo after.

4. ACP sessions never cleaned up (acp/session.ts:9)

The sessions Map has no delete(), no clear(), no dispose callback. Sessions accumulate forever.

Fix: Add a dispose callback to the Instance.state() call that clears the map.

5. PermissionNext and Question missing disposal callbacks

Unlike the legacy Permission module (which properly rejects pending promises on dispose), both PermissionNext (permission/next.ts:108) and Question (question/index.ts:82) register Instance.state() without a dispose callback. Pending promises and their closures leak on instance disposal.

Fix: Add dispose callbacks that reject all pending promises, matching the pattern already used in the legacy Permission module.

6. FileTime tracking grows unboundedly (file/time.ts:11-22)

The read object tracks [sessionID][path] -> Date and entries are never evicted.

Fix: Add cleanup when sessions end, or add a dispose callback.

Proposed Implementation (~8 files)

File Change
index.ts Add SIGTERM/SIGINT/SIGHUP handlers that call disposeAll() before exit
tool/bash.ts Cap output accumulation with a ring buffer or disk streaming
session/prompt.ts Replace clone(msgs) with targeted shallow copies
acp/session.ts Add dispose callback to clear sessions map
permission/next.ts Add dispose callback to reject pending promises
question/index.ts Add dispose callback to reject pending promises
file/time.ts Add dispose callback to clear read tracking
util/queue.ts Add close() method to drain pending resolvers

The signal handling fix alone would eliminate the zombie process problem (the 10-process / 9.4 GB scenario from the forensic analysis). Combined with the bash output cap and prompt clone fix, this covers the per-process memory growth. The remaining fixes are defense-in-depth for long-running sessions.

This should address ~90% of the memory exhaustion and kernel panic symptoms reported in this issue.

@dzianisv commented on GitHub (Feb 11, 2026): ## Root Cause Analysis and Proposed Fix After a deep dive into the codebase, here are the actual root causes ranked by impact, and a concrete fix plan. ### 1. No graceful shutdown on signals (THE #1 cause of zombie processes) This is the biggest problem and explains the 10 zombie processes in the forensic analysis. `index.ts:153` has a `finally` block that calls `process.exit()` but **never calls `Instance.disposeAll()`**. There are **zero** `SIGTERM`/`SIGINT`/`SIGHUP` handlers anywhere in the codebase. When you close a terminal: - The shell sends `SIGHUP` to opencode - Bun's default handler exits immediately - No cleanup runs — no LSP shutdown, no MCP close, no bash kill - Bash tool processes are spawned with `detached: true` (`bash.ts:168`), so they **survive** the parent dying - LSP servers (up to 25 per instance) may also linger since cleanup never fires This single gap explains both the zombie processes AND the multiplying memory. Each time you open a terminal, run opencode, and close the terminal without a clean exit, you leave behind orphaned LSP/MCP/bash processes. **Fix:** Add `SIGTERM`/`SIGINT`/`SIGHUP` handlers that call `Instance.disposeAll()` before exiting. Call `disposeAll()` in the `finally` block. ### 2. Unbounded `output += chunk` in bash tool (`bash.ts:171-182`) This is the #1 per-process memory growth cause. Every bash command accumulates its entire output as a string via `output += chunk.toString()`. For verbose builds or large `find` outputs, this creates O(n²) memory pressure from string concatenation. The output is only truncated *after* the command finishes — by then, memory has already spiked. **Fix:** Stream to disk above a threshold (like PR #9693 proposes), or use a `Buffer[]` ring buffer with a cap. ### 3. Deep clone of entire message history every prompt loop iteration (`prompt.ts:577`) `clone(msgs)` deep-copies **every message including all tool outputs** on every loop turn. For a long conversation with large tool results, this temporarily doubles memory. Combined with `MessageV2.stream()` reloading all messages from disk each iteration (line 276), each turn has massive transient allocations. **Fix:** Replace the deep clone with shallow copies of only the parts that get mutated (the `<system-reminder>` wrapping), or mutate in place and undo after. ### 4. ACP sessions never cleaned up (`acp/session.ts:9`) The sessions Map has no `delete()`, no `clear()`, no dispose callback. Sessions accumulate forever. **Fix:** Add a dispose callback to the `Instance.state()` call that clears the map. ### 5. PermissionNext and Question missing disposal callbacks Unlike the legacy `Permission` module (which properly rejects pending promises on dispose), both `PermissionNext` (`permission/next.ts:108`) and `Question` (`question/index.ts:82`) register `Instance.state()` **without** a dispose callback. Pending promises and their closures leak on instance disposal. **Fix:** Add dispose callbacks that reject all pending promises, matching the pattern already used in the legacy `Permission` module. ### 6. FileTime tracking grows unboundedly (`file/time.ts:11-22`) The `read` object tracks `[sessionID][path] -> Date` and entries are never evicted. **Fix:** Add cleanup when sessions end, or add a dispose callback. --- ### Proposed Implementation (~8 files) | File | Change | |------|--------| | `index.ts` | Add `SIGTERM`/`SIGINT`/`SIGHUP` handlers that call `disposeAll()` before exit | | `tool/bash.ts` | Cap output accumulation with a ring buffer or disk streaming | | `session/prompt.ts` | Replace `clone(msgs)` with targeted shallow copies | | `acp/session.ts` | Add dispose callback to clear sessions map | | `permission/next.ts` | Add dispose callback to reject pending promises | | `question/index.ts` | Add dispose callback to reject pending promises | | `file/time.ts` | Add dispose callback to clear read tracking | | `util/queue.ts` | Add `close()` method to drain pending resolvers | The signal handling fix alone would eliminate the zombie process problem (the 10-process / 9.4 GB scenario from the forensic analysis). Combined with the bash output cap and prompt clone fix, this covers the per-process memory growth. The remaining fixes are defense-in-depth for long-running sessions. This should address ~90% of the memory exhaustion and kernel panic symptoms reported in this issue.
yindo commented 2026-02-16 18:10:58 -05:00
Author
Owner
Copy Link
Copy Source

@dzianisv commented on GitHub (Feb 11, 2026):

Implementation Plan

Fix 1: Graceful shutdown via signal handlers (Critical — eliminates zombie processes)

Problem: No SIGTERM/SIGINT/SIGHUP handlers exist anywhere. When a terminal is closed, SIGHUP kills the process instantly. Instance.disposeAll() never runs. Child processes spawned with detached: true (bash tool) survive indefinitely. LSP/MCP servers are orphaned.

The only reliable cleanup path today is the TUI's graceful exit flow (ExitProvider -> shutdown RPC -> worker Instance.disposeAll()). Every other exit path (signals, serve command, errors) skips cleanup entirely.

Changes:

  • index.ts: Add SIGTERM/SIGINT/SIGHUP handlers that call Instance.disposeAll() before exiting. Also call disposeAll() in the existing finally block before process.exit().
  • worker.ts: Add matching signal handlers in the worker thread, since the worker owns the Instance state and runs disposeAll().
  • serve.ts: Replace the unreachable cleanup code (await new Promise(() => {}) blocks forever, making lines 48-56 dead code) with a signal-based exit that properly cleans up.

Fix 2: Bash/shell output accumulation (High — largest per-process memory spike)

Problem: bash.ts:171 uses let output = "" with output += chunk.toString() on every data event. The full output string grows unboundedly during execution. The same pattern exists in prompt.ts shell() method (~line 1499). While not a persistent leak (freed when the function returns), for verbose commands this creates massive transient memory spikes with O(n²) GC pressure from string concatenation.

Changes:

  • bash.ts: Accumulate output in a Buffer[] array. Cap total size (e.g., 10MB). Drop oldest chunks when exceeded. Join only at the end for the return value.
  • prompt.ts shell(): Apply the same ring buffer pattern.

Fix 3: Deep clone of message history in prompt loop (High — doubles memory per turn)

Problem: prompt.ts:577 calls clone(msgs) (remeda deep clone) on the entire session message history every loop iteration. This exists because lines 580-596 mutate part.text to wrap it in <system-reminder> tags. For long sessions with large tool outputs, this temporarily doubles memory.

Changes:

  • prompt.ts: Instead of deep-cloning everything, create the reminder-wrapped text without mutating the original messages. Build a map of partID -> wrappedText and apply it during model message construction, or shallow-copy only the specific parts that need mutation.

Fix 4: Missing dispose callbacks (High — pending promises leak on shutdown)

Problem: PermissionNext (permission/next.ts:108) and Question (question/index.ts:82) register Instance.state() without dispose callbacks. The legacy Permission (permission/index.ts:53-83) does it correctly — it provides a dispose callback that rejects all pending promises. When an instance is disposed, pending promise closures (holding references to session data, tool contexts, etc.) are never freed.

Changes:

  • permission/next.ts: Add dispose callback matching the pattern in permission/index.ts.
  • question/index.ts: Add dispose callback that rejects all pending question promises.

Fix 5: ACP session cleanup (Medium)

Problem: ACPSessionManager (acp/session.ts:9) stores sessions in a Map with no delete() or dispose() method. Sessions accumulate forever.

Changes:

  • acp/session.ts: Add a dispose() method that clears the sessions map. Wire it into the instance lifecycle.

Fix 6: AsyncQueue close mechanism (Medium)

Problem: util/queue.ts has no close()/termination mechanism. The async iterator is infinite (while(true)). If consumers stop iterating, pending resolver callbacks stay in memory as unresolved promises.

Changes:

  • util/queue.ts: Add a close() method that sets a flag, resolves/rejects all pending resolvers, and causes the async iterator to terminate.

Fix 7: FileTime cleanup (Low-Medium)

Problem: file/time.ts tracks [sessionID][path] -> Date for every file read, with no eviction or cleanup.

Changes:

  • file/time.ts: Add a dispose callback. Add a clear(sessionID) method for session-level cleanup.

Expected impact

Fix Addresses % of reported symptoms
Signal handlers 10 zombie processes, 9.4 GB combined, orphaned LSP/MCP/bash ~50-60%
Bash output cap 244 MB → 1.5 GB in 35 min growth pattern ~15-20%
Prompt loop clone Memory doubling per turn in long sessions ~10%
Dispose callbacks Slow leak from pending promises on abandoned sessions ~5%
ACP/Queue/FileTime Defense-in-depth for long-running sessions ~5%

Total estimated coverage: ~90% of reported symptoms.

The signal handler fix alone eliminates the multi-process accumulation problem. Combined with the bash output cap, it addresses the two primary mechanisms that cause kernel panics on 16 GB machines.

@dzianisv commented on GitHub (Feb 11, 2026): ## Implementation Plan ### Fix 1: Graceful shutdown via signal handlers (Critical — eliminates zombie processes) **Problem:** No `SIGTERM`/`SIGINT`/`SIGHUP` handlers exist anywhere. When a terminal is closed, `SIGHUP` kills the process instantly. `Instance.disposeAll()` never runs. Child processes spawned with `detached: true` (bash tool) survive indefinitely. LSP/MCP servers are orphaned. **The only reliable cleanup path today is the TUI's graceful exit flow** (`ExitProvider` -> `shutdown` RPC -> worker `Instance.disposeAll()`). Every other exit path (signals, `serve` command, errors) skips cleanup entirely. **Changes:** - **`index.ts`**: Add `SIGTERM`/`SIGINT`/`SIGHUP` handlers that call `Instance.disposeAll()` before exiting. Also call `disposeAll()` in the existing `finally` block before `process.exit()`. - **`worker.ts`**: Add matching signal handlers in the worker thread, since the worker owns the Instance state and runs `disposeAll()`. - **`serve.ts`**: Replace the unreachable cleanup code (`await new Promise(() => {})` blocks forever, making lines 48-56 dead code) with a signal-based exit that properly cleans up. ### Fix 2: Bash/shell output accumulation (High — largest per-process memory spike) **Problem:** `bash.ts:171` uses `let output = ""` with `output += chunk.toString()` on every data event. The full output string grows unboundedly during execution. The same pattern exists in `prompt.ts` `shell()` method (~line 1499). While not a persistent leak (freed when the function returns), for verbose commands this creates massive transient memory spikes with O(n²) GC pressure from string concatenation. **Changes:** - **`bash.ts`**: Accumulate output in a `Buffer[]` array. Cap total size (e.g., 10MB). Drop oldest chunks when exceeded. Join only at the end for the return value. - **`prompt.ts` `shell()`**: Apply the same ring buffer pattern. ### Fix 3: Deep clone of message history in prompt loop (High — doubles memory per turn) **Problem:** `prompt.ts:577` calls `clone(msgs)` (remeda deep clone) on the **entire** session message history every loop iteration. This exists because lines 580-596 mutate `part.text` to wrap it in `<system-reminder>` tags. For long sessions with large tool outputs, this temporarily doubles memory. **Changes:** - **`prompt.ts`**: Instead of deep-cloning everything, create the reminder-wrapped text without mutating the original messages. Build a map of `partID -> wrappedText` and apply it during model message construction, or shallow-copy only the specific parts that need mutation. ### Fix 4: Missing dispose callbacks (High — pending promises leak on shutdown) **Problem:** `PermissionNext` (`permission/next.ts:108`) and `Question` (`question/index.ts:82`) register `Instance.state()` **without** dispose callbacks. The legacy `Permission` (`permission/index.ts:53-83`) does it correctly — it provides a dispose callback that rejects all pending promises. When an instance is disposed, pending promise closures (holding references to session data, tool contexts, etc.) are never freed. **Changes:** - **`permission/next.ts`**: Add dispose callback matching the pattern in `permission/index.ts`. - **`question/index.ts`**: Add dispose callback that rejects all pending question promises. ### Fix 5: ACP session cleanup (Medium) **Problem:** `ACPSessionManager` (`acp/session.ts:9`) stores sessions in a `Map` with no `delete()` or `dispose()` method. Sessions accumulate forever. **Changes:** - **`acp/session.ts`**: Add a `dispose()` method that clears the sessions map. Wire it into the instance lifecycle. ### Fix 6: AsyncQueue close mechanism (Medium) **Problem:** `util/queue.ts` has no `close()`/termination mechanism. The async iterator is infinite (`while(true)`). If consumers stop iterating, pending resolver callbacks stay in memory as unresolved promises. **Changes:** - **`util/queue.ts`**: Add a `close()` method that sets a flag, resolves/rejects all pending resolvers, and causes the async iterator to terminate. ### Fix 7: FileTime cleanup (Low-Medium) **Problem:** `file/time.ts` tracks `[sessionID][path] -> Date` for every file read, with no eviction or cleanup. **Changes:** - **`file/time.ts`**: Add a dispose callback. Add a `clear(sessionID)` method for session-level cleanup. --- ### Expected impact | Fix | Addresses | % of reported symptoms | |-----|-----------|----------------------| | Signal handlers | 10 zombie processes, 9.4 GB combined, orphaned LSP/MCP/bash | ~50-60% | | Bash output cap | 244 MB → 1.5 GB in 35 min growth pattern | ~15-20% | | Prompt loop clone | Memory doubling per turn in long sessions | ~10% | | Dispose callbacks | Slow leak from pending promises on abandoned sessions | ~5% | | ACP/Queue/FileTime | Defense-in-depth for long-running sessions | ~5% | **Total estimated coverage: ~90% of reported symptoms.** The signal handler fix alone eliminates the multi-process accumulation problem. Combined with the bash output cap, it addresses the two primary mechanisms that cause kernel panics on 16 GB machines.
yindo commented 2026-02-16 18:10:58 -05:00
Author
Owner
Copy Link
Copy Source

@dzianisv commented on GitHub (Feb 11, 2026):

PR #13183 — Review and Cleanup

After a thorough self-review of the initial commit, I found and corrected several issues:

Problems caught and fixed

  1. serve.ts imported modules that don't exist on dev — Whisper, Coqui, Tunnel were accidentally included from a local feature branch. This would have broken the upstream build. Removed all voice/tunnel code, kept only the signal-based exit + Instance.disposeAll() fix.

  2. bash.ts metadata preview bug — chunks.slice(0, 1).toString() calls Array.toString() (joins with commas), not Buffer.concat(). This would produce garbled TUI output. Replaced with a separate preview string built incrementally and capped at 30KB.

  3. prompt.ts shell metadata called Buffer.concat() on every data chunk — This is O(n²), the same problem as the original output +=. Fixed with the same capped preview approach.

  4. Removed ACPSessionManager.dispose() — Nobody calls it. The class is instantiated in ACPAgentBridge with no lifecycle hook to the Instance state system. Cosmetic change that fixes nothing.

  5. Removed AsyncQueue.close() — Nobody calls it. The two AsyncQueue instances in tui.ts are module-level singletons with no dispose lifecycle. Cosmetic change that fixes nothing.

Final PR: 8 files, all meaningful

Fix Files What it prevents
Signal handlers + Instance.disposeAll() index.ts, worker.ts, serve.ts Zombie processes — the primary cause of #12687
Buffer[] ring buffer + capped preview bash.ts, prompt.ts OOM from O(n²) string growth on verbose commands
Shallow copy vs deep clone prompt.ts Memory spikes from clone(msgs) every prompt loop
Dispose callbacks permission/next.ts, question/index.ts Promise closure leaks on instance disposal
FileTime cleanup file/time.ts Unbounded growth of session read records

Testing

  • tsgo --noEmit passes across all 17 packages
  • 844 tests pass across 59 files, 0 failures
  • No behavioral changes — all fixes follow existing codebase patterns

PR: https://github.com/anomalyco/opencode/pull/13183

@dzianisv commented on GitHub (Feb 11, 2026): ## PR #13183 — Review and Cleanup After a thorough self-review of the initial commit, I found and corrected several issues: ### Problems caught and fixed 1. **`serve.ts` imported modules that don't exist on `dev`** — `Whisper`, `Coqui`, `Tunnel` were accidentally included from a local feature branch. This would have broken the upstream build. Removed all voice/tunnel code, kept only the signal-based exit + `Instance.disposeAll()` fix. 2. **`bash.ts` metadata preview bug** — `chunks.slice(0, 1).toString()` calls `Array.toString()` (joins with commas), not `Buffer.concat()`. This would produce garbled TUI output. Replaced with a separate `preview` string built incrementally and capped at 30KB. 3. **`prompt.ts` shell metadata called `Buffer.concat()` on every data chunk** — This is O(n²), the same problem as the original `output +=`. Fixed with the same capped preview approach. 4. **Removed `ACPSessionManager.dispose()`** — Nobody calls it. The class is instantiated in `ACPAgentBridge` with no lifecycle hook to the Instance state system. Cosmetic change that fixes nothing. 5. **Removed `AsyncQueue.close()`** — Nobody calls it. The two `AsyncQueue` instances in `tui.ts` are module-level singletons with no dispose lifecycle. Cosmetic change that fixes nothing. ### Final PR: 8 files, all meaningful | Fix | Files | What it prevents | |-----|-------|-----------------| | Signal handlers + `Instance.disposeAll()` | `index.ts`, `worker.ts`, `serve.ts` | **Zombie processes** — the primary cause of #12687 | | Buffer[] ring buffer + capped preview | `bash.ts`, `prompt.ts` | OOM from O(n²) string growth on verbose commands | | Shallow copy vs deep clone | `prompt.ts` | Memory spikes from `clone(msgs)` every prompt loop | | Dispose callbacks | `permission/next.ts`, `question/index.ts` | Promise closure leaks on instance disposal | | FileTime cleanup | `file/time.ts` | Unbounded growth of session read records | ### Testing - `tsgo --noEmit` passes across all 17 packages - 844 tests pass across 59 files, 0 failures - No behavioral changes — all fixes follow existing codebase patterns PR: https://github.com/anomalyco/opencode/pull/13183
yindo commented 2026-02-16 18:10:58 -05:00
Author
Owner
Copy Link
Copy Source

@dzianisv commented on GitHub (Feb 13, 2026):

Follow-up: Crash Log Analysis After PR #13186

After submitting PR #13186, I examined fresh macOS diagnostic reports to validate whether the fixes address the actual crash mechanisms. Here are the findings.

1. Node LSP Subprocess OOM (NEW — not addressed by PR #13186)

Report: ~/Library/Logs/DiagnosticReports/node-2026-02-12-101515.ips

A node subprocess (NOT the main opencode/bun process) crashed with EXC_CRASH / SIGABRT after only 7 minutes of runtime:

Exception Type:  EXC_CRASH (SIGABRT)
Termination Reason: V8::FatalProcessOutOfMemory → Heap::CollectGarbage failed

Stack trace shows:

  • node::OOMErrorHandler → v8::internal::V8::FatalProcessOutOfMemory
  • v8::internal::Heap::FatalProcessOutOfMemory → GC failed to reclaim enough memory
  • Allocation was in young generation (Runtime_AllocateInYoungGeneration) — rapid small allocations overwhelmed the nursery
  • Multiple V8 sweeper threads active (ConcurrentMajorSweeper) trying to reclaim memory but failing
  • Process launched from Terminal, lived ~7 minutes (10:07 → 10:15)

This is an LSP server (likely TypeScript server) spawned by opencode. It died from V8 heap exhaustion independently of anything our PR fixes. This aligns with @Thejuampi's report on PR #10392 that memory fixes didn't help for Java/jdtls projects — the LSP servers themselves are heavy memory consumers.

2. Revised Jetsam Analysis (page count math)

Re-examining the jetsam report with page count × 16KB conversion:

Process lifetimeMax (pages) Estimated Peak
opencode (largest) 90,891 ~1.4 GB
opencode 33,580 ~537 MB
opencode 27,978 ~448 MB
node (LSP) 11,027 ~176 MB
node (LSP) 10,935 ~175 MB
node (LSP) 5,913 ~94 MB
node (LSP) 5,881 ~94 MB
node (LSP) 2,906 ~46 MB

Multiple opencode + node processes consuming several GB combined. The zombie process problem (no signal handlers → orphaned processes accumulate) is confirmed as the primary driver.

3. What PR #13186 Does and Doesn't Fix

Addresses (~60-70% of the problem):

  • ✅ Zombie process accumulation — signal handlers now call Instance.disposeAll() on SIGTERM/SIGINT/SIGHUP
  • ✅ O(n²) string concat in 8 hot paths — reduces transient GC pressure significantly
  • ✅ Per-token serialization overhead — throttled flush in processor.ts
  • ✅ Missing dispose callbacks — permission/next.ts, question/index.ts, file/time.ts

Does NOT address (remaining ~30-40%):

  • ❌ LSP server memory — node subprocesses OOM independently. No --max-old-space-size limit is set when spawning LSP servers. Consider adding NODE_OPTIONS=--max-old-space-size=512 or similar.
  • ❌ Message re-loading — MessageV2.filterCompacted(MessageV2.stream(sessionID)) reads ALL messages from storage on every LLM round-trip. For long sessions, this grows unboundedly.
  • ❌ Deep clone in prompt loop — clone(msgs) at prompt.ts:577 temporarily doubles memory every iteration
  • ❌ Unbounded provider/instance caches — no LRU/TTL eviction
  • ❌ SSE broadcasting — JSON.stringify of full state on every event to every connected client

4. Recommended Next Steps

  1. Merge PR #13186 — the signal handler fix alone prevents the most catastrophic failure mode (10 zombie processes × ~1 GB each = kernel panic)
  2. Add LSP memory limits — cap node subprocess heap when spawning LSP servers (investigating this now)
  3. Tier 2 PR — message caching, prompt loop clone removal, provider cache eviction

The signal handlers + O(n²) fixes won't eliminate all memory growth, but they should prevent the kernel panics and forced reboots that make this issue critical.

@dzianisv commented on GitHub (Feb 13, 2026): ## Follow-up: Crash Log Analysis After PR #13186 After submitting PR #13186, I examined fresh macOS diagnostic reports to validate whether the fixes address the actual crash mechanisms. Here are the findings. --- ### 1. Node LSP Subprocess OOM (NEW — not addressed by PR #13186) **Report:** `~/Library/Logs/DiagnosticReports/node-2026-02-12-101515.ips` A `node` subprocess (NOT the main opencode/bun process) crashed with `EXC_CRASH / SIGABRT` after only **7 minutes** of runtime: ``` Exception Type: EXC_CRASH (SIGABRT) Termination Reason: V8::FatalProcessOutOfMemory → Heap::CollectGarbage failed ``` Stack trace shows: - `node::OOMErrorHandler` → `v8::internal::V8::FatalProcessOutOfMemory` - `v8::internal::Heap::FatalProcessOutOfMemory` → GC failed to reclaim enough memory - Allocation was in **young generation** (`Runtime_AllocateInYoungGeneration`) — rapid small allocations overwhelmed the nursery - Multiple V8 sweeper threads active (`ConcurrentMajorSweeper`) trying to reclaim memory but failing - Process launched from Terminal, lived ~7 minutes (10:07 → 10:15) This is an **LSP server** (likely TypeScript server) spawned by opencode. It died from V8 heap exhaustion independently of anything our PR fixes. This aligns with @Thejuampi's report on PR #10392 that memory fixes didn't help for Java/jdtls projects — the LSP servers themselves are heavy memory consumers. ### 2. Revised Jetsam Analysis (page count math) Re-examining the jetsam report with page count × 16KB conversion: | Process | `lifetimeMax` (pages) | Estimated Peak | |---------|----------------------|----------------| | opencode (largest) | 90,891 | ~1.4 GB | | opencode | 33,580 | ~537 MB | | opencode | 27,978 | ~448 MB | | node (LSP) | 11,027 | ~176 MB | | node (LSP) | 10,935 | ~175 MB | | node (LSP) | 5,913 | ~94 MB | | node (LSP) | 5,881 | ~94 MB | | node (LSP) | 2,906 | ~46 MB | Multiple opencode + node processes consuming **several GB combined**. The zombie process problem (no signal handlers → orphaned processes accumulate) is confirmed as the primary driver. ### 3. What PR #13186 Does and Doesn't Fix **Addresses (~60-70% of the problem):** - ✅ Zombie process accumulation — signal handlers now call `Instance.disposeAll()` on `SIGTERM`/`SIGINT`/`SIGHUP` - ✅ O(n²) string concat in 8 hot paths — reduces transient GC pressure significantly - ✅ Per-token serialization overhead — throttled flush in `processor.ts` - ✅ Missing dispose callbacks — `permission/next.ts`, `question/index.ts`, `file/time.ts` **Does NOT address (remaining ~30-40%):** - ❌ LSP server memory — node subprocesses OOM independently. No `--max-old-space-size` limit is set when spawning LSP servers. Consider adding `NODE_OPTIONS=--max-old-space-size=512` or similar. - ❌ Message re-loading — `MessageV2.filterCompacted(MessageV2.stream(sessionID))` reads ALL messages from storage on every LLM round-trip. For long sessions, this grows unboundedly. - ❌ Deep clone in prompt loop — `clone(msgs)` at `prompt.ts:577` temporarily doubles memory every iteration - ❌ Unbounded provider/instance caches — no LRU/TTL eviction - ❌ SSE broadcasting — `JSON.stringify` of full state on every event to every connected client ### 4. Recommended Next Steps 1. **Merge PR #13186** — the signal handler fix alone prevents the most catastrophic failure mode (10 zombie processes × ~1 GB each = kernel panic) 2. **Add LSP memory limits** — cap node subprocess heap when spawning LSP servers (investigating this now) 3. **Tier 2 PR** — message caching, prompt loop clone removal, provider cache eviction The signal handlers + O(n²) fixes won't eliminate all memory growth, but they should prevent the kernel panics and forced reboots that make this issue critical.
yindo commented 2026-02-16 18:10:58 -05:00
Author
Owner
Copy Link
Copy Source

@dzianisv commented on GitHub (Feb 15, 2026):

Forensic Investigation: macOS Kernel Panic Caused by opencode Memory Usage

Summary

On Feb 14 at 20:16, my MacBook Pro M1 (16 GB RAM) suffered a kernel panic caused by a watchdog timeout. Root cause: 25 simultaneous opencode processes consuming a combined 18.5 GB of memory on a 16 GB machine, which caused the kernel's memory compressor to bloat and starve the watchdog daemon.

Reboot Root Cause

Event Time Cause
Kernel panic Feb 14, 20:16 Watchdog timeout — watchdogd failed to check in for 94 seconds
Manual reset Feb 15, 13:08 Button reset (not a panic)

Memory state at panic time:

  • Physical RAM: 16 GB
  • Compressor bloated to 11.6 GB
  • Free memory: 14 MB
  • Wired: 3.4 GB
  • 742M compressions / 676M decompressions — CPU was thrashing on memory compression
  • Kernel couldn't schedule watchdogd → panic

opencode Memory Footprint

From Jetsam event logs near the panic:

  • 25 opencode processes running simultaneously (all in coalition 3713)
  • Total opencode memory: ~18.5 GB across 25 instances
  • Largest single instances: 4.8 GB, 4.1 GB, 2.9 GB
  • On Feb 13, a single opencode process reached 9.0 GB (lifetime max 10.2 GB)
  • opencode was flagged as largestProcess in multiple Jetsam events

Additionally, 6 diagnostic reports flagged opencode for excessive disk writes — up to 8.6 GB written in 30 minutes at 4.7 MB/s, with observed footprint growth from 868 MB → 1.6 GB (max 1.8 GB) in a single report window.

Causal Chain

25 opencode instances (18.5 GB total)
  → exceeded 16 GB physical RAM
  → kernel compressor bloated to 11.6 GB trying to keep up
  → CPU thrashing on compress/decompress (742M ops)
  → free memory dropped to 14 MB
  → watchdogd starved for 94 seconds
  → kernel panic (watchdog timeout)

Codebase Analysis: Memory Growth Sources

I analyzed the opencode codebase to identify what causes unbounded memory growth per session.

HIGH RISK

  1. Message cache in prompt loop (packages/opencode/src/session/prompt.ts, lines 274-376)

    • cache variable holds { order: string[], map: Map<string, MessageV2.WithParts> } that persists across the entire while(true) prompt loop
    • Grows with every message, never trimmed
    • Each MessageV2.WithParts contains all parts including full tool outputs, file contents as data URLs, etc.
    • In a long session this can hold hundreds of messages with multi-MB tool outputs

  2. toModelMessages() full reconstruction (packages/opencode/src/session/message-v2.ts, lines 491-713)

    • Called every iteration of the prompt loop
    • Reconstructs the full message array with all tool outputs from scratch
    • Creates duplicate copies of all messages + tool outputs simultaneously in memory alongside the cache

MODERATE RISK

  1. Streaming chunk accumulation (packages/opencode/src/session/processor.ts, lines 86-124) — chunks array per part ID grows during long streams, cleaned on finalize but large during streaming

  2. Session.messages() full DB load (packages/opencode/src/session/index.ts, lines 492-506) — loads ALL messages into memory, called by compaction/revert/summary

  3. Bus subscriptions (packages/opencode/src/bus/index.ts, lines 89-104) — can leak if callers don't properly unsubscribe

  4. FileTime state (packages/opencode/src/file/time.ts, lines 11-30) — grows with every file read, never pruned during session lifetime

Properly Bounded (low risk)

  • PTY buffer: 2 MB BUFFER_LIMIT
  • Tool output truncation: 50 KB limit (src/tool/truncation.ts)
  • LSP diagnostics: small entries

Next Steps

I'm experimenting with different approaches to decrease memory/VRAM usage on the dzianisv/main branch. Will report back with results.

Key areas to address:

  • Implement an LRU or sliding-window eviction policy on the message cache in the prompt loop
  • Avoid full message reconstruction on every loop iteration (incremental append instead)
  • Add session-level memory budgets or message count caps
  • Consider streaming tool outputs to disk instead of holding in memory
@dzianisv commented on GitHub (Feb 15, 2026): ## Forensic Investigation: macOS Kernel Panic Caused by opencode Memory Usage ### Summary On **Feb 14 at 20:16**, my MacBook Pro M1 (16 GB RAM) suffered a **kernel panic** caused by a watchdog timeout. Root cause: **25 simultaneous opencode processes** consuming a combined **18.5 GB of memory** on a 16 GB machine, which caused the kernel's memory compressor to bloat and starve the watchdog daemon. --- ### Reboot Root Cause | Event | Time | Cause | |---|---|---| | Kernel panic | Feb 14, 20:16 | Watchdog timeout — `watchdogd` failed to check in for 94 seconds | | Manual reset | Feb 15, 13:08 | Button reset (not a panic) | **Memory state at panic time:** - Physical RAM: 16 GB - Compressor bloated to **11.6 GB** - Free memory: **14 MB** - Wired: 3.4 GB - 742M compressions / 676M decompressions — CPU was thrashing on memory compression - Kernel couldn't schedule `watchdogd` → panic ### opencode Memory Footprint From Jetsam event logs near the panic: - **25 opencode processes** running simultaneously (all in coalition 3713) - Total opencode memory: **~18.5 GB** across 25 instances - Largest single instances: 4.8 GB, 4.1 GB, 2.9 GB - On Feb 13, a single opencode process reached **9.0 GB** (lifetime max 10.2 GB) - opencode was flagged as `largestProcess` in multiple Jetsam events Additionally, 6 diagnostic reports flagged opencode for **excessive disk writes** — up to 8.6 GB written in 30 minutes at 4.7 MB/s, with observed footprint growth from 868 MB → 1.6 GB (max 1.8 GB) in a single report window. ### Causal Chain ``` 25 opencode instances (18.5 GB total) → exceeded 16 GB physical RAM → kernel compressor bloated to 11.6 GB trying to keep up → CPU thrashing on compress/decompress (742M ops) → free memory dropped to 14 MB → watchdogd starved for 94 seconds → kernel panic (watchdog timeout) ``` --- ### Codebase Analysis: Memory Growth Sources I analyzed the opencode codebase to identify what causes unbounded memory growth per session. #### HIGH RISK 1. **Message cache in prompt loop** (`packages/opencode/src/session/prompt.ts`, lines 274-376) - `cache` variable holds `{ order: string[], map: Map<string, MessageV2.WithParts> }` that persists across the entire `while(true)` prompt loop - Grows with every message, **never trimmed** - Each `MessageV2.WithParts` contains all parts including full tool outputs, file contents as data URLs, etc. - In a long session this can hold hundreds of messages with multi-MB tool outputs 2. **`toModelMessages()` full reconstruction** (`packages/opencode/src/session/message-v2.ts`, lines 491-713) - Called every iteration of the prompt loop - Reconstructs the **full message array** with all tool outputs from scratch - Creates duplicate copies of all messages + tool outputs simultaneously in memory alongside the cache #### MODERATE RISK 3. **Streaming chunk accumulation** (`packages/opencode/src/session/processor.ts`, lines 86-124) — chunks array per part ID grows during long streams, cleaned on finalize but large during streaming 4. **`Session.messages()` full DB load** (`packages/opencode/src/session/index.ts`, lines 492-506) — loads ALL messages into memory, called by compaction/revert/summary 5. **Bus subscriptions** (`packages/opencode/src/bus/index.ts`, lines 89-104) — can leak if callers don't properly unsubscribe 6. **FileTime state** (`packages/opencode/src/file/time.ts`, lines 11-30) — grows with every file read, never pruned during session lifetime #### Properly Bounded (low risk) - PTY buffer: 2 MB `BUFFER_LIMIT` - Tool output truncation: 50 KB limit (`src/tool/truncation.ts`) - LSP diagnostics: small entries --- ### Next Steps I'm experimenting with different approaches to decrease memory/VRAM usage on the `dzianisv/main` branch. Will report back with results. Key areas to address: - Implement an LRU or sliding-window eviction policy on the message cache in the prompt loop - Avoid full message reconstruction on every loop iteration (incremental append instead) - Add session-level memory budgets or message count caps - Consider streaming tool outputs to disk instead of holding in memory
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label bug perf
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#8831
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?