[GH-ISSUE #338] [Bug]: ZIP downloads buffer the entire archive in memory [Introduced in 2.1.0] #111

New Issue

2026-06-06T22:09:11-04:00

yindo commented

2026-06-06 22:09:11 -04:00

Originally created by @john-ramsey on GitHub (Jun 5, 2026).
Original GitHub issue: https://github.com/vxcontrol/pentagi/issues/338

Originally assigned to: @asdek on GitHub.

Affected Component

Core Services (Frontend UI/Backend API)

Describe the bug

What happened:
Directory and multi-path ZIP downloads build the entire archive in a bytes.Buffer and only then send it, so the whole compressed archive is held on the heap before any response byte is written. Combined with the 2 GB upload/pull limits, an authenticated user with default User-role resource/flow-file permissions can store (or pull) a large, incompressible directory and request it as a ZIP, forcing the API process to allocate memory proportional to the archive size. A handful of concurrent requests can exhaust process/host memory and crash or severely degrade the API.

Single-file downloads are fine since they stream from disk. Only the directory / multi-path ZIP paths are affected.

WHat should happen:
We should probably create the zip writer over the response writer (zw := zip.NewWriter(c.Writer)) after setting Content-Type: application/zip and Content-Disposition, write entries, then zw.Close().

Steps to Reproduce

As a User-role account, upload (or pull from a flow container) a directory of incompressible data.
Request the directory (or a multi-path selection) from the ZIP download endpoint.
The handler calls ZipResources / ZipDirectory / ZipRelativePaths with a bytes.Buffer; resident heap grows to roughly the archive size before the response starts.
Issue a few concurrent requests → Go runtime fatal out-of-memory.

A standalone harness that calls the real ZipResources with a bytes.Buffer sink shows heap growth proportional to archive size (≈32 MiB incompressible input → ≈33.5 MiB ZIP → ≈105 MiB max heap) and crashes with a runtime OOM under a constrained ulimit -v, with the stack going through bytes.growSlice → bytes.(*Buffer).Write → archive/zip → io.Copy → resources.ZipResources.

System Configuration

Docker compose

Logs and Artifacts

No response

Screenshots or Recordings

No response

Verification

I have checked that this issue hasn't been already reported
I have provided all relevant configuration files (with sensitive data removed)
I have included relevant logs and error messages
I am running the latest version of PentAGI

Originally created by @john-ramsey on GitHub (Jun 5, 2026). Original GitHub issue: https://github.com/vxcontrol/pentagi/issues/338 Originally assigned to: @asdek on GitHub. ### Affected Component Core Services (Frontend UI/Backend API) ### Describe the bug What happened: Directory and multi-path ZIP downloads build the **entire** archive in a `bytes.Buffer` and only then send it, so the whole compressed archive is held on the heap before any response byte is written. Combined with the 2 GB upload/pull limits, an authenticated user with default `User`-role resource/flow-file permissions can store (or pull) a large, incompressible directory and request it as a ZIP, forcing the API process to allocate memory proportional to the archive size. A handful of concurrent requests can exhaust process/host memory and crash or severely degrade the API. Single-file downloads are fine since they stream from disk. Only the directory / multi-path ZIP paths are affected. WHat should happen: We should probably create the zip writer over the response writer (`zw := zip.NewWriter(c.Writer)`) after setting `Content-Type: application/zip` and `Content-Disposition`, write entries, then `zw.Close()`. ### Steps to Reproduce 1. As a `User`-role account, upload (or pull from a flow container) a directory of incompressible data. 2. Request the directory (or a multi-path selection) from the ZIP download endpoint. 3. The handler calls `ZipResources` / `ZipDirectory` / `ZipRelativePaths` with a `bytes.Buffer`; resident heap grows to roughly the archive size before the response starts. 4. Issue a few concurrent requests → Go runtime fatal out-of-memory. A standalone harness that calls the real `ZipResources` with a `bytes.Buffer` sink shows heap growth proportional to archive size (≈32 MiB incompressible input → ≈33.5 MiB ZIP → ≈105 MiB max heap) and crashes with a runtime OOM under a constrained `ulimit -v`, with the stack going through `bytes.growSlice` → `bytes.(*Buffer).Write` → `archive/zip` → `io.Copy` → `resources.ZipResources`. ### System Configuration - [x] Docker compose ### Logs and Artifacts _No response_ ### Screenshots or Recordings _No response_ ### Verification - [x] I have checked that this issue hasn't been already reported - [x] I have provided all relevant configuration files (with sensitive data removed) - [ ] I have included relevant logs and error messages - [x] I am running the latest version of PentAGI

yindo added the bug label 2026-06-06 22:09:11 -04:00

yindo referenced this issue

2026-06-06 22:09:31 -04:00

[PR #111] [MERGED] Upgrade GitHub Actions for Node 24 compatibility #163

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#111