langchain-ai/auto-evaluator
1
0
Fork 0
mirror of https://github.com/langchain-ai/auto-evaluator.git synced 2026-07-19 14:43:34 -04:00
Code Issues 21 Packages Projects Releases Wiki Activity

[GH-ISSUE #72] Prod (likely) hitting OOM #37

New Issue
Closed
opened 2026-02-15 18:17:08 -05:00 by yindo · 1 comment
yindo commented 2026-02-15 18:17:08 -05:00
Owner
Copy Link
Copy Source

Originally created by @rlancemartin on GitHub (Apr 28, 2023).
Original GitHub issue: https://github.com/langchain-ai/auto-evaluator/issues/72

TL;DR we see OOM (or server returning

These are 15 large pdfs fail (503 server error) on prod, but work on local:

curl -X POST \
-F "files=@2005.14165.pdf" \
-F "files=@2101.03961.pdf" \
-F "files=@2112.00861.pdf" \
-F "files=@2112.04426.pdf" \
-F "files=@2112.06905.pdf" \
-F "files=@2112.11446.pdf" \
-F "files=@2201.08239.pdf" \
-F "files=@2203.07814.pdf" \
-F "files=@2203.15556.pdf" \
-F "files=@2204.02311.pdf" \
-F "files=@2204.06745.pdf" \
-F "files=@2204.14198.pdf" \
-F "files=@2205.06175.pdf" \
-F "files=@2211.05100.pdf" \
-F "files=@2211.09085.pdf" \
-F "num_eval_questions=1" \
-F "chunk_chars=1000" \
-F "overlap=100" \
-F "split_method=RecursiveTextSplitter" \
-F "retriever_type=similarity-search" \
-F "embeddings=OpenAI" \
-F "model_version=gpt-3.5-turbo" \
-F "grade_prompt=Fast" \
-F "num_neighbors=3" \
https://auto-evaluator-production.up.railway.app/evaluator-stream

The first 7 work on prod w/ similarity-search:

curl -X POST \                                                                                        
-F "files=@2005.14165.pdf" \
-F "files=@2101.03961.pdf" \
-F "files=@2112.00861.pdf" \
-F "files=@2112.04426.pdf" \
-F "files=@2112.06905.pdf" \
-F "files=@2112.11446.pdf" \
-F "files=@2201.08239.pdf" \
-F "num_eval_questions=1" \
-F "chunk_chars=1000" \
-F "overlap=100" \
-F "split_method=RecursiveTextSplitter" \
-F "retriever_type=similarity-search" \
-F "embeddings=OpenAI" \
-F "model_version=gpt-3.5-turbo" \
-F "grade_prompt=Fast" \
-F "num_neighbors=3" \
https://auto-evaluator-production.up.railway.app/evaluator-stream

They are ~17 MB.

Adding any additional files fail (e.g., below fails):

curl -X POST \
-F "files=@2005.14165.pdf" \
-F "files=@2101.03961.pdf" \
-F "files=@2112.00861.pdf" \
-F "files=@2112.04426.pdf" \
-F "files=@2112.06905.pdf" \
-F "files=@2112.11446.pdf" \
-F "files=@2201.08239.pdf" \
-F "files=@2203.07814.pdf" \
-F "files=@2203.15556.pdf" \
-F "num_eval_questions=1" \
-F "chunk_chars=1000" \
-F "overlap=100" \
-F "split_method=RecursiveTextSplitter" \
-F "retriever_type=similarity-search" \
-F "embeddings=OpenAI" \
-F "model_version=gpt-3.5-turbo" \
-F "grade_prompt=Fast" \
-F "num_neighbors=3" \
https://auto-evaluator-production.up.railway.app/evaluator-stream
Originally created by @rlancemartin on GitHub (Apr 28, 2023). Original GitHub issue: https://github.com/langchain-ai/auto-evaluator/issues/72 TL;DR we see OOM (or server returning These are 15 large pdfs **fail** (`503 server error`) on prod, but work on local: ``` curl -X POST \ -F "files=@2005.14165.pdf" \ -F "files=@2101.03961.pdf" \ -F "files=@2112.00861.pdf" \ -F "files=@2112.04426.pdf" \ -F "files=@2112.06905.pdf" \ -F "files=@2112.11446.pdf" \ -F "files=@2201.08239.pdf" \ -F "files=@2203.07814.pdf" \ -F "files=@2203.15556.pdf" \ -F "files=@2204.02311.pdf" \ -F "files=@2204.06745.pdf" \ -F "files=@2204.14198.pdf" \ -F "files=@2205.06175.pdf" \ -F "files=@2211.05100.pdf" \ -F "files=@2211.09085.pdf" \ -F "num_eval_questions=1" \ -F "chunk_chars=1000" \ -F "overlap=100" \ -F "split_method=RecursiveTextSplitter" \ -F "retriever_type=similarity-search" \ -F "embeddings=OpenAI" \ -F "model_version=gpt-3.5-turbo" \ -F "grade_prompt=Fast" \ -F "num_neighbors=3" \ https://auto-evaluator-production.up.railway.app/evaluator-stream ``` The first 7 **work** on `prod` w/ `similarity-search`: ``` curl -X POST \ -F "files=@2005.14165.pdf" \ -F "files=@2101.03961.pdf" \ -F "files=@2112.00861.pdf" \ -F "files=@2112.04426.pdf" \ -F "files=@2112.06905.pdf" \ -F "files=@2112.11446.pdf" \ -F "files=@2201.08239.pdf" \ -F "num_eval_questions=1" \ -F "chunk_chars=1000" \ -F "overlap=100" \ -F "split_method=RecursiveTextSplitter" \ -F "retriever_type=similarity-search" \ -F "embeddings=OpenAI" \ -F "model_version=gpt-3.5-turbo" \ -F "grade_prompt=Fast" \ -F "num_neighbors=3" \ https://auto-evaluator-production.up.railway.app/evaluator-stream ``` **They are ~17 MB.** Adding any additional files fail (e.g., below fails): ``` curl -X POST \ -F "files=@2005.14165.pdf" \ -F "files=@2101.03961.pdf" \ -F "files=@2112.00861.pdf" \ -F "files=@2112.04426.pdf" \ -F "files=@2112.06905.pdf" \ -F "files=@2112.11446.pdf" \ -F "files=@2201.08239.pdf" \ -F "files=@2203.07814.pdf" \ -F "files=@2203.15556.pdf" \ -F "num_eval_questions=1" \ -F "chunk_chars=1000" \ -F "overlap=100" \ -F "split_method=RecursiveTextSplitter" \ -F "retriever_type=similarity-search" \ -F "embeddings=OpenAI" \ -F "model_version=gpt-3.5-turbo" \ -F "grade_prompt=Fast" \ -F "num_neighbors=3" \ https://auto-evaluator-production.up.railway.app/evaluator-stream ```
yindo closed this issue 2026-02-15 18:17:08 -05:00
yindo commented 2026-02-15 18:17:09 -05:00
Author
Owner
Copy Link
Copy Source

@rlancemartin commented on GitHub (Apr 28, 2023):

It was not OOM:

1/ Prod has been periodically crashing. It may be related to: https://github.com/langchain-ai/auto-evaluator/issues/76.

2/ Prod is slow with file loading, and the client will time-out: https://github.com/langchain-ai/auto-evaluator/issues/74

@rlancemartin commented on GitHub (Apr 28, 2023): It was not OOM: 1/ Prod has been periodically crashing. It may be related to: https://github.com/langchain-ai/auto-evaluator/issues/76. 2/ Prod is slow with file loading, and the client will time-out: https://github.com/langchain-ai/auto-evaluator/issues/74
yindo changed title from Prod (likely) hitting OOM to [GH-ISSUE #72] Prod (likely) hitting OOM 2026-06-05 17:14:10 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
good first issue
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/auto-evaluator#37
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?