langchain-ai/langserve
1
0
Fork 0
mirror of https://github.com/langchain-ai/langserve.git synced 2026-07-16 09:34:30 -04:00
Code Issues 136 Packages Projects Releases 10 Wiki Activity

Scaling to production -> OSError: [Errno 24] Too many open files socket.accept() out of system resource #211

New Issue
Closed
opened 2026-02-16 00:19:26 -05:00 by yindo · 1 comment
yindo commented 2026-02-16 00:19:26 -05:00
Owner
Copy Link
Copy Source

Originally created by @lukasugar on GitHub (Jul 24, 2024).

Problem

When my LangServe app gets ~1000 concurrent requests, it breaks with error:

OSError: [Errno 24] Too many open files
socket.accept() out of system resource

Mitigation/quickfix

I've checked the soft ulimit of the VM, it was only 1024, while the hard limit is 524288. I've increased the soft limit to be 100000, which should mitigate the issue for now.

Better way of doing it?

I'm curious if there's a better way of handling this issue. Even with the increased limit of allowed open files, is there something I can do in my app to make it better/more resilient?

What my code looks like

I define chains like this, straightforward:

review_text_chain = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            REVIEW_SYSTEM_PROMPT,
        ),
        ("user", "{text}"),
    ]
) | ChatOpenAI(model="gpt-4o")

and pass them to the router:

# python
from fastapi import APIRouter
from langserve import add_routes

router = APIRouter()

add_routes(router, review_text_chain, path="/api/v1/review_text")

I'm calling this service from a separatenestjs application, like this:

// typescript
private async callAiReview(document: string): Promise<any> {
    const analysisResponse = await fetch('www.path_to_my_endpoint/api/v1/review_text/invoke', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: document,
    });

    return await analysisResponse.json();
  }

There's a bunch of documents, and for each I'm calling callAiReview method.

Are there things in the app that can be improved? Maybe async?

Should I use async in LangServe? How?

I'm aware that I could use batch instead of invoke, but other than that, are there improvements to be made?

How to make LangServe work with production load?

Originally created by @lukasugar on GitHub (Jul 24, 2024). # Problem When my LangServe app gets ~1000 concurrent requests, it breaks with error: ``` OSError: [Errno 24] Too many open files socket.accept() out of system resource ``` # Mitigation/quickfix I've checked the soft `ulimit` of the VM, it was only `1024`, while the hard limit is `524288`. I've increased the soft limit to be `100000`, which should mitigate the issue for now. # Better way of doing it? I'm curious if there's a better way of handling this issue. Even with the increased limit of allowed open files, is there something I can do in my app to make it better/more resilient? # What my code looks like I define chains like this, straightforward: ``` review_text_chain = ChatPromptTemplate.from_messages( [ ( "system", REVIEW_SYSTEM_PROMPT, ), ("user", "{text}"), ] ) | ChatOpenAI(model="gpt-4o") ``` and pass them to the router: ``` # python from fastapi import APIRouter from langserve import add_routes router = APIRouter() add_routes(router, review_text_chain, path="/api/v1/review_text") ``` I'm calling this service from a separate`nestjs` application, like this: ``` // typescript private async callAiReview(document: string): Promise<any> { const analysisResponse = await fetch('www.path_to_my_endpoint/api/v1/review_text/invoke', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: document, }); return await analysisResponse.json(); } ``` There's a bunch of documents, and for each I'm calling `callAiReview` method. # Are there things in the app that can be improved? Maybe **async**? Should I use `async` in LangServe? How? I'm aware that I could use `batch` instead of `invoke`, but other than that, are there improvements to be made? How to make LangServe work with production load?
yindo closed this issue 2026-02-16 00:19:26 -05:00
yindo commented 2026-02-16 00:19:26 -05:00
Author
Owner
Copy Link
Copy Source

@eyurtsev commented on GitHub (Aug 1, 2024):

Duplicate of https://github.com/langchain-ai/langserve/issues/717.

Issue likely stemmed either using a misconfigured langsmith tracing client or being rate limited by the langsmith client.

@eyurtsev commented on GitHub (Aug 1, 2024): Duplicate of https://github.com/langchain-ai/langserve/issues/717. Issue likely stemmed either using a misconfigured langsmith tracing client or being rate limited by the langsmith client.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
duplicate
enhancement
help wanted
in-progress
investigate
not langserve
playground
pull-request
Mirrored from GitHub Pull Request
 question
stale
streaming
unable-to-reproduce
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langserve#211
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?