run-llama/llama_cloud_services
1
0
Fork 0
mirror of https://github.com/run-llama/llama_cloud_services.git synced 2026-08-26 16:56:36 -04:00
Code Issues 352 Packages Projects Releases 10 Wiki Activity

Misuse of httpx Client creating unnecessary overhead. #393

New Issue
Closed
opened 2026-02-16 00:17:42 -05:00 by yindo · 1 comment
yindo commented 2026-02-16 00:17:42 -05:00
Owner
Copy Link
Copy Source

Originally created by @apostolos-geyer on GitHub (Dec 25, 2024).

in llama_parse/base.py there are several instances where the usage of the client is such that some of the main benefits of using the httpx library are completely circumvented.

Methods such as _create_job, aload_data, aget_json, aget_assets use the client_context() context manager, which—unless a custom client is provided—create a new client on each invocation.

This means that HTTPX's strengths in connection pooling cannot be used as we recreate the client instance for each request, hence the TCP and TLS handshakes must be repeated. Obviously TCP is fairly lightweight but the the TLS handshake in particular is not inexpensive when parse jobs are even moderately sized.

E.g. on my home connection, where the total time for the TCP + TLS Handshake is about 63ms:

2024-12-25 14:42:17,194 - DEBUG - connect_tcp.started host='api.cloud.llamaindex.ai' port=443 local_address=None timeout=2000 socket_options=None
2024-12-25 14:42:17,224 - DEBUG - connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x12c085010>
2024-12-25 14:42:17,224 - DEBUG - start_tls.started ssl_context=<ssl.SSLContext object at 0x14dea0f50> server_hostname='api.cloud.llamaindex.ai' timeout=2000
2024-12-25 14:42:17,257 - DEBUG - start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x12c099430>

Over something like 5000 jobs, this would amount to 5+ minutes of JUST connection establishment.

Given the lack of bindings for any kind of batch job endpoint, I would say this is not an immaterial issue.

This is inefficient not only for your end users, but your own servers. TLS handshake should be done at most once per user-session.

I would be happy to begin contributing improvements if the maintainers are open, however based on the volume of issues with no responses I'd rather wait until I get a response from a maintainer on this than jump in right away.

Originally created by @apostolos-geyer on GitHub (Dec 25, 2024). in `llama_parse/base.py` there are several instances where the usage of the client is such that some of the main benefits of using the `httpx` library are completely circumvented. Methods such as `_create_job`, `aload_data`, `aget_json`, `aget_assets` use the `client_context()` context manager, which—unless a custom client is provided—create a new client on each invocation. This means that HTTPX's strengths in connection pooling cannot be used as we recreate the client instance for each request, hence the TCP and TLS handshakes must be repeated. Obviously TCP is fairly lightweight but the the TLS handshake in particular is not inexpensive when parse jobs are even moderately sized. E.g. on my home connection, where the total time for the TCP + TLS Handshake is about 63ms: ``` 2024-12-25 14:42:17,194 - DEBUG - connect_tcp.started host='api.cloud.llamaindex.ai' port=443 local_address=None timeout=2000 socket_options=None 2024-12-25 14:42:17,224 - DEBUG - connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x12c085010> 2024-12-25 14:42:17,224 - DEBUG - start_tls.started ssl_context=<ssl.SSLContext object at 0x14dea0f50> server_hostname='api.cloud.llamaindex.ai' timeout=2000 2024-12-25 14:42:17,257 - DEBUG - start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x12c099430> ``` Over something like 5000 jobs, this would amount to **5+ minutes** of **JUST** connection establishment. Given the lack of bindings for any kind of batch job endpoint, I would say this is not an immaterial issue. This is inefficient not only for your end users, but your own servers. TLS handshake should be done at most once per user-session. I would be happy to begin contributing improvements if the maintainers are open, however based on the volume of issues with no responses I'd rather wait until I get a response from a maintainer on this than jump in right away.
yindo closed this issue 2026-02-16 00:17:42 -05:00
yindo commented 2026-02-16 00:17:43 -05:00
Author
Owner
Copy Link
Copy Source

@logan-markewich commented on GitHub (Dec 25, 2024):

Feel free to make a pr! Sounds like you understand best what should be changed

@logan-markewich commented on GitHub (Dec 25, 2024): Feel free to make a pr! Sounds like you understand best what should be changed
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
LlamaCloud
LlamaExtract
LlamaParse
bug
client
complex layout
documentation
duplicate
enhancement
figure
font
header footer
help wanted
ocr
parser
pdf_debug
pdf_debug
pull-request
Mirrored from GitHub Pull Request
 python-lib
question
server_issue
server_issue
ui
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/llama_cloud_services#393
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?