[PR #265] [MERGED] feat: use a thread pool to orchestrate deployments #361

New Issue

2026-02-16T01:17:02-05:00

yindo commented

2026-02-16 01:17:02 -05:00

📋 Pull Request Information

Original PR: https://github.com/run-llama/llama_deploy/pull/265
Author: @masci
Created: 9/23/2024
Status: ✅ Merged
Merged: 9/23/2024
Merged by: @logan-markewich

Base: main ← Head: massi/threadpool

📝 Commits (2)

5c3b3d5 use a thread pool to orchestrate deployments
47f6271 remove dead code

📊 Changes

3 files changed (+59 additions, -32 deletions)

View changed files

📝 llama_deploy/apiserver/deployment.py (+27 -19)
📝 tests/apiserver/data/git_service.yaml (+1 -1)
📝 tests/apiserver/test_deployment.py (+31 -12)

📄 Description

Instead of manually managing threads for each deployment, we use a ThreadPool to do that.

Notes for the reviewer:

The responsibility of spawning the thread to accomodate the asyncio loop was removed from the Deployment class, that now is only responsible for implementing the start task
The thread pool has a fixed size, I took the chance to reflect this into the manager API with the max_deployments parameter because it makes sense to cap the number of deployments.
The manager has now a serve method that runs forever to keep the thread pool alive indefinitely

Usage exampe of this version of the manager:

manager = Manager(tmp_path)

# Run in the background, we use threads here but it could be a different process
t = threading.Thread(target=asyncio.run, args=(manager.serve(),))
t.start()

# While the manager is running it accepts new deployments
manager.deploy(config1)
manager.deploy(config2)

# Block until CTRL+C
t.join()

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/run-llama/llama_deploy/pull/265 **Author:** [@masci](https://github.com/masci) **Created:** 9/23/2024 **Status:** ✅ Merged **Merged:** 9/23/2024 **Merged by:** [@logan-markewich](https://github.com/logan-markewich) **Base:** `main` ← **Head:** `massi/threadpool` --- ### 📝 Commits (2) - [`5c3b3d5`](https://github.com/run-llama/llama_deploy/commit/5c3b3d52a1e5244a76de2aeea7bef1651af720e4) use a thread pool to orchestrate deployments - [`47f6271`](https://github.com/run-llama/llama_deploy/commit/47f6271044904caa76a0334a4c8bfc08989cedb4) remove dead code ### 📊 Changes **3 files changed** (+59 additions, -32 deletions) <details> <summary>View changed files</summary> 📝 `llama_deploy/apiserver/deployment.py` (+27 -19) 📝 `tests/apiserver/data/git_service.yaml` (+1 -1) 📝 `tests/apiserver/test_deployment.py` (+31 -12) </details> ### 📄 Description Instead of manually managing threads for each deployment, we use a `ThreadPool` to do that. Notes for the reviewer: - The responsibility of spawning the thread to accomodate the asyncio loop was removed from the `Deployment` class, that now is only responsible for implementing the `start` task - The thread pool has a fixed size, I took the chance to reflect this into the manager API with the `max_deployments` parameter because it makes sense to cap the number of deployments. - The manager has now a `serve` method that runs forever to keep the thread pool alive indefinitely Usage exampe of this version of the manager: ``` manager = Manager(tmp_path) # Run in the background, we use threads here but it could be a different process t = threading.Thread(target=asyncio.run, args=(manager.serve(),)) t.start() # While the manager is running it accepts new deployments manager.deploy(config1) manager.deploy(config2) # Block until CTRL+C t.join() ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-02-16 01:17:02 -05:00

yindo closed this issue

2026-02-16 01:17:02 -05:00

yindo referenced this issue

2026-02-16 01:17:24 -05:00

[PR #413] [MERGED] refact: Simple message queue refactoring #448

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: run-llama/llama_deploy#361