[PR #353] [MERGED] feat: make the kafka topic configurable #413

New Issue

2026-02-16T01:17:14-05:00

yindo commented

2026-02-16 01:17:14 -05:00

📋 Pull Request Information

Original PR: https://github.com/run-llama/llama_deploy/pull/353
Author: @masci
Created: 11/7/2024
Status: ✅ Merged
Merged: 11/9/2024
Merged by: @logan-markewich

Base: main ← Head: massi/349

📝 Commits (8)

b65ee1b feat: make the kafka topic configurable
3d3016a temporary ignore for return type
b713795 mark test as e2e
0426b6b remove cwd
a7e9b11 do not use docker-compose alias
f00c538 clean shutdown for message queue
f9d58be address review feedback
ac5a92a adjust unit test

📊 Changes

11 files changed (+148 additions, -114 deletions)

View changed files

➕ e2e_tests/message_queues/__init__.py (+0 -0)
➕ e2e_tests/message_queues/message_queue_kafka/__init__.py (+0 -0)
➕ e2e_tests/message_queues/message_queue_kafka/docker-compose.yml (+25 -0)
➕ e2e_tests/message_queues/message_queue_kafka/test_message_queue.py (+54 -0)
📝 llama_deploy/apiserver/deployment.py (+1 -1)
📝 llama_deploy/deploy/deploy.py (+8 -10)
📝 llama_deploy/message_queues/__init__.py (+3 -2)
📝 llama_deploy/message_queues/apache_kafka.py (+35 -85)
📝 llama_deploy/message_queues/base.py (+11 -9)
📝 tests/apiserver/test_deployment.py (+1 -1)
📝 tests/message_queues/test_apache_kafka.py (+10 -6)

📄 Description

Part of #349 (not a fix until we propagate the topic settings up to the control plane and api server)

Problem

Currently we use the name of the "message type" string to create the Kafka topic that'll be used in the message queue. Problem is that different instances of control planes using the same Kafka instances will step into each other, with catastrific effects. Note that this problem is common to other message queue brokers like S3, so eventually this fix will be rolled out for all the supported queues.

Solution

Make the Kafka topic name configurable, so that different control planes will use different topics. This PR only makes the topic configurable, another PR will make use of this new feature and resolve the original issue.

Notes

While working at the fix, I noticed a few things we can improve and already implemented in this PR in a backward compatible manner:

I am not sure the base class for queues should be a pydantic model, that's very limiting when you need to treat the class as a regular class (get and set local instance variables for example). I think a plain abstract class derived from ABC will make the job.
There's a lot of duplication in the configuration settings: the class constructor, the config settings model, the static methods... I think we should have a single source of truth, being the config settings model. For Kafka I made the constructor take the config model instance directly.
I moved the original code behind the if __name__ == "__main__" that was used for testing into a proper e2e test. Tests for message queues will most certainly need docker to be avaiable, I'll add the required CI bits (this part is WIP, you'll see it's done when tests are all green).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/run-llama/llama_deploy/pull/353 **Author:** [@masci](https://github.com/masci) **Created:** 11/7/2024 **Status:** ✅ Merged **Merged:** 11/9/2024 **Merged by:** [@logan-markewich](https://github.com/logan-markewich) **Base:** `main` ← **Head:** `massi/349` --- ### 📝 Commits (8) - [`b65ee1b`](https://github.com/run-llama/llama_deploy/commit/b65ee1bb66010713231bd515b8c113b3fae30c31) feat: make the kafka topic configurable - [`3d3016a`](https://github.com/run-llama/llama_deploy/commit/3d3016ae37f0dbda3923cc0ce8e382acc0087e95) temporary ignore for return type - [`b713795`](https://github.com/run-llama/llama_deploy/commit/b71379533db518b04eaa680b978c7be5b869cd85) mark test as e2e - [`0426b6b`](https://github.com/run-llama/llama_deploy/commit/0426b6b3a46f5a45fea5cf0b2af5005d4a47d3bc) remove cwd - [`a7e9b11`](https://github.com/run-llama/llama_deploy/commit/a7e9b118afb172bfad01637a95510d3ec3b0b196) do not use docker-compose alias - [`f00c538`](https://github.com/run-llama/llama_deploy/commit/f00c538afd3b3aa84d7a55f6668f437128ee8ec7) clean shutdown for message queue - [`f9d58be`](https://github.com/run-llama/llama_deploy/commit/f9d58beec97f666d7fc846d5c6b9bc352ad41ab5) address review feedback - [`ac5a92a`](https://github.com/run-llama/llama_deploy/commit/ac5a92a81547131449964e13bd656b027ed9e202) adjust unit test ### 📊 Changes **11 files changed** (+148 additions, -114 deletions) <details> <summary>View changed files</summary> ➕ `e2e_tests/message_queues/__init__.py` (+0 -0) ➕ `e2e_tests/message_queues/message_queue_kafka/__init__.py` (+0 -0) ➕ `e2e_tests/message_queues/message_queue_kafka/docker-compose.yml` (+25 -0) ➕ `e2e_tests/message_queues/message_queue_kafka/test_message_queue.py` (+54 -0) 📝 `llama_deploy/apiserver/deployment.py` (+1 -1) 📝 `llama_deploy/deploy/deploy.py` (+8 -10) 📝 `llama_deploy/message_queues/__init__.py` (+3 -2) 📝 `llama_deploy/message_queues/apache_kafka.py` (+35 -85) 📝 `llama_deploy/message_queues/base.py` (+11 -9) 📝 `tests/apiserver/test_deployment.py` (+1 -1) 📝 `tests/message_queues/test_apache_kafka.py` (+10 -6) </details> ### 📄 Description Part of #349 (not a fix until we propagate the topic settings up to the control plane and api server) ## Problem Currently we use the name of the "message type" string to create the Kafka topic that'll be used in the message queue. Problem is that different instances of control planes using the same Kafka instances will step into each other, with catastrific effects. Note that this problem is common to other message queue brokers like S3, so eventually this fix will be rolled out for all the supported queues. ## Solution Make the Kafka topic name configurable, so that different control planes will use different topics. This PR only makes the topic configurable, another PR will make use of this new feature and resolve the original issue. ## Notes While working at the fix, I noticed a few things we can improve and already implemented in this PR in a backward compatible manner: - I am not sure the base class for queues should be a pydantic model, that's very limiting when you need to treat the class as a regular class (get and set local instance variables for example). I think a plain abstract class derived from `ABC` will make the job. - There's a lot of duplication in the configuration settings: the class constructor, the config settings model, the static methods... I think we should have a single source of truth, being the config settings model. For Kafka I made the constructor take the config model instance directly. - I moved the original code behind the `if __name__ == "__main__"` that was used for testing into a proper e2e test. Tests for message queues will most certainly need docker to be avaiable, I'll add the required CI bits (this part is WIP, you'll see it's done when tests are all green). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-02-16 01:17:14 -05:00

yindo closed this issue

2026-02-16 01:17:15 -05:00

yindo referenced this issue

2026-02-16 01:17:24 -05:00

[PR #413] [MERGED] refact: Simple message queue refactoring #448

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: run-llama/llama_deploy#413