langchain-ai/helm
1
0
Fork 0
mirror of https://github.com/langchain-ai/helm.git synced 2026-07-22 14:45:24 -04:00
Code Issues 24 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #568] ls_timestamp label on pod templates causes cascading rollouts during slow deployments #24

New Issue
Closed
opened 2026-02-16 01:16:41 -05:00 by yindo · 10 comments
yindo commented 2026-02-16 01:16:41 -05:00
Owner
Copy Link
Copy Source

Originally created by @KranthiKRK on GitHub (Feb 4, 2026).
Original GitHub issue: https://github.com/langchain-ai/helm/issues/568

Description

The LangGraph dataplane operator propagates the ls_timestamp label from the LGP CR to pod templates. Since this label is updated during deployment reconciliation, it triggers new rolling deployments while a rollout is still in progress, causing cascading restarts.

Impact

  • A deployment that should take ~5 minutes took 37 minutes due to 6 cascading rollouts
  • Test cluster shows 150 deployment revisions in 105 days, many triggered by timestamp-only changes

Evidence

During an incident, we observed 6 rollouts in 30 minutes with the same image - only ls_timestamp changed:

Revision Created (UTC) ls_timestamp Image Changed?
8 19:57:03 1770235023 No
9 19:58:49 1770235129 No
10 20:07:34 1770235654 No
11 20:11:18 1770235878 No
12 20:18:07 1770236287 No
13 20:27:39 1770236859 No

Diffing consecutive ReplicaSets confirms only ls_timestamp changed:

- "ls_timestamp": "1770236287"
+ "ls_timestamp": "1770236859"

Root Cause

The listener updates ls_timestamp in the LGP CR while waiting for deployments to become ready. Since this label is propagated to pod templates, Kubernetes interprets it as a spec change and triggers a new rollout, resetting deployment progress.

Expected Behavior

Mutable metadata like ls_timestamp should either:

  1. Be stored in annotations instead of labels (annotations don't trigger rollouts)
  2. Be excluded from pod template labels
  3. Not be updated while a deployment is in progress

Environment

  • langgraph-operator: 0.1.21
  • langgraph-dataplane chart: 0.2.13
  • Kubernetes: EKS 1.33
Originally created by @KranthiKRK on GitHub (Feb 4, 2026). Original GitHub issue: https://github.com/langchain-ai/helm/issues/568 ## Description The LangGraph dataplane operator propagates the `ls_timestamp` label from the LGP CR to pod templates. Since this label is updated during deployment reconciliation, it triggers new rolling deployments while a rollout is still in progress, causing cascading restarts. ## Impact - A deployment that should take ~5 minutes took **37 minutes** due to 6 cascading rollouts - Test cluster shows **150 deployment revisions** in 105 days, many triggered by timestamp-only changes ## Evidence During an incident, we observed 6 rollouts in 30 minutes with the **same image** - only `ls_timestamp` changed: | Revision | Created (UTC) | ls_timestamp | Image Changed? | |----------|---------------|--------------|----------------| | 8 | 19:57:03 | 1770235023 | No | | 9 | 19:58:49 | 1770235129 | No | | 10 | 20:07:34 | 1770235654 | No | | 11 | 20:11:18 | 1770235878 | No | | 12 | 20:18:07 | 1770236287 | No | | 13 | 20:27:39 | 1770236859 | No | Diffing consecutive ReplicaSets confirms only `ls_timestamp` changed: ```diff - "ls_timestamp": "1770236287" + "ls_timestamp": "1770236859" ``` ## Root Cause The listener updates `ls_timestamp` in the LGP CR while waiting for deployments to become ready. Since this label is propagated to pod templates, Kubernetes interprets it as a spec change and triggers a new rollout, resetting deployment progress. ## Expected Behavior Mutable metadata like `ls_timestamp` should either: 1. Be stored in **annotations** instead of labels (annotations don't trigger rollouts) 2. Be excluded from pod template labels 3. Not be updated while a deployment is in progress ## Environment - langgraph-operator: `0.1.21` - langgraph-dataplane chart: `0.2.13` - Kubernetes: EKS 1.33
yindo closed this issue 2026-02-16 01:16:41 -05:00
yindo commented 2026-02-16 01:16:42 -05:00
Author
Owner
Copy Link
Copy Source

@langchain-infra commented on GitHub (Feb 5, 2026):

Hi @KranthiKRK this is intended. We set this on every revision created by the control plane to force a new rollout (even if image version was static for example). Could you explain more about the issue you were seeing? Did you have a bunch of queued revisions?

@langchain-infra commented on GitHub (Feb 5, 2026): Hi @KranthiKRK this is intended. We set this on every revision created by the control plane to force a new rollout (even if image version was static for example). Could you explain more about the issue you were seeing? Did you have a bunch of queued revisions?
yindo commented 2026-02-16 01:16:42 -05:00
Author
Owner
Copy Link
Copy Source

@KranthiKRK commented on GitHub (Feb 5, 2026):

Thanks for the quick response @langchain-infra.

We understand the intent to force rollouts via ls_timestamp. The issue is that during a single deployment, the control plane creates multiple revisions while the rollout is still in progress, causing cascading restarts.

Here is what we observed in our APJC (Sydney) production cluster during a deployment of the same image:

Revision Created (UTC) ls_timestamp
8 19:57:03 1770235023
9 19:58:49 1770235129
10 20:07:34 1770235654
11 20:11:18 1770235878
12 20:18:07 1770236287
13 20:27:39 1770236859

6 revisions in 30 minutes for the same image. Each revision reset the rolling deployment progress, turning a ~5 minute deployment into 37 minutes.

We confirmed the same pattern in our test cluster. For example, on Feb 3:

Revision Created (UTC) ls_timestamp
145 19:29:13 1770146953
146 19:33:26 1770147206
147 19:39:56 1770147596
148 19:40:37 1770147637

4 revisions in 11 minutes - again for the same image.

Meanwhile, our EU cluster deployed cleanly with just 1 revision change in 9 minutes because no cascading occurred.

The listener's reconcile_projects cron runs frequently and creates a new deploy_image job while the previous rollout has not completed. Each job updates ls_timestamp, which triggers a new rollout via the operator, resetting progress.

The core issue: new revisions should not be created while a rollout is still in progress, or at minimum, ls_timestamp updates should be deferred until the current rollout completes.

@KranthiKRK commented on GitHub (Feb 5, 2026): Thanks for the quick response @langchain-infra. We understand the intent to force rollouts via `ls_timestamp`. The issue is that during a **single deployment**, the control plane creates **multiple revisions** while the rollout is still in progress, causing cascading restarts. Here is what we observed in our APJC (Sydney) production cluster during a deployment of the same image: | Revision | Created (UTC) | ls_timestamp | |----------|---------------|--------------| | 8 | 19:57:03 | 1770235023 | | 9 | 19:58:49 | 1770235129 | | 10 | 20:07:34 | 1770235654 | | 11 | 20:11:18 | 1770235878 | | 12 | 20:18:07 | 1770236287 | | 13 | 20:27:39 | 1770236859 | **6 revisions in 30 minutes for the same image.** Each revision reset the rolling deployment progress, turning a ~5 minute deployment into 37 minutes. We confirmed the same pattern in our test cluster. For example, on Feb 3: | Revision | Created (UTC) | ls_timestamp | |----------|---------------|--------------| | 145 | 19:29:13 | 1770146953 | | 146 | 19:33:26 | 1770147206 | | 147 | 19:39:56 | 1770147596 | | 148 | 19:40:37 | 1770147637 | **4 revisions in 11 minutes** - again for the same image. Meanwhile, our EU cluster deployed cleanly with just 1 revision change in 9 minutes because no cascading occurred. The listener's `reconcile_projects` cron runs frequently and creates a new `deploy_image` job while the previous rollout has not completed. Each job updates `ls_timestamp`, which triggers a new rollout via the operator, resetting progress. The core issue: **new revisions should not be created while a rollout is still in progress**, or at minimum, `ls_timestamp` updates should be deferred until the current rollout completes.
yindo commented 2026-02-16 01:16:42 -05:00
Author
Owner
Copy Link
Copy Source

@langchain-infra commented on GitHub (Feb 5, 2026):

Hi @KranthiKRK, is it possible you have multiple listeners pointed at the same redis? This seems lke we are re-running the jobs that trigger a revision (not that our operator is incorrectly creating new revisions). Or do you potentially have multiple operators? Typically only one revision can happen at a time/additional revisions shold be queued.

@langchain-infra commented on GitHub (Feb 5, 2026): Hi @KranthiKRK, is it possible you have multiple listeners pointed at the same redis? This seems lke we are re-running the jobs that trigger a revision (not that our operator is incorrectly creating new revisions). Or do you potentially have multiple operators? Typically only one revision can happen at a time/additional revisions shold be queued.
yindo commented 2026-02-16 01:16:42 -05:00
Author
Owner
Copy Link
Copy Source

@KranthiKRK commented on GitHub (Feb 6, 2026):

Hi @langchain-infra, we have verified both clusters:

APJC Prod: 1 listener (replicas: 1), 1 operator (replicas: 1)
Test cluster: 1 listener (replicas: 1), 1 operator (replicas: 1)

No shared Redis across clusters - each cluster has its own Redis endpoint.

From the APJC incident logs we captured during the event, the deploy_image job had a single job_id and single revision_id throughout:

job_id: "saq:job:host:deploy_image:2466d2fb-3695-4f7d-99f2-e2c011b5f62e"
job_attempts: 6
revision_id: "2466d2fb-3695-4f7d-99f2-e2c011b5f62e"

And reconcile_projects was reporting the same job as active:

"Deploy Job ID saq:job:host:deploy_image:2466d2fb-... (active) in progress for revision ID 2466d2fb-..."

On the Kubernetes side, 6 ReplicaSets were created in 30 minutes with the same image but different ls_timestamp values. So something was updating the ls_timestamp label on the LGP CR multiple times during this single deployment, causing the operator to trigger new rollouts.

Could the deploy job retries (attempts 1-6) be re-applying the LGP CR spec with a new ls_timestamp each time? That would explain the 6 ReplicaSets matching the 6 attempts.

@KranthiKRK commented on GitHub (Feb 6, 2026): Hi @langchain-infra, we have verified both clusters: **APJC Prod:** 1 listener (replicas: 1), 1 operator (replicas: 1) **Test cluster:** 1 listener (replicas: 1), 1 operator (replicas: 1) No shared Redis across clusters - each cluster has its own Redis endpoint. From the APJC incident logs we captured during the event, the `deploy_image` job had a single `job_id` and single `revision_id` throughout: ``` job_id: "saq:job:host:deploy_image:2466d2fb-3695-4f7d-99f2-e2c011b5f62e" job_attempts: 6 revision_id: "2466d2fb-3695-4f7d-99f2-e2c011b5f62e" ``` And `reconcile_projects` was reporting the same job as active: ``` "Deploy Job ID saq:job:host:deploy_image:2466d2fb-... (active) in progress for revision ID 2466d2fb-..." ``` On the Kubernetes side, 6 ReplicaSets were created in 30 minutes with the same image but different `ls_timestamp` values. So something was updating the `ls_timestamp` label on the LGP CR multiple times during this single deployment, causing the operator to trigger new rollouts. Could the deploy job retries (attempts 1-6) be re-applying the LGP CR spec with a new `ls_timestamp` each time? That would explain the 6 ReplicaSets matching the 6 attempts.
yindo commented 2026-02-16 01:16:42 -05:00
Author
Owner
Copy Link
Copy Source

@langchain-infra commented on GitHub (Feb 6, 2026):

Yea this is due to retries. But not sure why the job itself was getting retried (as we should be handling errors and updating revision status instead). Were there any error logs?

@langchain-infra commented on GitHub (Feb 6, 2026): Yea this is due to retries. But not sure why the job itself was getting retried (as we should be handling errors and updating revision status instead). Were there any error logs?
yindo commented 2026-02-16 01:16:42 -05:00
Author
Owner
Copy Link
Copy Source

@KranthiKRK commented on GitHub (Feb 6, 2026):

Unfortunately the listener pod has been restarted since the incident, and previous container logs are not available. The current pod only has logs from 2026-02-06T05:13:40Z onward (well after the incident on Feb 4).

No error-level logs exist in the current listener pod - only info and one benign warning about StructuredPrompt being in beta.

From the logs we captured during the incident, the deploy_image job showed job_attempts: 6 but there were no error messages in the output we captured - only "Waiting for deployment ... to be ready" messages. This suggests the job may have been timing out rather than hitting an explicit error, then retrying with a new ls_timestamp each time.

Is there a configurable timeout for the deploy_image job that could explain the retries? In our case, with 30 queue replicas and a node failure mid-rollout, each rollout cycle was taking longer than expected.

@KranthiKRK commented on GitHub (Feb 6, 2026): Unfortunately the listener pod has been restarted since the incident, and previous container logs are not available. The current pod only has logs from `2026-02-06T05:13:40Z` onward (well after the incident on Feb 4). No error-level logs exist in the current listener pod - only `info` and one benign `warning` about `StructuredPrompt` being in beta. From the logs we captured during the incident, the deploy_image job showed `job_attempts: 6` but there were no error messages in the output we captured - only "Waiting for deployment ... to be ready" messages. This suggests the job may have been timing out rather than hitting an explicit error, then retrying with a new `ls_timestamp` each time. Is there a configurable timeout for the `deploy_image` job that could explain the retries? In our case, with 30 queue replicas and a node failure mid-rollout, each rollout cycle was taking longer than expected.
yindo commented 2026-02-16 01:16:43 -05:00
Author
Owner
Copy Link
Copy Source

@langchain-infra commented on GitHub (Feb 6, 2026):

Hi @KranthiKRK instead i think we will just make the deploy_image job more idempotent, we should not be trying to update ls_timestamp if the revision id is also not changing.

@langchain-infra commented on GitHub (Feb 6, 2026): Hi @KranthiKRK instead i think we will just make the deploy_image job more idempotent, we should not be trying to update ls_timestamp if the revision id is also not changing.
yindo commented 2026-02-16 01:16:43 -05:00
Author
Owner
Copy Link
Copy Source

@KranthiKRK commented on GitHub (Feb 6, 2026):

That sounds like the right fix - skipping the ls_timestamp update when the revision_id hasn't changed should prevent the cascading rollouts on retries. Thanks for looking into this @langchain-infra!

@KranthiKRK commented on GitHub (Feb 6, 2026): That sounds like the right fix - skipping the `ls_timestamp` update when the `revision_id` hasn't changed should prevent the cascading rollouts on retries. Thanks for looking into this @langchain-infra\!
yindo commented 2026-02-16 01:16:43 -05:00
Author
Owner
Copy Link
Copy Source

@langchain-infra commented on GitHub (Feb 9, 2026):

@KranthiKRK this should be solved in the latest release of our dataplane.

@langchain-infra commented on GitHub (Feb 9, 2026): @KranthiKRK this should be solved in the latest release of our dataplane.
yindo commented 2026-02-16 01:16:44 -05:00
Author
Owner
Copy Link
Copy Source

@KranthiKRK commented on GitHub (Feb 9, 2026):

Thanks @langchain-infra. We are on the hybrid LangSmith setup, so we use the langgraph-dataplane chart (not the langsmith chart).

The latest langgraph-dataplane release is 0.2.15, which still ships with listener 0.12.4 and operator 0.1.34. The fixed image tags (0.13.9 / 0.1.36) are included in the langsmith-0.13.9 chart but have not been released as a new langgraph-dataplane chart version yet.

Could you cut a new langgraph-dataplane release with the updated images so we can pick up the fix?

@KranthiKRK commented on GitHub (Feb 9, 2026): Thanks @langchain-infra. We are on the **hybrid** LangSmith setup, so we use the `langgraph-dataplane` chart (not the `langsmith` chart). The latest `langgraph-dataplane` release is [0.2.15](https://github.com/langchain-ai/helm/releases/tag/langgraph-dataplane-0.2.15), which still ships with listener `0.12.4` and operator `0.1.34`. The fixed image tags (`0.13.9` / `0.1.36`) are included in the `langsmith-0.13.9` chart but have not been released as a new `langgraph-dataplane` chart version yet. Could you cut a new `langgraph-dataplane` release with the updated images so we can pick up the fix?
yindo changed title from ls_timestamp label on pod templates causes cascading rollouts during slow deployments to [GH-ISSUE #568] ls_timestamp label on pod templates causes cascading rollouts during slow deployments 2026-06-05 19:14:52 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/helm#24
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?