langchain-ai/docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/docs.git synced 2026-07-22 12:25:55 -04:00
Code Issues 490 Packages Projects Releases Wiki Activity

[GH-ISSUE #4070] metadata.run_id → is_resuming idempotency: undocumented but critical for self-hosted checkpointer users #2776

New Issue
Open
opened 2026-06-05 17:26:44 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 17:26:44 -04:00
Owner
Copy Link
Copy Source

Originally created by @ganeshan007 on GitHub (May 19, 2026).
Original GitHub issue: https://github.com/langchain-ai/docs/issues/4070

Type of issue

question

Language

Python

Description

PR #3668 added a run_id match check to PregelLoop._first() that prevents input re-application when the same run_id re-enters a graph with an existing checkpoint. The source comment says:

Same run_id: re-entry into an ongoing run (e.g. stream reconnect)

This mechanism is exactly what self-hosted users need for retry idempotency (e.g., Temporal/Celery retries, k8s pod restarts), but it's completely undocumented. We discovered it by reading _loop.py source after debugging a production incident.

The problem we hit
We run LangGraph agents via Temporal workflows. When a KEDA autoscaler killed a WFE worker pod mid-InitializeAgent, Temporal retried the activity. The retry re-invoked compiled_graph.astream(input, config) on a thread that already had a checkpoint from the first (successful) execution.

Without metadata["run_id"]: LangGraph treated the retry as new input → appended a duplicate HumanMessage → the checkpoint now had orphaned tool_use blocks without tool_result → Anthropic returned 400 → thread permanently stuck.

With metadata["run_id"] set to our stable run identifier: is_resuming=True → input was not re-applied → checkpoint unchanged → Temporal got the same response back. One-line fix.

What we found
CheckpointMetadata defines run_id: str as a typed field (reference)
get_checkpoint_metadata() copies config["metadata"] keys into checkpoint metadata — so metadata["run_id"] flows through
PregelLoop._first() checks config.metadata.run_id == checkpoint_metadata.run_id to set is_resuming
But RunnableConfig["metadata"] is dict[str, Any] with no schema — nothing tells users that run_id is a semantically meaningful key
No documentation page mentions setting metadata["run_id"] for retry safety
The stream() / invoke() docstrings don't mention this behavior
CONFIG_KEY_RESUMING is marked deprecated/private in v1.0
We verified against our production DB: 10.4M checkpoints, zero have run_id set. The is_resuming check has been a dead branch for our entire deployment.

Questions
Is metadata["run_id"] a supported API for self-hosted users? Or is it internal to LangGraph Platform only?

Should this be documented? If users are expected to set metadata["run_id"] for retry idempotency, it should be in the Persistence or Fault Tolerance docs with an example like:

config = {
    "configurable": {"thread_id": thread_id},
    "metadata": {"run_id": my_stable_run_id},  # enables retry idempotency
}

Is this the right mechanism for retry idempotency? The PR title says "retrying a previously attempted run" and the source comment says "stream reconnect." Are these the same use case, or does stream reconnect have different semantics?

What happens when is_resuming=True but the graph is interrupted? Does it re-emit the interrupt? Skip to the pending tasks? Our tests show it preserves the interrupt state, but we want to confirm this is intended.

Will this remain stable across versions? The check survived 14+ months since PR #3668 (March 2025) and the comments were expanded on main. But since it's not in public docs, there's no semver contract.

Related
PR #3668 — "When retrying a previously attempted run, resume from previous checkpoint" (the commit that added this)
Issue #7417 — "Long tool calls (~180s+) silently re-executed from checkpoint on LangGraph Cloud" (same class of problem on LangGraph Cloud itself)
Issue #4230 — "run_id and other config parameters are not persisted" (related confusion about run_id)
Issue #6728 — "Duplicate subgraph task execution with identical run_id"

Originally created by @ganeshan007 on GitHub (May 19, 2026). Original GitHub issue: https://github.com/langchain-ai/docs/issues/4070 ### Type of issue question ### Language Python ### Description PR [#3668](https://github.com/langchain-ai/langgraph/pull/3668) added a run_id match check to PregelLoop._first() that prevents input re-application when the same run_id re-enters a graph with an existing checkpoint. The source comment says: # Same run_id: re-entry into an ongoing run (e.g. stream reconnect) This mechanism is exactly what self-hosted users need for retry idempotency (e.g., Temporal/Celery retries, k8s pod restarts), but it's completely undocumented. We discovered it by reading _loop.py source after debugging a production incident. The problem we hit We run LangGraph agents via Temporal workflows. When a KEDA autoscaler killed a WFE worker pod mid-InitializeAgent, Temporal retried the activity. The retry re-invoked compiled_graph.astream(input, config) on a thread that already had a checkpoint from the first (successful) execution. Without metadata["run_id"]: LangGraph treated the retry as new input → appended a duplicate HumanMessage → the checkpoint now had orphaned tool_use blocks without tool_result → Anthropic returned 400 → thread permanently stuck. With metadata["run_id"] set to our stable run identifier: is_resuming=True → input was not re-applied → checkpoint unchanged → Temporal got the same response back. One-line fix. What we found CheckpointMetadata defines run_id: str as a typed field ([reference](https://reference.langchain.com/python/langgraph.checkpoint/base/CheckpointMetadata)) get_checkpoint_metadata() copies config["metadata"] keys into checkpoint metadata — so metadata["run_id"] flows through PregelLoop._first() checks config.metadata.run_id == checkpoint_metadata.run_id to set is_resuming But RunnableConfig["metadata"] is dict[str, Any] with no schema — nothing tells users that run_id is a semantically meaningful key No documentation page mentions setting metadata["run_id"] for retry safety The stream() / invoke() docstrings don't mention this behavior CONFIG_KEY_RESUMING is marked deprecated/private in v1.0 We verified against our production DB: 10.4M checkpoints, zero have run_id set. The is_resuming check has been a dead branch for our entire deployment. Questions Is metadata["run_id"] a supported API for self-hosted users? Or is it internal to LangGraph Platform only? Should this be documented? If users are expected to set metadata["run_id"] for retry idempotency, it should be in the Persistence or Fault Tolerance docs with an example like: ``` config = { "configurable": {"thread_id": thread_id}, "metadata": {"run_id": my_stable_run_id}, # enables retry idempotency } ``` Is this the right mechanism for retry idempotency? The PR title says "retrying a previously attempted run" and the source comment says "stream reconnect." Are these the same use case, or does stream reconnect have different semantics? What happens when is_resuming=True but the graph is interrupted? Does it re-emit the interrupt? Skip to the pending tasks? Our tests show it preserves the interrupt state, but we want to confirm this is intended. Will this remain stable across versions? The check survived 14+ months since PR #3668 (March 2025) and the comments were expanded on main. But since it's not in public docs, there's no semver contract. Related PR [#3668](https://github.com/langchain-ai/langgraph/pull/3668) — "When retrying a previously attempted run, resume from previous checkpoint" (the commit that added this) Issue [#7417](https://github.com/langchain-ai/langgraph/issues/7417) — "Long tool calls (~180s+) silently re-executed from checkpoint on LangGraph Cloud" (same class of problem on LangGraph Cloud itself) Issue [#4230](https://github.com/langchain-ai/langgraph/issues/4230) — "run_id and other config parameters are not persisted" (related confusion about run_id) Issue [#6728](https://github.com/langchain-ai/langgraph/issues/6728) — "Duplicate subgraph task execution with identical run_id"
yindo added the langgraphexternal labels 2026-06-05 17:26:44 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
deepagents
docs-infra
external
external
help wanted
integration
internal
internal
javascript
js/ts
langchain
langgraph
langsmith
needs-title
pull-request
Mirrored from GitHub Pull Request
 python
reference
No Label external langgraph
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/docs#2776
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?