langchain-ai/langgraph
1
0
Fork 0
mirror of https://github.com/langchain-ai/langgraph.git synced 2026-07-15 12:35:43 -04:00
Code Issues 314 Packages Projects Releases 17 Wiki Activity

[PR #275] [MERGED] Issue #274: Fix LLMCompiler Bugs #1349

New Issue
Closed
opened 2026-02-20 17:44:31 -05:00 by yindo · 0 comments
yindo commented 2026-02-20 17:44:31 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/langgraph/pull/275
Author: @lantern-eight
Created: 4/3/2024
Status: ✅ Merged
Merged: 4/5/2024
Merged by: @hinthornw

Base: main ← Head: fix-llmcompiler-bugs

📝 Commits (4)

  • 2394b95 Update LLMCompiler.ipynb
  • bb35c05 Fix looping function calls because no arg context
  • 631b39c Fix tool index count and range
  • 0cb3da3 Fix unhandled empty iterator exception

📊 Changes

1 file changed (+28 additions, -14 deletions)

View changed files

📝 examples/llm-compiler/LLMCompiler.ipynb (+28 -14)

📄 Description

Fixes 4/5 bugs, (Specifically 1,2,4,5) described in Issue #274.

Issue text copied here for reference:
I noticed some bugs in the tutorial for LLMCompiler:

  1. Main problem: the code doesn't actually handle when there's more than one dependency listed in an argument, which is the whole point of LLMCompiler. The example of multi-step math used in the code doesn't work because the third step has more than one dependency in its args and the code only handles one:
User query: "What's ((3*(4+5)/0.5)+3245) + 8? What's 32/4.23? What's the sum of those two values?"
1. math(problem="((3*(4+5)/0.5)+3245) + 8")
2. math(problem="32/4.23")
3. math(problem="${1}+${2}")
4. join()<END_OFPLAN>

Step 3 fails, and does so silently too. It correctly parses strings that are "${1}", but not multi-argument strings like "${1}+${2}"
Luckily, this is a relatively easy fix:
Previous code:

def _resolve_arg(arg: Union[str, Any], observations: Dict[int, Any]):
    if isinstance(arg, str) and arg.startswith("$"):
        try:
            stripped = arg[1:].replace(".output", "").strip("{}")
            idx = int(stripped)
        except Exception:
            return str(arg)
        return str(observations[idx])
    elif isinstance(arg, list):
        return [_resolve_arg(a, observations) for a in arg]
    else:
        return str(arg)

Possible fix:

def _resolve_arg(arg: Union[str, Any], observations: Dict[int, Any]):
  '''Resolve arguments. This is mostly to get and replace ${idx} arguments that are
  dependencies on other tasks. Otherwise we just return the arguments again.
  '''

  # $1 or ${1} -> 1
  ID_PATTERN = r"\$\{?(\d+)\}?"

  def replace_match(match):
    # If the string is ${123}, match.group(0) is ${123}, and match.group(1) is 123.

    # Return the match group, in this case the index, from the string. This is the index
    # number we get back.
    idx = int(match.group(1))
    # Retrieves the value associated with the index idx from the
    # observations dictionary. If the index is not found in the dictionary, it returns
    # the placeholder itself (match.group(0)).
    return str(observations.get(idx, match.group(0)))

  # For dependencies on other tasks
  if isinstance(arg, str):
    # re.sub() searches the whole string and replaces all occurrences where the pattern
    # is matched. For each match, it calls the function and uses the match as input,
    # then replaces the match with the returned value from the function.
    return re.sub(ID_PATTERN, replace_match, arg)

  elif isinstance(arg, list):
    return [_resolve_arg(a, observations) for a in arg]

  else:
    return str(arg)
  1. Task Fetching Unit, when it inserts the tool_messages, the function args don't get logged with function calls, this causes the replanner to loop and use the same function parameters and never corrects itself.
  tool_messages = [
    FunctionMessage(name=name, content=str(obs), additional_kwargs={'idx': k})
    for k, (name, obs) in new_observations.items()
  ]

Possible fix:

  tasks = {}
  args_for_tasks = {}
  ...
        tasks[task["idx"]] = (
        task["tool"] if isinstance(task["tool"], str) else task["tool"].name
      )
      args_for_tasks[task["idx"]] = (task["args"])
  ...
    new_observations = {
    k: (tasks[k], args_for_tasks[k], observations[k])
    for k in sorted(observations.keys() - originals)
  }
  ...
  tool_messages = [
    FunctionMessage(name=name, content=str(obs), additional_kwargs={'idx': k, 'args':task_args})
    for k, (name, task_args, obs) in new_observations.items()
  ]
  1. Replan prompt is never actually inserted into the prompt sent to the LLM.
  2. In the planner prompt formatting, the 'num_tools' variable needs to have a +1 added to it because we're appending the join() function, without +1 it says there's one less available tools than there actually is because it only counts the passed in functions and not the join() function. And add +1 for listing otherwise it starts at a 0 offset.
    Possible fix:
  tool_descriptions = "\n".join(
    f"{i+1}. {tool.description}\n" # +1 to offset the 0 starting index, we want it count normally from 1.
    for i, tool in enumerate(tools)
  )
  ...
    planner_prompt = base_prompt.partial(
    replan="",
    num_tools=len(tools)+1,# add one because we're adding the join() tool at the end.
    tool_descriptions=tool_descriptions,
  )
  ...
    replanner_prompt = base_prompt.partial(
    replan=replan_prompt,
    num_tools=len(tools)+1,# add one because we're adding the join() tool at the end.
    tool_descriptions=tool_descriptions,
  )
  1. The task fetching unit, when it calls the tools it doesn't handle when there's not tasks or reaches it's end:
    Current code: tasks = itertools.chain([next(tasks)], tasks)
    A possible fix, wrap in try/except and set tasks to empty:
    try:
        tasks = itertools.chain([next(tasks)], tasks)
    except StopIteration:
        # Handle the case where 'tasks' is empty or has reached its end.
        tasks = iter([])

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/langgraph/pull/275 **Author:** [@lantern-eight](https://github.com/lantern-eight) **Created:** 4/3/2024 **Status:** ✅ Merged **Merged:** 4/5/2024 **Merged by:** [@hinthornw](https://github.com/hinthornw) **Base:** `main` ← **Head:** `fix-llmcompiler-bugs` --- ### 📝 Commits (4) - [`2394b95`](https://github.com/langchain-ai/langgraph/commit/2394b959de57361e184a2b39801280e700f69d3c) Update LLMCompiler.ipynb - [`bb35c05`](https://github.com/langchain-ai/langgraph/commit/bb35c05835772a28c0ec41c6a4d52199ab6812f3) Fix looping function calls because no arg context - [`631b39c`](https://github.com/langchain-ai/langgraph/commit/631b39c6987714baf156dee094e006cb8db19aa0) Fix tool index count and range - [`0cb3da3`](https://github.com/langchain-ai/langgraph/commit/0cb3da3afe3052bd0324b44ec68ce83d01c9aabe) Fix unhandled empty iterator exception ### 📊 Changes **1 file changed** (+28 additions, -14 deletions) <details> <summary>View changed files</summary> 📝 `examples/llm-compiler/LLMCompiler.ipynb` (+28 -14) </details> ### 📄 Description Fixes 4/5 bugs, (Specifically 1,2,4,5) described in [Issue #274](https://github.com/langchain-ai/langgraph/issues/274). Issue text copied here for reference: I noticed some bugs in the tutorial for LLMCompiler: 1) Main problem: the code doesn't actually handle when there's more than one dependency listed in an argument, which is the whole point of LLMCompiler. The example of multi-step math used in the code doesn't work because the third step has more than one dependency in its args and the code only handles one: ``` User query: "What's ((3*(4+5)/0.5)+3245) + 8? What's 32/4.23? What's the sum of those two values?" 1. math(problem="((3*(4+5)/0.5)+3245) + 8") 2. math(problem="32/4.23") 3. math(problem="${1}+${2}") 4. join()<END_OFPLAN> ``` Step 3 fails, and does so silently too. It correctly parses strings that are "${1}", but not multi-argument strings like "${1}+${2}" Luckily, this is a relatively easy fix: Previous code: ```python def _resolve_arg(arg: Union[str, Any], observations: Dict[int, Any]): if isinstance(arg, str) and arg.startswith("$"): try: stripped = arg[1:].replace(".output", "").strip("{}") idx = int(stripped) except Exception: return str(arg) return str(observations[idx]) elif isinstance(arg, list): return [_resolve_arg(a, observations) for a in arg] else: return str(arg) ``` Possible fix: ```python def _resolve_arg(arg: Union[str, Any], observations: Dict[int, Any]): '''Resolve arguments. This is mostly to get and replace ${idx} arguments that are dependencies on other tasks. Otherwise we just return the arguments again. ''' # $1 or ${1} -> 1 ID_PATTERN = r"\$\{?(\d+)\}?" def replace_match(match): # If the string is ${123}, match.group(0) is ${123}, and match.group(1) is 123. # Return the match group, in this case the index, from the string. This is the index # number we get back. idx = int(match.group(1)) # Retrieves the value associated with the index idx from the # observations dictionary. If the index is not found in the dictionary, it returns # the placeholder itself (match.group(0)). return str(observations.get(idx, match.group(0))) # For dependencies on other tasks if isinstance(arg, str): # re.sub() searches the whole string and replaces all occurrences where the pattern # is matched. For each match, it calls the function and uses the match as input, # then replaces the match with the returned value from the function. return re.sub(ID_PATTERN, replace_match, arg) elif isinstance(arg, list): return [_resolve_arg(a, observations) for a in arg] else: return str(arg) ``` 2) Task Fetching Unit, when it inserts the tool_messages, the function args don't get logged with function calls, this causes the replanner to loop and use the same function parameters and never corrects itself. ```python tool_messages = [ FunctionMessage(name=name, content=str(obs), additional_kwargs={'idx': k}) for k, (name, obs) in new_observations.items() ] ``` Possible fix: ```python tasks = {} args_for_tasks = {} ... tasks[task["idx"]] = ( task["tool"] if isinstance(task["tool"], str) else task["tool"].name ) args_for_tasks[task["idx"]] = (task["args"]) ... new_observations = { k: (tasks[k], args_for_tasks[k], observations[k]) for k in sorted(observations.keys() - originals) } ... tool_messages = [ FunctionMessage(name=name, content=str(obs), additional_kwargs={'idx': k, 'args':task_args}) for k, (name, task_args, obs) in new_observations.items() ] ``` 3) Replan prompt is never actually inserted into the prompt sent to the LLM. 4) In the planner prompt formatting, the 'num_tools' variable needs to have a +1 added to it because we're appending the join() function, without +1 it says there's one less available tools than there actually is because it only counts the passed in functions and not the join() function. And add +1 for listing otherwise it starts at a 0 offset. Possible fix: ```python tool_descriptions = "\n".join( f"{i+1}. {tool.description}\n" # +1 to offset the 0 starting index, we want it count normally from 1. for i, tool in enumerate(tools) ) ... planner_prompt = base_prompt.partial( replan="", num_tools=len(tools)+1,# add one because we're adding the join() tool at the end. tool_descriptions=tool_descriptions, ) ... replanner_prompt = base_prompt.partial( replan=replan_prompt, num_tools=len(tools)+1,# add one because we're adding the join() tool at the end. tool_descriptions=tool_descriptions, ) ``` 5) The task fetching unit, when it calls the tools it doesn't handle when there's not tasks or reaches it's end: Current code: `tasks = itertools.chain([next(tasks)], tasks)` A possible fix, wrap in try/except and set tasks to empty: ```python try: tasks = itertools.chain([next(tasks)], tasks) except StopIteration: # Handle the case where 'tasks' is empty or has reached its end. tasks = iter([]) ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-20 17:44:31 -05:00
yindo closed this issue 2026-02-20 17:44:31 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agents
awaiting author revision
bug
change
chat models
checkpointer
ci
code quality
code quality
dependencies
documentation
drawing
duplicate
enhancement
good first issue
help wanted
invalid
maintainer
meta
missing information
not maintained by langchain
open-swe-auto
open-swe-max
open-swe-max-auto
pending
performance
pull-request
Mirrored from GitHub Pull Request
 question
requires design
sdk
server
stale
studio
tests
type checking
usage question
v0.5
v1.1
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langgraph#1349
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?