Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3675] [BUG]: Agent web search consumes massive input tokens without producing results #2372

New Issue
Closed
opened 2026-02-22 18:29:23 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:29:23 -05:00
Owner
Copy Link
Copy Source

Originally created by @LinusAurel on GitHub (Apr 18, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3675

How are you running AnythingLLM?

Docker (local)

What happened?

When using an agent (@agent) command requiring web access (e.g., finding recent information, summarizing articles), the agent enters a state of extreme inefficiency, consuming massive amounts of resources without completing the task:

  1. Runaway Input Token Consumption per Step: The primary issue is that the agent makes numerous, consecutive calls to the underlying LLM (e.g., Claude 3.7 Sonnet), where nearly every single internal step/API call consumes an extremely high number of input tokens (consistently observed between 6,000 - 8,000+ tokens per call). This is not for processing large documents, but seemingly for intermediate reasoning, planning, or review steps within the agent's workflow.
  2. Rapid Repetition of Expensive Calls: API logs clearly show dozens of these high-input-token calls occurring in rapid succession, often within the same minute (see evidence section). This indicates the agent might be stuck in a loop or breaking down its process into excessively granular, high-cost steps.
  3. Failure to Produce Output: Despite this enormous token expenditure on input, the agent makes little to no progress towards the final goal. It fails to provide the requested summary or answer. The output tokens per call are minimal (e.g., 100-300), suggesting these are internal thoughts or minor actions, not the final result.
  4. Inevitable Rate Limiting: As a direct and unavoidable consequence of this runaway input token consumption, the LLM provider's API rate limits are quickly exceeded, halting the process with a 429 error. This error is a symptom of the catastrophic inefficiency, not the root cause.

Evidence from API Logs:

Image

API call logs reveal a pattern of intense activity. Within a single minute (e.g., timestamp 18:08), numerous (20+) distinct API requests are made to the configured LLM (Claude 3.7 Sonnet). Critically, almost every one of these requests consumes between ~5,600 and ~7,600 input tokens, while generating only small amounts of output tokens (typically 90-300). This pattern persists over time when the agent is active on a web search task.

Expected Behavior:

I expected the agent to:

  1. Execute its internal reasoning, searching, and processing steps with reasonable and efficient token usage. Individual internal steps or calls should not consistently cost thousands of input tokens each.
  2. Structure its interaction with the LLM more effectively, potentially using fewer, more targeted calls.
  3. Make tangible progress towards summarizing information or answering the query.
  4. Complete the task successfully and provide a useful response to the user, operating well within standard API rate limits due to efficient token use.

Are there known steps to reproduce?

  1. Set up AnythingLLM using Docker on a resource-constrained environment like a Raspberry Pi 5 (the issue might be more pronounced here, but could potentially occur elsewhere).
  2. Configure an Agent with web search capabilities, using an LLM like Claude 3.7 Sonnet (or potentially others).
  3. Have access to monitor the LLM provider's API usage dashboard or detailed logs to observe the backend token consumption per call.
  4. Invoke the agent in chat with a query requiring web searching and synthesis. Use a query known to trigger the behavior, like:
    @agent Summarize the latest scientific opinions on Topic X.
  5. Observe the agent's activity from two perspectives:
    • A) Agent Chat Logs (in AnythingLLM Interface): Watch the agent's progress messages in the chat. You may see steps like searching, reviewing results (potentially with estimated token counts like ~993 tokens just for review), attempting to scrape pages, and indicating lengthy content requires summarization, before it likely stalls. Example snippet: 
      Agent @agent invoked. Swapping over to agent chat. Type /exit to exit agent execution loop early.
@agent: Searching on Google for "..."
@agent: I found 10 results - reviewing the results now. (~993 tokens)
@agent: Searching on Google for "..."
@agent: I found 10 results - reviewing the results now. (~851 tokens)
@agent: Scraping the content of https://...
@agent: This page's content is way too long. I will summarize it right now.
# <-- Agent often stalls or loops after this point -->
    • B) Backend API Logs (LLM Provider Dashboard): Simultaneously, observe the stream of API calls being made to the LLM provider. Note the extremely high input token count (consistently 6k-8k+ observed with Claude 3.7 Sonnet) for nearly every single call, occurring rapidly.
  6. Observe that the agent likely fails to produce a final answer in the chat and eventually hits a 429 rate_limit_error (reported both in the chat and visible in API logs) due to the excessive input token consumption shown in the backend logs.
Originally created by @LinusAurel on GitHub (Apr 18, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3675 ### How are you running AnythingLLM? Docker (local) ### What happened? When using an agent (`@agent`) command requiring web access (e.g., finding recent information, summarizing articles), the agent enters a state of extreme inefficiency, consuming massive amounts of resources without completing the task: 1. **Runaway Input Token Consumption per Step:** The primary issue is that the agent makes numerous, consecutive calls to the underlying LLM (e.g., Claude 3.7 Sonnet), where **nearly every single internal step/API call consumes an extremely high number of *input* tokens (consistently observed between 6,000 - 8,000+ tokens per call)**. This is *not* for processing large documents, but seemingly for intermediate reasoning, planning, or review steps within the agent's workflow. 2. **Rapid Repetition of Expensive Calls:** API logs clearly show dozens of these high-input-token calls occurring in rapid succession, often within the same minute (see evidence section). This indicates the agent might be stuck in a loop or breaking down its process into excessively granular, high-cost steps. 3. **Failure to Produce Output:** Despite this enormous token expenditure on input, the agent makes little to no progress towards the final goal. It fails to provide the requested summary or answer. The output tokens per call are minimal (e.g., 100-300), suggesting these are internal thoughts or minor actions, not the final result. 4. **Inevitable Rate Limiting:** As a direct and unavoidable consequence of this runaway input token consumption, the LLM provider's API rate limits are quickly exceeded, halting the process with a `429 error`. This error is a *symptom* of the catastrophic inefficiency, not the root cause. **Evidence from API Logs:** <img width="873" alt="Image" src="https://github.com/user-attachments/assets/7b6fdd12-c9d5-4a87-912f-7a5aafcae5eb" /> API call logs reveal a pattern of intense activity. Within a single minute (e.g., timestamp 18:08), numerous (20+) distinct API requests are made to the configured LLM (Claude 3.7 Sonnet). Critically, almost **every one of these requests consumes between ~5,600 and ~7,600 input tokens**, while generating only small amounts of output tokens (typically 90-300). This pattern persists over time when the agent is active on a web search task. **Expected Behavior:** I expected the agent to: 1. Execute its internal reasoning, searching, and processing steps with **reasonable and efficient** token usage. Individual internal steps or calls should not consistently cost thousands of input tokens each. 2. Structure its interaction with the LLM more effectively, potentially using fewer, more targeted calls. 3. Make tangible progress towards summarizing information or answering the query. 4. Complete the task successfully and provide a useful response to the user, operating well within standard API rate limits due to efficient token use. ### Are there known steps to reproduce? 1. Set up AnythingLLM using Docker on a resource-constrained environment like a Raspberry Pi 5 (the issue might be more pronounced here, but could potentially occur elsewhere). 2. Configure an Agent with web search capabilities, using an LLM like Claude 3.7 Sonnet (or potentially others). 3. Have access to monitor the LLM provider's API usage dashboard or detailed logs to observe the backend token consumption per call. 4. Invoke the agent in chat with a query requiring web searching and synthesis. Use a query known to trigger the behavior, like: `@agent Summarize the latest scientific opinions on Topic X.` 5. **Observe the agent's activity from two perspectives:** * **A) Agent Chat Logs (in AnythingLLM Interface):** Watch the agent's progress messages in the chat. You may see steps like searching, reviewing results (potentially with estimated token counts like `~993 tokens` just for review), attempting to scrape pages, and indicating lengthy content requires summarization, before it likely stalls. *Example snippet:* ``` Agent @agent invoked. Swapping over to agent chat. Type /exit to exit agent execution loop early. @agent: Searching on Google for "..." @agent: I found 10 results - reviewing the results now. (~993 tokens) @agent: Searching on Google for "..." @agent: I found 10 results - reviewing the results now. (~851 tokens) @agent: Scraping the content of https://... @agent: This page's content is way too long. I will summarize it right now. # <-- Agent often stalls or loops after this point --> ``` * **B) Backend API Logs (LLM Provider Dashboard):** Simultaneously, observe the stream of API calls being made to the LLM provider. Note the extremely high *input token* count (consistently 6k-8k+ observed with Claude 3.7 Sonnet) for nearly every single call, occurring rapidly. 6. Observe that the agent likely fails to produce a final answer in the chat and eventually hits a `429 rate_limit_error` (reported both in the chat and visible in API logs) due to the excessive input token consumption shown in the backend logs.
yindo added the possible bug label 2026-02-22 18:29:23 -05:00
yindo closed this issue 2026-02-22 18:29:24 -05:00
yindo commented 2026-02-22 18:29:24 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 18, 2025):

In this case, I wouldn't even say this is a bug - your model is calling tools and it is totally within its abilities to not only call a tool but chain those tools together to produce a result. It is widely known that 3.7 is insanely chatty and thinks far too much to do even trivial tasks.

I would be willing to bet that 3.5 sonnet does exactly what you want in minimal calls and without infinite looping. We push these thoughts to the UI so runaway should not be hidden from you during execution. What is the prompt you are using - it likely thinks this amount of "deep research" is required.

Here, anythingllm is giving the model tools, it is calling them, and the model, for whatever reason, will not stop calling more tools because it thinks it has yet to arrive at a solution.

We are simply allowing the model to execute - I think the most we can do here is set a tool iteration max to prevent situations like this.

I'm not saying what you experienced is appropriate, but what I am saying is that it does operate within the bounds and rules of the agent system.

@timothycarambat commented on GitHub (Apr 18, 2025): In this case, I wouldn't even say this is a bug - your model _is calling tools_ and it is totally within its abilities to not only call a tool but chain those tools together to produce a result. It is [widely known that 3.7 is insanely chatty](https://www.reddit.com/r/ClaudeAI/comments/1iyyabe/i_am_massively_disappointed_and_feel_utterly/) and thinks far too much to do even trivial tasks. I would be willing to bet that 3.5 sonnet does exactly what you want in minimal calls and without infinite looping. We push these thoughts to the UI so runaway should not be hidden from you during execution. What is the prompt you are using - it likely thinks this amount of "deep research" is required. Here, anythingllm is giving the model tools, it is calling them, and the model, for whatever reason, will not stop calling more tools because it thinks it has yet to arrive at a solution. We are simply allowing the model to execute - I think the most we can do here is set a tool iteration max to prevent situations like this. I'm not saying what you experienced is _appropriate_, but what I am saying is that it does operate within the bounds and rules of the agent system.
yindo commented 2026-02-22 18:29:24 -05:00
Author
Owner
Copy Link
Copy Source

@LinusAurel commented on GitHub (Apr 18, 2025):

Hey @timothycarambat, thanks for the reply and for digging into the issue! I tested the agent with Claude 3.5 Sonnet and Gemini 2.5 Flash, and both worked well, finishing tasks quickly without burning through tokens or getting stuck in loops.

It does seem like 3.7’s behavior is the main culprit. Your idea of a tool iteration cap sounds like a solid fix to stop this kind of overuse with models like 3.7. Maybe something like Gemini’s thinking budget could also work here?

@LinusAurel commented on GitHub (Apr 18, 2025): Hey @timothycarambat, thanks for the reply and for digging into the issue! I tested the agent with Claude 3.5 Sonnet and Gemini 2.5 Flash, and both worked well, finishing tasks quickly without burning through tokens or getting stuck in loops. It does seem like 3.7’s behavior is the main culprit. Your idea of a tool iteration cap sounds like a solid fix to stop this kind of overuse with models like 3.7. Maybe something like [Gemini’s thinking budget](https://ai.google.dev/gemini-api/docs/thinking) could also work here?
yindo changed title from [BUG]: Agent web search consumes massive input tokens without producing results to [GH-ISSUE #3675] [BUG]: Agent web search consumes massive input tokens without producing results 2026-06-05 14:46:11 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2372
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?