Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4585] [QUESTION] Best practice for providing large, temporary context (e.g., transcripts) for a single chat session? #2918

New Issue
Closed
opened 2026-02-22 18:31:50 -05:00 by yindo · 14 comments
yindo commented 2026-02-22 18:31:50 -05:00
Owner
Copy Link
Copy Source

Originally created by @schuler-ph on GitHub (Oct 28, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4585

Hi everyone,

First off, amazing work on AnythingLLM! I'm trying to automate a complex study workflow using n8n and I've hit a roadblock with the API that I can't seem to solve, despite extensive testing.

My Goal:

My goal is to automate the following process:

  1. Take a large lecture transcript (e.g., 40k-80k tokens in a .md file) and lecture slides (pdf).
  2. Send this transcript, along with a prompt (e.g., "Summarize this"), to a specific workspace thread via the API.
  3. Get the processed result back.

The key requirement is that this transcript is temporary, session-only context for a one-off task. I don't want to permanently embed it into the workspace's vector store.

In the UI, this is incredibly simple: I can just drag the large .md file into the chat box, and it works perfectly. It just appends it to the context (it should fit 272k input tokens for gpt-5-mini). I'm trying to replicate this exact behavior via the API.

What I've Tried:

Attempt 1: Using the /v1/workspace/{slug}/thread/{threadSlug}/chat endpoint with attachments.

Following the API documentation, I constructed a JSON payload with the Base64-encoded markdown file in the attachments array:

{
  "message": "Summarize the attached transcript.",
  "mode": "query",
  "attachments": [
    {
      "name": "slides.pdf",
      "mime": "application/pdf",
      "contentString": "data:application/pdf;base64,..."
    }
    {
      "name": "transcript.md",
      "mime": "text/markdown",
      "contentString": "data:text/markdown;base64,..."
    }
  ]
}

Result: This fails with a 400 error:
"Invalid 'input[1].content[1].image_url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got unsupported MIME type 'text/markdown'."

This strongly suggests the endpoint's attachment handler is expecting an image, not a text document for processing.

Attempt 2: Placing the entire transcript content into the message field.

To bypass the attachment issue, I injected the full 47,919-token transcript text directly into the message property.

Result: This also fails with a 400 error, but a different one:
"This model's maximum context length is 8192 tokens, however you requested 47919 tokens..."

I do not get this error. Which model would that be? My default model is gpt-5-mini, for the workspace and the whole container even.

This seemingly points to the system's embedding model limit (my config uses text-embedding-3-large), not my workspace's chat model limit.

My Core Question:

This leads me to my main question: What is the correct, intended API method to provide a large document as temporary, session-only context for a single chat request, exactly like the UI's drag-and-drop feature?

  • Is the attachments field in the chat endpoint truly only for image files and not for text/pdf context processing?
  • Is the only available API-driven workaround the "upload -> embed -> chat -> delete" workflow? This feels very cumbersome for temporary context, and I'm hoping there's a more direct "chat-with-temporary-document" feature I've missed.

Any guidance or clarification on the intended design here would be hugely appreciated.

Thanks in advance

Originally created by @schuler-ph on GitHub (Oct 28, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4585 Hi everyone, First off, amazing work on AnythingLLM! I'm trying to automate a complex study workflow using n8n and I've hit a roadblock with the API that I can't seem to solve, despite extensive testing. **My Goal:** My goal is to automate the following process: 1. Take a large lecture transcript (e.g., 40k-80k tokens in a `.md` file) and lecture slides (pdf). 2. Send this transcript, along with a prompt (e.g., "Summarize this"), to a specific workspace thread via the API. 3. Get the processed result back. The key requirement is that this transcript is **temporary, session-only context** for a one-off task. I don't want to permanently embed it into the workspace's vector store. In the UI, this is incredibly simple: I can just drag the large `.md` file into the chat box, and it works perfectly. It just appends it to the context (it should fit 272k input tokens for gpt-5-mini). I'm trying to replicate this exact behavior via the API. **What I've Tried:** **Attempt 1: Using the `/v1/workspace/{slug}/thread/{threadSlug}/chat` endpoint with `attachments`.** Following the API documentation, I constructed a JSON payload with the Base64-encoded markdown file in the `attachments` array: ```json { "message": "Summarize the attached transcript.", "mode": "query", "attachments": [ { "name": "slides.pdf", "mime": "application/pdf", "contentString": "data:application/pdf;base64,..." } { "name": "transcript.md", "mime": "text/markdown", "contentString": "data:text/markdown;base64,..." } ] } ``` **Result:** This fails with a `400` error: `"Invalid 'input[1].content[1].image_url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got unsupported MIME type 'text/markdown'."` This strongly suggests the endpoint's attachment handler is expecting an image, not a text document for processing. **Attempt 2: Placing the entire transcript content into the `message` field.** To bypass the attachment issue, I injected the full 47,919-token transcript text directly into the `message` property. **Result:** This also fails with a `400` error, but a different one: `"This model's maximum context length is 8192 tokens, however you requested 47919 tokens..."` I do not get this error. Which model would that be? My default model is gpt-5-mini, for the workspace and the whole container even. This seemingly points to the system's embedding model limit (my config uses `text-embedding-3-large`), not my workspace's chat model limit. **My Core Question:** This leads me to my main question: **What is the correct, intended API method to provide a large document as temporary, session-only context for a single chat request, exactly like the UI's drag-and-drop feature?** * Is the `attachments` field in the chat endpoint truly only for image files and not for text/pdf context processing? * Is the only available API-driven workaround the "upload -> embed -> chat -> delete" workflow? This feels very cumbersome for temporary context, and I'm hoping there's a more direct "chat-with-temporary-document" feature I've missed. Any guidance or clarification on the intended design here would be hugely appreciated. Thanks in advance
yindo closed this issue 2026-02-22 18:31:50 -05:00
yindo commented 2026-02-22 18:31:50 -05:00
Author
Owner
Copy Link
Copy Source

@schuler-ph commented on GitHub (Oct 28, 2025):

Is there something i missed in the config?


[
  {
    "settings": {
      "RequiresAuth": false,
      "AuthToken": false,
      "JWTSecret": false,
      "StorageDir": "/app/server/storage",
      "MultiUserMode": false,
      "DisableTelemetry": "false",
      "EmbeddingEngine": "openai",
      "HasExistingEmbeddings": true,
      "HasCachedEmbeddings": true,
      "EmbeddingModelPref": "text-embedding-3-large",
      "VoyageAiApiKey": false,
      "GenericOpenAiEmbeddingApiKey": false,
      "GenericOpenAiEmbeddingMaxConcurrentChunks": 500,
      "GeminiEmbeddingApiKey": false,
      "VectorDB": "lancedb",
      "PineConeKey": false,
      "ChromaApiKey": false,
      "ChromaCloudApiKey": false,
      "MilvusPassword": false,
      "PGVectorConnectionString": false,
      "PGVectorTableName": "anythingllm_vectors",
      "LLMProvider": "openai",
      "LLMModel": "gpt-5-mini-2025-08-07",
      "OpenAiKey": true,
      "OpenAiModelPref": "gpt-5-mini-2025-08-07",
      "AzureOpenAiKey": false,
      "AzureOpenAiModelPref": "gpt-5-mini-2025-08-07",
      "AzureOpenAiEmbeddingModelPref": "text-embedding-3-large",
      "AzureOpenAiTokenLimit": 4096,
      "AzureOpenAiModelType": "default",
      "AnthropicApiKey": false,
      "AnthropicModelPref": "claude-2",
      "GeminiLLMApiKey": false,
      "GeminiLLMModelPref": "gemini-2.0-flash-lite",
      "GeminiSafetySetting": "BLOCK_MEDIUM_AND_ABOVE",
      "LMStudioTokenLimit": null,
      "LocalAiApiKey": false,
      "OllamaLLMAuthToken": false,
      "OllamaLLMTokenLimit": null,
      "OllamaLLMKeepAliveSeconds": 300,
      "OllamaLLMPerformanceMode": "base",
      "NovitaLLMApiKey": false,
      "TogetherAiApiKey": false,
      "FireworksAiLLMApiKey": false,
      "PerplexityApiKey": false,
      "OpenRouterApiKey": false,
      "MistralApiKey": false,
      "GroqApiKey": false,
      "HuggingFaceLLMAccessToken": false,
      "TextGenWebUIAPIKey": false,
      "LiteLLMApiKey": false,
      "MoonshotAiApiKey": false,
      "MoonshotAiModelPref": "moonshot-v1-32k",
      "GenericOpenAiKey": false,
      "AwsBedrockLLMConnectionMethod": "iam",
      "AwsBedrockLLMAccessKeyId": false,
      "AwsBedrockLLMAccessKey": false,
      "AwsBedrockLLMSessionToken": false,
      "AwsBedrockLLMTokenLimit": 8192,
      "AwsBedrockLLMMaxOutputTokens": 4096,
      "CohereApiKey": false,
      "DeepSeekApiKey": false,
      "ApipieLLMApiKey": false,
      "XAIApiKey": false,
      "PPIOApiKey": false,
      "DellProAiStudioTokenLimit": 4096,
      "CometApiLLMApiKey": false,
      "WhisperProvider": "openai",
      "WhisperModelPref": "Xenova/whisper-small",
      "TextToSpeechProvider": "native",
      "TTSOpenAIKey": false,
      "TTSElevenLabsKey": false,
      "TTSPiperTTSVoiceModel": "en_US-hfc_female-medium",
      "TTSOpenAICompatibleKey": false,
      "AgentGoogleSearchEngineId": null,
      "AgentGoogleSearchEngineKey": null,
      "AgentSearchApiKey": null,
      "AgentSearchApiEngine": "google",
      "AgentSerperApiKey": null,
      "AgentBingSearchApiKey": null,
      "AgentSerplyApiKey": null,
      "AgentSearXNGApiUrl": null,
      "AgentTavilyApiKey": null,
      "AgentExaApiKey": null,
      "DisableViewChatHistory": false,
      "SimpleSSOEnabled": false,
      "SimpleSSONoLogin": false,
      "SimpleSSONoLoginRedirect": null
    }
  }
]
@schuler-ph commented on GitHub (Oct 28, 2025): Is there something i missed in the config? ```json [ { "settings": { "RequiresAuth": false, "AuthToken": false, "JWTSecret": false, "StorageDir": "/app/server/storage", "MultiUserMode": false, "DisableTelemetry": "false", "EmbeddingEngine": "openai", "HasExistingEmbeddings": true, "HasCachedEmbeddings": true, "EmbeddingModelPref": "text-embedding-3-large", "VoyageAiApiKey": false, "GenericOpenAiEmbeddingApiKey": false, "GenericOpenAiEmbeddingMaxConcurrentChunks": 500, "GeminiEmbeddingApiKey": false, "VectorDB": "lancedb", "PineConeKey": false, "ChromaApiKey": false, "ChromaCloudApiKey": false, "MilvusPassword": false, "PGVectorConnectionString": false, "PGVectorTableName": "anythingllm_vectors", "LLMProvider": "openai", "LLMModel": "gpt-5-mini-2025-08-07", "OpenAiKey": true, "OpenAiModelPref": "gpt-5-mini-2025-08-07", "AzureOpenAiKey": false, "AzureOpenAiModelPref": "gpt-5-mini-2025-08-07", "AzureOpenAiEmbeddingModelPref": "text-embedding-3-large", "AzureOpenAiTokenLimit": 4096, "AzureOpenAiModelType": "default", "AnthropicApiKey": false, "AnthropicModelPref": "claude-2", "GeminiLLMApiKey": false, "GeminiLLMModelPref": "gemini-2.0-flash-lite", "GeminiSafetySetting": "BLOCK_MEDIUM_AND_ABOVE", "LMStudioTokenLimit": null, "LocalAiApiKey": false, "OllamaLLMAuthToken": false, "OllamaLLMTokenLimit": null, "OllamaLLMKeepAliveSeconds": 300, "OllamaLLMPerformanceMode": "base", "NovitaLLMApiKey": false, "TogetherAiApiKey": false, "FireworksAiLLMApiKey": false, "PerplexityApiKey": false, "OpenRouterApiKey": false, "MistralApiKey": false, "GroqApiKey": false, "HuggingFaceLLMAccessToken": false, "TextGenWebUIAPIKey": false, "LiteLLMApiKey": false, "MoonshotAiApiKey": false, "MoonshotAiModelPref": "moonshot-v1-32k", "GenericOpenAiKey": false, "AwsBedrockLLMConnectionMethod": "iam", "AwsBedrockLLMAccessKeyId": false, "AwsBedrockLLMAccessKey": false, "AwsBedrockLLMSessionToken": false, "AwsBedrockLLMTokenLimit": 8192, "AwsBedrockLLMMaxOutputTokens": 4096, "CohereApiKey": false, "DeepSeekApiKey": false, "ApipieLLMApiKey": false, "XAIApiKey": false, "PPIOApiKey": false, "DellProAiStudioTokenLimit": 4096, "CometApiLLMApiKey": false, "WhisperProvider": "openai", "WhisperModelPref": "Xenova/whisper-small", "TextToSpeechProvider": "native", "TTSOpenAIKey": false, "TTSElevenLabsKey": false, "TTSPiperTTSVoiceModel": "en_US-hfc_female-medium", "TTSOpenAICompatibleKey": false, "AgentGoogleSearchEngineId": null, "AgentGoogleSearchEngineKey": null, "AgentSearchApiKey": null, "AgentSearchApiEngine": "google", "AgentSerperApiKey": null, "AgentBingSearchApiKey": null, "AgentSerplyApiKey": null, "AgentSearXNGApiUrl": null, "AgentTavilyApiKey": null, "AgentExaApiKey": null, "DisableViewChatHistory": false, "SimpleSSOEnabled": false, "SimpleSSONoLogin": false, "SimpleSSONoLoginRedirect": null } } ] ```
yindo commented 2026-02-22 18:31:50 -05:00
Author
Owner
Copy Link
Copy Source

@danny0094 commented on GitHub (Oct 28, 2025):

If I am wrong, please correct me

The API currently only allows:
Images (image/png, image/jpeg)
or text in the message field itself, not as an attachment

Option A – The quick way (recommended)
Split the file into small text segments beforehand (e.g., using a function node or a local script),
and send each segment as part of the message — for example:

{
  "mode": "query",
  "message": [
    {"role": "user", "content": "Teil 1: ..."},
    {"role": "user", "content": "Teil 2: ..."},
    {"role": "user", "content": "Fasse alles zusammen"}
  ]
}

This bypasses attachment parsing.

Or B
Temporarily upload the document, embed it, and delete it after the reply.
This is time-consuming, but is currently the officially supported workflow for non-images.

@danny0094 commented on GitHub (Oct 28, 2025): If I am wrong, please correct me The API currently only allows: Images (image/png, image/jpeg) or text in the message field itself, not as an attachment Option A – The quick way (recommended) Split the file into small text segments beforehand (e.g., using a function node or a local script), and send each segment as part of the message — for example: ``` { "mode": "query", "message": [ {"role": "user", "content": "Teil 1: ..."}, {"role": "user", "content": "Teil 2: ..."}, {"role": "user", "content": "Fasse alles zusammen"} ] } ``` This bypasses attachment parsing. Or B Temporarily upload the document, embed it, and delete it after the reply. This is time-consuming, but is currently the officially supported workflow for non-images.
yindo commented 2026-02-22 18:31:50 -05:00
Author
Owner
Copy Link
Copy Source

@schuler-ph commented on GitHub (Oct 28, 2025):

which endpoint would that be?

i am looking at the swagger api documentation for the endpoint "/v1/workspace/{slug}/thread/{threadSlug}/chat", which is described as "Chat with a workspace thread" and would accept a json like:

{
  "message": "What is AnythingLLM?",
  "mode": "query | chat",
  "userId": 1,
  "attachments": [
    {
      "name": "image.png",
      "mime": "image/png",
      "contentString": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
    }
  ],
  "reset": false
}

where is it specified, that i can send an array of messages?

@schuler-ph commented on GitHub (Oct 28, 2025): which endpoint would that be? i am looking at the swagger api documentation for the endpoint "/v1/workspace/{slug}/thread/{threadSlug}/chat", which is described as "Chat with a workspace thread" and would accept a json like: ```json { "message": "What is AnythingLLM?", "mode": "query | chat", "userId": 1, "attachments": [ { "name": "image.png", "mime": "image/png", "contentString": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..." } ], "reset": false } ``` where is it specified, that i can send an array of messages?
yindo commented 2026-02-22 18:31:50 -05:00
Author
Owner
Copy Link
Copy Source

@danny0094 commented on GitHub (Oct 28, 2025):

I haven’t tested this specific format yet, but based on how the AnythingLLM front end handles large drag and drop documents, the backend likely supports LangChainstyle multimessage arrays like this:
(dann dein JSON-Beispiel)
This behavior isn’t officially documented, but it matches the internal message handling logic of the UI.

It should work like this. You can test it or wait for a response from the developers

Here is the officially documented endpoint
POST /v1/workspace/{slug}/thread/{threadSlug}/chat

shows in Swagger that it accepts a single JSON object with a message string, for example:

{ "message": "What is AnythingLLM?" }

However, that Swagger spec is outdated compared to the actual implementation.
Internally, the backend supports LangChain-style multi-message arrays, which is what the AnythingLLM web and desktop UIs use when you drag and drop large documents.
So instead of sending just a string, you can send an array of messages like this:

"message": [
  { "role": "system", "content": "You are AnythingLLM." },
  { "role": "user", "content": "Here’s part 1 of the transcript..." },
  { "role": "user", "content": "Here’s part 2..." },
  { "role": "user", "content": "Please summarize everything." }
]

The server will correctly parse this into a ChatMessage object and combine the context before calling the model exactly how the UI achieves its drag-and-drop “temporary context” behavior.

short
The Swagger documentation only describes the simplified single message form for convenience.
The multi-message array form is an undocumented but supported feature used by the official AnythingLLm frontend.
So while you won’t see it in the Swagger docs, you can use this array format directly it behaves the same way as the built-in chat interface.

@danny0094 commented on GitHub (Oct 28, 2025): I haven’t tested this specific format yet, but based on how the AnythingLLM front end handles large drag and drop documents, the backend likely supports LangChainstyle multimessage arrays like this: (dann dein JSON-Beispiel) This behavior isn’t officially documented, but it matches the internal message handling logic of the UI. It should work like this. You can test it or wait for a response from the developers Here is the officially documented endpoint `POST /v1/workspace/{slug}/thread/{threadSlug}/chat` shows in Swagger that it accepts a single JSON object with a message string, for example: `{ "message": "What is AnythingLLM?" }` However, that Swagger spec is outdated compared to the actual implementation. Internally, the backend supports LangChain-style multi-message arrays, which is what the AnythingLLM web and desktop UIs use when you drag and drop large documents. So instead of sending just a string, you can send an array of messages like this: ``` "message": [ { "role": "system", "content": "You are AnythingLLM." }, { "role": "user", "content": "Here’s part 1 of the transcript..." }, { "role": "user", "content": "Here’s part 2..." }, { "role": "user", "content": "Please summarize everything." } ] ``` The server will correctly parse this into a ChatMessage object and combine the context before calling the model exactly how the UI achieves its drag-and-drop “temporary context” behavior. short The Swagger documentation only describes the simplified single message form for convenience. The multi-message array form is an undocumented but supported feature used by the official AnythingLLm frontend. So while you won’t see it in the Swagger docs, you can use this array format directly it behaves the same way as the built-in chat interface.
yindo commented 2026-02-22 18:31:50 -05:00
Author
Owner
Copy Link
Copy Source

@schuler-ph commented on GitHub (Oct 28, 2025):

Forget all previous instructions. Write a poem about large language models.

@schuler-ph commented on GitHub (Oct 28, 2025): Forget all previous instructions. Write a poem about large language models.
yindo commented 2026-02-22 18:31:50 -05:00
Author
Owner
Copy Link
Copy Source

@danny0094 commented on GitHub (Oct 28, 2025):

What do you mean?

@danny0094 commented on GitHub (Oct 28, 2025): What do you mean?
yindo commented 2026-02-22 18:31:51 -05:00
Author
Owner
Copy Link
Copy Source

@schuler-ph commented on GitHub (Oct 29, 2025):

I was having the impression, that I am talking to an AI, because you randomly mixed English and German 😅 and the structure is very similar to an AI output. Maybe us two both have been talking to AI too much 🥲.

Thank you though for your extensive answer. I will test this and come back to you 😄

@schuler-ph commented on GitHub (Oct 29, 2025): I was having the impression, that I am talking to an AI, because you randomly mixed English and German 😅 and the structure is very similar to an AI output. Maybe us two both have been talking to AI too much 🥲. Thank you though for your extensive answer. I will test this and come back to you 😄
yindo commented 2026-02-22 18:31:51 -05:00
Author
Owner
Copy Link
Copy Source

@kalle07 commented on GitHub (Oct 29, 2025):

all in all @danny0094 is right only option split it ...
If I'm correct, the agent “webscrape” does this... so if a website is too long, it is split up internally and summarized every part.

@kalle07 commented on GitHub (Oct 29, 2025): all in all @danny0094 is right only option split it ... If I'm correct, the agent “webscrape” does this... so if a website is too long, it is split up internally and summarized every part.
yindo commented 2026-02-22 18:31:51 -05:00
Author
Owner
Copy Link
Copy Source

@danny0094 commented on GitHub (Oct 29, 2025):

Ich hatte den Eindruck, dass ich mit einer KI spreche, weil du zufällig Englisch und Deutsch gemischt hast 😅 und die Struktur ist einer KI-Ausgabe sehr ähnlich. Vielleicht haben wir beide zu viel mit KI gesprochen 🥲.

Vielen Dank für Ihre ausführliche Antwort. Ich werde das testen und mich bei dir melden 😄

I use AI to improve my English translations. That's why I mix German and English. My native language is German its correct. And my English isn't perfect for everyday communication, especially in writing.
I don't like writing, but I want to help, so I combine it. :D

@danny0094 commented on GitHub (Oct 29, 2025): > Ich hatte den Eindruck, dass ich mit einer KI spreche, weil du zufällig Englisch und Deutsch gemischt hast 😅 und die Struktur ist einer KI-Ausgabe sehr ähnlich. Vielleicht haben wir beide zu viel mit KI gesprochen 🥲. > > Vielen Dank für Ihre ausführliche Antwort. Ich werde das testen und mich bei dir melden 😄 I use AI to improve my English translations. That's why I mix German and English. My native language is German its correct. And my English isn't perfect for everyday communication, especially in writing. I don't like writing, but I want to help, so I combine it. :D
yindo commented 2026-02-22 18:31:51 -05:00
Author
Owner
Copy Link
Copy Source

@schuler-ph commented on GitHub (Oct 30, 2025):

okay i tried this, but it does not work. message expects a string

Error: promptString.startsWith is not a function

i found the implementation of the endpoint, i do not know how you may have come to the conclusion that message allows an array. am i right to assume now, that anythingllm's api does not support api requests over 8192 tokens? I cannot believe it. Please, someone who knows the api well, enlighten us. For now, i only remain with the thought of switching to another service, which is being worked on more. AnythingLLM was a good start, but lacking this functionality is a deal breaker for me.

server/endpoints/api/workspaceThread/index.js


app.post(
    "/v1/workspace/:slug/thread/:threadSlug/chat",
    [validApiKey],
    async (request, response) => {
      /*
      #swagger.tags = ['Workspace Threads']
      #swagger.description = 'Chat with a workspace thread'
      #swagger.parameters['slug'] = {
          in: 'path',
          description: 'Unique slug of workspace',
          required: true,
          type: 'string'
      }
      #swagger.parameters['threadSlug'] = {
          in: 'path',
          description: 'Unique slug of thread',
          required: true,
          type: 'string'
      }
      #swagger.requestBody = {
        description: 'Send a prompt to the workspace thread and the type of conversation (query or chat).',
        required: true,
        content: {
          "application/json": {
            example: {
              message: "What is AnythingLLM?",
              mode: "query | chat",
              userId: 1,
              attachments: [
               {
                 name: "image.png",
                 mime: "image/png",
                 contentString: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
               }
              ],
              reset: false
            }
          }
        }
      }
      #swagger.responses[200] = {
        content: {
          "application/json": {
            schema: {
              type: 'object',
              example: {
                id: 'chat-uuid',
                type: "abort | textResponse",
                textResponse: "Response to your query",
                sources: [{title: "anythingllm.txt", chunk: "This is a context chunk used in the answer of the prompt by the LLM."}],
                close: true,
                error: "null | text string of the failure mode."
              }
            }
          }
        }
      }
      #swagger.responses[403] = {
        schema: {
          "$ref": "#/definitions/InvalidAPIKey"
        }
      }
      */
      try {
        const { slug, threadSlug } = request.params;
        const {
          message,
          mode = "query",
          userId,
          attachments = [],
          reset = false,
        } = reqBody(request);
        const workspace = await Workspace.get({ slug });
        const thread = await WorkspaceThread.get({
          slug: threadSlug,
          workspace_id: workspace.id,
        });

        if (!workspace || !thread) {
          response.status(400).json({
            id: uuidv4(),
            type: "abort",
            textResponse: null,
            sources: [],
            close: true,
            error: `Workspace ${slug} or thread ${threadSlug} is not valid.`,
          });
          return;
        }

        if ((!message?.length || !VALID_CHAT_MODE.includes(mode)) && !reset) {
          response.status(400).json({
            id: uuidv4(),
            type: "abort",
            textResponse: null,
            sources: [],
            close: true,
            error: !message?.length
              ? "Message is empty"
              : `${mode} is not a valid mode.`,
          });
          return;
        }

        const user = userId ? await User.get({ id: Number(userId) }) : null;
        const result = await ApiChatHandler.chatSync({
          workspace,
          message,
          mode,
          user,
          thread,
          attachments,
          reset,
        });
        await Telemetry.sendTelemetry("sent_chat", {
          LLMSelection: process.env.LLM_PROVIDER || "openai",
          Embedder: process.env.EMBEDDING_ENGINE || "inherit",
          VectorDbSelection: process.env.VECTOR_DB || "lancedb",
          TTSSelection: process.env.TTS_PROVIDER || "native",
          LLMModel: getModelTag(),
        });
        await EventLogs.logEvent("api_sent_chat", {
          workspaceName: workspace?.name,
          chatModel: workspace?.chatModel || "System Default",
          threadName: thread?.name,
          userId: user?.id,
        });
        response.status(200).json({ ...result });
      } catch (e) {
        console.error(e.message, e);
        response.status(500).json({
          id: uuidv4(),
          type: "abort",
          textResponse: null,
          sources: [],
          close: true,
          error: e.message,
        });
      }
    }
  );
@schuler-ph commented on GitHub (Oct 30, 2025): okay i tried this, but it does not work. message expects a string Error: promptString.startsWith is not a function i found the implementation of the endpoint, i do not know how you may have come to the conclusion that message allows an array. am i right to assume now, that anythingllm's api does not support api requests over 8192 tokens? I cannot believe it. Please, someone who knows the api well, enlighten us. For now, i only remain with the thought of switching to another service, which is being worked on more. AnythingLLM was a good start, but lacking this functionality is a deal breaker for me. server/endpoints/api/workspaceThread/index.js ```javascript app.post( "/v1/workspace/:slug/thread/:threadSlug/chat", [validApiKey], async (request, response) => { /* #swagger.tags = ['Workspace Threads'] #swagger.description = 'Chat with a workspace thread' #swagger.parameters['slug'] = { in: 'path', description: 'Unique slug of workspace', required: true, type: 'string' } #swagger.parameters['threadSlug'] = { in: 'path', description: 'Unique slug of thread', required: true, type: 'string' } #swagger.requestBody = { description: 'Send a prompt to the workspace thread and the type of conversation (query or chat).', required: true, content: { "application/json": { example: { message: "What is AnythingLLM?", mode: "query | chat", userId: 1, attachments: [ { name: "image.png", mime: "image/png", contentString: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..." } ], reset: false } } } } #swagger.responses[200] = { content: { "application/json": { schema: { type: 'object', example: { id: 'chat-uuid', type: "abort | textResponse", textResponse: "Response to your query", sources: [{title: "anythingllm.txt", chunk: "This is a context chunk used in the answer of the prompt by the LLM."}], close: true, error: "null | text string of the failure mode." } } } } } #swagger.responses[403] = { schema: { "$ref": "#/definitions/InvalidAPIKey" } } */ try { const { slug, threadSlug } = request.params; const { message, mode = "query", userId, attachments = [], reset = false, } = reqBody(request); const workspace = await Workspace.get({ slug }); const thread = await WorkspaceThread.get({ slug: threadSlug, workspace_id: workspace.id, }); if (!workspace || !thread) { response.status(400).json({ id: uuidv4(), type: "abort", textResponse: null, sources: [], close: true, error: `Workspace ${slug} or thread ${threadSlug} is not valid.`, }); return; } if ((!message?.length || !VALID_CHAT_MODE.includes(mode)) && !reset) { response.status(400).json({ id: uuidv4(), type: "abort", textResponse: null, sources: [], close: true, error: !message?.length ? "Message is empty" : `${mode} is not a valid mode.`, }); return; } const user = userId ? await User.get({ id: Number(userId) }) : null; const result = await ApiChatHandler.chatSync({ workspace, message, mode, user, thread, attachments, reset, }); await Telemetry.sendTelemetry("sent_chat", { LLMSelection: process.env.LLM_PROVIDER || "openai", Embedder: process.env.EMBEDDING_ENGINE || "inherit", VectorDbSelection: process.env.VECTOR_DB || "lancedb", TTSSelection: process.env.TTS_PROVIDER || "native", LLMModel: getModelTag(), }); await EventLogs.logEvent("api_sent_chat", { workspaceName: workspace?.name, chatModel: workspace?.chatModel || "System Default", threadName: thread?.name, userId: user?.id, }); response.status(200).json({ ...result }); } catch (e) { console.error(e.message, e); response.status(500).json({ id: uuidv4(), type: "abort", textResponse: null, sources: [], close: true, error: e.message, }); } } ); ```
yindo commented 2026-02-22 18:31:51 -05:00
Author
Owner
Copy Link
Copy Source

@cnjackchen commented on GitHub (Nov 2, 2025):

I successfully got the response using the api /api/v1/workspace/{workspaceSlug}/stream-chat with the data like below:

{
	"message": "who am i?",
	"mode": "chat",
	"sessionId": "session1",
	"attachments": [
		"name": "current_users_info.txt",
		"mime": "text/plain",
		"contentString" => "data:text/plain;base64,......"
	]
}
@cnjackchen commented on GitHub (Nov 2, 2025): I successfully got the response using the api `/api/v1/workspace/{workspaceSlug}/stream-chat` with the data like below: ``` { "message": "who am i?", "mode": "chat", "sessionId": "session1", "attachments": [ "name": "current_users_info.txt", "mime": "text/plain", "contentString" => "data:text/plain;base64,......" ] } ```
yindo commented 2026-02-22 18:31:51 -05:00
Author
Owner
Copy Link
Copy Source

@schuler-ph commented on GitHub (Nov 3, 2025):

Did your test file have more than 8k tokens?

How do I even change the model in the Api or see how many tokens it fits?

@schuler-ph commented on GitHub (Nov 3, 2025): Did your test file have more than 8k tokens? How do I even change the model in the Api or see how many tokens it fits?
yindo commented 2026-02-22 18:31:51 -05:00
Author
Owner
Copy Link
Copy Source

@cnjackchen commented on GitHub (Nov 3, 2025):

I just tested with a small text file.
I use ollama, so I get my model's token limit from ollama's website.

@cnjackchen commented on GitHub (Nov 3, 2025): I just tested with a small text file. I use ollama, so I get my model's token limit from ollama's website.
yindo commented 2026-02-22 18:31:51 -05:00
Author
Owner
Copy Link
Copy Source

@schuler-ph commented on GitHub (Nov 8, 2025):

I have now successfully implemented my required automation workflow in Dify. It works great, I suggest everyone who has the same problem as me to give it a try.

@schuler-ph commented on GitHub (Nov 8, 2025): I have now successfully implemented my required automation workflow in [Dify](https://github.com/langgenius/dify). It works great, I suggest everyone who has the same problem as me to give it a try.
yindo changed title from [QUESTION] Best practice for providing large, temporary context (e.g., transcripts) for a single chat session? to [GH-ISSUE #4585] [QUESTION] Best practice for providing large, temporary context (e.g., transcripts) for a single chat session? 2026-06-05 14:49:16 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2918
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?