[GH-ISSUE #4611] [BUG]: forwardExtensionRequest causes X-Integrity signature mismatch #2932

Closed
opened 2026-02-22 18:31:54 -05:00 by yindo · 4 comments
Owner

Originally created by @DocterJac on GitHub (Nov 4, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4611

Description

The forwardExtensionRequest function in server/utils/collectorApi/index.js is not consistently handling body serialization, which causes X-Integrity signature mismatches when the body parameter is passed as an object instead of a pre-stringified string.

Impact

  • Extension endpoints (YouTube transcript import, website scraping) fail with signature errors
  • The X-Integrity header is calculated on a different value than what's sent in the request body
  • Inconsistent behavior compared to other methods in the same file

Root Cause

In the current implementation:

async forwardExtensionRequest({ endpoint, method, body }) {
  return await fetch(`${this.endpoint}${endpoint}`, {
    method,
    body, // ← May be object or string
    headers: {
      "Content-Type": "application/json",
      "X-Integrity": this.comkey.sign(body), // ← Signature on potentially unstringified body
      ...
    }
  });
}``

When body is an object:

  1. fetch() will stringify it automatically for the request
  2. But this.comkey.sign(body) may calculate signature on the object directly
  3. This causes a mismatch between the signature and the actual request body

Proposed Fix
Add explicit stringification to match the pattern used in other methods (processRawText, processLink):

async forwardExtensionRequest({ endpoint, method, body }) {
  // Stringify body if it's an object (same pattern as other methods)
  const data = typeof body === 'string' ? body : JSON.stringify(body);
  
  return await fetch(`${this.endpoint}${endpoint}`, {
    method,
    body: data,
    headers: {
      "Content-Type": "application/json",
      "X-Integrity": this.comkey.sign(data), // ← Signature on stringified data
      "X-Payload-Signer": this.comkey.encrypt(
        new EncryptionManager().xPayload
      ),
    },
  });
}

Testing
Tested with:
YouTube transcript import endpoint (/api/ext/youtube/transcript)
Website scraping endpoint (/api/ext/website-depth)
• Both work correctly after fix

File Location
server/utils/collectorApi/index.js - Line ~179-195

Fix Available
I have implemented and tested this fix in my fork: https://github.com/DocterJac/anything-llm/commit/407e0192

Environment
• AnythingLLM version: Latest (master branch)
• Node.js version: Latest
• Docker deployment

Originally created by @DocterJac on GitHub (Nov 4, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4611 ## Description The `forwardExtensionRequest` function in `server/utils/collectorApi/index.js` is not consistently handling body serialization, which causes X-Integrity signature mismatches when the body parameter is passed as an object instead of a pre-stringified string. ## Impact - Extension endpoints (YouTube transcript import, website scraping) fail with signature errors - The X-Integrity header is calculated on a different value than what's sent in the request body - Inconsistent behavior compared to other methods in the same file ## Root Cause In the current implementation: ```javascript async forwardExtensionRequest({ endpoint, method, body }) { return await fetch(`${this.endpoint}${endpoint}`, { method, body, // ← May be object or string headers: { "Content-Type": "application/json", "X-Integrity": this.comkey.sign(body), // ← Signature on potentially unstringified body ... } }); }`` ``` When body is an object: 1. fetch() will stringify it automatically for the request 2. But this.comkey.sign(body) may calculate signature on the object directly 3. This causes a mismatch between the signature and the actual request body Proposed Fix Add explicit stringification to match the pattern used in other methods (processRawText, processLink): ``` async forwardExtensionRequest({ endpoint, method, body }) { // Stringify body if it's an object (same pattern as other methods) const data = typeof body === 'string' ? body : JSON.stringify(body); return await fetch(`${this.endpoint}${endpoint}`, { method, body: data, headers: { "Content-Type": "application/json", "X-Integrity": this.comkey.sign(data), // ← Signature on stringified data "X-Payload-Signer": this.comkey.encrypt( new EncryptionManager().xPayload ), }, }); } ``` Testing Tested with: • ✅ YouTube transcript import endpoint (/api/ext/youtube/transcript) • ✅ Website scraping endpoint (/api/ext/website-depth) • Both work correctly after fix File Location server/utils/collectorApi/index.js - Line ~179-195 Fix Available I have implemented and tested this fix in my fork: https://github.com/DocterJac/anything-llm/commit/407e0192 Environment • AnythingLLM version: Latest (master branch) • Node.js version: Latest • Docker deployment
yindo added the needs info / can't replicateinvestigating labels 2026-02-22 18:31:54 -05:00
yindo closed this issue 2026-02-22 18:31:54 -05:00
Author
Owner

@timothycarambat commented on GitHub (Nov 20, 2025):

Extension endpoints (YouTube transcript import, website scraping) fail with signature errors

I have not been able to reproduce this with any query thus far or through any code path. How are you reproducing this? Closing until we have a known good reproduction

@timothycarambat commented on GitHub (Nov 20, 2025): > Extension endpoints (YouTube transcript import, website scraping) fail with signature errors I have not been able to reproduce this with any query thus far or through any code path. How are you reproducing this? Closing until we have a known good reproduction
Author
Owner

@timothycarambat commented on GitHub (Nov 20, 2025):

The only way to replicate that is to curl to the frontend API directly - this is not possible from our UI. I suppose it is worth fixing even though it is from not from official use

curl --location 'http://localhost:3001/api/ext/youtube/transcript' \
--header 'Content-Type: application/json' \ # this is the issue
--data '{ "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ" }'
@timothycarambat commented on GitHub (Nov 20, 2025): The only way to replicate that is to curl to the frontend API directly - this is not possible from our UI. I suppose it is worth fixing even though it is from not from official use ```bash curl --location 'http://localhost:3001/api/ext/youtube/transcript' \ --header 'Content-Type: application/json' \ # this is the issue --data '{ "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ" }' ```
Author
Owner

@DocterJac commented on GitHub (Nov 21, 2025):

AnythingLLM Bug Report Documentation

We discovered and fixed a critical bug in AnythingLLM's forwardExtensionRequest() function that was causing X-Integrity signature mismatches. This bug prevented YouTube transcript imports and website scraping from working correctly in our CogniBase V4 RAG document management system.


How We Found the Bug

Context

We were implementing a comprehensive RAG document management system in our project CogniBase V4 that integrates with AnythingLLM for document storage and embedding. Our system includes:

  1. Document Upload - Direct file uploads to AnythingLLM
  2. YouTube Import - Import video transcripts via AnythingLLM's extension API
  3. Website Scraping - Bulk website scraping via AnythingLLM's extension API
  4. Storage Management - View, organize, and clean up documents

Discovery Process

Step 1: Testing YouTube Import (November 3, 2024)

  • Implemented YouTube transcript import feature in CogniBase
  • Backend endpoint: /api/rag/import/youtube
  • Called AnythingLLM's /api/ext/youtube/transcript endpoint
  • Result: Failed with authentication/signature errors

Step 2: Testing Website Scraping

  • Implemented bulk website scraper in CogniBase
  • Backend endpoint: /api/rag/import/website
  • Called AnythingLLM's /api/ext/website-depth endpoint
  • Result: Also failed with similar signature errors

Step 3: Investigation

  • Reviewed AnythingLLM API documentation
  • Checked authentication token (AUTH_TOKEN) - correct
  • Examined JWT token generation - working correctly
  • Reviewed request headers and payload - properly formatted
  • Hypothesis: Problem must be in how AnythingLLM handles extension API requests

Step 4: Deep Dive into AnythingLLM Code

  • Located the issue in server/utils/collectorApi/index.js
  • Function: forwardExtensionRequest()
  • Root cause identified:
    • Function receives body parameter (can be object or string)
    • Calculates X-Integrity signature: this.comkey.sign(body)
    • Sends body to fetch: fetch(..., { body })
    • When body is an object:
      • fetch() internally stringifies it
      • But signature was calculated on the original object
      • Signature mismatch!

Step 5: Verification

  • Compared with other methods in same file:
    • processRawText() - properly stringifies before signing
    • processLink() - properly stringifies before signing
    • forwardExtensionRequest() - does NOT stringify
  • Pattern inconsistency confirmed

The Bug Explained

Technical Details

File: server/utils/collectorApi/index.js
Function: forwardExtensionRequest()
Line: ~179-195

Original Buggy Code:

async forwardExtensionRequest({ endpoint, method, body }) {
  return await fetch(`${this.endpoint}${endpoint}`, {
    method,
    body, // ← Problem: may be object or string
    headers: {
      "Content-Type": "application/json",
      "X-Integrity": this.comkey.sign(body), // ← Signature on potentially unstringified body
      "X-Payload-Signer": this.comkey.encrypt(
        new EncryptionManager().xPayload
      ),
    },
  });
}

Why It Fails

  1. When body is an object:

    body = { url: "https://youtube.com/watch?v=abc123" }
    
  2. Signature calculation:

    signature = this.comkey.sign(body)
    // Signs: [object Object] or similar representation
    
  3. Fetch sends:

    fetch(..., { body: body })
    // Internally stringifies: '{"url":"https://youtube.com/watch?v=abc123"}'
    
  4. Server receives:

    • Body: '{"url":"https://youtube.com/watch?v=abc123"}'
    • X-Integrity signature: calculated on unstringified object
    • Signature verification fails!

Impact

Broken Features:

  • ✗ YouTube transcript import
  • ✗ Website bulk scraping
  • ✗ Any extension API endpoints
  • ✗ Third-party integrations using forwardExtensionRequest

Affected Systems:

  • CogniBase V4 RAG document management
  • Possibly any system using AnythingLLM's extension APIs

Why We Had to Fix It

Business Requirements

CogniBase V4 Storage Management System Requirements:

  1. Document Diversity - Need multiple input sources:

    • Direct uploads (PDFs, DOCX, etc.)
    • YouTube transcripts (via extension API) Blocked by bug
    • Website content (via extension API) Blocked by bug
  2. Storage Management - Need to track and clean up documents:

    • Can't manage documents we can't import
    • Storage monitoring requires all import methods working
  3. User Experience - Can't ship incomplete features:

    • Users need YouTube import for educational content
    • Users need website scraping for research
    • Half-working features = bad UX

Why We Couldn't Wait for Upstream Fix

Timeline Pressure:

  • Implementing complete RAG management system
  • Users expecting YouTube and website import features
  • Can't pause development waiting for upstream response
  • Issue filing → Review → Fix → Release could take weeks/months

Self-Service Solution:

  • We have the expertise to fix it ourselves
  • Fix is straightforward (5 lines of code)
  • We can test thoroughly in our environment
  • Can contribute back to community via PR

Our Fix

Solution Implementation

Fixed Code:

async forwardExtensionRequest({ endpoint, method, body }) {
  // Stringify body if it's an object (same pattern as other methods)
  const data = typeof body === 'string' ? body : JSON.stringify(body);
  
  return await fetch(`${this.endpoint}${endpoint}`, {
    method,
    body: data, // ← Now always a string
    headers: {
      "Content-Type": "application/json",
      "X-Integrity": this.comkey.sign(data), // ← Signature on stringified data
      "X-Payload-Signer": this.comkey.encrypt(
        new EncryptionManager().xPayload
      ),
    },
  });
}

Why This Fix Works

  1. Explicit Stringification:

    • Check if body is already a string
    • If not, stringify it explicitly
    • Store result in data variable
  2. Consistent Signature:

    • Both body: data and sign(data) use same value
    • Signature calculated on exact string being sent
    • No more mismatch!
  3. Pattern Consistency:

    • Matches processRawText() implementation
    • Matches processLink() implementation
    • Follows established codebase patterns

Testing Performed

Test 1: YouTube Transcript Import

  • Endpoint: /api/ext/youtube/transcript
  • Test URL: Various YouTube videos
  • Result: Working perfectly
  • Transcripts imported successfully
  • Documents appear in AnythingLLM storage
  • CogniBase can list and manage imported transcripts

Test 2: Website Scraping

  • Endpoint: /api/ext/website-depth
  • Test URLs: Multiple websites at various crawl depths
  • Result: Working perfectly
  • Pages scraped and stored correctly
  • All scraped content accessible
  • CogniBase storage management shows all files

Test 3: Backward Compatibility

  • Direct uploads still work
  • Regular document APIs still work
  • No regression in existing functionality

Implementation in CogniBase V4

Backend Integration

File: backend/services/rag_service.py

YouTube Import Method:

def import_youtube_transcript(self, url: str) -> Dict:
    jwt_token = self._get_jwt_token()
    
    response = requests.post(
        f"{self.endpoint}/api/ext/youtube/transcript",
        headers={
            "Authorization": f"Bearer {jwt_token}",
            "Content-Type": "application/json"
        },
        json={"url": url},  # ← Body as object, AnythingLLM now handles correctly
        timeout=60
    )
    
    return response.json()

Website Scraping Method:

def scrape_website(self, url: str, depth: int = 1, max_pages: int = 20) -> Dict:
    jwt_token = self._get_jwt_token()
    
    response = requests.post(
        f"{self.endpoint}/api/ext/website-depth",
        headers={
            "Authorization": f"Bearer {jwt_token}",
            "Content-Type": "application/json"
        },
        json={
            "url": url,
            "depth": depth,
            "maxLinks": max_pages
        },  # ← Body as object, AnythingLLM now handles correctly
        timeout=120
    )
    
    return response.json()

Frontend Integration

Component: frontend/src/components/DataConnectors.jsx

Features:

  • YouTube URL input with validation
  • Website URL input with crawl depth and max pages settings
  • Real-time import status
  • Error handling and user feedback
  • Integration with RAG document management

Commit Details

Our Fork Commit

Repository: https://github.com/DocterJac/anything-llm
Branch: master
Commit: 407e0192
Date: November 3, 2024

Commit Message:

Fix: Properly stringify body in forwardExtensionRequest

Bug: The forwardExtensionRequest function was not consistently handling
body serialization, which caused issues with the X-Integrity signature
when the body was passed as an object instead of a string.

Fix: Added explicit body stringification check and use the stringified
version for both the request body and X-Integrity signature calculation.

This ensures the signature is always calculated on the same data that
gets sent in the request body, preventing signature mismatch errors.

Pattern matches other methods in the same file (e.g., processRawText,
processLink) which already do proper stringification.

@DocterJac commented on GitHub (Nov 21, 2025): # AnythingLLM Bug Report Documentation We discovered and fixed a critical bug in AnythingLLM's `forwardExtensionRequest()` function that was causing X-Integrity signature mismatches. This bug prevented YouTube transcript imports and website scraping from working correctly in our CogniBase V4 RAG document management system. --- ## How We Found the Bug ### Context We were implementing a comprehensive RAG document management system in our project CogniBase V4 that integrates with AnythingLLM for document storage and embedding. Our system includes: 1. **Document Upload** - Direct file uploads to AnythingLLM 2. **YouTube Import** - Import video transcripts via AnythingLLM's extension API 3. **Website Scraping** - Bulk website scraping via AnythingLLM's extension API 4. **Storage Management** - View, organize, and clean up documents ### Discovery Process **Step 1: Testing YouTube Import (November 3, 2024)** - Implemented YouTube transcript import feature in CogniBase - Backend endpoint: `/api/rag/import/youtube` - Called AnythingLLM's `/api/ext/youtube/transcript` endpoint - **Result:** Failed with authentication/signature errors **Step 2: Testing Website Scraping** - Implemented bulk website scraper in CogniBase - Backend endpoint: `/api/rag/import/website` - Called AnythingLLM's `/api/ext/website-depth` endpoint - **Result:** Also failed with similar signature errors **Step 3: Investigation** - Reviewed AnythingLLM API documentation - Checked authentication token (AUTH_TOKEN) - correct - Examined JWT token generation - working correctly - Reviewed request headers and payload - properly formatted - **Hypothesis:** Problem must be in how AnythingLLM handles extension API requests **Step 4: Deep Dive into AnythingLLM Code** - Located the issue in `server/utils/collectorApi/index.js` - Function: `forwardExtensionRequest()` - **Root cause identified:** - Function receives `body` parameter (can be object or string) - Calculates X-Integrity signature: `this.comkey.sign(body)` - Sends body to fetch: `fetch(..., { body })` - When body is an object: - `fetch()` internally stringifies it - But signature was calculated on the original object - **Signature mismatch!** **Step 5: Verification** - Compared with other methods in same file: - `processRawText()` - properly stringifies before signing ✅ - `processLink()` - properly stringifies before signing ✅ - `forwardExtensionRequest()` - does NOT stringify ❌ - **Pattern inconsistency confirmed** --- ## The Bug Explained ### Technical Details **File:** `server/utils/collectorApi/index.js` **Function:** `forwardExtensionRequest()` **Line:** ~179-195 **Original Buggy Code:** ```javascript async forwardExtensionRequest({ endpoint, method, body }) { return await fetch(`${this.endpoint}${endpoint}`, { method, body, // ← Problem: may be object or string headers: { "Content-Type": "application/json", "X-Integrity": this.comkey.sign(body), // ← Signature on potentially unstringified body "X-Payload-Signer": this.comkey.encrypt( new EncryptionManager().xPayload ), }, }); } ``` ### Why It Fails 1. **When body is an object:** ```javascript body = { url: "https://youtube.com/watch?v=abc123" } ``` 2. **Signature calculation:** ```javascript signature = this.comkey.sign(body) // Signs: [object Object] or similar representation ``` 3. **Fetch sends:** ```javascript fetch(..., { body: body }) // Internally stringifies: '{"url":"https://youtube.com/watch?v=abc123"}' ``` 4. **Server receives:** - Body: `'{"url":"https://youtube.com/watch?v=abc123"}'` - X-Integrity signature: calculated on unstringified object - **Signature verification fails!** ### Impact **Broken Features:** - ✗ YouTube transcript import - ✗ Website bulk scraping - ✗ Any extension API endpoints - ✗ Third-party integrations using forwardExtensionRequest **Affected Systems:** - CogniBase V4 RAG document management - Possibly any system using AnythingLLM's extension APIs --- ## Why We Had to Fix It ### Business Requirements **CogniBase V4 Storage Management System Requirements:** 1. **Document Diversity** - Need multiple input sources: - Direct uploads (PDFs, DOCX, etc.) ✅ - YouTube transcripts (via extension API) ❌ Blocked by bug - Website content (via extension API) ❌ Blocked by bug 2. **Storage Management** - Need to track and clean up documents: - Can't manage documents we can't import - Storage monitoring requires all import methods working 3. **User Experience** - Can't ship incomplete features: - Users need YouTube import for educational content - Users need website scraping for research - Half-working features = bad UX ### Why We Couldn't Wait for Upstream Fix **Timeline Pressure:** - Implementing complete RAG management system - Users expecting YouTube and website import features - Can't pause development waiting for upstream response - Issue filing → Review → Fix → Release could take weeks/months **Self-Service Solution:** - We have the expertise to fix it ourselves - Fix is straightforward (5 lines of code) - We can test thoroughly in our environment - Can contribute back to community via PR --- ## Our Fix ### Solution Implementation **Fixed Code:** ```javascript async forwardExtensionRequest({ endpoint, method, body }) { // Stringify body if it's an object (same pattern as other methods) const data = typeof body === 'string' ? body : JSON.stringify(body); return await fetch(`${this.endpoint}${endpoint}`, { method, body: data, // ← Now always a string headers: { "Content-Type": "application/json", "X-Integrity": this.comkey.sign(data), // ← Signature on stringified data "X-Payload-Signer": this.comkey.encrypt( new EncryptionManager().xPayload ), }, }); } ``` ### Why This Fix Works 1. **Explicit Stringification:** - Check if body is already a string - If not, stringify it explicitly - Store result in `data` variable 2. **Consistent Signature:** - Both `body: data` and `sign(data)` use same value - Signature calculated on exact string being sent - No more mismatch! 3. **Pattern Consistency:** - Matches `processRawText()` implementation - Matches `processLink()` implementation - Follows established codebase patterns ### Testing Performed **Test 1: YouTube Transcript Import** - Endpoint: `/api/ext/youtube/transcript` - Test URL: Various YouTube videos - **Result:** ✅ Working perfectly - Transcripts imported successfully - Documents appear in AnythingLLM storage - CogniBase can list and manage imported transcripts **Test 2: Website Scraping** - Endpoint: `/api/ext/website-depth` - Test URLs: Multiple websites at various crawl depths - **Result:** ✅ Working perfectly - Pages scraped and stored correctly - All scraped content accessible - CogniBase storage management shows all files **Test 3: Backward Compatibility** - Direct uploads still work ✅ - Regular document APIs still work ✅ - No regression in existing functionality ✅ --- ## Implementation in CogniBase V4 ### Backend Integration **File:** `backend/services/rag_service.py` **YouTube Import Method:** ```python def import_youtube_transcript(self, url: str) -> Dict: jwt_token = self._get_jwt_token() response = requests.post( f"{self.endpoint}/api/ext/youtube/transcript", headers={ "Authorization": f"Bearer {jwt_token}", "Content-Type": "application/json" }, json={"url": url}, # ← Body as object, AnythingLLM now handles correctly timeout=60 ) return response.json() ``` **Website Scraping Method:** ```python def scrape_website(self, url: str, depth: int = 1, max_pages: int = 20) -> Dict: jwt_token = self._get_jwt_token() response = requests.post( f"{self.endpoint}/api/ext/website-depth", headers={ "Authorization": f"Bearer {jwt_token}", "Content-Type": "application/json" }, json={ "url": url, "depth": depth, "maxLinks": max_pages }, # ← Body as object, AnythingLLM now handles correctly timeout=120 ) return response.json() ``` ### Frontend Integration **Component:** `frontend/src/components/DataConnectors.jsx` **Features:** - YouTube URL input with validation - Website URL input with crawl depth and max pages settings - Real-time import status - Error handling and user feedback - Integration with RAG document management --- ## Commit Details ### Our Fork Commit **Repository:** https://github.com/DocterJac/anything-llm **Branch:** master **Commit:** 407e0192 **Date:** November 3, 2024 **Commit Message:** ``` Fix: Properly stringify body in forwardExtensionRequest Bug: The forwardExtensionRequest function was not consistently handling body serialization, which caused issues with the X-Integrity signature when the body was passed as an object instead of a string. Fix: Added explicit body stringification check and use the stringified version for both the request body and X-Integrity signature calculation. This ensures the signature is always calculated on the same data that gets sent in the request body, preventing signature mismatch errors. Pattern matches other methods in the same file (e.g., processRawText, processLink) which already do proper stringification. ``` ---
Author
Owner

@timothycarambat commented on GitHub (Nov 21, 2025):

Called AnythingLLM's /api/ext/youtube/transcript endpoint

Yeah, this is why - you are overloading the frontend endpoints which are specific to our UI and not for general developer use. You were probably passing Content-Type with it which was casuing the issue since in our UI we do the stringification before reaching that code path.

Either way, added the patch just for isolation and sanity checking

@timothycarambat commented on GitHub (Nov 21, 2025): > Called AnythingLLM's /api/ext/youtube/transcript endpoint Yeah, this is why - you are overloading the frontend endpoints which are specific to our UI and not for general developer use. You were probably passing Content-Type with it which was casuing the issue since in our UI we do the stringification before reaching that code path. Either way, added the patch just for isolation and sanity checking
yindo changed title from [BUG]: forwardExtensionRequest causes X-Integrity signature mismatch to [GH-ISSUE #4611] [BUG]: forwardExtensionRequest causes X-Integrity signature mismatch 2026-06-05 14:49:21 -04:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2932