Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #300] Local embeddings with sentence transformers #175

New Issue
Closed
opened 2026-02-22 18:18:14 -05:00 by yindo · 7 comments
yindo commented 2026-02-22 18:18:14 -05:00
Owner
Copy Link
Copy Source

Originally created by @Ananderz on GitHub (Oct 27, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/300

Hi,

I saw that there are local LLM:s in the roadmap. It won't make much sense if you don't also use local sentecetransformers (MTEB) like instructor or e5.

I would suggest to have this looked into before local LLM:s because you could potentially use an API endpoint for local LLM:s

Originally created by @Ananderz on GitHub (Oct 27, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/300 Hi, I saw that there are local LLM:s in the roadmap. It won't make much sense if you don't also use local sentecetransformers (MTEB) like instructor or e5. I would suggest to have this looked into before local LLM:s because you could potentially use an API endpoint for local LLM:s
yindo added the enhancement label 2026-02-22 18:18:14 -05:00
yindo closed this issue 2026-02-22 18:18:14 -05:00
yindo commented 2026-02-22 18:18:14 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 27, 2023):

We would like to explore this option but will not block LLM inferencing with this, we will just have OpenAI embeddings be the only option for non-embedder services (all LLM options except OpenAI & Azure).

So phase one is any LLM selection for inferencing with no option for embedding engine selection.

Then we add more Embedding engine selection when we can find a normalized interface for it. It would be a massive pain to do sentence transformation and each embedding engine has a different API. Most are running HF models it seems

Do you know of any services that normalize the embedding API like LiteLLM does for LLM inferencing?

@timothycarambat commented on GitHub (Oct 27, 2023): We would like to explore this option but will not block LLM inferencing with this, we will just have OpenAI embeddings be the only option for non-embedder services (all LLM options except OpenAI & Azure). So phase one is any LLM selection for inferencing with no option for embedding engine selection. Then we add more Embedding engine selection when we can find a normalized interface for it. It would be a massive pain to do sentence transformation and each embedding engine has a different API. Most are running HF models it seems Do you know of any services that normalize the embedding API like LiteLLM does for LLM inferencing?
yindo commented 2026-02-22 18:18:14 -05:00
Author
Owner
Copy Link
Copy Source

@krrishdholakia commented on GitHub (Oct 30, 2023):

hey @timothycarambat litellm supports the huggingface text embedding inference api which should help here

@krrishdholakia commented on GitHub (Oct 30, 2023): hey @timothycarambat litellm supports the huggingface text embedding inference api which should help here
yindo commented 2026-02-22 18:18:14 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 30, 2023):

hey @timothycarambat litellm supports the huggingface text embedding inference api which should help here

Indeed, I saw this, but it also requires us to move our backend to python or run the proxy server to have that interface available

@timothycarambat commented on GitHub (Oct 30, 2023): > hey @timothycarambat litellm supports the huggingface text embedding inference api which should help here Indeed, I saw this, but it also requires us to move our backend to python or run the proxy server to have that interface available
yindo commented 2026-02-22 18:18:15 -05:00
Author
Owner
Copy Link
Copy Source

@krrishdholakia commented on GitHub (Oct 30, 2023):

Would just be an add to the docker-compose I believe:

version: '3.9'

name: anythingllm

networks:
  anything-llm:
    driver: bridge

services:
  anything-llm:
    container_name: anything-llm
    image: anything-llm:latest
    platform: linux/amd64
    build:
      context: ../.
      dockerfile: ./docker/Dockerfile
      args:
        ARG_UID: ${UID}
        ARG_GID: ${GID}
    volumes:
      - "../server/storage:/app/server/storage"
      - "../collector/hotdir/:/app/collector/hotdir"
      - "../collector/outputs/:/app/collector/outputs"
    user: "${UID}:${GID}"
    ports:
      - "3001:3001"
    env_file:
      - .env
    networks:
      - anything-llm
    extra_hosts:
      - "host.docker.internal:host-gateway"
litellm:
    image: ghcr.io/berriai/litellm:latest
    ports:
      - '8000:8000'
    environment:
      - PORT=8000
      - OPENAI_API_KEY=your-api-key
@krrishdholakia commented on GitHub (Oct 30, 2023): Would just be an add to the docker-compose I believe: ```Dockerfile version: '3.9' name: anythingllm networks: anything-llm: driver: bridge services: anything-llm: container_name: anything-llm image: anything-llm:latest platform: linux/amd64 build: context: ../. dockerfile: ./docker/Dockerfile args: ARG_UID: ${UID} ARG_GID: ${GID} volumes: - "../server/storage:/app/server/storage" - "../collector/hotdir/:/app/collector/hotdir" - "../collector/outputs/:/app/collector/outputs" user: "${UID}:${GID}" ports: - "3001:3001" env_file: - .env networks: - anything-llm extra_hosts: - "host.docker.internal:host-gateway" litellm: image: ghcr.io/berriai/litellm:latest ports: - '8000:8000' environment: - PORT=8000 - OPENAI_API_KEY=your-api-key ```
yindo commented 2026-02-22 18:18:15 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Nov 14, 2023):

resolved by https://github.com/Mintplex-Labs/anything-llm/pull/361

@timothycarambat commented on GitHub (Nov 14, 2023): resolved by https://github.com/Mintplex-Labs/anything-llm/pull/361
yindo changed title from Local embeddings with sentence transformers to [GH-ISSUE #300] Local embeddings with sentence transformers 2026-06-05 14:33:53 -04:00
yindo commented 2026-06-05 14:33:54 -04:00
Author
Owner
Copy Link
Copy Source

@ghost commented on GitHub (Oct 30, 2023):

hey @timothycarambat litellm supports the huggingface text embedding inference api which should help here

<!-- gh-comment-id:1786004206 --> @ghost commented on GitHub (Oct 30, 2023): hey @timothycarambat litellm supports the huggingface text embedding inference api which should help here
yindo commented 2026-06-05 14:33:54 -04:00
Author
Owner
Copy Link
Copy Source

@ghost commented on GitHub (Oct 30, 2023):

Would just be an add to the docker-compose I believe:

version: '3.9'

name: anythingllm

networks:
  anything-llm:
    driver: bridge

services:
  anything-llm:
    container_name: anything-llm
    image: anything-llm:latest
    platform: linux/amd64
    build:
      context: ../.
      dockerfile: ./docker/Dockerfile
      args:
        ARG_UID: ${UID}
        ARG_GID: ${GID}
    volumes:
      - "../server/storage:/app/server/storage"
      - "../collector/hotdir/:/app/collector/hotdir"
      - "../collector/outputs/:/app/collector/outputs"
    user: "${UID}:${GID}"
    ports:
      - "3001:3001"
    env_file:
      - .env
    networks:
      - anything-llm
    extra_hosts:
      - "host.docker.internal:host-gateway"
litellm:
    image: ghcr.io/berriai/litellm:latest
    ports:
      - '8000:8000'
    environment:
      - PORT=8000
      - OPENAI_API_KEY=your-api-key
<!-- gh-comment-id:1786222013 --> @ghost commented on GitHub (Oct 30, 2023): Would just be an add to the docker-compose I believe: ```Dockerfile version: '3.9' name: anythingllm networks: anything-llm: driver: bridge services: anything-llm: container_name: anything-llm image: anything-llm:latest platform: linux/amd64 build: context: ../. dockerfile: ./docker/Dockerfile args: ARG_UID: ${UID} ARG_GID: ${GID} volumes: - "../server/storage:/app/server/storage" - "../collector/hotdir/:/app/collector/hotdir" - "../collector/outputs/:/app/collector/outputs" user: "${UID}:${GID}" ports: - "3001:3001" env_file: - .env networks: - anything-llm extra_hosts: - "host.docker.internal:host-gateway" litellm: image: ghcr.io/berriai/litellm:latest ports: - '8000:8000' environment: - PORT=8000 - OPENAI_API_KEY=your-api-key ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#175
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?