Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #360] LocalAI for Embeddings #208

New Issue
Closed
opened 2026-02-22 18:18:21 -05:00 by yindo · 9 comments
yindo commented 2026-02-22 18:18:21 -05:00
Owner
Copy Link
Copy Source

Originally created by @netandreus on GitHub (Nov 13, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/360

How can I use LocalAI for Embeddings? There is no option for this at Embedding Preference window.

localai_embeddings
Originally created by @netandreus on GitHub (Nov 13, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/360 How can I use LocalAI for Embeddings? There is no option for this at Embedding Preference window. <img width="666" alt="localai_embeddings" src="https://github.com/Mintplex-Labs/anything-llm/assets/313477/462bdf60-d2d3-4eba-bee2-c633c5449a50">
yindo added the feature request label 2026-02-22 18:18:21 -05:00
yindo closed this issue 2026-02-22 18:18:21 -05:00
yindo commented 2026-02-22 18:18:21 -05:00
Author
Owner
Copy Link
Copy Source

@tlandenberger commented on GitHub (Nov 13, 2023):

Hi @netandreus I've created a draft pull request with a working solution for my use case #361. Would love to have some feedback if that also solves your needs.

@tlandenberger commented on GitHub (Nov 13, 2023): Hi @netandreus I've created a draft pull request with a working solution for my use case #361. Would love to have some feedback if that also solves your needs.
yindo commented 2026-02-22 18:18:22 -05:00
Author
Owner
Copy Link
Copy Source

@netandreus commented on GitHub (Nov 13, 2023):

Hello! @tlandenberger ! Thank you for your efforts!
I'm just tried this branch and got an error when build. Maybe you forget to commit image ../../../media/llmprovider/localai.png?

Here is an output:

=> ERROR [anything build-stage 2/2] RUN cd ./frontend/ && yarn build && yarn cache clean                                                                                              1.4s
------
 > [anything build-stage 2/2] RUN cd ./frontend/ && yarn build && yarn cache clean:
0.277 yarn run v1.22.19
0.290 $ vite build
0.595 vite v4.3.9 building for production...
0.614 transforming...
1.351 ✓ 72 modules transformed.
1.351 ✓ built in 756ms
1.351 Could not resolve "../../../media/llmprovider/localai.png" from "src/pages/GeneralSettings/EmbeddingPreference/index.jsx"
1.351 file: /app/frontend/src/pages/GeneralSettings/EmbeddingPreference/index.jsx
1.352 error during build:
1.352 RollupError: Could not resolve "../../../media/llmprovider/localai.png" from "src/pages/GeneralSettings/EmbeddingPreference/index.jsx"
1.352     at error (file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:2213:30)
1.352     at ModuleLoader.handleInvalidResolvedId (file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:24215:24)
1.352     at file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:24177:26
1.372 error Command failed with exit code 1.
1.372 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
------
failed to solve: process "/bin/sh -c cd ./frontend/ && yarn build && yarn cache clean" did not complete successfully: exit code: 1
@netandreus commented on GitHub (Nov 13, 2023): Hello! @tlandenberger ! Thank you for your efforts! I'm just tried [this branch](https://github.com/Mintplex-Labs/anything-llm/pull/361) and got an error when build. Maybe you forget to commit image ```../../../media/llmprovider/localai.png```? Here is an output: ``` => ERROR [anything build-stage 2/2] RUN cd ./frontend/ && yarn build && yarn cache clean 1.4s ------ > [anything build-stage 2/2] RUN cd ./frontend/ && yarn build && yarn cache clean: 0.277 yarn run v1.22.19 0.290 $ vite build 0.595 vite v4.3.9 building for production... 0.614 transforming... 1.351 ✓ 72 modules transformed. 1.351 ✓ built in 756ms 1.351 Could not resolve "../../../media/llmprovider/localai.png" from "src/pages/GeneralSettings/EmbeddingPreference/index.jsx" 1.351 file: /app/frontend/src/pages/GeneralSettings/EmbeddingPreference/index.jsx 1.352 error during build: 1.352 RollupError: Could not resolve "../../../media/llmprovider/localai.png" from "src/pages/GeneralSettings/EmbeddingPreference/index.jsx" 1.352 at error (file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:2213:30) 1.352 at ModuleLoader.handleInvalidResolvedId (file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:24215:24) 1.352 at file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:24177:26 1.372 error Command failed with exit code 1. 1.372 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. ------ failed to solve: process "/bin/sh -c cd ./frontend/ && yarn build && yarn cache clean" did not complete successfully: exit code: 1 ```
yindo commented 2026-02-22 18:18:22 -05:00
Author
Owner
Copy Link
Copy Source

@tlandenberger commented on GitHub (Nov 13, 2023):

Thanks, I've added the missing image ✅

@tlandenberger commented on GitHub (Nov 13, 2023): Thanks, I've added the missing image ✅
yindo commented 2026-02-22 18:18:22 -05:00
Author
Owner
Copy Link
Copy Source

@netandreus commented on GitHub (Nov 13, 2023):

Just tried to build with this fix! Now it built successfull. But there are some new errors, during chatting through Anything. Seems like we need to define LLM model in the form near LocalAI address. Because now LocalAI try to search unexisted model:

Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder

Anything screenshot

Screenshot 2023-11-13 at 17 56 18

LocalAI logs

5:55PM DBG [whisper] Attempting to load
5:55PM DBG Loading model whisper from model-placeholder
5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder
5:55PM DBG Loading GRPC Model whisper: {backendString:whisper model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false}
5:55PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/whisper
5:55PM DBG GRPC Service for model-placeholder will be running at: '127.0.0.1:51095'
5:55PM DBG GRPC Service state dir: /var/folders/f9/1b1jz83s4ysfn9zfncbsb8y40000gn/T/go-processmanager1786773264
5:55PM DBG GRPC Service Started
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:51095: connect: connection refused"
5:55PM DBG GRPC(model-placeholder-127.0.0.1:51095): stderr 2023/11/13 17:55:51 gRPC Server listening at 127.0.0.1:51095
5:55PM DBG GRPC Service Ready
5:55PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:model-placeholder ContextSize:512 Seed:0 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:0 MainGPU: TensorSplit: Threads:4 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/Users/andrey/sandbox/llm/current/models/model-placeholder Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 Tokenizer: LoraBase: LoraAdapter: NoMulMatQ:false DraftModel: AudioPath: Quantization:}
5:55PM DBG [whisper] Fails: could not load model: rpc error: code = Unknown desc = stat /Users/andrey/sandbox/llm/current/models/model-placeholder: no such file or directory
5:55PM DBG [stablediffusion] Attempting to load
5:55PM DBG Loading model stablediffusion from model-placeholder
5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder
5:55PM DBG Loading GRPC Model stablediffusion: {backendString:stablediffusion model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false}
5:55PM DBG [stablediffusion] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
5:55PM DBG [piper] Attempting to load
5:55PM DBG Loading model piper from model-placeholder
5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder
5:55PM DBG Loading GRPC Model piper: {backendString:piper model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false}
5:55PM DBG [piper] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/piper. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
[127.0.0.1]:50930 500 - POST /v1/chat/completions
@netandreus commented on GitHub (Nov 13, 2023): Just tried to build with this fix! Now it built successfull. But there are some new errors, during chatting through ```Anything```. Seems like we need to define LLM model in the form near LocalAI address. Because now LocalAI try to search unexisted model: ``` Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder ``` **Anything screenshot** <img width="1314" alt="Screenshot 2023-11-13 at 17 56 18" src="https://github.com/Mintplex-Labs/anything-llm/assets/313477/5f4be6b0-3cbe-4425-ba9a-60dce99d518e"> **LocalAI logs** ``` 5:55PM DBG [whisper] Attempting to load 5:55PM DBG Loading model whisper from model-placeholder 5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder 5:55PM DBG Loading GRPC Model whisper: {backendString:whisper model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false} 5:55PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/whisper 5:55PM DBG GRPC Service for model-placeholder will be running at: '127.0.0.1:51095' 5:55PM DBG GRPC Service state dir: /var/folders/f9/1b1jz83s4ysfn9zfncbsb8y40000gn/T/go-processmanager1786773264 5:55PM DBG GRPC Service Started rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:51095: connect: connection refused" 5:55PM DBG GRPC(model-placeholder-127.0.0.1:51095): stderr 2023/11/13 17:55:51 gRPC Server listening at 127.0.0.1:51095 5:55PM DBG GRPC Service Ready 5:55PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:model-placeholder ContextSize:512 Seed:0 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:0 MainGPU: TensorSplit: Threads:4 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/Users/andrey/sandbox/llm/current/models/model-placeholder Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 Tokenizer: LoraBase: LoraAdapter: NoMulMatQ:false DraftModel: AudioPath: Quantization:} 5:55PM DBG [whisper] Fails: could not load model: rpc error: code = Unknown desc = stat /Users/andrey/sandbox/llm/current/models/model-placeholder: no such file or directory 5:55PM DBG [stablediffusion] Attempting to load 5:55PM DBG Loading model stablediffusion from model-placeholder 5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder 5:55PM DBG Loading GRPC Model stablediffusion: {backendString:stablediffusion model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false} 5:55PM DBG [stablediffusion] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS 5:55PM DBG [piper] Attempting to load 5:55PM DBG Loading model piper from model-placeholder 5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder 5:55PM DBG Loading GRPC Model piper: {backendString:piper model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false} 5:55PM DBG [piper] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/piper. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS [127.0.0.1]:50930 500 - POST /v1/chat/completions ```
yindo commented 2026-02-22 18:18:22 -05:00
Author
Owner
Copy Link
Copy Source

@tlandenberger commented on GitHub (Nov 13, 2023):

Currently, I set the model via the .env file (included it in the .env.example file inside the server directory):
EMBEDDING_MODEL_PREF='bert-embeddings'
If you start with docker you need to add it to the .env file inside the docker directory.

I like the idea of selecting the model in the UI. We could implement an endpoint to fetch the installed models inside LocalAI with openai.listModels().

@tlandenberger commented on GitHub (Nov 13, 2023): Currently, I set the model via the _.env_ file (included it in the _.env.example_ file inside the server directory): `EMBEDDING_MODEL_PREF='bert-embeddings'` If you start with docker you need to add it to the _.env_ file inside the docker directory. I like the idea of selecting the model in the UI. We could implement an endpoint to fetch the installed models inside LocalAI with `openai.listModels()`.
yindo commented 2026-02-22 18:18:22 -05:00
Author
Owner
Copy Link
Copy Source

@netandreus commented on GitHub (Nov 13, 2023):

I fill it with my model, and there is this error. Maybe somewhere in code model name is hardcoded to model-placeholder?

.env

SERVER_PORT=3001
CACHE_VECTORS="true"
VECTOR_DB="lancedb"
STORAGE_DIR="/app/server/storage"
UID='501'
GID='20'
EMBEDDING_BASE_PATH="http://host.docker.internal:8080/"
EMBEDDING_MODEL_PREF="text-embedding"
@netandreus commented on GitHub (Nov 13, 2023): I fill it with my model, and there is this error. Maybe somewhere in code model name is hardcoded to ```model-placeholder```? **.env** ``` SERVER_PORT=3001 CACHE_VECTORS="true" VECTOR_DB="lancedb" STORAGE_DIR="/app/server/storage" UID='501' GID='20' EMBEDDING_BASE_PATH="http://host.docker.internal:8080/" EMBEDDING_MODEL_PREF="text-embedding" ```
yindo commented 2026-02-22 18:18:22 -05:00
Author
Owner
Copy Link
Copy Source

@tlandenberger commented on GitHub (Nov 13, 2023):

I am not sure if this error is caused by the new embedding engine. The model-placeholder is something inside the LMStudio AiProvider so I assume your problem comes from a wrongly setup LMStudio connection.
Does it work when using LMStudio as LLMProvider with another embedding engine?

@tlandenberger commented on GitHub (Nov 13, 2023): I am not sure if this error is caused by the new embedding engine. The `model-placeholder` is something inside the LMStudio AiProvider so I assume your problem comes from a wrongly setup LMStudio connection. Does it work when using LMStudio as LLMProvider with another embedding engine?
yindo commented 2026-02-22 18:18:22 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Nov 14, 2023):

Resolved by https://github.com/Mintplex-Labs/anything-llm/pull/361

@timothycarambat commented on GitHub (Nov 14, 2023): Resolved by https://github.com/Mintplex-Labs/anything-llm/pull/361
yindo commented 2026-02-22 18:18:22 -05:00
Author
Owner
Copy Link
Copy Source

@JOduMonT commented on GitHub (Nov 9, 2024):

It would also be nice if Anything LLM would support LocalAI as provider for TTS.
For now, I see PiperTTS which is available in LocalAI.
image

Of course as a non-programmer; I believe the step is little. ;)

@JOduMonT commented on GitHub (Nov 9, 2024): It would also be nice if Anything LLM would support LocalAI as provider for TTS. For now, I see `PiperTTS` which is available in `LocalAI`. ![image](https://github.com/user-attachments/assets/6806824f-5687-4c64-97cd-a8c503b5965c) *Of course as a non-programmer; I believe the step is little. ;)*
yindo changed title from LocalAI for Embeddings to [GH-ISSUE #360] LocalAI for Embeddings 2026-06-05 14:34:04 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#208
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?