Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1153] [FEAT]: Integration of Vllm as model server #706

New Issue
Closed
opened 2026-02-22 18:20:55 -05:00 by yindo · 11 comments
yindo commented 2026-02-22 18:20:55 -05:00
Owner
Copy Link
Copy Source

Originally created by @flefevre on GitHub (Apr 20, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1153

What would you like to see?

it would be great to be able to configure AnythingLLM with a Vllm model
https://github.com/vllm-project/vllm

Originally created by @flefevre on GitHub (Apr 20, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1153 ### What would you like to see? it would be great to be able to configure AnythingLLM with a Vllm model https://github.com/vllm-project/vllm
yindo added the enhancementIntegration Requestfeature request labels 2026-02-22 18:20:55 -05:00
yindo closed this issue 2026-02-22 18:20:55 -05:00
yindo commented 2026-02-22 18:20:55 -05:00
Author
Owner
Copy Link
Copy Source

@mkhludnev commented on GitHub (Apr 24, 2024):

I were able to use vllm selecting Local AI in AnythingLLM LLM Settings.
Enjoy.

@mkhludnev commented on GitHub (Apr 24, 2024): I were able to use vllm selecting Local AI in AnythingLLM LLM Settings. Enjoy.
yindo commented 2026-02-22 18:20:55 -05:00
Author
Owner
Copy Link
Copy Source

@flefevre commented on GitHub (Apr 26, 2024):

Thanks for your advice.

I have tested but i failed.
I do confirm vllm instance is working fine since
https://vllm-mixtral.myserver.fr/v1/models

{"object":"list","data":[{"id":"mistralai/Mixtral-8x7B-Instruct-v0.1","object":"model","created":1714112327,"owned_by":"vllm","root":"mistralai/Mixtral-8x7B-Instruct-v0.1","parent":null,"permission":[{"id":"modelperm-76d249bf4f0e44698e3bb82a41424183","object":"model_permission","created":1714112327,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}

i have putted in the config: LocalAI with http://vllm-mistral:5002/v1
Anything is able to retrieve the model.

But when I tried to engage chat, i got an error:vCould not respond to message.

Request failed with status code 400

Looking to the log of Anythingllm , I ahev the following trace:

I would appreciate your help.
Thanks in advance.
Francois, from France


      _events: [Object: null prototype],

349
      _eventsCount: 1,

350
      _maxListeners: undefined,

351
      socket: [Socket],

352
      httpVersionMajor: 1,

353
      httpVersionMinor: 1,

354
      httpVersion: '1.1',

355
      complete: true,

356
      rawHeaders: [Array],

357
      rawTrailers: [],

358
      joinDuplicateHeaders: undefined,

359
      aborted: false,

360
      upgrade: false,

361
      url: '',

362
      method: null,

363
      statusCode: 400,

364
      statusMessage: 'Bad Request',

365
      client: [Socket],

366
      _consuming: false,

367
      _dumped: false,

368
      req: [ClientRequest],

369
      responseUrl: 'http://vllm-mixtral:5002/v1/chat/completions',

370
      redirects: [],

371
      [Symbol(kCapture)]: false,

372
      [Symbol(kHeaders)]: [Object],

373
      [Symbol(kHeadersCount)]: 10,

374
      [Symbol(kTrailers)]: null,

375
      [Symbol(kTrailersCount)]: 0

376
    }

377
  },

378
  isAxiosError: true,

379
  toJSON: [Function: toJSON]

380
}

On Wed, Apr 24, 2024 at 2:52 PM Mikhail Khludnev @.***>
wrote:

I were able to use vllm selecting Local AI in AnythingLLM LLM Settings.
Enjoy.

—
Reply to this email directly, view it on GitHub
https://github.com/Mintplex-Labs/anything-llm/issues/1153#issuecomment-2074877514,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKZRFGNVGLCW5HBLY7LT2DY66TI7AVCNFSM6AAAAABGQZ5LSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZUHA3TONJRGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

--

@flefevre commented on GitHub (Apr 26, 2024): Thanks for your advice. I have tested but i failed. I do confirm vllm instance is working fine since https://vllm-mixtral.myserver.fr/v1/models {"object":"list","data":[{"id":"mistralai/Mixtral-8x7B-Instruct-v0.1","object":"model","created":1714112327,"owned_by":"vllm","root":"mistralai/Mixtral-8x7B-Instruct-v0.1","parent":null,"permission":[{"id":"modelperm-76d249bf4f0e44698e3bb82a41424183","object":"model_permission","created":1714112327,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]} i have putted in the config: LocalAI with http://vllm-mistral:5002/v1 Anything is able to retrieve the model. But when I tried to engage chat, i got an error:vCould not respond to message. Request failed with status code 400 Looking to the log of Anythingllm , I ahev the following trace: I would appreciate your help. Thanks in advance. Francois, from France ``` _events: [Object: null prototype], 349 _eventsCount: 1, 350 _maxListeners: undefined, 351 socket: [Socket], 352 httpVersionMajor: 1, 353 httpVersionMinor: 1, 354 httpVersion: '1.1', 355 complete: true, 356 rawHeaders: [Array], 357 rawTrailers: [], 358 joinDuplicateHeaders: undefined, 359 aborted: false, 360 upgrade: false, 361 url: '', 362 method: null, 363 statusCode: 400, 364 statusMessage: 'Bad Request', 365 client: [Socket], 366 _consuming: false, 367 _dumped: false, 368 req: [ClientRequest], 369 responseUrl: 'http://vllm-mixtral:5002/v1/chat/completions', 370 redirects: [], 371 [Symbol(kCapture)]: false, 372 [Symbol(kHeaders)]: [Object], 373 [Symbol(kHeadersCount)]: 10, 374 [Symbol(kTrailers)]: null, 375 [Symbol(kTrailersCount)]: 0 376 } 377 }, 378 isAxiosError: true, 379 toJSON: [Function: toJSON] 380 } ``` On Wed, Apr 24, 2024 at 2:52 PM Mikhail Khludnev ***@***.***> wrote: > I were able to use vllm selecting Local AI in AnythingLLM LLM Settings. > Enjoy. > > — > Reply to this email directly, view it on GitHub > <https://github.com/Mintplex-Labs/anything-llm/issues/1153#issuecomment-2074877514>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABKZRFGNVGLCW5HBLY7LT2DY66TI7AVCNFSM6AAAAABGQZ5LSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZUHA3TONJRGQ> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > --
yindo commented 2026-02-22 18:20:56 -05:00
Author
Owner
Copy Link
Copy Source

@flefevre commented on GitHub (Apr 26, 2024):

If it could help you in your analysis:
it seems to ask for
https://vllm-mixtral.myserver.fr/v1/chat/completions

but when i go this url i got:
{"detail":"Method Not Allowed"}

it is a problem of api between vllm and anythingllm ?

@flefevre commented on GitHub (Apr 26, 2024): If it could help you in your analysis: it seems to ask for https://vllm-mixtral.myserver.fr/v1/chat/completions but when i go this url i got: `{"detail":"Method Not Allowed"}` it is a problem of api between vllm and anythingllm ?
yindo commented 2026-02-22 18:20:56 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 26, 2024):

That endpoint is POST only, not GET - which is part of the reason you got method not allowed when going to the URL directly.

@timothycarambat commented on GitHub (Apr 26, 2024): That endpoint is POST only, not GET - which is part of the reason you got method not allowed when going to the URL directly.
yindo commented 2026-02-22 18:20:56 -05:00
Author
Owner
Copy Link
Copy Source

@alceausu commented on GitHub (Apr 28, 2024):

I have the same issue, no mather the integration (Local AI or Generic OpenAI).
The vllm server replies with:
ERROR` serving_chat.py:60] Error in applying chat template from request: Conversation roles must alternate user/assistant/user/assistant/...
INFO:` "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

@alceausu commented on GitHub (Apr 28, 2024): I have the same issue, no mather the integration (Local AI or Generic OpenAI). The vllm server replies with: ``ERROR` serving_chat.py:60] Error in applying chat template from request: Conversation roles must alternate user/assistant/user/assistant/...`` ``INFO:` "POST /v1/chat/completions HTTP/1.1" 400 Bad Request``
yindo commented 2026-02-22 18:20:56 -05:00
Author
Owner
Copy Link
Copy Source

@flefevre commented on GitHub (Apr 28, 2024):

Dear all,
I have tested again and no way to use vllm directly even if I have exposed the port through the docker configuration.
Could you share exactly how did you do it?
Thanks again

@flefevre commented on GitHub (Apr 28, 2024): Dear all, I have tested again and no way to use vllm directly even if I have exposed the port through the docker configuration. Could you share exactly how did you do it? Thanks again
yindo commented 2026-02-22 18:20:56 -05:00
Author
Owner
Copy Link
Copy Source

@mkhludnev commented on GitHub (Apr 28, 2024):

@flefevre once again, I did it twice it works. Maybe it's a container connectivity issue? I remember that I use curl to check connectivity between containers. Can you tell how your containers,hosts and processes are aligned?
The simplest approach for me were launch vllm and AnythingLLM as a sibling containers under the single docker-compose.yml config and point anithingllm to vllm via container name.
Another approach was to run vllm in a host, then launch AnythingLLM via docker and point to vllm via host.docker.internal but iirc is a Docker Desktop only feature.

@mkhludnev commented on GitHub (Apr 28, 2024): @flefevre once again, I did it twice it works. Maybe it's a container connectivity issue? I remember that I use curl to check connectivity between containers. Can you tell how your containers,hosts and processes are aligned? The simplest approach for me were launch vllm and AnythingLLM as a sibling containers under the single docker-compose.yml config and point anithingllm to vllm via container name. Another approach was to run vllm in a host, then launch AnythingLLM via docker and point to vllm via `host.docker.internal` but iirc is a Docker Desktop only feature.
yindo commented 2026-02-22 18:20:56 -05:00
Author
Owner
Copy Link
Copy Source

@alceausu commented on GitHub (Apr 29, 2024):

It seems these are two different issues, one related to connectivity, and the other one on format.
Related to the request format, the anything-llm can reach vllm, but vllm throws an error of '400 Bad request'.
ERROR` serving_chat.py:60] Error in applying chat template from request: Conversation roles must alternate user/assistant/user/assistant/...
For some hints on why, see vllm discussion Mixtral instruct doesn't accept system prompt
Is there a way to modify the template on the anything-llm side?

@alceausu commented on GitHub (Apr 29, 2024): It seems these are two different issues, one related to connectivity, and the other one on format. Related to the request format, the anything-llm can reach vllm, but vllm throws an error of '400 Bad request'. ``ERROR` serving_chat.py:60] Error in applying chat template from request: Conversation roles must alternate user/assistant/user/assistant/...`` For some hints on why, see vllm discussion [Mixtral instruct doesn't accept system prompt](https://github.com/vllm-project/vllm/discussions/2112) Is there a way to modify the template on the anything-llm side?
yindo commented 2026-02-22 18:20:56 -05:00
Author
Owner
Copy Link
Copy Source

@flefevre commented on GitHub (May 1, 2024):

Dear all,
Normally i have simplified the test.

Docker configuration

  • Anythingllm a docker compose with the same network
  • Vllm a docker compose with the same network

Docker validation

When I connect to the anythingllm container, i am able to retrieve model of vllm through the command:

anythingllm@6de6c5255f33:~$ curl http://vllm-mixtral:5002/v1/completions -H "Content-Type: application/json" -d '{"model": "mistralai/Mixtral-8x7B-Instruct-v0.1", "prompt": "San Francisco is a", "max_tokens": 7,"temperature": 0}' {"id":"cmpl-0df1e0e95b4c46a78632936ba277e3ef","object":"text_completion","created":1714551853,"model":"mistralai/Mixtral-8x7B-Instruct-v0.1","choices":[{"index":0,"text":" city that is known for its steep","logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":5,"total_tokens":12,"completion_tokens":7}}anythingllm@6de6c5255f33:~$

Anythingllm Webui configuration
I am able to configure the default LLM preference by setting

Anythingllm Webui Test
When i create a workspace, open a new thread, and ask something, I got everytime a
Could not respond to message. Request failed with status code 400

When i look at the Anythingllm logs, i have the following trace

httpVersionMajor: 1, httpVersionMinor: 1, httpVersion: '1.1', complete: true, rawHeaders: [Array], rawTrailers: [], joinDuplicateHeaders: undefined, aborted: false, upgrade: false, url: '', method: null, statusCode: 400, statusMessage: 'Bad Request', client: [Socket], _consuming: false, _dumped: false, req: [ClientRequest], responseUrl: 'http://vllm-mixtral:5002/v1/chat/completions', redirects: [], [Symbol(kCapture)]: false, [Symbol(kHeaders)]: [Object], [Symbol(kHeadersCount)]: 10, [Symbol(kTrailers)]: null, [Symbol(kTrailersCount)]: 0 } }, isAxiosError: true, toJSON: [Function: toJSON]

Analysis
I agree with @alceausu , the problem seems not to come from a mis configuration of docker / vllm / anythingllm.
It seems more related to a misconfiguration between anlythingllm and vllm in the usage of a specific model which in my case mixtral8x7b.
A solution could be to be able to understand the specificity of each vllm/model system prompt, as proposed by @alceausu https://github.com/vllm-project/vllm/discussions/2112
Or perhaps to use a model proxy such as Litellm that will encapsulate the model interaction based on an uniform api which is inspired by openai.

I have created the following Feature proposal here #1154 , I do think it is the good solution.
Do you agree?

If yes, my ticket should perhaps invalidated since Anythingllm is compatible with vllm but not with all models served by vllm.
Mixtral8x7b is really a good model. It will be perfect to access to it through a proxy such as Litellm, ensuring the developer of Anythingllm do not have to adapt all their backend solution for prompting for each model.

Thanks for for your expertise.

@flefevre commented on GitHub (May 1, 2024): Dear all, Normally i have simplified the test. **Docker configuration** - Anythingllm a docker compose with the same network - Vllm a docker compose with the same network **Docker validation** When I connect to the anythingllm container, i am able to retrieve model of vllm through the command: `anythingllm@6de6c5255f33:~$ curl http://vllm-mixtral:5002/v1/completions -H "Content-Type: application/json" -d '{"model": "mistralai/Mixtral-8x7B-Instruct-v0.1", "prompt": "San Francisco is a", "max_tokens": 7,"temperature": 0}' {"id":"cmpl-0df1e0e95b4c46a78632936ba277e3ef","object":"text_completion","created":1714551853,"model":"mistralai/Mixtral-8x7B-Instruct-v0.1","choices":[{"index":0,"text":" city that is known for its steep","logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":5,"total_tokens":12,"completion_tokens":7}}anythingllm@6de6c5255f33:~$ ` **Anythingllm Webui configuration** I am able to configure the default LLM preference by setting - Local AI - http://vllm-mixtral:5002 >> it see the model and propose me mistralai/Mixtral-8x7B-Instruct-v0.1 **Anythingllm Webui Test** When i create a workspace, open a new thread, and ask something, I got everytime a ` Could not respond to message. Request failed with status code 400` When i look at the Anythingllm logs, i have the following trace ` httpVersionMajor: 1, httpVersionMinor: 1, httpVersion: '1.1', complete: true, rawHeaders: [Array], rawTrailers: [], joinDuplicateHeaders: undefined, aborted: false, upgrade: false, url: '', method: null, statusCode: 400, statusMessage: 'Bad Request', client: [Socket], _consuming: false, _dumped: false, req: [ClientRequest], responseUrl: 'http://vllm-mixtral:5002/v1/chat/completions', redirects: [], [Symbol(kCapture)]: false, [Symbol(kHeaders)]: [Object], [Symbol(kHeadersCount)]: 10, [Symbol(kTrailers)]: null, [Symbol(kTrailersCount)]: 0 } }, isAxiosError: true, toJSON: [Function: toJSON]` **Analysis** I agree with @alceausu , the problem seems not to come from a mis configuration of docker / vllm / anythingllm. It seems more related to a misconfiguration between anlythingllm and vllm in the usage of a specific model which in my case mixtral8x7b. A solution could be to be able to understand the specificity of each vllm/model system prompt, as proposed by @alceausu https://github.com/vllm-project/vllm/discussions/2112 Or perhaps to use a model proxy such as Litellm that will encapsulate the model interaction based on an uniform api which is inspired by openai. I have created the following Feature proposal here #1154 , I do think it is the good solution. Do you agree? If yes, my ticket should perhaps invalidated since Anythingllm is compatible with vllm but not with all models served by vllm. Mixtral8x7b is really a good model. It will be perfect to access to it through a proxy such as Litellm, ensuring the developer of Anythingllm do not have to adapt all their backend solution for prompting for each model. Thanks for for your expertise.
yindo commented 2026-02-22 18:20:56 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 10, 2025):

Closin as not planned for now since you can leverage the Generic OpenAI connector provider to service VLLM endpoints since they are OpenAI compat

@timothycarambat commented on GitHub (Jul 10, 2025): Closin as not planned for now since you can leverage the Generic OpenAI connector provider to service VLLM endpoints since they are OpenAI compat
yindo changed title from [FEAT]: Integration of Vllm as model server to [GH-ISSUE #1153] [FEAT]: Integration of Vllm as model server 2026-06-05 14:36:46 -04:00
yindo commented 2026-06-05 14:36:47 -04:00
Author
Owner
Copy Link
Copy Source

@mekler22 commented on GitHub (Apr 7, 2026):

Found the solution for this. You need to run seperate dockers for each one (vllm, and anythingllm), but they both need to share the same network. You may need to use sudo chown -R 1000:1000 /path/to/docker and sudo chmod -R 775 /path/to/docker to make sure the docker gets all the write rights it needs.
This is the anythingllm docker-compose (running as a portainer stack):
version: '3.8'

services:
anythingllm:
image: mintplexlabs/anythingllm:latest
container_name: anythingllm
ports:
- "3001:3001"
cap_add:
- SYS_ADMIN
environment:
- STORAGE_DIR=/app/server/storage
- JWT_SECRET=20characterssecretgenerated
- LLM_PROVIDER=generic-openai
- GENERIC_OPEN_AI_BASE_PATH=http://vllm:8000/v1
- GENERIC_OPEN_AI_MODEL_PREF=Qwen/Qwen3-8B-AWQ
- GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=8192
- GENERIC_OPEN_AI_API_KEY=sk-123abc
- EMBEDDING_ENGINE=ollama
- EMBEDDING_BASE_PATH=http://OLLAMA:11434
- EMBEDDING_MODEL_PREF=nomic-embed-text
- EMBEDDING_MODEL_MAX_CHUNK_LENGTH=8192
- VECTOR_DB=lancedb
- WHISPER_PROVIDER=local
- TTS_PROVIDER=native
- PASSWORDMINCHAR=8
#
volumes:
- /volume1/docker/anythingllm:/app/server/storage
restart: always
networks:
- ollama_default
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
ollama_default:
external: true

And this is the docker-compose for vllm (running as a portainer stack):
version: "3.9"

services:
vllm:
image: vllm/vllm-openai:v0.8.5
container_name: vllm
restart: always
ports:
- "8001:8000"
environment:
- HUGGING_FACE_HUB_TOKEN=hf_xxxxxx
- VLLM_ENABLE_CUDA_COMPATIBILITY=1
volumes:
- /volume1/docker/vllm/cache:/root/.cache/huggingface
ipc: host
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [compute,video,graphics,utility]
command: >
--model Qwen/Qwen3-8B-AWQ
--served-model-name Qwen/Qwen3-8B-AWQ
--enable-auto-tool-choice
--tool-call-parser hermes
--max-model-len 16384
--gpu-memory-utilization 0.85
--trust-remote-code
--enforce-eager
networks:
- ollama_default
extra_hosts:
- "host.docker.internal:host-gateway"

networks:
ollama_default:
external: true

This is engineered (through trial and error) for my own synology nas based server with RTX 3060 12g card, and driver limitation for Cuda 12.4 - that's why the vllm version is limited to 0.8.5 - as the newer versions are running with Cuda 13.0. Also, it limits which models you can use as some newer functions are not available so certain models simply will not run or require command parameters changes. Also notice that my embedding is running off my Ollama docker - so you may want to change that according to what you have. And of course, the relevant folders need to be created in advance. However, it all works great on my hardware. This is done with pieces of code that I found in vllm, and anythingllm related sites, with A LOT of tweaking.
I find that Vllm+Anythingllm is definitely faster in responding than Ollama+Openwebui. But.. with the former I can use latest images without issue, while with the latter I am more limited. Also downloading and switching between models is MUCH easier with Ollama+Openwebui.

Anyways, Enjoy! I hope it helps (and don't forget to enter your own HF token before running the stack).

<!-- gh-comment-id:4196928691 --> @mekler22 commented on GitHub (Apr 7, 2026): Found the solution for this. You need to run seperate dockers for each one (vllm, and anythingllm), but they both need to share the same network. You may need to use sudo chown -R 1000:1000 /path/to/docker and sudo chmod -R 775 /path/to/docker to make sure the docker gets all the write rights it needs. **This is the anythingllm docker-compose (running as a portainer stack):** version: '3.8' services: anythingllm: image: mintplexlabs/anythingllm:latest container_name: anythingllm ports: - "3001:3001" cap_add: - SYS_ADMIN environment: - STORAGE_DIR=/app/server/storage - JWT_SECRET=20characterssecretgenerated - LLM_PROVIDER=generic-openai - GENERIC_OPEN_AI_BASE_PATH=http://vllm:8000/v1 - GENERIC_OPEN_AI_MODEL_PREF=Qwen/Qwen3-8B-AWQ - GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=8192 - GENERIC_OPEN_AI_API_KEY=sk-123abc - EMBEDDING_ENGINE=ollama - EMBEDDING_BASE_PATH=http://OLLAMA:11434 - EMBEDDING_MODEL_PREF=nomic-embed-text - EMBEDDING_MODEL_MAX_CHUNK_LENGTH=8192 - VECTOR_DB=lancedb - WHISPER_PROVIDER=local - TTS_PROVIDER=native - PASSWORDMINCHAR=8 # volumes: - /volume1/docker/anythingllm:/app/server/storage restart: always networks: - ollama_default extra_hosts: - "host.docker.internal:host-gateway" networks: ollama_default: external: true **And this is the docker-compose for vllm (running as a portainer stack):** version: "3.9" services: vllm: image: vllm/vllm-openai:v0.8.5 container_name: vllm restart: always ports: - "8001:8000" environment: - HUGGING_FACE_HUB_TOKEN=hf_xxxxxx - VLLM_ENABLE_CUDA_COMPATIBILITY=1 volumes: - /volume1/docker/vllm/cache:/root/.cache/huggingface ipc: host deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [compute,video,graphics,utility] command: > --model Qwen/Qwen3-8B-AWQ --served-model-name Qwen/Qwen3-8B-AWQ --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 16384 --gpu-memory-utilization 0.85 --trust-remote-code --enforce-eager networks: - ollama_default extra_hosts: - "host.docker.internal:host-gateway" networks: ollama_default: external: true This is engineered (through trial and error) for my own synology nas based server with RTX 3060 12g card, and driver limitation for Cuda 12.4 - that's why the vllm version is limited to 0.8.5 - as the newer versions are running with Cuda 13.0. Also, it limits which models you can use as some newer functions are not available so certain models simply will not run or require command parameters changes. Also notice that my embedding is running off my Ollama docker - so you may want to change that according to what you have. And of course, the relevant folders need to be created in advance. However, it all works great on my hardware. This is done with pieces of code that I found in vllm, and anythingllm related sites, with A LOT of tweaking. I find that Vllm+Anythingllm is definitely faster in responding than Ollama+Openwebui. But.. with the former I can use latest images without issue, while with the latter I am more limited. Also downloading and switching between models is MUCH easier with Ollama+Openwebui. Anyways, Enjoy! I hope it helps (and don't forget to enter your own HF token before running the stack).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#706
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?