feat: azure openai endpoint support #2

Originally created by @gregrobison on GitHub (Jan 9, 2024).

Is your feature request related to a problem? Please describe.
I am unable to get my Azure OpenAI API endpoint to be recognized - are there any additional settings needed?

Describe the solution you'd like
Azure OpenAI endpoint cannot connect.

Describe alternatives you've considered
API works in other environments.

Additional context

Originally created by @gregrobison on GitHub (Jan 9, 2024). **Is your feature request related to a problem? Please describe.** I am unable to get my Azure OpenAI API endpoint to be recognized - are there any additional settings needed? **Describe the solution you'd like** Azure OpenAI endpoint cannot connect. **Describe alternatives you've considered** API works in other environments. **Additional context**

yindo added the enhancement good first issue labels 2026-02-15 18:17:14 -05:00

yindo closed this issue

yindo commented

@justinh-rahb commented on GitHub (Jan 9, 2024):

If I had to guess, it's likely that ollama-webui needs to be able to send the deployment_id and api_version in the requests for Azure endpoints, which it currently does not have settings variables for.

@justinh-rahb commented on GitHub (Jan 9, 2024): If I had to guess, it's likely that ollama-webui needs to be able to send the `deployment_id` and `api_version` in the requests for Azure endpoints, which it currently does not have settings variables for.

yindo commented

@justinh-rahb commented on GitHub (Jan 9, 2024):

One workaround you could do for the time being is using LiteLLM proxy to connect to your Azure endpoint, and then connect ollama-webui to litellm. Here's their documentation on setting up Azure endpoints: https://litellm.vercel.app/docs/providers/azure

And here's an example docker-compose.yaml I use to deploy both ollama-webui and litellm together in a stack:

version: '3.9'

services:
  webui:
    image: ghcr.io/ollama-webui/ollama-webui:main
    environment:
      - "OLLAMA_API_BASE_URL=${OLLAMA_API_BASE_URL}"
      - "OPENAI_API_BASE_URL=http://openai-proxy:8000/v1"
      - "OPENAI_API_KEY=${LITELLM_API_KEY}"
    ports:
      - 3000:8080
    volumes:
      - ./ollama-webui/data:/app/backend/data
    restart: unless-stopped

  openai-proxy:
    image: ghcr.io/berriai/litellm:main-latest
    environment:
      - "MASTER_KEY=${LITELLM_API_KEY}"
      - "OPENAI_API_KEY=${OPENAI_API_KEY}"
      - "MISTRAL_API_KEY=${MISTRAL_API_KEY}"
      - "ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}"
    ports:
      - 4000:8000
    volumes:
      - ./litellm/config.yaml:/app/config.yaml
    command: [ "--config", "/app/config.yaml", "--port", "8000" ]
    restart: unless-stopped

@justinh-rahb commented on GitHub (Jan 9, 2024): One workaround you could do for the time being is using [LiteLLM](https://github.com/BerriAI/litellm) proxy to connect to your Azure endpoint, and then connect ollama-webui to litellm. Here's their documentation on setting up Azure endpoints: https://litellm.vercel.app/docs/providers/azure And here's an example `docker-compose.yaml` I use to deploy both ollama-webui and litellm together in a stack: ``` version: '3.9' services: webui: image: ghcr.io/ollama-webui/ollama-webui:main environment: - "OLLAMA_API_BASE_URL=${OLLAMA_API_BASE_URL}" - "OPENAI_API_BASE_URL=http://openai-proxy:8000/v1" - "OPENAI_API_KEY=${LITELLM_API_KEY}" ports: - 3000:8080 volumes: - ./ollama-webui/data:/app/backend/data restart: unless-stopped openai-proxy: image: ghcr.io/berriai/litellm:main-latest environment: - "MASTER_KEY=${LITELLM_API_KEY}" - "OPENAI_API_KEY=${OPENAI_API_KEY}" - "MISTRAL_API_KEY=${MISTRAL_API_KEY}" - "ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}" ports: - 4000:8000 volumes: - ./litellm/config.yaml:/app/config.yaml command: [ "--config", "/app/config.yaml", "--port", "8000" ] restart: unless-stopped ```

yindo commented

@anuraagdjain commented on GitHub (Jan 28, 2024):

@tjbck - Is the idea to provide support for other LLM endpoints via LiteLLM in the webUI or have an explicit option for providing Azure OpenAI endpoint + key ?

@anuraagdjain commented on GitHub (Jan 28, 2024): @tjbck - Is the idea to provide support for other LLM endpoints via LiteLLM in the webUI or have an explicit option for providing Azure OpenAI endpoint + key ?

yindo commented

@isaacblum commented on GitHub (Feb 6, 2024):

I created this Ansible Playbook to address this exact need. Please take a look: https://github.com/SpaceTerran/ollama-webui_and_liteLLM-proxy

@isaacblum commented on GitHub (Feb 6, 2024): I created this Ansible Playbook to address this exact need. Please take a look: https://github.com/SpaceTerran/ollama-webui_and_liteLLM-proxy

yindo commented

@mateuszdrab commented on GitHub (Mar 20, 2024):

I think a quick solution (temporary) could also be hardcoding the API version which gets added if the url contains the Azure domain.
This would skip the need to update the UI and probably can be done with a few lines of code.

@mateuszdrab commented on GitHub (Mar 20, 2024): I think a quick solution (temporary) could also be hardcoding the API version which gets added if the url contains the Azure domain. This would skip the need to update the UI and probably can be done with a few lines of code.

yindo commented

@justinh-rahb commented on GitHub (Mar 20, 2024):

I think a quick solution (temporary) could also be hardcoding the API version which gets added if the url contains the Azure domain.
This would skip the need to update the UI and probably can be done with a few lines of code.

Personally I wouldn't be against doing this in my own code but I think we can strive for higher here. If it's to be modified, is it much more additional work to do it right? Because I can see the issue reports now from that one guy that uses a different API version 😆

@justinh-rahb commented on GitHub (Mar 20, 2024): > I think a quick solution (temporary) could also be hardcoding the API version which gets added if the url contains the Azure domain. > This would skip the need to update the UI and probably can be done with a few lines of code. Personally I wouldn't be against doing this in my own code but I think we can strive for higher here. If it's to be modified, is it much more additional work to do it right? Because I can see the issue reports now from that one guy that uses a different API version 😆

yindo commented

@Zulban commented on GitHub (Apr 16, 2024):

I thought I'd write here in case others find it useful.

I was able to get Open WebUI (v0.1.118) to use Azure OpenAI, gpt4. I did this through litellm (1.35.5), my docker compose file is just a combination of the four services: openwebui, postgresql, ollama, litellm. However one important gotcha is your litellm deployment cannot have a "LITELLM_MASTER_KEY" defined. Took me awhile to figure out. I'm still not sure how to get Open WebUI to use litellm with a LITELLM_MASTER_KEY. I'm still trying to determine if this problem is:

a bug, or
a missing feature, or
my mistake somewhere

@Zulban commented on GitHub (Apr 16, 2024): I thought I'd write here in case others find it useful. I was able to get Open WebUI (v0.1.118) to use Azure OpenAI, gpt4. I did this through litellm (1.35.5), my docker compose file is just a combination of the four services: openwebui, postgresql, ollama, litellm. However one important gotcha is your litellm deployment cannot have a "LITELLM_MASTER_KEY" defined. Took me awhile to figure out. I'm still not sure how to get Open WebUI to use litellm with a LITELLM_MASTER_KEY. I'm still trying to determine if this problem is: * a bug, or * a missing feature, or * my mistake somewhere

yindo commented

@Michelklingler commented on GitHub (Apr 19, 2024):

Same for me, I use OpenAI on azure for a while now. No issue at all with the liteLLM integration.

@Michelklingler commented on GitHub (Apr 19, 2024): Same for me, I use OpenAI on azure for a while now. No issue at all with the liteLLM integration.

yindo commented

@imbible commented on GitHub (May 5, 2024):

@Zulban @Michelklingler How do you make Azure Dall-E 3 work via LiteLLM in WebUI? The Image setting does not provide an input field to specify the model.

@imbible commented on GitHub (May 5, 2024): @Zulban @Michelklingler How do you make Azure Dall-E 3 work via LiteLLM in WebUI? The Image setting does not provide an input field to specify the model.

yindo commented

@jakoberpf commented on GitHub (May 6, 2024):

@Michelklingler I am currently trying to integrate some Azure OpenAI models with the liteLLM integration but somehow does not get it to work. Does my litellm config looks like this, but I am probably missing something. How did you get it to work?

model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/test-gpt-4-turbo
      api_base: https://<azure-open-ai-service>.openai.azure.com/
      api_key: "<redacted>"

@jakoberpf commented on GitHub (May 6, 2024): @Michelklingler I am currently trying to integrate some Azure OpenAI models with the liteLLM integration but somehow does not get it to work. Does my litellm config looks like this, but I am probably missing something. How did you get it to work? ```yaml model_list: - model_name: gpt-4 litellm_params: model: azure/test-gpt-4-turbo api_base: https://<azure-open-ai-service>.openai.azure.com/ api_key: "<redacted>" ```

yindo commented

@Michelklingler commented on GitHub (May 6, 2024):

@Michelklingler I am currently trying to integrate some Azure OpenAI models with the liteLLM integration but somehow does not get it to work. Does my litellm config looks like this, but I am probably missing something. How did you get it to work?
model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/test-gpt-4-turbo
      api_base: https://<azure-open-ai-service>.openai.azure.com/
      api_key: "<redacted>"

Hey @jakoberpf !

Below is my config, works like a charm with Mistral-large and GPT4-Turbo

model_list:
- litellm_params:
    api_base: https://OpenAiEndpoint.openai.azure.com/
    api_key: xxx
    model: azure/gpt4turbo
  model_info:
    id: f609d5af-7d43-45dc-81dd-b9c26212e259
  model_name: "Azure - GPT4-Turbo"
- litellm_params:
    api_base: https://Mistral-large-serverless.eastus2.inference.ai.azure.com/v1
    api_key: xxx
    model: mistral/Mistral-large
  model_info:
    id: ccaa5796-64f3-4691-b4a6-0598d2450ec7
  model_name: "Azure - Mistral-large"

I slightly change the link of the endpoint and removed the API key, but should work following this template!

Let me know if you get it work!
M.

@Michelklingler commented on GitHub (May 6, 2024): > @Michelklingler I am currently trying to integrate some Azure OpenAI models with the liteLLM integration but somehow does not get it to work. Does my litellm config looks like this, but I am probably missing something. How did you get it to work? > > ```yaml > model_list: > - model_name: gpt-4 > litellm_params: > model: azure/test-gpt-4-turbo > api_base: https://<azure-open-ai-service>.openai.azure.com/ > api_key: "<redacted>" > ``` Hey @jakoberpf ! Below is my config, works like a charm with Mistral-large and GPT4-Turbo ``` model_list: - litellm_params: api_base: https://OpenAiEndpoint.openai.azure.com/ api_key: xxx model: azure/gpt4turbo model_info: id: f609d5af-7d43-45dc-81dd-b9c26212e259 model_name: "Azure - GPT4-Turbo" - litellm_params: api_base: https://Mistral-large-serverless.eastus2.inference.ai.azure.com/v1 api_key: xxx model: mistral/Mistral-large model_info: id: ccaa5796-64f3-4691-b4a6-0598d2450ec7 model_name: "Azure - Mistral-large" ``` I slightly change the link of the endpoint and removed the API key, but should work following this template! Let me know if you get it work! M.

yindo commented

@Michelklingler commented on GitHub (May 6, 2024):

@Zulban @Michelklingler How do you make Azure Dall-E 3 work via LiteLLM in WebUI? The Image setting does not provide an input field to specify the model.

Mmm this is actually a good, point, not sure that at this point LiteLLM as it is configured can manage pulling an image from an endpoint, this is above my understanding of the code. But I would be interested to know as well.

@Michelklingler commented on GitHub (May 6, 2024): > @Zulban @Michelklingler How do you make Azure Dall-E 3 work via LiteLLM in WebUI? The Image setting does not provide an input field to specify the model. Mmm this is actually a good, point, not sure that at this point LiteLLM as it is configured can manage pulling an image from an endpoint, this is above my understanding of the code. But I would be interested to know as well.

yindo commented

@jakoberpf commented on GitHub (May 7, 2024):

@Michelklingler This worked great thank you. Do you know what the model_info is actually referencing?

@jakoberpf commented on GitHub (May 7, 2024): @Michelklingler This worked great thank you. Do you know what the `model_info ` is actually referencing?

yindo commented

@gelbzucht commented on GitHub (May 16, 2024):

thx @Michelklingler, that works really well. 👍🏻

@gelbzucht commented on GitHub (May 16, 2024): thx @Michelklingler, that works really well. 👍🏻

yindo commented

@gelbzucht commented on GitHub (May 16, 2024):

thx @Michelklingler, that works really well. 👍🏻

@gelbzucht commented on GitHub (May 16, 2024): thx @Michelklingler, that works really well. 👍🏻

yindo commented

@dino1729 commented on GitHub (May 24, 2024):

@Michelklingler This worked great thank you. Do you know what the model_info is actually referencing?

@Michelklingler Can you let us know where to get the model_id param?

@dino1729 commented on GitHub (May 24, 2024): > @Michelklingler This worked great thank you. Do you know what the `model_info ` is actually referencing? @Michelklingler Can you let us know where to get the model_id param?

yindo commented

@Michelklingler commented on GitHub (May 25, 2024):

@Michelklingler This worked great thank you. Do you know what the model_info is actually referencing?

@Michelklingler Can you let us know where to get the model_id param?

Hi!

I'm not actually completely sure what the model info is coming from. I think it is a value that auto populate when I use the WebUI interface to add new model.

For some reason I think it is a value that is automatically generated by liteLLM. I think as long as this value is different for every liteLLM model it's fine it's only for LiteLLM to identify each model internally.

It's my assumption. Let me know if it works!

@Michelklingler commented on GitHub (May 25, 2024): > > @Michelklingler This worked great thank you. Do you know what the `model_info ` is actually referencing? > > > > @Michelklingler Can you let us know where to get the model_id param? Hi! I'm not actually completely sure what the model info is coming from. I think it is a value that auto populate when I use the WebUI interface to add new model. For some reason I think it is a value that is automatically generated by liteLLM. I think as long as this value is different for every liteLLM model it's fine it's only for LiteLLM to identify each model internally. It's my assumption. Let me know if it works!

yindo commented

@0xCodeGuruX commented on GitHub (May 26, 2024):

I successfully used the project at https://github.com/haibbo/cf-openai-azure-proxy to convert Azure OpenAI to OpenAI, and it has been functioning perfectly.

@0xCodeGuruX commented on GitHub (May 26, 2024): I successfully used the project at https://github.com/haibbo/cf-openai-azure-proxy to convert Azure OpenAI to OpenAI, and it has been functioning perfectly.

yindo commented

@tjbck commented on GitHub (Jun 2, 2024):

Implemented as Pipeline: https://github.com/open-webui/pipelines/blob/main/examples/providers/azure_openai_pipeline.py

@tjbck commented on GitHub (Jun 2, 2024): Implemented as Pipeline: https://github.com/open-webui/pipelines/blob/main/examples/providers/azure_openai_pipeline.py

yindo commented

@kentsuiGitHub commented on GitHub (Jun 4, 2024):

tried the pipeline and find hit error 500. Further checked the request body JSON is not match with Azure OpenAI ChatCompletion API schema.

@kentsuiGitHub commented on GitHub (Jun 4, 2024): tried the pipeline and find hit error 500. Further checked the request body JSON is not match with Azure OpenAI ChatCompletion API schema.

yindo commented

@TDaubignyElsan commented on GitHub (Jun 6, 2024):

tried the pipeline and find hit error 500. Further checked the request body JSON is not match with Azure OpenAI ChatCompletion API schema.

Same here, is there a way to fix it ?

@TDaubignyElsan commented on GitHub (Jun 6, 2024): > tried the pipeline and find hit error 500. Further checked the request body JSON is not match with Azure OpenAI ChatCompletion API schema. Same here, is there a way to fix it ?

yindo commented

@tjbck commented on GitHub (Jun 6, 2024):

@TDaubignyElsan @kentsuiGitHub I don't use Azure myself (I don't have the keys to test it) so PR welcome!

@tjbck commented on GitHub (Jun 6, 2024): @TDaubignyElsan @kentsuiGitHub I don't use Azure myself (I don't have the keys to test it) so PR welcome!

yindo commented

@TDaubignyElsan commented on GitHub (Jun 6, 2024):

@TDaubignyElsan @kentsuiGitHub I don't use Azure myself (I don't have the keys to test it) so PR welcome!

Related documentation is located just here : https://learn.microsoft.com/en-us/azure/ai-services/openai/reference if it can help ?

@TDaubignyElsan commented on GitHub (Jun 6, 2024): > @TDaubignyElsan @kentsuiGitHub I don't use Azure myself (I don't have the keys to test it) so PR welcome! Related documentation is located just here : https://learn.microsoft.com/en-us/azure/ai-services/openai/reference if it can help ?

yindo commented

@justinh-rahb commented on GitHub (Jun 6, 2024):

Related documentation is located just here : https://learn.microsoft.com/en-us/azure/ai-services/openai/reference if it can help ?

The issue isn't with finding documentation, it's the inability for the team to test it since we do not have access. We could be open to someone offering us a limited scope key via Discord PM to move this forward.

@justinh-rahb commented on GitHub (Jun 6, 2024): > Related documentation is located just here : https://learn.microsoft.com/en-us/azure/ai-services/openai/reference if it can help ? The issue isn't with finding documentation, it's the inability for the team to _test_ it since we do not have access. We could be open to someone offering us a limited scope key via Discord PM to move this forward.

yindo commented

@Gyarbij commented on GitHub (Jun 14, 2024):

Related documentation is located just here : https://learn.microsoft.com/en-us/azure/ai-services/openai/reference if it can help ?

The issue isn't with finding documentation, it's the inability for the team to test it since we do not have access. We could be open to someone offering us a limited scope key via Discord PM to move this forward.

I can help with that, if someone wants to shoot me dm. on discord you can also find me under Gyarbij.

@Gyarbij commented on GitHub (Jun 14, 2024): > > Related documentation is located just here : https://learn.microsoft.com/en-us/azure/ai-services/openai/reference if it can help ? > > The issue isn't with finding documentation, it's the inability for the team to _test_ it since we do not have access. We could be open to someone offering us a limited scope key via Discord PM to move this forward. > I can help with that, if someone wants to shoot me dm. on discord you can also find me under Gyarbij.

yindo commented