how to load local modell??????????? #28

New Issue

2026-02-16T00:15:15-05:00

yindo commented

2026-02-16 00:15:15 -05:00

Originally created by @khalilxg on GitHub (Dec 24, 2023).

yindo commented

2026-02-16 00:15:16 -05:00

@0xM4sk commented on GitHub (Apr 24, 2024):

modified core/builder_config.py ---

from llama_index.llms.openai_like import OpenAILike

API_KEY = os.getenv('OPENAI_API_KEY')
BUILDER_LLM = OpenAILike(
api_base="[IP]:1337",
model="[model ID]",
is_chat_model=True,
max_tokens=None,
api_version="v1",
api_key=API_KEY,
)

using this method I was able to perform inference against local models hosted by Jan. Unfortunately my TensorRT Mistral model had streaming issues but I got other models working partially. .streamlit/secrets.toml does seem to need a valid openai API key, im not seeing any usage, but worth noting.

@0xM4sk commented on GitHub (Apr 24, 2024): modified core/builder_config.py --- from llama_index.llms.openai_like import OpenAILike API_KEY = os.getenv('OPENAI_API_KEY') BUILDER_LLM = OpenAILike( api_base="[IP]:1337", model="[model ID]", is_chat_model=True, max_tokens=None, api_version="v1", api_key=API_KEY, ) using this method I was able to perform inference against local models hosted by Jan. Unfortunately my TensorRT Mistral model had streaming issues but I got other models working partially. .streamlit/secrets.toml does seem to need a valid openai API key, im not seeing any usage, but worth noting.

yindo referenced this issue

2026-02-16 00:15:22 -05:00

[PR #28] [CLOSED] I synched my fork to the latest upgrade and refactored _resolve_llm #56

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: run-llama/rags#28