Bagatur/openllm ensure available (#6960)

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com> Co-authored-by: Aaron <29749331+aarnphm@users.noreply.github.com>
2026-07-19 21:33:31 -04:00 · 2023-06-30 00:54:23 -07:00
parent 521c6f0233
commit 60b0d6ea35
3 changed files with 539 additions and 31 deletions
@@ -166,11 +166,13 @@ class OpenLLM(LLM):
            # Runner with embedded when running the server. Instead, we will only set
            # the init_local here so that LangChain users can still use the LLM
            # in-process. Wrt to BentoML users, setting embedded=False is the expected
-            # behaviour to invoke the runners remotely
+            # behaviour to invoke the runners remotely.
+            # We need to also enable ensure_available to download and setup the model.
            runner = openllm.Runner(
                model_name=model_name,
                model_id=model_id,
                init_local=embedded,
+                ensure_available=True,
                **llm_kwargs,
            )
            super().__init__(