run-llama/create-llama
1
0
Fork 0
mirror of https://github.com/run-llama/create-llama.git synced 2026-07-15 05:08:15 -04:00
Code Issues 39 Packages Projects Releases 10 Wiki Activity

LLMAgent doesn't support Ollama (needs LITS update) #92

New Issue
Closed
opened 2026-02-15 19:15:50 -05:00 by yindo · 9 comments
yindo commented 2026-02-15 19:15:50 -05:00
Owner
Copy Link
Copy Source

Originally created by @joshuacox on GitHub (Oct 28, 2024).

Originally assigned to: @thucpn on GitHub.

How do I prevent openAI from being invoked at all?

[LlamaIndex] OpenAIError: The OPENAI_API_KEY environment variable is missing or empty; either provide it, or instantiate the OpenAI client with an apiKey option, like new OpenAI({ apiKey: 'My API Key' }).
    at new OpenAI (webpack-internal:///(rsc)/./node_modules/openai/index.mjs:69:19)
    at eval (webpack-internal:///(rsc)/./node_modules/@llamaindex/openai/dist/index.js:772:28)
    at async OpenAI.streamChat (webpack-internal:///(rsc)/./node_modules/@llamaindex/openai/dist/index.js:888:31)
    at async asyncGeneratorWrapper (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/decorator/dist/index.js:32:45)
    at async response.<computed> (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/decorator/dist/index.js:99:34)
    at async Object.start (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/agent/dist/index.js:35:30)
 POST /api/chat 500 in 3533ms
The data stream is hanging. Did you

.env

MODEL_PROVIDER=ollama
MODEL=llama3.2
EMBEDDING_MODEL="sentence-transformers/all-mpnet-base-v2"
EMBEDDING_DIM=1024
FILESERVER_URL_PREFIX=http://localhost:3000/api/files
NEXT_QUESTION_PROMPT="You're a helpful assistant! Your task is to suggest the next question that user might ask. 
Here is the conversation history
---------------------
{conversation}
---------------------
Given the conversation history, please give me 3 questions that you might ask next!
Your answer should be wrapped in three sticks which follows the following format:

<question 1>
<question 2>
<question 3>

SYSTEM_PROMPT="You are a DuckDuckGo search agent. 
You can use the duckduckgo search tool to get information from the web to answer user questions.
For better results, you can specify the region parameter to get results from a specific region but it's optional.
"
Originally created by @joshuacox on GitHub (Oct 28, 2024). Originally assigned to: @thucpn on GitHub. How do I prevent openAI from being invoked at all? ``` [LlamaIndex] OpenAIError: The OPENAI_API_KEY environment variable is missing or empty; either provide it, or instantiate the OpenAI client with an apiKey option, like new OpenAI({ apiKey: 'My API Key' }). at new OpenAI (webpack-internal:///(rsc)/./node_modules/openai/index.mjs:69:19) at eval (webpack-internal:///(rsc)/./node_modules/@llamaindex/openai/dist/index.js:772:28) at async OpenAI.streamChat (webpack-internal:///(rsc)/./node_modules/@llamaindex/openai/dist/index.js:888:31) at async asyncGeneratorWrapper (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/decorator/dist/index.js:32:45) at async response.<computed> (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/decorator/dist/index.js:99:34) at async Object.start (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/agent/dist/index.js:35:30) POST /api/chat 500 in 3533ms The data stream is hanging. Did you ``` .env ``` MODEL_PROVIDER=ollama MODEL=llama3.2 EMBEDDING_MODEL="sentence-transformers/all-mpnet-base-v2" EMBEDDING_DIM=1024 FILESERVER_URL_PREFIX=http://localhost:3000/api/files NEXT_QUESTION_PROMPT="You're a helpful assistant! Your task is to suggest the next question that user might ask. Here is the conversation history --------------------- {conversation} --------------------- Given the conversation history, please give me 3 questions that you might ask next! Your answer should be wrapped in three sticks which follows the following format: ``` <question 1> <question 2> <question 3> ```" SYSTEM_PROMPT="You are a DuckDuckGo search agent. You can use the duckduckgo search tool to get information from the web to answer user questions. For better results, you can specify the region parameter to get results from a specific region but it's optional. " ```
yindo closed this issue 2026-02-15 19:15:50 -05:00
yindo commented 2026-02-15 19:15:51 -05:00
Author
Owner
Copy Link
Copy Source

@marcusschiesser commented on GitHub (Oct 28, 2024):

@joshuacox we will upgrade llamaindex to fix this soon

@marcusschiesser commented on GitHub (Oct 28, 2024): @joshuacox we will upgrade llamaindex to fix this soon
yindo commented 2026-02-15 19:15:51 -05:00
Author
Owner
Copy Link
Copy Source

@joshuacox commented on GitHub (Oct 28, 2024):

I deleted all initializers aside from Ollama in settings.ts after taking this lines advice:

  // HINT: you can delete the initialization code for unused model providers

Which I did to no effect. It still complains about the missing API key. I tried setting a false key to get it to ignore the issue, but of course openAI rejects the API call. But I do not want it contacting openAI at all.

EDIT: of course I submitted that comment and then the refresh happened and I see the comment from you @marcusschiesser

@joshuacox commented on GitHub (Oct 28, 2024): I deleted all initializers aside from Ollama in settings.ts after taking this lines advice: ``` // HINT: you can delete the initialization code for unused model providers ``` Which I did to no effect. It still complains about the missing API key. I tried setting a false key to get it to ignore the issue, but of course openAI rejects the API call. But I do not want it contacting openAI at all. EDIT: of course I submitted that comment and then the refresh happened and I see the comment from you @marcusschiesser
yindo commented 2026-02-15 19:15:51 -05:00
Author
Owner
Copy Link
Copy Source

@mcavdar commented on GitHub (Oct 31, 2024):

hi @joshuacox
Can you please initialize with npx create-llama@latest --pro rather than npx create-llama@latest, and update .env? This works for me.

@mcavdar commented on GitHub (Oct 31, 2024): hi @joshuacox Can you please initialize with `npx create-llama@latest --pro` rather than `npx create-llama@latest`, and update .env? This works for me.
yindo commented 2026-02-15 19:15:51 -05:00
Author
Owner
Copy Link
Copy Source

@marcusschiesser commented on GitHub (Nov 1, 2024):

should be fixed in Release 0.3.10 - please have a try

@marcusschiesser commented on GitHub (Nov 1, 2024): should be fixed in Release 0.3.10 - please have a try
yindo commented 2026-02-15 19:15:52 -05:00
Author
Owner
Copy Link
Copy Source

@mcavdar commented on GitHub (Nov 1, 2024):

hi @marcusschiesser

Unfortunately, updating LlamaIndex to version 0.8.2 didn't resolve the issue. If it would be helpful, I can share the differences between the files when initialized with --pro versus without it.

without-pro/app/api/chat/engine:

import {
  BaseChatEngine,
  BaseToolWithCall,
  OpenAIAgent,
  QueryEngineTool,
} from "llamaindex";
import fs from "node:fs/promises";
import path from "node:path";
import { getDataSource } from "./index";
import { generateFilters } from "./queryFilter";
import { createTools } from "./tools";

export async function createChatEngine(documentIds?: string[], params?: any) {
  const tools: BaseToolWithCall[] = [];

  // Add a query engine tool if we have a data source
  // Delete this code if you don't have a data source
  const index = await getDataSource(params);
  if (index) {
    tools.push(
      new QueryEngineTool({
        queryEngine: index.asQueryEngine({
          preFilters: generateFilters(documentIds || []),
        }),
        metadata: {
          name: "data_query_engine",
          description: `A query engine for documents from your data source.`,
        },
      }),
    );
  }

  const configFile = path.join("config", "tools.json");
  let toolConfig: any;
  try {
    // add tools from config file if it exists
    toolConfig = JSON.parse(await fs.readFile(configFile, "utf8"));
  } catch (e) {
    console.info(`Could not read ${configFile} file. Using no tools.`);
  }
  if (toolConfig) {
    tools.push(...(await createTools(toolConfig)));
  }

  const agent = new OpenAIAgent({
    tools,
    systemPrompt: process.env.SYSTEM_PROMPT,
  }) as unknown as BaseChatEngine;

  return agent;
}

with-pro//app/api/chat/engine:

import { ContextChatEngine, Settings } from "llamaindex";
import { getDataSource } from "./index";
import { nodeCitationProcessor } from "./nodePostprocessors";
import { generateFilters } from "./queryFilter";

export async function createChatEngine(documentIds?: string[], params?: any) {
  const index = await getDataSource(params);
  if (!index) {
    throw new Error(
      `StorageContext is empty - call 'npm run generate' to generate the storage first`,
    );
  }
  const retriever = index.asRetriever({
    similarityTopK: process.env.TOP_K ? parseInt(process.env.TOP_K) : undefined,
    filters: generateFilters(documentIds || []),
  });

  const systemPrompt = process.env.SYSTEM_PROMPT;
  const citationPrompt = process.env.SYSTEM_CITATION_PROMPT;
  const prompt =
    [systemPrompt, citationPrompt].filter((p) => p).join("\n") || undefined;
  const nodePostprocessors = citationPrompt
    ? [nodeCitationProcessor]
    : undefined;

  return new ContextChatEngine({
    chatModel: Settings.llm,
    retriever,
    systemPrompt: prompt,
    nodePostprocessors,
  });
}

const agent = new OpenAIAgent I guess we need to fix this part.

@mcavdar commented on GitHub (Nov 1, 2024): hi @marcusschiesser Unfortunately, updating LlamaIndex to version 0.8.2 didn't resolve the issue. If it would be helpful, I can share the differences between the files when initialized with --pro versus without it. **without-pro/app/api/chat/engine:** ``` import { BaseChatEngine, BaseToolWithCall, OpenAIAgent, QueryEngineTool, } from "llamaindex"; import fs from "node:fs/promises"; import path from "node:path"; import { getDataSource } from "./index"; import { generateFilters } from "./queryFilter"; import { createTools } from "./tools"; export async function createChatEngine(documentIds?: string[], params?: any) { const tools: BaseToolWithCall[] = []; // Add a query engine tool if we have a data source // Delete this code if you don't have a data source const index = await getDataSource(params); if (index) { tools.push( new QueryEngineTool({ queryEngine: index.asQueryEngine({ preFilters: generateFilters(documentIds || []), }), metadata: { name: "data_query_engine", description: `A query engine for documents from your data source.`, }, }), ); } const configFile = path.join("config", "tools.json"); let toolConfig: any; try { // add tools from config file if it exists toolConfig = JSON.parse(await fs.readFile(configFile, "utf8")); } catch (e) { console.info(`Could not read ${configFile} file. Using no tools.`); } if (toolConfig) { tools.push(...(await createTools(toolConfig))); } const agent = new OpenAIAgent({ tools, systemPrompt: process.env.SYSTEM_PROMPT, }) as unknown as BaseChatEngine; return agent; } ``` **with-pro//app/api/chat/engine:** ``` import { ContextChatEngine, Settings } from "llamaindex"; import { getDataSource } from "./index"; import { nodeCitationProcessor } from "./nodePostprocessors"; import { generateFilters } from "./queryFilter"; export async function createChatEngine(documentIds?: string[], params?: any) { const index = await getDataSource(params); if (!index) { throw new Error( `StorageContext is empty - call 'npm run generate' to generate the storage first`, ); } const retriever = index.asRetriever({ similarityTopK: process.env.TOP_K ? parseInt(process.env.TOP_K) : undefined, filters: generateFilters(documentIds || []), }); const systemPrompt = process.env.SYSTEM_PROMPT; const citationPrompt = process.env.SYSTEM_CITATION_PROMPT; const prompt = [systemPrompt, citationPrompt].filter((p) => p).join("\n") || undefined; const nodePostprocessors = citationPrompt ? [nodeCitationProcessor] : undefined; return new ContextChatEngine({ chatModel: Settings.llm, retriever, systemPrompt: prompt, nodePostprocessors, }); } ``` `const agent = new OpenAIAgent `I guess we need to fix this part.
yindo commented 2026-02-15 19:15:52 -05:00
Author
Owner
Copy Link
Copy Source

@marcusschiesser commented on GitHub (Nov 3, 2024):

@mcavdar great catch, it should be actually generate to use the LLMAgent instead of the OpenAIAgent - I fixed this in https://github.com/run-llama/create-llama/pull/410

@marcusschiesser commented on GitHub (Nov 3, 2024): @mcavdar great catch, it should be actually generate to use the `LLMAgent` instead of the `OpenAIAgent` - I fixed this in https://github.com/run-llama/create-llama/pull/410
yindo commented 2026-02-15 19:15:52 -05:00
Author
Owner
Copy Link
Copy Source

@marcusschiesser commented on GitHub (Nov 3, 2024):

Unfortunately, the Ollama LLM class from LlamaIndexTS doesn't support tool calls yet, I added this in https://github.com/run-llama/LlamaIndexTS/issues/1430

Once, this is added to LITS, it will automatically work in create-llama (after update). I keep this ticket open for tracking

@marcusschiesser commented on GitHub (Nov 3, 2024): Unfortunately, the Ollama LLM class from LlamaIndexTS doesn't support tool calls yet, I added this in https://github.com/run-llama/LlamaIndexTS/issues/1430 Once, this is added to LITS, it will automatically work in create-llama (after update). I keep this ticket open for tracking
yindo commented 2026-02-15 19:15:52 -05:00
Author
Owner
Copy Link
Copy Source

@joshuacox commented on GitHub (Nov 11, 2024):

sorry for the delay, but this is working great for me now, closing. Feel free to re-open if there are some other issues.

@joshuacox commented on GitHub (Nov 11, 2024): sorry for the delay, but this is working great for me now, closing. Feel free to re-open if there are some other issues.
yindo commented 2026-02-15 19:15:52 -05:00
Author
Owner
Copy Link
Copy Source

@marcusschiesser commented on GitHub (Nov 12, 2024):

just for others, you need to update LllamaIndexTS if it doesn't work

@marcusschiesser commented on GitHub (Nov 12, 2024): just for others, you need to update LllamaIndexTS if it doesn't work
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/create-llama#92
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?