Release 0.1.28 (#206 )

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
fix: Vercel deployment (by including WASM files) (#201 )
2026-07-19 23:13:36 -04:00 · 2024-08-02 15:38:27 +07:00 · 2024-08-02 15:36:54 +07:00 · 2024-08-01 17:09:07 +02:00 · 2024-08-01 16:31:38 +02:00 · 2024-07-31 08:56:00 +02:00
79 changed files with 1751 additions and 509 deletions
@@ -62,6 +62,7 @@ jobs:
        run: pnpm run e2e
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          LLAMA_CLOUD_API_KEY: ${{ secrets.LLAMA_CLOUD_API_KEY }}
        working-directory: .

      - uses: actions/upload-artifact@v3
@@ -1,5 +1,67 @@
 # create-llama

+## 0.1.28
+
+### Patch Changes
+
+- 9a09e8c: Fix Vercel deployment
+
+## 0.1.27
+
+### Patch Changes
+
+- c5c7eee: Make components reusable for chat-llamaindex
+
+## 0.1.26
+
+### Patch Changes
+
+- f43399c: Add metadatafilters to context chat engine (Typescript)
+
+## 0.1.25
+
+### Patch Changes
+
+- c67daeb: fix: missing set private to false for default generate.py
+
+## 0.1.24
+
+### Patch Changes
+
+- 43474a5: Configure LlamaCloud organization ID for Python
+- cf11b23: Add Azure code interpreter for Python and TS
+- fd9fb42: Add Azure OpenAI as model provider
+- 5c13646: Fix starter questions not working in python backend
+
+## 0.1.23
+
+### Patch Changes
+
+- 6bd76fb: Add template for structured extraction
+
+## 0.1.22
+
+### Patch Changes
+
+- b0becaa: Add e2e testing for llamacloud datasource
+- df9cca5: Upgrade pdf viewer
+
+## 0.1.21
+
+### Patch Changes
+
+- bd4714c: Filter private documents for Typescript (Using MetadataFilters) and update to LlamaIndexTS 0.5.7
+- 58e6c15: Add using LlamaParse for private file uploader
+- 455ab68: Display files in sources using LlamaCloud indexes.
+- 23b7357: Use gpt-4o-mini as default model
+- 0900413: Add suggestions for next questions.
+
+## 0.1.20
+
+### Patch Changes
+
+- 624c721: Update to LlamaIndex 0.10.55
+
 ## 0.1.19

 ### Patch Changes
@@ -1,14 +1,20 @@
-# Create LlamaIndex App
+# Create Llama

 The easiest way to get started with [LlamaIndex](https://www.llamaindex.ai/) is by using `create-llama`. This CLI tool enables you to quickly start building a new LlamaIndex application, with everything set up for you.

+## Get started
+
 Just run

 ```bash
 npx create-llama@latest
 ```

-to get started, or see below for more options. Once your app is generated, run
+to get started, or watch this video for a demo session:
+
+https://github.com/user-attachments/assets/dd3edc36-4453-4416-91c2-d24326c6c167
+
+Once your app is generated, run

 ```bash
 npm run dev
@@ -18,16 +24,20 @@ to start the development server. You can then visit [http://localhost:3000](http

 ## What you'll get

- A Next.js-powered front-end using components from [shadcn/ui](https://ui.shadcn.com/). The app is set up as a chat interface that can answer questions about your data (see below)
+- A Next.js-powered front-end using components from [shadcn/ui](https://ui.shadcn.com/). The app is set up as a chat interface that can answer questions about your data or interact with your agent
 - Your choice of 3 back-ends:
  - **Next.js**: if you select this option, you’ll have a full-stack Next.js application that you can deploy to a host like [Vercel](https://vercel.com/) in just a few clicks. This uses [LlamaIndex.TS](https://www.npmjs.com/package/llamaindex), our TypeScript library.
  - **Express**: if you want a more traditional Node.js application you can generate an Express backend. This also uses LlamaIndex.TS.
-  - **Python FastAPI**: if you select this option, you’ll get a backend powered by the [llama-index python package](https://pypi.org/project/llama-index/), which you can deploy to a service like Render or fly.io.
+  - **Python FastAPI**: if you select this option, you’ll get a backend powered by the [llama-index Python package](https://pypi.org/project/llama-index/), which you can deploy to a service like Render or fly.io.
 - The back-end has two endpoints (one streaming, the other one non-streaming) that allow you to send the state of your chat and receive additional responses
 - You add arbitrary data sources to your chat, like local files, websites, or data retrieved from a database.
 - Turn your chat into an AI agent by adding tools (functions called by the LLM).
 - The app uses OpenAI by default, so you'll need an OpenAI API key, or you can customize it to use any of the dozens of LLMs we support.

+Here's how it looks like:
+
+https://github.com/user-attachments/assets/d57af1a1-d99b-4e9c-98d9-4cbd1327eff8
+
 ## Using your data

 You can supply your own data; the app will index it and answer questions. Your generated app will have a folder called `data` (If you're using Express or Python and generate a frontend, it will be `./backend/data`).
@@ -54,7 +64,7 @@ Optionally generate a frontend if you've selected the Python or Express back-end

 ## Customizing the AI models

-The app will default to OpenAI's `gpt-4-turbo` LLM and `text-embedding-3-large` embedding model.
+The app will default to OpenAI's `gpt-4o-mini` LLM and `text-embedding-3-large` embedding model.

 If you want to use different OpenAI models, add the `--ask-models` CLI parameter.

@@ -84,7 +94,7 @@ Need to install the following packages:
  create-llama@latest
 Ok to proceed? (y) y
 ✔ What is your project named? … my-app
-✔ Which template would you like to use? › Chat
+✔ Which template would you like to use? › Agentic RAG (single agent)
 ✔ Which framework would you like to use? › NextJS
 ✔ Would you like to set up observability? › No
 ✔ Please provide your OpenAI API key (leave blank to skip): …
@@ -92,6 +102,7 @@ Ok to proceed? (y) y
 ✔ Would you like to add another data source? › No
 ✔ Would you like to use LlamaParse (improved parser for RAG - requires API key)? … no / yes
 ✔ Would you like to use a vector database? › No, just store the data in the file system
+✔ Would you like to build an agent using tools? If so, select the tools here, otherwise just press enter › Weather
 ? How would you like to proceed? › - Use arrow-keys. Return to submit.
   Just generate code (~1 sec)
 ❯  Start in VSCode (~1 sec)
@@ -17,13 +17,16 @@ const templateFrameworks: TemplateFramework[] = [
  "express",
  "fastapi",
 ];
-const dataSources: string[] = ["--no-files", "--example-file"];
+const dataSources: string[] = ["--no-files", "--llamacloud"];
 const templateUIs: TemplateUI[] = ["shadcn", "html"];
 const templatePostInstallActions: TemplatePostInstallAction[] = [
  "none",
  "runApp",
 ];

+const llamaCloudProjectName = "create-llama";
+const llamaCloudIndexName = "e2e-test";
+
 for (const templateType of templateTypes) {
  for (const templateFramework of templateFrameworks) {
    for (const dataSource of dataSources) {
@@ -31,6 +34,10 @@ for (const templateType of templateTypes) {
        for (const templatePostInstallAction of templatePostInstallActions) {
          const appType: AppType =
            templateFramework === "nextjs" ? "" : "--frontend";
+          const userMessage =
+            dataSource !== "--no-files"
+              ? "Physical standard for letters"
+              : "Hello";
          test.describe(`try create-llama ${templateType} ${templateFramework} ${dataSource} ${templateUI} ${appType} ${templatePostInstallAction}`, async () => {
            let port: number;
            let externalPort: number;
@@ -55,6 +62,8 @@ for (const templateType of templateTypes) {
                port,
                externalPort,
                templatePostInstallAction,
+                llamaCloudProjectName,
+                llamaCloudIndexName,
              );
              name = result.projectName;
              appProcess = result.appProcess;
@@ -75,7 +84,7 @@ for (const templateType of templateTypes) {
            }) => {
              test.skip(templatePostInstallAction !== "runApp");
              await page.goto(`http://localhost:${port}`);
-              await page.fill("form input", "hello");
+              await page.fill("form input", userMessage);
              const [response] = await Promise.all([
                page.waitForResponse(
                  (res) => {
@@ -106,7 +115,7 @@ for (const templateType of templateTypes) {
                    messages: [
                      {
                        role: "user",
-                        content: "Hello",
+                        content: userMessage,
                      },
                    ],
                  },
@@ -72,9 +72,13 @@ export async function runCreateLlama(
  port: number,
  externalPort: number,
  postInstallAction: TemplatePostInstallAction,
+  llamaCloudProjectName: string,
+  llamaCloudIndexName: string,
 ): Promise<CreateLlamaResult> {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error("Setting OPENAI_API_KEY is mandatory to run tests");
+  if (!process.env.OPENAI_API_KEY || !process.env.LLAMA_CLOUD_API_KEY) {
+    throw new Error(
+      "Setting the OPENAI_API_KEY and LLAMA_CLOUD_API_KEY is mandatory to run tests",
+    );
  }
  const name = [
    templateType,
@@ -110,12 +114,16 @@ export async function runCreateLlama(
    "--no-llama-parse",
    "--observability",
    "none",
+    "--llama-cloud-key",
+    process.env.LLAMA_CLOUD_API_KEY,
  ].join(" ");
  console.log(`running command '${command}' in ${cwd}`);
  const appProcess = exec(command, {
    cwd,
    env: {
      ...process.env,
+      LLAMA_CLOUD_PROJECT_NAME: llamaCloudProjectName,
+      LLAMA_CLOUD_INDEX_NAME: llamaCloudIndexName,
    },
  });
  appProcess.stderr?.on("data", (data) => {
@@ -155,6 +155,11 @@ const getVectorDBEnvs = (
            "The base URL for the LlamaCloud API. Only change this for non-production environments",
          value: "https://api.cloud.llamaindex.ai",
        },
+        {
+          name: "LLAMA_CLOUD_ORGANIZATION_ID",
+          description:
+            "The organization ID for the LlamaCloud project (uses default organization if not specified - Python only)",
+        },
      ];
    case "chroma":
      const envs = [
@@ -274,6 +279,33 @@ const getModelEnvs = (modelConfig: ModelConfig): EnvVar[] => {
          },
        ]
      : []),
+    ...(modelConfig.provider === "azure-openai"
+      ? [
+          {
+            name: "AZURE_OPENAI_KEY",
+            description: "The Azure OpenAI key to use.",
+            value: modelConfig.apiKey,
+          },
+          {
+            name: "AZURE_OPENAI_ENDPOINT",
+            description: "The Azure OpenAI endpoint to use.",
+          },
+          {
+            name: "AZURE_OPENAI_API_VERSION",
+            description: "The Azure OpenAI API version to use.",
+          },
+          {
+            name: "AZURE_OPENAI_LLM_DEPLOYMENT",
+            description:
+              "The Azure OpenAI deployment to use for LLM deployment.",
+          },
+          {
+            name: "AZURE_OPENAI_EMBEDDING_DEPLOYMENT",
+            description:
+              "The Azure OpenAI deployment to use for embedding deployment.",
+          },
+        ]
+      : []),
    ...(modelConfig.provider === "t-systems"
      ? [
          {
@@ -23,6 +23,31 @@ import {
 } from "./types";
 import { installTSTemplate } from "./typescript";

+const checkForGenerateScript = (
+  modelConfig: ModelConfig,
+  vectorDb?: TemplateVectorDB,
+  llamaCloudKey?: string,
+  useLlamaParse?: boolean,
+) => {
+  const missingSettings = [];
+
+  if (!modelConfig.isConfigured()) {
+    missingSettings.push("your model provider API key");
+  }
+
+  const llamaCloudApiKey = llamaCloudKey ?? process.env["LLAMA_CLOUD_API_KEY"];
+  const isRequiredLlamaCloudKey = useLlamaParse || vectorDb === "llamacloud";
+  if (isRequiredLlamaCloudKey && !llamaCloudApiKey) {
+    missingSettings.push("your LLAMA_CLOUD_API_KEY");
+  }
+
+  if (vectorDb !== "none" && vectorDb !== "llamacloud") {
+    missingSettings.push("your Vector DB environment variables");
+  }
+
+  return missingSettings;
+};
+
 // eslint-disable-next-line max-params
 async function generateContextData(
  framework: TemplateFramework,
@@ -38,12 +63,15 @@ async function generateContextData(
        ? "poetry run generate"
        : `${packageManager} run generate`,
    )}`;
-    const modelConfigured = modelConfig.isConfigured();
-    const llamaCloudKeyConfigured = useLlamaParse
-      ? llamaCloudKey || process.env["LLAMA_CLOUD_API_KEY"]
-      : true;
-    const hasVectorDb = vectorDb && vectorDb !== "none";
-    if (modelConfigured && llamaCloudKeyConfigured && !hasVectorDb) {
+
+    const missingSettings = checkForGenerateScript(
+      modelConfig,
+      vectorDb,
+      llamaCloudKey,
+      useLlamaParse,
+    );
+
+    if (!missingSettings.length) {
      // If all the required environment variables are set, run the generate script
      if (framework === "fastapi") {
        if (isHavingPoetryLockFile()) {
@@ -63,15 +91,8 @@ async function generateContextData(
      }
    }

-    // generate the message of what to do to run the generate script manually
-    const settings = [];
-    if (!modelConfigured) settings.push("your model provider API key");
-    if (!llamaCloudKeyConfigured) settings.push("your Llama Cloud key");
-    if (hasVectorDb) settings.push("your Vector DB environment variables");
-    const settingsMessage =
-      settings.length > 0 ? `After setting ${settings.join(" and ")}, ` : "";
-    const generateMessage = `run ${runGenerate} to generate the context data.`;
-    console.log(`\n${settingsMessage}${generateMessage}\n\n`);
+    const settingsMessage = `After setting ${missingSettings.join(" and ")}, run ${runGenerate} to generate the context data.`;
+    console.log(`\n${settingsMessage}\n\n`);
  }
 }

@@ -142,7 +163,11 @@ export const installTemplate = async (
    // This is a backend, so we need to copy the test data and create the env file.

    // Copy the environment file to the target directory.
-    if (props.template === "streaming" || props.template === "multiagent") {
+    if (
+      props.template === "streaming" ||
+      props.template === "multiagent" ||
+      props.template === "extractor"
+    ) {
      await createBackendEnvFile(props.root, {
        modelConfig: props.modelConfig,
        llamaCloudKey: props.llamaCloudKey,
@@ -177,10 +202,9 @@ export const installTemplate = async (
    }

    // Create outputs directory
-    if (props.tools && props.tools.length > 0) {
-      await makeDir(path.join(props.root, "output/tools"));
-      await makeDir(path.join(props.root, "output/uploaded"));
-    }
+    await makeDir(path.join(props.root, "output/tools"));
+    await makeDir(path.join(props.root, "output/uploaded"));
+    await makeDir(path.join(props.root, "output/llamacloud"));
  } else {
    // this is a frontend for a full-stack app, create .env file with model information
    await createFrontendEnvFile(props.root, {
@@ -0,0 +1,118 @@
+import ciInfo from "ci-info";
+import prompts from "prompts";
+import { ModelConfigParams, ModelConfigQuestionsParams } from ".";
+import { questionHandlers } from "../../questions";
+
+const ALL_AZURE_OPENAI_CHAT_MODELS: Record<string, { openAIModel: string }> = {
+  "gpt-35-turbo": { openAIModel: "gpt-3.5-turbo" },
+  "gpt-35-turbo-16k": {
+    openAIModel: "gpt-3.5-turbo-16k",
+  },
+  "gpt-4o": { openAIModel: "gpt-4o" },
+  "gpt-4": { openAIModel: "gpt-4" },
+  "gpt-4-32k": { openAIModel: "gpt-4-32k" },
+  "gpt-4-turbo": {
+    openAIModel: "gpt-4-turbo",
+  },
+  "gpt-4-turbo-2024-04-09": {
+    openAIModel: "gpt-4-turbo",
+  },
+  "gpt-4-vision-preview": {
+    openAIModel: "gpt-4-vision-preview",
+  },
+  "gpt-4-1106-preview": {
+    openAIModel: "gpt-4-1106-preview",
+  },
+  "gpt-4o-2024-05-13": {
+    openAIModel: "gpt-4o-2024-05-13",
+  },
+};
+
+const ALL_AZURE_OPENAI_EMBEDDING_MODELS: Record<
+  string,
+  {
+    dimensions: number;
+    openAIModel: string;
+  }
+> = {
+  "text-embedding-ada-002": {
+    dimensions: 1536,
+    openAIModel: "text-embedding-ada-002",
+  },
+  "text-embedding-3-small": {
+    dimensions: 1536,
+    openAIModel: "text-embedding-3-small",
+  },
+  "text-embedding-3-large": {
+    dimensions: 3072,
+    openAIModel: "text-embedding-3-large",
+  },
+};
+
+const DEFAULT_MODEL = "gpt-4o";
+const DEFAULT_EMBEDDING_MODEL = "text-embedding-3-large";
+
+export async function askAzureQuestions({
+  openAiKey,
+  askModels,
+}: ModelConfigQuestionsParams): Promise<ModelConfigParams> {
+  const config: ModelConfigParams = {
+    apiKey: openAiKey || process.env.AZURE_OPENAI_KEY,
+    model: DEFAULT_MODEL,
+    embeddingModel: DEFAULT_EMBEDDING_MODEL,
+    dimensions: getDimensions(DEFAULT_EMBEDDING_MODEL),
+    isConfigured(): boolean {
+      // the Azure model provider can't be fully configured as endpoint and deployment names have to be configured with env variables
+      return false;
+    },
+  };
+
+  // use default model values in CI or if user should not be asked
+  const useDefaults = ciInfo.isCI || !askModels;
+  if (!useDefaults) {
+    const { model } = await prompts(
+      {
+        type: "select",
+        name: "model",
+        message: "Which LLM model would you like to use?",
+        choices: getAvailableModelChoices(),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.model = model;
+
+    const { embeddingModel } = await prompts(
+      {
+        type: "select",
+        name: "embeddingModel",
+        message: "Which embedding model would you like to use?",
+        choices: getAvailableEmbeddingModelChoices(),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.embeddingModel = embeddingModel;
+    config.dimensions = getDimensions(embeddingModel);
+  }
+
+  return config;
+}
+
+function getAvailableModelChoices() {
+  return Object.keys(ALL_AZURE_OPENAI_CHAT_MODELS).map((key) => ({
+    title: key,
+    value: key,
+  }));
+}
+
+function getAvailableEmbeddingModelChoices() {
+  return Object.keys(ALL_AZURE_OPENAI_EMBEDDING_MODELS).map((key) => ({
+    title: key,
+    value: key,
+  }));
+}
+
+function getDimensions(modelName: string) {
+  return ALL_AZURE_OPENAI_EMBEDDING_MODELS[modelName].dimensions;
+}
@@ -3,6 +3,7 @@ import prompts from "prompts";
 import { questionHandlers } from "../../questions";
 import { ModelConfig, ModelProvider, TemplateFramework } from "../types";
 import { askAnthropicQuestions } from "./anthropic";
+import { askAzureQuestions } from "./azure";
 import { askGeminiQuestions } from "./gemini";
 import { askGroqQuestions } from "./groq";
 import { askLLMHubQuestions } from "./llmhub";
@@ -34,6 +35,7 @@ export async function askModelConfig({
      { title: "Anthropic", value: "anthropic" },
      { title: "Gemini", value: "gemini" },
      { title: "Mistral", value: "mistral" },
+      { title: "AzureOpenAI", value: "azure-openai" },
    ];

    if (framework === "fastapi") {
@@ -69,6 +71,9 @@ export async function askModelConfig({
    case "mistral":
      modelConfig = await askMistralQuestions({ askModels });
      break;
+    case "azure-openai":
+      modelConfig = await askAzureQuestions({ askModels });
+      break;
    case "t-systems":
      modelConfig = await askLLMHubQuestions({ askModels });
      break;
@@ -8,7 +8,7 @@ import { questionHandlers } from "../../questions";

 const OPENAI_API_URL = "https://api.openai.com/v1";

-const DEFAULT_MODEL = "gpt-3.5-turbo";
+const DEFAULT_MODEL = "gpt-4o-mini";
 const DEFAULT_EMBEDDING_MODEL = "text-embedding-3-large";

 export async function askOpenAIQuestions({
@@ -121,7 +121,7 @@ const getAdditionalDependencies = (
        case "llamacloud":
          dependencies.push({
            name: "llama-index-indices-managed-llama-cloud",
-            version: "^0.2.1",
+            version: "^0.2.7",
          });
          break;
      }
@@ -193,6 +193,16 @@ const getAdditionalDependencies = (
        version: "0.1.4",
      });
      break;
+    case "azure-openai":
+      dependencies.push({
+        name: "llama-index-llms-azure-openai",
+        version: "0.1.10",
+      });
+      dependencies.push({
+        name: "llama-index-embeddings-azure-openai",
+        version: "0.1.11",
+      });
+      break;
    case "t-systems":
      dependencies.push({
        name: "llama-index-agent-openai",
@@ -188,6 +188,35 @@ For better results, you can specify the region parameter to get results from a s
      },
    ],
  },
+  {
+    display: "Azure Code Interpreter",
+    name: "azure_code_interpreter.AzureCodeInterpreterToolSpec",
+    supportedFrameworks: ["fastapi", "nextjs", "express"],
+    type: ToolType.LLAMAHUB,
+    dependencies: [
+      {
+        name: "llama-index-tools-azure-code-interpreter",
+        version: "0.2.0",
+      },
+    ],
+    envVars: [
+      {
+        name: "AZURE_POOL_MANAGEMENT_ENDPOINT",
+        description:
+          "Please follow this guideline to create and get the pool management endpoint: https://learn.microsoft.com/azure/container-apps/sessions?tabs=azure-cli",
+      },
+      {
+        name: TOOL_SYSTEM_PROMPT_ENV_VAR,
+        description: "System prompt for Azure code interpreter tool.",
+        value: `-You are a Python interpreter that can run any python code in a secure environment.
+- The python code runs in a Jupyter notebook. Every time you call the 'interpreter' tool, the python code is executed in a separate cell. 
+- You are given tasks to complete and you run python code to solve them.
+- It's okay to make multiple calls to interpreter tool. If you get an error or the result is not what you expected, you can call the tool again. Don't give up too soon!
+- Plot visualizations using matplotlib or any other visualization library directly in the notebook.
+- You can install any pip package (if it exists) by running a cell with pip install.`,
+      },
+    ],
+  },
 ];

 export const getTool = (toolName: string): Tool | undefined => {
@@ -8,6 +8,7 @@ export type ModelProvider =
  | "anthropic"
  | "gemini"
  | "mistral"
+  | "azure-openai"
  | "t-systems";
 export type ModelConfig = {
  provider: ModelProvider;
@@ -18,6 +19,7 @@ export type ModelConfig = {
  isConfigured(): boolean;
 };
 export type TemplateType =
+  | "extractor"
  | "streaming"
  | "community"
  | "llamapack"
@@ -9,7 +9,7 @@ import prompts from "prompts";
 import terminalLink from "terminal-link";
 import checkForUpdate from "update-check";
 import { createApp } from "./create-app";
-import { getDataSources } from "./helpers/datasources";
+import { EXAMPLE_FILE, getDataSources } from "./helpers/datasources";
 import { getPkgManager } from "./helpers/get-pkg-manager";
 import { isFolderEmpty } from "./helpers/is-folder-empty";
 import { initializeGlobalAgent } from "./helpers/proxy";
@@ -194,8 +194,16 @@ if (process.argv.includes("--no-llama-parse")) {
 program.askModels = process.argv.includes("--ask-models");
 if (process.argv.includes("--no-files")) {
  program.dataSources = [];
-} else {
+} else if (process.argv.includes("--example-file")) {
  program.dataSources = getDataSources(program.files, program.exampleFile);
+} else if (process.argv.includes("--llamacloud")) {
+  program.dataSources = [
+    {
+      type: "llamacloud",
+      config: {},
+    },
+    EXAMPLE_FILE,
+  ];
 }

 const packageManager = !!program.useNpm
@@ -1,6 +1,6 @@
 {
  "name": "create-llama",
-  "version": "0.1.19",
+  "version": "0.1.28",
  "description": "Create LlamaIndex-powered apps with one command",
  "keywords": [
    "rag",
@@ -342,6 +342,7 @@ export const askQuestions = async (
              title: "Multi-agent app (using llama-agents)",
              value: "multiagent",
            },
+            { title: "Structured Extractor", value: "extractor" },
            {
              title: `Community template from ${styledRepo}`,
              value: "community",
@@ -405,7 +406,7 @@ export const askQuestions = async (
    return; // early return - no further questions needed for llamapack projects
  }

-  if (program.template === "multiagent") {
+  if (program.template === "multiagent" || program.template === "extractor") {
    // TODO: multi-agents currently only supports FastAPI
    program.framework = preferences.framework = "fastapi";
  }
@@ -647,45 +648,51 @@ export const askQuestions = async (
    // default to use LlamaParse if using LlamaCloud
    program.useLlamaParse = preferences.useLlamaParse = true;
  } else {
-    if (program.dataSources.some((ds) => ds.type === "file")) {
-      if (ciInfo.isCI) {
-        program.useLlamaParse = getPrefOrDefault("useLlamaParse");
-      } else {
-        const { useLlamaParse } = await prompts(
-          {
-            type: "toggle",
-            name: "useLlamaParse",
-            message:
-              "Would you like to use LlamaParse (improved parser for RAG - requires API key)?",
-            initial: false,
-            active: "yes",
-            inactive: "no",
-          },
-          questionHandlers,
-        );
-        program.useLlamaParse = useLlamaParse;
-        preferences.useLlamaParse = useLlamaParse;
+    if (program.useLlamaParse === undefined) {
+      // if already set useLlamaParse, don't ask again
+      if (program.dataSources.some((ds) => ds.type === "file")) {
+        if (ciInfo.isCI) {
+          program.useLlamaParse = getPrefOrDefault("useLlamaParse");
+        } else {
+          const { useLlamaParse } = await prompts(
+            {
+              type: "toggle",
+              name: "useLlamaParse",
+              message:
+                "Would you like to use LlamaParse (improved parser for RAG - requires API key)?",
+              initial: false,
+              active: "yes",
+              inactive: "no",
+            },
+            questionHandlers,
+          );
+          program.useLlamaParse = useLlamaParse;
+          preferences.useLlamaParse = useLlamaParse;
+        }
      }
    }
  }

  // Ask for LlamaCloud API key when using a LlamaCloud index or LlamaParse
  if (isUsingLlamaCloud || program.useLlamaParse) {
-    if (ciInfo.isCI) {
-      program.llamaCloudKey = getPrefOrDefault("llamaCloudKey");
-    } else {
-      // Ask for LlamaCloud API key
-      const { llamaCloudKey } = await prompts(
-        {
-          type: "text",
-          name: "llamaCloudKey",
-          message:
-            "Please provide your LlamaCloud API key (leave blank to skip):",
-        },
-        questionHandlers,
-      );
-      program.llamaCloudKey = preferences.llamaCloudKey =
-        llamaCloudKey || process.env.LLAMA_CLOUD_API_KEY;
+    if (!program.llamaCloudKey) {
+      // if already set, don't ask again
+      if (ciInfo.isCI) {
+        program.llamaCloudKey = getPrefOrDefault("llamaCloudKey");
+      } else {
+        // Ask for LlamaCloud API key
+        const { llamaCloudKey } = await prompts(
+          {
+            type: "text",
+            name: "llamaCloudKey",
+            message:
+              "Please provide your LlamaCloud API key (leave blank to skip):",
+          },
+          questionHandlers,
+        );
+        program.llamaCloudKey = preferences.llamaCloudKey =
+          llamaCloudKey || process.env.LLAMA_CLOUD_API_KEY;
+      }
    }
  }

@@ -1,4 +1,10 @@
-import { BaseToolWithCall, OpenAIAgent, QueryEngineTool } from "llamaindex";
+import {
+  BaseToolWithCall,
+  MetadataFilter,
+  MetadataFilters,
+  OpenAIAgent,
+  QueryEngineTool,
+} from "llamaindex";
 import fs from "node:fs/promises";
 import path from "node:path";
 import { getDataSource } from "./index";
@@ -14,7 +20,7 @@ export async function createChatEngine(documentIds?: string[]) {
    tools.push(
      new QueryEngineTool({
        queryEngine: index.asQueryEngine({
-          preFilters: undefined, // TODO: Add filters once LITS supports it (getQueryFilters)
+          preFilters: generateFilters(documentIds || []),
        }),
        metadata: {
          name: "data_query_engine",
@@ -41,3 +47,27 @@ export async function createChatEngine(documentIds?: string[]) {
    systemPrompt: process.env.SYSTEM_PROMPT,
  });
 }
+
+export function generateFilters(documentIds: string[]): MetadataFilters {
+  // public documents don't have the "private" field or it's set to "false"
+  const publicDocumentsFilter: MetadataFilter = {
+    key: "private",
+    value: ["true"],
+    operator: "nin",
+  };
+
+  // if no documentIds are provided, only retrieve information from public documents
+  if (!documentIds.length) return { filters: [publicDocumentsFilter] };
+
+  const privateDocumentsFilter: MetadataFilter = {
+    key: "doc_id",
+    value: documentIds,
+    operator: "in",
+  };
+
+  // if documentIds are provided, retrieve information from public and private documents
+  return {
+    filters: [publicDocumentsFilter, privateDocumentsFilter],
+    condition: "or",
+  };
+}
@@ -1,4 +1,9 @@
-import { ContextChatEngine, Settings } from "llamaindex";
+import {
+  ContextChatEngine,
+  MetadataFilter,
+  MetadataFilters,
+  Settings,
+} from "llamaindex";
 import { getDataSource } from "./index";

 export async function createChatEngine(documentIds?: string[]) {
@@ -10,12 +15,36 @@ export async function createChatEngine(documentIds?: string[]) {
  }
  const retriever = index.asRetriever({
    similarityTopK: process.env.TOP_K ? parseInt(process.env.TOP_K) : 3,
+    filters: generateFilters(documentIds || []),
  });

  return new ContextChatEngine({
    chatModel: Settings.llm,
    retriever,
-    // disable as a custom system prompt disables the generated context
-    // systemPrompt: process.env.SYSTEM_PROMPT,
+    systemPrompt: process.env.SYSTEM_PROMPT,
  });
 }
+
+export function generateFilters(documentIds: string[]): MetadataFilters {
+  // public documents don't have the "private" field or it's set to "false"
+  const publicDocumentsFilter: MetadataFilter = {
+    key: "private",
+    value: ["true"],
+    operator: "nin",
+  };
+
+  // if no documentIds are provided, only retrieve information from public documents
+  if (!documentIds.length) return { filters: [publicDocumentsFilter] };
+
+  const privateDocumentsFilter: MetadataFilter = {
+    key: "doc_id",
+    value: documentIds,
+    operator: "in",
+  };
+
+  // if documentIds are provided, retrieve information from public and private documents
+  return {
+    filters: [publicDocumentsFilter, privateDocumentsFilter],
+    condition: "or",
+  };
+}
@@ -1,115 +0,0 @@
-import fs from "fs";
-import {
-  BaseNode,
-  Document,
-  IngestionPipeline,
-  Metadata,
-  Settings,
-  SimpleNodeParser,
-  storageContextFromDefaults,
-  VectorStoreIndex,
-} from "llamaindex";
-import { DocxReader } from "llamaindex/readers/DocxReader";
-import { PDFReader } from "llamaindex/readers/PDFReader";
-import { TextFileReader } from "llamaindex/readers/TextFileReader";
-import crypto from "node:crypto";
-import { getDataSource } from "../../engine";
-
-const MIME_TYPE_TO_EXT: Record<string, string> = {
-  "application/pdf": "pdf",
-  "text/plain": "txt",
-  "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
-    "docx",
-};
-
-export async function uploadDocument(raw: string): Promise<string[]> {
-  const [header, content] = raw.split(",");
-  const mimeType = header.replace("data:", "").replace(";base64", "");
-  const fileBuffer = Buffer.from(content, "base64");
-  const documents = await loadDocuments(fileBuffer, mimeType);
-  const { filename } = await saveDocument(fileBuffer, mimeType);
-  return await runPipeline(documents, filename);
-}
-
-async function runPipeline(
-  documents: Document[],
-  filename: string,
-): Promise<string[]> {
-  // mark documents to add to the vector store as private
-  for (const document of documents) {
-    document.metadata = {
-      ...document.metadata,
-      file_name: filename,
-      private: true,
-    };
-  }
-  const pipeline = new IngestionPipeline({
-    transformations: [
-      new SimpleNodeParser({
-        chunkSize: Settings.chunkSize,
-        chunkOverlap: Settings.chunkOverlap,
-      }),
-      Settings.embedModel,
-    ],
-  });
-  const nodes = await pipeline.run({ documents });
-  await addNodesToVectorStore(nodes);
-  return documents.map((document) => document.id_);
-}
-
-async function loadDocuments(fileBuffer: Buffer, mimeType: string) {
-  console.log(`Processing uploaded document of type: ${mimeType}`);
-  switch (mimeType) {
-    case "application/pdf": {
-      const pdfReader = new PDFReader();
-      return await pdfReader.loadDataAsContent(new Uint8Array(fileBuffer));
-    }
-    case "text/plain": {
-      const textReader = new TextFileReader();
-      return await textReader.loadDataAsContent(fileBuffer);
-    }
-    case "application/vnd.openxmlformats-officedocument.wordprocessingml.document": {
-      const docxReader = new DocxReader();
-      return await docxReader.loadDataAsContent(fileBuffer);
-    }
-    default:
-      throw new Error(`Unsupported document type: ${mimeType}`);
-  }
-}
-
-async function saveDocument(fileBuffer: Buffer, mimeType: string) {
-  const fileExt = MIME_TYPE_TO_EXT[mimeType];
-  if (!fileExt) throw new Error(`Unsupported document type: ${mimeType}`);
-
-  const folder = "output/uploaded";
-  const filename = `${crypto.randomUUID()}.${fileExt}`;
-  const filepath = `${folder}/${filename}`;
-  const fileurl = `${process.env.FILESERVER_URL_PREFIX}/${filepath}`;
-
-  if (!fs.existsSync(folder)) {
-    fs.mkdirSync(folder, { recursive: true });
-  }
-  await fs.promises.writeFile(filepath, fileBuffer);
-
-  console.log(`Saved document file to ${filepath}.\nURL: ${fileurl}`);
-  return {
-    filename,
-    filepath,
-    fileurl,
-  };
-}
-
-async function addNodesToVectorStore(nodes: BaseNode<Metadata>[]) {
-  let currentIndex = await getDataSource(); // always not null with an vectordb
-  if (currentIndex) {
-    await currentIndex.insertNodes(nodes);
-  } else {
-    // Not using vectordb and haven't generated local index yet
-    const storageContext = await storageContextFromDefaults({
-      persistDir: "./cache",
-    });
-    currentIndex = await VectorStoreIndex.init({ nodes, storageContext });
-  }
-  currentIndex.storageContext.docStore.persist();
-  console.log("Added nodes to the vector store.");
-}
@@ -0,0 +1,44 @@
+import fs from "fs";
+import crypto from "node:crypto";
+import { getExtractors } from "../../engine/loader";
+
+const MIME_TYPE_TO_EXT: Record<string, string> = {
+  "application/pdf": "pdf",
+  "text/plain": "txt",
+  "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
+    "docx",
+};
+
+const UPLOADED_FOLDER = "output/uploaded";
+
+export async function loadDocuments(fileBuffer: Buffer, mimeType: string) {
+  const extractors = getExtractors();
+  const reader = extractors[MIME_TYPE_TO_EXT[mimeType]];
+
+  if (!reader) {
+    throw new Error(`Unsupported document type: ${mimeType}`);
+  }
+  console.log(`Processing uploaded document of type: ${mimeType}`);
+  return await reader.loadDataAsContent(fileBuffer);
+}
+
+export async function saveDocument(fileBuffer: Buffer, mimeType: string) {
+  const fileExt = MIME_TYPE_TO_EXT[mimeType];
+  if (!fileExt) throw new Error(`Unsupported document type: ${mimeType}`);
+
+  const filename = `${crypto.randomUUID()}.${fileExt}`;
+  const filepath = `${UPLOADED_FOLDER}/${filename}`;
+  const fileurl = `${process.env.FILESERVER_URL_PREFIX}/${filepath}`;
+
+  if (!fs.existsSync(UPLOADED_FOLDER)) {
+    fs.mkdirSync(UPLOADED_FOLDER, { recursive: true });
+  }
+  await fs.promises.writeFile(filepath, fileBuffer);
+
+  console.log(`Saved document file to ${filepath}.\nURL: ${fileurl}`);
+  return {
+    filename,
+    filepath,
+    fileurl,
+  };
+}
@@ -0,0 +1,38 @@
+import {
+  Document,
+  IngestionPipeline,
+  Settings,
+  SimpleNodeParser,
+  VectorStoreIndex,
+} from "llamaindex";
+import { LlamaCloudIndex } from "llamaindex/cloud/LlamaCloudIndex";
+
+export async function runPipeline(
+  currentIndex: VectorStoreIndex | LlamaCloudIndex,
+  documents: Document[],
+) {
+  if (currentIndex instanceof LlamaCloudIndex) {
+    // LlamaCloudIndex processes the documents automatically
+    // so we don't need ingestion pipeline, just insert the documents directly
+    for (const document of documents) {
+      await currentIndex.insert(document);
+    }
+  } else {
+    // Use ingestion pipeline to process the documents into nodes and add them to the vector store
+    const pipeline = new IngestionPipeline({
+      transformations: [
+        new SimpleNodeParser({
+          chunkSize: Settings.chunkSize,
+          chunkOverlap: Settings.chunkOverlap,
+        }),
+        Settings.embedModel,
+      ],
+    });
+    const nodes = await pipeline.run({ documents });
+    await currentIndex.insertNodes(nodes);
+    currentIndex.storageContext.docStore.persist();
+    console.log("Added nodes to the vector store.");
+  }
+
+  return documents.map((document) => document.id_);
+}
@@ -0,0 +1,26 @@
+import { VectorStoreIndex } from "llamaindex";
+import { LlamaCloudIndex } from "llamaindex/cloud/LlamaCloudIndex";
+import { loadDocuments, saveDocument } from "./helper";
+import { runPipeline } from "./pipeline";
+
+export async function uploadDocument(
+  index: VectorStoreIndex | LlamaCloudIndex,
+  raw: string,
+): Promise<string[]> {
+  const [header, content] = raw.split(",");
+  const mimeType = header.replace("data:", "").replace(";base64", "");
+  const fileBuffer = Buffer.from(content, "base64");
+  const documents = await loadDocuments(fileBuffer, mimeType);
+  const { filename } = await saveDocument(fileBuffer, mimeType);
+
+  // Update documents with metadata
+  for (const document of documents) {
+    document.metadata = {
+      ...document.metadata,
+      file_name: filename,
+      private: "true", // to separate private uploads from public documents
+    };
+  }
+
+  return await runPipeline(index, documents);
+}
@@ -35,9 +35,9 @@ export function retrieveDocumentIds(annotations?: JSONValue[]): string[] {
    ) {
      const files = data.files as DocumentFile[];
      for (const file of files) {
-        if (Array.isArray(file.content)) {
+        if (Array.isArray(file.content.value)) {
          // it's an array, so it's an array of doc IDs
-          for (const id of file.content) {
+          for (const id of file.content.value) {
            ids.push(id);
          }
        }
@@ -6,23 +6,29 @@ import {
  ToolCall,
  ToolOutput,
 } from "llamaindex";
+import { LLamaCloudFileService } from "./service";

 export function appendSourceData(
  data: StreamData,
  sourceNodes?: NodeWithScore<Metadata>[],
 ) {
  if (!sourceNodes?.length) return;
-  data.appendMessageAnnotation({
-    type: "sources",
-    data: {
-      nodes: sourceNodes.map((node) => ({
-        ...node.node.toMutableJSON(),
-        id: node.node.id_,
-        score: node.score ?? null,
-        url: getNodeUrl(node.node.metadata),
-      })),
-    },
-  });
+  try {
+    const nodes = sourceNodes.map((node) => ({
+      ...node.node.toMutableJSON(),
+      id: node.node.id_,
+      score: node.score ?? null,
+      url: getNodeUrl(node.node.metadata),
+    }));
+    data.appendMessageAnnotation({
+      type: "sources",
+      data: {
+        nodes,
+      },
+    });
+  } catch (error) {
+    console.error("Error appending source data:", error);
+  }
 }

 export function appendEventData(data: StreamData, title?: string) {
@@ -69,17 +75,18 @@ export function createCallbackManager(stream: StreamData) {
  const callbackManager = new CallbackManager();

  callbackManager.on("retrieve-end", (data) => {
-    const { nodes, query } = data.detail.payload;
+    const { nodes, query } = data.detail;
    appendSourceData(stream, nodes);
    appendEventData(stream, `Retrieving context for query: '${query}'`);
    appendEventData(
      stream,
      `Retrieved ${nodes.length} sources to use as context for the query`,
    );
+    LLamaCloudFileService.downloadFiles(nodes); // don't await to avoid blocking chat streaming
  });

  callbackManager.on("llm-tool-call", (event) => {
-    const { name, input } = event.detail.payload.toolCall;
+    const { name, input } = event.detail.toolCall;
    const inputString = Object.entries(input)
      .map(([key, value]) => `${key}: ${value}`)
      .join(", ");
@@ -90,7 +97,7 @@ export function createCallbackManager(stream: StreamData) {
  });

  callbackManager.on("llm-tool-result", (event) => {
-    const { toolCall, toolResult } = event.detail.payload;
+    const { toolCall, toolResult } = event.detail;
    appendToolData(stream, toolCall, toolResult);
  });

@@ -98,18 +105,24 @@ export function createCallbackManager(stream: StreamData) {
 }

 function getNodeUrl(metadata: Metadata) {
-  const url = metadata["URL"];
-  if (url) return url;
-  const fileName = metadata["file_name"];
  if (!process.env.FILESERVER_URL_PREFIX) {
    console.warn(
      "FILESERVER_URL_PREFIX is not set. File URLs will not be generated.",
    );
-    return undefined;
  }
-  if (fileName) {
-    const folder = metadata["private"] ? "output/uploaded" : "data";
+  const fileName = metadata["file_name"];
+  if (fileName && process.env.FILESERVER_URL_PREFIX) {
+    // file_name exists and file server is configured
+    const pipelineId = metadata["pipeline_id"];
+    if (pipelineId && metadata["private"] == null) {
+      // file is from LlamaCloud and was not ingested locally
+      const name = LLamaCloudFileService.toDownloadedName(pipelineId, fileName);
+      return `${process.env.FILESERVER_URL_PREFIX}/output/llamacloud/${name}`;
+    }
+    const isPrivate = metadata["private"] === "true";
+    const folder = isPrivate ? "output/uploaded" : "data";
    return `${process.env.FILESERVER_URL_PREFIX}/${folder}/${fileName}`;
  }
-  return undefined;
+  // fallback to URL in metadata (e.g. for websites)
+  return metadata["URL"];
 }
@@ -0,0 +1,128 @@
+import { Metadata, NodeWithScore } from "llamaindex";
+import fs from "node:fs";
+import https from "node:https";
+import path from "node:path";
+
+const LLAMA_CLOUD_OUTPUT_DIR = "output/llamacloud";
+const LLAMA_CLOUD_BASE_URL = "https://cloud.llamaindex.ai/api/v1";
+const FILE_DELIMITER = "$"; // delimiter between pipelineId and filename
+
+interface LlamaCloudFile {
+  name: string;
+  file_id: string;
+  project_id: string;
+}
+
+export class LLamaCloudFileService {
+  public static async downloadFiles(nodes: NodeWithScore<Metadata>[]) {
+    const files = this.nodesToDownloadFiles(nodes);
+    if (!files.length) return;
+    console.log("Downloading files from LlamaCloud...");
+    for (const file of files) {
+      await this.downloadFile(file.pipelineId, file.fileName);
+    }
+  }
+
+  public static toDownloadedName(pipelineId: string, fileName: string) {
+    return `${pipelineId}${FILE_DELIMITER}${fileName}`;
+  }
+
+  /**
+   * This function will return an array of unique files to download from LlamaCloud
+   * We only download files that are uploaded directly in LlamaCloud datasources (don't have `private` in metadata)
+   * Files are uploaded directly in LlamaCloud datasources don't have `private` in metadata (public docs)
+   * Files are uploaded from local via `generate` command will have `private=false` (public docs)
+   * Files are uploaded from local via `/chat/upload` endpoint will have `private=true` (private docs)
+   *
+   * @param nodes
+   * @returns list of unique files to download
+   */
+  private static nodesToDownloadFiles(nodes: NodeWithScore<Metadata>[]) {
+    const downloadFiles: Array<{
+      pipelineId: string;
+      fileName: string;
+    }> = [];
+    for (const node of nodes) {
+      const isLocalFile = node.node.metadata["private"] != null;
+      const pipelineId = node.node.metadata["pipeline_id"];
+      const fileName = node.node.metadata["file_name"];
+      if (isLocalFile || !pipelineId || !fileName) continue;
+      const isDuplicate = downloadFiles.some(
+        (f) => f.pipelineId === pipelineId && f.fileName === fileName,
+      );
+      if (!isDuplicate) {
+        downloadFiles.push({ pipelineId, fileName });
+      }
+    }
+    return downloadFiles;
+  }
+
+  private static async downloadFile(pipelineId: string, fileName: string) {
+    try {
+      const downloadedName = this.toDownloadedName(pipelineId, fileName);
+      const downloadedPath = path.join(LLAMA_CLOUD_OUTPUT_DIR, downloadedName);
+
+      // Check if file already exists
+      if (fs.existsSync(downloadedPath)) return;
+
+      const urlToDownload = await this.getFileUrlByName(pipelineId, fileName);
+      if (!urlToDownload) throw new Error("File not found in LlamaCloud");
+
+      const file = fs.createWriteStream(downloadedPath);
+      https
+        .get(urlToDownload, (response) => {
+          response.pipe(file);
+          file.on("finish", () => {
+            file.close(() => {
+              console.log("File downloaded successfully");
+            });
+          });
+        })
+        .on("error", (err) => {
+          fs.unlink(downloadedPath, () => {
+            console.error("Error downloading file:", err);
+            throw err;
+          });
+        });
+    } catch (error) {
+      throw new Error(`Error downloading file from LlamaCloud: ${error}`);
+    }
+  }
+
+  private static async getFileUrlByName(
+    pipelineId: string,
+    name: string,
+  ): Promise<string | null> {
+    const files = await this.getAllFiles(pipelineId);
+    const file = files.find((file) => file.name === name);
+    if (!file) return null;
+    return await this.getFileUrlById(file.project_id, file.file_id);
+  }
+
+  private static async getFileUrlById(
+    projectId: string,
+    fileId: string,
+  ): Promise<string> {
+    const url = `${LLAMA_CLOUD_BASE_URL}/files/${fileId}/content?project_id=${projectId}`;
+    const headers = {
+      Accept: "application/json",
+      Authorization: `Bearer ${process.env.LLAMA_CLOUD_API_KEY}`,
+    };
+    const response = await fetch(url, { method: "GET", headers });
+    const data = (await response.json()) as { url: string };
+    return data.url;
+  }
+
+  private static async getAllFiles(
+    pipelineId: string,
+  ): Promise<LlamaCloudFile[]> {
+    const url = `${LLAMA_CLOUD_BASE_URL}/pipelines/${pipelineId}/files`;
+    const headers = {
+      Accept: "application/json",
+      Authorization: `Bearer ${process.env.LLAMA_CLOUD_API_KEY}`,
+    };
+    const response = await fetch(url, { method: "GET", headers });
+    const data = await response.json();
+    return data;
+  }
+}
@@ -5,34 +5,51 @@ import {
  trimStartOfStreamHelper,
  type AIStreamCallbacksAndOptions,
 } from "ai";
-import { EngineResponse } from "llamaindex";
+import { ChatMessage, EngineResponse } from "llamaindex";
+import { generateNextQuestions } from "./suggestion";

 export function LlamaIndexStream(
  response: AsyncIterable<EngineResponse>,
  data: StreamData,
+  chatHistory: ChatMessage[],
  opts?: {
    callbacks?: AIStreamCallbacksAndOptions;
  },
 ): ReadableStream<Uint8Array> {
-  return createParser(response, data)
+  return createParser(response, data, chatHistory)
    .pipeThrough(createCallbacksTransformer(opts?.callbacks))
    .pipeThrough(createStreamDataTransformer());
 }

-function createParser(res: AsyncIterable<EngineResponse>, data: StreamData) {
+function createParser(
+  res: AsyncIterable<EngineResponse>,
+  data: StreamData,
+  chatHistory: ChatMessage[],
+) {
  const it = res[Symbol.asyncIterator]();
  const trimStartOfStream = trimStartOfStreamHelper();
+  let llmTextResponse = "";

  return new ReadableStream<string>({
    async pull(controller): Promise<void> {
      const { value, done } = await it.next();
      if (done) {
        controller.close();
+        // LLM stream is done, generate the next questions with a new LLM call
+        chatHistory.push({ role: "assistant", content: llmTextResponse });
+        const questions: string[] = await generateNextQuestions(chatHistory);
+        if (questions.length > 0) {
+          data.appendMessageAnnotation({
+            type: "suggested_questions",
+            data: questions,
+          });
+        }
        data.close();
        return;
      }
      const text = trimStartOfStream(value.delta ?? "");
      if (text) {
+        llmTextResponse += text;
        controller.enqueue(text);
      }
    },
@@ -0,0 +1,55 @@
+import { ChatMessage, Settings } from "llamaindex";
+
+const NEXT_QUESTION_PROMPT_TEMPLATE = `You're a helpful assistant! Your task is to suggest the next question that user might ask. 
+Here is the conversation history
+---------------------
+$conversation
+---------------------
+Given the conversation history, please give me $number_of_questions questions that you might ask next!
+Your answer should be wrapped in three sticks which follows the following format:
+\`\`\`
+<question 1>
+<question 2>\`\`\`
+`;
+const N_QUESTIONS_TO_GENERATE = 3;
+
+export async function generateNextQuestions(
+  conversation: ChatMessage[],
+  numberOfQuestions: number = N_QUESTIONS_TO_GENERATE,
+) {
+  const llm = Settings.llm;
+
+  // Format conversation
+  const conversationText = conversation
+    .map((message) => `${message.role}: ${message.content}`)
+    .join("\n");
+  const message = NEXT_QUESTION_PROMPT_TEMPLATE.replace(
+    "$conversation",
+    conversationText,
+  ).replace("$number_of_questions", numberOfQuestions.toString());
+
+  try {
+    const response = await llm.complete({ prompt: message });
+    const questions = extractQuestions(response.text);
+    return questions;
+  } catch (error) {
+    console.error("Error: ", error);
+    throw error;
+  }
+}
+
+// TODO: instead of parsing the LLM's result we can use structured predict, once LITS supports it
+function extractQuestions(text: string): string[] {
+  // Extract the text inside the triple backticks
+  // @ts-ignore
+  const contentMatch = text.match(/```(.*?)```/s);
+  const content = contentMatch ? contentMatch[1] : "";
+
+  // Split the content by newlines to get each question
+  const questions = content
+    .split("\n")
+    .map((question) => question.trim())
+    .filter((question) => question !== "");
+
+  return questions;
+}
@@ -1,11 +1,9 @@
-import os
-import yaml
-import importlib
 import logging
-from typing import Dict
+
+import yaml
+from app.engine.loaders.db import DBLoaderConfig, get_db_documents
 from app.engine.loaders.file import FileLoaderConfig, get_file_documents
 from app.engine.loaders.web import WebLoaderConfig, get_web_documents
-from app.engine.loaders.db import DBLoaderConfig, get_db_documents

 logger = logging.getLogger(__name__)

@@ -62,7 +62,8 @@ def get_file_documents(config: FileLoaderConfig):
        )
        return reader.load_data()
    except Exception as e:
-        import sys, traceback
+        import sys
+        import traceback

        # Catch the error if the data dir is empty
        # and return as empty document list
@@ -1,7 +1,14 @@
-import { SimpleDirectoryReader } from "llamaindex";
+import {
+  FILE_EXT_TO_READER,
+  SimpleDirectoryReader,
+} from "llamaindex/readers/SimpleDirectoryReader";

 export const DATA_DIR = "./data";

+export function getExtractors() {
+  return FILE_EXT_TO_READER;
+}
+
 export async function getDocuments() {
  return await new SimpleDirectoryReader().loadData({
    directoryPath: DATA_DIR,
@@ -1,19 +1,30 @@
+import { LlamaParseReader } from "llamaindex/readers/LlamaParseReader";
 import {
  FILE_EXT_TO_READER,
-  LlamaParseReader,
  SimpleDirectoryReader,
-} from "llamaindex";
+} from "llamaindex/readers/SimpleDirectoryReader";

 export const DATA_DIR = "./data";

+export function getExtractors() {
+  const llamaParseParser = new LlamaParseReader({ resultType: "markdown" });
+  const extractors = FILE_EXT_TO_READER;
+  // Change all the supported extractors to LlamaParse
+  // except for .txt, it doesn't need to be parsed
+  for (const key in extractors) {
+    if (key === "txt") {
+      continue;
+    }
+    extractors[key] = llamaParseParser;
+  }
+  return extractors;
+}
+
 export async function getDocuments() {
  const reader = new SimpleDirectoryReader();
-  // Load PDFs using LlamaParseReader
+  const extractors = getExtractors();
  return await reader.loadData({
    directoryPath: DATA_DIR,
-    fileExtToReader: {
-      ...FILE_EXT_TO_READER,
-      pdf: new LlamaParseReader({ resultType: "markdown" }),
-    },
+    fileExtToReader: extractors,
  });
 }
@@ -75,27 +75,33 @@ def init_azure_openai():
    from llama_index.embeddings.azure_openai import AzureOpenAIEmbedding
    from llama_index.llms.azure_openai import AzureOpenAI

-    llm_deployment = os.getenv("AZURE_OPENAI_LLM_DEPLOYMENT")
-    embedding_deployment = os.getenv("AZURE_OPENAI_EMBEDDING_DEPLOYMENT")
+    llm_deployment = os.environ["AZURE_OPENAI_LLM_DEPLOYMENT"]
+    embedding_deployment = os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"]
    max_tokens = os.getenv("LLM_MAX_TOKENS")
-    api_key = os.getenv("AZURE_OPENAI_API_KEY")
-    llm_config = {
-        "api_key": api_key,
-        "deployment_name": llm_deployment,
-        "model": os.getenv("MODEL"),
-        "temperature": float(os.getenv("LLM_TEMPERATURE", DEFAULT_TEMPERATURE)),
-        "max_tokens": int(max_tokens) if max_tokens is not None else None,
-    }
-    Settings.llm = AzureOpenAI(**llm_config)
-
+    temperature = os.getenv("LLM_TEMPERATURE", DEFAULT_TEMPERATURE)
    dimensions = os.getenv("EMBEDDING_DIM")
-    embedding_config = {
-        "api_key": api_key,
-        "deployment_name": embedding_deployment,
-        "model": os.getenv("EMBEDDING_MODEL"),
-        "dimensions": int(dimensions) if dimensions is not None else None,
+
+    azure_config = {
+        "api_key": os.environ["AZURE_OPENAI_KEY"],
+        "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
+        "api_version": os.getenv("AZURE_OPENAI_API_VERSION")
+        or os.getenv("OPENAI_API_VERSION"),
    }
-    Settings.embed_model = AzureOpenAIEmbedding(**embedding_config)
+
+    Settings.llm = AzureOpenAI(
+        model=os.getenv("MODEL"),
+        max_tokens=int(max_tokens) if max_tokens is not None else None,
+        temperature=float(temperature),
+        deployment_name=llm_deployment,
+        **azure_config,
+    )
+
+    Settings.embed_model = AzureOpenAIEmbedding(
+        model=os.getenv("EMBEDDING_MODEL"),
+        dimensions=int(dimensions) if dimensions is not None else None,
+        deployment_name=embedding_deployment,
+        **azure_config,
+    )


 def init_fastembed():
@@ -108,7 +114,7 @@ def init_fastembed():
        # Small and multilingual
        "all-MiniLM-L6-v2": "sentence-transformers/all-MiniLM-L6-v2",
        # Large and multilingual
-        "paraphrase-multilingual-mpnet-base-v2": "sentence-transformers/paraphrase-multilingual-mpnet-base-v2",   # noqa: E501
+        "paraphrase-multilingual-mpnet-base-v2": "sentence-transformers/paraphrase-multilingual-mpnet-base-v2",  # noqa: E501
    }

    # This will download the model automatically if it is not already downloaded
@@ -116,6 +122,7 @@ def init_fastembed():
        model_name=embed_model_map[os.getenv("EMBEDDING_MODEL")]
    )

+
 def init_groq():
    from llama_index.llms.groq import Groq

@@ -125,7 +132,6 @@ def init_groq():
        "mixtral-8x7b": "mixtral-8x7b-32768",
    }

-
    Settings.llm = Groq(model=model_map[os.getenv("MODEL")])
    # Groq does not provide embeddings, so we use FastEmbed instead
    init_fastembed()
@@ -21,6 +21,7 @@ def generate_datasource():
    project_name = os.getenv("LLAMA_CLOUD_PROJECT_NAME")
    api_key = os.getenv("LLAMA_CLOUD_API_KEY")
    base_url = os.getenv("LLAMA_CLOUD_BASE_URL")
+    organization_id = os.getenv("LLAMA_CLOUD_ORGANIZATION_ID")

    if name is None or project_name is None or api_key is None:
        raise ValueError(
@@ -30,12 +31,17 @@ def generate_datasource():

    documents = get_documents()

+    # Set private=false to mark the document as public (required for filtering)
+    for doc in documents:
+        doc.metadata["private"] = "false"
+
    LlamaCloudIndex.from_documents(
        documents=documents,
        name=name,
        project_name=project_name,
        api_key=api_key,
        base_url=base_url,
+        organization_id=organization_id
    )

    logger.info("Finished generating the index")
@@ -11,6 +11,7 @@ def get_index():
    project_name = os.getenv("LLAMA_CLOUD_PROJECT_NAME")
    api_key = os.getenv("LLAMA_CLOUD_API_KEY")
    base_url = os.getenv("LLAMA_CLOUD_BASE_URL")
+    organization_id = os.getenv("LLAMA_CLOUD_ORGANIZATION_ID")

    if name is None or project_name is None or api_key is None:
        raise ValueError(
@@ -23,6 +24,7 @@ def get_index():
        project_name=project_name,
        api_key=api_key,
        base_url=base_url,
+        organization_id=organization_id
    )

    return index
@@ -2,14 +2,14 @@ from dotenv import load_dotenv

 load_dotenv()

-import os
 import logging
+import os
+
+from app.engine.loaders import get_documents
+from app.settings import init_settings
 from llama_index.core.indices import (
    VectorStoreIndex,
 )
-from app.engine.loaders import get_documents
-from app.settings import init_settings
-

 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger()
@@ -1,5 +1,6 @@
 import * as dotenv from "dotenv";
 import { LlamaCloudIndex } from "llamaindex";
+import { getDataSource } from "./index";
 import { getDocuments } from "./loader";
 import { initSettings } from "./settings";
 import { checkRequiredEnvVars } from "./shared";
@@ -8,6 +9,14 @@ dotenv.config();

 async function loadAndIndex() {
  const documents = await getDocuments();
+  // Set private=false to mark the document as public (required for filtering)
+  for (const document of documents) {
+    document.metadata = {
+      ...document.metadata,
+      private: "false",
+    };
+  }
+  await getDataSource();
  await LlamaCloudIndex.fromDocuments({
    documents,
    name: process.env.LLAMA_CLOUD_INDEX_NAME!,
@@ -4,7 +4,6 @@ const REQUIRED_ENV_VARS = [
  "MILVUS_ADDRESS",
  "MILVUS_USERNAME",
  "MILVUS_PASSWORD",
-  "MILVUS_COLLECTION",
 ];

 export function getMilvusClient() {
@@ -19,8 +18,13 @@ export function getMilvusClient() {
  });
 }

-export function checkRequiredEnvVars() {
-  const missingEnvVars = REQUIRED_ENV_VARS.filter((envVar) => {
+export function checkRequiredEnvVars(opts?: { checkCollectionEnv?: boolean }) {
+  const shouldCheckCollectionEnv = opts?.checkCollectionEnv ?? true; // default to true
+  const requiredEnvVars = [...REQUIRED_ENV_VARS]; // create a copy of the array
+  if (shouldCheckCollectionEnv) {
+    requiredEnvVars.push("MILVUS_COLLECTION");
+  }
+  const missingEnvVars = requiredEnvVars.filter((envVar) => {
    return !process.env[envVar];
  });

@@ -25,6 +25,11 @@ async function generateDatasource() {
      persistDir: STORAGE_CACHE_DIR,
    });
    const documents = await getDocuments();
+    //  Set private=false to mark the document as public (required for filtering)
+    documents.forEach((doc) => {
+      doc.metadata["private"] = "false";
+    });
+
    await VectorStoreIndex.fromDocuments(documents, {
      storageContext,
    });
@@ -0,0 +1,68 @@
+This is a [LlamaIndex](https://www.llamaindex.ai/) project using [FastAPI](https://fastapi.tiangolo.com/) bootstrapped with [`create-llama`](https://github.com/run-llama/LlamaIndexTS/tree/main/packages/create-llama) featuring [structured extraction](https://docs.llamaindex.ai/en/stable/examples/structured_outputs/structured_outputs/?h=structured+output).
+
+## Getting Started
+
+First, setup the environment with poetry:
+
+> **_Note:_** This step is not needed if you are using the dev-container.
+
+```shell
+poetry install
+poetry shell
+```
+
+Then check the parameters that have been pre-configured in the `.env` file in this directory. (E.g. you might need to configure an `OPENAI_API_KEY` if you're using OpenAI as model provider).
+
+Second, generate the embeddings of the documents in the `./data` directory (if this folder exists - otherwise, skip this step):
+
+```shell
+poetry run generate
+```
+
+Third, run the API in one command:
+
+```shell
+poetry run python main.py
+```
+
+The example provides the `/api/extractor/query` API endpoint.
+
+This query endpoint returns structured data in the format of the [Output](./app/api/routers/output.py) class. Modify this class to change the output format.
+
+You can test the endpoint with the following curl request:
+
+```shell
+curl --location 'localhost:8000/api/extractor/query' \
+--header 'Content-Type: application/json' \
+--data '{ "query": "What is the maximum weight for a parcel?" }'
+```
+
+Which will return a response that the RAG pipeline is confident about the answer.
+
+Try
+
+```shell
+curl --location 'localhost:8000/api/extractor/query' \
+--header 'Content-Type: application/json' \
+--data '{ "query": "What is the weather today?" }'
+```
+
+To retrieve a response with low confidence since the question is not related to the provided document in the `./data` directory.
+
+You can start editing the API endpoint by modifying [`extractor.py`](./app/api/routers/extractor.py). The endpoints auto-update as you save the file.
+
+Open [http://localhost:8000/docs](http://localhost:8000/docs) with your browser to see the Swagger UI of the API.
+
+The API allows CORS for all origins to simplify development. You can change this behavior by setting the `ENVIRONMENT` environment variable to `prod`:
+
+```
+ENVIRONMENT=prod python main.py
+```
+
+## Learn More
+
+To learn more about LlamaIndex, take a look at the following resources:
+
+- [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex.
+
+You can check out [the LlamaIndex GitHub repository](https://github.com/run-llama/llama_index) - your feedback and contributions are welcome!
@@ -0,0 +1,58 @@
+import logging
+import os
+
+from fastapi import APIRouter, HTTPException
+from llama_index.core.settings import Settings
+from pydantic import BaseModel
+
+from app.api.routers.output import Output
+from app.engine.index import get_index
+
+extractor_router = r = APIRouter()
+
+logger = logging.getLogger("uvicorn")
+
+
+class RequestData(BaseModel):
+    query: str
+
+    class Config:
+        json_schema_extra = {
+            "examples": [
+                {"query": "What's the maximum weight for a parcel?"},
+            ],
+        }
+
+
+@r.post("/query")
+async def query_request(
+    data: RequestData,
+):
+    # Create a query engine using that returns responses in the format of the Output class
+    query_engine = get_query_engine(Output)
+
+    response = await query_engine.aquery(data.query)
+
+    output_data = response.response.dict()
+    return Output(**output_data)
+
+
+def get_query_engine(output_cls: BaseModel):
+    top_k = os.getenv("TOP_K", 3)
+
+    index = get_index()
+    if index is None:
+        raise HTTPException(
+            status_code=500,
+            detail=str(
+                "StorageContext is empty - call 'poetry run generate' to generate the storage first"
+            ),
+        )
+
+    sllm = Settings.llm.as_structured_llm(output_cls)
+
+    return index.as_query_engine(
+        similarity_top_k=int(top_k),
+        llm=sllm,
+        response_mode="tree_summarize",
+    )
@@ -0,0 +1,32 @@
+import logging
+from llama_index.core.schema import BaseModel, Field
+from typing import List
+
+logger = logging.getLogger("uvicorn")
+
+
+class Output(BaseModel):
+    response: str = Field(..., description="The answer to the question.")
+    page_numbers: List[int] = Field(
+        ...,
+        description="The page numbers of the sources used to answer this question. Do not include a page number if the context is irrelevant.",
+    )
+    confidence: float = Field(
+        ...,
+        ge=0,
+        le=1,
+        description="Confidence value between 0-1 of the correctness of the result.",
+    )
+    confidence_explanation: str = Field(
+        ..., description="Explanation for the confidence score"
+    )
+
+    class Config:
+        json_schema_extra = {
+            "example": {
+                "response": "This is an example answer.",
+                "page_numbers": [1, 2, 3],
+                "confidence": 0.85,
+                "confidence_explanation": "This is an explanation for the confidence score.",
+            }
+        }
@@ -0,0 +1,45 @@
+from dotenv import load_dotenv
+
+load_dotenv()
+
+import logging
+import os
+import uvicorn
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import RedirectResponse
+from app.api.routers.extractor import extractor_router
+from app.settings import init_settings
+
+
+app = FastAPI()
+
+init_settings()
+
+environment = os.getenv("ENVIRONMENT", "dev")  # Default to 'development' if not set
+logger = logging.getLogger("uvicorn")
+
+if environment == "dev":
+    logger.warning("Running in development mode - allowing CORS for all origins")
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=["*"],
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+    # Redirect to documentation page when accessing base URL
+    @app.get("/")
+    async def redirect_to_docs():
+        return RedirectResponse(url="/docs")
+
+
+app.include_router(extractor_router, prefix="/api/extractor")
+
+if __name__ == "__main__":
+    app_host = os.getenv("APP_HOST", "0.0.0.0")
+    app_port = int(os.getenv("APP_PORT", "8000"))
+    reload = True if environment == "dev" else False
+
+    uvicorn.run(app="main:app", host=app_host, port=app_port, reload=reload)
@@ -0,0 +1,21 @@
+[tool.poetry]
+name = "app"
+version = "0.1.0"
+description = ""
+authors = ["Marcus Schiesser <mail@marcusschiesser.de>"]
+readme = "README.md"
+
+[tool.poetry.scripts]
+generate = "app.engine.generate:generate_datasource"
+
+[tool.poetry.dependencies]
+python = "^3.11,<3.12"
+fastapi = "^0.109.1"
+uvicorn = { extras = ["standard"], version = "^0.23.2" }
+python-dotenv = "^1.0.0"
+llama-index = "^0.10.58"
+cachetools = "^5.3.3"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
@@ -20,7 +20,7 @@
    "dotenv": "^16.3.1",
    "duck-duck-scrape": "^2.2.5",
    "express": "^4.18.2",
-    "llamaindex": "0.4.6",
+    "llamaindex": "0.5.12",
    "pdf2json": "3.0.5",
    "ajv": "^8.12.0",
    "@e2b/code-interpreter": "^0.0.5",
@@ -39,7 +39,7 @@
    "prettier": "^3.2.5",
    "prettier-plugin-organize-imports": "^3.2.4",
    "tsx": "^4.7.2",
-    "tsup": "^8.0.1",
+    "tsup": "8.1.0",
    "typescript": "^5.3.2"
  }
 }
@@ -1,5 +1,6 @@
 import { Request, Response } from "express";
-import { uploadDocument } from "./llamaindex/documents/documents";
+import { getDataSource } from "./engine";
+import { uploadDocument } from "./llamaindex/documents/upload";

 export const chatUpload = async (req: Request, res: Response) => {
  const { base64 }: { base64: string } = req.body;
@@ -8,5 +9,6 @@ export const chatUpload = async (req: Request, res: Response) => {
      error: "base64 is required in the request body",
    });
  }
-  return res.status(200).json(await uploadDocument(base64));
+  const index = await getDataSource();
+  return res.status(200).json(await uploadDocument(index, base64));
 };
@@ -67,7 +67,11 @@ export const chat = async (req: Request, res: Response) => {
    });

    // Return a stream, which can be consumed by the Vercel/AI client
-    const stream = LlamaIndexStream(response, vercelStreamData);
+    const stream = LlamaIndexStream(
+      response,
+      vercelStreamData,
+      messages as ChatMessage[],
+    );

    return streamToResponse(stream, res, {}, vercelStreamData);
  } catch (error) {
@@ -45,6 +45,9 @@ export const initSettings = async () => {
    case "mistral":
      initMistralAI();
      break;
+    case "azure-openai":
+      initAzureOpenAI();
+      break;
    default:
      initOpenAI();
      break;
@@ -55,7 +58,7 @@ export const initSettings = async () => {

 function initOpenAI() {
  Settings.llm = new OpenAI({
-    model: process.env.MODEL ?? "gpt-3.5-turbo",
+    model: process.env.MODEL ?? "gpt-4o-mini",
    maxTokens: process.env.LLM_MAX_TOKENS
      ? Number(process.env.LLM_MAX_TOKENS)
      : undefined,
@@ -68,6 +71,53 @@ function initOpenAI() {
  });
 }

+function initAzureOpenAI() {
+  // Map Azure OpenAI model names to OpenAI model names (only for TS)
+  const AZURE_OPENAI_MODEL_MAP: Record<string, string> = {
+    "gpt-35-turbo": "gpt-3.5-turbo",
+    "gpt-35-turbo-16k": "gpt-3.5-turbo-16k",
+    "gpt-4o": "gpt-4o",
+    "gpt-4": "gpt-4",
+    "gpt-4-32k": "gpt-4-32k",
+    "gpt-4-turbo": "gpt-4-turbo",
+    "gpt-4-turbo-2024-04-09": "gpt-4-turbo",
+    "gpt-4-vision-preview": "gpt-4-vision-preview",
+    "gpt-4-1106-preview": "gpt-4-1106-preview",
+    "gpt-4o-2024-05-13": "gpt-4o-2024-05-13",
+  };
+
+  const azureConfig = {
+    apiKey: process.env.AZURE_OPENAI_KEY,
+    endpoint: process.env.AZURE_OPENAI_ENDPOINT,
+    apiVersion:
+      process.env.AZURE_OPENAI_API_VERSION || process.env.OPENAI_API_VERSION,
+  };
+
+  Settings.llm = new OpenAI({
+    model:
+      AZURE_OPENAI_MODEL_MAP[process.env.MODEL ?? "gpt-35-turbo"] ??
+      "gpt-3.5-turbo",
+    maxTokens: process.env.LLM_MAX_TOKENS
+      ? Number(process.env.LLM_MAX_TOKENS)
+      : undefined,
+    azure: {
+      ...azureConfig,
+      deployment: process.env.AZURE_OPENAI_LLM_DEPLOYMENT,
+    },
+  });
+
+  Settings.embedModel = new OpenAIEmbedding({
+    model: process.env.EMBEDDING_MODEL,
+    dimensions: process.env.EMBEDDING_DIM
+      ? parseInt(process.env.EMBEDDING_DIM)
+      : undefined,
+    azure: {
+      ...azureConfig,
+      deployment: process.env.AZURE_OPENAI_EMBEDDING_DEPLOYMENT,
+    },
+  });
+}
+
 function initOllama() {
  const config = {
    host: process.env.OLLAMA_BASE_URL ?? "http://127.0.0.1:11434",
@@ -1,87 +0,0 @@
-import os
-import base64
-import mimetypes
-from uuid import uuid4
-from pathlib import Path
-from typing import List, Dict
-from llama_index.core import VectorStoreIndex
-from llama_index.readers.file import FlatReader
-from llama_index.core.readers.file.base import (
-    _try_loading_included_file_formats as get_file_loaders_map,
-    default_file_metadata_func,
-)
-from llama_index.core.schema import Document
-from llama_index.core.ingestion import IngestionPipeline
-from app.engine.index import get_index
-
-
-
-def file_metadata_func(*args, **kwargs) -> Dict:
-    default_meta = default_file_metadata_func(*args, **kwargs)
-    default_meta["private"] = "true"
-    return default_meta
-
-def file_loaders_map():
-    default_loaders = get_file_loaders_map()
-    default_loaders[".txt"] = FlatReader
-    return default_loaders
-
-
-
-class FileController:
-
-    PRIVATE_STORE_PATH="output/uploaded"
-
-    @staticmethod
-    def preprocess_base64_file(base64_content: str) -> tuple:
-        header, data = base64_content.split(",", 1)
-        mime_type = header.split(";")[0].split(":", 1)[1]
-        extension = mimetypes.guess_extension(mime_type)
-        # File data as bytes
-        data = base64.b64decode(data)
-        return data, extension
-
-    @staticmethod
-    def store_and_parse_file(file_data, extension) -> List[Document]:
-        # Store file to the private directory
-        os.makedirs(FileController.PRIVATE_STORE_PATH, exist_ok=True)
-
-        # random file name
-        file_name = f"{uuid4().hex}{extension}"
-        file_path = Path(os.path.join(FileController.PRIVATE_STORE_PATH, file_name))
-
-        # write file
-        with open(file_path, "wb") as f:
-            f.write(file_data)
-
-        # Load file to documents
-        reader_cls = file_loaders_map().get(extension)
-        if reader_cls is None:
-            raise ValueError(f"File extension {extension} is not supported")
-        documents = reader_cls().load_data(file_path)
-        # Add custom metadata
-        for doc in documents:
-            doc.metadata["private"] = "true"
-        return documents
-
-    @staticmethod
-    def process_file(base64_content: str) -> List[str]:
-        file_data, extension = FileController.preprocess_base64_file(base64_content)
-        documents = FileController.store_and_parse_file(file_data, extension)
-
-        # Only process nodes, no store the index
-        pipeline = IngestionPipeline()
-        nodes = pipeline.run(documents=documents)
-
-        # Add the nodes to the index and persist it
-        current_index = get_index()
-        if current_index is None:
-            current_index = VectorStoreIndex(nodes=nodes)
-        else:
-            current_index.insert_nodes(nodes=nodes)
-        current_index.storage_context.persist(
-            persist_dir=os.environ.get("STORAGE_DIR", "storage")
-        )
-
-        # Return the document ids
-        return [doc.doc_id for doc in documents]
@@ -1,32 +1,50 @@
-import os
 import logging
+import os
+from typing import List

-from aiostream import stream
-from fastapi import APIRouter, Depends, HTTPException, Request, status
+from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request, status
+from llama_index.core.chat_engine.types import BaseChatEngine, NodeWithScore
 from llama_index.core.llms import MessageRole
-from llama_index.core.chat_engine.types import BaseChatEngine
-from llama_index.core.vector_stores.types import MetadataFilters, MetadataFilter
-from app.engine import get_chat_engine
-from app.api.routers.vercel_response import VercelStreamResponse
+from llama_index.core.vector_stores.types import MetadataFilter, MetadataFilters
+
 from app.api.routers.events import EventCallbackHandler
 from app.api.routers.models import (
-    ChatData,
    ChatConfig,
-    SourceNodes,
-    Result,
+    ChatData,
    Message,
+    Result,
+    SourceNodes,
 )
+from app.api.routers.vercel_response import VercelStreamResponse
+from app.api.services.llama_cloud import LLamaCloudFileService
+from app.engine import get_chat_engine

 chat_router = r = APIRouter()

 logger = logging.getLogger("uvicorn")


+def process_response_nodes(
+    nodes: List[NodeWithScore],
+    background_tasks: BackgroundTasks,
+):
+    """
+    Start background tasks on the source nodes if needed.
+    """
+    files_to_download = SourceNodes.get_download_files(nodes)
+    for file in files_to_download:
+        background_tasks.add_task(
+            LLamaCloudFileService.download_llamacloud_pipeline_file, file
+        )
+
+
 # streaming endpoint - delete if not needed
@r.post("")
 async def chat(
    request: Request,
    data: ChatData,
+    background_tasks: BackgroundTasks,
+    chat_engine: BaseChatEngine = Depends(get_chat_engine),
 ):
    try:
        last_message_content = data.get_last_message_content()
@@ -41,8 +59,9 @@ async def chat(
        chat_engine.callback_manager.handlers.append(event_handler)  # type: ignore

        response = await chat_engine.astream_chat(last_message_content, messages)
+        process_response_nodes(response.source_nodes, background_tasks)

-        return VercelStreamResponse(request, event_handler, response)
+        return VercelStreamResponse(request, event_handler, response, data)
    except Exception as e:
        logger.exception("Error in chat engine", exc_info=True)
        raise HTTPException(
@@ -57,8 +76,8 @@ def generate_filters(doc_ids):
            filters=[
                MetadataFilter(
                    key="private",
-                    value="true",
-                    operator="!=",  # type: ignore
+                    value=["true"],
+                    operator="nin",  # type: ignore
                ),
                MetadataFilter(
                    key="doc_id",
@@ -70,11 +89,12 @@ def generate_filters(doc_ids):
        )
    else:
        filters = MetadataFilters(
+            # Use the "NIN" - "not in" operator to include all public documents (don't have the private key set)
            filters=[
                MetadataFilter(
                    key="private",
-                    value="true",
-                    operator="!=",  # type: ignore
+                    value=["true"],
+                    operator="nin",  # type: ignore
                ),
            ]
        )
@@ -1,11 +1,11 @@
-import os
 import logging
+import os
+from typing import Any, Dict, List, Literal, Optional, Set
+
+from llama_index.core.llms import ChatMessage, MessageRole
+from llama_index.core.schema import NodeWithScore
 from pydantic import BaseModel, Field, validator
 from pydantic.alias_generators import to_camel
-from typing import List, Any, Optional, Dict, Literal
-from llama_index.core.schema import NodeWithScore
-from llama_index.core.llms import ChatMessage, MessageRole
-

 logger = logging.getLogger("uvicorn")

@@ -25,7 +25,7 @@ class File(BaseModel):
    filetype: str


-class AnnotationData(BaseModel):
+class AnnotationFileData(BaseModel):
    files: List[File] = Field(
        default=[],
        description="List of files",
@@ -50,7 +50,7 @@ class AnnotationData(BaseModel):

 class Annotation(BaseModel):
    type: str
-    data: AnnotationData
+    data: AnnotationFileData | List[str]

    def to_content(self) -> str | None:
        if self.type == "document_file":
@@ -96,7 +96,8 @@ class ChatData(BaseModel):

    def get_last_message_content(self) -> str:
        """
-        Get the content of the last message along with the data content if available. Fallback to use data content from previous messages
+        Get the content of the last message along with the data content if available.
+        Fallback to use data content from previous messages
        """
        if len(self.messages) == 0:
            raise ValueError("There is not any message in the chat")
@@ -131,7 +132,7 @@ class ChatData(BaseModel):
        """
        Get the document IDs from the chat messages
        """
-        document_ids = []
+        document_ids: List[str] = []
        for message in self.messages:
            if message.role == MessageRole.USER and message.annotations is not None:
                for annotation in message.annotations:
@@ -145,6 +146,21 @@ class ChatData(BaseModel):
        return list(set(document_ids))


+class LlamaCloudFile(BaseModel):
+    file_name: str
+    pipeline_id: str
+
+    def __eq__(self, other):
+        if not isinstance(other, LlamaCloudFile):
+            return NotImplemented
+        return (
+            self.file_name == other.file_name and self.pipeline_id == other.pipeline_id
+        )
+
+    def __hash__(self):
+        return hash((self.file_name, self.pipeline_id))
+
+
 class SourceNodes(BaseModel):
    id: str
    metadata: Dict[str, Any]
@@ -155,21 +171,7 @@ class SourceNodes(BaseModel):
    @classmethod
    def from_source_node(cls, source_node: NodeWithScore):
        metadata = source_node.node.metadata
-        url = metadata.get("URL")
-
-        if not url:
-            file_name = metadata.get("file_name")
-            is_private = metadata.get("private", "false") == "true"
-            url_prefix = os.getenv("FILESERVER_URL_PREFIX")
-            if not url_prefix:
-                logger.warning(
-                    "Warning: FILESERVER_URL_PREFIX not set in environment variables"
-                )
-            if file_name and url_prefix:
-                if is_private:
-                    url = f"{url_prefix}/output/uploaded/{file_name}"
-                else:
-                    url = f"{url_prefix}/data/{file_name}"
+        url = cls.get_url_from_metadata(metadata)

        return cls(
            id=source_node.node.node_id,
@@ -179,10 +181,52 @@ class SourceNodes(BaseModel):
            url=url,
        )

+    @classmethod
+    def get_url_from_metadata(cls, metadata: Dict[str, Any]) -> str:
+        url_prefix = os.getenv("FILESERVER_URL_PREFIX")
+        if not url_prefix:
+            logger.warning(
+                "Warning: FILESERVER_URL_PREFIX not set in environment variables. Can't use file server"
+            )
+        file_name = metadata.get("file_name")
+        if file_name and url_prefix:
+            # file_name exists and file server is configured
+            pipeline_id = metadata.get("pipeline_id")
+            if pipeline_id and metadata.get("private") is None:
+                # file is from LlamaCloud and was not ingested locally
+                file_name = f"{pipeline_id}${file_name}"
+                return f"{url_prefix}/output/llamacloud/{file_name}"
+            is_private = metadata.get("private", "false") == "true"
+            if is_private:
+                return f"{url_prefix}/output/uploaded/{file_name}"
+            return f"{url_prefix}/data/{file_name}"
+        else:
+            # fallback to URL in metadata (e.g. for websites)
+            return metadata.get("URL")
+
    @classmethod
    def from_source_nodes(cls, source_nodes: List[NodeWithScore]):
        return [cls.from_source_node(node) for node in source_nodes]

+    @staticmethod
+    def get_download_files(nodes: List[NodeWithScore]) -> Set[LlamaCloudFile]:
+        source_nodes = SourceNodes.from_source_nodes(nodes)
+        llama_cloud_files = [
+            LlamaCloudFile(
+                file_name=node.metadata.get("file_name"),
+                pipeline_id=node.metadata.get("pipeline_id"),
+            )
+            for node in source_nodes
+            if (
+                node.metadata.get("private")
+                is None  # Only download files are from LlamaCloud and were not ingested locally
+                and node.metadata.get("pipeline_id") is not None
+                and node.metadata.get("file_name") is not None
+            )
+        ]
+        # Remove duplicates and return
+        return set(llama_cloud_files)
+

 class Result(BaseModel):
    result: Message
@@ -193,6 +237,7 @@ class ChatConfig(BaseModel):
    starter_questions: Optional[List[str]] = Field(
        default=None,
        description="List of starter questions",
+        serialization_alias="starterQuestions"
    )

    class Config:
@@ -204,4 +249,3 @@ class ChatConfig(BaseModel):
                ]
            }
        }
-        alias_generator = to_camel
@@ -1,9 +1,10 @@
 import logging
 from typing import List
+
+from fastapi import APIRouter, HTTPException
 from pydantic import BaseModel
-from fastapi import HTTPException
-from fastapi import APIRouter
-from app.api.controllers.file import FileController
+
+from app.api.services.file import PrivateFileService

 file_upload_router = r = APIRouter()

@@ -18,7 +19,7 @@ class FileUploadRequest(BaseModel):
 def upload_file(request: FileUploadRequest) -> List[str]:
    try:
        logger.info("Processing file")
-        return FileController.process_file(request.base64)
+        return PrivateFileService.process_file(request.base64)
    except Exception as e:
        logger.error(f"Error processing file: {e}", exc_info=True)
        raise HTTPException(status_code=500, detail="Error processing file")
@@ -1,13 +1,13 @@
 import json
-from typing import Any
+
+from aiostream import stream
+from fastapi import Request
 from fastapi.responses import StreamingResponse
+from llama_index.core.chat_engine.types import StreamingAgentChatResponse

 from app.api.routers.events import EventCallbackHandler
-from aiostream import stream
-
-from app.api.routers.models import SourceNodes
-from fastapi import Request
-from llama_index.core.chat_engine.types import StreamingAgentChatResponse
+from app.api.routers.models import ChatData, Message, SourceNodes
+from app.api.services.suggestion import NextQuestionSuggestion


 class VercelStreamResponse(StreamingResponse):
@@ -34,9 +34,10 @@ class VercelStreamResponse(StreamingResponse):
        request: Request,
        event_handler: EventCallbackHandler,
        response: StreamingAgentChatResponse,
+        chat_data: ChatData,
    ):
        content = VercelStreamResponse.content_generator(
-            request, event_handler, response
+            request, event_handler, response, chat_data
        )
        super().__init__(content=content)

@@ -46,16 +47,35 @@ class VercelStreamResponse(StreamingResponse):
        request: Request,
        event_handler: EventCallbackHandler,
        response: StreamingAgentChatResponse,
+        chat_data: ChatData,
    ):
        # Yield the text response
        async def _chat_response_generator():
+            final_response = ""
            async for token in response.async_response_gen():
+                final_response += token
                yield VercelStreamResponse.convert_text(token)
+
+            # Generate questions that user might interested to
+            conversation = chat_data.messages + [
+                Message(role="assistant", content=final_response)
+            ]
+            questions = await NextQuestionSuggestion.suggest_next_questions(
+                conversation
+            )
+            if len(questions) > 0:
+                yield VercelStreamResponse.convert_data(
+                    {
+                        "type": "suggested_questions",
+                        "data": questions,
+                    }
+                )
+
            # the text_generator is the leading stream, once it's finished, also finish the event stream
            event_handler.is_done = True

            # Yield the source nodes
-            yield VercelStreamResponse.convert_data(
+            yield cls.convert_data(
                {
                    "type": "sources",
                    "data": {
@@ -0,0 +1,113 @@
+import base64
+import mimetypes
+import os
+from pathlib import Path
+from typing import Dict, List
+from uuid import uuid4
+
+from app.engine.index import get_index
+from llama_index.core import VectorStoreIndex
+from llama_index.core.ingestion import IngestionPipeline
+from llama_index.core.readers.file.base import (
+    _try_loading_included_file_formats as get_file_loaders_map,
+)
+from llama_index.core.readers.file.base import (
+    default_file_metadata_func,
+)
+from llama_index.core.schema import Document
+from llama_index.indices.managed.llama_cloud.base import LlamaCloudIndex
+from llama_index.readers.file import FlatReader
+
+
+def get_llamaparse_parser():
+    from app.engine.loaders import load_configs
+    from app.engine.loaders.file import FileLoaderConfig, llama_parse_parser
+
+    config = load_configs()
+    file_loader_config = FileLoaderConfig(**config["file"])
+    if file_loader_config.use_llama_parse:
+        return llama_parse_parser()
+    else:
+        return None
+
+
+def default_file_loaders_map():
+    default_loaders = get_file_loaders_map()
+    default_loaders[".txt"] = FlatReader
+    return default_loaders
+
+
+class PrivateFileService:
+    PRIVATE_STORE_PATH = "output/uploaded"
+
+    @staticmethod
+    def preprocess_base64_file(base64_content: str) -> tuple:
+        header, data = base64_content.split(",", 1)
+        mime_type = header.split(";")[0].split(":", 1)[1]
+        extension = mimetypes.guess_extension(mime_type)
+        # File data as bytes
+        return base64.b64decode(data), extension
+
+    @staticmethod
+    def store_and_parse_file(file_data, extension) -> List[Document]:
+        # Store file to the private directory
+        os.makedirs(PrivateFileService.PRIVATE_STORE_PATH, exist_ok=True)
+
+        # random file name
+        file_name = f"{uuid4().hex}{extension}"
+        file_path = Path(os.path.join(PrivateFileService.PRIVATE_STORE_PATH, file_name))
+
+        # write file
+        with open(file_path, "wb") as f:
+            f.write(file_data)
+
+        # Load file to documents
+        # If LlamaParse is enabled, use it to parse the file
+        # Otherwise, use the default file loaders
+        reader = get_llamaparse_parser()
+        if reader is None:
+            reader_cls = default_file_loaders_map().get(extension)
+            if reader_cls is None:
+                raise ValueError(f"File extension {extension} is not supported")
+            reader = reader_cls()
+        documents = reader.load_data(file_path)
+        # Add custom metadata
+        for doc in documents:
+            doc.metadata["file_name"] = file_name
+            doc.metadata["private"] = "true"
+        return documents
+
+    @staticmethod
+    def process_file(base64_content: str) -> List[str]:
+        file_data, extension = PrivateFileService.preprocess_base64_file(base64_content)
+        documents = PrivateFileService.store_and_parse_file(file_data, extension)
+
+        # Only process nodes, no store the index
+        pipeline = IngestionPipeline()
+        nodes = pipeline.run(documents=documents)
+
+        # Add the nodes to the index and persist it
+        current_index = get_index()
+
+        # Insert the documents into the index
+        if isinstance(current_index, LlamaCloudIndex):
+            # LlamaCloudIndex is a managed index so we don't need to process the nodes
+            # just insert the documents
+            for doc in documents:
+                current_index.insert(doc)
+        else:
+            # Only process nodes, no store the index
+            pipeline = IngestionPipeline()
+            nodes = pipeline.run(documents=documents)
+
+            # Add the nodes to the index and persist it
+            if current_index is None:
+                current_index = VectorStoreIndex(nodes=nodes)
+            else:
+                current_index.insert_nodes(nodes=nodes)
+            current_index.storage_context.persist(
+                persist_dir=os.environ.get("STORAGE_DIR", "storage")
+            )
+
+        # Return the document ids
+        return [doc.doc_id for doc in documents]
@@ -0,0 +1,88 @@
+import logging
+import os
+from typing import Any, Dict, List, Optional
+
+import requests
+from app.api.routers.models import LlamaCloudFile
+
+logger = logging.getLogger("uvicorn")
+
+
+class LLamaCloudFileService:
+    LLAMA_CLOUD_URL = "https://cloud.llamaindex.ai/api/v1"
+    LOCAL_STORE_PATH = "output/llamacloud"
+
+    DOWNLOAD_FILE_NAME_TPL = "{pipeline_id}${filename}"
+
+    @classmethod
+    def _get_files(cls, pipeline_id: str) -> List[Dict[str, Any]]:
+        url = f"{cls.LLAMA_CLOUD_URL}/pipelines/{pipeline_id}/files"
+        return cls._make_request(url)
+
+    @classmethod
+    def _get_file_detail(cls, project_id: str, file_id: str) -> Dict[str, Any]:
+        url = f"{cls.LLAMA_CLOUD_URL}/files/{file_id}/content?project_id={project_id}"
+        return cls._make_request(url)
+
+    @classmethod
+    def _download_file(cls, url: str, local_file_path: str):
+        logger.info(f"Downloading file to {local_file_path}")
+        # Create directory if it doesn't exist
+        os.makedirs(cls.LOCAL_STORE_PATH, exist_ok=True)
+        # Download the file
+        with requests.get(url, stream=True) as r:
+            r.raise_for_status()
+            with open(local_file_path, "wb") as f:
+                for chunk in r.iter_content(chunk_size=8192):
+                    f.write(chunk)
+        logger.info("File downloaded successfully")
+
+    @classmethod
+    def download_llamacloud_pipeline_file(
+        cls,
+        file: LlamaCloudFile,
+        force_download: bool = False,
+    ):
+        file_name = file.file_name
+        pipeline_id = file.pipeline_id
+
+        # Check is the file already exists
+        downloaded_file_path = cls.get_file_path(file_name, pipeline_id)
+        if os.path.exists(downloaded_file_path) and not force_download:
+            logger.debug(f"File {file_name} already exists in local storage")
+            return
+        try:
+            logger.info(f"Downloading file {file_name} for pipeline {pipeline_id}")
+            files = cls._get_files(pipeline_id)
+            if not files or not isinstance(files, list):
+                raise Exception("No files found in LlamaCloud")
+            for file_entry in files:
+                if file_entry["name"] == file_name:
+                    file_id = file_entry["file_id"]
+                    project_id = file_entry["project_id"]
+                    file_detail = cls._get_file_detail(project_id, file_id)
+                    cls._download_file(file_detail["url"], downloaded_file_path)
+                    break
+        except Exception as error:
+            logger.info(f"Error fetching file from LlamaCloud: {error}")
+
+    @classmethod
+    def get_file_name(cls, name: str, pipeline_id: str) -> str:
+        return cls.DOWNLOAD_FILE_NAME_TPL.format(pipeline_id=pipeline_id, filename=name)
+
+    @classmethod
+    def get_file_path(cls, name: str, pipeline_id: str) -> str:
+        return os.path.join(cls.LOCAL_STORE_PATH, cls.get_file_name(name, pipeline_id))
+
+    @staticmethod
+    def _make_request(
+        url: str, data=None, headers: Optional[Dict] = None, method: str = "get"
+    ):
+        if headers is None:
+            headers = {
+                "Accept": "application/json",
+                "Authorization": f'Bearer {os.getenv("LLAMA_CLOUD_API_KEY")}',
+            }
+        response = requests.request(method, url, headers=headers, data=data)
+        response.raise_for_status()
+        return response.json()
@@ -0,0 +1,48 @@
+from typing import List
+
+from app.api.routers.models import Message
+from llama_index.core.prompts import PromptTemplate
+from llama_index.core.settings import Settings
+from pydantic import BaseModel
+
+NEXT_QUESTIONS_SUGGESTION_PROMPT = PromptTemplate(
+    "You're a helpful assistant! Your task is to suggest the next question that user might ask. "
+    "\nHere is the conversation history"
+    "\n---------------------\n{conversation}\n---------------------"
+    "Given the conversation history, please give me $number_of_questions questions that you might ask next!"
+)
+N_QUESTION_TO_GENERATE = 3
+
+
+class NextQuestions(BaseModel):
+    """A list of questions that user might ask next"""
+
+    questions: List[str]
+
+
+class NextQuestionSuggestion:
+    @staticmethod
+    async def suggest_next_questions(
+        messages: List[Message],
+        number_of_questions: int = N_QUESTION_TO_GENERATE,
+    ) -> List[str]:
+        # Reduce the cost by only using the last two messages
+        last_user_message = None
+        last_assistant_message = None
+        for message in reversed(messages):
+            if message.role == "user":
+                last_user_message = f"User: {message.content}"
+            elif message.role == "assistant":
+                last_assistant_message = f"Assistant: {message.content}"
+            if last_user_message and last_assistant_message:
+                break
+        conversation: str = f"{last_user_message}\n{last_assistant_message}"
+
+        output: NextQuestions = await Settings.llm.astructured_predict(
+            NextQuestions,
+            prompt=NEXT_QUESTIONS_SUGGESTION_PROMPT,
+            conversation=conversation,
+            nun_questions=number_of_questions,
+        )
+
+        return output.questions
@@ -1,5 +0,0 @@
-from llama_index.core.chat_engine import SimpleChatEngine
-
-
-def get_chat_engine():
-    return SimpleChatEngine.from_defaults()
@@ -2,17 +2,17 @@ from dotenv import load_dotenv

 load_dotenv()

-import os
 import logging
-from llama_index.core.settings import Settings
-from llama_index.core.ingestion import IngestionPipeline
-from llama_index.core.node_parser import SentenceSplitter
-from llama_index.core.storage.docstore import SimpleDocumentStore
-from llama_index.core.storage import StorageContext
-from app.settings import init_settings
+import os
+
 from app.engine.loaders import get_documents
 from app.engine.vectordb import get_vector_store
-
+from app.settings import init_settings
+from llama_index.core.ingestion import IngestionPipeline
+from llama_index.core.node_parser import SentenceSplitter
+from llama_index.core.settings import Settings
+from llama_index.core.storage import StorageContext
+from llama_index.core.storage.docstore import SimpleDocumentStore

 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger()
@@ -64,7 +64,7 @@ def generate_datasource():

    # Get the stores and documents or create new ones
    documents = get_documents()
-    # Set the metadata for the documents
+    # Set private=false to mark the document as public (required for filtering)
    for doc in documents:
        doc.metadata["private"] = "false"
    docstore = get_doc_store()
@@ -1,61 +0,0 @@
-from llama_index.embeddings.openai import OpenAIEmbedding
-from llama_index.core.settings import Settings
-from typing import Dict
-import os
-
-DEFAULT_MODEL = "gpt-3.5-turbo"
-DEFAULT_EMBEDDING_MODEL = "text-embedding-3-large"
-
-class TSIEmbedding(OpenAIEmbedding):
-    def __init__(self, **kwargs):
-        super().__init__(**kwargs)
-        self._query_engine = self._text_engine = self.model_name
-
-def llm_config_from_env() -> Dict:
-    from llama_index.core.constants import DEFAULT_TEMPERATURE
-
-    model = os.getenv("MODEL", DEFAULT_MODEL)
-    temperature = os.getenv("LLM_TEMPERATURE", DEFAULT_TEMPERATURE)
-    max_tokens = os.getenv("LLM_MAX_TOKENS")
-    api_key = os.getenv("T_SYSTEMS_LLMHUB_API_KEY")
-    api_base = os.getenv("T_SYSTEMS_LLMHUB_BASE_URL")
-
-    config = {
-        "model": model,
-        "api_key": api_key,
-        "api_base": api_base,
-        "temperature": float(temperature),
-        "max_tokens": int(max_tokens) if max_tokens is not None else None,
-    }
-    return config
-
-
-def embedding_config_from_env() -> Dict:
-    from llama_index.core.constants import DEFAULT_EMBEDDING_DIM
-    
-    model = os.getenv("EMBEDDING_MODEL", DEFAULT_EMBEDDING_MODEL)
-    dimension = os.getenv("EMBEDDING_DIM", DEFAULT_EMBEDDING_DIM)
-    api_key = os.getenv("T_SYSTEMS_LLMHUB_API_KEY")
-    api_base = os.getenv("T_SYSTEMS_LLMHUB_BASE_URL")
-
-    config = {
-        "model_name": model,
-        "dimension": int(dimension) if dimension is not None else None,
-        "api_key": api_key,
-        "api_base": api_base,
-    }
-    return config
-
-def init_llmhub():
-    from llama_index.llms.openai_like import OpenAILike
-
-    llm_configs = llm_config_from_env()
-    embedding_configs = embedding_config_from_env()
-
-    Settings.embed_model = TSIEmbedding(**embedding_configs)
-    Settings.llm = OpenAILike(
-        **llm_configs,
-        is_chat_model=True,
-        is_function_calling_model=False,
-        context_window=4096,
-    )
@@ -21,9 +21,9 @@ init_settings()
 init_observability()

 environment = os.getenv("ENVIRONMENT", "dev")  # Default to 'development' if not set
+logger = logging.getLogger("uvicorn")

 if environment == "dev":
-    logger = logging.getLogger("uvicorn")
    logger.warning("Running in development mode - allowing CORS for all origins")
    app.add_middleware(
        CORSMiddleware,
@@ -41,7 +41,11 @@ if environment == "dev":

 def mount_static_files(directory, path):
    if os.path.exists(directory):
-        app.mount(path, StaticFiles(directory=directory), name=f"{directory}-static")
+        for dir, _, _ in os.walk(directory):
+            relative_path = os.path.relpath(dir, directory)
+            mount_path = path if relative_path == "." else f"{path}/{relative_path}"
+            logger.info(f"Mounting static files '{dir}' at {mount_path}")
+            app.mount(mount_path, StaticFiles(directory=dir), name=f"{dir}-static")


 # Mount the data files to serve the file viewer
@@ -14,7 +14,7 @@ fastapi = "^0.109.1"
 uvicorn = { extras = ["standard"], version = "^0.23.2" }
 python-dotenv = "^1.0.0"
 aiostream = "^0.5.2"
-llama-index = "0.10.54post1"
+llama-index = "0.10.58"
 cachetools = "^5.3.3"

 [build-system]
@@ -45,6 +45,9 @@ export const initSettings = async () => {
    case "mistral":
      initMistralAI();
      break;
+    case "azure-openai":
+      initAzureOpenAI();
+      break;
    default:
      initOpenAI();
      break;
@@ -55,7 +58,7 @@ export const initSettings = async () => {

 function initOpenAI() {
  Settings.llm = new OpenAI({
-    model: process.env.MODEL ?? "gpt-3.5-turbo",
+    model: process.env.MODEL ?? "gpt-4o-mini",
    maxTokens: process.env.LLM_MAX_TOKENS
      ? Number(process.env.LLM_MAX_TOKENS)
      : undefined,
@@ -68,6 +71,53 @@ function initOpenAI() {
  });
 }

+function initAzureOpenAI() {
+  // Map Azure OpenAI model names to OpenAI model names (only for TS)
+  const AZURE_OPENAI_MODEL_MAP: Record<string, string> = {
+    "gpt-35-turbo": "gpt-3.5-turbo",
+    "gpt-35-turbo-16k": "gpt-3.5-turbo-16k",
+    "gpt-4o": "gpt-4o",
+    "gpt-4": "gpt-4",
+    "gpt-4-32k": "gpt-4-32k",
+    "gpt-4-turbo": "gpt-4-turbo",
+    "gpt-4-turbo-2024-04-09": "gpt-4-turbo",
+    "gpt-4-vision-preview": "gpt-4-vision-preview",
+    "gpt-4-1106-preview": "gpt-4-1106-preview",
+    "gpt-4o-2024-05-13": "gpt-4o-2024-05-13",
+  };
+
+  const azureConfig = {
+    apiKey: process.env.AZURE_OPENAI_KEY,
+    endpoint: process.env.AZURE_OPENAI_ENDPOINT,
+    apiVersion:
+      process.env.AZURE_OPENAI_API_VERSION || process.env.OPENAI_API_VERSION,
+  };
+
+  Settings.llm = new OpenAI({
+    model:
+      AZURE_OPENAI_MODEL_MAP[process.env.MODEL ?? "gpt-35-turbo"] ??
+      "gpt-3.5-turbo",
+    maxTokens: process.env.LLM_MAX_TOKENS
+      ? Number(process.env.LLM_MAX_TOKENS)
+      : undefined,
+    azure: {
+      ...azureConfig,
+      deployment: process.env.AZURE_OPENAI_LLM_DEPLOYMENT,
+    },
+  });
+
+  Settings.embedModel = new OpenAIEmbedding({
+    model: process.env.EMBEDDING_MODEL,
+    dimensions: process.env.EMBEDDING_DIM
+      ? parseInt(process.env.EMBEDDING_DIM)
+      : undefined,
+    azure: {
+      ...azureConfig,
+      deployment: process.env.AZURE_OPENAI_EMBEDDING_DEPLOYMENT,
+    },
+  });
+}
+
 function initOllama() {
  const config = {
    host: process.env.OLLAMA_BASE_URL ?? "http://127.0.0.1:11434",
@@ -80,7 +80,11 @@ export async function POST(request: NextRequest) {
    });

    // Transform LlamaIndex stream to Vercel/AI format
-    const stream = LlamaIndexStream(response, vercelStreamData);
+    const stream = LlamaIndexStream(
+      response,
+      vercelStreamData,
+      messages as ChatMessage[],
+    );

    // Return a StreamingTextResponse, which can be consumed by the Vercel/AI client
    return new StreamingTextResponse(stream, {}, vercelStreamData);
@@ -1,6 +1,7 @@
 import { NextRequest, NextResponse } from "next/server";
+import { getDataSource } from "../engine";
 import { initSettings } from "../engine/settings";
-import { uploadDocument } from "../llamaindex/documents/documents";
+import { uploadDocument } from "../llamaindex/documents/upload";

 initSettings();

@@ -16,7 +17,13 @@ export async function POST(request: NextRequest) {
        { status: 400 },
      );
    }
-    return NextResponse.json(await uploadDocument(base64));
+    const index = await getDataSource();
+    if (!index) {
+      throw new Error(
+        `StorageContext is empty - call 'npm run generate' to generate the storage first`,
+      );
+    }
+    return NextResponse.json(await uploadDocument(index, base64));
  } catch (error) {
    console.error("[Upload API]", error);
    return NextResponse.json(
@@ -21,7 +21,9 @@ export default function ChatInput(
    | "messages"
    | "setInput"
    | "append"
-  >,
+  > & {
+    requestParams?: any;
+  },
 ) {
  const {
    imageUrl,
@@ -64,7 +66,7 @@ export default function ChatInput(
      return;
    }
    try {
-      await uploadFile(file);
+      await uploadFile(file, props.requestParams);
      props.onFileUpload?.(file);
    } catch (error: any) {
      props.onFileError?.(error.message);
@@ -90,7 +90,11 @@ function NodeInfo({ nodeInfo }: { nodeInfo: NodeInfo }) {
    // add a link to view its URL and a button to copy the URL to the clipboard
    return (
      <div className="flex items-center my-2">
-        <a className="hover:text-blue-900" href={nodeInfo.url} target="_blank">
+        <a
+          className="hover:text-blue-900 truncate"
+          href={nodeInfo.url}
+          target="_blank"
+        >
          <span>{nodeInfo.url}</span>
        </a>
        <Button
@@ -0,0 +1,32 @@
+import { useState } from "react";
+import { ChatHandler, SuggestedQuestionsData } from "..";
+
+export function SuggestedQuestions({
+  questions,
+  append,
+}: {
+  questions: SuggestedQuestionsData;
+  append: Pick<ChatHandler, "append">["append"];
+}) {
+  const [showQuestions, setShowQuestions] = useState(questions.length > 0);
+
+  return (
+    showQuestions &&
+    append !== undefined && (
+      <div className="flex flex-col space-y-2">
+        {questions.map((question, index) => (
+          <a
+            key={index}
+            onClick={() => {
+              append({ role: "user", content: question });
+              setShowQuestions(false);
+            }}
+            className="text-sm italic hover:underline cursor-pointer"
+          >
+            {"->"} {question}
+          </a>
+        ))}
+      </div>
+    )
+  );
+}
@@ -5,12 +5,14 @@ import { Fragment } from "react";
 import { Button } from "../../button";
 import { useCopyToClipboard } from "../hooks/use-copy-to-clipboard";
 import {
+  ChatHandler,
  DocumentFileData,
  EventData,
  ImageData,
  MessageAnnotation,
  MessageAnnotationType,
  SourceData,
+  SuggestedQuestionsData,
  ToolData,
  getAnnotationData,
 } from "../index";
@@ -19,6 +21,7 @@ import { ChatEvents } from "./chat-events";
 import { ChatFiles } from "./chat-files";
 import { ChatImage } from "./chat-image";
 import { ChatSources } from "./chat-sources";
+import { SuggestedQuestions } from "./chat-suggestedQuestions";
 import ChatTools from "./chat-tools";
 import Markdown from "./markdown";

@@ -30,9 +33,11 @@ type ContentDisplayConfig = {
 function ChatMessageContent({
  message,
  isLoading,
+  append,
 }: {
  message: Message;
  isLoading: boolean;
+  append: Pick<ChatHandler, "append">["append"];
 }) {
  const annotations = message.annotations as MessageAnnotation[] | undefined;
  if (!annotations?.length) return <Markdown content={message.content} />;
@@ -57,6 +62,10 @@ function ChatMessageContent({
    annotations,
    MessageAnnotationType.TOOLS,
  );
+  const suggestedQuestionsData = getAnnotationData<SuggestedQuestionsData>(
+    annotations,
+    MessageAnnotationType.SUGGESTED_QUESTIONS,
+  );

  const contents: ContentDisplayConfig[] = [
    {
@@ -88,6 +97,15 @@ function ChatMessageContent({
      order: 3,
      component: sourceData[0] ? <ChatSources data={sourceData[0]} /> : null,
    },
+    {
+      order: 4,
+      component: suggestedQuestionsData[0] ? (
+        <SuggestedQuestions
+          questions={suggestedQuestionsData[0]}
+          append={append}
+        />
+      ) : null,
+    },
  ];

  return (
@@ -104,16 +122,22 @@ function ChatMessageContent({
 export default function ChatMessage({
  chatMessage,
  isLoading,
+  append,
 }: {
  chatMessage: Message;
  isLoading: boolean;
+  append: Pick<ChatHandler, "append">["append"];
 }) {
  const { isCopied, copyToClipboard } = useCopyToClipboard({ timeout: 2000 });
  return (
    <div className="flex items-start gap-4 pr-5 pt-5">
      <ChatAvatar role={chatMessage.role} />
      <div className="group flex flex-1 justify-between gap-2">
-        <ChatMessageContent message={chatMessage} isLoading={isLoading} />
+        <ChatMessageContent
+          message={chatMessage}
+          isLoading={isLoading}
+          append={append}
+        />
        <Button
          onClick={() => copyToClipboard(chatMessage.content)}
          size="icon"
@@ -53,6 +53,7 @@ export default function ChatMessages(
              key={m.id}
              chatMessage={m}
              isLoading={isLoadingMessage}
+              append={props.append!}
            />
          );
        })}
@@ -48,7 +48,10 @@ export function useFile() {
    files.length && setFiles([]);
  };

-  const uploadContent = async (base64: string): Promise<string[]> => {
+  const uploadContent = async (
+    base64: string,
+    requestParams: any = {},
+  ): Promise<string[]> => {
    const uploadAPI = `${backend}/api/chat/upload`;
    const response = await fetch(uploadAPI, {
      method: "POST",
@@ -57,6 +60,7 @@ export function useFile() {
      },
      body: JSON.stringify({
        base64,
+        ...requestParams,
      }),
    });
    if (!response.ok) throw new Error("Failed to upload document.");
@@ -98,7 +102,7 @@ export function useFile() {
    return content;
  };

-  const uploadFile = async (file: File) => {
+  const uploadFile = async (file: File, requestParams: any = {}) => {
    if (file.type.startsWith("image/")) {
      const base64 = await readContent({ file, asUrl: true });
      return setImageUrl(base64);
@@ -125,7 +129,7 @@ export function useFile() {
      }
      default: {
        const base64 = await readContent({ file, asUrl: true });
-        const ids = await uploadContent(base64);
+        const ids = await uploadContent(base64, requestParams);
        return addDoc({
          ...newDoc,
          content: {
@@ -11,6 +11,7 @@ export enum MessageAnnotationType {
  SOURCES = "sources",
  EVENTS = "events",
  TOOLS = "tools",
+  SUGGESTED_QUESTIONS = "suggested_questions",
 }

 export type ImageData = {
@@ -67,12 +68,15 @@ export type ToolData = {
  };
 };

+export type SuggestedQuestionsData = string[];
+
 export type AnnotationData =
  | ImageData
  | DocumentFileData
  | SourceData
  | EventData
-  | ToolData;
+  | ToolData
+  | SuggestedQuestionsData;

 export type MessageAnnotation = {
  type: MessageAnnotationType;
@@ -1,4 +1,4 @@
-import { PDFViewer, PdfFocusProvider } from "@llamaindex/pdf-viewer";
+import dynamic from "next/dynamic";
 import { Button } from "../../button";
 import {
  Drawer,
@@ -16,6 +16,18 @@ export interface PdfDialogProps {
  trigger: React.ReactNode;
 }

+// Dynamic imports for client-side rendering only
+const PDFViewer = dynamic(
+  () => import("@llamaindex/pdf-viewer").then((module) => module.PDFViewer),
+  { ssr: false },
+);
+
+const PdfFocusProvider = dynamic(
+  () =>
+    import("@llamaindex/pdf-viewer").then((module) => module.PdfFocusProvider),
+  { ssr: false },
+);
+
 export default function PdfDialog(props: PdfDialogProps) {
  return (
    <Drawer direction="left">
@@ -1,8 +1,8 @@
 {
  "experimental": {
    "outputFileTracingIncludes": {
-      "/*": ["./cache/**/*"]
-    },
-    "serverComponentsExternalPackages": ["sharp", "onnxruntime-node"]
+      "/*": ["./cache/**/*"],
+      "/api/**/*": ["./node_modules/**/*.wasm"]
+    }
  }
 }
@@ -1,8 +1,10 @@
 /** @type {import('next').NextConfig} */
 import fs from "fs";
+import withLlamaIndex from "llamaindex/next";
 import webpack from "./webpack.config.mjs";

 const nextConfig = JSON.parse(fs.readFileSync("./next.config.json", "utf-8"));
 nextConfig.webpack = webpack;

-export default nextConfig;
+// use withLlamaIndex to add necessary modifications for llamaindex library
+export default withLlamaIndex(nextConfig);
@@ -12,7 +12,7 @@
  "dependencies": {
    "@apidevtools/swagger-parser": "^10.1.0",
    "@e2b/code-interpreter": "^0.0.5",
-    "@llamaindex/pdf-viewer": "^1.1.1",
+    "@llamaindex/pdf-viewer": "^1.1.3",
    "@radix-ui/react-collapsible": "^1.0.3",
    "@radix-ui/react-hover-card": "^1.0.7",
    "@radix-ui/react-slot": "^1.0.2",
@@ -24,7 +24,7 @@
    "duck-duck-scrape": "^2.2.5",
    "formdata-node": "^6.0.3",
    "got": "^14.4.1",
-    "llamaindex": "0.4.6",
+    "llamaindex": "0.5.12",
    "lucide-react": "^0.294.0",
    "next": "^14.2.4",
    "react": "^18.2.0",
@@ -1,6 +1,5 @@
 {
  "compilerOptions": {
-    "target": "es5",
    "lib": ["dom", "dom.iterable", "esnext"],
    "allowJs": true,
    "skipLibCheck": true,
@@ -20,8 +19,7 @@
    ],
    "paths": {
      "@/*": ["./*"]
-    },
-    "forceConsistentCasingInFileNames": true
+    }
  },
  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
  "exclude": ["node_modules"]
@@ -4,12 +4,5 @@ export default function webpack(config) {
    aws4: false,
  };

-  // Following lines will fix issues with onnxruntime-node when using pnpm
-  // See: https://github.com/vercel/next.js/issues/43433
-  config.externals.push({
-    "onnxruntime-node": "commonjs onnxruntime-node",
-    sharp: "commonjs sharp",
-  });
-
  return config;
 }
Author	SHA1	Message	Date
github-actions[bot]	dcf52abdba	Release 0.1.28 (#206 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-08-02 15:38:27 +07:00
Marcus Schiesser	9a09e8c7e2	fix: Vercel deployment (by including WASM files) (#201 )	2024-08-02 15:36:54 +07:00
github-actions[bot]	a4a55239e9	Release 0.1.27 (#204 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-08-01 17:09:07 +02:00
Thuc Pham	c5c7eee04d	refactor: make components resuable for chat llm (#202 ) --------- Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de>	2024-08-01 16:31:38 +02:00
github-actions[bot]	8b89ac547f	Release 0.1.26 (#199 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-07-31 08:56:00 +02:00
Marcus Schiesser	f43399cc18	fix: Add metadatafilters to context chat engine (Typescript) (#196 )	2024-07-31 08:55:06 +02:00
github-actions[bot]	df51361ca1	Release 0.1.25 (#195 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-07-30 15:09:21 +07:00
Huu Le	c67daeb2be	fix: missing set private to false for default generate.py (#194 )	2024-07-30 15:06:04 +07:00
github-actions[bot]	af6ac9a444	Release 0.1.24 (#188 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-07-29 21:01:51 +07:00
Marcus Schiesser	22245ca9fd	chore: remove azure openai key question	2024-07-29 15:51:58 +02:00
Marcus Schiesser	81b67794ef	fix: throw errors if azure deployments are no	2024-07-29 15:51:58 +02:00
Bartłomiej Szczygło	5c13646e55	Fix starter questions not working in python backend (#189 ) * Fix starter questions not working in python backend * add changeset --------- Co-authored-by: Arputikos <arputikos11@op.pl> Co-authored-by: leehuwuj <leehuwuj@gmail.com>	2024-07-29 15:55:00 +07:00
Thuc Pham	43474a51ff	feat: config llamacloud organization ID (#192 ) --------- Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de>	2024-07-29 15:20:18 +07:00
Huu Le	cf11b233c6	feat: support using azure code interpreter in create-llama (#158 ) --------- Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de> Co-authored-by: Wassim Chegham <github@wassim.dev> Co-authored-by: Thuc Pham <51660321+thucpn@users.noreply.github.com>	2024-07-26 17:34:36 +07:00
Thuc Pham	fd9fb42ace	feat: add azure model provider (#184 )	2024-07-26 17:32:14 +07:00
Thuc Pham	92798f73dd	fix: dont ask useLlamaParse if --no-llama-parse in command (#187 )	2024-07-26 14:58:34 +07:00
Marcus Schiesser	e71d8bd6e2	fix: lint	2024-07-25 15:25:53 +02:00
github-actions[bot]	e25e112873	Release 0.1.23 (#186 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-07-25 20:23:04 +07:00
Marcus Schiesser	048187cce3	Update README.md	2024-07-25 20:21:19 +07:00
Marcus Schiesser	6bd76fbfb1	feat: Add template for structured extraction (#185 )	2024-07-25 19:56:26 +07:00
github-actions[bot]	a553d5051e	Release 0.1.22 (#180 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-07-25 15:57:55 +07:00
Thuc Pham	b0becaa8dc	feat: add e2e testing for llamacloud datasource (#181 ) --------- Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de>	2024-07-25 15:56:23 +07:00
Marcus Schiesser	6a42542642	chore: sync tsconfig with create-next-app (#182 )	2024-07-25 15:11:05 +07:00
Thuc Pham	f936a470f3	fix: ignore ts check grammar for regex (#183 )	2024-07-25 14:40:34 +07:00
Thuc Pham	df9cca5a52	feat: upgrade pdf viewer (#179 ) --------- Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de>	2024-07-24 16:03:58 +07:00
Marcus Schiesser	dc9ee895a7	Add video to README.md	2024-07-23 16:38:24 +02:00
github-actions[bot]	98ff3c2e77	Release 0.1.21 (#169 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-07-23 19:01:32 +07:00
Huu Le	0900413689	Add next questions suggestion to the user (#170 ) --------- Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de>	2024-07-23 17:04:53 +07:00
Marcus Schiesser	8dc6a2bf5a	fix: simplify webpack config using 0.57 (#174 )	2024-07-23 16:16:38 +07:00
Huu Le	23b735717d	chore: Use gpt-4o-mini as default (#173 )	2024-07-22 21:42:40 +07:00
Thuc Pham	bd4714ca8d	feat: add filter for query in ts templates (#172 ) --------- Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de>	2024-07-22 21:42:20 +07:00
Thuc Pham	455ab6862e	feat: display files from llamacloud (#153 ) --------- Co-authored-by: leehuwuj <leehuwuj@gmail.com> Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de>	2024-07-18 22:46:06 +07:00
Huu Le	58e6c150c0	feat: Use LlamaParse to parse the private files (#167 )	2024-07-17 19:15:43 +07:00
Thuc Pham	e57e9813dd	fix: use stable tsup version (#168 )	2024-07-17 17:07:40 +07:00
github-actions[bot]	7302880c5f	Release 0.1.20 (#166 ) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2024-07-15 22:17:09 +07:00
Marcus Schiesser	624c721ac4	chore: update to llamaindex 0.10.55 (#165 )	2024-07-15 21:51:15 +07:00