Release 0.1.37 (#239)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

.github/workflows/e2e.yml

@@ -17,7 +17,9 @@ jobs:
       matrix:
         node-version: [18, 20]
         python-version: ["3.11"]
-        os: [macos-latest, windows-latest]
+        os: [macos-latest, windows-latest, ubuntu-22.04]
+        frameworks: ["nextjs", "express", "fastapi"]
+        datasources: ["--no-files", "--example-file"]
     defaults:
       run:
         shell: bash
@@ -62,6 +64,9 @@ jobs:
         run: pnpm run e2e
         env:
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          LLAMA_CLOUD_API_KEY: ${{ secrets.LLAMA_CLOUD_API_KEY }}
+          FRAMEWORK: ${{ matrix.frameworks }}
+          DATASOURCE: ${{ matrix.datasources }}
         working-directory: .
 
       - uses: actions/upload-artifact@v3

.github/workflows/lint_on_push_or_pull.yml

@@ -30,3 +30,13 @@ jobs:
 
       - name: Run Prettier
         run: pnpm run format
+
+      - name: Run Python format check
+        uses: chartboost/ruff-action@v1
+        with:
+          args: "format --check"
+
+      - name: Run Python lint
+        uses: chartboost/ruff-action@v1
+        with:
+          args: "check"

.gitignore

@@ -46,5 +46,8 @@ e2e/cache
 # intellij
 **/.idea
 
+# Python
+.mypy_cache/
+
 # build artifacts
 create-llama-*.tgz

CHANGELOG.md

@@ -1,5 +1,170 @@
 # create-llama
 
+## 0.1.37
+
+### Patch Changes
+
+- 9fd832c: Add in-text citation references
+
+## 0.1.36
+
+### Patch Changes
+
+- 2b7a5d8: Fix: private file upload not working in Python without LlamaCloud
+
+## 0.1.35
+
+### Patch Changes
+
+- 81ef7f0: Use LlamaCloud pipeline for data ingestion (private file uploads and generate script)
+
+## 0.1.34
+
+### Patch Changes
+
+- c49a5e1: Add error handling for generating the next question
+- c49a5e1: Fix wrong api key variable in Azure OpenAI provider
+
+## 0.1.33
+
+### Patch Changes
+
+- d746c75: Add Weaviate vector store (Typescript)
+
+## 0.1.32
+
+### Patch Changes
+
+- 3ec5163: Add Weaviate vector database support (Python)
+
+## 0.1.31
+
+### Patch Changes
+
+- 04a9c71: Cluster nodes by document
+
+## 0.1.30
+
+### Patch Changes
+
+- 09e3022: Add support for LlamaTrace (Python)
+- c06ec4f: Fix imports for MongoDB
+- b6dd7a9: Always send chat data when submit message
+
+## 0.1.29
+
+### Patch Changes
+
+- 8890e27: Let user change indexes in LlamaCloud projects
+
+## 0.1.28
+
+### Patch Changes
+
+- 9a09e8c: Fix Vercel deployment
+
+## 0.1.27
+
+### Patch Changes
+
+- c5c7eee: Make components reusable for chat-llamaindex
+
+## 0.1.26
+
+### Patch Changes
+
+- f43399c: Add metadatafilters to context chat engine (Typescript)
+
+## 0.1.25
+
+### Patch Changes
+
+- c67daeb: fix: missing set private to false for default generate.py
+
+## 0.1.24
+
+### Patch Changes
+
+- 43474a5: Configure LlamaCloud organization ID for Python
+- cf11b23: Add Azure code interpreter for Python and TS
+- fd9fb42: Add Azure OpenAI as model provider
+- 5c13646: Fix starter questions not working in python backend
+
+## 0.1.23
+
+### Patch Changes
+
+- 6bd76fb: Add template for structured extraction
+
+## 0.1.22
+
+### Patch Changes
+
+- b0becaa: Add e2e testing for llamacloud datasource
+- df9cca5: Upgrade pdf viewer
+
+## 0.1.21
+
+### Patch Changes
+
+- bd4714c: Filter private documents for Typescript (Using MetadataFilters) and update to LlamaIndexTS 0.5.7
+- 58e6c15: Add using LlamaParse for private file uploader
+- 455ab68: Display files in sources using LlamaCloud indexes.
+- 23b7357: Use gpt-4o-mini as default model
+- 0900413: Add suggestions for next questions.
+
+## 0.1.20
+
+### Patch Changes
+
+- 624c721: Update to LlamaIndex 0.10.55
+
+## 0.1.19
+
+### Patch Changes
+
+- df96159: Use Qdrant FastEmbed as local embedding provider
+- 32fb32a: Support upload document files: pdf, docx, txt
+
+## 0.1.18
+
+### Patch Changes
+
+- d1026ea: support Mistral as llm and embedding
+- a221cfc: Use LlamaParse for all the file types that it supports (if activated)
+
+## 0.1.17
+
+### Patch Changes
+
+- 9ecd061: Add new template for a multi-agents app
+
+## 0.1.16
+
+### Patch Changes
+
+- a0aab03: Add T-System's LLMHUB as a model provider
+
+## 0.1.15
+
+### Patch Changes
+
+- 64732f0: Fix the issue of images not showing with the sandbox URL from OpenAI's models
+- aeb6fef: use llamacloud for chat
+
+## 0.1.14
+
+### Patch Changes
+
+- f2c3389: chore: update to llamaindex 0.4.3
+- 5093b37: Remove non-working file selectors for Linux
+
+## 0.1.13
+
+### Patch Changes
+
+- b3c969d: Add image generator tool
+
 ## 0.1.12
 
 ### Patch Changes

README.md

@@ -1,14 +1,20 @@
-# Create LlamaIndex App
+# Create Llama
 
 The easiest way to get started with [LlamaIndex](https://www.llamaindex.ai/) is by using `create-llama`. This CLI tool enables you to quickly start building a new LlamaIndex application, with everything set up for you.
 
+## Get started
+
 Just run
 
 ```bash
 npx create-llama@latest
 ```
 
-to get started, or see below for more options. Once your app is generated, run
+to get started, or watch this video for a demo session:
+
+https://github.com/user-attachments/assets/dd3edc36-4453-4416-91c2-d24326c6c167
+
+Once your app is generated, run
 
 ```bash
 npm run dev
@@ -18,16 +24,20 @@ to start the development server. You can then visit [http://localhost:3000](http
 
 ## What you'll get
 
-- A Next.js-powered front-end using components from [shadcn/ui](https://ui.shadcn.com/). The app is set up as a chat interface that can answer questions about your data (see below)
+- A Next.js-powered front-end using components from [shadcn/ui](https://ui.shadcn.com/). The app is set up as a chat interface that can answer questions about your data or interact with your agent
 - Your choice of 3 back-ends:
   - **Next.js**: if you select this option, you’ll have a full-stack Next.js application that you can deploy to a host like [Vercel](https://vercel.com/) in just a few clicks. This uses [LlamaIndex.TS](https://www.npmjs.com/package/llamaindex), our TypeScript library.
   - **Express**: if you want a more traditional Node.js application you can generate an Express backend. This also uses LlamaIndex.TS.
-  - **Python FastAPI**: if you select this option, you’ll get a backend powered by the [llama-index python package](https://pypi.org/project/llama-index/), which you can deploy to a service like Render or fly.io.
+  - **Python FastAPI**: if you select this option, you’ll get a backend powered by the [llama-index Python package](https://pypi.org/project/llama-index/), which you can deploy to a service like Render or fly.io.
 - The back-end has two endpoints (one streaming, the other one non-streaming) that allow you to send the state of your chat and receive additional responses
 - You add arbitrary data sources to your chat, like local files, websites, or data retrieved from a database.
 - Turn your chat into an AI agent by adding tools (functions called by the LLM).
 - The app uses OpenAI by default, so you'll need an OpenAI API key, or you can customize it to use any of the dozens of LLMs we support.
 
+Here's how it looks like:
+
+https://github.com/user-attachments/assets/d57af1a1-d99b-4e9c-98d9-4cbd1327eff8
+
 ## Using your data
 
 You can supply your own data; the app will index it and answer questions. Your generated app will have a folder called `data` (If you're using Express or Python and generate a frontend, it will be `./backend/data`).
@@ -54,7 +64,7 @@ Optionally generate a frontend if you've selected the Python or Express back-end
 
 ## Customizing the AI models
 
-The app will default to OpenAI's `gpt-4-turbo` LLM and `text-embedding-3-large` embedding model.
+The app will default to OpenAI's `gpt-4o-mini` LLM and `text-embedding-3-large` embedding model.
 
 If you want to use different OpenAI models, add the `--ask-models` CLI parameter.
 
@@ -84,7 +94,7 @@ Need to install the following packages:
   create-llama@latest
 Ok to proceed? (y) y
 ✔ What is your project named? … my-app
-✔ Which template would you like to use? › Chat
+✔ Which template would you like to use? › Agentic RAG (single agent)
 ✔ Which framework would you like to use? › NextJS
 ✔ Would you like to set up observability? › No
 ✔ Please provide your OpenAI API key (leave blank to skip): …
@@ -92,6 +102,7 @@ Ok to proceed? (y) y
 ✔ Would you like to add another data source? › No
 ✔ Would you like to use LlamaParse (improved parser for RAG - requires API key)? … no / yes
 ✔ Would you like to use a vector database? › No, just store the data in the file system
+✔ Would you like to build an agent using tools? If so, select the tools here, otherwise just press enter › Weather
 ? How would you like to proceed? › - Use arrow-keys. Return to submit.
    Just generate code (~1 sec)
 ❯  Start in VSCode (~1 sec)

create-app.ts

@@ -9,7 +9,7 @@ import { makeDir } from "./helpers/make-dir";
 
 import fs from "fs";
 import terminalLink from "terminal-link";
-import type { InstallTemplateArgs } from "./helpers";
+import type { InstallTemplateArgs, TemplateObservability } from "./helpers";
 import { installTemplate } from "./helpers";
 import { writeDevcontainer } from "./helpers/devcontainer";
 import { templatesDir } from "./helpers/dir";
@@ -142,14 +142,42 @@ export async function createApp({
     )} and learn how to get started.`,
   );
 
-  if (args.observability === "opentelemetry") {
+  outputObservability(args.observability);
+
+  if (
+    dataSources.some((dataSource) => dataSource.type === "file") &&
+    process.platform === "linux"
+  ) {
     console.log(
-      `\n${yellow("Observability")}: Visit the ${terminalLink(
-        "documentation",
-        "https://traceloop.com/docs/openllmetry/integrations",
-      )} to set up the environment variables and start seeing execution traces.`,
+      yellow(
+        `You can add your own data files to ${terminalLink(
+          "data",
+          `file://${root}/data`,
+        )} folder manually.`,
+      ),
     );
   }
 
   console.log();
 }
+
+function outputObservability(observability?: TemplateObservability) {
+  switch (observability) {
+    case "traceloop":
+      console.log(
+        `\n${yellow("Observability")}: Visit the ${terminalLink(
+          "documentation",
+          "https://traceloop.com/docs/openllmetry/integrations",
+        )} to set up the environment variables and start seeing execution traces.`,
+      );
+      break;
+    case "llamatrace":
+      console.log(
+        `\n${yellow("Observability")}: LlamaTrace has been configured for your project. Visit the ${terminalLink(
+          "LlamaTrace dashboard",
+          "https://llamatrace.com/login",
+        )} to view your traces and monitor your application.`,
+      );
+      break;
+  }
+}

e2e/basic.spec.ts

@@ -11,119 +11,110 @@ import type {
 } from "../helpers";
 import { createTestDir, runCreateLlama, type AppType } from "./utils";
 
-const templateTypes: TemplateType[] = ["streaming"];
-const templateFrameworks: TemplateFramework[] = [
-  "nextjs",
-  "express",
-  "fastapi",
-];
-const dataSources: string[] = ["--no-files", "--example-file"];
-const templateUIs: TemplateUI[] = ["shadcn", "html"];
-const templatePostInstallActions: TemplatePostInstallAction[] = [
-  "none",
-  "runApp",
-];
+const templateType: TemplateType = "streaming";
+const templateFramework: TemplateFramework = process.env.FRAMEWORK
+  ? (process.env.FRAMEWORK as TemplateFramework)
+  : "fastapi";
+const dataSource: string = process.env.DATASOURCE
+  ? process.env.DATASOURCE
+  : "--example-file";
+const templateUI: TemplateUI = "shadcn";
+const templatePostInstallAction: TemplatePostInstallAction = "runApp";
 
-for (const templateType of templateTypes) {
-  for (const templateFramework of templateFrameworks) {
-    for (const dataSource of dataSources) {
-      for (const templateUI of templateUIs) {
-        for (const templatePostInstallAction of templatePostInstallActions) {
-          const appType: AppType =
-            templateFramework === "nextjs" ? "" : "--frontend";
-          test.describe(`try create-llama ${templateType} ${templateFramework} ${dataSource} ${templateUI} ${appType} ${templatePostInstallAction}`, async () => {
-            let port: number;
-            let externalPort: number;
-            let cwd: string;
-            let name: string;
-            let appProcess: ChildProcess;
-            // Only test without using vector db for now
-            const vectorDb = "none";
+const llamaCloudProjectName = "create-llama";
+const llamaCloudIndexName = "e2e-test";
 
-            test.beforeAll(async () => {
-              port = Math.floor(Math.random() * 10000) + 10000;
-              externalPort = port + 1;
-              cwd = await createTestDir();
-              const result = await runCreateLlama(
-                cwd,
-                templateType,
-                templateFramework,
-                dataSource,
-                templateUI,
-                vectorDb,
-                appType,
-                port,
-                externalPort,
-                templatePostInstallAction,
-              );
-              name = result.projectName;
-              appProcess = result.appProcess;
-            });
+const appType: AppType = templateFramework === "nextjs" ? "" : "--frontend";
+const userMessage =
+  dataSource !== "--no-files" ? "Physical standard for letters" : "Hello";
+test.describe(`try create-llama ${templateType} ${templateFramework} ${dataSource} ${templateUI} ${appType} ${templatePostInstallAction}`, async () => {
+  let port: number;
+  let externalPort: number;
+  let cwd: string;
+  let name: string;
+  let appProcess: ChildProcess;
+  // Only test without using vector db for now
+  const vectorDb = "none";
 
-            test("App folder should exist", async () => {
-              const dirExists = fs.existsSync(path.join(cwd, name));
-              expect(dirExists).toBeTruthy();
-            });
-            test("Frontend should have a title", async ({ page }) => {
-              test.skip(templatePostInstallAction !== "runApp");
-              await page.goto(`http://localhost:${port}`);
-              await expect(page.getByText("Built by LlamaIndex")).toBeVisible();
-            });
+  test.beforeAll(async () => {
+    port = Math.floor(Math.random() * 10000) + 10000;
+    externalPort = port + 1;
+    cwd = await createTestDir();
+    const result = await runCreateLlama(
+      cwd,
+      templateType,
+      templateFramework,
+      dataSource,
+      templateUI,
+      vectorDb,
+      appType,
+      port,
+      externalPort,
+      templatePostInstallAction,
+      llamaCloudProjectName,
+      llamaCloudIndexName,
+    );
+    name = result.projectName;
+    appProcess = result.appProcess;
+  });
 
-            test("Frontend should be able to submit a message and receive a response", async ({
-              page,
-            }) => {
-              test.skip(templatePostInstallAction !== "runApp");
-              await page.goto(`http://localhost:${port}`);
-              await page.fill("form input", "hello");
-              const [response] = await Promise.all([
-                page.waitForResponse(
-                  (res) => {
-                    return (
-                      res.url().includes("/api/chat") && res.status() === 200
-                    );
-                  },
-                  {
-                    timeout: 1000 * 60,
-                  },
-                ),
-                page.click("form button[type=submit]"),
-              ]);
-              const text = await response.text();
-              console.log("AI response when submitting message: ", text);
-              expect(response.ok()).toBeTruthy();
-            });
+  test("App folder should exist", async () => {
+    const dirExists = fs.existsSync(path.join(cwd, name));
+    expect(dirExists).toBeTruthy();
+  });
+  test("Frontend should have a title", async ({ page }) => {
+    test.skip(templatePostInstallAction !== "runApp");
+    await page.goto(`http://localhost:${port}`);
+    await expect(page.getByText("Built by LlamaIndex")).toBeVisible();
+  });
 
-            test("Backend frameworks should response when calling non-streaming chat API", async ({
-              request,
-            }) => {
-              test.skip(templatePostInstallAction !== "runApp");
-              test.skip(templateFramework === "nextjs");
-              const response = await request.post(
-                `http://localhost:${externalPort}/api/chat/request`,
-                {
-                  data: {
-                    messages: [
-                      {
-                        role: "user",
-                        content: "Hello",
-                      },
-                    ],
-                  },
-                },
-              );
-              const text = await response.text();
-              console.log("AI response when calling API: ", text);
-              expect(response.ok()).toBeTruthy();
-            });
+  test("Frontend should be able to submit a message and receive a response", async ({
+    page,
+  }) => {
+    test.skip(templatePostInstallAction !== "runApp");
+    await page.goto(`http://localhost:${port}`);
+    await page.fill("form input", userMessage);
+    const [response] = await Promise.all([
+      page.waitForResponse(
+        (res) => {
+          return res.url().includes("/api/chat") && res.status() === 200;
+        },
+        {
+          timeout: 1000 * 60,
+        },
+      ),
+      page.click("form button[type=submit]"),
+    ]);
+    const text = await response.text();
+    console.log("AI response when submitting message: ", text);
+    expect(response.ok()).toBeTruthy();
+  });
 
-            // clean processes
-            test.afterAll(async () => {
-              appProcess?.kill();
-            });
-          });
-        }
-      }
-    }
-  }
-}
+  test("Backend frameworks should response when calling non-streaming chat API", async ({
+    request,
+  }) => {
+    test.skip(templatePostInstallAction !== "runApp");
+    test.skip(templateFramework === "nextjs");
+    const response = await request.post(
+      `http://localhost:${externalPort}/api/chat/request`,
+      {
+        data: {
+          messages: [
+            {
+              role: "user",
+              content: userMessage,
+            },
+          ],
+        },
+      },
+    );
+    const text = await response.text();
+    console.log("AI response when calling API: ", text);
+    expect(response.ok()).toBeTruthy();
+  });
+
+  // clean processes
+  test.afterAll(async () => {
+    appProcess?.kill();
+  });
+});

e2e/utils.ts

@@ -18,48 +18,6 @@ export type CreateLlamaResult = {
   appProcess: ChildProcess;
 };
 
-// eslint-disable-next-line max-params
-export async function checkAppHasStarted(
-  frontend: boolean,
-  framework: TemplateFramework,
-  port: number,
-  externalPort: number,
-  timeout: number,
-) {
-  if (frontend) {
-    await Promise.all([
-      waitPort({
-        host: "localhost",
-        port: port,
-        timeout,
-      }),
-      waitPort({
-        host: "localhost",
-        port: externalPort,
-        timeout,
-      }),
-    ]).catch((err) => {
-      console.error(err);
-      throw err;
-    });
-  } else {
-    let wPort: number;
-    if (framework === "nextjs") {
-      wPort = port;
-    } else {
-      wPort = externalPort;
-    }
-    await waitPort({
-      host: "localhost",
-      port: wPort,
-      timeout,
-    }).catch((err) => {
-      console.error(err);
-      throw err;
-    });
-  }
-}
-
 // eslint-disable-next-line max-params
 export async function runCreateLlama(
   cwd: string,
@@ -72,9 +30,13 @@ export async function runCreateLlama(
   port: number,
   externalPort: number,
   postInstallAction: TemplatePostInstallAction,
+  llamaCloudProjectName: string,
+  llamaCloudIndexName: string,
 ): Promise<CreateLlamaResult> {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error("Setting OPENAI_API_KEY is mandatory to run tests");
+  if (!process.env.OPENAI_API_KEY || !process.env.LLAMA_CLOUD_API_KEY) {
+    throw new Error(
+      "Setting the OPENAI_API_KEY and LLAMA_CLOUD_API_KEY is mandatory to run tests",
+    );
   }
   const name = [
     templateType,
@@ -110,12 +72,16 @@ export async function runCreateLlama(
     "--no-llama-parse",
     "--observability",
     "none",
+    "--llama-cloud-key",
+    process.env.LLAMA_CLOUD_API_KEY,
   ].join(" ");
   console.log(`running command '${command}' in ${cwd}`);
   const appProcess = exec(command, {
     cwd,
     env: {
       ...process.env,
+      LLAMA_CLOUD_PROJECT_NAME: llamaCloudProjectName,
+      LLAMA_CLOUD_INDEX_NAME: llamaCloudIndexName,
     },
   });
   appProcess.stderr?.on("data", (data) => {
@@ -134,25 +100,10 @@ export async function runCreateLlama(
       templateFramework,
       port,
       externalPort,
-      1000 * 60 * 5,
     );
   } else {
-    // wait create-llama to exit
-    // we don't test install dependencies for now, so just set timeout for 10 seconds
-    await new Promise((resolve, reject) => {
-      const timeout = setTimeout(() => {
-        reject(new Error("create-llama timeout error"));
-      }, 1000 * 10);
-      appProcess.on("exit", (code) => {
-        if (code !== 0 && code !== null) {
-          clearTimeout(timeout);
-          reject(new Error("create-llama command was failed!"));
-        } else {
-          clearTimeout(timeout);
-          resolve(undefined);
-        }
-      });
-    });
+    // wait 10 seconds for create-llama to exit
+    await waitForProcess(appProcess, 1000 * 10);
   }
 
   return {
@@ -166,3 +117,53 @@ export async function createTestDir() {
   await mkdir(cwd, { recursive: true });
   return cwd;
 }
+
+// eslint-disable-next-line max-params
+async function checkAppHasStarted(
+  frontend: boolean,
+  framework: TemplateFramework,
+  port: number,
+  externalPort: number,
+) {
+  const portsToWait = frontend
+    ? [port, externalPort]
+    : [framework === "nextjs" ? port : externalPort];
+  await waitPorts(portsToWait);
+}
+
+async function waitPorts(ports: number[]): Promise<void> {
+  const waitForPort = async (port: number): Promise<void> => {
+    await waitPort({
+      host: "localhost",
+      port: port,
+      // wait max. 5 mins for start up of app
+      timeout: 1000 * 60 * 5,
+    });
+  };
+  try {
+    await Promise.all(ports.map(waitForPort));
+  } catch (err) {
+    console.error(err);
+    throw err;
+  }
+}
+
+async function waitForProcess(
+  process: ChildProcess,
+  timeoutMs: number,
+): Promise<void> {
+  return new Promise((resolve, reject) => {
+    const timeout = setTimeout(() => {
+      reject(new Error("Process timeout error"));
+    }, timeoutMs);
+
+    process.on("exit", (code) => {
+      clearTimeout(timeout);
+      if (code !== 0 && code !== null) {
+        reject(new Error("Process exited with non-zero code"));
+      } else {
+        resolve();
+      }
+    });
+  });
+}

helpers/env-variables.ts

@@ -2,12 +2,17 @@ import fs from "fs/promises";
 import path from "path";
 import { TOOL_SYSTEM_PROMPT_ENV_VAR, Tool } from "./tools";
 import {
+  InstallTemplateArgs,
   ModelConfig,
   TemplateDataSource,
   TemplateFramework,
+  TemplateObservability,
+  TemplateType,
   TemplateVectorDB,
 } from "./types";
 
+import { TSYSTEMS_LLMHUB_API_URL } from "./providers/llmhub";
+
 export type EnvVar = {
   name?: string;
   description?: string;
@@ -133,6 +138,42 @@ const getVectorDBEnvs = (
             "Optional API key for authenticating requests to Qdrant.",
         },
       ];
+    case "llamacloud":
+      return [
+        {
+          name: "LLAMA_CLOUD_INDEX_NAME",
+          description:
+            "The name of the LlamaCloud index to use (part of the LlamaCloud project).",
+          value: "test",
+        },
+        {
+          name: "LLAMA_CLOUD_PROJECT_NAME",
+          description: "The name of the LlamaCloud project.",
+          value: "Default",
+        },
+        {
+          name: "LLAMA_CLOUD_BASE_URL",
+          description:
+            "The base URL for the LlamaCloud API. Only change this for non-production environments",
+          value: "https://api.cloud.llamaindex.ai",
+        },
+        {
+          name: "LLAMA_CLOUD_ORGANIZATION_ID",
+          description:
+            "The organization ID for the LlamaCloud project (uses default organization if not specified - Python only)",
+        },
+        ...(framework === "nextjs"
+          ? // activate index selector per default (not needed for non-NextJS backends as it's handled by createFrontendEnvFile)
+            [
+              {
+                name: "NEXT_PUBLIC_USE_LLAMACLOUD",
+                description:
+                  "Let's the user change indexes in LlamaCloud projects",
+                value: "true",
+              },
+            ]
+          : []),
+      ];
     case "chroma":
       const envs = [
         {
@@ -158,6 +199,23 @@ Otherwise, use CHROMA_HOST and CHROMA_PORT config above`,
         });
       }
       return envs;
+    case "weaviate":
+      return [
+        {
+          name: "WEAVIATE_CLUSTER_URL",
+          description:
+            "The URL of the Weaviate cloud cluster, see: https://weaviate.io/developers/wcs/connect",
+        },
+        {
+          name: "WEAVIATE_API_KEY",
+          description: "The API key for the Weaviate cloud cluster",
+        },
+        {
+          name: "WEAVIATE_INDEX_NAME",
+          description:
+            "(Optional) The collection name to use, default is LlamaIndex if not specified",
+        },
+      ];
     default:
       return [];
   }
@@ -242,6 +300,57 @@ const getModelEnvs = (modelConfig: ModelConfig): EnvVar[] => {
           },
         ]
       : []),
+    ...(modelConfig.provider === "mistral"
+      ? [
+          {
+            name: "MISTRAL_API_KEY",
+            description: "The Mistral API key to use.",
+            value: modelConfig.apiKey,
+          },
+        ]
+      : []),
+    ...(modelConfig.provider === "azure-openai"
+      ? [
+          {
+            name: "AZURE_OPENAI_API_KEY",
+            description: "The Azure OpenAI key to use.",
+            value: modelConfig.apiKey,
+          },
+          {
+            name: "AZURE_OPENAI_ENDPOINT",
+            description: "The Azure OpenAI endpoint to use.",
+          },
+          {
+            name: "AZURE_OPENAI_API_VERSION",
+            description: "The Azure OpenAI API version to use.",
+          },
+          {
+            name: "AZURE_OPENAI_LLM_DEPLOYMENT",
+            description:
+              "The Azure OpenAI deployment to use for LLM deployment.",
+          },
+          {
+            name: "AZURE_OPENAI_EMBEDDING_DEPLOYMENT",
+            description:
+              "The Azure OpenAI deployment to use for embedding deployment.",
+          },
+        ]
+      : []),
+    ...(modelConfig.provider === "t-systems"
+      ? [
+          {
+            name: "T_SYSTEMS_LLMHUB_BASE_URL",
+            description:
+              "The base URL for the T-Systems AI Foundation Model API. Eg: http://localhost:11434",
+            value: TSYSTEMS_LLMHUB_API_URL,
+          },
+          {
+            name: "T_SYSTEMS_LLMHUB_API_KEY",
+            description: "API Key for T-System's AI Foundation Model.",
+            value: modelConfig.apiKey,
+          },
+        ]
+      : []),
   ];
 };
 
@@ -315,7 +424,11 @@ const getToolEnvs = (tools?: Tool[]): EnvVar[] => {
   return toolEnvs;
 };
 
-const getSystemPromptEnv = (tools?: Tool[]): EnvVar => {
+const getSystemPromptEnv = (
+  tools?: Tool[],
+  dataSources?: TemplateDataSource[],
+  framework?: TemplateFramework,
+): EnvVar[] => {
   const defaultSystemPrompt =
     "You are a helpful assistant who helps users with their questions.";
 
@@ -334,24 +447,110 @@ const getSystemPromptEnv = (tools?: Tool[]): EnvVar => {
     ? `\"${toolSystemPrompt}\"`
     : defaultSystemPrompt;
 
-  return {
-    name: "SYSTEM_PROMPT",
-    description: "The system prompt for the AI model.",
-    value: systemPrompt,
-  };
+  const systemPromptEnv = [
+    {
+      name: "SYSTEM_PROMPT",
+      description: "The system prompt for the AI model.",
+      value: systemPrompt,
+    },
+  ];
+
+  // Citation only works with FastAPI along with the chat engine and data source provided for now.
+  if (
+    framework === "fastapi" &&
+    tools?.length == 0 &&
+    (dataSources?.length ?? 0 > 0)
+  ) {
+    const citationPrompt = `'You have provided information from a knowledge base that has been passed to you in nodes of information.
+Each node has useful metadata such as node ID, file name, page, etc.
+Please add the citation to the data node for each sentence or paragraph that you reference in the provided information.
+The citation format is: . [citation:<node_id>]()
+Where the <node_id> is the unique identifier of the data node.
+
+Example:
+We have two nodes:
+  node_id: xyz
+  file_name: llama.pdf
+  
+  node_id: abc
+  file_name: animal.pdf
+
+User question: Tell me a fun fact about Llama.
+Your answer:
+A baby llama is called "Cria" [citation:xyz]().
+It often live in desert [citation:abc]().
+It\\'s cute animal.
+'`;
+    systemPromptEnv.push({
+      name: "SYSTEM_CITATION_PROMPT",
+      description:
+        "An additional system prompt to add citation when responding to user questions.",
+      value: citationPrompt,
+    });
+  }
+
+  return systemPromptEnv;
+};
+
+const getTemplateEnvs = (template?: TemplateType): EnvVar[] => {
+  if (template === "multiagent") {
+    return [
+      {
+        name: "MESSAGE_QUEUE_PORT",
+      },
+      {
+        name: "CONTROL_PLANE_PORT",
+      },
+      {
+        name: "HUMAN_CONSUMER_PORT",
+      },
+      {
+        name: "AGENT_QUERY_ENGINE_PORT",
+        value: "8003",
+      },
+      {
+        name: "AGENT_QUERY_ENGINE_DESCRIPTION",
+        value: "Query information from the provided data",
+      },
+      {
+        name: "AGENT_DUMMY_PORT",
+        value: "8004",
+      },
+    ];
+  } else {
+    return [];
+  }
+};
+
+const getObservabilityEnvs = (
+  observability?: TemplateObservability,
+): EnvVar[] => {
+  if (observability === "llamatrace") {
+    return [
+      {
+        name: "PHOENIX_API_KEY",
+        description:
+          "API key for LlamaTrace observability. Retrieve from https://llamatrace.com/login",
+      },
+    ];
+  }
+  return [];
 };
 
 export const createBackendEnvFile = async (
   root: string,
-  opts: {
-    llamaCloudKey?: string;
-    vectorDb?: TemplateVectorDB;
-    modelConfig: ModelConfig;
-    framework: TemplateFramework;
-    dataSources?: TemplateDataSource[];
-    port?: number;
-    tools?: Tool[];
-  },
+  opts: Pick<
+    InstallTemplateArgs,
+    | "llamaCloudKey"
+    | "vectorDb"
+    | "modelConfig"
+    | "framework"
+    | "dataSources"
+    | "template"
+    | "externalPort"
+    | "tools"
+    | "observability"
+  >,
 ) => {
   // Init env values
   const envFileName = ".env";
@@ -361,15 +560,15 @@ export const createBackendEnvFile = async (
       description: `The Llama Cloud API key.`,
       value: opts.llamaCloudKey,
     },
-    // Add model environment variables
+    // Add environment variables of each component
     ...getModelEnvs(opts.modelConfig),
-    // Add engine environment variables
     ...getEngineEnvs(),
-    // Add vector database environment variables
     ...getVectorDBEnvs(opts.vectorDb, opts.framework),
-    ...getFrameworkEnvs(opts.framework, opts.port),
+    ...getFrameworkEnvs(opts.framework, opts.externalPort),
     ...getToolEnvs(opts.tools),
-    getSystemPromptEnv(opts.tools),
+    ...getTemplateEnvs(opts.template),
+    ...getObservabilityEnvs(opts.observability),
+    ...getSystemPromptEnv(opts.tools, opts.dataSources, opts.framework),
   ];
   // Render and write env file
   const content = renderEnvVar(envVars);
@@ -381,6 +580,7 @@ export const createFrontendEnvFile = async (
   root: string,
   opts: {
     customApiPath?: string;
+    vectorDb?: TemplateVectorDB;
   },
 ) => {
   const defaultFrontendEnvs = [
@@ -391,6 +591,11 @@ export const createFrontendEnvFile = async (
         ? opts.customApiPath
         : "http://localhost:8000/api/chat",
     },
+    {
+      name: "NEXT_PUBLIC_USE_LLAMACLOUD",
+      description: "Let's the user change indexes in LlamaCloud projects",
+      value: opts.vectorDb === "llamacloud" ? "true" : "false",
+    },
   ];
   const content = renderEnvVar(defaultFrontendEnvs);
   await fs.writeFile(path.join(root, ".env"), content);

helpers/index.ts

@@ -8,6 +8,7 @@ import { writeLoadersConfig } from "./datasources";
 import { createBackendEnvFile, createFrontendEnvFile } from "./env-variables";
 import { PackageManager } from "./get-pkg-manager";
 import { installLlamapackProject } from "./llama-pack";
+import { makeDir } from "./make-dir";
 import { isHavingPoetryLockFile, tryPoetryRun } from "./poetry";
 import { installPythonTemplate } from "./python";
 import { downloadAndExtractRepo } from "./repo";
@@ -22,6 +23,31 @@ import {
 } from "./types";
 import { installTSTemplate } from "./typescript";
 
+const checkForGenerateScript = (
+  modelConfig: ModelConfig,
+  vectorDb?: TemplateVectorDB,
+  llamaCloudKey?: string,
+  useLlamaParse?: boolean,
+) => {
+  const missingSettings = [];
+
+  if (!modelConfig.isConfigured()) {
+    missingSettings.push("your model provider API key");
+  }
+
+  const llamaCloudApiKey = llamaCloudKey ?? process.env["LLAMA_CLOUD_API_KEY"];
+  const isRequiredLlamaCloudKey = useLlamaParse || vectorDb === "llamacloud";
+  if (isRequiredLlamaCloudKey && !llamaCloudApiKey) {
+    missingSettings.push("your LLAMA_CLOUD_API_KEY");
+  }
+
+  if (vectorDb !== "none" && vectorDb !== "llamacloud") {
+    missingSettings.push("your Vector DB environment variables");
+  }
+
+  return missingSettings;
+};
+
 // eslint-disable-next-line max-params
 async function generateContextData(
   framework: TemplateFramework,
@@ -37,12 +63,15 @@ async function generateContextData(
         ? "poetry run generate"
         : `${packageManager} run generate`,
     )}`;
-    const modelConfigured = modelConfig.isConfigured();
-    const llamaCloudKeyConfigured = useLlamaParse
-      ? llamaCloudKey || process.env["LLAMA_CLOUD_API_KEY"]
-      : true;
-    const hasVectorDb = vectorDb && vectorDb !== "none";
-    if (modelConfigured && llamaCloudKeyConfigured && !hasVectorDb) {
+
+    const missingSettings = checkForGenerateScript(
+      modelConfig,
+      vectorDb,
+      llamaCloudKey,
+      useLlamaParse,
+    );
+
+    if (!missingSettings.length) {
       // If all the required environment variables are set, run the generate script
       if (framework === "fastapi") {
         if (isHavingPoetryLockFile()) {
@@ -62,15 +91,8 @@ async function generateContextData(
       }
     }
 
-    // generate the message of what to do to run the generate script manually
-    const settings = [];
-    if (!modelConfigured) settings.push("your model provider API key");
-    if (!llamaCloudKeyConfigured) settings.push("your Llama Cloud key");
-    if (hasVectorDb) settings.push("your Vector DB environment variables");
-    const settingsMessage =
-      settings.length > 0 ? `After setting ${settings.join(" and ")}, ` : "";
-    const generateMessage = `run ${runGenerate} to generate the context data.`;
-    console.log(`\n${settingsMessage}${generateMessage}\n\n`);
+    const settingsMessage = `After setting ${missingSettings.join(" and ")}, run ${runGenerate} to generate the context data.`;
+    console.log(`\n${settingsMessage}\n\n`);
   }
 }
 
@@ -120,12 +142,15 @@ export const installTemplate = async (
 
   if (props.framework === "fastapi") {
     await installPythonTemplate(props);
-    // write loaders configuration (currently Python only)
-    await writeLoadersConfig(
-      props.root,
-      props.dataSources,
-      props.useLlamaParse,
-    );
+    if (props.vectorDb !== "llamacloud") {
+      // write loaders configuration (currently Python only)
+      // not needed for LlamaCloud as it has its own loaders
+      await writeLoadersConfig(
+        props.root,
+        props.dataSources,
+        props.useLlamaParse,
+      );
+    }
   } else {
     await installTSTemplate(props);
   }
@@ -141,15 +166,13 @@ export const installTemplate = async (
     // This is a backend, so we need to copy the test data and create the env file.
 
     // Copy the environment file to the target directory.
-    await createBackendEnvFile(props.root, {
-      modelConfig: props.modelConfig,
-      llamaCloudKey: props.llamaCloudKey,
-      vectorDb: props.vectorDb,
-      framework: props.framework,
-      dataSources: props.dataSources,
-      port: props.externalPort,
-      tools: props.tools,
-    });
+    if (
+      props.template === "streaming" ||
+      props.template === "multiagent" ||
+      props.template === "extractor"
+    ) {
+      await createBackendEnvFile(props.root, props);
+    }
 
     if (props.dataSources.length > 0) {
       console.log("\nGenerating context data...\n");
@@ -172,14 +195,15 @@ export const installTemplate = async (
       }
     }
 
-    // Create tool-output directory
-    if (props.tools && props.tools.length > 0) {
-      await fsExtra.mkdir(path.join(props.root, "tool-output"));
-    }
+    // Create outputs directory
+    await makeDir(path.join(props.root, "output/tools"));
+    await makeDir(path.join(props.root, "output/uploaded"));
+    await makeDir(path.join(props.root, "output/llamacloud"));
   } else {
     // this is a frontend for a full-stack app, create .env file with model information
     await createFrontendEnvFile(props.root, {
       customApiPath: props.customApiPath,
+      vectorDb: props.vectorDb,
     });
   }
 };

helpers/providers/azure.ts

@@ -0,0 +1,118 @@
+import ciInfo from "ci-info";
+import prompts from "prompts";
+import { ModelConfigParams, ModelConfigQuestionsParams } from ".";
+import { questionHandlers } from "../../questions";
+
+const ALL_AZURE_OPENAI_CHAT_MODELS: Record<string, { openAIModel: string }> = {
+  "gpt-35-turbo": { openAIModel: "gpt-3.5-turbo" },
+  "gpt-35-turbo-16k": {
+    openAIModel: "gpt-3.5-turbo-16k",
+  },
+  "gpt-4o": { openAIModel: "gpt-4o" },
+  "gpt-4o-mini": { openAIModel: "gpt-4o-mini" },
+  "gpt-4": { openAIModel: "gpt-4" },
+  "gpt-4-32k": { openAIModel: "gpt-4-32k" },
+  "gpt-4-turbo": {
+    openAIModel: "gpt-4-turbo",
+  },
+  "gpt-4-turbo-2024-04-09": {
+    openAIModel: "gpt-4-turbo",
+  },
+  "gpt-4-vision-preview": {
+    openAIModel: "gpt-4-vision-preview",
+  },
+  "gpt-4-1106-preview": {
+    openAIModel: "gpt-4-1106-preview",
+  },
+  "gpt-4o-2024-05-13": {
+    openAIModel: "gpt-4o-2024-05-13",
+  },
+  "gpt-4o-mini-2024-07-18": {
+    openAIModel: "gpt-4o-mini-2024-07-18",
+  },
+};
+
+const ALL_AZURE_OPENAI_EMBEDDING_MODELS: Record<
+  string,
+  {
+    dimensions: number;
+    openAIModel: string;
+  }
+> = {
+  "text-embedding-3-small": {
+    dimensions: 1536,
+    openAIModel: "text-embedding-3-small",
+  },
+  "text-embedding-3-large": {
+    dimensions: 3072,
+    openAIModel: "text-embedding-3-large",
+  },
+};
+
+const DEFAULT_MODEL = "gpt-4o";
+const DEFAULT_EMBEDDING_MODEL = "text-embedding-3-large";
+
+export async function askAzureQuestions({
+  openAiKey,
+  askModels,
+}: ModelConfigQuestionsParams): Promise<ModelConfigParams> {
+  const config: ModelConfigParams = {
+    apiKey: openAiKey || process.env.AZURE_OPENAI_KEY,
+    model: DEFAULT_MODEL,
+    embeddingModel: DEFAULT_EMBEDDING_MODEL,
+    dimensions: getDimensions(DEFAULT_EMBEDDING_MODEL),
+    isConfigured(): boolean {
+      // the Azure model provider can't be fully configured as endpoint and deployment names have to be configured with env variables
+      return false;
+    },
+  };
+
+  // use default model values in CI or if user should not be asked
+  const useDefaults = ciInfo.isCI || !askModels;
+  if (!useDefaults) {
+    const { model } = await prompts(
+      {
+        type: "select",
+        name: "model",
+        message: "Which LLM model would you like to use?",
+        choices: getAvailableModelChoices(),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.model = model;
+
+    const { embeddingModel } = await prompts(
+      {
+        type: "select",
+        name: "embeddingModel",
+        message: "Which embedding model would you like to use?",
+        choices: getAvailableEmbeddingModelChoices(),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.embeddingModel = embeddingModel;
+    config.dimensions = getDimensions(embeddingModel);
+  }
+
+  return config;
+}
+
+function getAvailableModelChoices() {
+  return Object.keys(ALL_AZURE_OPENAI_CHAT_MODELS).map((key) => ({
+    title: key,
+    value: key,
+  }));
+}
+
+function getAvailableEmbeddingModelChoices() {
+  return Object.keys(ALL_AZURE_OPENAI_EMBEDDING_MODELS).map((key) => ({
+    title: key,
+    value: key,
+  }));
+}
+
+function getDimensions(modelName: string) {
+  return ALL_AZURE_OPENAI_EMBEDDING_MODELS[modelName].dimensions;
+}

helpers/providers/index.ts

@@ -1,10 +1,13 @@
 import ciInfo from "ci-info";
 import prompts from "prompts";
 import { questionHandlers } from "../../questions";
-import { ModelConfig, ModelProvider } from "../types";
+import { ModelConfig, ModelProvider, TemplateFramework } from "../types";
 import { askAnthropicQuestions } from "./anthropic";
+import { askAzureQuestions } from "./azure";
 import { askGeminiQuestions } from "./gemini";
 import { askGroqQuestions } from "./groq";
+import { askLLMHubQuestions } from "./llmhub";
+import { askMistralQuestions } from "./mistral";
 import { askOllamaQuestions } from "./ollama";
 import { askOpenAIQuestions } from "./openai";
 
@@ -13,6 +16,7 @@ const DEFAULT_MODEL_PROVIDER = "openai";
 export type ModelConfigQuestionsParams = {
   openAiKey?: string;
   askModels: boolean;
+  framework?: TemplateFramework;
 };
 
 export type ModelConfigParams = Omit<ModelConfig, "provider">;
@@ -20,24 +24,29 @@ export type ModelConfigParams = Omit<ModelConfig, "provider">;
 export async function askModelConfig({
   askModels,
   openAiKey,
+  framework,
 }: ModelConfigQuestionsParams): Promise<ModelConfig> {
   let modelProvider: ModelProvider = DEFAULT_MODEL_PROVIDER;
   if (askModels && !ciInfo.isCI) {
+    let choices = [
+      { title: "OpenAI", value: "openai" },
+      { title: "Groq", value: "groq" },
+      { title: "Ollama", value: "ollama" },
+      { title: "Anthropic", value: "anthropic" },
+      { title: "Gemini", value: "gemini" },
+      { title: "Mistral", value: "mistral" },
+      { title: "AzureOpenAI", value: "azure-openai" },
+    ];
+
+    if (framework === "fastapi") {
+      choices.push({ title: "T-Systems", value: "t-systems" });
+    }
     const { provider } = await prompts(
       {
         type: "select",
         name: "provider",
         message: "Which model provider would you like to use",
-        choices: [
-          {
-            title: "OpenAI",
-            value: "openai",
-          },
-          { title: "Groq", value: "groq" },
-          { title: "Ollama", value: "ollama" },
-          { title: "Anthropic", value: "anthropic" },
-          { title: "Gemini", value: "gemini" },
-        ],
+        choices: choices,
         initial: 0,
       },
       questionHandlers,
@@ -59,6 +68,15 @@ export async function askModelConfig({
     case "gemini":
       modelConfig = await askGeminiQuestions({ askModels });
       break;
+    case "mistral":
+      modelConfig = await askMistralQuestions({ askModels });
+      break;
+    case "azure-openai":
+      modelConfig = await askAzureQuestions({ askModels });
+      break;
+    case "t-systems":
+      modelConfig = await askLLMHubQuestions({ askModels });
+      break;
     default:
       modelConfig = await askOpenAIQuestions({
         openAiKey,

helpers/providers/llmhub.ts

@@ -0,0 +1,169 @@
+import ciInfo from "ci-info";
+import got from "got";
+import ora from "ora";
+import { red } from "picocolors";
+import prompts from "prompts";
+import { ModelConfigParams } from ".";
+import { questionHandlers } from "../../questions";
+
+export const TSYSTEMS_LLMHUB_API_URL =
+  "https://llm-server.llmhub.t-systems.net/v2";
+
+const DEFAULT_MODEL = "gpt-3.5-turbo";
+const DEFAULT_EMBEDDING_MODEL = "text-embedding-3-large";
+
+const LLMHUB_MODELS = [
+  "gpt-35-turbo",
+  "gpt-4-32k-1",
+  "gpt-4-32k-canada",
+  "gpt-4-32k-france",
+  "gpt-4-turbo-128k-france",
+  "Llama2-70b-Instruct",
+  "Llama-3-70B-Instruct",
+  "Mixtral-8x7B-Instruct-v0.1",
+  "mistral-large-32k-france",
+  "CodeLlama-2",
+];
+const LLMHUB_EMBEDDING_MODELS = [
+  "text-embedding-ada-002",
+  "text-embedding-ada-002-france",
+  "jina-embeddings-v2-base-de",
+  "jina-embeddings-v2-base-code",
+  "text-embedding-bge-m3",
+];
+
+type LLMHubQuestionsParams = {
+  apiKey?: string;
+  askModels: boolean;
+};
+
+export async function askLLMHubQuestions({
+  askModels,
+  apiKey,
+}: LLMHubQuestionsParams): Promise<ModelConfigParams> {
+  const config: ModelConfigParams = {
+    apiKey,
+    model: DEFAULT_MODEL,
+    embeddingModel: DEFAULT_EMBEDDING_MODEL,
+    dimensions: getDimensions(DEFAULT_EMBEDDING_MODEL),
+    isConfigured(): boolean {
+      if (config.apiKey) {
+        return true;
+      }
+      if (process.env["T_SYSTEMS_LLMHUB_API_KEY"]) {
+        return true;
+      }
+      return false;
+    },
+  };
+
+  if (!config.apiKey) {
+    const { key } = await prompts(
+      {
+        type: "text",
+        name: "key",
+        message: askModels
+          ? "Please provide your LLMHub API key (or leave blank to use T_SYSTEMS_LLMHUB_API_KEY env variable):"
+          : "Please provide your LLMHub API key (leave blank to skip):",
+        validate: (value: string) => {
+          if (askModels && !value) {
+            if (process.env.T_SYSTEMS_LLMHUB_API_KEY) {
+              return true;
+            }
+            return "T_SYSTEMS_LLMHUB_API_KEY env variable is not set - key is required";
+          }
+          return true;
+        },
+      },
+      questionHandlers,
+    );
+    config.apiKey = key || process.env.T_SYSTEMS_LLMHUB_API_KEY;
+  }
+
+  // use default model values in CI or if user should not be asked
+  const useDefaults = ciInfo.isCI || !askModels;
+  if (!useDefaults) {
+    const { model } = await prompts(
+      {
+        type: "select",
+        name: "model",
+        message: "Which LLM model would you like to use?",
+        choices: await getAvailableModelChoices(false, config.apiKey),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.model = model;
+
+    const { embeddingModel } = await prompts(
+      {
+        type: "select",
+        name: "embeddingModel",
+        message: "Which embedding model would you like to use?",
+        choices: await getAvailableModelChoices(true, config.apiKey),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.embeddingModel = embeddingModel;
+    config.dimensions = getDimensions(embeddingModel);
+  }
+
+  return config;
+}
+
+async function getAvailableModelChoices(
+  selectEmbedding: boolean,
+  apiKey?: string,
+) {
+  if (!apiKey) {
+    throw new Error("Need LLMHub key to retrieve model choices");
+  }
+  const isLLMModel = (modelId: string) => {
+    return LLMHUB_MODELS.includes(modelId);
+  };
+
+  const isEmbeddingModel = (modelId: string) => {
+    return LLMHUB_EMBEDDING_MODELS.includes(modelId);
+  };
+
+  const spinner = ora("Fetching available models").start();
+  try {
+    const response = await got(`${TSYSTEMS_LLMHUB_API_URL}/models`, {
+      headers: {
+        Authorization: "Bearer " + apiKey,
+      },
+      timeout: 5000,
+      responseType: "json",
+    });
+    const data: any = await response.body;
+    spinner.stop();
+    return data.data
+      .filter((model: any) =>
+        selectEmbedding ? isEmbeddingModel(model.id) : isLLMModel(model.id),
+      )
+      .map((el: any) => {
+        return {
+          title: el.id,
+          value: el.id,
+        };
+      });
+  } catch (error) {
+    spinner.stop();
+    if ((error as any).response?.statusCode === 401) {
+      console.log(
+        red(
+          "Invalid LLMHub API key provided! Please provide a valid key and try again!",
+        ),
+      );
+    } else {
+      console.log(red("Request failed: " + error));
+    }
+    process.exit(1);
+  }
+}
+
+function getDimensions(modelName: string) {
+  // Assuming dimensions similar to OpenAI for simplicity. Update if different.
+  return modelName === "text-embedding-004" ? 768 : 1536;
+}

helpers/providers/mistral.ts

@@ -0,0 +1,86 @@
+import ciInfo from "ci-info";
+import prompts from "prompts";
+import { ModelConfigParams } from ".";
+import { questionHandlers, toChoice } from "../../questions";
+
+const MODELS = ["mistral-tiny", "mistral-small", "mistral-medium"];
+type ModelData = {
+  dimensions: number;
+};
+const EMBEDDING_MODELS: Record<string, ModelData> = {
+  "mistral-embed": { dimensions: 1024 },
+};
+
+const DEFAULT_MODEL = MODELS[0];
+const DEFAULT_EMBEDDING_MODEL = Object.keys(EMBEDDING_MODELS)[0];
+const DEFAULT_DIMENSIONS = Object.values(EMBEDDING_MODELS)[0].dimensions;
+
+type MistralQuestionsParams = {
+  apiKey?: string;
+  askModels: boolean;
+};
+
+export async function askMistralQuestions({
+  askModels,
+  apiKey,
+}: MistralQuestionsParams): Promise<ModelConfigParams> {
+  const config: ModelConfigParams = {
+    apiKey,
+    model: DEFAULT_MODEL,
+    embeddingModel: DEFAULT_EMBEDDING_MODEL,
+    dimensions: DEFAULT_DIMENSIONS,
+    isConfigured(): boolean {
+      if (config.apiKey) {
+        return true;
+      }
+      if (process.env["MISTRAL_API_KEY"]) {
+        return true;
+      }
+      return false;
+    },
+  };
+
+  if (!config.apiKey) {
+    const { key } = await prompts(
+      {
+        type: "text",
+        name: "key",
+        message:
+          "Please provide your Mistral API key (or leave blank to use MISTRAL_API_KEY env variable):",
+      },
+      questionHandlers,
+    );
+    config.apiKey = key || process.env.MISTRAL_API_KEY;
+  }
+
+  // use default model values in CI or if user should not be asked
+  const useDefaults = ciInfo.isCI || !askModels;
+  if (!useDefaults) {
+    const { model } = await prompts(
+      {
+        type: "select",
+        name: "model",
+        message: "Which LLM model would you like to use?",
+        choices: MODELS.map(toChoice),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.model = model;
+
+    const { embeddingModel } = await prompts(
+      {
+        type: "select",
+        name: "embeddingModel",
+        message: "Which embedding model would you like to use?",
+        choices: Object.keys(EMBEDDING_MODELS).map(toChoice),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.embeddingModel = embeddingModel;
+    config.dimensions = EMBEDDING_MODELS[embeddingModel].dimensions;
+  }
+
+  return config;
+}

helpers/providers/openai.ts

@@ -8,7 +8,7 @@ import { questionHandlers } from "../../questions";
 
 const OPENAI_API_URL = "https://api.openai.com/v1";
 
-const DEFAULT_MODEL = "gpt-3.5-turbo";
+const DEFAULT_MODEL = "gpt-4o-mini";
 const DEFAULT_EMBEDDING_MODEL = "text-embedding-3-large";
 
 export async function askOpenAIQuestions({

helpers/python.ts

@@ -55,11 +55,11 @@ const getAdditionalDependencies = (
     case "milvus": {
       dependencies.push({
         name: "llama-index-vector-stores-milvus",
-        version: "^0.1.6",
+        version: "^0.1.20",
       });
       dependencies.push({
         name: "pymilvus",
-        version: "2.3.7",
+        version: "2.4.4",
       });
       break;
     }
@@ -84,6 +84,13 @@ const getAdditionalDependencies = (
       });
       break;
     }
+    case "weaviate": {
+      dependencies.push({
+        name: "llama-index-vector-stores-weaviate",
+        version: "^1.0.2",
+      });
+      break;
+    }
   }
 
   // Add data source dependencies
@@ -118,6 +125,12 @@ const getAdditionalDependencies = (
             version: "^2.9.9",
           });
           break;
+        case "llamacloud":
+          dependencies.push({
+            name: "llama-index-indices-managed-llama-cloud",
+            version: "^0.2.7",
+          });
+          break;
       }
     }
   }
@@ -147,14 +160,24 @@ const getAdditionalDependencies = (
         version: "0.2.6",
       });
       break;
+    case "groq":
+      dependencies.push({
+        name: "llama-index-llms-groq",
+        version: "0.1.4",
+      });
+      dependencies.push({
+        name: "llama-index-embeddings-fastembed",
+        version: "^0.1.4",
+      });
+      break;
     case "anthropic":
       dependencies.push({
         name: "llama-index-llms-anthropic",
         version: "0.1.10",
       });
       dependencies.push({
-        name: "llama-index-embeddings-huggingface",
-        version: "0.2.0",
+        name: "llama-index-embeddings-fastembed",
+        version: "^0.1.4",
       });
       break;
     case "gemini":
@@ -167,6 +190,36 @@ const getAdditionalDependencies = (
         version: "0.1.6",
       });
       break;
+    case "mistral":
+      dependencies.push({
+        name: "llama-index-llms-mistralai",
+        version: "0.1.17",
+      });
+      dependencies.push({
+        name: "llama-index-embeddings-mistralai",
+        version: "0.1.4",
+      });
+      break;
+    case "azure-openai":
+      dependencies.push({
+        name: "llama-index-llms-azure-openai",
+        version: "0.1.10",
+      });
+      dependencies.push({
+        name: "llama-index-embeddings-azure-openai",
+        version: "0.1.11",
+      });
+      break;
+    case "t-systems":
+      dependencies.push({
+        name: "llama-index-agent-openai",
+        version: "0.2.2",
+      });
+      dependencies.push({
+        name: "llama-index-llms-openai-like",
+        version: "0.1.3",
+      });
+      break;
   }
 
   return dependencies;
@@ -297,26 +350,36 @@ export const installPythonTemplate = async ({
     cwd: path.join(compPath, "vectordbs", "python", vectorDb ?? "none"),
   });
 
-  // Copy all loaders to enginePath
-  const loaderPath = path.join(enginePath, "loaders");
-  await copy("**", loaderPath, {
-    parents: true,
-    cwd: path.join(compPath, "loaders", "python"),
+  if (vectorDb !== "llamacloud") {
+    // Copy all loaders to enginePath
+    // Not needed for LlamaCloud as it has its own loaders
+    const loaderPath = path.join(enginePath, "loaders");
+    await copy("**", loaderPath, {
+      parents: true,
+      cwd: path.join(compPath, "loaders", "python"),
+    });
+  }
+
+  // Copy settings.py to app
+  await copy("**", path.join(root, "app"), {
+    cwd: path.join(compPath, "settings", "python"),
   });
 
-  // Select and copy engine code based on data sources and tools
-  let engine;
-  tools = tools ?? [];
-  if (dataSources.length > 0 && tools.length === 0) {
-    console.log("\nNo tools selected - use optimized context chat engine\n");
-    engine = "chat";
-  } else {
-    engine = "agent";
+  if (template === "streaming") {
+    // For the streaming template only:
+    // Select and copy engine code based on data sources and tools
+    let engine;
+    if (dataSources.length > 0 && (!tools || tools.length === 0)) {
+      console.log("\nNo tools selected - use optimized context chat engine\n");
+      engine = "chat";
+    } else {
+      engine = "agent";
+    }
+    await copy("**", enginePath, {
+      parents: true,
+      cwd: path.join(compPath, "engines", "python", engine),
+    });
   }
-  await copy("**", enginePath, {
-    parents: true,
-    cwd: path.join(compPath, "engines", "python", engine),
-  });
 
   console.log("Adding additional dependencies");
 
@@ -327,18 +390,27 @@ export const installPythonTemplate = async ({
     tools,
   );
 
-  if (observability === "opentelemetry") {
-    addOnDependencies.push({
-      name: "traceloop-sdk",
-      version: "^0.15.11",
-    });
+  if (observability && observability !== "none") {
+    if (observability === "traceloop") {
+      addOnDependencies.push({
+        name: "traceloop-sdk",
+        version: "^0.15.11",
+      });
+    }
+
+    if (observability === "llamatrace") {
+      addOnDependencies.push({
+        name: "llama-index-callbacks-arize-phoenix",
+        version: "^0.1.6",
+      });
+    }
 
     const templateObservabilityPath = path.join(
       templatesDir,
       "components",
       "observability",
       "python",
-      "opentelemetry",
+      observability,
     );
     await copy("**", path.join(root, "app"), {
       cwd: templateObservabilityPath,

helpers/tools.ts

@@ -170,6 +170,53 @@ For better results, you can specify the region parameter to get results from a s
       },
     ],
   },
+  {
+    display: "Image Generator",
+    name: "img_gen",
+    supportedFrameworks: ["fastapi", "express", "nextjs"],
+    type: ToolType.LOCAL,
+    envVars: [
+      {
+        name: "STABILITY_API_KEY",
+        description:
+          "STABILITY_API_KEY key is required to run image generator. Get it here: https://platform.stability.ai/account/keys",
+      },
+      {
+        name: TOOL_SYSTEM_PROMPT_ENV_VAR,
+        description: "System prompt for image generator tool.",
+        value: `You are an image generator agent. You help users to generate images using the Stability API.`,
+      },
+    ],
+  },
+  {
+    display: "Azure Code Interpreter",
+    name: "azure_code_interpreter.AzureCodeInterpreterToolSpec",
+    supportedFrameworks: ["fastapi", "nextjs", "express"],
+    type: ToolType.LLAMAHUB,
+    dependencies: [
+      {
+        name: "llama-index-tools-azure-code-interpreter",
+        version: "0.2.0",
+      },
+    ],
+    envVars: [
+      {
+        name: "AZURE_POOL_MANAGEMENT_ENDPOINT",
+        description:
+          "Please follow this guideline to create and get the pool management endpoint: https://learn.microsoft.com/azure/container-apps/sessions?tabs=azure-cli",
+      },
+      {
+        name: TOOL_SYSTEM_PROMPT_ENV_VAR,
+        description: "System prompt for Azure code interpreter tool.",
+        value: `-You are a Python interpreter that can run any python code in a secure environment.
+- The python code runs in a Jupyter notebook. Every time you call the 'interpreter' tool, the python code is executed in a separate cell. 
+- You are given tasks to complete and you run python code to solve them.
+- It's okay to make multiple calls to interpreter tool. If you get an error or the result is not what you expected, you can call the tool again. Don't give up too soon!
+- Plot visualizations using matplotlib or any other visualization library directly in the notebook.
+- You can install any pip package (if it exists) by running a cell with pip install.`,
+      },
+    ],
+  },
 ];
 
 export const getTool = (toolName: string): Tool | undefined => {

helpers/types.ts

@@ -6,7 +6,10 @@ export type ModelProvider =
   | "groq"
   | "ollama"
   | "anthropic"
-  | "gemini";
+  | "gemini"
+  | "mistral"
+  | "azure-openai"
+  | "t-systems";
 export type ModelConfig = {
   provider: ModelProvider;
   apiKey?: string;
@@ -15,7 +18,12 @@ export type ModelConfig = {
   dimensions: number;
   isConfigured(): boolean;
 };
-export type TemplateType = "streaming" | "community" | "llamapack";
+export type TemplateType =
+  | "extractor"
+  | "streaming"
+  | "community"
+  | "llamapack"
+  | "multiagent";
 export type TemplateFramework = "nextjs" | "express" | "fastapi";
 export type TemplateUI = "html" | "shadcn";
 export type TemplateVectorDB =
@@ -26,7 +34,9 @@ export type TemplateVectorDB =
   | "milvus"
   | "astra"
   | "qdrant"
-  | "chroma";
+  | "chroma"
+  | "llamacloud"
+  | "weaviate";
 export type TemplatePostInstallAction =
   | "none"
   | "VSCode"
@@ -36,8 +46,8 @@ export type TemplateDataSource = {
   type: TemplateDataSourceType;
   config: TemplateDataSourceConfig;
 };
-export type TemplateDataSourceType = "file" | "web" | "db";
-export type TemplateObservability = "none" | "opentelemetry";
+export type TemplateDataSourceType = "file" | "web" | "db" | "llamacloud";
+export type TemplateObservability = "none" | "traceloop" | "llamatrace";
 // Config for both file and folder
 export type FileSourceConfig = {
   path: string;

helpers/typescript.ts

@@ -1,7 +1,7 @@
 import fs from "fs/promises";
 import os from "os";
 import path from "path";
-import { bold, cyan } from "picocolors";
+import { bold, cyan, yellow } from "picocolors";
 import { assetRelocator, copy } from "../helpers/copy";
 import { callPackageManager } from "../helpers/install";
 import { templatesDir } from "./dir";
@@ -70,7 +70,7 @@ export const installTSTemplate = async ({
     );
 
     const webpackConfigOtelFile = path.join(root, "webpack.config.o11y.mjs");
-    if (observability === "opentelemetry") {
+    if (observability === "traceloop") {
       const webpackConfigDefaultFile = path.join(root, "webpack.config.mjs");
       await fs.rm(webpackConfigDefaultFile);
       await fs.rename(webpackConfigOtelFile, webpackConfigDefaultFile);
@@ -104,8 +104,20 @@ export const installTSTemplate = async ({
       : path.join("src", "controllers");
   const enginePath = path.join(root, relativeEngineDestPath, "engine");
 
+  // copy llamaindex code for TS templates
+  await copy("**", path.join(root, relativeEngineDestPath, "llamaindex"), {
+    parents: true,
+    cwd: path.join(compPath, "llamaindex", "typescript"),
+  });
+
   // copy vector db component
-  console.log("\nUsing vector DB:", vectorDb ?? "none", "\n");
+  if (vectorDb === "llamacloud") {
+    console.log(
+      `\nUsing managed index from LlamaCloud. Ensure the ${yellow("LLAMA_CLOUD_* environment variables are set correctly.")}`,
+    );
+  } else {
+    console.log("\nUsing vector DB:", vectorDb ?? "none");
+  }
   await copy("**", enginePath, {
     parents: true,
     cwd: path.join(compPath, "vectordbs", "typescript", vectorDb ?? "none"),
@@ -236,7 +248,7 @@ async function updatePackageJson({
     };
   }
 
-  if (observability === "opentelemetry") {
+  if (observability === "traceloop") {
     packageJson.dependencies = {
       ...packageJson.dependencies,
       "@traceloop/node-server-sdk": "^0.5.19",

index.ts

@@ -9,7 +9,7 @@ import prompts from "prompts";
 import terminalLink from "terminal-link";
 import checkForUpdate from "update-check";
 import { createApp } from "./create-app";
-import { getDataSources } from "./helpers/datasources";
+import { EXAMPLE_FILE, getDataSources } from "./helpers/datasources";
 import { getPkgManager } from "./helpers/get-pkg-manager";
 import { isFolderEmpty } from "./helpers/is-folder-empty";
 import { initializeGlobalAgent } from "./helpers/proxy";
@@ -194,8 +194,16 @@ if (process.argv.includes("--no-llama-parse")) {
 program.askModels = process.argv.includes("--ask-models");
 if (process.argv.includes("--no-files")) {
   program.dataSources = [];
-} else {
+} else if (process.argv.includes("--example-file")) {
   program.dataSources = getDataSources(program.files, program.exampleFile);
+} else if (process.argv.includes("--llamacloud")) {
+  program.dataSources = [
+    {
+      type: "llamacloud",
+      config: {},
+    },
+    EXAMPLE_FILE,
+  ];
 }
 
 const packageManager = !!program.useNpm

package.json

@@ -1,6 +1,6 @@
 {
   "name": "create-llama",
-  "version": "0.1.12",
+  "version": "0.1.37",
   "description": "Create LlamaIndex-powered apps with one command",
   "keywords": [
     "rag",
@@ -9,7 +9,7 @@
   ],
   "repository": {
     "type": "git",
-    "url": "https://github.com/run-llama/LlamaIndexTS",
+    "url": "https://github.com/run-llama/create-llama",
     "directory": "packages/create-llama"
   },
   "license": "MIT",

questions.ts

@@ -9,6 +9,7 @@ import {
   TemplateDataSource,
   TemplateDataSourceType,
   TemplateFramework,
+  TemplateType,
 } from "./helpers";
 import { COMMUNITY_OWNER, COMMUNITY_REPO } from "./helpers/constant";
 import { EXAMPLE_FILE } from "./helpers/datasources";
@@ -102,6 +103,7 @@ const getVectorDbChoices = (framework: TemplateFramework) => {
     { title: "Astra", value: "astra" },
     { title: "Qdrant", value: "qdrant" },
     { title: "ChromaDB", value: "chroma" },
+    { title: "Weaviate", value: "weaviate" },
   ];
 
   const vectordbLang = framework === "fastapi" ? "python" : "typescript";
@@ -122,8 +124,15 @@ const getVectorDbChoices = (framework: TemplateFramework) => {
 export const getDataSourceChoices = (
   framework: TemplateFramework,
   selectedDataSource: TemplateDataSource[],
+  template?: TemplateType,
 ) => {
+  // If LlamaCloud is already selected, don't show any other options
+  if (selectedDataSource.find((s) => s.type === "llamacloud")) {
+    return [];
+  }
+
   const choices = [];
+
   if (selectedDataSource.length > 0) {
     choices.push({
       title: "No",
@@ -131,29 +140,37 @@ export const getDataSourceChoices = (
     });
   }
   if (selectedDataSource === undefined || selectedDataSource.length === 0) {
+    if (template !== "multiagent") {
+      choices.push({
+        title: "No datasource",
+        value: "none",
+      });
+    }
     choices.push({
-      title: "No data, just a simple chat or agent",
-      value: "none",
-    });
-    choices.push({
-      title: "Use an example PDF",
+      title:
+        process.platform !== "linux"
+          ? "Use an example PDF"
+          : "Use an example PDF (you can add your own data files later)",
       value: "exampleFile",
     });
   }
 
-  choices.push(
-    {
-      title: `Use local files (${supportedContextFileTypes.join(", ")})`,
-      value: "file",
-    },
-    {
-      title:
-        process.platform === "win32"
-          ? "Use a local folder"
-          : "Use local folders",
-      value: "folder",
-    },
-  );
+  // Linux has many distros so we won't support file/folder picker for now
+  if (process.platform !== "linux") {
+    choices.push(
+      {
+        title: `Use local files (${supportedContextFileTypes.join(", ")})`,
+        value: "file",
+      },
+      {
+        title:
+          process.platform === "win32"
+            ? "Use a local folder"
+            : "Use local folders",
+        value: "folder",
+      },
+    );
+  }
 
   if (framework === "fastapi") {
     choices.push({
@@ -165,6 +182,13 @@ export const getDataSourceChoices = (
       value: "db",
     });
   }
+
+  if (!selectedDataSource.length) {
+    choices.push({
+      title: "Use managed index from LlamaCloud",
+      value: "llamacloud",
+    });
+  }
   return choices;
 };
 
@@ -262,25 +286,27 @@ export const askQuestions = async (
           },
         ];
 
-        const modelConfigured =
-          !program.llamapack && program.modelConfig.isConfigured();
-        // If using LlamaParse, require LlamaCloud API key
-        const llamaCloudKeyConfigured = program.useLlamaParse
-          ? program.llamaCloudKey || process.env["LLAMA_CLOUD_API_KEY"]
-          : true;
-        const hasVectorDb = program.vectorDb && program.vectorDb !== "none";
-        // Can run the app if all tools do not require configuration
-        if (
-          !hasVectorDb &&
-          modelConfigured &&
-          llamaCloudKeyConfigured &&
-          !toolsRequireConfig(program.tools)
-        ) {
-          actionChoices.push({
-            title:
-              "Generate code, install dependencies, and run the app (~2 min)",
-            value: "runApp",
-          });
+        if (program.template !== "multiagent") {
+          const modelConfigured =
+            !program.llamapack && program.modelConfig.isConfigured();
+          // If using LlamaParse, require LlamaCloud API key
+          const llamaCloudKeyConfigured = program.useLlamaParse
+            ? program.llamaCloudKey || process.env["LLAMA_CLOUD_API_KEY"]
+            : true;
+          const hasVectorDb = program.vectorDb && program.vectorDb !== "none";
+          // Can run the app if all tools do not require configuration
+          if (
+            !hasVectorDb &&
+            modelConfigured &&
+            llamaCloudKeyConfigured &&
+            !toolsRequireConfig(program.tools)
+          ) {
+            actionChoices.push({
+              title:
+                "Generate code, install dependencies, and run the app (~2 min)",
+              value: "runApp",
+            });
+          }
         }
 
         const { action } = await prompts(
@@ -312,7 +338,12 @@ export const askQuestions = async (
           name: "template",
           message: "Which template would you like to use?",
           choices: [
-            { title: "Chat", value: "streaming" },
+            { title: "Agentic RAG (single agent)", value: "streaming" },
+            {
+              title: "Multi-agent app (using llama-agents)",
+              value: "multiagent",
+            },
+            { title: "Structured Extractor", value: "extractor" },
             {
               title: `Community template from ${styledRepo}`,
               value: "community",
@@ -376,6 +407,10 @@ export const askQuestions = async (
     return; // early return - no further questions needed for llamapack projects
   }
 
+  if (program.template === "multiagent" || program.template === "extractor") {
+    // TODO: multi-agents currently only supports FastAPI
+    program.framework = preferences.framework = "fastapi";
+  }
   if (!program.framework) {
     if (ciInfo.isCI) {
       program.framework = getPrefOrDefault("framework");
@@ -401,7 +436,10 @@ export const askQuestions = async (
     }
   }
 
-  if (program.framework === "express" || program.framework === "fastapi") {
+  if (
+    (program.framework === "express" || program.framework === "fastapi") &&
+    program.template === "streaming"
+  ) {
     // if a backend-only framework is selected, ask whether we should create a frontend
     if (program.frontend === undefined) {
       if (ciInfo.isCI) {
@@ -438,7 +476,7 @@ export const askQuestions = async (
     }
   }
 
-  if (!program.observability) {
+  if (!program.observability && program.template === "streaming") {
     if (ciInfo.isCI) {
       program.observability = getPrefOrDefault("observability");
     } else {
@@ -449,7 +487,10 @@ export const askQuestions = async (
           message: "Would you like to set up observability?",
           choices: [
             { title: "No", value: "none" },
-            { title: "OpenTelemetry", value: "opentelemetry" },
+            ...(program.framework === "fastapi"
+              ? [{ title: "LlamaTrace", value: "llamatrace" }]
+              : []),
+            { title: "Traceloop", value: "traceloop" },
           ],
           initial: 0,
         },
@@ -465,6 +506,7 @@ export const askQuestions = async (
     const modelConfig = await askModelConfig({
       openAiKey,
       askModels: program.askModels ?? false,
+      framework: program.framework,
     });
     program.modelConfig = modelConfig;
     preferences.modelConfig = modelConfig;
@@ -478,6 +520,12 @@ export const askQuestions = async (
       // continue asking user for data sources if none are initially provided
       while (true) {
         const firstQuestion = program.dataSources.length === 0;
+        const choices = getDataSourceChoices(
+          program.framework,
+          program.dataSources,
+          program.template,
+        );
+        if (choices.length === 0) break;
         const { selectedSource } = await prompts(
           {
             type: "select",
@@ -485,10 +533,7 @@ export const askQuestions = async (
             message: firstQuestion
               ? "Which data source would you like to use?"
               : "Would you like to add another data source?",
-            choices: getDataSourceChoices(
-              program.framework,
-              program.dataSources,
-            ),
+            choices,
             initial: firstQuestion ? 1 : 0,
           },
           questionHandlers,
@@ -585,51 +630,82 @@ export const askQuestions = async (
               config: await prompts(dbPrompts, questionHandlers),
             });
           }
+          case "llamacloud": {
+            program.dataSources.push({
+              type: "llamacloud",
+              config: {},
+            });
+            program.dataSources.push(EXAMPLE_FILE);
+            break;
+          }
         }
       }
     }
   }
 
-  // Asking for LlamaParse if user selected file or folder data source
-  if (
-    program.dataSources.some((ds) => ds.type === "file") &&
-    program.useLlamaParse === undefined
-  ) {
-    if (ciInfo.isCI) {
-      program.useLlamaParse = getPrefOrDefault("useLlamaParse");
-      program.llamaCloudKey = getPrefOrDefault("llamaCloudKey");
-    } else {
-      const { useLlamaParse } = await prompts(
-        {
-          type: "toggle",
-          name: "useLlamaParse",
-          message:
-            "Would you like to use LlamaParse (improved parser for RAG - requires API key)?",
-          initial: false,
-          active: "yes",
-          inactive: "no",
-        },
-        questionHandlers,
-      );
-      program.useLlamaParse = useLlamaParse;
+  const isUsingLlamaCloud = program.dataSources.some(
+    (ds) => ds.type === "llamacloud",
+  );
 
-      // Ask for LlamaCloud API key
-      if (useLlamaParse && program.llamaCloudKey === undefined) {
+  // Asking for LlamaParse if user selected file data source
+  if (isUsingLlamaCloud) {
+    // default to use LlamaParse if using LlamaCloud
+    program.useLlamaParse = preferences.useLlamaParse = true;
+  } else {
+    if (program.useLlamaParse === undefined) {
+      // if already set useLlamaParse, don't ask again
+      if (program.dataSources.some((ds) => ds.type === "file")) {
+        if (ciInfo.isCI) {
+          program.useLlamaParse = getPrefOrDefault("useLlamaParse");
+        } else {
+          const { useLlamaParse } = await prompts(
+            {
+              type: "toggle",
+              name: "useLlamaParse",
+              message:
+                "Would you like to use LlamaParse (improved parser for RAG - requires API key)?",
+              initial: false,
+              active: "yes",
+              inactive: "no",
+            },
+            questionHandlers,
+          );
+          program.useLlamaParse = useLlamaParse;
+          preferences.useLlamaParse = useLlamaParse;
+        }
+      }
+    }
+  }
+
+  // Ask for LlamaCloud API key when using a LlamaCloud index or LlamaParse
+  if (isUsingLlamaCloud || program.useLlamaParse) {
+    if (!program.llamaCloudKey) {
+      // if already set, don't ask again
+      if (ciInfo.isCI) {
+        program.llamaCloudKey = getPrefOrDefault("llamaCloudKey");
+      } else {
+        // Ask for LlamaCloud API key
         const { llamaCloudKey } = await prompts(
           {
             type: "text",
             name: "llamaCloudKey",
             message:
-              "Please provide your LlamaIndex Cloud API key (leave blank to skip):",
+              "Please provide your LlamaCloud API key (leave blank to skip):",
           },
           questionHandlers,
         );
-        program.llamaCloudKey = llamaCloudKey;
+        program.llamaCloudKey = preferences.llamaCloudKey =
+          llamaCloudKey || process.env.LLAMA_CLOUD_API_KEY;
       }
     }
   }
 
-  if (program.dataSources.length > 0 && !program.vectorDb) {
+  if (isUsingLlamaCloud) {
+    // When using a LlamaCloud index, don't ask for vector database and use code in `llamacloud` folder for vector database
+    const vectorDb = "llamacloud";
+    program.vectorDb = vectorDb;
+    preferences.vectorDb = vectorDb;
+  } else if (program.dataSources.length > 0 && !program.vectorDb) {
     if (ciInfo.isCI) {
       program.vectorDb = getPrefOrDefault("vectorDb");
     } else {
@@ -648,7 +724,8 @@ export const askQuestions = async (
     }
   }
 
-  if (!program.tools) {
+  if (!program.tools && program.template === "streaming") {
+    // TODO: allow to select tools also for multi-agent framework
     if (ciInfo.isCI) {
       program.tools = getPrefOrDefault("tools");
     } else {

templates/components/engines/python/agent/__init__.py

@@ -6,7 +6,7 @@ from app.engine.tools import ToolFactory
 from app.engine.index import get_index
 
 
-def get_chat_engine():
+def get_chat_engine(filters=None, params=None):
     system_prompt = os.getenv("SYSTEM_PROMPT")
     top_k = os.getenv("TOP_K", "3")
     tools = []
@@ -14,7 +14,9 @@ def get_chat_engine():
     # Add query tool if index exists
     index = get_index()
     if index is not None:
-        query_engine = index.as_query_engine(similarity_top_k=int(top_k))
+        query_engine = index.as_query_engine(
+            similarity_top_k=int(top_k), filters=filters
+        )
         query_engine_tool = QueryEngineTool.from_defaults(query_engine=query_engine)
         tools.append(query_engine_tool)
 

templates/components/engines/python/agent/tools/__init__.py

@@ -1,8 +1,6 @@
 import os
 import yaml
-import json
 import importlib
-from cachetools import cached, LRUCache
 from llama_index.core.tools.tool_spec.base import BaseToolSpec
 from llama_index.core.tools.function_tool import FunctionTool
 
@@ -13,7 +11,6 @@ class ToolType:
 
 
 class ToolFactory:
-
     TOOL_SOURCE_PACKAGE_MAP = {
         ToolType.LLAMAHUB: "llama_index.tools",
         ToolType.LOCAL: "app.engine.tools",
@@ -31,7 +28,7 @@ class ToolFactory:
                 return tool_spec.to_tool_list()
             else:
                 module = importlib.import_module(f"{source_package}.{tool_name}")
-                tools = module.get_tools()
+                tools = module.get_tools(**config)
                 if not all(isinstance(tool, FunctionTool) for tool in tools):
                     raise ValueError(
                         f"The module {module} does not contain valid tools"

templates/components/engines/python/agent/tools/duckduckgo.py

@@ -32,5 +32,5 @@ def duckduckgo_search(
     return results
 
 
-def get_tools():
+def get_tools(**kwargs):
     return [FunctionTool.from_defaults(duckduckgo_search)]

templates/components/engines/python/agent/tools/img_gen.py

@@ -0,0 +1,108 @@
+import os
+import uuid
+import logging
+import requests
+from typing import Optional
+from pydantic import BaseModel, Field
+from llama_index.core.tools import FunctionTool
+
+logger = logging.getLogger(__name__)
+
+
+class ImageGeneratorToolOutput(BaseModel):
+    is_success: bool = Field(
+        ...,
+        description="Whether the image generation was successful.",
+    )
+    image_url: Optional[str] = Field(
+        None,
+        description="The URL of the generated image.",
+    )
+    error_message: Optional[str] = Field(
+        None,
+        description="The error message if the image generation failed.",
+    )
+
+
+class ImageGeneratorTool:
+    _IMG_OUTPUT_FORMAT = "webp"
+    _IMG_OUTPUT_DIR = "output/tool"
+    _IMG_GEN_API = "https://api.stability.ai/v2beta/stable-image/generate/core"
+
+    def __init__(self, api_key: str = None):
+        if not api_key:
+            api_key = os.getenv("STABILITY_API_KEY")
+        self._api_key = api_key
+        self.fileserver_url_prefix = os.getenv("FILESERVER_URL_PREFIX")
+        if self._api_key is None:
+            raise ValueError(
+                "STABILITY_API_KEY key is required to run image generator. Get it here: https://platform.stability.ai/account/keys"
+            )
+        if self.fileserver_url_prefix is None:
+            raise ValueError("FILESERVER_URL_PREFIX is required.")
+
+    def _prepare_output_dir(self):
+        """
+        Create the output directory if it doesn't exist
+        """
+        if not os.path.exists(self._IMG_OUTPUT_DIR):
+            os.makedirs(self._IMG_OUTPUT_DIR, exist_ok=True)
+
+    def _save_image(self, image_data: bytes):
+        self._prepare_output_dir()
+        filename = f"{uuid.uuid4()}.{self._IMG_OUTPUT_FORMAT}"
+        output_path = os.path.join(self._IMG_OUTPUT_DIR, filename)
+        with open(output_path, "wb") as f:
+            f.write(image_data)
+        url = f"{os.getenv('FILESERVER_URL_PREFIX')}/{self._IMG_OUTPUT_DIR}/{filename}"
+        logger.info(f"Saved image to {output_path}.\nURL: {url}")
+        return url
+
+    def _call_stability_api(self, prompt: str):
+        headers = {
+            "authorization": f"Bearer {self._api_key}",
+            "accept": "image/*",
+        }
+        data = {
+            "prompt": prompt,
+            "output_format": self._IMG_OUTPUT_FORMAT,
+        }
+
+        response = requests.post(
+            self._IMG_GEN_API,
+            headers=headers,
+            files={"none": ""},
+            data=data,
+        )
+        response.raise_for_status()
+
+        return response
+
+    def generate_image(self, prompt: str) -> ImageGeneratorToolOutput:
+        """
+        Use this tool to generate an image based on the prompt.
+        Args:
+            prompt (str): The prompt to generate the image from.
+        """
+
+        try:
+            # Call the Stability API
+            response = self._call_stability_api(prompt)
+
+            # Save the image and get the URL
+            image_url = self._save_image(response.content)
+
+            return ImageGeneratorToolOutput(
+                is_success=True,
+                image_url=image_url,
+            )
+        except Exception as e:
+            logger.exception(e, exc_info=True)
+            return ImageGeneratorToolOutput(
+                is_success=False,
+                error_message=str(e),
+            )
+
+
+def get_tools(**kwargs):
+    return [FunctionTool.from_defaults(ImageGeneratorTool(**kwargs).generate_image)]

templates/components/engines/python/agent/tools/interpreter.py

@@ -3,7 +3,7 @@ import logging
 import base64
 import uuid
 from pydantic import BaseModel
-from typing import List, Tuple, Dict, Optional
+from typing import List, Dict, Optional
 from llama_index.core.tools import FunctionTool
 from e2b_code_interpreter import CodeInterpreter
 from e2b_code_interpreter.models import Logs
@@ -26,11 +26,11 @@ class E2BToolOutput(BaseModel):
 
 
 class E2BCodeInterpreter:
+    output_dir = "output/tool"
 
-    output_dir = "tool-output"
-
-    def __init__(self):
-        api_key = os.getenv("E2B_API_KEY")
+    def __init__(self, api_key: str = None):
+        if api_key is None:
+            api_key = os.getenv("E2B_API_KEY")
         filesever_url_prefix = os.getenv("FILESERVER_URL_PREFIX")
         if not api_key:
             raise ValueError(
@@ -138,5 +138,5 @@ class E2BCodeInterpreter:
         return output
 
 
-def get_tools():
-    return [FunctionTool.from_defaults(E2BCodeInterpreter().interpret)]
+def get_tools(**kwargs):
+    return [FunctionTool.from_defaults(E2BCodeInterpreter(**kwargs).interpret)]

templates/components/engines/python/agent/tools/weather.py

@@ -69,5 +69,5 @@ class OpenMeteoWeather:
         return response.json()
 
 
-def get_tools():
+def get_tools(**kwargs):
     return [FunctionTool.from_defaults(OpenMeteoWeather.get_weather_information)]

templates/components/engines/python/chat/__init__.py

@@ -1,13 +1,22 @@
 import os
+
 from app.engine.index import get_index
+from app.engine.node_postprocessors import NodeCitationProcessor
 from fastapi import HTTPException
+from llama_index.core.chat_engine import CondensePlusContextChatEngine
 
 
-def get_chat_engine():
+def get_chat_engine(filters=None, params=None):
     system_prompt = os.getenv("SYSTEM_PROMPT")
-    top_k = os.getenv("TOP_K", 3)
+    citation_prompt = os.getenv("SYSTEM_CITATION_PROMPT", None)
+    top_k = int(os.getenv("TOP_K", 3))
 
-    index = get_index()
+    node_postprocessors = []
+    if citation_prompt:
+        node_postprocessors = [NodeCitationProcessor()]
+        system_prompt = f"{system_prompt}\n{citation_prompt}"
+
+    index = get_index(params)
     if index is None:
         raise HTTPException(
             status_code=500,
@@ -16,8 +25,13 @@ def get_chat_engine():
             ),
         )
 
-    return index.as_chat_engine(
-        similarity_top_k=int(top_k),
-        system_prompt=system_prompt,
-        chat_mode="condense_plus_context",
+    retriever = index.as_retriever(
+        similarity_top_k=top_k,
+        filters=filters,
+    )
+
+    return CondensePlusContextChatEngine.from_defaults(
+        system_prompt=system_prompt,
+        retriever=retriever,
+        node_postprocessors=node_postprocessors,
     )

templates/components/engines/python/chat/node_postprocessors.py

@@ -0,0 +1,21 @@
+from typing import List, Optional
+
+from llama_index.core import QueryBundle
+from llama_index.core.postprocessor.types import BaseNodePostprocessor
+from llama_index.core.schema import NodeWithScore
+
+
+class NodeCitationProcessor(BaseNodePostprocessor):
+    """
+    Append node_id into metadata for citation purpose.
+    Config SYSTEM_CITATION_PROMPT in your runtime environment variable to enable this feature.
+    """
+
+    def _postprocess_nodes(
+        self,
+        nodes: List[NodeWithScore],
+        query_bundle: Optional[QueryBundle] = None,
+    ) -> List[NodeWithScore]:
+        for node_score in nodes:
+            node_score.node.metadata["node_id"] = node_score.node.node_id
+        return nodes

templates/components/engines/typescript/agent/chat.ts

@@ -2,22 +2,24 @@ import { BaseToolWithCall, OpenAIAgent, QueryEngineTool } from "llamaindex";
 import fs from "node:fs/promises";
 import path from "node:path";
 import { getDataSource } from "./index";
-import { STORAGE_CACHE_DIR } from "./shared";
+import { generateFilters } from "./queryFilter";
 import { createTools } from "./tools";
 
-export async function createChatEngine() {
+export async function createChatEngine(documentIds?: string[], params?: any) {
   const tools: BaseToolWithCall[] = [];
 
   // Add a query engine tool if we have a data source
   // Delete this code if you don't have a data source
-  const index = await getDataSource();
+  const index = await getDataSource(params);
   if (index) {
     tools.push(
       new QueryEngineTool({
-        queryEngine: index.asQueryEngine(),
+        queryEngine: index.asQueryEngine({
+          preFilters: generateFilters(documentIds || []),
+        }),
         metadata: {
           name: "data_query_engine",
-          description: `A query engine for documents in storage folder: ${STORAGE_CACHE_DIR}`,
+          description: `A query engine for documents from your data source.`,
         },
       }),
     );

templates/components/engines/typescript/agent/tools/img-gen.ts

@@ -0,0 +1,112 @@
+import type { JSONSchemaType } from "ajv";
+import { FormData } from "formdata-node";
+import fs from "fs";
+import got from "got";
+import { BaseTool, ToolMetadata } from "llamaindex";
+import path from "node:path";
+import { Readable } from "stream";
+
+export type ImgGeneratorParameter = {
+  prompt: string;
+};
+
+export type ImgGeneratorToolParams = {
+  metadata?: ToolMetadata<JSONSchemaType<ImgGeneratorParameter>>;
+};
+
+export type ImgGeneratorToolOutput = {
+  isSuccess: boolean;
+  imageUrl?: string;
+  errorMessage?: string;
+};
+
+const DEFAULT_META_DATA: ToolMetadata<JSONSchemaType<ImgGeneratorParameter>> = {
+  name: "image_generator",
+  description: `Use this function to generate an image based on the prompt.`,
+  parameters: {
+    type: "object",
+    properties: {
+      prompt: {
+        type: "string",
+        description: "The prompt to generate the image",
+      },
+    },
+    required: ["prompt"],
+  },
+};
+
+export class ImgGeneratorTool implements BaseTool<ImgGeneratorParameter> {
+  readonly IMG_OUTPUT_FORMAT = "webp";
+  readonly IMG_OUTPUT_DIR = "output/tool";
+  readonly IMG_GEN_API =
+    "https://api.stability.ai/v2beta/stable-image/generate/core";
+
+  metadata: ToolMetadata<JSONSchemaType<ImgGeneratorParameter>>;
+
+  constructor(params?: ImgGeneratorToolParams) {
+    this.checkRequiredEnvVars();
+    this.metadata = params?.metadata || DEFAULT_META_DATA;
+  }
+
+  async call(input: ImgGeneratorParameter): Promise<ImgGeneratorToolOutput> {
+    return await this.generateImage(input.prompt);
+  }
+
+  private generateImage = async (
+    prompt: string,
+  ): Promise<ImgGeneratorToolOutput> => {
+    try {
+      const buffer = await this.promptToImgBuffer(prompt);
+      const imageUrl = this.saveImage(buffer);
+      return { isSuccess: true, imageUrl };
+    } catch (error) {
+      console.error(error);
+      return {
+        isSuccess: false,
+        errorMessage: "Failed to generate image. Please try again.",
+      };
+    }
+  };
+
+  private promptToImgBuffer = async (prompt: string) => {
+    const form = new FormData();
+    form.append("prompt", prompt);
+    form.append("output_format", this.IMG_OUTPUT_FORMAT);
+    const buffer = await got
+      .post(this.IMG_GEN_API, {
+        // Not sure why it shows an type error when passing form to body
+        // Although I follow document: https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#body
+        // Tt still works fine, so I make casting to unknown to avoid the typescript warning
+        // Found a similar issue: https://github.com/sindresorhus/got/discussions/1877
+        body: form as unknown as Buffer | Readable | string,
+        headers: {
+          Authorization: `Bearer ${process.env.STABILITY_API_KEY}`,
+          Accept: "image/*",
+        },
+      })
+      .buffer();
+    return buffer;
+  };
+
+  private saveImage = (buffer: Buffer) => {
+    const filename = `${crypto.randomUUID()}.${this.IMG_OUTPUT_FORMAT}`;
+    const outputPath = path.join(this.IMG_OUTPUT_DIR, filename);
+    fs.writeFileSync(outputPath, buffer);
+    const url = `${process.env.FILESERVER_URL_PREFIX}/${this.IMG_OUTPUT_DIR}/${filename}`;
+    console.log(`Saved image to ${outputPath}.\nURL: ${url}`);
+    return url;
+  };
+
+  private checkRequiredEnvVars = () => {
+    if (!process.env.STABILITY_API_KEY) {
+      throw new Error(
+        "STABILITY_API_KEY key is required to run image generator. Get it here: https://platform.stability.ai/account/keys",
+      );
+    }
+    if (!process.env.FILESERVER_URL_PREFIX) {
+      throw new Error(
+        "FILESERVER_URL_PREFIX is required to display file output after generation",
+      );
+    }
+  };
+}

templates/components/engines/typescript/agent/tools/index.ts

@@ -1,6 +1,7 @@
 import { BaseToolWithCall } from "llamaindex";
 import { ToolsFactory } from "llamaindex/tools/ToolsFactory";
 import { DuckDuckGoSearchTool, DuckDuckGoToolParams } from "./duckduckgo";
+import { ImgGeneratorTool, ImgGeneratorToolParams } from "./img-gen";
 import { InterpreterTool, InterpreterToolParams } from "./interpreter";
 import { OpenAPIActionTool } from "./openapi-action";
 import { WeatherTool, WeatherToolParams } from "./weather";
@@ -39,6 +40,9 @@ const toolFactory: Record<string, ToolCreator> = {
   duckduckgo: async (config: unknown) => {
     return [new DuckDuckGoSearchTool(config as DuckDuckGoToolParams)];
   },
+  img_gen: async (config: unknown) => {
+    return [new ImgGeneratorTool(config as ImgGeneratorToolParams)];
+  },
 };
 
 async function createLocalTools(

templates/components/engines/typescript/agent/tools/interpreter.ts

@@ -56,7 +56,7 @@ const DEFAULT_META_DATA: ToolMetadata<JSONSchemaType<InterpreterParameter>> = {
 };
 
 export class InterpreterTool implements BaseTool<InterpreterParameter> {
-  private readonly outputDir = "tool-output";
+  private readonly outputDir = "output/tool";
   private apiKey?: string;
   private fileServerURLPrefix?: string;
   metadata: ToolMetadata<JSONSchemaType<InterpreterParameter>>;

templates/components/engines/typescript/chat/chat.ts

@@ -1,17 +1,18 @@
 import { ContextChatEngine, Settings } from "llamaindex";
 import { getDataSource } from "./index";
+import { generateFilters } from "./queryFilter";
 
-export async function createChatEngine() {
-  const index = await getDataSource();
+export async function createChatEngine(documentIds?: string[], params?: any) {
+  const index = await getDataSource(params);
   if (!index) {
     throw new Error(
       `StorageContext is empty - call 'npm run generate' to generate the storage first`,
     );
   }
-  const retriever = index.asRetriever();
-  retriever.similarityTopK = process.env.TOP_K
-    ? parseInt(process.env.TOP_K)
-    : 3;
+  const retriever = index.asRetriever({
+    similarityTopK: process.env.TOP_K ? parseInt(process.env.TOP_K) : 3,
+    filters: generateFilters(documentIds || []),
+  });
 
   return new ContextChatEngine({
     chatModel: Settings.llm,

templates/components/llamaindex/typescript/documents/helper.ts

@@ -0,0 +1,44 @@
+import fs from "fs";
+import crypto from "node:crypto";
+import { getExtractors } from "../../engine/loader";
+
+const MIME_TYPE_TO_EXT: Record<string, string> = {
+  "application/pdf": "pdf",
+  "text/plain": "txt",
+  "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
+    "docx",
+};
+
+const UPLOADED_FOLDER = "output/uploaded";
+
+export async function loadDocuments(fileBuffer: Buffer, mimeType: string) {
+  const extractors = getExtractors();
+  const reader = extractors[MIME_TYPE_TO_EXT[mimeType]];
+
+  if (!reader) {
+    throw new Error(`Unsupported document type: ${mimeType}`);
+  }
+  console.log(`Processing uploaded document of type: ${mimeType}`);
+  return await reader.loadDataAsContent(fileBuffer);
+}
+
+export async function saveDocument(fileBuffer: Buffer, mimeType: string) {
+  const fileExt = MIME_TYPE_TO_EXT[mimeType];
+  if (!fileExt) throw new Error(`Unsupported document type: ${mimeType}`);
+
+  const filename = `${crypto.randomUUID()}.${fileExt}`;
+  const filepath = `${UPLOADED_FOLDER}/${filename}`;
+  const fileurl = `${process.env.FILESERVER_URL_PREFIX}/${filepath}`;
+
+  if (!fs.existsSync(UPLOADED_FOLDER)) {
+    fs.mkdirSync(UPLOADED_FOLDER, { recursive: true });
+  }
+  await fs.promises.writeFile(filepath, fileBuffer);
+
+  console.log(`Saved document file to ${filepath}.\nURL: ${fileurl}`);
+  return {
+    filename,
+    filepath,
+    fileurl,
+  };
+}

templates/components/llamaindex/typescript/documents/pipeline.ts

@@ -0,0 +1,38 @@
+import {
+  Document,
+  IngestionPipeline,
+  Settings,
+  SimpleNodeParser,
+  VectorStoreIndex,
+} from "llamaindex";
+import { LlamaCloudIndex } from "llamaindex/cloud/LlamaCloudIndex";
+
+export async function runPipeline(
+  currentIndex: VectorStoreIndex | LlamaCloudIndex,
+  documents: Document[],
+) {
+  if (currentIndex instanceof LlamaCloudIndex) {
+    // LlamaCloudIndex processes the documents automatically
+    // so we don't need ingestion pipeline, just insert the documents directly
+    for (const document of documents) {
+      await currentIndex.insert(document);
+    }
+  } else {
+    // Use ingestion pipeline to process the documents into nodes and add them to the vector store
+    const pipeline = new IngestionPipeline({
+      transformations: [
+        new SimpleNodeParser({
+          chunkSize: Settings.chunkSize,
+          chunkOverlap: Settings.chunkOverlap,
+        }),
+        Settings.embedModel,
+      ],
+    });
+    const nodes = await pipeline.run({ documents });
+    await currentIndex.insertNodes(nodes);
+    currentIndex.storageContext.docStore.persist();
+    console.log("Added nodes to the vector store.");
+  }
+
+  return documents.map((document) => document.id_);
+}

templates/components/llamaindex/typescript/documents/upload.ts

@@ -0,0 +1,26 @@
+import { VectorStoreIndex } from "llamaindex";
+import { LlamaCloudIndex } from "llamaindex/cloud/LlamaCloudIndex";
+import { loadDocuments, saveDocument } from "./helper";
+import { runPipeline } from "./pipeline";
+
+export async function uploadDocument(
+  index: VectorStoreIndex | LlamaCloudIndex,
+  raw: string,
+): Promise<string[]> {
+  const [header, content] = raw.split(",");
+  const mimeType = header.replace("data:", "").replace(";base64", "");
+  const fileBuffer = Buffer.from(content, "base64");
+  const documents = await loadDocuments(fileBuffer, mimeType);
+  const { filename } = await saveDocument(fileBuffer, mimeType);
+
+  // Update documents with metadata
+  for (const document of documents) {
+    document.metadata = {
+      ...document.metadata,
+      file_name: filename,
+      private: "true", // to separate private uploads from public documents
+    };
+  }
+
+  return await runPipeline(index, documents);
+}

templates/components/llamaindex/typescript/streaming/annotations.ts

@@ -0,0 +1,124 @@
+import { JSONValue } from "ai";
+import { MessageContent, MessageContentDetail } from "llamaindex";
+
+export type DocumentFileType = "csv" | "pdf" | "txt" | "docx";
+
+export type DocumentFileContent = {
+  type: "ref" | "text";
+  value: string[] | string;
+};
+
+export type DocumentFile = {
+  id: string;
+  filename: string;
+  filesize: number;
+  filetype: DocumentFileType;
+  content: DocumentFileContent;
+};
+
+type Annotation = {
+  type: string;
+  data: object;
+};
+
+export function retrieveDocumentIds(annotations?: JSONValue[]): string[] {
+  if (!annotations) return [];
+
+  const ids: string[] = [];
+
+  for (const annotation of annotations) {
+    const { type, data } = getValidAnnotation(annotation);
+    if (
+      type === "document_file" &&
+      "files" in data &&
+      Array.isArray(data.files)
+    ) {
+      const files = data.files as DocumentFile[];
+      for (const file of files) {
+        if (Array.isArray(file.content.value)) {
+          // it's an array, so it's an array of doc IDs
+          for (const id of file.content.value) {
+            ids.push(id);
+          }
+        }
+      }
+    }
+  }
+
+  return ids;
+}
+
+export function convertMessageContent(
+  content: string,
+  annotations?: JSONValue[],
+): MessageContent {
+  if (!annotations) return content;
+  return [
+    {
+      type: "text",
+      text: content,
+    },
+    ...convertAnnotations(annotations),
+  ];
+}
+
+function convertAnnotations(annotations: JSONValue[]): MessageContentDetail[] {
+  const content: MessageContentDetail[] = [];
+  annotations.forEach((annotation: JSONValue) => {
+    const { type, data } = getValidAnnotation(annotation);
+    // convert image
+    if (type === "image" && "url" in data && typeof data.url === "string") {
+      content.push({
+        type: "image_url",
+        image_url: {
+          url: data.url,
+        },
+      });
+    }
+    // convert the content of files to a text message
+    if (
+      type === "document_file" &&
+      "files" in data &&
+      Array.isArray(data.files)
+    ) {
+      // get all CSV files and convert their whole content to one text message
+      // currently CSV files are the only files where we send the whole content - we don't use an index
+      const csvFiles: DocumentFile[] = data.files.filter(
+        (file: DocumentFile) => file.filetype === "csv",
+      );
+      if (csvFiles && csvFiles.length > 0) {
+        const csvContents = csvFiles.map((file: DocumentFile) => {
+          const fileContent = Array.isArray(file.content.value)
+            ? file.content.value.join("\n")
+            : file.content.value;
+          return "```csv\n" + fileContent + "\n```";
+        });
+        const text =
+          "Use the following CSV content:\n" + csvContents.join("\n\n");
+        content.push({
+          type: "text",
+          text,
+        });
+      }
+    }
+  });
+
+  return content;
+}
+
+function getValidAnnotation(annotation: JSONValue): Annotation {
+  if (
+    !(
+      annotation &&
+      typeof annotation === "object" &&
+      "type" in annotation &&
+      typeof annotation.type === "string" &&
+      "data" in annotation &&
+      annotation.data &&
+      typeof annotation.data === "object"
+    )
+  ) {
+    throw new Error("Client sent invalid annotation. Missing data and type");
+  }
+  return { type: annotation.type, data: annotation.data };
+}

templates/components/llamaindex/typescript/streaming/events.ts

@@ -2,43 +2,35 @@ import { StreamData } from "ai";
 import {
   CallbackManager,
   Metadata,
+  MetadataMode,
   NodeWithScore,
   ToolCall,
   ToolOutput,
 } from "llamaindex";
-
-function getNodeUrl(metadata: Metadata) {
-  const url = metadata["URL"];
-  if (url) return url;
-  const fileName = metadata["file_name"];
-  if (!process.env.FILESERVER_URL_PREFIX) {
-    console.warn(
-      "FILESERVER_URL_PREFIX is not set. File URLs will not be generated.",
-    );
-    return undefined;
-  }
-  if (fileName) {
-    return `${process.env.FILESERVER_URL_PREFIX}/data/${fileName}`;
-  }
-  return undefined;
-}
+import { LLamaCloudFileService } from "./service";
 
 export function appendSourceData(
   data: StreamData,
   sourceNodes?: NodeWithScore<Metadata>[],
 ) {
   if (!sourceNodes?.length) return;
-  data.appendMessageAnnotation({
-    type: "sources",
-    data: {
-      nodes: sourceNodes.map((node) => ({
-        ...node.node.toMutableJSON(),
-        id: node.node.id_,
-        score: node.score ?? null,
-        url: getNodeUrl(node.node.metadata),
-      })),
-    },
-  });
+  try {
+    const nodes = sourceNodes.map((node) => ({
+      metadata: node.node.metadata,
+      id: node.node.id_,
+      score: node.score ?? null,
+      url: getNodeUrl(node.node.metadata),
+      text: node.node.getContent(MetadataMode.NONE),
+    }));
+    data.appendMessageAnnotation({
+      type: "sources",
+      data: {
+        nodes,
+      },
+    });
+  } catch (error) {
+    console.error("Error appending source data:", error);
+  }
 }
 
 export function appendEventData(data: StreamData, title?: string) {
@@ -84,17 +76,19 @@ export function createStreamTimeout(stream: StreamData) {
 export function createCallbackManager(stream: StreamData) {
   const callbackManager = new CallbackManager();
 
-  callbackManager.on("retrieve", (data) => {
+  callbackManager.on("retrieve-end", (data) => {
     const { nodes, query } = data.detail;
+    appendSourceData(stream, nodes);
     appendEventData(stream, `Retrieving context for query: '${query}'`);
     appendEventData(
       stream,
       `Retrieved ${nodes.length} sources to use as context for the query`,
     );
+    LLamaCloudFileService.downloadFiles(nodes); // don't await to avoid blocking chat streaming
   });
 
   callbackManager.on("llm-tool-call", (event) => {
-    const { name, input } = event.detail.payload.toolCall;
+    const { name, input } = event.detail.toolCall;
     const inputString = Object.entries(input)
       .map(([key, value]) => `${key}: ${value}`)
       .join(", ");
@@ -105,16 +99,32 @@ export function createCallbackManager(stream: StreamData) {
   });
 
   callbackManager.on("llm-tool-result", (event) => {
-    const { toolCall, toolResult } = event.detail.payload;
+    const { toolCall, toolResult } = event.detail;
     appendToolData(stream, toolCall, toolResult);
   });
 
   return callbackManager;
 }
 
-export type CsvFile = {
-  content: string;
-  filename: string;
-  filesize: number;
-  id: string;
-};
+function getNodeUrl(metadata: Metadata) {
+  if (!process.env.FILESERVER_URL_PREFIX) {
+    console.warn(
+      "FILESERVER_URL_PREFIX is not set. File URLs will not be generated.",
+    );
+  }
+  const fileName = metadata["file_name"];
+  if (fileName && process.env.FILESERVER_URL_PREFIX) {
+    // file_name exists and file server is configured
+    const pipelineId = metadata["pipeline_id"];
+    if (pipelineId && metadata["private"] == null) {
+      // file is from LlamaCloud and was not ingested locally
+      const name = LLamaCloudFileService.toDownloadedName(pipelineId, fileName);
+      return `${process.env.FILESERVER_URL_PREFIX}/output/llamacloud/${name}`;
+    }
+    const isPrivate = metadata["private"] === "true";
+    const folder = isPrivate ? "output/uploaded" : "data";
+    return `${process.env.FILESERVER_URL_PREFIX}/${folder}/${fileName}`;
+  }
+  // fallback to URL in metadata (e.g. for websites)
+  return metadata["URL"];
+}

templates/components/llamaindex/typescript/streaming/service.ts

@@ -0,0 +1,187 @@
+import { Metadata, NodeWithScore } from "llamaindex";
+import fs from "node:fs";
+import https from "node:https";
+import path from "node:path";
+
+const LLAMA_CLOUD_OUTPUT_DIR = "output/llamacloud";
+const LLAMA_CLOUD_BASE_URL = "https://cloud.llamaindex.ai/api/v1";
+const FILE_DELIMITER = "$"; // delimiter between pipelineId and filename
+
+type LlamaCloudFile = {
+  name: string;
+  file_id: string;
+  project_id: string;
+};
+
+type LLamaCloudProject = {
+  id: string;
+  organization_id: string;
+  name: string;
+  is_default: boolean;
+};
+
+type LLamaCloudPipeline = {
+  id: string;
+  name: string;
+  project_id: string;
+};
+
+export class LLamaCloudFileService {
+  private static readonly headers = {
+    Accept: "application/json",
+    Authorization: `Bearer ${process.env.LLAMA_CLOUD_API_KEY}`,
+  };
+
+  public static async getAllProjectsWithPipelines() {
+    try {
+      const projects = await LLamaCloudFileService.getAllProjects();
+      const pipelines = await LLamaCloudFileService.getAllPipelines();
+      return projects.map((project) => ({
+        ...project,
+        pipelines: pipelines.filter((p) => p.project_id === project.id),
+      }));
+    } catch (error) {
+      console.error("Error listing projects and pipelines:", error);
+      return [];
+    }
+  }
+
+  public static async downloadFiles(nodes: NodeWithScore<Metadata>[]) {
+    const files = LLamaCloudFileService.nodesToDownloadFiles(nodes);
+    if (!files.length) return;
+    console.log("Downloading files from LlamaCloud...");
+    for (const file of files) {
+      await LLamaCloudFileService.downloadFile(file.pipelineId, file.fileName);
+    }
+  }
+
+  public static toDownloadedName(pipelineId: string, fileName: string) {
+    return `${pipelineId}${FILE_DELIMITER}${fileName}`;
+  }
+
+  /**
+   * This function will return an array of unique files to download from LlamaCloud
+   * We only download files that are uploaded directly in LlamaCloud datasources (don't have `private` in metadata)
+   * Files are uploaded directly in LlamaCloud datasources don't have `private` in metadata (public docs)
+   * Files are uploaded from local via `generate` command will have `private=false` (public docs)
+   * Files are uploaded from local via `/chat/upload` endpoint will have `private=true` (private docs)
+   *
+   * @param nodes
+   * @returns list of unique files to download
+   */
+  private static nodesToDownloadFiles(nodes: NodeWithScore<Metadata>[]) {
+    const downloadFiles: Array<{
+      pipelineId: string;
+      fileName: string;
+    }> = [];
+    for (const node of nodes) {
+      const isLocalFile = node.node.metadata["private"] != null;
+      const pipelineId = node.node.metadata["pipeline_id"];
+      const fileName = node.node.metadata["file_name"];
+      if (isLocalFile || !pipelineId || !fileName) continue;
+      const isDuplicate = downloadFiles.some(
+        (f) => f.pipelineId === pipelineId && f.fileName === fileName,
+      );
+      if (!isDuplicate) {
+        downloadFiles.push({ pipelineId, fileName });
+      }
+    }
+    return downloadFiles;
+  }
+
+  private static async downloadFile(pipelineId: string, fileName: string) {
+    try {
+      const downloadedName = LLamaCloudFileService.toDownloadedName(
+        pipelineId,
+        fileName,
+      );
+      const downloadedPath = path.join(LLAMA_CLOUD_OUTPUT_DIR, downloadedName);
+
+      // Check if file already exists
+      if (fs.existsSync(downloadedPath)) return;
+
+      const urlToDownload = await LLamaCloudFileService.getFileUrlByName(
+        pipelineId,
+        fileName,
+      );
+      if (!urlToDownload) throw new Error("File not found in LlamaCloud");
+
+      const file = fs.createWriteStream(downloadedPath);
+      https
+        .get(urlToDownload, (response) => {
+          response.pipe(file);
+          file.on("finish", () => {
+            file.close(() => {
+              console.log("File downloaded successfully");
+            });
+          });
+        })
+        .on("error", (err) => {
+          fs.unlink(downloadedPath, () => {
+            console.error("Error downloading file:", err);
+            throw err;
+          });
+        });
+    } catch (error) {
+      throw new Error(`Error downloading file from LlamaCloud: ${error}`);
+    }
+  }
+
+  private static async getFileUrlByName(
+    pipelineId: string,
+    name: string,
+  ): Promise<string | null> {
+    const files = await LLamaCloudFileService.getAllFiles(pipelineId);
+    const file = files.find((file) => file.name === name);
+    if (!file) return null;
+    return await LLamaCloudFileService.getFileUrlById(
+      file.project_id,
+      file.file_id,
+    );
+  }
+
+  private static async getFileUrlById(
+    projectId: string,
+    fileId: string,
+  ): Promise<string> {
+    const url = `${LLAMA_CLOUD_BASE_URL}/files/${fileId}/content?project_id=${projectId}`;
+    const response = await fetch(url, {
+      method: "GET",
+      headers: LLamaCloudFileService.headers,
+    });
+    const data = (await response.json()) as { url: string };
+    return data.url;
+  }
+
+  private static async getAllFiles(
+    pipelineId: string,
+  ): Promise<LlamaCloudFile[]> {
+    const url = `${LLAMA_CLOUD_BASE_URL}/pipelines/${pipelineId}/files`;
+    const response = await fetch(url, {
+      method: "GET",
+      headers: LLamaCloudFileService.headers,
+    });
+    const data = await response.json();
+    return data;
+  }
+
+  private static async getAllProjects(): Promise<LLamaCloudProject[]> {
+    const url = `${LLAMA_CLOUD_BASE_URL}/projects`;
+    const response = await fetch(url, {
+      method: "GET",
+      headers: LLamaCloudFileService.headers,
+    });
+    const data = (await response.json()) as LLamaCloudProject[];
+    return data;
+  }
+
+  private static async getAllPipelines(): Promise<LLamaCloudPipeline[]> {
+    const url = `${LLAMA_CLOUD_BASE_URL}/pipelines`;
+    const response = await fetch(url, {
+      method: "GET",
+      headers: LLamaCloudFileService.headers,
+    });
+    const data = (await response.json()) as LLamaCloudPipeline[];
+    return data;
+  }
+}

templates/components/llamaindex/typescript/streaming/stream.ts

@@ -0,0 +1,57 @@
+import {
+  StreamData,
+  createCallbacksTransformer,
+  createStreamDataTransformer,
+  trimStartOfStreamHelper,
+  type AIStreamCallbacksAndOptions,
+} from "ai";
+import { ChatMessage, EngineResponse } from "llamaindex";
+import { generateNextQuestions } from "./suggestion";
+
+export function LlamaIndexStream(
+  response: AsyncIterable<EngineResponse>,
+  data: StreamData,
+  chatHistory: ChatMessage[],
+  opts?: {
+    callbacks?: AIStreamCallbacksAndOptions;
+  },
+): ReadableStream<Uint8Array> {
+  return createParser(response, data, chatHistory)
+    .pipeThrough(createCallbacksTransformer(opts?.callbacks))
+    .pipeThrough(createStreamDataTransformer());
+}
+
+function createParser(
+  res: AsyncIterable<EngineResponse>,
+  data: StreamData,
+  chatHistory: ChatMessage[],
+) {
+  const it = res[Symbol.asyncIterator]();
+  const trimStartOfStream = trimStartOfStreamHelper();
+  let llmTextResponse = "";
+
+  return new ReadableStream<string>({
+    async pull(controller): Promise<void> {
+      const { value, done } = await it.next();
+      if (done) {
+        controller.close();
+        // LLM stream is done, generate the next questions with a new LLM call
+        chatHistory.push({ role: "assistant", content: llmTextResponse });
+        const questions: string[] = await generateNextQuestions(chatHistory);
+        if (questions.length > 0) {
+          data.appendMessageAnnotation({
+            type: "suggested_questions",
+            data: questions,
+          });
+        }
+        data.close();
+        return;
+      }
+      const text = trimStartOfStream(value.delta ?? "");
+      if (text) {
+        llmTextResponse += text;
+        controller.enqueue(text);
+      }
+    },
+  });
+}

templates/components/llamaindex/typescript/streaming/suggestion.ts

@@ -0,0 +1,55 @@
+import { ChatMessage, Settings } from "llamaindex";
+
+const NEXT_QUESTION_PROMPT_TEMPLATE = `You're a helpful assistant! Your task is to suggest the next question that user might ask. 
+Here is the conversation history
+---------------------
+$conversation
+---------------------
+Given the conversation history, please give me $number_of_questions questions that you might ask next!
+Your answer should be wrapped in three sticks which follows the following format:
+\`\`\`
+<question 1>
+<question 2>\`\`\`
+`;
+const N_QUESTIONS_TO_GENERATE = 3;
+
+export async function generateNextQuestions(
+  conversation: ChatMessage[],
+  numberOfQuestions: number = N_QUESTIONS_TO_GENERATE,
+) {
+  const llm = Settings.llm;
+
+  // Format conversation
+  const conversationText = conversation
+    .map((message) => `${message.role}: ${message.content}`)
+    .join("\n");
+  const message = NEXT_QUESTION_PROMPT_TEMPLATE.replace(
+    "$conversation",
+    conversationText,
+  ).replace("$number_of_questions", numberOfQuestions.toString());
+
+  try {
+    const response = await llm.complete({ prompt: message });
+    const questions = extractQuestions(response.text);
+    return questions;
+  } catch (error) {
+    console.error("Error when generating the next questions: ", error);
+    return [];
+  }
+}
+
+// TODO: instead of parsing the LLM's result we can use structured predict, once LITS supports it
+function extractQuestions(text: string): string[] {
+  // Extract the text inside the triple backticks
+  // @ts-ignore
+  const contentMatch = text.match(/```(.*?)```/s);
+  const content = contentMatch ? contentMatch[1] : "";
+
+  // Split the content by newlines to get each question
+  const questions = content
+    .split("\n")
+    .map((question) => question.trim())
+    .filter((question) => question !== "");
+
+  return questions;
+}

templates/components/loaders/python/__init__.py

@@ -1,11 +1,9 @@
-import os
-import yaml
-import importlib
 import logging
-from typing import Dict
+
+import yaml
+from app.engine.loaders.db import DBLoaderConfig, get_db_documents
 from app.engine.loaders.file import FileLoaderConfig, get_file_documents
 from app.engine.loaders.web import WebLoaderConfig, get_web_documents
-from app.engine.loaders.db import DBLoaderConfig, get_db_documents
 
 logger = logging.getLogger(__name__)
 

templates/components/loaders/python/db.py

@@ -1,8 +1,6 @@
-import os
 import logging
 from typing import List
-from pydantic import BaseModel, validator
-from llama_index.core.indices.vector_store import VectorStoreIndex
+from pydantic import BaseModel
 
 logger = logging.getLogger(__name__)
 

templates/components/loaders/python/file.py

@@ -1,5 +1,6 @@
 import os
 import logging
+from typing import Dict
 from llama_parse import LlamaParse
 from pydantic import BaseModel, validator
 
@@ -32,13 +33,18 @@ def llama_parse_parser():
     return parser
 
 
+def llama_parse_extractor() -> Dict[str, LlamaParse]:
+    from llama_parse.utils import SUPPORTED_FILE_TYPES
+
+    parser = llama_parse_parser()
+    return {file_type: parser for file_type in SUPPORTED_FILE_TYPES}
+
+
 def get_file_documents(config: FileLoaderConfig):
     from llama_index.core.readers import SimpleDirectoryReader
 
     try:
-        reader = SimpleDirectoryReader(
-            config.data_dir, recursive=True, filename_as_id=True, raise_on_error=True
-        )
+        file_extractor = None
         if config.use_llama_parse:
             # LlamaParse is async first,
             # so we need to use nest_asyncio to run it in sync mode
@@ -46,11 +52,18 @@ def get_file_documents(config: FileLoaderConfig):
 
             nest_asyncio.apply()
 
-            parser = llama_parse_parser()
-            reader.file_extractor = {".pdf": parser}
+            file_extractor = llama_parse_extractor()
+        reader = SimpleDirectoryReader(
+            config.data_dir,
+            recursive=True,
+            filename_as_id=True,
+            raise_on_error=True,
+            file_extractor=file_extractor,
+        )
         return reader.load_data()
     except Exception as e:
-        import sys, traceback
+        import sys
+        import traceback
 
         # Catch the error if the data dir is empty
         # and return as empty document list

templates/components/loaders/python/web.py

@@ -1,5 +1,3 @@
-import os
-import json
 from pydantic import BaseModel, Field
 
 

templates/components/loaders/typescript/file/loader.ts

@@ -1,9 +1,24 @@
-import { SimpleDirectoryReader } from "llamaindex";
+import {
+  FILE_EXT_TO_READER,
+  SimpleDirectoryReader,
+} from "llamaindex/readers/SimpleDirectoryReader";
 
 export const DATA_DIR = "./data";
 
+export function getExtractors() {
+  return FILE_EXT_TO_READER;
+}
+
 export async function getDocuments() {
-  return await new SimpleDirectoryReader().loadData({
+  const documents = await new SimpleDirectoryReader().loadData({
     directoryPath: DATA_DIR,
   });
+  // Set private=false to mark the document as public (required for filtering)
+  for (const document of documents) {
+    document.metadata = {
+      ...document.metadata,
+      private: "false",
+    };
+  }
+  return documents;
 }

templates/components/loaders/typescript/llama_parse/loader.ts

@@ -1,19 +1,38 @@
+import { LlamaParseReader } from "llamaindex/readers/LlamaParseReader";
 import {
   FILE_EXT_TO_READER,
-  LlamaParseReader,
   SimpleDirectoryReader,
-} from "llamaindex";
+} from "llamaindex/readers/SimpleDirectoryReader";
 
 export const DATA_DIR = "./data";
 
+export function getExtractors() {
+  const llamaParseParser = new LlamaParseReader({ resultType: "markdown" });
+  const extractors = FILE_EXT_TO_READER;
+  // Change all the supported extractors to LlamaParse
+  // except for .txt, it doesn't need to be parsed
+  for (const key in extractors) {
+    if (key === "txt") {
+      continue;
+    }
+    extractors[key] = llamaParseParser;
+  }
+  return extractors;
+}
+
 export async function getDocuments() {
   const reader = new SimpleDirectoryReader();
-  // Load PDFs using LlamaParseReader
-  return await reader.loadData({
+  const extractors = getExtractors();
+  const documents = await reader.loadData({
     directoryPath: DATA_DIR,
-    fileExtToReader: {
-      ...FILE_EXT_TO_READER,
-      pdf: new LlamaParseReader({ resultType: "markdown" }),
-    },
+    fileExtToReader: extractors,
   });
+  // Set private=false to mark the document as public (required for filtering)
+  for (const document of documents) {
+    document.metadata = {
+      ...document.metadata,
+      private: "false",
+    };
+  }
+  return documents;
 }

templates/components/observability/python/llamatrace/observability.py

@@ -0,0 +1,12 @@
+import llama_index.core
+import os
+
+
+def init_observability():
+    PHOENIX_API_KEY = os.getenv("PHOENIX_API_KEY")
+    if not PHOENIX_API_KEY:
+        raise ValueError("PHOENIX_API_KEY environment variable is not set")
+    os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = f"api_key={PHOENIX_API_KEY}"
+    llama_index.core.set_global_handler(
+        "arize_phoenix", endpoint="https://llamatrace.com/v1/traces"
+    )

templates/components/sample-projects/llamapack/pyproject.toml

@@ -6,7 +6,7 @@ authors = ["Marcus Schiesser <mail@marcusschiesser.de>"]
 readme = "README.md"
 
 [tool.poetry.dependencies]
-python = "^3.11,<3.12"
+python = "^3.11,<4.0"
 llama-index = "^0.10.6"
 llama-index-readers-file = "^0.1.3"
 python-dotenv = "^1.0.0"

templates/components/settings/python/llmhub.py

@@ -0,0 +1,64 @@
+from llama_index.embeddings.openai import OpenAIEmbedding
+from llama_index.core.settings import Settings
+from typing import Dict
+import os
+
+DEFAULT_MODEL = "gpt-3.5-turbo"
+DEFAULT_EMBEDDING_MODEL = "text-embedding-3-large"
+
+
+class TSIEmbedding(OpenAIEmbedding):
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self._query_engine = self._text_engine = self.model_name
+
+
+def llm_config_from_env() -> Dict:
+    from llama_index.core.constants import DEFAULT_TEMPERATURE
+
+    model = os.getenv("MODEL", DEFAULT_MODEL)
+    temperature = os.getenv("LLM_TEMPERATURE", DEFAULT_TEMPERATURE)
+    max_tokens = os.getenv("LLM_MAX_TOKENS")
+    api_key = os.getenv("T_SYSTEMS_LLMHUB_API_KEY")
+    api_base = os.getenv("T_SYSTEMS_LLMHUB_BASE_URL")
+
+    config = {
+        "model": model,
+        "api_key": api_key,
+        "api_base": api_base,
+        "temperature": float(temperature),
+        "max_tokens": int(max_tokens) if max_tokens is not None else None,
+    }
+    return config
+
+
+def embedding_config_from_env() -> Dict:
+    from llama_index.core.constants import DEFAULT_EMBEDDING_DIM
+
+    model = os.getenv("EMBEDDING_MODEL", DEFAULT_EMBEDDING_MODEL)
+    dimension = os.getenv("EMBEDDING_DIM", DEFAULT_EMBEDDING_DIM)
+    api_key = os.getenv("T_SYSTEMS_LLMHUB_API_KEY")
+    api_base = os.getenv("T_SYSTEMS_LLMHUB_BASE_URL")
+
+    config = {
+        "model_name": model,
+        "dimension": int(dimension) if dimension is not None else None,
+        "api_key": api_key,
+        "api_base": api_base,
+    }
+    return config
+
+
+def init_llmhub():
+    from llama_index.llms.openai_like import OpenAILike
+
+    llm_configs = llm_config_from_env()
+    embedding_configs = embedding_config_from_env()
+
+    Settings.embed_model = TSIEmbedding(**embedding_configs)
+    Settings.llm = OpenAILike(
+        **llm_configs,
+        is_chat_model=True,
+        is_function_calling_model=False,
+        context_window=4096,
+    )

templates/components/settings/python/settings.py

@@ -17,10 +17,17 @@ def init_settings():
             init_anthropic()
         case "gemini":
             init_gemini()
+        case "mistral":
+            init_mistral()
         case "azure-openai":
             init_azure_openai()
+        case "t-systems":
+            from .llmhub import init_llmhub
+
+            init_llmhub()
         case _:
             raise ValueError(f"Invalid model provider: {model_provider}")
+
     Settings.chunk_size = int(os.getenv("CHUNK_SIZE", "1024"))
     Settings.chunk_overlap = int(os.getenv("CHUNK_OVERLAP", "20"))
 
@@ -68,31 +75,55 @@ def init_azure_openai():
     from llama_index.embeddings.azure_openai import AzureOpenAIEmbedding
     from llama_index.llms.azure_openai import AzureOpenAI
 
-    llm_deployment = os.getenv("AZURE_OPENAI_LLM_DEPLOYMENT")
-    embedding_deployment = os.getenv("AZURE_OPENAI_EMBEDDING_DEPLOYMENT")
+    llm_deployment = os.environ["AZURE_OPENAI_LLM_DEPLOYMENT"]
+    embedding_deployment = os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"]
     max_tokens = os.getenv("LLM_MAX_TOKENS")
-    api_key = os.getenv("AZURE_OPENAI_API_KEY")
-    llm_config = {
-        "api_key": api_key,
-        "deployment_name": llm_deployment,
-        "model": os.getenv("MODEL"),
-        "temperature": float(os.getenv("LLM_TEMPERATURE", DEFAULT_TEMPERATURE)),
-        "max_tokens": int(max_tokens) if max_tokens is not None else None,
-    }
-    Settings.llm = AzureOpenAI(**llm_config)
-
+    temperature = os.getenv("LLM_TEMPERATURE", DEFAULT_TEMPERATURE)
     dimensions = os.getenv("EMBEDDING_DIM")
-    embedding_config = {
-        "api_key": api_key,
-        "deployment_name": embedding_deployment,
-        "model": os.getenv("EMBEDDING_MODEL"),
-        "dimensions": int(dimensions) if dimensions is not None else None,
+
+    azure_config = {
+        "api_key": os.environ["AZURE_OPENAI_API_KEY"],
+        "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
+        "api_version": os.getenv("AZURE_OPENAI_API_VERSION")
+        or os.getenv("OPENAI_API_VERSION"),
     }
-    Settings.embed_model = AzureOpenAIEmbedding(**embedding_config)
+
+    Settings.llm = AzureOpenAI(
+        model=os.getenv("MODEL"),
+        max_tokens=int(max_tokens) if max_tokens is not None else None,
+        temperature=float(temperature),
+        deployment_name=llm_deployment,
+        **azure_config,
+    )
+
+    Settings.embed_model = AzureOpenAIEmbedding(
+        model=os.getenv("EMBEDDING_MODEL"),
+        dimensions=int(dimensions) if dimensions is not None else None,
+        deployment_name=embedding_deployment,
+        **azure_config,
+    )
+
+
+def init_fastembed():
+    """
+    Use Qdrant Fastembed as the local embedding provider.
+    """
+    from llama_index.embeddings.fastembed import FastEmbedEmbedding
+
+    embed_model_map: Dict[str, str] = {
+        # Small and multilingual
+        "all-MiniLM-L6-v2": "sentence-transformers/all-MiniLM-L6-v2",
+        # Large and multilingual
+        "paraphrase-multilingual-mpnet-base-v2": "sentence-transformers/paraphrase-multilingual-mpnet-base-v2",  # noqa: E501
+    }
+
+    # This will download the model automatically if it is not already downloaded
+    Settings.embed_model = FastEmbedEmbedding(
+        model_name=embed_model_map[os.getenv("EMBEDDING_MODEL")]
+    )
 
 
 def init_groq():
-    from llama_index.embeddings.huggingface import HuggingFaceEmbedding
     from llama_index.llms.groq import Groq
 
     model_map: Dict[str, str] = {
@@ -101,19 +132,12 @@ def init_groq():
         "mixtral-8x7b": "mixtral-8x7b-32768",
     }
 
-    embed_model_map: Dict[str, str] = {
-        "all-MiniLM-L6-v2": "sentence-transformers/all-MiniLM-L6-v2",
-        "all-mpnet-base-v2": "sentence-transformers/all-mpnet-base-v2",
-    }
-
     Settings.llm = Groq(model=model_map[os.getenv("MODEL")])
-    Settings.embed_model = HuggingFaceEmbedding(
-        model_name=embed_model_map[os.getenv("EMBEDDING_MODEL")]
-    )
+    # Groq does not provide embeddings, so we use FastEmbed instead
+    init_fastembed()
 
 
 def init_anthropic():
-    from llama_index.embeddings.huggingface import HuggingFaceEmbedding
     from llama_index.llms.anthropic import Anthropic
 
     model_map: Dict[str, str] = {
@@ -124,15 +148,9 @@ def init_anthropic():
         "claude-instant-1.2": "claude-instant-1.2",
     }
 
-    embed_model_map: Dict[str, str] = {
-        "all-MiniLM-L6-v2": "sentence-transformers/all-MiniLM-L6-v2",
-        "all-mpnet-base-v2": "sentence-transformers/all-mpnet-base-v2",
-    }
-
     Settings.llm = Anthropic(model=model_map[os.getenv("MODEL")])
-    Settings.embed_model = HuggingFaceEmbedding(
-        model_name=embed_model_map[os.getenv("EMBEDDING_MODEL")]
-    )
+    # Anthropic does not provide embeddings, so we use FastEmbed instead
+    init_fastembed()
 
 
 def init_gemini():
@@ -144,3 +162,11 @@ def init_gemini():
 
     Settings.llm = Gemini(model=model_name)
     Settings.embed_model = GeminiEmbedding(model_name=embed_model_name)
+
+
+def init_mistral():
+    from llama_index.embeddings.mistralai import MistralAIEmbedding
+    from llama_index.llms.mistralai import MistralAI
+
+    Settings.llm = MistralAI(model=os.getenv("MODEL"))
+    Settings.embed_model = MistralAIEmbedding(model_name=os.getenv("EMBEDDING_MODEL"))

templates/components/ui/html/chat/hooks/use-config.ts

@@ -1,30 +1,24 @@
 "use client";
 
-import { useEffect, useMemo, useState } from "react";
-
 export interface ChatConfig {
-  chatAPI?: string;
-  starterQuestions?: string[];
+  backend?: string;
 }
 
-export function useClientConfig() {
-  const API_ROUTE = "/api/chat/config";
+function getBackendOrigin(): string {
   const chatAPI = process.env.NEXT_PUBLIC_CHAT_API;
-  const [config, setConfig] = useState<ChatConfig>({
-    chatAPI,
-  });
-
-  const configAPI = useMemo(() => {
-    const backendOrigin = chatAPI ? new URL(chatAPI).origin : "";
-    return `${backendOrigin}${API_ROUTE}`;
-  }, [chatAPI]);
-
-  useEffect(() => {
-    fetch(configAPI)
-      .then((response) => response.json())
-      .then((data) => setConfig({ ...data, chatAPI }))
-      .catch((error) => console.error("Error fetching config", error));
-  }, [chatAPI, configAPI]);
-
-  return config;
+  if (chatAPI) {
+    return new URL(chatAPI).origin;
+  } else {
+    if (typeof window !== "undefined") {
+      // Use BASE_URL from window.ENV
+      return (window as any).ENV?.BASE_URL || "";
+    }
+    return "";
+  }
+}
+
+export function useClientConfig(): ChatConfig {
+  return {
+    backend: getBackendOrigin(),
+  };
 }

templates/components/vectordbs/python/llamacloud/generate.py

@@ -0,0 +1,50 @@
+# flake8: noqa: E402
+from dotenv import load_dotenv
+
+from app.engine.index import get_index
+
+load_dotenv()
+
+import logging
+from llama_index.core.readers import SimpleDirectoryReader
+from app.engine.service import LLamaCloudFileService
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger()
+
+
+def generate_datasource():
+    logger.info("Generate index for the provided data")
+
+    index = get_index()
+    project_id = index._get_project_id()
+    pipeline_id = index._get_pipeline_id()
+
+    # use SimpleDirectoryReader to retrieve the files to process
+    reader = SimpleDirectoryReader(
+        "data",
+        recursive=True,
+    )
+    files_to_process = reader.input_files
+
+    # add each file to the LlamaCloud pipeline
+    for input_file in files_to_process:
+        with open(input_file, "rb") as f:
+            logger.info(
+                f"Adding file {input_file} to pipeline {index.name} in project {index.project_name}"
+            )
+            LLamaCloudFileService.add_file_to_pipeline(
+                project_id,
+                pipeline_id,
+                f,
+                custom_metadata={
+                    # Set private=false to mark the document as public (required for filtering)
+                    "private": "false",
+                },
+            )
+
+    logger.info("Finished generating the index")
+
+
+if __name__ == "__main__":
+    generate_datasource()

templates/components/vectordbs/python/llamacloud/index.py

@@ -0,0 +1,41 @@
+import logging
+import os
+from llama_index.indices.managed.llama_cloud import LlamaCloudIndex
+from llama_index.core.ingestion.api_utils import (
+    get_client as llama_cloud_get_client,
+)
+
+logger = logging.getLogger("uvicorn")
+
+
+def get_client():
+    return llama_cloud_get_client(
+        os.getenv("LLAMA_CLOUD_API_KEY"),
+        os.getenv("LLAMA_CLOUD_BASE_URL"),
+    )
+
+
+def get_index(params=None):
+    configParams = params or {}
+    pipelineConfig = configParams.get("llamaCloudPipeline", {})
+    name = pipelineConfig.get("pipeline", os.getenv("LLAMA_CLOUD_INDEX_NAME"))
+    project_name = pipelineConfig.get("project", os.getenv("LLAMA_CLOUD_PROJECT_NAME"))
+    api_key = os.getenv("LLAMA_CLOUD_API_KEY")
+    base_url = os.getenv("LLAMA_CLOUD_BASE_URL")
+    organization_id = os.getenv("LLAMA_CLOUD_ORGANIZATION_ID")
+
+    if name is None or project_name is None or api_key is None:
+        raise ValueError(
+            "Please set LLAMA_CLOUD_INDEX_NAME, LLAMA_CLOUD_PROJECT_NAME and LLAMA_CLOUD_API_KEY"
+            " to your environment variables or config them in .env file"
+        )
+
+    index = LlamaCloudIndex(
+        name=name,
+        project_name=project_name,
+        api_key=api_key,
+        base_url=base_url,
+        organization_id=organization_id,
+    )
+
+    return index

templates/components/vectordbs/python/llamacloud/query_filter.py

@@ -0,0 +1,35 @@
+from llama_index.core.vector_stores.types import MetadataFilter, MetadataFilters
+
+
+def generate_filters(doc_ids):
+    """
+    Generate public/private document filters based on the doc_ids and the vector store.
+    """
+    # Using "nin" filter to include the documents don't have the "private" key because they're uploaded in LlamaCloud UI
+    public_doc_filter = MetadataFilter(
+        key="private",
+        value=["true"],
+        operator="nin",  # type: ignore
+    )
+    selected_doc_filter = MetadataFilter(
+        key="file_id",  # Note: LLamaCloud uses "file_id" to reference private document ids as "doc_id" is a restricted field in LlamaCloud
+        value=doc_ids,
+        operator="in",  # type: ignore
+    )
+    if len(doc_ids) > 0:
+        # If doc_ids are provided, we will select both public and selected documents
+        filters = MetadataFilters(
+            filters=[
+                public_doc_filter,
+                selected_doc_filter,
+            ],
+            condition="or",  # type: ignore
+        )
+    else:
+        filters = MetadataFilters(
+            filters=[
+                public_doc_filter,
+            ]
+        )
+
+    return filters

templates/components/vectordbs/python/llamacloud/service.py

@@ -0,0 +1,173 @@
+from io import BytesIO
+import logging
+import os
+import time
+from typing import Any, Dict, List, Optional, Set, Tuple, Union
+import typing
+
+from fastapi import BackgroundTasks
+from llama_cloud import ManagedIngestionStatus, PipelineFileCreateCustomMetadataValue
+from pydantic import BaseModel
+import requests
+from app.api.routers.models import SourceNodes
+from app.engine.index import get_client
+from llama_index.core.schema import NodeWithScore
+
+
+logger = logging.getLogger("uvicorn")
+
+
+class LlamaCloudFile(BaseModel):
+    file_name: str
+    pipeline_id: str
+
+    def __eq__(self, other):
+        if not isinstance(other, LlamaCloudFile):
+            return NotImplemented
+        return (
+            self.file_name == other.file_name and self.pipeline_id == other.pipeline_id
+        )
+
+    def __hash__(self):
+        return hash((self.file_name, self.pipeline_id))
+
+
+class LLamaCloudFileService:
+    LOCAL_STORE_PATH = "output/llamacloud"
+    DOWNLOAD_FILE_NAME_TPL = "{pipeline_id}${filename}"
+
+    @classmethod
+    def get_all_projects_with_pipelines(cls) -> List[Dict[str, Any]]:
+        try:
+            client = get_client()
+            projects = client.projects.list_projects()
+            pipelines = client.pipelines.search_pipelines()
+            return [
+                {
+                    **(project.dict()),
+                    "pipelines": [
+                        {"id": p.id, "name": p.name}
+                        for p in pipelines
+                        if p.project_id == project.id
+                    ],
+                }
+                for project in projects
+            ]
+        except Exception as error:
+            logger.error(f"Error listing projects and pipelines: {error}")
+            return []
+
+    @classmethod
+    def add_file_to_pipeline(
+        cls,
+        project_id: str,
+        pipeline_id: str,
+        upload_file: Union[typing.IO, Tuple[str, BytesIO]],
+        custom_metadata: Optional[Dict[str, PipelineFileCreateCustomMetadataValue]],
+    ) -> str:
+        client = get_client()
+        file = client.files.upload_file(project_id=project_id, upload_file=upload_file)
+        files = [
+            {
+                "file_id": file.id,
+                "custom_metadata": {"file_id": file.id, **(custom_metadata or {})},
+            }
+        ]
+        files = client.pipelines.add_files_to_pipeline(pipeline_id, request=files)
+
+        # Wait 2s for the file to be processed
+        max_attempts = 20
+        attempt = 0
+        while attempt < max_attempts:
+            result = client.pipelines.get_pipeline_file_status(pipeline_id, file.id)
+            if result.status == ManagedIngestionStatus.ERROR:
+                raise Exception(f"File processing failed: {str(result)}")
+            if result.status == ManagedIngestionStatus.SUCCESS:
+                # File is ingested - return the file id
+                return file.id
+            attempt += 1
+            time.sleep(0.1)  # Sleep for 100ms
+        raise Exception(
+            f"File processing did not complete after {max_attempts} attempts."
+        )
+
+    @classmethod
+    def download_pipeline_file(
+        cls,
+        file: LlamaCloudFile,
+        force_download: bool = False,
+    ):
+        client = get_client()
+        file_name = file.file_name
+        pipeline_id = file.pipeline_id
+
+        # Check is the file already exists
+        downloaded_file_path = cls._get_file_path(file_name, pipeline_id)
+        if os.path.exists(downloaded_file_path) and not force_download:
+            logger.debug(f"File {file_name} already exists in local storage")
+            return
+        try:
+            logger.info(f"Downloading file {file_name} for pipeline {pipeline_id}")
+            files = client.pipelines.list_pipeline_files(pipeline_id)
+            if not files or not isinstance(files, list):
+                raise Exception("No files found in LlamaCloud")
+            for file_entry in files:
+                if file_entry.name == file_name:
+                    file_id = file_entry.file_id
+                    project_id = file_entry.project_id
+                    file_detail = client.files.read_file_content(
+                        file_id, project_id=project_id
+                    )
+                    cls._download_file(file_detail.url, downloaded_file_path)
+                    break
+        except Exception as error:
+            logger.info(f"Error fetching file from LlamaCloud: {error}")
+
+    @classmethod
+    def download_files_from_nodes(
+        cls, nodes: List[NodeWithScore], background_tasks: BackgroundTasks
+    ):
+        files = cls._get_files_to_download(nodes)
+        for file in files:
+            logger.info(f"Adding download of {file.file_name} to background tasks")
+            background_tasks.add_task(
+                LLamaCloudFileService.download_pipeline_file, file
+            )
+
+    @classmethod
+    def _get_files_to_download(cls, nodes: List[NodeWithScore]) -> Set[LlamaCloudFile]:
+        source_nodes = SourceNodes.from_source_nodes(nodes)
+        llama_cloud_files = [
+            LlamaCloudFile(
+                file_name=node.metadata.get("file_name"),
+                pipeline_id=node.metadata.get("pipeline_id"),
+            )
+            for node in source_nodes
+            if (
+                node.metadata.get("pipeline_id") is not None
+                and node.metadata.get("file_name") is not None
+            )
+        ]
+        # Remove duplicates and return
+        return set(llama_cloud_files)
+
+    @classmethod
+    def _get_file_name(cls, name: str, pipeline_id: str) -> str:
+        return cls.DOWNLOAD_FILE_NAME_TPL.format(pipeline_id=pipeline_id, filename=name)
+
+    @classmethod
+    def _get_file_path(cls, name: str, pipeline_id: str) -> str:
+        return os.path.join(cls.LOCAL_STORE_PATH, cls._get_file_name(name, pipeline_id))
+
+    @classmethod
+    def _download_file(cls, url: str, local_file_path: str):
+        logger.info(f"Saving file to {local_file_path}")
+        # Create directory if it doesn't exist
+        os.makedirs(cls.LOCAL_STORE_PATH, exist_ok=True)
+        # Download the file
+        with requests.get(url, stream=True) as r:
+            r.raise_for_status()
+            with open(local_file_path, "wb") as f:
+                for chunk in r.iter_content(chunk_size=8192):
+                    f.write(chunk)
+        logger.info("File downloaded successfully")

templates/components/vectordbs/python/none/generate.py

@@ -1,15 +1,16 @@
+# flake8: noqa: E402
 from dotenv import load_dotenv
 
 load_dotenv()
 
-import os
 import logging
+import os
+
+from app.engine.loaders import get_documents
+from app.settings import init_settings
 from llama_index.core.indices import (
     VectorStoreIndex,
 )
-from app.engine.loaders import get_documents
-from app.settings import init_settings
-
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger()
@@ -21,6 +22,9 @@ def generate_datasource():
     storage_dir = os.environ.get("STORAGE_DIR", "storage")
     # load the documents and create the index
     documents = get_documents()
+    # Set private=false to mark the document as public (required for filtering)
+    for doc in documents:
+        doc.metadata["private"] = "false"
     index = VectorStoreIndex.from_documents(
         documents,
     )

templates/components/vectordbs/python/none/index.py

@@ -17,7 +17,7 @@ def get_storage_context(persist_dir: str) -> StorageContext:
     return StorageContext.from_defaults(persist_dir=persist_dir)
 
 
-def get_index():
+def get_index(params=None):
     storage_dir = os.getenv("STORAGE_DIR", "storage")
     # check if storage already exists
     if not os.path.exists(storage_dir):

templates/components/vectordbs/python/weaviate/query_filter.py

@@ -0,0 +1,36 @@
+from llama_index.core.vector_stores.types import MetadataFilter, MetadataFilters
+
+
+def generate_filters(doc_ids):
+    """
+    Generate public/private document filters based on the doc_ids and the vector store.
+    """
+    public_doc_filter = MetadataFilter(
+        key="private",
+        value="true",
+        operator="!=",  # type: ignore
+    )
+    # Weaviate doesn't support "in" filter right now, so use "any" instead - it has the same behavior.
+    # TODO: Use "in" operator, once Weaviate supports it
+    selected_doc_filter = MetadataFilter(
+        key="doc_id",
+        value=doc_ids,
+        operator="any",  # type: ignore
+    )
+    if len(doc_ids) > 0:
+        # If doc_ids are provided, we will select both public and selected documents
+        filters = MetadataFilters(
+            filters=[
+                public_doc_filter,
+                selected_doc_filter,
+            ],
+            condition="or",  # type: ignore
+        )
+    else:
+        filters = MetadataFilters(
+            filters=[
+                public_doc_filter,
+            ]
+        )
+
+    return filters

templates/components/vectordbs/python/weaviate/vectordb.py

@@ -0,0 +1,35 @@
+import os
+
+import weaviate
+from llama_index.vector_stores.weaviate import WeaviateVectorStore
+
+DEFAULT_INDEX_NAME = "LlamaIndex"
+
+
+def _create_weaviate_client():
+    cluster_url = os.getenv("WEAVIATE_CLUSTER_URL")
+    api_key = os.getenv("WEAVIATE_API_KEY")
+    if not cluster_url or not api_key:
+        raise ValueError(
+            "Environment variables: WEAVIATE_CLUSTER_URL and WEAVIATE_API_KEY are required."
+        )
+    auth_credentials = weaviate.auth.AuthApiKey(api_key)
+    client = weaviate.connect_to_weaviate_cloud(cluster_url, auth_credentials)
+    return client
+
+
+# Global variable to store the Weaviate client
+client = None
+
+
+def get_vector_store():
+    global client
+    if client is None:
+        client = _create_weaviate_client()
+
+    index_name = os.getenv("WEAVIATE_INDEX_NAME", DEFAULT_INDEX_NAME)
+    vector_store = WeaviateVectorStore(
+        weaviate_client=client,
+        index_name=index_name,
+    )
+    return vector_store

templates/components/vectordbs/typescript/astra/index.ts

@@ -3,7 +3,7 @@ import { VectorStoreIndex } from "llamaindex";
 import { AstraDBVectorStore } from "llamaindex/storage/vectorStore/AstraDBVectorStore";
 import { checkRequiredEnvVars } from "./shared";
 
-export async function getDataSource() {
+export async function getDataSource(params?: any) {
   checkRequiredEnvVars();
   const store = new AstraDBVectorStore();
   await store.connect(process.env.ASTRA_DB_COLLECTION!);

templates/components/vectordbs/typescript/chroma/index.ts

@@ -3,7 +3,7 @@ import { VectorStoreIndex } from "llamaindex";
 import { ChromaVectorStore } from "llamaindex/storage/vectorStore/ChromaVectorStore";
 import { checkRequiredEnvVars } from "./shared";
 
-export async function getDataSource() {
+export async function getDataSource(params?: any) {
   checkRequiredEnvVars();
   const chromaUri = `http://${process.env.CHROMA_HOST}:${process.env.CHROMA_PORT}`;
 

templates/components/vectordbs/typescript/llamacloud/generate.ts

@@ -0,0 +1,29 @@
+import * as dotenv from "dotenv";
+import { LlamaCloudIndex } from "llamaindex";
+import { getDataSource } from "./index";
+import { getDocuments } from "./loader";
+import { initSettings } from "./settings";
+import { checkRequiredEnvVars } from "./shared";
+
+dotenv.config();
+
+async function loadAndIndex() {
+  const documents = await getDocuments();
+
+  await getDataSource();
+  await LlamaCloudIndex.fromDocuments({
+    documents,
+    name: process.env.LLAMA_CLOUD_INDEX_NAME!,
+    projectName: process.env.LLAMA_CLOUD_PROJECT_NAME!,
+    apiKey: process.env.LLAMA_CLOUD_API_KEY,
+    baseUrl: process.env.LLAMA_CLOUD_BASE_URL,
+  });
+  console.log(`Successfully created embeddings!`);
+}
+
+(async () => {
+  checkRequiredEnvVars();
+  initSettings();
+  await loadAndIndex();
+  console.log("Finished generating storage.");
+})();

templates/components/vectordbs/typescript/llamacloud/index.ts

@@ -0,0 +1,27 @@
+import { LlamaCloudIndex } from "llamaindex/cloud/LlamaCloudIndex";
+
+type LlamaCloudDataSourceParams = {
+  llamaCloudPipeline?: {
+    project: string;
+    pipeline: string;
+  };
+};
+
+export async function getDataSource(params?: LlamaCloudDataSourceParams) {
+  const { project, pipeline } = params?.llamaCloudPipeline ?? {};
+  const projectName = project ?? process.env.LLAMA_CLOUD_PROJECT_NAME;
+  const pipelineName = pipeline ?? process.env.LLAMA_CLOUD_INDEX_NAME;
+  const apiKey = process.env.LLAMA_CLOUD_API_KEY;
+  if (!projectName || !pipelineName || !apiKey) {
+    throw new Error(
+      "Set project, pipeline, and api key in the params or as environment variables.",
+    );
+  }
+  const index = new LlamaCloudIndex({
+    name: pipelineName,
+    projectName,
+    apiKey,
+    baseUrl: process.env.LLAMA_CLOUD_BASE_URL,
+  });
+  return index;
+}

templates/components/vectordbs/typescript/llamacloud/queryFilter.ts

@@ -0,0 +1,25 @@
+import { MetadataFilter, MetadataFilters } from "llamaindex";
+
+export function generateFilters(documentIds: string[]): MetadataFilters {
+  // public documents don't have the "private" field or it's set to "false"
+  const publicDocumentsFilter: MetadataFilter = {
+    key: "private",
+    value: ["true"],
+    operator: "nin",
+  };
+
+  // if no documentIds are provided, only retrieve information from public documents
+  if (!documentIds.length) return { filters: [publicDocumentsFilter] };
+
+  const privateDocumentsFilter: MetadataFilter = {
+    key: "doc_id",
+    value: documentIds,
+    operator: "in",
+  };
+
+  // if documentIds are provided, retrieve information from public and private documents
+  return {
+    filters: [publicDocumentsFilter, privateDocumentsFilter],
+    condition: "or",
+  };
+}

templates/components/vectordbs/typescript/llamacloud/shared.ts

@@ -0,0 +1,22 @@
+const REQUIRED_ENV_VARS = [
+  "LLAMA_CLOUD_INDEX_NAME",
+  "LLAMA_CLOUD_PROJECT_NAME",
+  "LLAMA_CLOUD_API_KEY",
+];
+
+export function checkRequiredEnvVars() {
+  const missingEnvVars = REQUIRED_ENV_VARS.filter((envVar) => {
+    return !process.env[envVar];
+  });
+
+  if (missingEnvVars.length > 0) {
+    console.log(
+      `The following environment variables are required but missing: ${missingEnvVars.join(
+        ", ",
+      )}`,
+    );
+    throw new Error(
+      `Missing environment variables: ${missingEnvVars.join(", ")}`,
+    );
+  }
+}

templates/components/vectordbs/typescript/milvus/index.ts

@@ -2,7 +2,7 @@ import { VectorStoreIndex } from "llamaindex";
 import { MilvusVectorStore } from "llamaindex/storage/vectorStore/MilvusVectorStore";
 import { checkRequiredEnvVars, getMilvusClient } from "./shared";
 
-export async function getDataSource() {
+export async function getDataSource(params?: any) {
   checkRequiredEnvVars();
   const milvusClient = getMilvusClient();
   const store = new MilvusVectorStore({ milvusClient });

templates/components/vectordbs/typescript/milvus/shared.ts

@@ -4,7 +4,6 @@ const REQUIRED_ENV_VARS = [
   "MILVUS_ADDRESS",
   "MILVUS_USERNAME",
   "MILVUS_PASSWORD",
-  "MILVUS_COLLECTION",
 ];
 
 export function getMilvusClient() {
@@ -19,8 +18,13 @@ export function getMilvusClient() {
   });
 }
 
-export function checkRequiredEnvVars() {
-  const missingEnvVars = REQUIRED_ENV_VARS.filter((envVar) => {
+export function checkRequiredEnvVars(opts?: { checkCollectionEnv?: boolean }) {
+  const shouldCheckCollectionEnv = opts?.checkCollectionEnv ?? true; // default to true
+  const requiredEnvVars = [...REQUIRED_ENV_VARS]; // create a copy of the array
+  if (shouldCheckCollectionEnv) {
+    requiredEnvVars.push("MILVUS_COLLECTION");
+  }
+  const missingEnvVars = requiredEnvVars.filter((envVar) => {
     return !process.env[envVar];
   });
 

templates/components/vectordbs/typescript/mongo/generate.ts

@@ -1,7 +1,10 @@
 /* eslint-disable turbo/no-undeclared-env-vars */
 import * as dotenv from "dotenv";
-import { VectorStoreIndex, storageContextFromDefaults } from "llamaindex";
-import { MongoDBAtlasVectorSearch } from "llamaindex/storage/vectorStore/MongoDBAtlasVectorSearch";
+import {
+  MongoDBAtlasVectorSearch,
+  VectorStoreIndex,
+  storageContextFromDefaults,
+} from "llamaindex";
 import { MongoClient } from "mongodb";
 import { getDocuments } from "./loader";
 import { initSettings } from "./settings";

templates/components/vectordbs/typescript/mongo/index.ts

@@ -1,10 +1,9 @@
 /* eslint-disable turbo/no-undeclared-env-vars */
-import { VectorStoreIndex } from "llamaindex";
-import { MongoDBAtlasVectorSearch } from "llamaindex/storage/vectorStore/MongoDBAtlasVectorSearch";
+import { MongoDBAtlasVectorSearch, VectorStoreIndex } from "llamaindex";
 import { MongoClient } from "mongodb";
 import { checkRequiredEnvVars } from "./shared";
 
-export async function getDataSource() {
+export async function getDataSource(params?: any) {
   checkRequiredEnvVars();
   const client = new MongoClient(process.env.MONGO_URI!);
   const store = new MongoDBAtlasVectorSearch({

templates/components/vectordbs/typescript/none/generate.ts

@@ -25,6 +25,7 @@ async function generateDatasource() {
       persistDir: STORAGE_CACHE_DIR,
     });
     const documents = await getDocuments();
+
     await VectorStoreIndex.fromDocuments(documents, {
       storageContext,
     });

templates/components/vectordbs/typescript/none/index.ts

@@ -2,7 +2,7 @@ import { SimpleDocumentStore, VectorStoreIndex } from "llamaindex";
 import { storageContextFromDefaults } from "llamaindex/storage/StorageContext";
 import { STORAGE_CACHE_DIR } from "./shared";
 
-export async function getDataSource() {
+export async function getDataSource(params?: any) {
   const storageContext = await storageContextFromDefaults({
     persistDir: `${STORAGE_CACHE_DIR}`,
   });

templates/components/vectordbs/typescript/pg/index.ts

@@ -7,7 +7,7 @@ import {
   checkRequiredEnvVars,
 } from "./shared";
 
-export async function getDataSource() {
+export async function getDataSource(params?: any) {
   checkRequiredEnvVars();
   const pgvs = new PGVectorStore({
     connectionString: process.env.PG_CONNECTION_STRING,

templates/components/vectordbs/typescript/pinecone/index.ts

@@ -3,7 +3,7 @@ import { VectorStoreIndex } from "llamaindex";
 import { PineconeVectorStore } from "llamaindex/storage/vectorStore/PineconeVectorStore";
 import { checkRequiredEnvVars } from "./shared";
 
-export async function getDataSource() {
+export async function getDataSource(params?: any) {
   checkRequiredEnvVars();
   const store = new PineconeVectorStore();
   return await VectorStoreIndex.fromVectorStore(store);

templates/components/vectordbs/typescript/qdrant/index.ts

@@ -5,7 +5,7 @@ import { checkRequiredEnvVars, getQdrantClient } from "./shared";
 
 dotenv.config();
 
-export async function getDataSource() {
+export async function getDataSource(params?: any) {
   checkRequiredEnvVars();
   const collectionName = process.env.QDRANT_COLLECTION;
   const store = new QdrantVectorStore({

templates/components/vectordbs/typescript/weaviate/generate.ts

@@ -0,0 +1,33 @@
+/* eslint-disable turbo/no-undeclared-env-vars */
+import * as dotenv from "dotenv";
+import {
+  VectorStoreIndex,
+  WeaviateVectorStore,
+  storageContextFromDefaults,
+} from "llamaindex";
+import { getDocuments } from "./loader";
+import { initSettings } from "./settings";
+import { DEFAULT_INDEX_NAME, checkRequiredEnvVars } from "./shared";
+dotenv.config();
+
+async function loadAndIndex() {
+  const indexName = process.env.WEAVIATE_INDEX_NAME || DEFAULT_INDEX_NAME;
+
+  // load objects from storage and convert them into LlamaIndex Document objects
+  const documents = await getDocuments();
+
+  const vectorStore = new WeaviateVectorStore({ indexName });
+
+  const storageContext = await storageContextFromDefaults({ vectorStore });
+  await VectorStoreIndex.fromDocuments(documents, {
+    storageContext: storageContext,
+  });
+  console.log(`Successfully upload embeddings to Weaviate index ${indexName}.`);
+}
+
+(async () => {
+  checkRequiredEnvVars();
+  initSettings();
+  await loadAndIndex();
+  console.log("Finished generating storage.");
+})();

templates/components/vectordbs/typescript/weaviate/index.ts

@@ -0,0 +1,14 @@
+import * as dotenv from "dotenv";
+import { VectorStoreIndex } from "llamaindex";
+import { WeaviateVectorStore } from "llamaindex/storage/vectorStore/WeaviateVectorStore";
+import { checkRequiredEnvVars, DEFAULT_INDEX_NAME } from "./shared";
+
+dotenv.config();
+
+export async function getDataSource(params?: any) {
+  checkRequiredEnvVars();
+  const indexName = process.env.WEAVIATE_INDEX_NAME || DEFAULT_INDEX_NAME;
+  const store = new WeaviateVectorStore({ indexName });
+
+  return await VectorStoreIndex.fromVectorStore(store);
+}

templates/components/vectordbs/typescript/weaviate/queryFilter.ts

@@ -0,0 +1,26 @@
+import { MetadataFilter, MetadataFilters } from "llamaindex";
+
+export function generateFilters(documentIds: string[]): MetadataFilters {
+  // filter all documents have the private metadata key set to true
+  const publicDocumentsFilter: MetadataFilter = {
+    key: "private",
+    value: "true",
+    operator: "!=",
+  };
+
+  // if no documentIds are provided, only retrieve information from public documents
+  if (!documentIds.length) return { filters: [publicDocumentsFilter] };
+
+  // Weaviate uses 'any' instead of 'in' for the operator
+  const privateDocumentsFilter: MetadataFilter = {
+    key: "doc_id",
+    value: documentIds,
+    operator: "any",
+  };
+
+  // if documentIds are provided, retrieve information from public and private documents
+  return {
+    filters: [publicDocumentsFilter, privateDocumentsFilter],
+    condition: "or",
+  };
+}

templates/components/vectordbs/typescript/weaviate/shared.ts

@@ -0,0 +1,20 @@
+const REQUIRED_ENV_VARS = ["WEAVIATE_CLUSTER_URL", "WEAVIATE_API_KEY"];
+
+export const DEFAULT_INDEX_NAME = "LlamaIndex";
+
+export function checkRequiredEnvVars() {
+  const missingEnvVars = REQUIRED_ENV_VARS.filter((envVar) => {
+    return !process.env[envVar];
+  });
+
+  if (missingEnvVars.length > 0) {
+    console.log(
+      `The following environment variables are required but missing: ${missingEnvVars.join(
+        ", ",
+      )}`,
+    );
+    throw new Error(
+      `Missing environment variables: ${missingEnvVars.join(", ")}`,
+    );
+  }
+}

templates/types/extractor/fastapi/README-template.md

@@ -0,0 +1,68 @@
+This is a [LlamaIndex](https://www.llamaindex.ai/) project using [FastAPI](https://fastapi.tiangolo.com/) bootstrapped with [`create-llama`](https://github.com/run-llama/LlamaIndexTS/tree/main/packages/create-llama) featuring [structured extraction](https://docs.llamaindex.ai/en/stable/examples/structured_outputs/structured_outputs/?h=structured+output).
+
+## Getting Started
+
+First, setup the environment with poetry:
+
+> **_Note:_** This step is not needed if you are using the dev-container.
+
+```shell
+poetry install
+poetry shell
+```
+
+Then check the parameters that have been pre-configured in the `.env` file in this directory. (E.g. you might need to configure an `OPENAI_API_KEY` if you're using OpenAI as model provider).
+
+Second, generate the embeddings of the documents in the `./data` directory (if this folder exists - otherwise, skip this step):
+
+```shell
+poetry run generate
+```
+
+Third, run the API in one command:
+
+```shell
+poetry run python main.py
+```
+
+The example provides the `/api/extractor/query` API endpoint.
+
+This query endpoint returns structured data in the format of the [Output](./app/api/routers/output.py) class. Modify this class to change the output format.
+
+You can test the endpoint with the following curl request:
+
+```shell
+curl --location 'localhost:8000/api/extractor/query' \
+--header 'Content-Type: application/json' \
+--data '{ "query": "What is the maximum weight for a parcel?" }'
+```
+
+Which will return a response that the RAG pipeline is confident about the answer.
+
+Try
+
+```shell
+curl --location 'localhost:8000/api/extractor/query' \
+--header 'Content-Type: application/json' \
+--data '{ "query": "What is the weather today?" }'
+```
+
+To retrieve a response with low confidence since the question is not related to the provided document in the `./data` directory.
+
+You can start editing the API endpoint by modifying [`extractor.py`](./app/api/routers/extractor.py). The endpoints auto-update as you save the file.
+
+Open [http://localhost:8000/docs](http://localhost:8000/docs) with your browser to see the Swagger UI of the API.
+
+The API allows CORS for all origins to simplify development. You can change this behavior by setting the `ENVIRONMENT` environment variable to `prod`:
+
+```
+ENVIRONMENT=prod python main.py
+```
+
+## Learn More
+
+To learn more about LlamaIndex, take a look at the following resources:
+
+- [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex.
+
+You can check out [the LlamaIndex GitHub repository](https://github.com/run-llama/llama_index) - your feedback and contributions are welcome!

templates/types/extractor/fastapi/app/api/routers/extractor.py

@@ -0,0 +1,58 @@
+import logging
+import os
+
+from fastapi import APIRouter, HTTPException
+from llama_index.core.settings import Settings
+from pydantic import BaseModel
+
+from app.api.routers.output import Output
+from app.engine.index import get_index
+
+extractor_router = r = APIRouter()
+
+logger = logging.getLogger("uvicorn")
+
+
+class RequestData(BaseModel):
+    query: str
+
+    class Config:
+        json_schema_extra = {
+            "examples": [
+                {"query": "What's the maximum weight for a parcel?"},
+            ],
+        }
+
+
+@r.post("/query")
+async def query_request(
+    data: RequestData,
+):
+    # Create a query engine using that returns responses in the format of the Output class
+    query_engine = get_query_engine(Output)
+
+    response = await query_engine.aquery(data.query)
+
+    output_data = response.response.dict()
+    return Output(**output_data)
+
+
+def get_query_engine(output_cls: BaseModel):
+    top_k = os.getenv("TOP_K", 3)
+
+    index = get_index()
+    if index is None:
+        raise HTTPException(
+            status_code=500,
+            detail=str(
+                "StorageContext is empty - call 'poetry run generate' to generate the storage first"
+            ),
+        )
+
+    sllm = Settings.llm.as_structured_llm(output_cls)
+
+    return index.as_query_engine(
+        similarity_top_k=int(top_k),
+        llm=sllm,
+        response_mode="tree_summarize",
+    )

templates/types/extractor/fastapi/app/api/routers/output.py

@@ -0,0 +1,32 @@
+import logging
+from llama_index.core.schema import BaseModel, Field
+from typing import List
+
+logger = logging.getLogger("uvicorn")
+
+
+class Output(BaseModel):
+    response: str = Field(..., description="The answer to the question.")
+    page_numbers: List[int] = Field(
+        ...,
+        description="The page numbers of the sources used to answer this question. Do not include a page number if the context is irrelevant.",
+    )
+    confidence: float = Field(
+        ...,
+        ge=0,
+        le=1,
+        description="Confidence value between 0-1 of the correctness of the result.",
+    )
+    confidence_explanation: str = Field(
+        ..., description="Explanation for the confidence score"
+    )
+
+    class Config:
+        json_schema_extra = {
+            "example": {
+                "response": "This is an example answer.",
+                "page_numbers": [1, 2, 3],
+                "confidence": 0.85,
+                "confidence_explanation": "This is an explanation for the confidence score.",
+            }
+        }

templates/types/extractor/fastapi/main.py

@@ -0,0 +1,46 @@
+# flake8: noqa: E402
+from dotenv import load_dotenv
+
+load_dotenv()
+
+import logging
+import os
+import uvicorn
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import RedirectResponse
+from app.api.routers.extractor import extractor_router
+from app.settings import init_settings
+
+
+app = FastAPI()
+
+init_settings()
+
+environment = os.getenv("ENVIRONMENT", "dev")  # Default to 'development' if not set
+logger = logging.getLogger("uvicorn")
+
+if environment == "dev":
+    logger.warning("Running in development mode - allowing CORS for all origins")
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=["*"],
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+    # Redirect to documentation page when accessing base URL
+    @app.get("/")
+    async def redirect_to_docs():
+        return RedirectResponse(url="/docs")
+
+
+app.include_router(extractor_router, prefix="/api/extractor")
+
+if __name__ == "__main__":
+    app_host = os.getenv("APP_HOST", "0.0.0.0")
+    app_port = int(os.getenv("APP_PORT", "8000"))
+    reload = True if environment == "dev" else False
+
+    uvicorn.run(app="main:app", host=app_host, port=app_port, reload=reload)

templates/types/extractor/fastapi/pyproject.toml

@@ -0,0 +1,21 @@
+[tool.poetry]
+name = "app"
+version = "0.1.0"
+description = ""
+authors = ["Marcus Schiesser <mail@marcusschiesser.de>"]
+readme = "README.md"
+
+[tool.poetry.scripts]
+generate = "app.engine.generate:generate_datasource"
+
+[tool.poetry.dependencies]
+python = "^3.11,<4.0"
+fastapi = "^0.109.1"
+uvicorn = { extras = ["standard"], version = "^0.23.2" }
+python-dotenv = "^1.0.0"
+llama-index = "^0.10.58"
+cachetools = "^5.3.3"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

templates/types/multiagent/fastapi/README-template.md

@@ -0,0 +1,50 @@
+This is a [LlamaIndex](https://www.llamaindex.ai/) project using [FastAPI](https://fastapi.tiangolo.com/) bootstrapped with [`create-llama`](https://github.com/run-llama/LlamaIndexTS/tree/main/packages/create-llama).
+
+## Getting Started
+
+First, setup the environment with poetry:
+
+> **_Note:_** This step is not needed if you are using the dev-container.
+
+```shell
+poetry install
+poetry shell
+```
+
+Then check the parameters that have been pre-configured in the `.env` file in this directory. (E.g. you might need to configure an `OPENAI_API_KEY` if you're using OpenAI as model provider).
+
+Second, generate the embeddings of the documents in the `./data` directory (if this folder exists - otherwise, skip this step):
+
+```shell
+poetry run generate
+```
+
+Third, run all the services in one command:
+
+```shell
+poetry run python main.py
+```
+
+You can monitor and test the agent services with `llama-agents` monitor TUI:
+
+```shell
+poetry run llama-agents monitor --control-plane-url http://127.0.0.1:8001
+```
+
+## Services:
+
+- Message queue (port 8000): To exchange the message between services
+- Control plane (port 8001): A gateway to manage the tasks and services.
+- Human consumer (port 8002): To handle result when the task is completed.
+- Agent service `query_engine` (port 8003): Agent that can query information from the configured LlamaIndex index.
+- Agent service `dummy_agent` (port 8004): A dummy agent that does nothing. Good starting point to add more agents.
+
+The ports listed above are set by default, but you can change them in the `.env` file.
+
+## Learn More
+
+To learn more about LlamaIndex, take a look at the following resources:
+
+- [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex.
+
+You can check out [the LlamaIndex GitHub repository](https://github.com/run-llama/llama_index) - your feedback and contributions are welcome!

templates/types/multiagent/fastapi/app/agents/dummy/agent.py

@@ -0,0 +1,33 @@
+from llama_agents import AgentService, SimpleMessageQueue
+from llama_index.core.agent import FunctionCallingAgentWorker
+from llama_index.core.tools import FunctionTool
+from llama_index.core.settings import Settings
+from app.utils import load_from_env
+
+
+DEFAULT_DUMMY_AGENT_DESCRIPTION = "I'm a dummy agent which does nothing."
+
+
+def dummy_function():
+    """
+    This function does nothing.
+    """
+    return ""
+
+
+def init_dummy_agent(message_queue: SimpleMessageQueue) -> AgentService:
+    agent = FunctionCallingAgentWorker(
+        tools=[FunctionTool.from_defaults(fn=dummy_function)],
+        llm=Settings.llm,
+        prefix_messages=[],
+    ).as_agent()
+
+    return AgentService(
+        service_name="dummy_agent",
+        agent=agent,
+        message_queue=message_queue.client,
+        description=load_from_env("AGENT_DUMMY_DESCRIPTION", throw_error=False)
+        or DEFAULT_DUMMY_AGENT_DESCRIPTION,
+        host=load_from_env("AGENT_DUMMY_HOST", throw_error=False) or "127.0.0.1",
+        port=int(load_from_env("AGENT_DUMMY_PORT")),
+    )

templates/types/multiagent/fastapi/app/agents/query_engine/agent.py

@@ -0,0 +1,52 @@
+import os
+from llama_agents import AgentService, SimpleMessageQueue
+from llama_index.core.agent import FunctionCallingAgentWorker
+from llama_index.core.tools import QueryEngineTool, ToolMetadata
+from llama_index.core.settings import Settings
+from app.engine.index import get_index
+from app.utils import load_from_env
+
+
+DEFAULT_QUERY_ENGINE_AGENT_DESCRIPTION = (
+    "Used to answer the questions using the provided context data."
+)
+
+
+def get_query_engine_tool() -> QueryEngineTool:
+    """
+    Provide an agent worker that can be used to query the index.
+    """
+    index = get_index()
+    if index is None:
+        raise ValueError("Index not found. Please create an index first.")
+    query_engine = index.as_query_engine(similarity_top_k=int(os.getenv("TOP_K", 3)))
+    return QueryEngineTool(
+        query_engine=query_engine,
+        metadata=ToolMetadata(
+            name="context_data",
+            description="""
+                Provide the provided context information. 
+                Use a detailed plain text question as input to the tool.
+            """,
+        ),
+    )
+
+
+def init_query_engine_agent(
+    message_queue: SimpleMessageQueue,
+) -> AgentService:
+    """
+    Initialize the agent service.
+    """
+    agent = FunctionCallingAgentWorker(
+        tools=[get_query_engine_tool()], llm=Settings.llm, prefix_messages=[]
+    ).as_agent()
+    return AgentService(
+        service_name="context_query_agent",
+        agent=agent,
+        message_queue=message_queue.client,
+        description=load_from_env("AGENT_QUERY_ENGINE_DESCRIPTION", throw_error=False)
+        or DEFAULT_QUERY_ENGINE_AGENT_DESCRIPTION,
+        host=load_from_env("AGENT_QUERY_ENGINE_HOST", throw_error=False) or "127.0.0.1",
+        port=int(load_from_env("AGENT_QUERY_ENGINE_PORT")),
+    )

templates/types/multiagent/fastapi/app/core/control_plane.py

@@ -0,0 +1,19 @@
+from llama_index.llms.openai import OpenAI
+from llama_agents import AgentOrchestrator, ControlPlaneServer
+from app.core.message_queue import message_queue
+from app.utils import load_from_env
+
+
+control_plane_host = (
+    load_from_env("CONTROL_PLANE_HOST", throw_error=False) or "127.0.0.1"
+)
+control_plane_port = load_from_env("CONTROL_PLANE_PORT", throw_error=False) or "8001"
+
+
+# setup control plane
+control_plane = ControlPlaneServer(
+    message_queue=message_queue,
+    orchestrator=AgentOrchestrator(llm=OpenAI()),
+    host=control_plane_host,
+    port=int(control_plane_port) if control_plane_port else None,
+)

templates/types/multiagent/fastapi/app/core/message_queue.py

@@ -0,0 +1,12 @@
+from llama_agents import SimpleMessageQueue
+from app.utils import load_from_env
+
+message_queue_host = (
+    load_from_env("MESSAGE_QUEUE_HOST", throw_error=False) or "127.0.0.1"
+)
+message_queue_port = load_from_env("MESSAGE_QUEUE_PORT", throw_error=False) or "8000"
+
+message_queue = SimpleMessageQueue(
+    host=message_queue_host,
+    port=int(message_queue_port) if message_queue_port else None,
+)

templates/types/multiagent/fastapi/app/core/task_result.py

@@ -0,0 +1,88 @@
+import json
+from logging import getLogger
+from pathlib import Path
+from fastapi import FastAPI
+from typing import Dict, Optional
+from llama_agents import CallableMessageConsumer, QueueMessage
+from llama_agents.message_queues.base import BaseMessageQueue
+from llama_agents.message_consumers.base import BaseMessageQueueConsumer
+from llama_agents.message_consumers.remote import RemoteMessageConsumer
+from app.utils import load_from_env
+from app.core.message_queue import message_queue
+
+
+logger = getLogger(__name__)
+
+
+class TaskResultService:
+    def __init__(
+        self,
+        message_queue: BaseMessageQueue,
+        name: str = "human",
+        host: str = "127.0.0.1",
+        port: Optional[int] = 8002,
+    ) -> None:
+        self.name = name
+        self.host = host
+        self.port = port
+
+        self._message_queue = message_queue
+
+        # app
+        self._app = FastAPI()
+        self._app.add_api_route(
+            "/", self.home, methods=["GET"], tags=["Human Consumer"]
+        )
+        self._app.add_api_route(
+            "/process_message",
+            self.process_message,
+            methods=["POST"],
+            tags=["Human Consumer"],
+        )
+
+    @property
+    def message_queue(self) -> BaseMessageQueue:
+        return self._message_queue
+
+    def as_consumer(self, remote: bool = False) -> BaseMessageQueueConsumer:
+        if remote:
+            return RemoteMessageConsumer(
+                url=(
+                    f"http://{self.host}:{self.port}/process_message"
+                    if self.port
+                    else f"http://{self.host}/process_message"
+                ),
+                message_type=self.name,
+            )
+
+        return CallableMessageConsumer(
+            message_type=self.name,
+            handler=self.process_message,
+        )
+
+    async def process_message(self, message: QueueMessage) -> None:
+        Path("task_results").mkdir(exist_ok=True)
+        with open("task_results/task_results.json", "+a") as f:
+            json.dump(message.model_dump(), f)
+            f.write("\n")
+
+    async def home(self) -> Dict[str, str]:
+        return {"message": "hello, human."}
+
+    async def register_to_message_queue(self) -> None:
+        """Register to the message queue."""
+        await self.message_queue.register_consumer(self.as_consumer(remote=True))
+
+
+human_consumer_host = (
+    load_from_env("HUMAN_CONSUMER_HOST", throw_error=False) or "127.0.0.1"
+)
+human_consumer_port = load_from_env("HUMAN_CONSUMER_PORT", throw_error=False) or "8002"
+
+
+human_consumer_server = TaskResultService(
+    message_queue=message_queue,
+    host=human_consumer_host,
+    port=int(human_consumer_port) if human_consumer_port else None,
+    name="human",
+)

templates/types/multiagent/fastapi/app/utils.py

@@ -0,0 +1,8 @@
+import os
+
+
+def load_from_env(var: str, throw_error: bool = True) -> str:
+    res = os.getenv(var)
+    if res is None and throw_error:
+        raise ValueError(f"Missing environment variable: {var}")
+    return res

templates/types/multiagent/fastapi/main.py

@@ -0,0 +1,28 @@
+# flake8: noqa: E402
+from dotenv import load_dotenv
+from app.settings import init_settings
+
+load_dotenv()
+init_settings()
+
+from llama_agents import ServerLauncher
+from app.core.message_queue import message_queue
+from app.core.control_plane import control_plane
+from app.core.task_result import human_consumer_server
+from app.agents.query_engine.agent import init_query_engine_agent
+from app.agents.dummy.agent import init_dummy_agent
+
+agents = [
+    init_query_engine_agent(message_queue),
+    init_dummy_agent(message_queue),
+]
+
+launcher = ServerLauncher(
+    agents,
+    control_plane,
+    message_queue,
+    additional_consumers=[human_consumer_server.as_consumer()],
+)
+
+if __name__ == "__main__":
+    launcher.launch_servers()