feat: Generate NEXT_PUBLIC_CHAT_API for NextJS backend to specify alternative backend

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

.changeset/great-students-appear.md

@@ -1,5 +0,0 @@
----
-"create-llama": patch
----
-
-docs: chroma env variables

.changeset/perfect-days-smoke.md

@@ -0,0 +1,5 @@
+---
+"create-llama": patch
+---
+
+Generate NEXT_PUBLIC_CHAT_API for NextJS backend to specify alternative backend

.github/workflows/e2e.yml

@@ -86,7 +86,7 @@ jobs:
         python-version: ["3.11"]
         os: [macos-latest, windows-latest, ubuntu-22.04]
         frameworks: ["nextjs", "express"]
-        datasources: ["--no-files", "--example-file"]
+        datasources: ["--no-files", "--example-file", "--llamacloud"]
     defaults:
       run:
         shell: bash

.gitignore

@@ -51,3 +51,7 @@ e2e/cache
 
 # build artifacts
 create-llama-*.tgz
+
+# vscode
+.vscode
+!.vscode/settings.json

CHANGELOG.md

@@ -1,5 +1,117 @@
 # create-llama
 
+## 0.3.15
+
+### Patch Changes
+
+- fc5b266: Improve DX for Python template (use one deployment instead of two)
+- f8f97d2: Add support for python 3.13
+
+## 0.3.14
+
+### Patch Changes
+
+- 00f0b3a: fix: dont include user message in chat history
+- 4663dec: chore: bump react19 rc
+- 44b34fb: chore: update eslint 9, nextjs 15, react 19
+- 6925676: feat: use latest chat UI
+
+## 0.3.13
+
+### Patch Changes
+
+- 282eaa0: Ensure that the index and document store are created when uploading a file with no available index.
+
+## 0.3.12
+
+### Patch Changes
+
+- 6edea6a: Optimize generated workflow code for Python
+- 8431b78: Optimize Typescript multi-agent code
+- 8431b78: Add form filling use case (Typescript)
+
+## 0.3.11
+
+### Patch Changes
+
+- 2b8aaa8: Add support for local models via Hugging Face
+- b9570b2: Fix: use generic LLMAgent instead of OpenAIAgent (adds support for Gemini and Anthropic for Agentic RAG)
+- 1fe21f8: Fix the highlight.js issue with the Next.js static build
+- 00009ae: feat: import pdf css
+
+## 0.3.10
+
+### Patch Changes
+
+- 9172fed: feat: bump LITS 0.8.2
+- 78ccde7: feat: use llamaindex chat-ui for nextjs frontend
+
+## 0.3.9
+
+### Patch Changes
+
+- ed59927: Add form filling use case (Python)
+
+## 0.3.8
+
+### Patch Changes
+
+- 4a83469: Add multi-agent financial report for Typescript (and update LITS to 0.7.10)
+
+## 0.3.7
+
+### Patch Changes
+
+- fa80378: DocumentInfo working with relative URLs
+
+## 0.3.6
+
+### Patch Changes
+
+- 0182368: Fix the streaming issue to prevent the UI from hanging.
+
+## 0.3.5
+
+### Patch Changes
+
+- 2209409: Add financial report as the default use case in the multi-agent template (Python).
+
+## 0.3.4
+
+### Patch Changes
+
+- 384a136: Fix import error if the artifact tool is selected
+
+## 0.3.3
+
+### Patch Changes
+
+- 99b8247: Simplify and unify handling file uploads
+
+## 0.3.2
+
+### Patch Changes
+
+- 6d1b6b9: Update README.md for pro mode
+
+## 0.3.1
+
+### Patch Changes
+
+- f3577c5: Fix event streaming is blocked
+- f3577c5: Add upload file to sandbox (artifact and code interpreter)
+
+## 0.3.0
+
+### Minor Changes
+
+- 7562cb4: Simplified default questions and added pro mode
+
+### Patch Changes
+
+- 0a69fe0: fix: missing params when init Astra vectorstore
+- 98a82b0: docs: chroma env variables
+
 ## 0.2.19
 
 ### Patch Changes

README.md

@@ -12,7 +12,7 @@ npx create-llama@latest
 
 to get started, or watch this video for a demo session:
 
-https://github.com/user-attachments/assets/dd3edc36-4453-4416-91c2-d24326c6c167
+<img src="https://github.com/user-attachments/assets/c4a7fe18-8e30-498a-96f8-78127dd706b9" width="100%">
 
 Once your app is generated, run
 
@@ -24,14 +24,14 @@ to start the development server. You can then visit [http://localhost:3000](http
 
 ## What you'll get
 
+- A set of pre-configured use cases to get you started, e.g. Agentic RAG, Data Analysis, Report Generation, etc.
 - A Next.js-powered front-end using components from [shadcn/ui](https://ui.shadcn.com/). The app is set up as a chat interface that can answer questions about your data or interact with your agent
-- Your choice of 3 back-ends:
+- Your choice of two back-ends:
   - **Next.js**: if you select this option, you’ll have a full-stack Next.js application that you can deploy to a host like [Vercel](https://vercel.com/) in just a few clicks. This uses [LlamaIndex.TS](https://www.npmjs.com/package/llamaindex), our TypeScript library.
-  - **Express**: if you want a more traditional Node.js application you can generate an Express backend. This also uses LlamaIndex.TS.
-  - **Python FastAPI**: if you select this option, you’ll get a backend powered by the [llama-index Python package](https://pypi.org/project/llama-index/), which you can deploy to a service like Render or fly.io.
-- The back-end has two endpoints (one streaming, the other one non-streaming) that allow you to send the state of your chat and receive additional responses
-- You add arbitrary data sources to your chat, like local files, websites, or data retrieved from a database.
-- Turn your chat into an AI agent by adding tools (functions called by the LLM).
+  - **Python FastAPI**: if you select this option, you’ll get a separate backend powered by the [llama-index Python package](https://pypi.org/project/llama-index/), which you can deploy to a service like [Render](https://render.com/) or [fly.io](https://fly.io/). The separate Next.js front-end will connect to this backend.
+- Each back-end has two endpoints:
+  - One streaming chat endpoint, that allow you to send the state of your chat and receive additional responses
+  - One endpoint to upload private files which can be used in your chat
 - The app uses OpenAI by default, so you'll need an OpenAI API key, or you can customize it to use any of the dozens of LLMs we support.
 
 Here's how it looks like:
@@ -40,9 +40,9 @@ https://github.com/user-attachments/assets/d57af1a1-d99b-4e9c-98d9-4cbd1327eff8
 
 ## Using your data
 
-You can supply your own data; the app will index it and answer questions. Your generated app will have a folder called `data` (If you're using Express or Python and generate a frontend, it will be `./backend/data`).
+Optionally, you can supply your own data; the app will index it and make use of it, e.g. to answer questions. Your generated app will have a folder called `data` (If you're using Express or Python and generate a frontend, it will be `./backend/data`).
 
-The app will ingest any supported files you put in this directory. Your Next.js and Express apps use LlamaIndex.TS so they will be able to ingest any PDF, text, CSV, Markdown, Word and HTML files. The Python backend can read even more types, including video and audio files.
+The app will ingest any supported files you put in this directory. Your Next.js and Express apps use LlamaIndex.TS, so they will be able to ingest any PDF, text, CSV, Markdown, Word and HTML files. The Python backend can read even more types, including video and audio files.
 
 Before you can use your data, you need to index it. If you're using the Next.js or Express apps, run:
 
@@ -58,10 +58,6 @@ If you're using the Python backend, you can trigger indexing of your data by cal
 poetry run generate
 ```
 
-## Want a front-end?
-
-Optionally generate a frontend if you've selected the Python or Express back-ends. If you do so, `create-llama` will generate two folders: `frontend`, for your Next.js-based frontend code, and `backend` containing your API.
-
 ## Customizing the AI models
 
 The app will default to OpenAI's `gpt-4o-mini` LLM and `text-embedding-3-large` embedding model.
@@ -94,46 +90,40 @@ Need to install the following packages:
   create-llama@latest
 Ok to proceed? (y) y
 ✔ What is your project named? … my-app
-✔ Which template would you like to use? › Agentic RAG (e.g. chat with docs)
-✔ Which framework would you like to use? › NextJS
-✔ Would you like to set up observability? › No
+✔ What app do you want to build? › Agentic RAG
+✔ What language do you want to use? › Python (FastAPI)
+✔ Do you want to use LlamaCloud services? … No / Yes
+✔ Please provide your LlamaCloud API key (leave blank to skip): …
 ✔ Please provide your OpenAI API key (leave blank to skip): …
-✔ Which data source would you like to use? › Use an example PDF
-✔ Would you like to add another data source? › No
-✔ Would you like to use LlamaParse (improved parser for RAG - requires API key)? … no / yes
-✔ Would you like to use a vector database? › No, just store the data in the file system
-✔ Would you like to build an agent using tools? If so, select the tools here, otherwise just press enter › Weather
 ? How would you like to proceed? › - Use arrow-keys. Return to submit.
-   Just generate code (~1 sec)
-❯  Start in VSCode (~1 sec)
-   Generate code and install dependencies (~2 min)
-   Generate code, install dependencies, and run the app (~2 min)
+    Just generate code (~1 sec)
+❯   Start in VSCode (~1 sec)
+    Generate code and install dependencies (~2 min)
 ```
 
 ### Running non-interactively
 
 You can also pass command line arguments to set up a new project
-non-interactively. See `create-llama --help`:
+non-interactively. For a list of the latest options, call `create-llama --help`.
 
-```bash
-create-llama <project-directory> [options]
+### Running in pro mode
 
-Options:
-  -V, --version                      output the version number
+If you prefer more advanced customization options, you can run `create-llama` in pro mode using the `--pro` flag.
 
-  --use-npm
+In pro mode, instead of selecting a predefined use case, you'll be prompted to select each technical component of your project. This allows for greater flexibility in customizing your project, including:
 
-    Explicitly tell the CLI to bootstrap the app using npm
+- **Vector Store**: Choose from a variety of vector stores for keeping your documents, including MongoDB, Pinecone, Weaviate, Qdrant and Chroma.
+- **Tools**: Choose from a variety of agent tools (functions called by the LLM), such as:
+  - Code Interpreter: Executes Python code in a secure Jupyter notebook environment
+  - Artifact Code Generator: Generates code artifacts that can be run in a sandbox
+  - OpenAPI Action: Facilitates requests to a provided OpenAPI schema
+  - Image Generator: Creates images based on text descriptions
+  - Web Search: Performs web searches to retrieve up-to-date information
+- **Data Sources**: Integrate various data sources into your chat application, including local files, websites, or database-retrieved data.
+- **Backend Options**: Besides using Next.js or FastAPI, you can also select to use Express for a more traditional Node.js application.
+- **Observability**: Choose from a variety of LLM observability tools, including LlamaTrace and Traceloop.
 
-  --use-pnpm
-
-    Explicitly tell the CLI to bootstrap the app using pnpm
-
-  --use-yarn
-
-    Explicitly tell the CLI to bootstrap the app using Yarn
-
-```
+Pro mode is ideal for developers who want fine-grained control over their project's configuration and are comfortable with more technical setup options.
 
 ## LlamaIndex Documentation
 

create-app.ts

@@ -7,7 +7,6 @@ import { getOnline } from "./helpers/is-online";
 import { isWriteable } from "./helpers/is-writeable";
 import { makeDir } from "./helpers/make-dir";
 
-import fs from "fs";
 import terminalLink from "terminal-link";
 import type { InstallTemplateArgs, TemplateObservability } from "./helpers";
 import { installTemplate } from "./helpers";
@@ -17,7 +16,7 @@ import { toolsRequireConfig } from "./helpers/tools";
 
 export type InstallAppArgs = Omit<
   InstallTemplateArgs,
-  "appName" | "root" | "isOnline" | "customApiPath"
+  "appName" | "root" | "isOnline" | "port"
 > & {
   appPath: string;
   frontend: boolean;
@@ -35,12 +34,12 @@ export async function createApp({
   communityProjectConfig,
   llamapack,
   vectorDb,
-  externalPort,
   postInstallAction,
   dataSources,
   tools,
   useLlamaParse,
   observability,
+  agents,
 }: InstallAppArgs): Promise<void> {
   const root = path.resolve(appPath);
 
@@ -80,36 +79,27 @@ export async function createApp({
     communityProjectConfig,
     llamapack,
     vectorDb,
-    externalPort,
     postInstallAction,
     dataSources,
     tools,
     useLlamaParse,
     observability,
+    agents,
   };
 
-  if (frontend) {
-    // install backend
-    const backendRoot = path.join(root, "backend");
-    await makeDir(backendRoot);
-    await installTemplate({ ...args, root: backendRoot, backend: true });
+  // Install backend
+  await installTemplate({ ...args, backend: true });
+
+  if (frontend && framework === "fastapi") {
     // install frontend
-    const frontendRoot = path.join(root, "frontend");
+    const frontendRoot = path.join(root, ".frontend");
     await makeDir(frontendRoot);
     await installTemplate({
       ...args,
       root: frontendRoot,
       framework: "nextjs",
-      customApiPath: `http://localhost:${externalPort ?? 8000}/api/chat`,
       backend: false,
     });
-    // copy readme for fullstack
-    await fs.promises.copyFile(
-      path.join(templatesDir, "README-fullstack.md"),
-      path.join(root, "README.md"),
-    );
-  } else {
-    await installTemplate({ ...args, backend: true });
   }
 
   await writeDevcontainer(root, templatesDir, framework, frontend);

e2e/python/resolve_dependencies.spec.ts

@@ -63,7 +63,6 @@ if (
             vectorDb,
             tools: "none",
             port: 3000,
-            externalPort: 8000,
             postInstallAction: "none",
             templateUI: undefined,
             appType: "--no-frontend",
@@ -101,7 +100,6 @@ if (
             vectorDb: "none",
             tools: tool,
             port: 3000,
-            externalPort: 8000,
             postInstallAction: "none",
             templateUI: undefined,
             appType: "--no-frontend",
@@ -135,7 +133,6 @@ if (
             vectorDb: "none",
             tools: "none",
             port: 3000,
-            externalPort: 8000,
             postInstallAction: "none",
             templateUI: undefined,
             appType: "--no-frontend",
@@ -169,7 +166,6 @@ if (
             vectorDb: "none",
             tools: "none",
             port: 3000,
-            externalPort: 8000,
             postInstallAction: "none",
             templateUI: undefined,
             appType: "--no-frontend",

e2e/shared/extractor_template.spec.ts

@@ -20,8 +20,7 @@ if (
   dataSource === "--example-file"
 ) {
   test.describe("Test extractor template", async () => {
-    let frontendPort: number;
-    let backendPort: number;
+    let appPort: number;
     let name: string;
     let appProcess: ChildProcess;
     let cwd: string;
@@ -29,16 +28,14 @@ if (
     // Create extractor app
     test.beforeAll(async () => {
       cwd = await createTestDir();
-      frontendPort = Math.floor(Math.random() * 10000) + 10000;
-      backendPort = frontendPort + 1;
+      appPort = Math.floor(Math.random() * 10000) + 10000;
       const result = await runCreateLlama({
         cwd,
         templateType: "extractor",
         templateFramework: "fastapi",
         dataSource: "--example-file",
         vectorDb: "none",
-        port: frontendPort,
-        externalPort: backendPort,
+        port: appPort,
         postInstallAction: "runApp",
       });
       name = result.projectName;
@@ -54,7 +51,7 @@ if (
       expect(dirExists).toBeTruthy();
     });
     test("Frontend should have a title", async ({ page }) => {
-      await page.goto(`http://localhost:${frontendPort}`);
+      await page.goto(`http://localhost:${appPort}`);
       await expect(page.getByText("Built by LlamaIndex")).toBeVisible({
         timeout: 2000 * 60,
       });

e2e/shared/multiagent_template.spec.ts

@@ -16,70 +16,82 @@ const templateFramework: TemplateFramework = process.env.FRAMEWORK
 const dataSource: string = "--example-file";
 const templateUI: TemplateUI = "shadcn";
 const templatePostInstallAction: TemplatePostInstallAction = "runApp";
-const appType: AppType = templateFramework === "nextjs" ? "" : "--frontend";
+const appType: AppType = templateFramework === "fastapi" ? "--frontend" : "";
 const userMessage = "Write a blog post about physical standards for letters";
+const templateAgents = ["financial_report", "blog", "form_filling"];
 
-test.describe(`Test multiagent template ${templateFramework} ${dataSource} ${templateUI} ${appType} ${templatePostInstallAction}`, async () => {
-  test.skip(
-    process.platform !== "linux" || process.env.DATASOURCE === "--no-files",
-    "The multiagent template currently only works with files. We also only run on Linux to speed up tests.",
-  );
-  let port: number;
-  let externalPort: number;
-  let cwd: string;
-  let name: string;
-  let appProcess: ChildProcess;
-  // Only test without using vector db for now
-  const vectorDb = "none";
-
-  test.beforeAll(async () => {
-    port = Math.floor(Math.random() * 10000) + 10000;
-    externalPort = port + 1;
-    cwd = await createTestDir();
-    const result = await runCreateLlama({
-      cwd,
-      templateType: "multiagent",
-      templateFramework,
-      dataSource,
-      vectorDb,
-      port,
-      externalPort,
-      postInstallAction: templatePostInstallAction,
-      templateUI,
-      appType,
-    });
-    name = result.projectName;
-    appProcess = result.appProcess;
-  });
-
-  test("App folder should exist", async () => {
-    const dirExists = fs.existsSync(path.join(cwd, name));
-    expect(dirExists).toBeTruthy();
-  });
-
-  test("Frontend should have a title", async ({ page }) => {
-    await page.goto(`http://localhost:${port}`);
-    await expect(page.getByText("Built by LlamaIndex")).toBeVisible();
-  });
-
-  test("Frontend should be able to submit a message and receive the start of a streamed response", async ({
-    page,
-  }) => {
-    await page.goto(`http://localhost:${port}`);
-    await page.fill("form textarea", userMessage);
-
-    const responsePromise = page.waitForResponse((res) =>
-      res.url().includes("/api/chat"),
+for (const agents of templateAgents) {
+  test.describe(`Test multiagent template ${agents} ${templateFramework} ${dataSource} ${templateUI} ${appType} ${templatePostInstallAction}`, async () => {
+    test.skip(
+      process.platform !== "linux" || process.env.DATASOURCE === "--no-files",
+      "The multiagent template currently only works with files. We also only run on Linux to speed up tests.",
     );
+    let port: number;
+    let cwd: string;
+    let name: string;
+    let appProcess: ChildProcess;
+    // Only test without using vector db for now
+    const vectorDb = "none";
 
-    await page.click("form button[type=submit]");
+    test.beforeAll(async () => {
+      port = Math.floor(Math.random() * 10000) + 10000;
+      cwd = await createTestDir();
+      const result = await runCreateLlama({
+        cwd,
+        templateType: "multiagent",
+        templateFramework,
+        dataSource,
+        vectorDb,
+        port,
+        postInstallAction: templatePostInstallAction,
+        templateUI,
+        appType,
+        agents,
+      });
+      name = result.projectName;
+      appProcess = result.appProcess;
+    });
 
-    const response = await responsePromise;
-    expect(response.ok()).toBeTruthy();
+    test("App folder should exist", async () => {
+      const dirExists = fs.existsSync(path.join(cwd, name));
+      expect(dirExists).toBeTruthy();
+    });
+
+    test("Frontend should have a title", async ({ page }) => {
+      test.skip(
+        templatePostInstallAction !== "runApp" ||
+          templateFramework === "express",
+      );
+      await page.goto(`http://localhost:${port}`);
+      await expect(page.getByText("Built by LlamaIndex")).toBeVisible();
+    });
+
+    test("Frontend should be able to submit a message and receive the start of a streamed response", async ({
+      page,
+    }) => {
+      test.skip(
+        templatePostInstallAction !== "runApp" ||
+          agents === "financial_report" ||
+          agents === "form_filling" ||
+          templateFramework === "express",
+        "Skip chat tests for financial report and form filling.",
+      );
+      await page.goto(`http://localhost:${port}`);
+      await page.fill("form textarea", userMessage);
+
+      const responsePromise = page.waitForResponse((res) =>
+        res.url().includes("/api/chat"),
+      );
+
+      await page.click("form button[type=submit]");
+
+      const response = await responsePromise;
+      expect(response.ok()).toBeTruthy();
+    });
+
+    // clean processes
+    test.afterAll(async () => {
+      appProcess?.kill();
+    });
   });
-
-  // clean processes
-  test.afterAll(async () => {
-    appProcess?.kill();
-  });
-});
+}

e2e/shared/streaming_template.spec.ts

@@ -22,7 +22,7 @@ const templatePostInstallAction: TemplatePostInstallAction = "runApp";
 const llamaCloudProjectName = "create-llama";
 const llamaCloudIndexName = "e2e-test";
 
-const appType: AppType = templateFramework === "nextjs" ? "" : "--frontend";
+const appType: AppType = templateFramework === "fastapi" ? "--frontend" : "";
 const userMessage =
   dataSource !== "--no-files" ? "Physical standard for letters" : "Hello";
 
@@ -35,7 +35,6 @@ test.describe(`Test streaming template ${templateFramework} ${dataSource} ${temp
   }
 
   let port: number;
-  let externalPort: number;
   let cwd: string;
   let name: string;
   let appProcess: ChildProcess;
@@ -44,7 +43,6 @@ test.describe(`Test streaming template ${templateFramework} ${dataSource} ${temp
 
   test.beforeAll(async () => {
     port = Math.floor(Math.random() * 10000) + 10000;
-    externalPort = port + 1;
     cwd = await createTestDir();
     const result = await runCreateLlama({
       cwd,
@@ -53,7 +51,6 @@ test.describe(`Test streaming template ${templateFramework} ${dataSource} ${temp
       dataSource,
       vectorDb,
       port,
-      externalPort,
       postInstallAction: templatePostInstallAction,
       templateUI,
       appType,
@@ -68,8 +65,11 @@ test.describe(`Test streaming template ${templateFramework} ${dataSource} ${temp
     const dirExists = fs.existsSync(path.join(cwd, name));
     expect(dirExists).toBeTruthy();
   });
+
   test("Frontend should have a title", async ({ page }) => {
-    test.skip(templatePostInstallAction !== "runApp");
+    test.skip(
+      templatePostInstallAction !== "runApp" || templateFramework === "express",
+    );
     await page.goto(`http://localhost:${port}`);
     await expect(page.getByText("Built by LlamaIndex")).toBeVisible();
   });
@@ -77,7 +77,9 @@ test.describe(`Test streaming template ${templateFramework} ${dataSource} ${temp
   test("Frontend should be able to submit a message and receive a response", async ({
     page,
   }) => {
-    test.skip(templatePostInstallAction !== "runApp");
+    test.skip(
+      templatePostInstallAction !== "runApp" || templateFramework === "express",
+    );
     await page.goto(`http://localhost:${port}`);
     await page.fill("form textarea", userMessage);
     const [response] = await Promise.all([
@@ -102,7 +104,7 @@ test.describe(`Test streaming template ${templateFramework} ${dataSource} ${temp
     test.skip(templatePostInstallAction !== "runApp");
     test.skip(templateFramework === "nextjs");
     const response = await request.post(
-      `http://localhost:${externalPort}/api/chat/request`,
+      `http://localhost:${port}/api/chat/request`,
       {
         data: {
           messages: [

e2e/typescript/resolve_dependencies.spec.ts

@@ -56,7 +56,6 @@ test.describe("Test resolve TS dependencies", () => {
       dataSource: dataSource,
       vectorDb: vectorDb,
       port: 3000,
-      externalPort: 8000,
       postInstallAction: "none",
       templateUI: undefined,
       appType: templateFramework === "nextjs" ? "" : "--no-frontend",

e2e/utils.ts

@@ -25,7 +25,6 @@ export type RunCreateLlamaOptions = {
   dataSource: string;
   vectorDb: TemplateVectorDB;
   port: number;
-  externalPort: number;
   postInstallAction: TemplatePostInstallAction;
   templateUI?: TemplateUI;
   appType?: AppType;
@@ -34,6 +33,7 @@ export type RunCreateLlamaOptions = {
   tools?: string;
   useLlamaParse?: boolean;
   observability?: string;
+  agents?: string;
 };
 
 export async function runCreateLlama({
@@ -43,7 +43,6 @@ export async function runCreateLlama({
   dataSource,
   vectorDb,
   port,
-  externalPort,
   postInstallAction,
   templateUI,
   appType,
@@ -52,6 +51,7 @@ export async function runCreateLlama({
   tools,
   useLlamaParse,
   observability,
+  agents,
 }: RunCreateLlamaOptions): Promise<CreateLlamaResult> {
   if (!process.env.OPENAI_API_KEY || !process.env.LLAMA_CLOUD_API_KEY) {
     throw new Error(
@@ -88,21 +88,15 @@ export async function runCreateLlama({
     ...dataSourceArgs,
     "--vector-db",
     vectorDb,
-    "--open-ai-key",
-    process.env.OPENAI_API_KEY,
     "--use-pnpm",
     "--port",
     port,
-    "--external-port",
-    externalPort,
     "--post-install-action",
     postInstallAction,
     "--tools",
     tools ?? "none",
     "--observability",
     "none",
-    "--llama-cloud-key",
-    process.env.LLAMA_CLOUD_API_KEY,
   ];
 
   if (templateUI) {
@@ -119,6 +113,9 @@ export async function runCreateLlama({
   if (observability) {
     commandArgs.push("--observability", observability);
   }
+  if (templateType === "multiagent" && agents) {
+    commandArgs.push("--agents", agents);
+  }
 
   const command = commandArgs.join(" ");
   console.log(`running command '${command}' in ${cwd}`);
@@ -141,12 +138,7 @@ export async function runCreateLlama({
 
   // Wait for app to start
   if (postInstallAction === "runApp") {
-    await checkAppHasStarted(
-      appType === "--frontend",
-      templateFramework,
-      port,
-      externalPort,
-    );
+    await waitPorts([port]);
   } else if (postInstallAction === "dependencies") {
     await waitForProcess(appProcess, 1000 * 60); // wait 1 min for dependencies to be resolved
   } else {
@@ -166,19 +158,6 @@ export async function createTestDir() {
   return cwd;
 }
 
-// eslint-disable-next-line max-params
-async function checkAppHasStarted(
-  frontend: boolean,
-  framework: TemplateFramework,
-  port: number,
-  externalPort: number,
-) {
-  const portsToWait = frontend
-    ? [port, externalPort]
-    : [framework === "nextjs" ? port : externalPort];
-  await waitPorts(portsToWait);
-}
-
 async function waitPorts(ports: number[]): Promise<void> {
   const waitForPort = async (port: number): Promise<void> => {
     await waitPort({

helpers/datasources.ts

@@ -11,6 +11,25 @@ export const EXAMPLE_FILE: TemplateDataSource = {
   },
 };
 
+export const EXAMPLE_10K_SEC_FILES: TemplateDataSource[] = [
+  {
+    type: "file",
+    config: {
+      url: new URL(
+        "https://s2.q4cdn.com/470004039/files/doc_earnings/2023/q4/filing/_10-K-Q4-2023-As-Filed.pdf",
+      ),
+    },
+  },
+  {
+    type: "file",
+    config: {
+      url: new URL(
+        "https://ir.tesla.com/_flysystem/s3/sec/000162828024002390/tsla-20231231-gen.pdf",
+      ),
+    },
+  },
+];
+
 export function getDataSources(
   files?: string,
   exampleFile?: boolean,

helpers/devcontainer.ts

@@ -5,36 +5,21 @@ import { TemplateFramework } from "./types";
 function renderDevcontainerContent(
   templatesDir: string,
   framework: TemplateFramework,
-  frontend: boolean,
 ) {
   const devcontainerJson: any = JSON.parse(
     fs.readFileSync(path.join(templatesDir, "devcontainer.json"), "utf8"),
   );
 
   // Modify postCreateCommand
-  if (frontend) {
-    devcontainerJson.postCreateCommand =
-      framework === "fastapi"
-        ? "cd backend && poetry install && cd ../frontend && npm install"
-        : "cd backend && npm install && cd ../frontend && npm install";
-  } else {
-    devcontainerJson.postCreateCommand =
-      framework === "fastapi" ? "poetry install" : "npm install";
-  }
+  devcontainerJson.postCreateCommand =
+    framework === "fastapi" ? "poetry install" : "npm install";
 
   // Modify containerEnv
   if (framework === "fastapi") {
-    if (frontend) {
-      devcontainerJson.containerEnv = {
-        ...devcontainerJson.containerEnv,
-        PYTHONPATH: "${PYTHONPATH}:${workspaceFolder}/backend",
-      };
-    } else {
-      devcontainerJson.containerEnv = {
-        ...devcontainerJson.containerEnv,
-        PYTHONPATH: "${PYTHONPATH}:${workspaceFolder}",
-      };
-    }
+    devcontainerJson.containerEnv = {
+      ...devcontainerJson.containerEnv,
+      PYTHONPATH: "${PYTHONPATH}:${workspaceFolder}",
+    };
   }
 
   return JSON.stringify(devcontainerJson, null, 2);
@@ -54,7 +39,6 @@ export const writeDevcontainer = async (
   const devcontainerContent = renderDevcontainerContent(
     templatesDir,
     framework,
-    frontend,
   );
   fs.mkdirSync(devcontainerDir);
   await fs.promises.writeFile(

helpers/env-variables.ts

@@ -217,7 +217,13 @@ Otherwise, use CHROMA_HOST and CHROMA_PORT config above`,
         },
       ];
     default:
-      return [];
+      return [
+        {
+          name: "STORAGE_CACHE_DIR",
+          description: "The directory to store the local storage cache.",
+          value: ".cache",
+        },
+      ];
   }
 };
 
@@ -336,6 +342,20 @@ const getModelEnvs = (modelConfig: ModelConfig): EnvVar[] => {
           },
         ]
       : []),
+    ...(modelConfig.provider === "huggingface"
+      ? [
+          {
+            name: "EMBEDDING_BACKEND",
+            description:
+              "The backend to use for the Sentence Transformers embedding model, either 'torch', 'onnx', or 'openvino'. Defaults to 'onnx'.",
+          },
+          {
+            name: "EMBEDDING_TRUST_REMOTE_CODE",
+            description:
+              "Whether to trust remote code for the embedding model, required for some models with custom code.",
+          },
+        ]
+      : []),
     ...(modelConfig.provider === "t-systems"
       ? [
           {
@@ -387,6 +407,13 @@ const getFrameworkEnvs = (
       ],
     );
   }
+  if (framework === "nextjs") {
+    result.push({
+      name: "NEXT_PUBLIC_CHAT_API",
+      description:
+        "The API for the chat endpoint. Set when using a custom backend (e.g. Express). Use full URL like http://localhost:8000/api/chat",
+    });
+  }
   return result;
 };
 
@@ -533,7 +560,7 @@ export const createBackendEnvFile = async (
     | "framework"
     | "dataSources"
     | "template"
-    | "externalPort"
+    | "port"
     | "tools"
     | "observability"
   >,
@@ -550,7 +577,7 @@ export const createBackendEnvFile = async (
     ...getModelEnvs(opts.modelConfig),
     ...getEngineEnvs(),
     ...getVectorDBEnvs(opts.vectorDb, opts.framework),
-    ...getFrameworkEnvs(opts.framework, opts.externalPort),
+    ...getFrameworkEnvs(opts.framework, opts.port),
     ...getToolEnvs(opts.tools),
     ...getTemplateEnvs(opts.template),
     ...getObservabilityEnvs(opts.observability),
@@ -565,18 +592,10 @@ export const createBackendEnvFile = async (
 export const createFrontendEnvFile = async (
   root: string,
   opts: {
-    customApiPath?: string;
     vectorDb?: TemplateVectorDB;
   },
 ) => {
   const defaultFrontendEnvs = [
-    {
-      name: "NEXT_PUBLIC_CHAT_API",
-      description: "The backend API for chat endpoint.",
-      value: opts.customApiPath
-        ? opts.customApiPath
-        : "http://localhost:8000/api/chat",
-    },
     {
       name: "NEXT_PUBLIC_USE_LLAMACLOUD",
       description: "Let's the user change indexes in LlamaCloud projects",

helpers/index.ts

@@ -96,6 +96,12 @@ async function generateContextData(
   }
 }
 
+const downloadFile = async (url: string, destPath: string) => {
+  const response = await fetch(url);
+  const fileBuffer = await response.arrayBuffer();
+  await fsExtra.writeFile(destPath, Buffer.from(fileBuffer));
+};
+
 const prepareContextData = async (
   root: string,
   dataSources: TemplateDataSource[],
@@ -103,12 +109,28 @@ const prepareContextData = async (
   await makeDir(path.join(root, "data"));
   for (const dataSource of dataSources) {
     const dataSourceConfig = dataSource?.config as FileSourceConfig;
-    // Copy local data
-    const dataPath = dataSourceConfig.path;
-
-    const destPath = path.join(root, "data", path.basename(dataPath));
-    console.log("Copying data from path:", dataPath);
-    await fsExtra.copy(dataPath, destPath);
+    // If the path is URLs, download the data and save it to the data directory
+    if ("url" in dataSourceConfig) {
+      console.log(
+        "Downloading file from URL:",
+        dataSourceConfig.url.toString(),
+      );
+      const destPath = path.join(
+        root,
+        "data",
+        path.basename(dataSourceConfig.url.toString()),
+      );
+      await downloadFile(dataSourceConfig.url.toString(), destPath);
+    } else {
+      // Copy local data
+      console.log("Copying data from path:", dataSourceConfig.path);
+      const destPath = path.join(
+        root,
+        "data",
+        path.basename(dataSourceConfig.path),
+      );
+      await fsExtra.copy(dataSourceConfig.path, destPath);
+    }
   }
 };
 
@@ -203,7 +225,6 @@ export const installTemplate = async (
   } else {
     // this is a frontend for a full-stack app, create .env file with model information
     await createFrontendEnvFile(props.root, {
-      customApiPath: props.customApiPath,
       vectorDb: props.vectorDb,
     });
   }

helpers/providers/anthropic.ts

@@ -1,4 +1,3 @@
-import ciInfo from "ci-info";
 import prompts from "prompts";
 import { ModelConfigParams } from ".";
 import { questionHandlers, toChoice } from "../../questions/utils";
@@ -70,9 +69,7 @@ export async function askAnthropicQuestions({
     config.apiKey = key || process.env.ANTHROPIC_API_KEY;
   }
 
-  // use default model values in CI or if user should not be asked
-  const useDefaults = ciInfo.isCI || !askModels;
-  if (!useDefaults) {
+  if (askModels) {
     const { model } = await prompts(
       {
         type: "select",

helpers/providers/azure.ts

@@ -1,4 +1,3 @@
-import ciInfo from "ci-info";
 import prompts from "prompts";
 import { ModelConfigParams, ModelConfigQuestionsParams } from ".";
 import { questionHandlers } from "../../questions/utils";
@@ -67,9 +66,7 @@ export async function askAzureQuestions({
     },
   };
 
-  // use default model values in CI or if user should not be asked
-  const useDefaults = ciInfo.isCI || !askModels;
-  if (!useDefaults) {
+  if (askModels) {
     const { model } = await prompts(
       {
         type: "select",

helpers/providers/gemini.ts

@@ -1,4 +1,3 @@
-import ciInfo from "ci-info";
 import prompts from "prompts";
 import { ModelConfigParams } from ".";
 import { questionHandlers, toChoice } from "../../questions/utils";
@@ -54,9 +53,7 @@ export async function askGeminiQuestions({
     config.apiKey = key || process.env.GOOGLE_API_KEY;
   }
 
-  // use default model values in CI or if user should not be asked
-  const useDefaults = ciInfo.isCI || !askModels;
-  if (!useDefaults) {
+  if (askModels) {
     const { model } = await prompts(
       {
         type: "select",

helpers/providers/groq.ts

@@ -1,4 +1,3 @@
-import ciInfo from "ci-info";
 import prompts from "prompts";
 import { ModelConfigParams } from ".";
 import { questionHandlers, toChoice } from "../../questions/utils";
@@ -110,9 +109,7 @@ export async function askGroqQuestions({
     config.apiKey = key || process.env.GROQ_API_KEY;
   }
 
-  // use default model values in CI or if user should not be asked
-  const useDefaults = ciInfo.isCI || !askModels;
-  if (!useDefaults) {
+  if (askModels) {
     const modelChoices = await getAvailableModelChoicesGroq(config.apiKey!);
 
     const { model } = await prompts(

helpers/providers/huggingface.ts

@@ -0,0 +1,68 @@
+import prompts from "prompts";
+import { ModelConfigParams } from ".";
+import { questionHandlers, toChoice } from "../../questions/utils";
+
+const MODELS = ["HuggingFaceH4/zephyr-7b-alpha"];
+type ModelData = {
+  dimensions: number;
+};
+const EMBEDDING_MODELS: Record<string, ModelData> = {
+  "BAAI/bge-small-en-v1.5": { dimensions: 384 },
+  "BAAI/bge-base-en-v1.5": { dimensions: 768 },
+  "BAAI/bge-large-en-v1.5": { dimensions: 1024 },
+  "sentence-transformers/all-MiniLM-L6-v2": { dimensions: 384 },
+  "sentence-transformers/all-mpnet-base-v2": { dimensions: 768 },
+  "intfloat/multilingual-e5-large": { dimensions: 1024 },
+  "mixedbread-ai/mxbai-embed-large-v1": { dimensions: 1024 },
+  "nomic-ai/nomic-embed-text-v1.5": { dimensions: 768 },
+};
+
+const DEFAULT_MODEL = MODELS[0];
+const DEFAULT_EMBEDDING_MODEL = Object.keys(EMBEDDING_MODELS)[0];
+const DEFAULT_DIMENSIONS = Object.values(EMBEDDING_MODELS)[0].dimensions;
+
+type HuggingfaceQuestionsParams = {
+  askModels: boolean;
+};
+
+export async function askHuggingfaceQuestions({
+  askModels,
+}: HuggingfaceQuestionsParams): Promise<ModelConfigParams> {
+  const config: ModelConfigParams = {
+    model: DEFAULT_MODEL,
+    embeddingModel: DEFAULT_EMBEDDING_MODEL,
+    dimensions: DEFAULT_DIMENSIONS,
+    isConfigured(): boolean {
+      return true;
+    },
+  };
+
+  if (askModels) {
+    const { model } = await prompts(
+      {
+        type: "select",
+        name: "model",
+        message: "Which Hugging Face model would you like to use?",
+        choices: MODELS.map(toChoice),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.model = model;
+
+    const { embeddingModel } = await prompts(
+      {
+        type: "select",
+        name: "embeddingModel",
+        message: "Which embedding model would you like to use?",
+        choices: Object.keys(EMBEDDING_MODELS).map(toChoice),
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    config.embeddingModel = embeddingModel;
+    config.dimensions = EMBEDDING_MODELS[embeddingModel].dimensions;
+  }
+
+  return config;
+}

helpers/providers/index.ts

@@ -5,6 +5,7 @@ import { askAnthropicQuestions } from "./anthropic";
 import { askAzureQuestions } from "./azure";
 import { askGeminiQuestions } from "./gemini";
 import { askGroqQuestions } from "./groq";
+import { askHuggingfaceQuestions } from "./huggingface";
 import { askLLMHubQuestions } from "./llmhub";
 import { askMistralQuestions } from "./mistral";
 import { askOllamaQuestions } from "./ollama";
@@ -39,6 +40,7 @@ export async function askModelConfig({
 
     if (framework === "fastapi") {
       choices.push({ title: "T-Systems", value: "t-systems" });
+      choices.push({ title: "Huggingface", value: "huggingface" });
     }
     const { provider } = await prompts(
       {
@@ -76,6 +78,9 @@ export async function askModelConfig({
     case "t-systems":
       modelConfig = await askLLMHubQuestions({ askModels });
       break;
+    case "huggingface":
+      modelConfig = await askHuggingfaceQuestions({ askModels });
+      break;
     default:
       modelConfig = await askOpenAIQuestions({
         openAiKey,

helpers/providers/llmhub.ts

@@ -1,4 +1,3 @@
-import ciInfo from "ci-info";
 import got from "got";
 import ora from "ora";
 import { red } from "picocolors";
@@ -80,9 +79,7 @@ export async function askLLMHubQuestions({
     config.apiKey = key || process.env.T_SYSTEMS_LLMHUB_API_KEY;
   }
 
-  // use default model values in CI or if user should not be asked
-  const useDefaults = ciInfo.isCI || !askModels;
-  if (!useDefaults) {
+  if (askModels) {
     const { model } = await prompts(
       {
         type: "select",

helpers/providers/mistral.ts

@@ -1,4 +1,3 @@
-import ciInfo from "ci-info";
 import prompts from "prompts";
 import { ModelConfigParams } from ".";
 import { questionHandlers, toChoice } from "../../questions/utils";
@@ -53,9 +52,7 @@ export async function askMistralQuestions({
     config.apiKey = key || process.env.MISTRAL_API_KEY;
   }
 
-  // use default model values in CI or if user should not be asked
-  const useDefaults = ciInfo.isCI || !askModels;
-  if (!useDefaults) {
+  if (askModels) {
     const { model } = await prompts(
       {
         type: "select",

helpers/providers/ollama.ts

@@ -1,4 +1,3 @@
-import ciInfo from "ci-info";
 import ollama, { type ModelResponse } from "ollama";
 import { red } from "picocolors";
 import prompts from "prompts";
@@ -34,9 +33,7 @@ export async function askOllamaQuestions({
     },
   };
 
-  // use default model values in CI or if user should not be asked
-  const useDefaults = ciInfo.isCI || !askModels;
-  if (!useDefaults) {
+  if (askModels) {
     const { model } = await prompts(
       {
         type: "select",

helpers/providers/openai.ts

@@ -1,9 +1,9 @@
-import ciInfo from "ci-info";
 import got from "got";
 import ora from "ora";
 import { red } from "picocolors";
 import prompts from "prompts";
 import { ModelConfigParams, ModelConfigQuestionsParams } from ".";
+import { isCI } from "../../questions";
 import { questionHandlers } from "../../questions/utils";
 
 const OPENAI_API_URL = "https://api.openai.com/v1";
@@ -31,7 +31,7 @@ export async function askOpenAIQuestions({
     },
   };
 
-  if (!config.apiKey) {
+  if (!config.apiKey && !isCI) {
     const { key } = await prompts(
       {
         type: "text",
@@ -54,9 +54,7 @@ export async function askOpenAIQuestions({
     config.apiKey = key || process.env.OPENAI_API_KEY;
   }
 
-  // use default model values in CI or if user should not be asked
-  const useDefaults = ciInfo.isCI || !askModels;
-  if (!useDefaults) {
+  if (askModels) {
     const { model } = await prompts(
       {
         type: "select",

helpers/python.ts

@@ -20,6 +20,7 @@ interface Dependency {
   name: string;
   version?: string;
   extras?: string[];
+  constraints?: Record<string, string>;
 }
 
 const getAdditionalDependencies = (
@@ -51,6 +52,9 @@ const getAdditionalDependencies = (
       dependencies.push({
         name: "llama-index-vector-stores-pinecone",
         version: "^0.2.1",
+        constraints: {
+          python: ">=3.11,<3.13",
+        },
       });
       break;
     }
@@ -76,6 +80,9 @@ const getAdditionalDependencies = (
       dependencies.push({
         name: "llama-index-vector-stores-qdrant",
         version: "^0.3.0",
+        constraints: {
+          python: ">=3.11,<3.13",
+        },
       });
       break;
     }
@@ -234,6 +241,21 @@ const getAdditionalDependencies = (
         version: "0.2.4",
       });
       break;
+    case "huggingface":
+      dependencies.push({
+        name: "llama-index-llms-huggingface",
+        version: "^0.3.5",
+      });
+      dependencies.push({
+        name: "llama-index-embeddings-huggingface",
+        version: "^0.3.1",
+      });
+      dependencies.push({
+        name: "optimum",
+        version: "^1.23.3",
+        extras: ["onnxruntime"],
+      });
+      break;
     case "t-systems":
       dependencies.push({
         name: "llama-index-agent-openai",
@@ -264,14 +286,19 @@ const mergePoetryDependencies = (
     value.version = dependency.version ?? value.version;
     value.extras = dependency.extras ?? value.extras;
 
+    // Merge constraints if they exist
+    if (dependency.constraints) {
+      value = { ...value, ...dependency.constraints };
+    }
+
     if (value.version === undefined) {
       throw new Error(
         `Dependency "${dependency.name}" is missing attribute "version"!`,
       );
     }
 
-    // Serialize separately only if extras are provided
-    if (value.extras && value.extras.length > 0) {
+    // Serialize as object if there are any additional properties
+    if (Object.keys(value).length > 1) {
       existingDependencies[dependency.name] = value;
     } else {
       // Otherwise, serialize just the version string
@@ -362,6 +389,7 @@ export const installPythonTemplate = async ({
   postInstallAction,
   observability,
   modelConfig,
+  agents,
 }: Pick<
   InstallTemplateArgs,
   | "root"
@@ -373,6 +401,7 @@ export const installPythonTemplate = async ({
   | "postInstallAction"
   | "observability"
   | "modelConfig"
+  | "agents"
 >) => {
   console.log("\nInitializing Python project with template:", template, "\n");
   let templatePath;
@@ -443,6 +472,24 @@ export const installPythonTemplate = async ({
       cwd: path.join(compPath, "engines", "python", engine),
     });
 
+    // Copy agent code
+    if (template === "multiagent") {
+      if (agents) {
+        await copy("**", path.join(root), {
+          parents: true,
+          cwd: path.join(compPath, "agents", "python", agents),
+          rename: assetRelocator,
+        });
+      } else {
+        console.log(
+          red(
+            "There is no agent selected for multi-agent template. Please pick an agent to use via --agents flag.",
+          ),
+        );
+        process.exit(1);
+      }
+    }
+
     // Copy router code
     await copyRouterCode(root, tools ?? []);
   }
@@ -478,6 +525,9 @@ export const installPythonTemplate = async ({
       addOnDependencies.push({
         name: "llama-index-callbacks-arize-phoenix",
         version: "^0.2.1",
+        constraints: {
+          python: ">=3.11,<3.13",
+        },
       });
     }
 

helpers/run-app.ts

@@ -1,40 +1,39 @@
-import { ChildProcess, SpawnOptions, spawn } from "child_process";
-import path from "path";
+import { SpawnOptions, spawn } from "child_process";
 import { TemplateFramework } from "./types";
 
 const createProcess = (
   command: string,
   args: string[],
   options: SpawnOptions,
-) => {
-  return spawn(command, args, {
-    ...options,
-    shell: true,
-  })
-    .on("exit", function (code) {
-      if (code !== 0) {
-        console.log(`Child process exited with code=${code}`);
-        process.exit(1);
-      }
+): Promise<void> => {
+  return new Promise((resolve, reject) => {
+    spawn(command, args, {
+      ...options,
+      shell: true,
     })
-    .on("error", function (err) {
-      console.log("Error when running chill process: ", err);
-      process.exit(1);
-    });
+      .on("exit", function (code) {
+        if (code !== 0) {
+          console.log(`Child process exited with code=${code}`);
+          reject(code);
+        } else {
+          resolve();
+        }
+      })
+      .on("error", function (err) {
+        console.log("Error when running child process: ", err);
+        reject(err);
+      });
+  });
 };
 
-export function runReflexApp(
-  appPath: string,
-  frontendPort?: number,
-  backendPort?: number,
-) {
-  const commandArgs = ["run", "reflex", "run"];
-  if (frontendPort) {
-    commandArgs.push("--frontend-port", frontendPort.toString());
-  }
-  if (backendPort) {
-    commandArgs.push("--backend-port", backendPort.toString());
-  }
+export function runReflexApp(appPath: string, port: number) {
+  const commandArgs = [
+    "run",
+    "reflex",
+    "run",
+    "--frontend-port",
+    port.toString(),
+  ];
   return createProcess("poetry", commandArgs, {
     stdio: "inherit",
     cwd: appPath,
@@ -42,11 +41,10 @@ export function runReflexApp(
 }
 
 export function runFastAPIApp(appPath: string, port: number) {
-  const commandArgs = ["run", "uvicorn", "main:app", "--port=" + port];
-
-  return createProcess("poetry", commandArgs, {
+  return createProcess("poetry", ["run", "dev"], {
     stdio: "inherit",
     cwd: appPath,
+    env: { ...process.env, APP_PORT: `${port}` },
   });
 }
 
@@ -61,39 +59,23 @@ export function runTSApp(appPath: string, port: number) {
 export async function runApp(
   appPath: string,
   template: string,
-  frontend: boolean,
   framework: TemplateFramework,
   port?: number,
-  externalPort?: number,
-): Promise<any> {
-  const processes: ChildProcess[] = [];
+): Promise<void> {
+  try {
+    // Start the app
+    const defaultPort =
+      framework === "nextjs" || template === "extractor" ? 3000 : 8000;
 
-  // Callback to kill all sub processes if the main process is killed
-  process.on("exit", () => {
-    console.log("Killing app processes...");
-    processes.forEach((p) => p.kill());
-  });
-
-  // Default sub app paths
-  const backendPath = path.join(appPath, "backend");
-  const frontendPath = path.join(appPath, "frontend");
-
-  if (template === "extractor") {
-    processes.push(runReflexApp(appPath, port, externalPort));
+    const appRunner =
+      template === "extractor"
+        ? runReflexApp
+        : framework === "fastapi"
+          ? runFastAPIApp
+          : runTSApp;
+    await appRunner(appPath, port || defaultPort);
+  } catch (error) {
+    console.error("Failed to run app:", error);
+    throw error;
   }
-  if (template === "streaming" || template === "multiagent") {
-    if (framework === "fastapi" || framework === "express") {
-      const backendRunner = framework === "fastapi" ? runFastAPIApp : runTSApp;
-      if (frontend) {
-        processes.push(backendRunner(backendPath, externalPort || 8000));
-        processes.push(runTSApp(frontendPath, port || 3000));
-      } else {
-        processes.push(backendRunner(appPath, externalPort || 8000));
-      }
-    } else if (framework === "nextjs") {
-      processes.push(runTSApp(appPath, port || 3000));
-    }
-  }
-
-  return Promise.all(processes);
 }

helpers/tools.ts

@@ -62,7 +62,7 @@ export const supportedTools: Tool[] = [
     dependencies: [
       {
         name: "duckduckgo-search",
-        version: "6.1.7",
+        version: "^6.3.5",
       },
     ],
     supportedFrameworks: ["fastapi", "nextjs", "express"],
@@ -139,7 +139,7 @@ For better results, you can specify the region parameter to get results from a s
     dependencies: [
       {
         name: "e2b_code_interpreter",
-        version: "0.0.10",
+        version: "0.0.11b38",
       },
     ],
     supportedFrameworks: ["fastapi", "express", "nextjs"],
@@ -170,7 +170,7 @@ For better results, you can specify the region parameter to get results from a s
     dependencies: [
       {
         name: "e2b_code_interpreter",
-        version: "^0.0.11b38",
+        version: "0.0.11b38",
       },
     ],
     supportedFrameworks: ["fastapi", "express", "nextjs"],
@@ -267,6 +267,22 @@ For better results, you can specify the region parameter to get results from a s
       },
     ],
   },
+  {
+    display: "Form Filling",
+    name: "form_filling",
+    supportedFrameworks: ["fastapi"],
+    type: ToolType.LOCAL,
+    dependencies: [
+      {
+        name: "pandas",
+        version: "^2.2.3",
+      },
+      {
+        name: "tabulate",
+        version: "^0.9.0",
+      },
+    ],
+  },
 ];
 
 export const getTool = (toolName: string): Tool | undefined => {

helpers/types.ts

@@ -9,6 +9,7 @@ export type ModelProvider =
   | "gemini"
   | "mistral"
   | "azure-openai"
+  | "huggingface"
   | "t-systems";
 export type ModelConfig = {
   provider: ModelProvider;
@@ -48,10 +49,15 @@ export type TemplateDataSource = {
 };
 export type TemplateDataSourceType = "file" | "web" | "db";
 export type TemplateObservability = "none" | "traceloop" | "llamatrace";
+export type TemplateAgents = "financial_report" | "blog" | "form_filling";
 // Config for both file and folder
-export type FileSourceConfig = {
-  path: string;
-};
+export type FileSourceConfig =
+  | {
+      path: string;
+    }
+  | {
+      url: URL;
+    };
 export type WebSourceConfig = {
   baseUrl?: string;
   prefix?: string;
@@ -83,15 +89,15 @@ export interface InstallTemplateArgs {
   framework: TemplateFramework;
   ui: TemplateUI;
   dataSources: TemplateDataSource[];
-  customApiPath?: string;
   modelConfig: ModelConfig;
   llamaCloudKey?: string;
   useLlamaParse?: boolean;
   communityProjectConfig?: CommunityProjectConfig;
   llamapack?: string;
   vectorDb?: TemplateVectorDB;
-  externalPort?: number;
+  port?: number;
   postInstallAction?: TemplatePostInstallAction;
   tools?: Tool[];
   observability?: TemplateObservability;
+  agents?: TemplateAgents;
 }

helpers/typescript.ts

@@ -1,7 +1,7 @@
 import fs from "fs/promises";
 import os from "os";
 import path from "path";
-import { bold, cyan, yellow } from "picocolors";
+import { bold, cyan, red, yellow } from "picocolors";
 import { assetRelocator, copy } from "../helpers/copy";
 import { callPackageManager } from "../helpers/install";
 import { templatesDir } from "./dir";
@@ -26,6 +26,7 @@ export const installTSTemplate = async ({
   tools,
   dataSources,
   useLlamaParse,
+  agents,
 }: InstallTemplateArgs & { backend: boolean }) => {
   console.log(bold(`Using ${packageManager}.`));
 
@@ -57,11 +58,9 @@ export const installTSTemplate = async ({
       console.log("\nUsing static site generation\n");
     } else {
       if (vectorDb === "milvus") {
-        nextConfigJson.experimental.serverComponentsExternalPackages =
-          nextConfigJson.experimental.serverComponentsExternalPackages ?? [];
-        nextConfigJson.experimental.serverComponentsExternalPackages.push(
-          "@zilliz/milvus2-sdk-node",
-        );
+        nextConfigJson.serverExternalPackages =
+          nextConfigJson.serverExternalPackages ?? [];
+        nextConfigJson.serverExternalPackages.push("@zilliz/milvus2-sdk-node");
       }
     }
     await fs.writeFile(
@@ -132,6 +131,34 @@ export const installTSTemplate = async ({
       cwd: path.join(multiagentPath, "workflow"),
     });
 
+    // Copy agents use case code for multiagent template
+    if (agents) {
+      console.log("\nCopying agent:", agents, "\n");
+      const useCasePath = path.join(compPath, "agents", "typescript", agents);
+      const agentsCodePath = path.join(useCasePath, "workflow");
+
+      // Copy agent codes
+      await copy("**", path.join(root, relativeEngineDestPath, "workflow"), {
+        parents: true,
+        cwd: agentsCodePath,
+        rename: assetRelocator,
+      });
+
+      // Copy use case files to project root
+      await copy("*.*", path.join(root), {
+        parents: true,
+        cwd: useCasePath,
+        rename: assetRelocator,
+      });
+    } else {
+      console.log(
+        red(
+          "There is no agent selected for multi-agent template. Please pick an agent to use via --agents flag.",
+        ),
+      );
+      process.exit(1);
+    }
+
     if (framework === "nextjs") {
       // patch route.ts file
       await copy("**", path.join(root, relativeEngineDestPath), {
@@ -214,7 +241,10 @@ export const installTSTemplate = async ({
     vectorDb,
   });
 
-  if (postInstallAction === "runApp" || postInstallAction === "dependencies") {
+  if (
+    backend &&
+    (postInstallAction === "runApp" || postInstallAction === "dependencies")
+  ) {
     await installTSDependencies(packageJson, packageManager, isOnline);
   }
 
@@ -279,12 +309,7 @@ async function updatePackageJson({
       "remark-gfm": undefined,
       "remark-math": undefined,
       "react-markdown": undefined,
-      "react-syntax-highlighter": undefined,
-    };
-
-    packageJson.devDependencies = {
-      ...packageJson.devDependencies,
-      "@types/react-syntax-highlighter": undefined,
+      "highlight.js": undefined,
     };
   }
 

index.ts

@@ -134,13 +134,6 @@ const program = new Command(packageJson.name)
     `
 
   Select UI port.
-`,
-  )
-  .option(
-    "--external-port <external>",
-    `
-
-  Select external port.
 `,
   )
   .option(
@@ -208,6 +201,13 @@ const program = new Command(packageJson.name)
 `,
     false,
   )
+  .option(
+    "--agents <agents>",
+    `
+
+  Select which agents to use for the multi-agent template (e.g: financial_report, blog).
+`,
+  )
   .allowUnknownOption()
   .parse(process.argv);
 
@@ -326,7 +326,6 @@ async function run(): Promise<void> {
     ...answers,
     appPath: resolvedProjectPath,
     packageManager,
-    externalPort: options.externalPort,
   });
 
   if (answers.postInstallAction === "VSCode") {
@@ -355,14 +354,7 @@ Please check ${cyan(
     }
   } else if (answers.postInstallAction === "runApp") {
     console.log(`Running app in ${root}...`);
-    await runApp(
-      root,
-      answers.template,
-      answers.frontend,
-      answers.framework,
-      options.port,
-      options.externalPort,
-    );
+    await runApp(root, answers.template, answers.framework, options.port);
   }
 }
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "create-llama",
-  "version": "0.2.19",
+  "version": "0.3.15",
   "description": "Create LlamaIndex-powered apps with one command",
   "keywords": [
     "rag",

questions/index.ts

@@ -4,10 +4,12 @@ import { askProQuestions } from "./questions";
 import { askSimpleQuestions } from "./simple";
 import { QuestionArgs, QuestionResults } from "./types";
 
+export const isCI = ciInfo.isCI || process.env.PLAYWRIGHT_TEST === "1";
+
 export const askQuestions = async (
   args: QuestionArgs,
 ): Promise<QuestionResults> => {
-  if (ciInfo.isCI) {
+  if (isCI) {
     return await getCIQuestionResults(args);
   } else if (args.pro) {
     // TODO: refactor pro questions to return a result object

questions/questions.ts

@@ -1,5 +1,6 @@
-import { blue, green } from "picocolors";
+import { blue } from "picocolors";
 import prompts from "prompts";
+import { isCI } from ".";
 import { COMMUNITY_OWNER, COMMUNITY_REPO } from "../helpers/constant";
 import { EXAMPLE_FILE } from "../helpers/datasources";
 import { getAvailableLlamapackOptions } from "../helpers/llama-pack";
@@ -88,7 +89,9 @@ export const askProQuestions = async (program: QuestionArgs) => {
       questionHandlers,
     );
     program.llamapack = llamapack;
-    program.postInstallAction = await askPostInstallAction(program);
+    if (!program.postInstallAction) {
+      program.postInstallAction = await askPostInstallAction(program);
+    }
     return; // early return - no further questions needed for llamapack projects
   }
 
@@ -120,24 +123,17 @@ export const askProQuestions = async (program: QuestionArgs) => {
   }
 
   if (
-    (program.framework === "express" || program.framework === "fastapi") &&
+    program.framework === "fastapi" &&
     (program.template === "streaming" || program.template === "multiagent")
   ) {
     // if a backend-only framework is selected, ask whether we should create a frontend
     if (program.frontend === undefined) {
       const styledNextJS = blue("NextJS");
-      const styledBackend = green(
-        program.framework === "express"
-          ? "Express "
-          : program.framework === "fastapi"
-            ? "FastAPI (Python) "
-            : "",
-      );
       const { frontend } = await prompts({
         onState: onPromptState,
         type: "toggle",
         name: "frontend",
-        message: `Would you like to generate a ${styledNextJS} frontend for your ${styledBackend}backend?`,
+        message: `Would you like to generate a ${styledNextJS} frontend for your FastAPI backend?`,
         initial: false,
         active: "Yes",
         inactive: "No",
@@ -175,6 +171,34 @@ export const askProQuestions = async (program: QuestionArgs) => {
     program.observability = observability;
   }
 
+  // Ask agents
+  if (program.template === "multiagent" && !program.agents) {
+    const { agents } = await prompts(
+      {
+        type: "select",
+        name: "agents",
+        message: "Which agents would you like to use?",
+        choices: [
+          {
+            title: "Financial report (generate a financial report)",
+            value: "financial_report",
+          },
+          {
+            title: "Form filling (fill missing value in a CSV file)",
+            value: "form_filling",
+          },
+          {
+            title: "Blog writer (Write a blog post)",
+            value: "blog_writer",
+          },
+        ],
+        initial: 0,
+      },
+      questionHandlers,
+    );
+    program.agents = agents;
+  }
+
   if (!program.modelConfig) {
     const modelConfig = await askModelConfig({
       openAiKey: program.openAiKey,
@@ -356,7 +380,7 @@ export const askProQuestions = async (program: QuestionArgs) => {
 
   // Ask for LlamaCloud API key when using a LlamaCloud index or LlamaParse
   if (isUsingLlamaCloud || program.useLlamaParse) {
-    if (!program.llamaCloudKey) {
+    if (!program.llamaCloudKey && !isCI) {
       // if already set, don't ask again
       // Ask for LlamaCloud API key
       const { llamaCloudKey } = await prompts(
@@ -396,5 +420,7 @@ export const askProQuestions = async (program: QuestionArgs) => {
     program.tools = tools;
   }
 
-  program.postInstallAction = await askPostInstallAction(program);
+  if (!program.postInstallAction) {
+    program.postInstallAction = await askPostInstallAction(program);
+  }
 };

questions/simple.ts

@@ -1,18 +1,24 @@
 import prompts from "prompts";
-import { EXAMPLE_FILE } from "../helpers/datasources";
+import { EXAMPLE_10K_SEC_FILES, EXAMPLE_FILE } from "../helpers/datasources";
 import { askModelConfig } from "../helpers/providers";
 import { getTools } from "../helpers/tools";
 import { ModelConfig, TemplateFramework } from "../helpers/types";
 import { PureQuestionArgs, QuestionResults } from "./types";
 import { askPostInstallAction, questionHandlers } from "./utils";
-type AppType = "rag" | "code_artifact" | "multiagent" | "extractor";
+
+type AppType =
+  | "rag"
+  | "code_artifact"
+  | "financial_report_agent"
+  | "form_filling"
+  | "extractor"
+  | "data_scientist";
 
 type SimpleAnswers = {
   appType: AppType;
   language: TemplateFramework;
   useLlamaCloud: boolean;
   llamaCloudKey?: string;
-  modelConfig: ModelConfig;
 };
 
 export const askSimpleQuestions = async (
@@ -25,17 +31,28 @@ export const askSimpleQuestions = async (
       message: "What app do you want to build?",
       choices: [
         { title: "Agentic RAG", value: "rag" },
+        { title: "Data Scientist", value: "data_scientist" },
+        {
+          title: "Financial Report Generator (using Workflows)",
+          value: "financial_report_agent",
+        },
+        {
+          title: "Form Filler (using Workflows)",
+          value: "form_filling",
+        },
         { title: "Code Artifact Agent", value: "code_artifact" },
-        { title: "Multi-Agent Report Gen", value: "multiagent" },
-        { title: "Structured extraction", value: "extractor" },
+        { title: "Information Extractor", value: "extractor" },
       ],
     },
     questionHandlers,
   );
 
   let language: TemplateFramework = "fastapi";
+  let llamaCloudKey = args.llamaCloudKey;
+  let useLlamaCloud = false;
+
   if (appType !== "extractor") {
-    const res = await prompts(
+    const { language: newLanguage } = await prompts(
       {
         type: "select",
         name: "language",
@@ -47,59 +64,70 @@ export const askSimpleQuestions = async (
       },
       questionHandlers,
     );
-    language = res.language;
-  }
+    language = newLanguage;
 
-  const { useLlamaCloud } = await prompts(
-    {
-      type: "toggle",
-      name: "useLlamaCloud",
-      message: "Do you want to use LlamaCloud services?",
-      initial: false,
-      active: "Yes",
-      inactive: "No",
-      hint: "see https://www.llamaindex.ai/enterprise for more info",
-    },
-    questionHandlers,
-  );
-
-  let llamaCloudKey = args.llamaCloudKey;
-  if (useLlamaCloud && !llamaCloudKey) {
-    // Ask for LlamaCloud API key, if not set
-    const { llamaCloudKey: newLlamaCloudKey } = await prompts(
+    const { useLlamaCloud: newUseLlamaCloud } = await prompts(
       {
-        type: "text",
-        name: "llamaCloudKey",
-        message:
-          "Please provide your LlamaCloud API key (leave blank to skip):",
+        type: "toggle",
+        name: "useLlamaCloud",
+        message: "Do you want to use LlamaCloud services?",
+        initial: false,
+        active: "Yes",
+        inactive: "No",
+        hint: "see https://www.llamaindex.ai/enterprise for more info",
       },
       questionHandlers,
     );
-    llamaCloudKey = newLlamaCloudKey || process.env.LLAMA_CLOUD_API_KEY;
+    useLlamaCloud = newUseLlamaCloud;
+
+    if (useLlamaCloud && !llamaCloudKey) {
+      // Ask for LlamaCloud API key, if not set
+      const { llamaCloudKey: newLlamaCloudKey } = await prompts(
+        {
+          type: "text",
+          name: "llamaCloudKey",
+          message:
+            "Please provide your LlamaCloud API key (leave blank to skip):",
+        },
+        questionHandlers,
+      );
+      llamaCloudKey = newLlamaCloudKey || process.env.LLAMA_CLOUD_API_KEY;
+    }
   }
 
-  const modelConfig = await askModelConfig({
-    openAiKey: args.openAiKey,
-    askModels: args.askModels ?? false,
-    framework: language,
-  });
-
-  const results = convertAnswers({
+  const results = await convertAnswers(args, {
     appType,
     language,
     useLlamaCloud,
     llamaCloudKey,
-    modelConfig,
   });
 
   results.postInstallAction = await askPostInstallAction(results);
   return results;
 };
 
-const convertAnswers = (answers: SimpleAnswers): QuestionResults => {
+const convertAnswers = async (
+  args: PureQuestionArgs,
+  answers: SimpleAnswers,
+): Promise<QuestionResults> => {
+  const MODEL_GPT4o: ModelConfig = {
+    provider: "openai",
+    apiKey: args.openAiKey,
+    model: "gpt-4o",
+    embeddingModel: "text-embedding-3-large",
+    dimensions: 1536,
+    isConfigured(): boolean {
+      return !!args.openAiKey;
+    },
+  };
   const lookup: Record<
     AppType,
-    Pick<QuestionResults, "template" | "tools" | "frontend" | "dataSources">
+    Pick<
+      QuestionResults,
+      "template" | "tools" | "frontend" | "dataSources" | "agents"
+    > & {
+      modelConfig?: ModelConfig;
+    }
   > = {
     rag: {
       template: "streaming",
@@ -107,22 +135,35 @@ const convertAnswers = (answers: SimpleAnswers): QuestionResults => {
       frontend: true,
       dataSources: [EXAMPLE_FILE],
     },
+    data_scientist: {
+      template: "streaming",
+      tools: getTools(["interpreter", "document_generator"]),
+      frontend: true,
+      dataSources: [],
+      modelConfig: MODEL_GPT4o,
+    },
     code_artifact: {
       template: "streaming",
       tools: getTools(["artifact"]),
       frontend: true,
       dataSources: [],
+      modelConfig: MODEL_GPT4o,
     },
-    multiagent: {
+    financial_report_agent: {
       template: "multiagent",
-      tools: getTools([
-        "document_generator",
-        "wikipedia.WikipediaToolSpec",
-        "duckduckgo",
-        "img_gen",
-      ]),
+      agents: "financial_report",
+      tools: getTools(["document_generator", "interpreter"]),
+      dataSources: EXAMPLE_10K_SEC_FILES,
       frontend: true,
-      dataSources: [EXAMPLE_FILE],
+      modelConfig: MODEL_GPT4o,
+    },
+    form_filling: {
+      template: "multiagent",
+      agents: "form_filling",
+      tools: getTools(["form_filling"]),
+      dataSources: EXAMPLE_10K_SEC_FILES,
+      frontend: true,
+      modelConfig: MODEL_GPT4o,
     },
     extractor: {
       template: "extractor",
@@ -138,11 +179,16 @@ const convertAnswers = (answers: SimpleAnswers): QuestionResults => {
     llamaCloudKey: answers.llamaCloudKey,
     useLlamaParse: answers.useLlamaCloud,
     llamapack: "",
-    postInstallAction: "none",
     vectorDb: answers.useLlamaCloud ? "llamacloud" : "none",
-    modelConfig: answers.modelConfig,
     observability: "none",
     ...results,
+    modelConfig:
+      results.modelConfig ??
+      (await askModelConfig({
+        openAiKey: args.openAiKey,
+        askModels: args.askModels ?? false,
+        framework: answers.language,
+      })),
     frontend: answers.language === "nextjs" ? false : results.frontend,
   };
 };

questions/types.ts

@@ -2,7 +2,7 @@ import { InstallAppArgs } from "../create-app";
 
 export type QuestionResults = Omit<
   InstallAppArgs,
-  "appPath" | "packageManager" | "externalPort"
+  "appPath" | "packageManager"
 >;
 
 export type PureQuestionArgs = {

templates/.gitignore

@@ -1,3 +0,0 @@
-__pycache__
-poetry.lock
-storage

templates/README-fullstack.md

@@ -1,18 +0,0 @@
-This is a [LlamaIndex](https://www.llamaindex.ai/) project bootstrapped with [`create-llama`](https://github.com/run-llama/LlamaIndexTS/tree/main/packages/create-llama).
-
-## Getting Started
-
-First, startup the backend as described in the [backend README](./backend/README.md).
-
-Second, run the development server of the frontend as described in the [frontend README](./frontend/README.md).
-
-Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
-
-## Learn More
-
-To learn more about LlamaIndex, take a look at the following resources:
-
-- [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex (Python features).
-- [LlamaIndexTS Documentation](https://ts.llamaindex.ai) - learn about LlamaIndex (Typescript features).
-
-You can check out [the LlamaIndexTS GitHub repository](https://github.com/run-llama/LlamaIndexTS) - your feedback and contributions are welcome!

templates/components/agents/python/blog/README-template.md

@@ -1,5 +1,3 @@
-This is a [LlamaIndex](https://www.llamaindex.ai/) multi-agents project using [Workflows](https://docs.llamaindex.ai/en/stable/understanding/workflows/).
-
 ## Overview
 
 This example is using three agents to generate a blog post:
@@ -10,9 +8,9 @@ This example is using three agents to generate a blog post:
 
 There are three different methods how the agents can interact to reach their goal:
 
-1. [Choreography](./app/examples/choreography.py) - the agents decide themselves to delegate a task to another agent
-1. [Orchestrator](./app/examples/orchestrator.py) - a central orchestrator decides which agent should execute a task
-1. [Explicit Workflow](./app/examples/workflow.py) - a pre-defined workflow specific for the task is used to execute the tasks
+1. [Choreography](./app/agents/choreography.py) - the agents decide themselves to delegate a task to another agent
+1. [Orchestrator](./app/agents/orchestrator.py) - a central orchestrator decides which agent should execute a task
+1. [Explicit Workflow](./app/agents/workflow.py) - a pre-defined workflow specific for the task is used to execute the tasks
 
 ## Getting Started
 
@@ -25,7 +23,6 @@ poetry install
 ```
 
 Then check the parameters that have been pre-configured in the `.env` file in this directory. (E.g. you might need to configure an `OPENAI_API_KEY` if you're using OpenAI as model provider).
-
 Second, generate the embeddings of the documents in the `./data` directory:
 
 ```shell
@@ -35,11 +32,10 @@ poetry run generate
 Third, run the development server:
 
 ```shell
-poetry run python main.py
+poetry run dev
 ```
 
 Per default, the example is using the explicit workflow. You can change the example by setting the `EXAMPLE_TYPE` environment variable to `choreography` or `orchestrator`.
-
 The example provides one streaming API endpoint `/api/chat`.
 You can test the endpoint with the following curl request:
 
@@ -51,12 +47,12 @@ curl --location 'localhost:8000/api/chat' \
 
 You can start editing the API by modifying `app/api/routers/chat.py` or `app/examples/workflow.py`. The API auto-updates as you save the files.
 
-Open [http://localhost:8000/docs](http://localhost:8000/docs) with your browser to see the Swagger UI of the API.
+Open [http://localhost:8000](http://localhost:8000) with your browser to start the app.
 
-The API allows CORS for all origins to simplify development. You can change this behavior by setting the `ENVIRONMENT` environment variable to `prod`:
+To start the app optimized for **production**, run:
 
 ```
-ENVIRONMENT=prod poetry run python main.py
+poetry run prod
 ```
 
 ## Learn More
@@ -65,5 +61,4 @@ To learn more about LlamaIndex, take a look at the following resources:
 
 - [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex.
 - [Workflows Introduction](https://docs.llamaindex.ai/en/stable/understanding/workflows/) - learn about LlamaIndex workflows.
-
-You can check out [the LlamaIndex GitHub repository](https://github.com/run-llama/llama_index) - your feedback and contributions are welcome!
+  You can check out [the LlamaIndex GitHub repository](https://github.com/run-llama/llama_index) - your feedback and contributions are welcome!

templates/components/agents/python/blog/app/agents/choreography.py

@@ -1,10 +1,10 @@
 from textwrap import dedent
 from typing import List, Optional
 
-from app.agents.multi import AgentCallingAgent
-from app.agents.single import FunctionCallingAgent
-from app.examples.publisher import create_publisher
-from app.examples.researcher import create_researcher
+from app.agents.publisher import create_publisher
+from app.agents.researcher import create_researcher
+from app.workflows.multi import AgentCallingAgent
+from app.workflows.single import FunctionCallingAgent
 from llama_index.core.chat_engine.types import ChatMessage
 
 

templates/components/agents/python/blog/app/agents/orchestrator.py

@@ -1,10 +1,10 @@
 from textwrap import dedent
 from typing import List, Optional
 
-from app.agents.multi import AgentOrchestrator
-from app.agents.single import FunctionCallingAgent
-from app.examples.publisher import create_publisher
-from app.examples.researcher import create_researcher
+from app.agents.publisher import create_publisher
+from app.agents.researcher import create_researcher
+from app.workflows.multi import AgentOrchestrator
+from app.workflows.single import FunctionCallingAgent
 from llama_index.core.chat_engine.types import ChatMessage
 
 

templates/components/agents/python/blog/app/agents/publisher.py

@@ -1,8 +1,8 @@
 from textwrap import dedent
 from typing import List, Tuple
 
-from app.agents.single import FunctionCallingAgent
 from app.engine.tools import ToolFactory
+from app.workflows.single import FunctionCallingAgent
 from llama_index.core.chat_engine.types import ChatMessage
 from llama_index.core.tools import FunctionTool
 
@@ -11,11 +11,11 @@ def get_publisher_tools() -> Tuple[List[FunctionTool], str, str]:
     tools = []
     # Get configured tools from the tools.yaml file
     configured_tools = ToolFactory.from_env(map_result=True)
-    if "document_generator" in configured_tools.keys():
-        tools.extend(configured_tools["document_generator"])
+    if "generate_document" in configured_tools.keys():
+        tools.append(configured_tools["generate_document"])
         prompt_instructions = dedent("""
             Normally, reply the blog post content to the user directly. 
-            But if user requested to generate a file, use the document_generator tool to generate the file and reply the link to the file.
+            But if user requested to generate a file, use the generate_document tool to generate the file and reply the link to the file.
         """)
         description = "Expert in publishing the blog post, able to publish the blog post in PDF or HTML format."
     else:

templates/components/agents/python/blog/app/agents/researcher.py

@@ -2,9 +2,9 @@ import os
 from textwrap import dedent
 from typing import List
 
-from app.agents.single import FunctionCallingAgent
 from app.engine.index import IndexConfig, get_index
 from app.engine.tools import ToolFactory
+from app.workflows.single import FunctionCallingAgent
 from llama_index.core.chat_engine.types import ChatMessage
 from llama_index.core.tools import QueryEngineTool, ToolMetadata
 
@@ -42,11 +42,15 @@ def _get_research_tools(**kwargs) -> QueryEngineTool:
     query_engine_tool = _create_query_engine_tool(**kwargs)
     if query_engine_tool is not None:
         tools.append(query_engine_tool)
-    researcher_tool_names = ["duckduckgo", "wikipedia.WikipediaToolSpec"]
+    researcher_tool_names = [
+        "duckduckgo_search",
+        "duckduckgo_image_search",
+        "wikipedia.WikipediaToolSpec",
+    ]
     configured_tools = ToolFactory.from_env(map_result=True)
     for tool_name, tool in configured_tools.items():
         if tool_name in researcher_tool_names:
-            tools.extend(tool)
+            tools.append(tool)
     return tools
 
 

templates/components/agents/python/blog/app/agents/workflow.py

@@ -1,9 +1,9 @@
 from textwrap import dedent
 from typing import AsyncGenerator, List, Optional
 
-from app.agents.single import AgentRunEvent, AgentRunResult, FunctionCallingAgent
-from app.examples.publisher import create_publisher
-from app.examples.researcher import create_researcher
+from app.agents.publisher import create_publisher
+from app.agents.researcher import create_researcher
+from app.workflows.single import AgentRunEvent, AgentRunResult, FunctionCallingAgent
 from llama_index.core.chat_engine.types import ChatMessage
 from llama_index.core.prompts import PromptTemplate
 from llama_index.core.settings import Settings
@@ -238,7 +238,9 @@ class BlogPostWorkflow(Workflow):
         publisher: FunctionCallingAgent,
     ) -> StopEvent:
         try:
-            result: AgentRunResult = await self.run_agent(ctx, publisher, ev.input)
+            result: AgentRunResult = await self.run_agent(
+                ctx, publisher, ev.input, streaming=ctx.data["streaming"]
+            )
             return StopEvent(result=result)
         except Exception as e:
             ctx.write_event_to_stream(

templates/components/agents/python/blog/app/workflows/__init__.py

@@ -0,0 +1,3 @@
+from .blog import create_workflow
+
+__all__ = ["create_workflow"]

templates/components/agents/python/blog/app/workflows/blog.py

@@ -2,19 +2,20 @@ import logging
 import os
 from typing import List, Optional
 
-from app.examples.choreography import create_choreography
-from app.examples.orchestrator import create_orchestrator
-from app.examples.workflow import create_workflow
+from app.agents.choreography import create_choreography
+from app.agents.orchestrator import create_orchestrator
+from app.agents.workflow import create_workflow as create_blog_workflow
 from llama_index.core.chat_engine.types import ChatMessage
 from llama_index.core.workflow import Workflow
 
 logger = logging.getLogger("uvicorn")
 
 
-def get_chat_engine(
+def create_workflow(
     chat_history: Optional[List[ChatMessage]] = None, **kwargs
 ) -> Workflow:
-    # TODO: the EXAMPLE_TYPE could be passed as a chat config parameter?
+    # Chat filters are not supported yet
+    kwargs.pop("filters", None)
     agent_type = os.getenv("EXAMPLE_TYPE", "").lower()
     match agent_type:
         case "choreography":
@@ -22,7 +23,7 @@ def get_chat_engine(
         case "orchestrator":
             agent = create_orchestrator(chat_history, **kwargs)
         case _:
-            agent = create_workflow(chat_history, **kwargs)
+            agent = create_blog_workflow(chat_history, **kwargs)
 
     logger.info(f"Using agent pattern: {agent_type}")
 

templates/components/agents/python/blog/app/workflows/multi.py

@@ -1,7 +1,7 @@
 from typing import Any, List
 
-from app.agents.planner import StructuredPlannerAgent
-from app.agents.single import (
+from app.workflows.planner import StructuredPlannerAgent
+from app.workflows.single import (
     AgentRunResult,
     ContextAwareTool,
     FunctionCallingAgent,

templates/components/agents/python/blog/app/workflows/planner.py

@@ -2,7 +2,7 @@ import uuid
 from enum import Enum
 from typing import Any, AsyncGenerator, Dict, List, Optional, Tuple, Union
 
-from app.agents.single import AgentRunEvent, AgentRunResult, FunctionCallingAgent
+from app.workflows.single import AgentRunEvent, AgentRunResult, FunctionCallingAgent
 from llama_index.core.agent.runner.planner import (
     DEFAULT_INITIAL_PLAN_PROMPT,
     DEFAULT_PLAN_REFINE_PROMPT,

templates/components/agents/python/blog/app/workflows/single.py

@@ -1,4 +1,5 @@
 from abc import abstractmethod
+from enum import Enum
 from typing import Any, AsyncGenerator, List, Optional
 
 from llama_index.core.llms import ChatMessage, ChatResponse
@@ -15,7 +16,7 @@ from llama_index.core.workflow import (
     Workflow,
     step,
 )
-from pydantic import BaseModel
+from pydantic import BaseModel, Field
 
 
 class InputEvent(Event):
@@ -26,17 +27,27 @@ class ToolCallEvent(Event):
     tool_calls: list[ToolSelection]
 
 
+class AgentRunEventType(Enum):
+    TEXT = "text"
+    PROGRESS = "progress"
+
+
 class AgentRunEvent(Event):
     name: str
-    _msg: str
+    msg: str
+    event_type: AgentRunEventType = Field(default=AgentRunEventType.TEXT)
+    data: Optional[dict] = None
 
-    @property
-    def msg(self):
-        return self._msg
-
-    @msg.setter
-    def msg(self, value):
-        self._msg = value
+    def to_response(self) -> dict:
+        return {
+            "type": "agent",
+            "data": {
+                "agent": self.name,
+                "type": self.event_type.value,
+                "text": self.msg,
+                "data": self.data,
+            },
+        }
 
 
 class AgentRunResult(BaseModel):

templates/components/agents/python/financial_report/README-template.md

@@ -0,0 +1,53 @@
+This is a [LlamaIndex](https://www.llamaindex.ai/) multi-agents project using [Workflows](https://docs.llamaindex.ai/en/stable/understanding/workflows/).
+
+## Getting Started
+
+First, setup the environment with poetry:
+
+> **_Note:_** This step is not needed if you are using the dev-container.
+
+```shell
+poetry install
+```
+
+Then check the parameters that have been pre-configured in the `.env` file in this directory. (E.g. you might need to configure an `OPENAI_API_KEY` if you're using OpenAI as model provider and `E2B_API_KEY` for the [E2B's code interpreter tool](https://e2b.dev/docs)).
+
+Second, generate the embeddings of the documents in the `./data` directory:
+
+```shell
+poetry run generate
+```
+
+Third, run the development server:
+
+```shell
+poetry run dev
+```
+
+The example provides one streaming API endpoint `/api/chat`.
+You can test the endpoint with the following curl request:
+
+```
+curl --location 'localhost:8000/api/chat' \
+--header 'Content-Type: application/json' \
+--data '{ "messages": [{ "role": "user", "content": "Create a report comparing the finances of Apple and Tesla" }] }'
+```
+
+You can start editing the API by modifying `app/api/routers/chat.py` or `app/workflows/financial_report.py`. The API auto-updates as you save the files.
+
+Open [http://localhost:8000](http://localhost:8000) with your browser to start the app.
+
+To start the app optimized for **production**, run:
+
+```
+poetry run prod
+```
+
+## Learn More
+
+To learn more about LlamaIndex, take a look at the following resources:
+
+- [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex.
+- [Workflows Introduction](https://docs.llamaindex.ai/en/stable/understanding/workflows/) - learn about LlamaIndex workflows.
+
+You can check out [the LlamaIndex GitHub repository](https://github.com/run-llama/llama_index) - your feedback and contributions are welcome!

templates/components/agents/python/financial_report/app/workflows/__init__.py

@@ -0,0 +1,3 @@
+from .financial_report import create_workflow
+
+__all__ = ["create_workflow"]

templates/components/agents/python/financial_report/app/workflows/financial_report.py

@@ -0,0 +1,298 @@
+import os
+from typing import Any, Dict, List, Optional
+
+from app.engine.index import IndexConfig, get_index
+from app.engine.tools import ToolFactory
+from app.workflows.events import AgentRunEvent
+from app.workflows.tools import (
+    call_tools,
+    chat_with_tools,
+)
+from llama_index.core import Settings
+from llama_index.core.base.llms.types import ChatMessage, MessageRole
+from llama_index.core.indices.vector_store import VectorStoreIndex
+from llama_index.core.llms.function_calling import FunctionCallingLLM
+from llama_index.core.memory import ChatMemoryBuffer
+from llama_index.core.tools import FunctionTool, QueryEngineTool, ToolSelection
+from llama_index.core.workflow import (
+    Context,
+    Event,
+    StartEvent,
+    StopEvent,
+    Workflow,
+    step,
+)
+
+
+def create_workflow(
+    chat_history: Optional[List[ChatMessage]] = None,
+    params: Optional[Dict[str, Any]] = None,
+    filters: Optional[List[Any]] = None,
+) -> Workflow:
+    index_config = IndexConfig(**params)
+    index: VectorStoreIndex = get_index(config=index_config)
+    if index is None:
+        query_engine_tool = None
+    else:
+        top_k = int(os.getenv("TOP_K", 10))
+        query_engine = index.as_query_engine(similarity_top_k=top_k, filters=filters)
+        query_engine_tool = QueryEngineTool.from_defaults(query_engine=query_engine)
+
+    configured_tools: Dict[str, FunctionTool] = ToolFactory.from_env(map_result=True)  # type: ignore
+    code_interpreter_tool = configured_tools.get("interpret")
+    document_generator_tool = configured_tools.get("generate_document")
+
+    return FinancialReportWorkflow(
+        query_engine_tool=query_engine_tool,
+        code_interpreter_tool=code_interpreter_tool,
+        document_generator_tool=document_generator_tool,
+        chat_history=chat_history,
+    )
+
+
+class InputEvent(Event):
+    input: List[ChatMessage]
+    response: bool = False
+
+
+class ResearchEvent(Event):
+    input: list[ToolSelection]
+
+
+class AnalyzeEvent(Event):
+    input: list[ToolSelection] | ChatMessage
+
+
+class ReportEvent(Event):
+    input: list[ToolSelection]
+
+
+class FinancialReportWorkflow(Workflow):
+    """
+    A workflow to generate a financial report using indexed documents.
+
+    Requirements:
+    - Indexed documents containing financial data and a query engine tool to search them
+    - A code interpreter tool to analyze data and generate reports
+    - A document generator tool to create report files
+
+    Steps:
+    1. LLM Input: The LLM determines the next step based on function calling.
+       For example, if the model requests the query engine tool, it returns a ResearchEvent;
+       if it requests document generation, it returns a ReportEvent.
+    2. Research: Uses the query engine to find relevant chunks from indexed documents.
+       After gathering information, it requests analysis (step 3).
+    3. Analyze: Uses a custom prompt to analyze research results and can call the code
+       interpreter tool for visualization or calculation. Returns results to the LLM.
+    4. Report: Uses the document generator tool to create a report. Returns results to the LLM.
+    """
+
+    _default_system_prompt = """
+    You are a financial analyst who are given a set of tools to help you.
+    It's good to using appropriate tools for the user request and always use the information from the tools, don't make up anything yourself.
+    For the query engine tool, you should break down the user request into a list of queries and call the tool with the queries.
+    """
+
+    def __init__(
+        self,
+        query_engine_tool: QueryEngineTool,
+        code_interpreter_tool: FunctionTool,
+        document_generator_tool: FunctionTool,
+        llm: Optional[FunctionCallingLLM] = None,
+        timeout: int = 360,
+        chat_history: Optional[List[ChatMessage]] = None,
+        system_prompt: Optional[str] = None,
+    ):
+        super().__init__(timeout=timeout)
+        self.system_prompt = system_prompt or self._default_system_prompt
+        self.chat_history = chat_history or []
+        self.query_engine_tool = query_engine_tool
+        self.code_interpreter_tool = code_interpreter_tool
+        self.document_generator_tool = document_generator_tool
+        assert (
+            query_engine_tool is not None
+        ), "Query engine tool is not found. Try run generation script or upload a document file first."
+        assert code_interpreter_tool is not None, "Code interpreter tool is required"
+        assert (
+            document_generator_tool is not None
+        ), "Document generator tool is required"
+        self.tools = [
+            self.query_engine_tool,
+            self.code_interpreter_tool,
+            self.document_generator_tool,
+        ]
+        self.llm: FunctionCallingLLM = llm or Settings.llm
+        assert isinstance(self.llm, FunctionCallingLLM)
+        self.memory = ChatMemoryBuffer.from_defaults(
+            llm=self.llm, chat_history=self.chat_history
+        )
+
+    @step()
+    async def prepare_chat_history(self, ctx: Context, ev: StartEvent) -> InputEvent:
+        ctx.data["input"] = ev.input
+
+        if self.system_prompt:
+            system_msg = ChatMessage(
+                role=MessageRole.SYSTEM, content=self.system_prompt
+            )
+            self.memory.put(system_msg)
+
+        # Add user input to memory
+        self.memory.put(ChatMessage(role=MessageRole.USER, content=ev.input))
+
+        return InputEvent(input=self.memory.get())
+
+    @step()
+    async def handle_llm_input(  # type: ignore
+        self,
+        ctx: Context,
+        ev: InputEvent,
+    ) -> ResearchEvent | AnalyzeEvent | ReportEvent | StopEvent:
+        """
+        Handle an LLM input and decide the next step.
+        """
+        # Always use the latest chat history from the input
+        chat_history: list[ChatMessage] = ev.input
+
+        # Get tool calls
+        response = await chat_with_tools(
+            self.llm,
+            self.tools,  # type: ignore
+            chat_history,
+        )
+        if not response.has_tool_calls():
+            # If no tool call, return the response generator
+            return StopEvent(result=response.generator)
+        # calling different tools at the same time is not supported at the moment
+        # add an error message to tell the AI to process step by step
+        if response.is_calling_different_tools():
+            self.memory.put(
+                ChatMessage(
+                    role=MessageRole.ASSISTANT,
+                    content="Cannot call different tools at the same time. Try calling one tool at a time.",
+                )
+            )
+            return InputEvent(input=self.memory.get())
+        self.memory.put(response.tool_call_message)
+        match response.tool_name():
+            case self.code_interpreter_tool.metadata.name:
+                return AnalyzeEvent(input=response.tool_calls)
+            case self.document_generator_tool.metadata.name:
+                return ReportEvent(input=response.tool_calls)
+            case self.query_engine_tool.metadata.name:
+                return ResearchEvent(input=response.tool_calls)
+            case _:
+                raise ValueError(f"Unknown tool: {response.tool_name()}")
+
+    @step()
+    async def research(self, ctx: Context, ev: ResearchEvent) -> AnalyzeEvent:
+        """
+        Do a research to gather information for the user's request.
+        A researcher should have these tools: query engine, search engine, etc.
+        """
+        ctx.write_event_to_stream(
+            AgentRunEvent(
+                name="Researcher",
+                msg="Starting research",
+            )
+        )
+        tool_calls = ev.input
+
+        tool_messages = await call_tools(
+            ctx=ctx,
+            agent_name="Researcher",
+            tools=[self.query_engine_tool],
+            tool_calls=tool_calls,
+        )
+        self.memory.put_messages(tool_messages)
+        return AnalyzeEvent(
+            input=ChatMessage(
+                role=MessageRole.ASSISTANT,
+                content="I've finished the research. Please analyze the result.",
+            ),
+        )
+
+    @step()
+    async def analyze(self, ctx: Context, ev: AnalyzeEvent) -> InputEvent:
+        """
+        Analyze the research result.
+        """
+        ctx.write_event_to_stream(
+            AgentRunEvent(
+                name="Analyst",
+                msg="Starting analysis",
+            )
+        )
+        event_requested_by_workflow_llm = isinstance(ev.input, list)
+        # Requested by the workflow LLM Input step, it's a tool call
+        if event_requested_by_workflow_llm:
+            # Set the tool calls
+            tool_calls = ev.input
+        else:
+            # Otherwise, it's triggered by the research step
+            # Use a custom prompt and independent memory for the analyst agent
+            analysis_prompt = """
+            You are a financial analyst, you are given a research result and a set of tools to help you.
+            Always use the given information, don't make up anything yourself. If there is not enough information, you can asking for more information.
+            If you have enough numerical information, it's good to include some charts/visualizations to the report so you can use the code interpreter tool to generate a report.
+            """
+            # This is handled by analyst agent
+            # Clone the shared memory to avoid conflicting with the workflow.
+            chat_history = self.memory.get()
+            chat_history.append(
+                ChatMessage(role=MessageRole.SYSTEM, content=analysis_prompt)
+            )
+            chat_history.append(ev.input)  # type: ignore
+            # Check if the analyst agent needs to call tools
+            response = await chat_with_tools(
+                self.llm,
+                [self.code_interpreter_tool],
+                chat_history,
+            )
+            if not response.has_tool_calls():
+                # If no tool call, fallback analyst message to the workflow
+                analyst_msg = ChatMessage(
+                    role=MessageRole.ASSISTANT,
+                    content=await response.full_response(),
+                )
+                self.memory.put(analyst_msg)
+                return InputEvent(input=self.memory.get())
+            else:
+                # Set the tool calls and the tool call message to the memory
+                tool_calls = response.tool_calls
+                self.memory.put(response.tool_call_message)
+
+        # Call tools
+        tool_messages = await call_tools(
+            ctx=ctx,
+            agent_name="Analyst",
+            tools=[self.code_interpreter_tool],
+            tool_calls=tool_calls,  # type: ignore
+        )
+        self.memory.put_messages(tool_messages)
+
+        # Fallback to the input with the latest chat history
+        return InputEvent(input=self.memory.get())
+
+    @step()
+    async def report(self, ctx: Context, ev: ReportEvent) -> InputEvent:
+        """
+        Generate a report based on the analysis result.
+        """
+        ctx.write_event_to_stream(
+            AgentRunEvent(
+                name="Reporter",
+                msg="Starting report generation",
+            )
+        )
+        tool_calls = ev.input
+        tool_messages = await call_tools(
+            ctx=ctx,
+            agent_name="Reporter",
+            tools=[self.document_generator_tool],
+            tool_calls=tool_calls,
+        )
+        self.memory.put_messages(tool_messages)
+
+        # After the tool calls, fallback to the input with the latest chat history
+        return InputEvent(input=self.memory.get())

templates/components/agents/python/form_filling/README-template.md

@@ -0,0 +1,59 @@
+This is a [LlamaIndex](https://www.llamaindex.ai/) multi-agents project using [Workflows](https://docs.llamaindex.ai/en/stable/understanding/workflows/).
+
+## Getting Started
+
+First, setup the environment with poetry:
+
+> **_Note:_** This step is not needed if you are using the dev-container.
+
+```shell
+poetry install
+```
+
+Then check the parameters that have been pre-configured in the `.env` file in this directory.
+Make sure you have the `OPENAI_API_KEY` set.
+
+Second, run the development server:
+
+```shell
+poetry run dev
+```
+
+## Use Case: Filling Financial CSV Template
+
+To reproduce the use case, start the [frontend](../frontend/README.md) and follow these steps in the frontend:
+
+1. Upload the Apple and Tesla financial reports from the [data](./data) directory. Just send an empty message.
+2. Upload the CSV file [sec_10k_template.csv](./sec_10k_template.csv) and send the message "Fill the missing cells in the CSV file".
+
+The agent will fill the missing cells by retrieving the information from the uploaded financial reports and return a new CSV file with the filled cells.
+
+### API endpoints
+
+The example provides one streaming API endpoint `/api/chat`.
+You can test the endpoint with the following curl request:
+
+```
+curl --location 'localhost:8000/api/chat' \
+--header 'Content-Type: application/json' \
+--data '{ "messages": [{ "role": "user", "content": "What can you do?" }] }'
+```
+
+You can start editing the API by modifying `app/api/routers/chat.py` or `app/workflows/form_filling.py`. The API auto-updates as you save the files.
+
+Open [http://localhost:8000](http://localhost:8000) with your browser to start the app.
+
+To start the app optimized for **production**, run:
+
+```
+poetry run prod
+```
+
+## Learn More
+
+To learn more about LlamaIndex, take a look at the following resources:
+
+- [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex.
+- [Workflows Introduction](https://docs.llamaindex.ai/en/stable/understanding/workflows/) - learn about LlamaIndex workflows.
+
+You can check out [the LlamaIndex GitHub repository](https://github.com/run-llama/llama_index) - your feedback and contributions are welcome!

templates/components/agents/python/form_filling/app/workflows/__init__.py

@@ -0,0 +1,3 @@
+from .form_filling import create_workflow
+
+__all__ = ["create_workflow"]

templates/components/agents/python/form_filling/app/workflows/form_filling.py

@@ -0,0 +1,241 @@
+import os
+from typing import Any, Dict, List, Optional
+
+from app.engine.index import IndexConfig, get_index
+from app.engine.tools import ToolFactory
+from app.workflows.events import AgentRunEvent
+from app.workflows.tools import (
+    call_tools,
+    chat_with_tools,
+)
+from llama_index.core import Settings
+from llama_index.core.base.llms.types import ChatMessage, MessageRole
+from llama_index.core.indices.vector_store import VectorStoreIndex
+from llama_index.core.llms.function_calling import FunctionCallingLLM
+from llama_index.core.memory import ChatMemoryBuffer
+from llama_index.core.tools import FunctionTool, QueryEngineTool, ToolSelection
+from llama_index.core.workflow import (
+    Context,
+    Event,
+    StartEvent,
+    StopEvent,
+    Workflow,
+    step,
+)
+
+
+def create_workflow(
+    chat_history: Optional[List[ChatMessage]] = None,
+    params: Optional[Dict[str, Any]] = None,
+    filters: Optional[List[Any]] = None,
+) -> Workflow:
+    if params is None:
+        params = {}
+    if filters is None:
+        filters = []
+    index_config = IndexConfig(**params)
+    index: VectorStoreIndex = get_index(config=index_config)
+    if index is None:
+        query_engine_tool = None
+    else:
+        top_k = int(os.getenv("TOP_K", 10))
+        query_engine = index.as_query_engine(similarity_top_k=top_k, filters=filters)
+        query_engine_tool = QueryEngineTool.from_defaults(query_engine=query_engine)
+
+    configured_tools = ToolFactory.from_env(map_result=True)
+    extractor_tool = configured_tools.get("extract_questions")  # type: ignore
+    filling_tool = configured_tools.get("fill_form")  # type: ignore
+
+    workflow = FormFillingWorkflow(
+        query_engine_tool=query_engine_tool,
+        extractor_tool=extractor_tool,  # type: ignore
+        filling_tool=filling_tool,  # type: ignore
+        chat_history=chat_history,
+    )
+
+    return workflow
+
+
+class InputEvent(Event):
+    input: List[ChatMessage]
+    response: bool = False
+
+
+class ExtractMissingCellsEvent(Event):
+    tool_calls: list[ToolSelection]
+
+
+class FindAnswersEvent(Event):
+    tool_calls: list[ToolSelection]
+
+
+class FillEvent(Event):
+    tool_calls: list[ToolSelection]
+
+
+class FormFillingWorkflow(Workflow):
+    """
+    A predefined workflow for filling missing cells in a CSV file.
+    Required tools:
+    - query_engine: A query engine to query for the answers to the questions.
+    - extract_question: Extract missing cells in a CSV file and generate questions to fill them.
+    - answer_question: Query for the answers to the questions.
+
+    Flow:
+    1. Extract missing cells in a CSV file and generate questions to fill them.
+    2. Query for the answers to the questions.
+    3. Fill the missing cells with the answers.
+    """
+
+    _default_system_prompt = """
+    You are a helpful assistant who helps fill missing cells in a CSV file.
+    Only extract missing cells from CSV files.
+    Only use provided data - never make up any information yourself. Fill N/A if an answer is not found.
+    If there is no query engine tool or the gathered information has many N/A values indicating the questions don't match the data, respond with a warning and ask the user to upload a different file or connect to a knowledge base.
+    """
+
+    def __init__(
+        self,
+        query_engine_tool: Optional[QueryEngineTool],
+        extractor_tool: FunctionTool,
+        filling_tool: FunctionTool,
+        llm: Optional[FunctionCallingLLM] = None,
+        timeout: int = 360,
+        chat_history: Optional[List[ChatMessage]] = None,
+        system_prompt: Optional[str] = None,
+    ):
+        super().__init__(timeout=timeout)
+        self.system_prompt = system_prompt or self._default_system_prompt
+        self.chat_history = chat_history or []
+        self.query_engine_tool = query_engine_tool
+        self.extractor_tool = extractor_tool
+        self.filling_tool = filling_tool
+        if self.extractor_tool is None or self.filling_tool is None:
+            raise ValueError("Extractor and filling tools are required.")
+        self.tools = [self.extractor_tool, self.filling_tool]
+        if self.query_engine_tool is not None:
+            self.tools.append(self.query_engine_tool)  # type: ignore
+        self.llm: FunctionCallingLLM = llm or Settings.llm
+        if not isinstance(self.llm, FunctionCallingLLM):
+            raise ValueError("FormFillingWorkflow only supports FunctionCallingLLM.")
+        self.memory = ChatMemoryBuffer.from_defaults(
+            llm=self.llm, chat_history=self.chat_history
+        )
+
+    @step()
+    async def start(self, ctx: Context, ev: StartEvent) -> InputEvent:
+        ctx.data["input"] = ev.input
+
+        if self.system_prompt:
+            system_msg = ChatMessage(
+                role=MessageRole.SYSTEM, content=self.system_prompt
+            )
+            self.memory.put(system_msg)
+
+        user_input = ev.input
+        user_msg = ChatMessage(role=MessageRole.USER, content=user_input)
+        self.memory.put(user_msg)
+
+        chat_history = self.memory.get()
+        return InputEvent(input=chat_history)
+
+    @step()
+    async def handle_llm_input(  # type: ignore
+        self,
+        ctx: Context,
+        ev: InputEvent,
+    ) -> ExtractMissingCellsEvent | FillEvent | StopEvent:
+        """
+        Handle an LLM input and decide the next step.
+        """
+        chat_history: list[ChatMessage] = ev.input
+        response = await chat_with_tools(
+            self.llm,
+            self.tools,
+            chat_history,
+        )
+        if not response.has_tool_calls():
+            return StopEvent(result=response.generator)
+        # calling different tools at the same time is not supported at the moment
+        # add an error message to tell the AI to process step by step
+        if response.is_calling_different_tools():
+            self.memory.put(
+                ChatMessage(
+                    role=MessageRole.ASSISTANT,
+                    content="Cannot call different tools at the same time. Try calling one tool at a time.",
+                )
+            )
+            return InputEvent(input=self.memory.get())
+        self.memory.put(response.tool_call_message)
+        match response.tool_name():
+            case self.extractor_tool.metadata.name:
+                return ExtractMissingCellsEvent(tool_calls=response.tool_calls)
+            case self.query_engine_tool.metadata.name:
+                return FindAnswersEvent(tool_calls=response.tool_calls)
+            case self.filling_tool.metadata.name:
+                return FillEvent(tool_calls=response.tool_calls)
+            case _:
+                raise ValueError(f"Unknown tool: {response.tool_name()}")
+
+    @step()
+    async def extract_missing_cells(
+        self, ctx: Context, ev: ExtractMissingCellsEvent
+    ) -> InputEvent | FindAnswersEvent:
+        """
+        Extract missing cells in a CSV file and generate questions to fill them.
+        """
+        ctx.write_event_to_stream(
+            AgentRunEvent(
+                name="Extractor",
+                msg="Extracting missing cells",
+            )
+        )
+        # Call the extract questions tool
+        tool_messages = await call_tools(
+            agent_name="Extractor",
+            tools=[self.extractor_tool],
+            ctx=ctx,
+            tool_calls=ev.tool_calls,
+        )
+        self.memory.put_messages(tool_messages)
+        return InputEvent(input=self.memory.get())
+
+    @step()
+    async def find_answers(self, ctx: Context, ev: FindAnswersEvent) -> InputEvent:
+        """
+        Call answer questions tool to query for the answers to the questions.
+        """
+        ctx.write_event_to_stream(
+            AgentRunEvent(
+                name="Researcher",
+                msg="Finding answers for missing cells",
+            )
+        )
+        tool_messages = await call_tools(
+            ctx=ctx,
+            agent_name="Researcher",
+            tools=[self.query_engine_tool],
+            tool_calls=ev.tool_calls,
+        )
+        self.memory.put_messages(tool_messages)
+        return InputEvent(input=self.memory.get())
+
+    @step()
+    async def fill_cells(self, ctx: Context, ev: FillEvent) -> InputEvent:
+        """
+        Call fill cells tool to fill the missing cells with the answers.
+        """
+        ctx.write_event_to_stream(
+            AgentRunEvent(
+                name="Processor",
+                msg="Filling missing cells",
+            )
+        )
+        tool_messages = await call_tools(
+            agent_name="Processor",
+            tools=[self.filling_tool],
+            ctx=ctx,
+            tool_calls=ev.tool_calls,
+        )
+        self.memory.put_messages(tool_messages)
+        return InputEvent(input=self.memory.get())

templates/components/agents/python/form_filling/sec_10k_template.csv

@@ -0,0 +1,17 @@
+Parameter,2023 Apple (AAPL),2023 Tesla (TSLA)
+Revenue,,
+Net Income,,
+Earnings Per Share (EPS),,
+Debt-to-Equity Ratio,,
+Current Ratio,,
+Gross Margin,,
+Operating Margin,,
+Net Profit Margin,,
+Inventory Turnover,,
+Accounts Receivable Turnover,,
+Capital Expenditure,,
+Research and Development Expense,,
+Market Cap,,
+Price to Earnings Ratio,,
+Dividend Yield,,
+Year-over-Year Growth Rate,,

templates/components/agents/typescript/blog/workflow/agents.ts

@@ -1,19 +1,16 @@
 import { ChatMessage } from "llamaindex";
+import { getTool } from "../engine/tools";
 import { FunctionCallingAgent } from "./single-agent";
-import { getQueryEngineTool, lookupTools } from "./tools";
+import { getQueryEngineTools } from "./tools";
 
-export const createResearcher = async (
-  chatHistory: ChatMessage[],
-  params?: any,
-) => {
-  const queryEngineTool = await getQueryEngineTool(params);
-  const tools = (
-    await lookupTools([
-      "wikipedia_tool",
-      "duckduckgo_search",
-      "image_generator",
-    ])
-  ).concat(queryEngineTool ? [queryEngineTool] : []);
+export const createResearcher = async (chatHistory: ChatMessage[]) => {
+  const queryEngineTools = await getQueryEngineTools();
+  const tools = [
+    await getTool("wikipedia_tool"),
+    await getTool("duckduckgo_search"),
+    await getTool("image_generator"),
+    ...(queryEngineTools ? queryEngineTools : []),
+  ].filter((tool) => tool !== undefined);
 
   return new FunctionCallingAgent({
     name: "researcher",
@@ -81,17 +78,17 @@ Example:
 };
 
 export const createPublisher = async (chatHistory: ChatMessage[]) => {
-  const tools = await lookupTools(["document_generator"]);
+  const tool = await getTool("document_generator");
   let systemPrompt = `You are an expert in publishing blog posts. You are given a task to publish a blog post. 
 If the writer says that there was an error, you should reply with the error and not publish the post.`;
-  if (tools.length > 0) {
+  if (tool) {
     systemPrompt = `${systemPrompt}. 
 If the user requests to generate a file, use the document_generator tool to generate the file and reply with the link to the file.
 Otherwise, simply return the content of the post.`;
   }
   return new FunctionCallingAgent({
     name: "publisher",
-    tools: tools,
+    tools: tool ? [tool] : [],
     systemPrompt: systemPrompt,
     chatHistory,
   });

templates/components/agents/typescript/blog/workflow/factory.ts

@@ -0,0 +1,291 @@
+import {
+  HandlerContext,
+  StartEvent,
+  StopEvent,
+  Workflow,
+  WorkflowContext,
+  WorkflowEvent,
+} from "@llamaindex/workflow";
+import { ChatMessage, ChatResponseChunk, Settings } from "llamaindex";
+import {
+  createPublisher,
+  createResearcher,
+  createReviewer,
+  createWriter,
+} from "./agents";
+import {
+  FunctionCallingAgent,
+  FunctionCallingAgentInput,
+} from "./single-agent";
+import { AgentInput, AgentRunEvent } from "./type";
+
+const TIMEOUT = 360 * 1000;
+const MAX_ATTEMPTS = 2;
+
+class ResearchEvent extends WorkflowEvent<{ input: string }> {}
+class WriteEvent extends WorkflowEvent<{
+  input: string;
+  isGood: boolean;
+}> {}
+class ReviewEvent extends WorkflowEvent<{ input: string }> {}
+class PublishEvent extends WorkflowEvent<{ input: string }> {}
+
+type BlogContext = {
+  task: string;
+  attempts: number;
+  result: string;
+};
+
+export const createWorkflow = ({
+  chatHistory,
+  params,
+}: {
+  chatHistory: ChatMessage[];
+  params?: any;
+}) => {
+  const runAgent = async (
+    context: HandlerContext<BlogContext>,
+    agent: FunctionCallingAgent,
+    input: FunctionCallingAgentInput,
+  ) => {
+    const agentContext = agent.run(input, {
+      streaming: input.streaming ?? false,
+    });
+    for await (const event of agentContext) {
+      if (event instanceof AgentRunEvent) {
+        context.sendEvent(event);
+      }
+      if (event instanceof StopEvent) {
+        return event;
+      }
+    }
+    return null;
+  };
+
+  const start = async (
+    context: HandlerContext<BlogContext>,
+    ev: StartEvent<AgentInput>,
+  ) => {
+    const chatHistoryStr = chatHistory
+      .map((msg) => `${msg.role}: ${msg.content}`)
+      .join("\n");
+
+    // Decision-making process
+    const decision = await decideWorkflow(
+      ev.data.message.toString(),
+      chatHistoryStr,
+    );
+
+    if (decision !== "publish") {
+      return new ResearchEvent({
+        input: `Research for this task: ${JSON.stringify(context.data.task)}`,
+      });
+    } else {
+      return new PublishEvent({
+        input: `Publish content based on the chat history\n${chatHistoryStr}\n\n and task: ${context.data.task}`,
+      });
+    }
+  };
+
+  const decideWorkflow = async (task: string, chatHistoryStr: string) => {
+    const llm = Settings.llm;
+
+    const prompt = `You are an expert in decision-making, helping people write and publish blog posts.
+If the user is asking for a file or to publish content, respond with 'publish'.
+If the user requests to write or update a blog post, respond with 'not_publish'.
+
+Here is the chat history:
+${chatHistoryStr}
+
+The current user request is:
+${task}
+
+Given the chat history and the new user request, decide whether to publish based on existing information.
+Decision (respond with either 'not_publish' or 'publish'):`;
+
+    const output = await llm.complete({ prompt: prompt });
+    const decision = output.text.trim().toLowerCase();
+    return decision === "publish" ? "publish" : "research";
+  };
+
+  const research = async (
+    context: HandlerContext<BlogContext>,
+    ev: ResearchEvent,
+  ) => {
+    const researcher = await createResearcher(chatHistory);
+    const researchRes = await runAgent(context, researcher, {
+      displayName: "Researcher",
+      message: ev.data.input,
+    });
+    const researchResult = researchRes?.data;
+
+    return new WriteEvent({
+      input: `Write a blog post given this task: ${JSON.stringify(
+        context.data.task,
+      )} using this research content: ${researchResult}`,
+      isGood: false,
+    });
+  };
+
+  const write = async (
+    context: HandlerContext<BlogContext>,
+    ev: WriteEvent,
+  ) => {
+    const writer = createWriter(chatHistory);
+    context.data.attempts = context.data.attempts + 1;
+    const tooManyAttempts = context.data.attempts > MAX_ATTEMPTS;
+    if (tooManyAttempts) {
+      context.sendEvent(
+        new AgentRunEvent({
+          agent: "writer",
+          text: `Too many attempts (${MAX_ATTEMPTS}) to write the blog post. Proceeding with the current version.`,
+          type: "text",
+        }),
+      );
+    }
+
+    if (ev.data.isGood || tooManyAttempts) {
+      // the blog post is good or too many attempts
+      // stream the final content
+      const result = await runAgent(context, writer, {
+        message: `Based on the reviewer's feedback, refine the post and return only the final version of the post. Here's the current version: ${ev.data.input}`,
+        displayName: "Writer",
+        streaming: true,
+      });
+      return result as unknown as StopEvent<AsyncGenerator<ChatResponseChunk>>;
+    }
+
+    const writeRes = await runAgent(context, writer, {
+      message: ev.data.input,
+      displayName: "Writer",
+      streaming: false,
+    });
+    const writeResult = writeRes?.data;
+    context.data.result = writeResult; // store the last result
+
+    return new ReviewEvent({ input: writeResult });
+  };
+
+  const review = async (
+    context: HandlerContext<BlogContext>,
+    ev: ReviewEvent,
+  ) => {
+    const reviewer = createReviewer(chatHistory);
+    const reviewResult = (await runAgent(context, reviewer, {
+      message: ev.data.input,
+      displayName: "Reviewer",
+      streaming: false,
+    })) as unknown as StopEvent<string>;
+    const reviewResultStr = reviewResult.data;
+    const oldContent = context.data.result;
+    const postIsGood = reviewResultStr.toLowerCase().includes("post is good");
+    context.sendEvent(
+      new AgentRunEvent({
+        agent: "reviewer",
+        text: `The post is ${postIsGood ? "" : "not "}good enough for publishing. Sending back to the writer${
+          postIsGood ? " for publication." : "."
+        }`,
+        type: "text",
+      }),
+    );
+    if (postIsGood) {
+      return new WriteEvent({
+        input: "",
+        isGood: true,
+      });
+    }
+
+    return new WriteEvent({
+      input: `Improve the writing of a given blog post by using a given review.
+            Blog post:
+            \`\`\`
+            ${oldContent}
+            \`\`\`
+
+            Review:
+            \`\`\`
+            ${reviewResult}
+            \`\`\``,
+      isGood: false,
+    });
+  };
+
+  const publish = async (
+    context: HandlerContext<BlogContext>,
+    ev: PublishEvent,
+  ) => {
+    const publisher = await createPublisher(chatHistory);
+
+    const publishResult = await runAgent(context, publisher, {
+      message: `${ev.data.input}`,
+      displayName: "Publisher",
+      streaming: true,
+    });
+    return publishResult as unknown as StopEvent<
+      AsyncGenerator<ChatResponseChunk>
+    >;
+  };
+
+  const workflow: Workflow<
+    BlogContext,
+    AgentInput,
+    string | AsyncGenerator<boolean | ChatResponseChunk>
+  > = new Workflow();
+
+  workflow.addStep(
+    {
+      inputs: [StartEvent<AgentInput>],
+      outputs: [ResearchEvent, PublishEvent],
+    },
+    start,
+  );
+
+  workflow.addStep(
+    {
+      inputs: [ResearchEvent],
+      outputs: [WriteEvent],
+    },
+    research,
+  );
+
+  workflow.addStep(
+    {
+      inputs: [WriteEvent],
+      outputs: [ReviewEvent, StopEvent<AsyncGenerator<ChatResponseChunk>>],
+    },
+    write,
+  );
+
+  workflow.addStep(
+    {
+      inputs: [ReviewEvent],
+      outputs: [WriteEvent],
+    },
+    review,
+  );
+
+  workflow.addStep(
+    {
+      inputs: [PublishEvent],
+      outputs: [StopEvent],
+    },
+    publish,
+  );
+
+  // Overload run method to initialize the context
+  workflow.run = function (
+    input: AgentInput,
+  ): WorkflowContext<
+    AgentInput,
+    string | AsyncGenerator<boolean | ChatResponseChunk>,
+    BlogContext
+  > {
+    return Workflow.prototype.run.call(workflow, new StartEvent(input), {
+      task: input.message.toString(),
+      attempts: 0,
+      result: "",
+    });
+  };
+
+  return workflow;
+};

templates/components/agents/typescript/financial_report/README-template.md

@@ -0,0 +1,47 @@
+This is a [LlamaIndex](https://www.llamaindex.ai/) project using [Next.js](https://nextjs.org/) bootstrapped with [`create-llama`](https://github.com/run-llama/LlamaIndexTS/tree/main/packages/create-llama).
+
+## Getting Started
+
+First, install the dependencies:
+
+```
+npm install
+```
+
+Then check the parameters that have been pre-configured in the `.env` file in this directory.
+Make sure you have the `OPENAI_API_KEY` set.
+
+Second, generate the embeddings of the documents in the `./data` directory:
+
+```
+npm run generate
+```
+
+Third, run the development server:
+
+```
+npm run dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the chat UI.
+
+## Use Case: Filling Financial CSV Template
+
+You can start by sending an request on the chat UI to create a report comparing the finances of Apple and Tesla.
+Or you can test the `/api/chat` endpoint with the following curl request:
+
+```
+curl --location 'localhost:3000/api/chat' \
+--header 'Content-Type: application/json' \
+--data '{ "messages": [{ "role": "user", "content": "Create a report comparing the finances of Apple and Tesla" }] }'
+```
+
+## Learn More
+
+To learn more about LlamaIndex, take a look at the following resources:
+
+- [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex (Python features).
+- [LlamaIndexTS Documentation](https://ts.llamaindex.ai/docs/llamaindex) - learn about LlamaIndex (Typescript features).
+- [Workflows Introduction](https://ts.llamaindex.ai/docs/llamaindex/guide/workflow) - learn about LlamaIndexTS workflows.
+
+You can check out [the LlamaIndexTS GitHub repository](https://github.com/run-llama/LlamaIndexTS) - your feedback and contributions are welcome!

templates/components/agents/typescript/financial_report/workflow/factory.ts

@@ -0,0 +1,20 @@
+import { ChatMessage, ToolCallLLM } from "llamaindex";
+import { getTool } from "../engine/tools";
+import { FinancialReportWorkflow } from "./fin-report";
+import { getQueryEngineTools } from "./tools";
+
+const TIMEOUT = 360 * 1000;
+
+export async function createWorkflow(options: {
+  chatHistory: ChatMessage[];
+  llm?: ToolCallLLM;
+}) {
+  return new FinancialReportWorkflow({
+    chatHistory: options.chatHistory,
+    queryEngineTools: (await getQueryEngineTools()) || [],
+    codeInterpreterTool: (await getTool("interpreter"))!,
+    documentGeneratorTool: (await getTool("document_generator"))!,
+    llm: options.llm,
+    timeout: TIMEOUT,
+  });
+}

templates/components/agents/typescript/financial_report/workflow/fin-report.ts

@@ -0,0 +1,322 @@
+import {
+  HandlerContext,
+  StartEvent,
+  StopEvent,
+  Workflow,
+  WorkflowEvent,
+} from "@llamaindex/workflow";
+import {
+  BaseToolWithCall,
+  ChatMemoryBuffer,
+  ChatMessage,
+  ChatResponseChunk,
+  Settings,
+  ToolCall,
+  ToolCallLLM,
+} from "llamaindex";
+import { callTools, chatWithTools } from "./tools";
+import { AgentInput, AgentRunEvent } from "./type";
+
+// Create a custom event type
+class InputEvent extends WorkflowEvent<{ input: ChatMessage[] }> {}
+
+class ResearchEvent extends WorkflowEvent<{
+  toolCalls: ToolCall[];
+}> {}
+
+class AnalyzeEvent extends WorkflowEvent<{
+  input: ChatMessage | ToolCall[];
+}> {}
+
+class ReportGenerationEvent extends WorkflowEvent<{
+  toolCalls: ToolCall[];
+}> {}
+
+const DEFAULT_SYSTEM_PROMPT = `
+You are a financial analyst who are given a set of tools to help you.
+It's good to using appropriate tools for the user request and always use the information from the tools, don't make up anything yourself.
+For the query engine tool, you should break down the user request into a list of queries and call the tool with the queries.
+`;
+
+export class FinancialReportWorkflow extends Workflow<
+  null,
+  AgentInput,
+  ChatResponseChunk
+> {
+  llm: ToolCallLLM;
+  memory: ChatMemoryBuffer;
+  queryEngineTools: BaseToolWithCall[];
+  codeInterpreterTool: BaseToolWithCall;
+  documentGeneratorTool: BaseToolWithCall;
+  systemPrompt?: string;
+
+  constructor(options: {
+    llm?: ToolCallLLM;
+    chatHistory: ChatMessage[];
+    queryEngineTools: BaseToolWithCall[];
+    codeInterpreterTool: BaseToolWithCall;
+    documentGeneratorTool: BaseToolWithCall;
+    systemPrompt?: string;
+    verbose?: boolean;
+    timeout?: number;
+  }) {
+    super({
+      verbose: options?.verbose ?? false,
+      timeout: options?.timeout ?? 360,
+    });
+
+    this.llm = options.llm ?? (Settings.llm as ToolCallLLM);
+    if (!(this.llm instanceof ToolCallLLM)) {
+      throw new Error("LLM is not a ToolCallLLM");
+    }
+    this.systemPrompt = options.systemPrompt ?? DEFAULT_SYSTEM_PROMPT;
+    this.queryEngineTools = options.queryEngineTools;
+    this.codeInterpreterTool = options.codeInterpreterTool;
+
+    this.documentGeneratorTool = options.documentGeneratorTool;
+    this.memory = new ChatMemoryBuffer({
+      llm: this.llm,
+      chatHistory: options.chatHistory,
+    });
+
+    // Add steps
+    this.addStep(
+      {
+        inputs: [StartEvent<AgentInput>],
+        outputs: [InputEvent],
+      },
+      this.prepareChatHistory,
+    );
+
+    this.addStep(
+      {
+        inputs: [InputEvent],
+        outputs: [
+          InputEvent,
+          ResearchEvent,
+          AnalyzeEvent,
+          ReportGenerationEvent,
+          StopEvent,
+        ],
+      },
+      this.handleLLMInput,
+    );
+
+    this.addStep(
+      {
+        inputs: [ResearchEvent],
+        outputs: [AnalyzeEvent],
+      },
+      this.handleResearch,
+    );
+
+    this.addStep(
+      {
+        inputs: [AnalyzeEvent],
+        outputs: [InputEvent],
+      },
+      this.handleAnalyze,
+    );
+
+    this.addStep(
+      {
+        inputs: [ReportGenerationEvent],
+        outputs: [InputEvent],
+      },
+      this.handleReportGeneration,
+    );
+  }
+
+  prepareChatHistory = async (
+    ctx: HandlerContext<null>,
+    ev: StartEvent<AgentInput>,
+  ): Promise<InputEvent> => {
+    const { message } = ev.data;
+
+    if (this.systemPrompt) {
+      this.memory.put({ role: "system", content: this.systemPrompt });
+    }
+    this.memory.put({ role: "user", content: message });
+
+    return new InputEvent({ input: this.memory.getMessages() });
+  };
+
+  handleLLMInput = async (
+    ctx: HandlerContext<null>,
+    ev: InputEvent,
+  ): Promise<
+    | InputEvent
+    | ResearchEvent
+    | AnalyzeEvent
+    | ReportGenerationEvent
+    | StopEvent
+  > => {
+    const chatHistory = ev.data.input;
+
+    const tools = [this.codeInterpreterTool, this.documentGeneratorTool];
+    if (this.queryEngineTools) {
+      tools.push(...this.queryEngineTools);
+    }
+
+    const toolCallResponse = await chatWithTools(this.llm, tools, chatHistory);
+
+    if (!toolCallResponse.hasToolCall()) {
+      return new StopEvent(toolCallResponse.responseGenerator);
+    }
+
+    if (toolCallResponse.hasMultipleTools()) {
+      this.memory.put({
+        role: "assistant",
+        content:
+          "Calling different tools is not allowed. Please only use multiple calls of the same tool.",
+      });
+      return new InputEvent({ input: this.memory.getMessages() });
+    }
+
+    // Put the LLM tool call message into the memory
+    // And trigger the next step according to the tool call
+    if (toolCallResponse.toolCallMessage) {
+      this.memory.put(toolCallResponse.toolCallMessage);
+    }
+    const toolName = toolCallResponse.getToolNames()[0];
+    switch (toolName) {
+      case this.codeInterpreterTool.metadata.name:
+        return new AnalyzeEvent({
+          input: toolCallResponse.toolCalls,
+        });
+      case this.documentGeneratorTool.metadata.name:
+        return new ReportGenerationEvent({
+          toolCalls: toolCallResponse.toolCalls,
+        });
+      default:
+        if (
+          this.queryEngineTools &&
+          this.queryEngineTools.some((tool) => tool.metadata.name === toolName)
+        ) {
+          return new ResearchEvent({
+            toolCalls: toolCallResponse.toolCalls,
+          });
+        }
+        throw new Error(`Unknown tool: ${toolName}`);
+    }
+  };
+
+  handleResearch = async (
+    ctx: HandlerContext<null>,
+    ev: ResearchEvent,
+  ): Promise<AnalyzeEvent> => {
+    ctx.sendEvent(
+      new AgentRunEvent({
+        agent: "Researcher",
+        text: "Researching data",
+        type: "text",
+      }),
+    );
+
+    const { toolCalls } = ev.data;
+
+    const toolMsgs = await callTools({
+      tools: this.queryEngineTools,
+      toolCalls,
+      ctx,
+      agentName: "Researcher",
+    });
+    for (const toolMsg of toolMsgs) {
+      this.memory.put(toolMsg);
+    }
+    return new AnalyzeEvent({
+      input: {
+        role: "assistant",
+        content:
+          "I have finished researching the data, please analyze the data.",
+      },
+    });
+  };
+
+  /**
+   * Analyze a research result or a tool call for code interpreter from the LLM
+   */
+  handleAnalyze = async (
+    ctx: HandlerContext<null>,
+    ev: AnalyzeEvent,
+  ): Promise<InputEvent> => {
+    ctx.sendEvent(
+      new AgentRunEvent({
+        agent: "Analyst",
+        text: `Starting analysis`,
+        type: "text",
+      }),
+    );
+    // Request by workflow LLM, input is a list of tool calls
+    let toolCalls: ToolCall[] = [];
+    if (Array.isArray(ev.data.input)) {
+      toolCalls = ev.data.input;
+    } else {
+      // Requested by Researcher, input is a ChatMessage
+      // We start new LLM chat specifically for analyzing the data
+      const analysisPrompt = `
+      You are an expert in analyzing financial data.
+      You are given a set of financial data to analyze. Your task is to analyze the financial data and return a report.
+      Your response should include a detailed analysis of the financial data, including any trends, patterns, or insights that you find.
+      Construct the analysis in textual format; including tables would be great!
+      Don't need to synthesize the data, just analyze and provide your findings.
+      `;
+
+      // Clone the current chat history
+      // Add the analysis system prompt and the message from the researcher
+      const newChatHistory = [
+        ...this.memory.getMessages(),
+        { role: "system", content: analysisPrompt },
+        ev.data.input,
+      ];
+      const toolCallResponse = await chatWithTools(
+        this.llm,
+        [this.codeInterpreterTool],
+        newChatHistory as ChatMessage[],
+      );
+
+      if (!toolCallResponse.hasToolCall()) {
+        this.memory.put(await toolCallResponse.asFullResponse());
+        return new InputEvent({
+          input: this.memory.getMessages(),
+        });
+      } else {
+        this.memory.put(toolCallResponse.toolCallMessage);
+        toolCalls = toolCallResponse.toolCalls;
+      }
+    }
+
+    // Call the tools
+    const toolMsgs = await callTools({
+      tools: [this.codeInterpreterTool],
+      toolCalls,
+      ctx,
+      agentName: "Analyst",
+    });
+    for (const toolMsg of toolMsgs) {
+      this.memory.put(toolMsg);
+    }
+
+    return new InputEvent({
+      input: this.memory.getMessages(),
+    });
+  };
+
+  handleReportGeneration = async (
+    ctx: HandlerContext<null>,
+    ev: ReportGenerationEvent,
+  ): Promise<InputEvent> => {
+    const { toolCalls } = ev.data;
+
+    const toolMsgs = await callTools({
+      tools: [this.documentGeneratorTool],
+      toolCalls,
+      ctx,
+      agentName: "Reporter",
+    });
+    for (const toolMsg of toolMsgs) {
+      this.memory.put(toolMsg);
+    }
+    return new InputEvent({ input: this.memory.getMessages() });
+  };
+}

templates/components/agents/typescript/form_filling/README-template.md

@@ -0,0 +1,37 @@
+This is a [LlamaIndex](https://www.llamaindex.ai/) project using [Next.js](https://nextjs.org/) bootstrapped with [`create-llama`](https://github.com/run-llama/LlamaIndexTS/tree/main/packages/create-llama).
+
+## Getting Started
+
+First, install the dependencies:
+
+```
+npm install
+```
+
+Then check the parameters that have been pre-configured in the `.env` file in this directory.
+Make sure you have the `OPENAI_API_KEY` set.
+
+Second, run the development server:
+
+```
+npm run dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the chat UI.
+
+## Use Case: Filling Financial CSV Template
+
+1. Upload the Apple and Tesla financial reports from the [data](./data) directory. Just send an empty message.
+2. Upload the CSV file [sec_10k_template.csv](./sec_10k_template.csv) and send the message "Fill the missing cells in the CSV file".
+
+The agent will fill the missing cells by retrieving the information from the uploaded financial reports and return a new CSV file with the filled cells.
+
+## Learn More
+
+To learn more about LlamaIndex, take a look at the following resources:
+
+- [LlamaIndex Documentation](https://docs.llamaindex.ai) - learn about LlamaIndex (Python features).
+- [LlamaIndexTS Documentation](https://ts.llamaindex.ai/docs/llamaindex) - learn about LlamaIndex (Typescript features).
+- [Workflows Introduction](https://ts.llamaindex.ai/docs/llamaindex/guide/workflow) - learn about LlamaIndexTS workflows.
+
+You can check out [the LlamaIndexTS GitHub repository](https://github.com/run-llama/LlamaIndexTS) - your feedback and contributions are welcome!

templates/components/agents/typescript/form_filling/sec_10k_template.csv

@@ -0,0 +1,17 @@
+Parameter,2023 Apple (AAPL),2023 Tesla (TSLA)
+Revenue,,
+Net Income,,
+Earnings Per Share (EPS),,
+Debt-to-Equity Ratio,,
+Current Ratio,,
+Gross Margin,,
+Operating Margin,,
+Net Profit Margin,,
+Inventory Turnover,,
+Accounts Receivable Turnover,,
+Capital Expenditure,,
+Research and Development Expense,,
+Market Cap,,
+Price to Earnings Ratio,,
+Dividend Yield,,
+Year-over-Year Growth Rate,,

templates/components/agents/typescript/form_filling/workflow/factory.ts

@@ -0,0 +1,20 @@
+import { ChatMessage, ToolCallLLM } from "llamaindex";
+import { getTool } from "../engine/tools";
+import { FormFillingWorkflow } from "./form-filling";
+import { getQueryEngineTools } from "./tools";
+
+const TIMEOUT = 360 * 1000;
+
+export async function createWorkflow(options: {
+  chatHistory: ChatMessage[];
+  llm?: ToolCallLLM;
+}) {
+  return new FormFillingWorkflow({
+    chatHistory: options.chatHistory,
+    queryEngineTools: (await getQueryEngineTools()) || [],
+    extractorTool: (await getTool("extract_missing_cells"))!,
+    fillMissingCellsTool: (await getTool("fill_missing_cells"))!,
+    llm: options.llm,
+    timeout: TIMEOUT,
+  });
+}

templates/components/agents/typescript/form_filling/workflow/form-filling.ts

@@ -0,0 +1,275 @@
+import {
+  HandlerContext,
+  StartEvent,
+  StopEvent,
+  Workflow,
+  WorkflowEvent,
+} from "@llamaindex/workflow";
+import {
+  BaseToolWithCall,
+  ChatMemoryBuffer,
+  ChatMessage,
+  ChatResponseChunk,
+  Settings,
+  ToolCall,
+  ToolCallLLM,
+} from "llamaindex";
+import { callTools, chatWithTools } from "./tools";
+import { AgentInput, AgentRunEvent } from "./type";
+
+// Create a custom event type
+class InputEvent extends WorkflowEvent<{ input: ChatMessage[] }> {}
+
+class ExtractMissingCellsEvent extends WorkflowEvent<{
+  toolCalls: ToolCall[];
+}> {}
+
+class FindAnswersEvent extends WorkflowEvent<{
+  toolCalls: ToolCall[];
+}> {}
+
+class FillMissingCellsEvent extends WorkflowEvent<{
+  toolCalls: ToolCall[];
+}> {}
+
+const DEFAULT_SYSTEM_PROMPT = `
+You are a helpful assistant who helps fill missing cells in a CSV file.
+Only use the information from the retriever tool - don't make up any information yourself. Fill N/A if an answer is not found.
+If there is no retriever tool or the gathered information has many N/A values indicating the questions don't match the data, respond with a warning and ask the user to upload a different file or connect to a knowledge base.
+You can make multiple tool calls at once but only call with the same tool.
+Only use the local file path for the tools.
+`;
+
+export class FormFillingWorkflow extends Workflow<
+  null,
+  AgentInput,
+  ChatResponseChunk
+> {
+  llm: ToolCallLLM;
+  memory: ChatMemoryBuffer;
+  extractorTool: BaseToolWithCall;
+  queryEngineTools?: BaseToolWithCall[];
+  fillMissingCellsTool: BaseToolWithCall;
+  systemPrompt?: string;
+
+  constructor(options: {
+    llm?: ToolCallLLM;
+    chatHistory: ChatMessage[];
+    extractorTool: BaseToolWithCall;
+    queryEngineTools?: BaseToolWithCall[];
+    fillMissingCellsTool: BaseToolWithCall;
+    systemPrompt?: string;
+    verbose?: boolean;
+    timeout?: number;
+  }) {
+    super({
+      verbose: options?.verbose ?? false,
+      timeout: options?.timeout ?? 360,
+    });
+
+    this.llm = options.llm ?? (Settings.llm as ToolCallLLM);
+    if (!(this.llm instanceof ToolCallLLM)) {
+      throw new Error("LLM is not a ToolCallLLM");
+    }
+    this.systemPrompt = options.systemPrompt ?? DEFAULT_SYSTEM_PROMPT;
+    this.extractorTool = options.extractorTool;
+    this.queryEngineTools = options.queryEngineTools;
+    this.fillMissingCellsTool = options.fillMissingCellsTool;
+
+    this.memory = new ChatMemoryBuffer({
+      llm: this.llm,
+      chatHistory: options.chatHistory,
+    });
+
+    // Add steps
+    this.addStep(
+      {
+        inputs: [StartEvent<AgentInput>],
+        outputs: [InputEvent],
+      },
+      this.prepareChatHistory,
+    );
+
+    this.addStep(
+      {
+        inputs: [InputEvent],
+        outputs: [
+          InputEvent,
+          ExtractMissingCellsEvent,
+          FindAnswersEvent,
+          FillMissingCellsEvent,
+          StopEvent,
+        ],
+      },
+      this.handleLLMInput,
+    );
+
+    this.addStep(
+      {
+        inputs: [ExtractMissingCellsEvent],
+        outputs: [InputEvent],
+      },
+      this.handleExtractMissingCells,
+    );
+
+    this.addStep(
+      {
+        inputs: [FindAnswersEvent],
+        outputs: [InputEvent],
+      },
+      this.handleFindAnswers,
+    );
+
+    this.addStep(
+      {
+        inputs: [FillMissingCellsEvent],
+        outputs: [InputEvent],
+      },
+      this.handleFillMissingCells,
+    );
+  }
+
+  prepareChatHistory = async (
+    ctx: HandlerContext<null>,
+    ev: StartEvent<AgentInput>,
+  ): Promise<InputEvent> => {
+    const { message } = ev.data;
+
+    if (this.systemPrompt) {
+      this.memory.put({ role: "system", content: this.systemPrompt });
+    }
+    this.memory.put({ role: "user", content: message });
+
+    return new InputEvent({ input: this.memory.getMessages() });
+  };
+
+  handleLLMInput = async (
+    ctx: HandlerContext<null>,
+    ev: InputEvent,
+  ): Promise<
+    | InputEvent
+    | ExtractMissingCellsEvent
+    | FindAnswersEvent
+    | FillMissingCellsEvent
+    | StopEvent
+  > => {
+    const chatHistory = ev.data.input;
+
+    const tools = [this.extractorTool, this.fillMissingCellsTool];
+    if (this.queryEngineTools) {
+      tools.push(...this.queryEngineTools);
+    }
+
+    const toolCallResponse = await chatWithTools(this.llm, tools, chatHistory);
+
+    if (!toolCallResponse.hasToolCall()) {
+      return new StopEvent(toolCallResponse.responseGenerator);
+    }
+
+    if (toolCallResponse.hasMultipleTools()) {
+      this.memory.put({
+        role: "assistant",
+        content:
+          "Calling different tools is not allowed. Please only use multiple calls of the same tool.",
+      });
+      return new InputEvent({ input: this.memory.getMessages() });
+    }
+
+    // Put the LLM tool call message into the memory
+    // And trigger the next step according to the tool call
+    if (toolCallResponse.toolCallMessage) {
+      this.memory.put(toolCallResponse.toolCallMessage);
+    }
+    const toolName = toolCallResponse.getToolNames()[0];
+    switch (toolName) {
+      case this.extractorTool.metadata.name:
+        return new ExtractMissingCellsEvent({
+          toolCalls: toolCallResponse.toolCalls,
+        });
+      case this.fillMissingCellsTool.metadata.name:
+        return new FillMissingCellsEvent({
+          toolCalls: toolCallResponse.toolCalls,
+        });
+      default:
+        if (
+          this.queryEngineTools &&
+          this.queryEngineTools.some((tool) => tool.metadata.name === toolName)
+        ) {
+          return new FindAnswersEvent({
+            toolCalls: toolCallResponse.toolCalls,
+          });
+        }
+        throw new Error(`Unknown tool: ${toolName}`);
+    }
+  };
+
+  handleExtractMissingCells = async (
+    ctx: HandlerContext<null>,
+    ev: ExtractMissingCellsEvent,
+  ): Promise<InputEvent> => {
+    ctx.sendEvent(
+      new AgentRunEvent({
+        agent: "CSVExtractor",
+        text: "Extracting missing cells",
+        type: "text",
+      }),
+    );
+    const { toolCalls } = ev.data;
+    const toolMsgs = await callTools({
+      tools: [this.extractorTool],
+      toolCalls,
+      ctx,
+      agentName: "CSVExtractor",
+    });
+    for (const toolMsg of toolMsgs) {
+      this.memory.put(toolMsg);
+    }
+    return new InputEvent({ input: this.memory.getMessages() });
+  };
+
+  handleFindAnswers = async (
+    ctx: HandlerContext<null>,
+    ev: FindAnswersEvent,
+  ): Promise<InputEvent> => {
+    const { toolCalls } = ev.data;
+    if (!this.queryEngineTools) {
+      throw new Error("Query engine tool is not available");
+    }
+    ctx.sendEvent(
+      new AgentRunEvent({
+        agent: "Researcher",
+        text: "Finding answers",
+        type: "text",
+      }),
+    );
+    const toolMsgs = await callTools({
+      tools: this.queryEngineTools,
+      toolCalls,
+      ctx,
+      agentName: "Researcher",
+    });
+
+    for (const toolMsg of toolMsgs) {
+      this.memory.put(toolMsg);
+    }
+    return new InputEvent({ input: this.memory.getMessages() });
+  };
+
+  handleFillMissingCells = async (
+    ctx: HandlerContext<null>,
+    ev: FillMissingCellsEvent,
+  ): Promise<InputEvent> => {
+    const { toolCalls } = ev.data;
+
+    const toolMsgs = await callTools({
+      tools: [this.fillMissingCellsTool],
+      toolCalls,
+      ctx,
+      agentName: "Processor",
+    });
+    for (const toolMsg of toolMsgs) {
+      this.memory.put(toolMsg);
+    }
+    return new InputEvent({ input: this.memory.getMessages() });
+  };
+}

templates/components/engines/python/agent/tools/__init__.py

@@ -56,7 +56,7 @@ class ToolFactory:
             A dictionary of tool names to lists of FunctionTools if map_result is True,
             otherwise a list of FunctionTools.
         """
-        tools: Union[Dict[str, List[FunctionTool]], List[FunctionTool]] = (
+        tools: Union[Dict[str, FunctionTool], List[FunctionTool]] = (
             {} if map_result else []
         )
 
@@ -69,7 +69,9 @@ class ToolFactory:
                             tool_type, tool_name, config
                         )
                         if map_result:
-                            tools[tool_name] = loaded_tools  # type: ignore
+                            tools.update(  # type: ignore
+                                {tool.metadata.name: tool for tool in loaded_tools}
+                            )
                         else:
                             tools.extend(loaded_tools)  # type: ignore
 

templates/components/engines/python/agent/tools/artifact.py

@@ -66,21 +66,29 @@ class CodeGeneratorTool:
     def __init__(self):
         pass
 
-    def artifact(self, query: str, old_code: Optional[str] = None) -> Dict:
-        """Generate a code artifact based on the input.
+    def artifact(
+        self,
+        query: str,
+        sandbox_files: Optional[List[str]] = None,
+        old_code: Optional[str] = None,
+    ) -> Dict:
+        """Generate a code artifact based on the provided input.
 
         Args:
-            query (str): The description of the application you want to build.
+            query (str): A description of the application you want to build.
+            sandbox_files (Optional[List[str]], optional): A list of sandbox file paths. Defaults to None. Include these files if the code requires them.
             old_code (Optional[str], optional): The existing code to be modified. Defaults to None.
 
         Returns:
-            Dict: A dictionary containing the generated artifact information.
+            Dict: A dictionary containing information about the generated artifact.
         """
 
         if old_code:
             user_message = f"{query}\n\nThe existing code is: \n```\n{old_code}\n```"
         else:
             user_message = query
+        if sandbox_files:
+            user_message += f"\n\nThe provided files are: \n{str(sandbox_files)}"
 
         messages: List[ChatMessage] = [
             ChatMessage(role="system", content=CODE_GENERATION_PROMPT),
@@ -90,7 +98,10 @@ class CodeGeneratorTool:
             sllm = Settings.llm.as_structured_llm(output_cls=CodeArtifact)  # type: ignore
             response = sllm.chat(messages)
             data: CodeArtifact = response.raw
-            return data.model_dump()
+            data_dict = data.model_dump()
+            if sandbox_files:
+                data_dict["files"] = sandbox_files
+            return data_dict
         except Exception as e:
             logger.error(f"Failed to generate artifact: {str(e)}")
             raise e

templates/components/engines/python/agent/tools/document_generator.py

@@ -105,7 +105,7 @@ class DocumentGenerator:
         Generate HTML content from the original markdown content.
         """
         try:
-            import markdown
+            import markdown  # type: ignore
         except ImportError:
             raise ImportError(
                 "Failed to import required modules. Please install markdown."

templates/components/engines/python/agent/tools/form_filling.py

@@ -0,0 +1,224 @@
+import logging
+import os
+import uuid
+from textwrap import dedent
+from typing import Optional
+
+import pandas as pd
+from app.services.file import FileService
+from llama_index.core import Settings
+from llama_index.core.prompts import PromptTemplate
+from llama_index.core.tools import FunctionTool
+from pydantic import BaseModel, Field
+
+logger = logging.getLogger(__name__)
+
+
+class MissingCell(BaseModel):
+    """
+    A missing cell in a table.
+    """
+
+    row_index: int = Field(description="The index of the row of the missing cell")
+    column_index: int = Field(description="The index of the column of the missing cell")
+    question_to_answer: str = Field(
+        description="The question to answer to fill the missing cell"
+    )
+
+
+class MissingCells(BaseModel):
+    """
+    A list of missing cells.
+    """
+
+    missing_cells: list[MissingCell] = Field(description="The missing cells")
+
+
+class CellValue(BaseModel):
+    row_index: int = Field(description="The row index of the cell")
+    column_index: int = Field(description="The column index of the cell")
+    value: str = Field(
+        description="The value of the cell. Should be a concise value (numerical value or specific value)"
+    )
+
+
+class FormFillingTool:
+    """
+    Fill out missing cells in a CSV file using information from the knowledge base.
+    """
+
+    save_dir: str = os.path.join("output", "tools")
+
+    # Default prompt for extracting questions
+    # Replace the default prompt with a custom prompt by setting the EXTRACT_QUESTIONS_PROMPT environment variable.
+    _default_extract_questions_prompt = dedent(
+        """
+        You are a data analyst. You are given a table with missing cells.
+        Your task is to identify the missing cells and the questions needed to fill them.
+        IMPORTANT: Column indices should be 0-based, where the first data column is index 1 
+        (index 0 is typically the row names/index column).
+
+        # Instructions:
+        - Understand the entire content of the table and the topics of the table.
+        - Identify the missing cells and the meaning of the data in the cells.
+        - For each missing cell, provide the row index and the correct column index (remember: first data column is 1).
+        - For each missing cell, provide the question needed to fill the cell (it's important to provide the question that is relevant to the topic of the table).
+        - Since the cell's value should be concise, the question should request a numerical answer or a specific value.
+
+        # Example:
+        # |    | Name | Age | City |
+        # |----|------|-----|------|
+        # | 0  | John |     | Paris|
+        # | 1  | Mary |     |      |
+        # | 2  |      | 30  |      |
+        #
+        # Your thoughts:
+        # - The table is about people's names, ages, and cities.
+        # - Row: 1, Column: 1 (Age column), Question: "How old is Mary? Please provide only the numerical answer."
+        # - Row: 1, Column: 2 (City column), Question: "In which city does Mary live? Please provide only the city name."
+        
+
+        Please provide your answer in the requested format.
+        # Here is your task:
+
+        - Table content:
+        {table_content}
+
+        - Your answer:
+        """
+    )
+
+    def extract_questions(
+        self,
+        file_path: Optional[str] = None,
+        file_content: Optional[str] = None,
+    ) -> dict:
+        """
+        Use this tool to extract missing cells in a CSV file and generate questions to fill them.
+        Pass either the path to the CSV file or the content of the CSV file.
+
+        Args:
+            file_path (Optional[str]): The local file path to the CSV file to extract missing cells from (Don't pass a sandbox path).
+            file_content (Optional[str]): The content of the CSV file to extract missing cells from.
+
+        Returns:
+            dict: A dictionary containing the missing cells and their corresponding questions.
+        """
+        extract_questions_prompt = os.getenv(
+            "EXTRACT_QUESTIONS_PROMPT", self._default_extract_questions_prompt
+        )
+        if file_path is None and file_content is None:
+            raise ValueError("Either `file_path` or `file_content` must be provided")
+
+        table_content = None
+
+        if file_path:
+            file_name, file_extension = self._get_file_name_and_extension(
+                file_path, file_content
+            )
+
+            try:
+                df = pd.read_csv(file_path)
+            except FileNotFoundError as e:
+                return {
+                    "error": str(e),
+                    "message": "Please check and update the file path and ensure it's a local path - not a sandbox path.",
+                }
+
+            table_content = df.to_markdown()
+            if table_content is None:
+                raise ValueError("Could not convert the table to markdown")
+        if file_content:
+            table_content = file_content
+
+        if table_content is None:
+            raise ValueError("Table content not found")
+
+        response: MissingCells = Settings.llm.structured_predict(
+            output_cls=MissingCells,
+            prompt=PromptTemplate(extract_questions_prompt),
+            table_content=table_content,
+        )
+        return response.model_dump()
+
+    def fill_form(
+        self,
+        cell_values: list[CellValue],
+        file_path: Optional[str] = None,
+        file_content: Optional[str] = None,
+    ) -> dict:
+        """
+        Use this tool to fill cell values into a CSV file.
+        Requires cell values to be used for filling out, as well as either the path to the CSV file or the content of the CSV file.
+
+        Args:
+            cell_values (list[CellValue]): The cell values used to fill out the CSV file (call `extract_questions` and query engine to construct the cell values).
+            file_path (Optional[str]): The local file path to the CSV file that should be filled out (not as sandbox path).
+            file_content (Optional[str]): The content of the CSV file that should be filled out.
+
+        Returns:
+            dict: A dictionary containing the content and metadata of the filled-out file.
+        """
+        file_name, file_extension = self._get_file_name_and_extension(
+            file_path, file_content
+        )
+        df = pd.read_csv(file_path)
+
+        # Fill the dataframe with the cell values
+        filled_df = df.copy()
+        for cell_value in cell_values:
+            if not isinstance(cell_value, CellValue):
+                cell_value = CellValue(**cell_value)
+            filled_df.iloc[cell_value.row_index, cell_value.column_index] = (
+                cell_value.value
+            )
+
+        # Save the filled table to a new CSV file
+        csv_content: str = filled_df.to_csv(index=False)
+        file_metadata = FileService.save_file(
+            content=csv_content,
+            file_name=f"{file_name}_filled.csv",
+            save_dir=self.save_dir,
+        )
+
+        new_content: str = filled_df.to_markdown()
+        result = {
+            "filled_content": new_content,
+            "filled_file": file_metadata,
+        }
+        return result
+
+    def _get_file_name_and_extension(
+        self, file_path: Optional[str], file_content: Optional[str]
+    ) -> tuple[str, str]:
+        if file_path is None and file_content is None:
+            raise ValueError("Either `file_path` or `file_content` must be provided")
+
+        if file_path is None:
+            file_name = str(uuid.uuid4())
+            file_extension = ".csv"
+        else:
+            file_name, file_extension = os.path.splitext(file_path)
+            if file_extension != ".csv":
+                raise ValueError("Form filling is only supported for CSV files")
+
+        return file_name, file_extension
+
+    def _save_output(self, file_name: str, output: str) -> dict:
+        """
+        Save the output to a file.
+        """
+        file_metadata = FileService.save_file(
+            content=output,
+            file_name=file_name,
+            save_dir=self.save_dir,
+        )
+        return file_metadata.model_dump()
+
+
+def get_tools(**kwargs):
+    tool = FormFillingTool()
+    return [
+        FunctionTool.from_defaults(tool.extract_questions),
+        FunctionTool.from_defaults(tool.fill_form),
+    ]

templates/components/engines/python/agent/tools/img_gen.py

@@ -3,7 +3,7 @@ import os
 import uuid
 from typing import Optional
 
-import requests
+import requests  # type: ignore
 from llama_index.core.tools import FunctionTool
 from pydantic import BaseModel, Field
 

templates/components/engines/python/agent/tools/interpreter.py

@@ -2,14 +2,15 @@ import base64
 import logging
 import os
 import uuid
-from typing import Dict, List, Optional
+from typing import List, Optional
 
+from app.services.file import DocumentFile, FileService
 from e2b_code_interpreter import CodeInterpreter
 from e2b_code_interpreter.models import Logs
 from llama_index.core.tools import FunctionTool
 from pydantic import BaseModel
 
-logger = logging.getLogger(__name__)
+logger = logging.getLogger("uvicorn")
 
 
 class InterpreterExtraResult(BaseModel):
@@ -22,13 +23,16 @@ class InterpreterExtraResult(BaseModel):
 class E2BToolOutput(BaseModel):
     is_error: bool
     logs: Logs
+    error_message: Optional[str] = None
     results: List[InterpreterExtraResult] = []
+    retry_count: int = 0
 
 
 class E2BCodeInterpreter:
     output_dir = "output/tools"
+    uploaded_files_dir = "output/uploaded"
 
-    def __init__(self, api_key: str = None):
+    def __init__(self, api_key: Optional[str] = None):
         if api_key is None:
             api_key = os.getenv("E2B_API_KEY")
         filesever_url_prefix = os.getenv("FILESERVER_URL_PREFIX")
@@ -42,40 +46,45 @@ class E2BCodeInterpreter:
             )
 
         self.filesever_url_prefix = filesever_url_prefix
-        self.interpreter = CodeInterpreter(api_key=api_key)
+        self.interpreter = None
+        self.api_key = api_key
 
     def __del__(self):
-        self.interpreter.close()
+        """
+        Kill the interpreter when the tool is no longer in use
+        """
+        if self.interpreter is not None:
+            self.interpreter.kill()
 
-    def get_output_path(self, filename: str) -> str:
-        # if output directory doesn't exist, create it
-        if not os.path.exists(self.output_dir):
-            os.makedirs(self.output_dir, exist_ok=True)
-        return os.path.join(self.output_dir, filename)
+    def _init_interpreter(self, sandbox_files: List[str] = []):
+        """
+        Lazily initialize the interpreter.
+        """
+        logger.info(f"Initializing interpreter with {len(sandbox_files)} files")
+        self.interpreter = CodeInterpreter(api_key=self.api_key)
+        if len(sandbox_files) > 0:
+            for file_path in sandbox_files:
+                file_name = os.path.basename(file_path)
+                local_file_path = os.path.join(self.uploaded_files_dir, file_name)
+                with open(local_file_path, "rb") as f:
+                    content = f.read()
+                    if self.interpreter and self.interpreter.files:
+                        self.interpreter.files.write(file_path, content)
+            logger.info(f"Uploaded {len(sandbox_files)} files to sandbox")
 
-    def save_to_disk(self, base64_data: str, ext: str) -> Dict:
-        filename = f"{uuid.uuid4()}.{ext}"  # generate a unique filename
+    def _save_to_disk(self, base64_data: str, ext: str) -> DocumentFile:
         buffer = base64.b64decode(base64_data)
-        output_path = self.get_output_path(filename)
 
-        try:
-            with open(output_path, "wb") as file:
-                file.write(buffer)
-        except IOError as e:
-            logger.error(f"Failed to write to file {output_path}: {str(e)}")
-            raise e
+        # Output from e2b doesn't have a name. Create a random name for it.
+        filename = f"e2b_file_{uuid.uuid4()}.{ext}"
 
-        logger.info(f"Saved file to {output_path}")
+        document_file = FileService.save_file(
+            buffer, file_name=filename, save_dir=self.output_dir
+        )
 
-        return {
-            "outputPath": output_path,
-            "filename": filename,
-        }
+        return document_file
 
-    def get_file_url(self, filename: str) -> str:
-        return f"{self.filesever_url_prefix}/{self.output_dir}/{filename}"
-
-    def parse_result(self, result) -> List[InterpreterExtraResult]:
+    def _parse_result(self, result) -> List[InterpreterExtraResult]:
         """
         The result could include multiple formats (e.g. png, svg, etc.) but encoded in base64
         We save each result to disk and return saved file metadata (extension, filename, url)
@@ -92,16 +101,20 @@ class E2BCodeInterpreter:
             for ext, data in zip(formats, results):
                 match ext:
                     case "png" | "svg" | "jpeg" | "pdf":
-                        result = self.save_to_disk(data, ext)
-                        filename = result["filename"]
+                        document_file = self._save_to_disk(data, ext)
                         output.append(
                             InterpreterExtraResult(
                                 type=ext,
-                                filename=filename,
-                                url=self.get_file_url(filename),
+                                filename=document_file.name,
+                                url=document_file.url,
                             )
                         )
                     case _:
+                        # Try serialize data to string
+                        try:
+                            data = str(data)
+                        except Exception as e:
+                            data = f"Error when serializing data: {e}"
                         output.append(
                             InterpreterExtraResult(
                                 type=ext,
@@ -114,28 +127,75 @@ class E2BCodeInterpreter:
 
         return output
 
-    def interpret(self, code: str) -> E2BToolOutput:
+    def interpret(
+        self,
+        code: str,
+        sandbox_files: List[str] = [],
+        retry_count: int = 0,
+    ) -> E2BToolOutput:
         """
-        Execute python code in a Jupyter notebook cell, the toll will return result, stdout, stderr, display_data, and error.
+        Execute Python code in a Jupyter notebook cell. The tool will return the result, stdout, stderr, display_data, and error.
+        If the code needs to use a file, ALWAYS pass the file path in the sandbox_files argument.
+        You have a maximum of 3 retries to get the code to run successfully.
 
         Parameters:
-            code (str): The python code to be executed in a single cell.
+            code (str): The Python code to be executed in a single cell.
+            sandbox_files (List[str]): List of local file paths to be used by the code. The tool will throw an error if a file is not found.
+            retry_count (int): Number of times the tool has been retried.
         """
-        logger.info(
-            f"\n{'='*50}\n> Running following AI-generated code:\n{code}\n{'='*50}"
-        )
-        exec = self.interpreter.notebook.exec_cell(code)
+        if retry_count > 2:
+            return E2BToolOutput(
+                is_error=True,
+                logs=Logs(
+                    stdout="",
+                    stderr="",
+                    display_data="",
+                    error="",
+                ),
+                error_message="Failed to execute the code after 3 retries. Explain the error to the user and suggest a fix.",
+                retry_count=retry_count,
+            )
 
-        if exec.error:
-            logger.error("Error when executing code", exec.error)
-            output = E2BToolOutput(is_error=True, logs=exec.logs, results=[])
-        else:
-            if len(exec.results) == 0:
-                output = E2BToolOutput(is_error=False, logs=exec.logs, results=[])
+        if self.interpreter is None:
+            self._init_interpreter(sandbox_files)
+
+        if self.interpreter and self.interpreter.notebook:
+            logger.info(
+                f"\n{'='*50}\n> Running following AI-generated code:\n{code}\n{'='*50}"
+            )
+            exec = self.interpreter.notebook.exec_cell(code)
+
+            if exec.error:
+                error_message = f"The code failed to execute successfully. Error: {exec.error}. Try to fix the code and run again."
+                logger.error(error_message)
+                # Calling the generated code caused an error. Kill the interpreter and return the error to the LLM so it can try to fix the error
+                try:
+                    self.interpreter.kill()  # type: ignore
+                except Exception:
+                    pass
+                finally:
+                    self.interpreter = None
+                output = E2BToolOutput(
+                    is_error=True,
+                    logs=exec.logs,
+                    results=[],
+                    error_message=error_message,
+                    retry_count=retry_count + 1,
+                )
             else:
-                results = self.parse_result(exec.results[0])
-                output = E2BToolOutput(is_error=False, logs=exec.logs, results=results)
-        return output
+                if len(exec.results) == 0:
+                    output = E2BToolOutput(is_error=False, logs=exec.logs, results=[])
+                else:
+                    results = self._parse_result(exec.results[0])
+                    output = E2BToolOutput(
+                        is_error=False,
+                        logs=exec.logs,
+                        results=results,
+                        retry_count=retry_count + 1,
+                    )
+            return output
+        else:
+            raise ValueError("Interpreter is not initialized.")
 
 
 def get_tools(**kwargs):

templates/components/engines/python/agent/tools/openapi_action.py

@@ -1,4 +1,5 @@
 from typing import Dict, List, Tuple
+
 from llama_index.tools.openapi import OpenAPIToolSpec
 from llama_index.tools.requests import RequestsToolSpec
 
@@ -43,11 +44,12 @@ class OpenAPIActionToolSpec(OpenAPIToolSpec, RequestsToolSpec):
         Returns:
             List[Document]: A list of Document objects.
         """
-        import yaml
         from urllib.parse import urlparse
 
+        import yaml  # type: ignore
+
         if uri.startswith("http"):
-            import requests
+            import requests  # type: ignore
 
             response = requests.get(uri)
             if response.status_code != 200:

templates/components/engines/python/agent/tools/weather.py

@@ -1,8 +1,9 @@
 """Open Meteo weather map tool spec."""
 
 import logging
-import requests
-import pytz
+
+import pytz  # type: ignore
+import requests  # type: ignore
 from llama_index.core.tools import FunctionTool
 
 logger = logging.getLogger(__name__)

templates/components/engines/typescript/agent/chat.ts

@@ -1,7 +1,7 @@
 import {
   BaseChatEngine,
   BaseToolWithCall,
-  OpenAIAgent,
+  LLMAgent,
   QueryEngineTool,
 } from "llamaindex";
 import fs from "node:fs/promises";
@@ -42,7 +42,7 @@ export async function createChatEngine(documentIds?: string[], params?: any) {
     tools.push(...(await createTools(toolConfig)));
   }
 
-  const agent = new OpenAIAgent({
+  const agent = new LLMAgent({
     tools,
     systemPrompt: process.env.SYSTEM_PROMPT,
   }) as unknown as BaseChatEngine;

templates/components/engines/typescript/agent/tools/code-generator.ts

@@ -48,11 +48,13 @@ export type CodeArtifact = {
   port: number | null;
   file_path: string;
   code: string;
+  files?: string[];
 };
 
 export type CodeGeneratorParameter = {
   requirement: string;
   oldCode?: string;
+  sandboxFiles?: string[];
 };
 
 export type CodeGeneratorToolParams = {
@@ -75,6 +77,15 @@ const DEFAULT_META_DATA: ToolMetadata<JSONSchemaType<CodeGeneratorParameter>> =
           description: "The existing code to be modified",
           nullable: true,
         },
+        sandboxFiles: {
+          type: "array",
+          description:
+            "A list of sandbox file paths. Include these files if the code requires them.",
+          items: {
+            type: "string",
+          },
+          nullable: true,
+        },
       },
       required: ["requirement"],
     },
@@ -93,6 +104,9 @@ export class CodeGeneratorTool implements BaseTool<CodeGeneratorParameter> {
         input.requirement,
         input.oldCode,
       );
+      if (input.sandboxFiles) {
+        artifact.files = input.sandboxFiles;
+      }
       return artifact as JSONValue;
     } catch (error) {
       return { isError: true };

templates/components/engines/typescript/agent/tools/form-filling.ts

@@ -0,0 +1,296 @@
+import { JSONSchemaType } from "ajv";
+import fs from "fs";
+import { BaseTool, Settings, ToolMetadata } from "llamaindex";
+import Papa from "papaparse";
+import path from "path";
+import { saveDocument } from "../../llamaindex/documents/helper";
+
+type ExtractMissingCellsParameter = {
+  filePath: string;
+};
+
+export type MissingCell = {
+  rowIndex: number;
+  columnIndex: number;
+  question: string;
+};
+
+const CSV_EXTRACTION_PROMPT = `You are a data analyst. You are given a table with missing cells.
+Your task is to identify the missing cells and the questions needed to fill them.
+IMPORTANT: Column indices should be 0-based
+
+# Instructions:
+- Understand the entire content of the table and the topics of the table.
+- Identify the missing cells and the meaning of the data in the cells.
+- For each missing cell, provide the row index and the correct column index (remember: first data column is 1).
+- For each missing cell, provide the question needed to fill the cell (it's important to provide the question that is relevant to the topic of the table).
+- Since the cell's value should be concise, the question should request a numerical answer or a specific value.
+- Finally, only return the answer in JSON format with the following schema:
+{
+  "missing_cells": [
+    {
+      "rowIndex": number,
+      "columnIndex": number,
+      "question": string
+    }
+  ]
+}
+- If there are no missing cells, return an empty array.
+- The answer is only the JSON object, nothing else and don't wrap it inside markdown code block.
+
+# Example:
+# |    | Name | Age | City |
+# |----|------|-----|------|
+# | 0  | John |     | Paris|
+# | 1  | Mary |     |      |
+# | 2  |      | 30  |      |
+#
+# Your thoughts:
+# - The table is about people's names, ages, and cities.
+# - Row: 1, Column: 2 (Age column), Question: "How old is Mary? Please provide only the numerical answer."
+# - Row: 1, Column: 3 (City column), Question: "In which city does Mary live? Please provide only the city name."
+# Your answer:
+# {
+#   "missing_cells": [
+#     {
+#       "rowIndex": 1,
+#       "columnIndex": 2,
+#       "question": "How old is Mary? Please provide only the numerical answer."
+#     },
+#     {
+#       "rowIndex": 1,
+#       "columnIndex": 3,
+#       "question": "In which city does Mary live? Please provide only the city name."
+#     }
+#   ]
+# }
+
+
+# Here is your task:
+
+- Table content:
+{table_content}
+
+- Your answer:
+`;
+
+const DEFAULT_METADATA: ToolMetadata<
+  JSONSchemaType<ExtractMissingCellsParameter>
+> = {
+  name: "extract_missing_cells",
+  description: `Use this tool to extract missing cells in a CSV file and generate questions to fill them. This tool only works with local file path.`,
+  parameters: {
+    type: "object",
+    properties: {
+      filePath: {
+        type: "string",
+        description: "The local file path to the CSV file.",
+      },
+    },
+    required: ["filePath"],
+  },
+};
+
+export interface ExtractMissingCellsParams {
+  metadata?: ToolMetadata<JSONSchemaType<ExtractMissingCellsParameter>>;
+}
+
+export class ExtractMissingCellsTool
+  implements BaseTool<ExtractMissingCellsParameter>
+{
+  metadata: ToolMetadata<JSONSchemaType<ExtractMissingCellsParameter>>;
+  defaultExtractionPrompt: string;
+
+  constructor(params: ExtractMissingCellsParams) {
+    this.metadata = params.metadata ?? DEFAULT_METADATA;
+    this.defaultExtractionPrompt = CSV_EXTRACTION_PROMPT;
+  }
+
+  private readCsvFile(filePath: string): Promise<string[][]> {
+    return new Promise((resolve, reject) => {
+      fs.readFile(filePath, "utf8", (err, data) => {
+        if (err) {
+          reject(err);
+          return;
+        }
+
+        const parsedData = Papa.parse<string[]>(data, {
+          skipEmptyLines: false,
+        });
+
+        if (parsedData.errors.length) {
+          reject(parsedData.errors);
+          return;
+        }
+
+        // Ensure all rows have the same number of columns as the header
+        const maxColumns = parsedData.data[0].length;
+        const paddedRows = parsedData.data.map((row) => {
+          return [...row, ...Array(maxColumns - row.length).fill("")];
+        });
+
+        resolve(paddedRows);
+      });
+    });
+  }
+
+  private formatToMarkdownTable(data: string[][]): string {
+    if (data.length === 0) return "";
+
+    const maxColumns = data[0].length;
+
+    const headerRow = `| ${data[0].join(" | ")} |`;
+    const separatorRow = `| ${Array(maxColumns).fill("---").join(" | ")} |`;
+
+    const dataRows = data.slice(1).map((row) => {
+      return `| ${row.join(" | ")} |`;
+    });
+
+    return [headerRow, separatorRow, ...dataRows].join("\n");
+  }
+
+  async call(input: ExtractMissingCellsParameter): Promise<MissingCell[]> {
+    const { filePath } = input;
+    let tableContent: string[][];
+    try {
+      tableContent = await this.readCsvFile(filePath);
+    } catch (error) {
+      throw new Error(
+        `Failed to read CSV file. Make sure that you are reading a local file path (not a sandbox path).`,
+      );
+    }
+
+    const prompt = this.defaultExtractionPrompt.replace(
+      "{table_content}",
+      this.formatToMarkdownTable(tableContent),
+    );
+
+    const llm = Settings.llm;
+    const response = await llm.complete({
+      prompt,
+    });
+    const rawAnswer = response.text;
+    const parsedResponse = JSON.parse(rawAnswer) as {
+      missing_cells: MissingCell[];
+    };
+    if (!parsedResponse.missing_cells) {
+      throw new Error(
+        "The answer is not in the correct format. There should be a missing_cells array.",
+      );
+    }
+    const answer = parsedResponse.missing_cells;
+
+    return answer;
+  }
+}
+
+type FillMissingCellsParameter = {
+  filePath: string;
+  cells: {
+    rowIndex: number;
+    columnIndex: number;
+    answer: string;
+  }[];
+};
+
+const FILL_CELLS_METADATA: ToolMetadata<
+  JSONSchemaType<FillMissingCellsParameter>
+> = {
+  name: "fill_missing_cells",
+  description: `Use this tool to fill missing cells in a CSV file with provided answers. This tool only works with local file path.`,
+  parameters: {
+    type: "object",
+    properties: {
+      filePath: {
+        type: "string",
+        description: "The local file path to the CSV file.",
+      },
+      cells: {
+        type: "array",
+        items: {
+          type: "object",
+          properties: {
+            rowIndex: { type: "number" },
+            columnIndex: { type: "number" },
+            answer: { type: "string" },
+          },
+          required: ["rowIndex", "columnIndex", "answer"],
+        },
+        description: "Array of cells to fill with their answers",
+      },
+    },
+    required: ["filePath", "cells"],
+  },
+};
+
+export interface FillMissingCellsParams {
+  metadata?: ToolMetadata<JSONSchemaType<FillMissingCellsParameter>>;
+}
+
+export class FillMissingCellsTool
+  implements BaseTool<FillMissingCellsParameter>
+{
+  metadata: ToolMetadata<JSONSchemaType<FillMissingCellsParameter>>;
+
+  constructor(params: FillMissingCellsParams = {}) {
+    this.metadata = params.metadata ?? FILL_CELLS_METADATA;
+  }
+
+  async call(input: FillMissingCellsParameter): Promise<string> {
+    const { filePath, cells } = input;
+
+    // Read the CSV file
+    const fileContent = await new Promise<string>((resolve, reject) => {
+      fs.readFile(filePath, "utf8", (err, data) => {
+        if (err) {
+          reject(err);
+        } else {
+          resolve(data);
+        }
+      });
+    });
+
+    // Parse CSV with PapaParse
+    const parseResult = Papa.parse<string[]>(fileContent, {
+      header: false, // Ensure the header is not treated as a separate object
+      skipEmptyLines: false, // Ensure empty lines are not skipped
+    });
+
+    if (parseResult.errors.length) {
+      throw new Error(
+        "Failed to parse CSV file: " + parseResult.errors[0].message,
+      );
+    }
+
+    const rows = parseResult.data;
+
+    // Fill the cells with answers
+    for (const cell of cells) {
+      // Adjust rowIndex to start from 1 for data rows
+      const adjustedRowIndex = cell.rowIndex + 1;
+      if (
+        adjustedRowIndex < rows.length &&
+        cell.columnIndex < rows[adjustedRowIndex].length
+      ) {
+        rows[adjustedRowIndex][cell.columnIndex] = cell.answer;
+      }
+    }
+
+    // Convert back to CSV format
+    const updatedContent = Papa.unparse(rows, {
+      delimiter: parseResult.meta.delimiter,
+    });
+
+    // Use the helper function to write the file
+    const parsedPath = path.parse(filePath);
+    const newFileName = `${parsedPath.name}-filled${parsedPath.ext}`;
+    const newFilePath = path.join("output/tools", newFileName);
+
+    const newFileUrl = await saveDocument(newFilePath, updatedContent);
+
+    return (
+      "Successfully filled missing cells in the CSV file. File URL to show to the user: " +
+      newFileUrl
+    );
+  }
+}

templates/components/engines/typescript/agent/tools/index.ts

@@ -1,11 +1,19 @@
 import { BaseToolWithCall } from "llamaindex";
 import { ToolsFactory } from "llamaindex/tools/ToolsFactory";
+import fs from "node:fs/promises";
+import path from "node:path";
 import { CodeGeneratorTool, CodeGeneratorToolParams } from "./code-generator";
 import {
   DocumentGenerator,
   DocumentGeneratorParams,
 } from "./document-generator";
 import { DuckDuckGoSearchTool, DuckDuckGoToolParams } from "./duckduckgo";
+import {
+  ExtractMissingCellsParams,
+  ExtractMissingCellsTool,
+  FillMissingCellsParams,
+  FillMissingCellsTool,
+} from "./form-filling";
 import { ImgGeneratorTool, ImgGeneratorToolParams } from "./img-gen";
 import { InterpreterTool, InterpreterToolParams } from "./interpreter";
 import { OpenAPIActionTool } from "./openapi-action";
@@ -54,6 +62,12 @@ const toolFactory: Record<string, ToolCreator> = {
   document_generator: async (config: unknown) => {
     return [new DocumentGenerator(config as DocumentGeneratorParams)];
   },
+  form_filling: async (config: unknown) => {
+    return [
+      new ExtractMissingCellsTool(config as ExtractMissingCellsParams),
+      new FillMissingCellsTool(config as FillMissingCellsParams),
+    ];
+  },
 };
 
 async function createLocalTools(
@@ -70,3 +84,19 @@ async function createLocalTools(
 
   return tools;
 }
+
+export async function getConfiguredTools(
+  configPath?: string,
+): Promise<BaseToolWithCall[]> {
+  const configFile = path.join(configPath ?? "config", "tools.json");
+  const toolConfig = JSON.parse(await fs.readFile(configFile, "utf8"));
+  const tools = await createTools(toolConfig);
+  return tools;
+}
+
+export async function getTool(
+  toolName: string,
+): Promise<BaseToolWithCall | undefined> {
+  const tools = await getConfiguredTools();
+  return tools.find((tool) => tool.metadata.name === toolName);
+}

templates/components/engines/typescript/agent/tools/interpreter.ts

@@ -7,6 +7,8 @@ import path from "node:path";
 
 export type InterpreterParameter = {
   code: string;
+  sandboxFiles?: string[];
+  retryCount?: number;
 };
 
 export type InterpreterToolParams = {
@@ -18,7 +20,9 @@ export type InterpreterToolParams = {
 export type InterpreterToolOutput = {
   isError: boolean;
   logs: Logs;
+  text?: string;
   extraResult: InterpreterExtraResult[];
+  retryCount?: number;
 };
 
 type InterpreterExtraType =
@@ -41,8 +45,10 @@ export type InterpreterExtraResult = {
 
 const DEFAULT_META_DATA: ToolMetadata<JSONSchemaType<InterpreterParameter>> = {
   name: "interpreter",
-  description:
-    "Execute python code in a Jupyter notebook cell and return any result, stdout, stderr, display_data, and error.",
+  description: `Execute python code in a Jupyter notebook cell and return any result, stdout, stderr, display_data, and error.
+If the code needs to use a file, ALWAYS pass the file path in the sandbox_files argument.
+You have a maximum of 3 retries to get the code to run successfully.
+`,
   parameters: {
     type: "object",
     properties: {
@@ -50,6 +56,21 @@ const DEFAULT_META_DATA: ToolMetadata<JSONSchemaType<InterpreterParameter>> = {
         type: "string",
         description: "The python code to execute in a single cell.",
       },
+      sandboxFiles: {
+        type: "array",
+        description:
+          "List of local file paths to be used by the code. The tool will throw an error if a file is not found.",
+        items: {
+          type: "string",
+        },
+        nullable: true,
+      },
+      retryCount: {
+        type: "number",
+        description: "The number of times the tool has been retried",
+        default: 0,
+        nullable: true,
+      },
     },
     required: ["code"],
   },
@@ -57,6 +78,7 @@ const DEFAULT_META_DATA: ToolMetadata<JSONSchemaType<InterpreterParameter>> = {
 
 export class InterpreterTool implements BaseTool<InterpreterParameter> {
   private readonly outputDir = "output/tools";
+  private readonly uploadedFilesDir = "output/uploaded";
   private apiKey?: string;
   private fileServerURLPrefix?: string;
   metadata: ToolMetadata<JSONSchemaType<InterpreterParameter>>;
@@ -80,33 +102,67 @@ export class InterpreterTool implements BaseTool<InterpreterParameter> {
     }
   }
 
-  public async initInterpreter() {
+  public async initInterpreter(input: InterpreterParameter) {
     if (!this.codeInterpreter) {
       this.codeInterpreter = await CodeInterpreter.create({
         apiKey: this.apiKey,
       });
     }
+    // upload files to sandbox
+    if (input.sandboxFiles) {
+      console.log(`Uploading ${input.sandboxFiles.length} files to sandbox`);
+      try {
+        for (const filePath of input.sandboxFiles) {
+          const fileName = path.basename(filePath);
+          const localFilePath = path.join(this.uploadedFilesDir, fileName);
+          const content = fs.readFileSync(localFilePath);
+          await this.codeInterpreter?.files.write(filePath, content);
+        }
+      } catch (error) {
+        console.error("Got error when uploading files to sandbox", error);
+      }
+    }
     return this.codeInterpreter;
   }
 
-  public async codeInterpret(code: string): Promise<InterpreterToolOutput> {
+  public async codeInterpret(
+    input: InterpreterParameter,
+  ): Promise<InterpreterToolOutput> {
     console.log(
-      `\n${"=".repeat(50)}\n> Running following AI-generated code:\n${code}\n${"=".repeat(50)}`,
+      `Sandbox files: ${input.sandboxFiles}. Retry count: ${input.retryCount}`,
     );
-    const interpreter = await this.initInterpreter();
-    const exec = await interpreter.notebook.execCell(code);
+
+    if (input.retryCount && input.retryCount >= 3) {
+      return {
+        isError: true,
+        logs: {
+          stdout: [],
+          stderr: [],
+        },
+        text: "Max retries reached",
+        extraResult: [],
+      };
+    }
+
+    console.log(
+      `\n${"=".repeat(50)}\n> Running following AI-generated code:\n${input.code}\n${"=".repeat(50)}`,
+    );
+    const interpreter = await this.initInterpreter(input);
+    const exec = await interpreter.notebook.execCell(input.code);
     if (exec.error) console.error("[Code Interpreter error]", exec.error);
     const extraResult = await this.getExtraResult(exec.results[0]);
     const result: InterpreterToolOutput = {
       isError: !!exec.error,
       logs: exec.logs,
+      text: exec.text,
       extraResult,
+      retryCount: input.retryCount ? input.retryCount + 1 : 1,
     };
     return result;
   }
 
   async call(input: InterpreterParameter): Promise<InterpreterToolOutput> {
-    const result = await this.codeInterpret(input.code);
+    const result = await this.codeInterpret(input);
     return result;
   }
 

templates/components/llamaindex/typescript/documents/helper.ts

@@ -1,27 +1,60 @@
+import { Document } from "llamaindex";
+import crypto from "node:crypto";
 import fs from "node:fs";
 import path from "node:path";
 import { getExtractors } from "../../engine/loader";
+import { DocumentFile } from "../streaming/annotations";
 
 const MIME_TYPE_TO_EXT: Record<string, string> = {
   "application/pdf": "pdf",
   "text/plain": "txt",
+  "text/csv": "csv",
   "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
     "docx",
 };
 
-const UPLOADED_FOLDER = "output/uploaded";
+export const UPLOADED_FOLDER = "output/uploaded";
 
 export async function storeAndParseFile(
-  filename: string,
+  name: string,
+  fileBuffer: Buffer,
+  mimeType: string,
+): Promise<DocumentFile> {
+  const file = await storeFile(name, fileBuffer, mimeType);
+  const documents: Document[] = await parseFile(fileBuffer, name, mimeType);
+  // Update document IDs in the file metadata
+  file.refs = documents.map((document) => document.id_ as string);
+  return file;
+}
+
+export async function storeFile(
+  name: string,
   fileBuffer: Buffer,
   mimeType: string,
 ) {
   const fileExt = MIME_TYPE_TO_EXT[mimeType];
   if (!fileExt) throw new Error(`Unsupported document type: ${mimeType}`);
 
+  const fileId = crypto.randomUUID();
+  const newFilename = `${sanitizeFileName(name)}_${fileId}.${fileExt}`;
+  const filepath = path.join(UPLOADED_FOLDER, newFilename);
+  const fileUrl = await saveDocument(filepath, fileBuffer);
+  return {
+    id: fileId,
+    name: newFilename,
+    size: fileBuffer.length,
+    type: fileExt,
+    url: fileUrl,
+    refs: [] as string[],
+  } as DocumentFile;
+}
+
+export async function parseFile(
+  fileBuffer: Buffer,
+  filename: string,
+  mimeType: string,
+) {
   const documents = await loadDocuments(fileBuffer, mimeType);
-  const filepath = path.join(UPLOADED_FOLDER, filename);
-  await saveDocument(filepath, fileBuffer);
   for (const document of documents) {
     document.metadata = {
       ...document.metadata,
@@ -48,12 +81,6 @@ export async function saveDocument(filepath: string, content: string | Buffer) {
   if (path.isAbsolute(filepath)) {
     throw new Error("Absolute file paths are not allowed.");
   }
-  const fileName = path.basename(filepath);
-  if (!/^[a-zA-Z0-9_.-]+$/.test(fileName)) {
-    throw new Error(
-      "File name is not allowed to contain any special characters.",
-    );
-  }
   if (!process.env.FILESERVER_URL_PREFIX) {
     throw new Error("FILESERVER_URL_PREFIX environment variable is not set.");
   }
@@ -71,3 +98,8 @@ export async function saveDocument(filepath: string, content: string | Buffer) {
   console.log(`Saved document to ${filepath}. Reachable at URL: ${fileurl}`);
   return fileurl;
 }
+
+function sanitizeFileName(fileName: string) {
+  // Remove file extension and sanitize
+  return fileName.split(".")[0].replace(/[^a-zA-Z0-9_-]/g, "_");
+}

templates/components/llamaindex/typescript/documents/pipeline.ts

@@ -3,11 +3,12 @@ import {
   IngestionPipeline,
   Settings,
   SimpleNodeParser,
+  storageContextFromDefaults,
   VectorStoreIndex,
 } from "llamaindex";
 
 export async function runPipeline(
-  currentIndex: VectorStoreIndex,
+  currentIndex: VectorStoreIndex | null,
   documents: Document[],
 ) {
   // Use ingestion pipeline to process the documents into nodes and add them to the vector store
@@ -21,8 +22,27 @@ export async function runPipeline(
     ],
   });
   const nodes = await pipeline.run({ documents });
-  await currentIndex.insertNodes(nodes);
-  currentIndex.storageContext.docStore.persist();
-  console.log("Added nodes to the vector store.");
-  return documents.map((document) => document.id_);
+  if (currentIndex) {
+    await currentIndex.insertNodes(nodes);
+    currentIndex.storageContext.docStore.persist();
+    console.log("Added nodes to the vector store.");
+    return documents.map((document) => document.id_);
+  } else {
+    // Initialize a new index with the documents
+    console.log(
+      "Got empty index, created new index with the uploaded documents",
+    );
+    const persistDir = process.env.STORAGE_CACHE_DIR;
+    if (!persistDir) {
+      throw new Error("STORAGE_CACHE_DIR environment variable is required!");
+    }
+    const storageContext = await storageContextFromDefaults({
+      persistDir,
+    });
+    const newIndex = await VectorStoreIndex.fromDocuments(documents, {
+      storageContext,
+    });
+    await newIndex.storageContext.docStore.persist();
+    return documents.map((document) => document.id_);
+  }
 }

templates/components/llamaindex/typescript/documents/upload.ts

@@ -1,27 +1,36 @@
-import { LLamaCloudFileService, VectorStoreIndex } from "llamaindex";
+import { Document, LLamaCloudFileService, VectorStoreIndex } from "llamaindex";
 import { LlamaCloudIndex } from "llamaindex/cloud/LlamaCloudIndex";
-import { storeAndParseFile } from "./helper";
+import { DocumentFile } from "../streaming/annotations";
+import { parseFile, storeFile } from "./helper";
 import { runPipeline } from "./pipeline";
 
 export async function uploadDocument(
-  index: VectorStoreIndex | LlamaCloudIndex,
-  filename: string,
+  index: VectorStoreIndex | LlamaCloudIndex | null,
+  name: string,
   raw: string,
-): Promise<string[]> {
+): Promise<DocumentFile> {
   const [header, content] = raw.split(",");
   const mimeType = header.replace("data:", "").replace(";base64", "");
   const fileBuffer = Buffer.from(content, "base64");
 
+  // Store file
+  const fileMetadata = await storeFile(name, fileBuffer, mimeType);
+
+  // Do not index csv files
+  if (mimeType === "text/csv") {
+    return fileMetadata;
+  }
+  let documentIds: string[] = [];
   if (index instanceof LlamaCloudIndex) {
     // trigger LlamaCloudIndex API to upload the file and run the pipeline
     const projectId = await index.getProjectId();
     const pipelineId = await index.getPipelineId();
     try {
-      return [
+      documentIds = [
         await LLamaCloudFileService.addFileToPipeline(
           projectId,
           pipelineId,
-          new File([fileBuffer], filename, { type: mimeType }),
+          new File([fileBuffer], name, { type: mimeType }),
           { private: "true" },
         ),
       ];
@@ -36,9 +45,17 @@ export async function uploadDocument(
       }
       throw error;
     }
+  } else {
+    // run the pipeline for other vector store indexes
+    const documents: Document[] = await parseFile(
+      fileBuffer,
+      fileMetadata.name,
+      mimeType,
+    );
+    documentIds = await runPipeline(index, documents);
   }
 
-  // run the pipeline for other vector store indexes
-  const documents = await storeAndParseFile(filename, fileBuffer, mimeType);
-  return runPipeline(index, documents);
+  // Update file metadata with document IDs
+  fileMetadata.refs = documentIds;
+  return fileMetadata;
 }

templates/components/llamaindex/typescript/streaming/annotations.ts

@@ -1,19 +1,21 @@
 import { JSONValue, Message } from "ai";
-import { MessageContent, MessageContentDetail } from "llamaindex";
+import {
+  ChatMessage,
+  MessageContent,
+  MessageContentDetail,
+  MessageType,
+} from "llamaindex";
+import { UPLOADED_FOLDER } from "../documents/helper";
 
 export type DocumentFileType = "csv" | "pdf" | "txt" | "docx";
 
-export type DocumentFileContent = {
-  type: "ref" | "text";
-  value: string[] | string;
-};
-
 export type DocumentFile = {
   id: string;
-  filename: string;
-  filesize: number;
-  filetype: DocumentFileType;
-  content: DocumentFileContent;
+  name: string;
+  size: number;
+  type: string;
+  url: string;
+  refs?: string[];
 };
 
 type Annotation = {
@@ -29,28 +31,25 @@ export function isValidMessages(messages: Message[]): boolean {
 
 export function retrieveDocumentIds(messages: Message[]): string[] {
   // retrieve document Ids from the annotations of all messages (if any)
+  const documentFiles = retrieveDocumentFiles(messages);
+  return documentFiles.map((file) => file.refs || []).flat();
+}
+
+export function retrieveDocumentFiles(messages: Message[]): DocumentFile[] {
   const annotations = getAllAnnotations(messages);
   if (annotations.length === 0) return [];
 
-  const ids: string[] = [];
-
+  const files: DocumentFile[] = [];
   for (const { type, data } of annotations) {
     if (
       type === "document_file" &&
       "files" in data &&
       Array.isArray(data.files)
     ) {
-      const files = data.files as DocumentFile[];
-      for (const file of files) {
-        if (Array.isArray(file.content.value)) {
-          // it's an array, so it's an array of doc IDs
-          ids.push(...file.content.value);
-        }
-      }
+      files.push(...data.files);
     }
   }
-
-  return ids;
+  return files;
 }
 
 export function retrieveMessageContent(messages: Message[]): MessageContent {
@@ -65,6 +64,78 @@ export function retrieveMessageContent(messages: Message[]): MessageContent {
   ];
 }
 
+export function convertToChatHistory(messages: Message[]): ChatMessage[] {
+  if (!messages || !Array.isArray(messages)) {
+    return [];
+  }
+  const agentHistory = retrieveAgentHistoryMessage(messages);
+  if (agentHistory) {
+    const previousMessages = messages.slice(0, -1);
+    return [...previousMessages, agentHistory].map((msg) => ({
+      role: msg.role as MessageType,
+      content: msg.content,
+    }));
+  }
+  return messages.map((msg) => ({
+    role: msg.role as MessageType,
+    content: msg.content,
+  }));
+}
+
+function retrieveAgentHistoryMessage(
+  messages: Message[],
+  maxAgentMessages = 10,
+): ChatMessage | null {
+  const agentAnnotations = getAnnotations<{ agent: string; text: string }>(
+    messages,
+    { role: "assistant", type: "agent" },
+  ).slice(-maxAgentMessages);
+
+  if (agentAnnotations.length > 0) {
+    const messageContent =
+      "Here is the previous conversation of agents:\n" +
+      agentAnnotations.map((annotation) => annotation.data.text).join("\n");
+    return {
+      role: "assistant",
+      content: messageContent,
+    };
+  }
+  return null;
+}
+
+function getFileContent(file: DocumentFile): string {
+  let defaultContent = `=====File: ${file.name}=====\n`;
+  // Include file URL if it's available
+  const urlPrefix = process.env.FILESERVER_URL_PREFIX;
+  let urlContent = "";
+  if (urlPrefix) {
+    if (file.url) {
+      urlContent = `File URL: ${file.url}\n`;
+    } else {
+      urlContent = `File URL (instruction: do not update this file URL yourself): ${urlPrefix}/output/uploaded/${file.name}\n`;
+    }
+  } else {
+    console.warn(
+      "Warning: FILESERVER_URL_PREFIX not set in environment variables. Can't use file server",
+    );
+  }
+  defaultContent += urlContent;
+
+  // Include document IDs if it's available
+  if (file.refs) {
+    defaultContent += `Document IDs: ${file.refs}\n`;
+  }
+  // Include sandbox file paths
+  const sandboxFilePath = `/tmp/${file.name}`;
+  defaultContent += `Sandbox file path (instruction: only use sandbox path for artifact or code interpreter tool): ${sandboxFilePath}\n`;
+
+  // Include local file path
+  const localFilePath = `${UPLOADED_FOLDER}/${file.name}`;
+  defaultContent += `Local file path (instruction: use for local tool that requires a local path): ${localFilePath}\n`;
+
+  return defaultContent;
+}
+
 function getAllAnnotations(messages: Message[]): Annotation[] {
   return messages.flatMap((message) =>
     (message.annotations ?? []).map((annotation) =>
@@ -105,13 +176,10 @@ function retrieveLatestArtifact(messages: Message[]): MessageContentDetail[] {
 }
 
 function convertAnnotations(messages: Message[]): MessageContentDetail[] {
-  // annotations from the last user message that has annotations
-  const annotations: Annotation[] =
-    messages
-      .slice()
-      .reverse()
-      .find((message) => message.role === "user" && message.annotations)
-      ?.annotations?.map(getValidAnnotation) || [];
+  // get all annotations from user messages
+  const annotations: Annotation[] = messages
+    .filter((message) => message.role === "user" && message.annotations)
+    .flatMap((message) => message.annotations?.map(getValidAnnotation) || []);
   if (annotations.length === 0) return [];
 
   const content: MessageContentDetail[] = [];
@@ -131,25 +199,11 @@ function convertAnnotations(messages: Message[]): MessageContentDetail[] {
       "files" in data &&
       Array.isArray(data.files)
     ) {
-      // get all CSV files and convert their whole content to one text message
-      // currently CSV files are the only files where we send the whole content - we don't use an index
-      const csvFiles: DocumentFile[] = data.files.filter(
-        (file: DocumentFile) => file.filetype === "csv",
-      );
-      if (csvFiles && csvFiles.length > 0) {
-        const csvContents = csvFiles.map((file: DocumentFile) => {
-          const fileContent = Array.isArray(file.content.value)
-            ? file.content.value.join("\n")
-            : file.content.value;
-          return "```csv\n" + fileContent + "\n```";
-        });
-        const text =
-          "Use the following CSV content:\n" + csvContents.join("\n\n");
-        content.push({
-          type: "text",
-          text,
-        });
-      }
+      const fileContent = data.files.map(getFileContent).join("\n");
+      content.push({
+        type: "text",
+        text: fileContent,
+      });
     }
   });
 

templates/components/loaders/typescript/file/loader.ts

@@ -1,7 +1,7 @@
 import {
   FILE_EXT_TO_READER,
   SimpleDirectoryReader,
-} from "llamaindex/readers/SimpleDirectoryReader";
+} from "llamaindex/readers/index";
 
 export const DATA_DIR = "./data";
 

templates/components/loaders/typescript/llama_parse/loader.ts

@@ -2,7 +2,7 @@ import { LlamaParseReader } from "llamaindex";
 import {
   FILE_EXT_TO_READER,
   SimpleDirectoryReader,
-} from "llamaindex/readers/SimpleDirectoryReader";
+} from "llamaindex/readers/index";
 
 export const DATA_DIR = "./data";
 

templates/components/multiagent/python/app/api/routers/chat.py

@@ -4,7 +4,8 @@ from app.api.routers.models import (
     ChatData,
 )
 from app.api.routers.vercel_response import VercelStreamResponse
-from app.engine.engine import get_chat_engine
+from app.engine.query_filter import generate_filters
+from app.workflows import create_workflow
 from fastapi import APIRouter, BackgroundTasks, HTTPException, Request, status
 
 chat_router = r = APIRouter()
@@ -22,19 +23,20 @@ async def chat(
         last_message_content = data.get_last_message_content()
         messages = data.get_history_messages(include_agent_messages=True)
 
-        # The chat API supports passing private document filters and chat params
-        # but agent workflow does not support them yet
-        # ignore chat params and use all documents for now
-        # TODO: generate filters based on doc_ids
+        doc_ids = data.get_chat_document_ids()
+        filters = generate_filters(doc_ids)
         params = data.data or {}
-        engine = get_chat_engine(chat_history=messages, params=params)
 
-        event_handler = engine.run(input=last_message_content, streaming=True)
+        workflow = create_workflow(
+            chat_history=messages, params=params, filters=filters
+        )
+
+        event_handler = workflow.run(input=last_message_content, streaming=True)
         return VercelStreamResponse(
             request=request,
             chat_data=data,
             event_handler=event_handler,
-            events=engine.stream_events(),
+            events=workflow.stream_events(),
         )
     except Exception as e:
         logger.exception("Error in chat engine", exc_info=True)

templates/components/multiagent/python/app/api/routers/vercel_response.py

@@ -1,10 +1,9 @@
 import asyncio
 import json
 import logging
-from typing import AsyncGenerator, List
+from typing import AsyncGenerator, Awaitable, List
 
 from aiostream import stream
-from app.agents.single import AgentRunEvent, AgentRunResult
 from app.api.routers.models import ChatData, Message
 from app.api.services.suggestion import NextQuestionSuggestion
 from fastapi import Request
@@ -28,7 +27,6 @@ class VercelStreamResponse(StreamingResponse):
         super().__init__(content=content)
 
     async def content_generator(self, event_handler, events):
-        logger.info("Starting content_generator")
         stream = self._create_stream(
             self.request, self.chat_data, event_handler, events
         )
@@ -56,8 +54,8 @@ class VercelStreamResponse(StreamingResponse):
         self,
         request: Request,
         chat_data: ChatData,
-        event_handler: AgentRunResult | AsyncGenerator,
-        events: AsyncGenerator[AgentRunEvent, None],
+        event_handler: Awaitable,
+        events: AsyncGenerator,
         verbose: bool = True,
     ):
         # Yield the text response
@@ -65,15 +63,17 @@ class VercelStreamResponse(StreamingResponse):
             result = await event_handler
             final_response = ""
 
-            if isinstance(result, AgentRunResult):
-                for token in result.response.message.content:
-                    final_response += token
-                    yield self.convert_text(token)
-
             if isinstance(result, AsyncGenerator):
                 async for token in result:
-                    final_response += token.delta
+                    final_response += str(token.delta)
                     yield self.convert_text(token.delta)
+            else:
+                if hasattr(result, "response"):
+                    content = result.response.message.content
+                    if content:
+                        for token in content:
+                            final_response += str(token)
+                            yield self.convert_text(token)
 
             # Generate next questions if next question prompt is configured
             question_data = await self._generate_next_questions(
@@ -87,7 +87,7 @@ class VercelStreamResponse(StreamingResponse):
         # Yield the events from the event handler
         async def _event_generator():
             async for event in events:
-                event_response = self._event_to_response(event)
+                event_response = event.to_response()
                 if verbose:
                     logger.debug(event_response)
                 if event_response is not None:
@@ -96,13 +96,6 @@ class VercelStreamResponse(StreamingResponse):
         combine = stream.merge(_chat_response_generator(), _event_generator())
         return combine
 
-    @staticmethod
-    def _event_to_response(event: AgentRunEvent) -> dict:
-        return {
-            "type": "agent",
-            "data": {"agent": event.name, "text": event.msg},
-        }
-
     @classmethod
     def convert_text(cls, token: str):
         # Escape newlines and double quotes to avoid breaking the stream

templates/components/multiagent/python/app/workflows/events.py

@@ -0,0 +1,27 @@
+from enum import Enum
+from typing import Optional
+
+from llama_index.core.workflow import Event
+
+
+class AgentRunEventType(Enum):
+    TEXT = "text"
+    PROGRESS = "progress"
+
+
+class AgentRunEvent(Event):
+    name: str
+    msg: str
+    event_type: AgentRunEventType = AgentRunEventType.TEXT
+    data: Optional[dict] = None
+
+    def to_response(self) -> dict:
+        return {
+            "type": "agent",
+            "data": {
+                "agent": self.name,
+                "type": self.event_type.value,
+                "text": self.msg,
+                "data": self.data,
+            },
+        }

templates/components/multiagent/python/app/workflows/function_calling_agent.py

@@ -0,0 +1,121 @@
+from typing import Any, List, Optional
+
+from app.workflows.events import AgentRunEvent
+from app.workflows.tools import ToolCallResponse, call_tools, chat_with_tools
+from llama_index.core.base.llms.types import ChatMessage
+from llama_index.core.llms.function_calling import FunctionCallingLLM
+from llama_index.core.memory import ChatMemoryBuffer
+from llama_index.core.settings import Settings
+from llama_index.core.tools.types import BaseTool
+from llama_index.core.workflow import (
+    Context,
+    Event,
+    StartEvent,
+    StopEvent,
+    Workflow,
+    step,
+)
+
+
+class InputEvent(Event):
+    input: list[ChatMessage]
+
+
+class ToolCallEvent(Event):
+    input: ToolCallResponse
+
+
+class FunctionCallingAgent(Workflow):
+    """
+    A simple workflow to request LLM with tools independently.
+    You can share the previous chat history to provide the context for the LLM.
+    """
+
+    def __init__(
+        self,
+        *args: Any,
+        llm: FunctionCallingLLM | None = None,
+        chat_history: Optional[List[ChatMessage]] = None,
+        tools: List[BaseTool] | None = None,
+        system_prompt: str | None = None,
+        verbose: bool = False,
+        timeout: float = 360.0,
+        name: str,
+        write_events: bool = True,
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(*args, verbose=verbose, timeout=timeout, **kwargs)  # type: ignore
+        self.tools = tools or []
+        self.name = name
+        self.write_events = write_events
+
+        if llm is None:
+            llm = Settings.llm
+        self.llm = llm
+        if not self.llm.metadata.is_function_calling_model:
+            raise ValueError("The provided LLM must support function calling.")
+
+        self.system_prompt = system_prompt
+
+        self.memory = ChatMemoryBuffer.from_defaults(
+            llm=self.llm, chat_history=chat_history
+        )
+        self.sources = []  # type: ignore
+
+    @step()
+    async def prepare_chat_history(self, ctx: Context, ev: StartEvent) -> InputEvent:
+        # clear sources
+        self.sources = []
+
+        # set streaming
+        ctx.data["streaming"] = getattr(ev, "streaming", False)
+
+        # set system prompt
+        if self.system_prompt is not None:
+            system_msg = ChatMessage(role="system", content=self.system_prompt)
+            self.memory.put(system_msg)
+
+        # get user input
+        user_input = ev.input
+        user_msg = ChatMessage(role="user", content=user_input)
+        self.memory.put(user_msg)
+
+        if self.write_events:
+            ctx.write_event_to_stream(
+                AgentRunEvent(name=self.name, msg=f"Start to work on: {user_input}")
+            )
+
+        return InputEvent(input=self.memory.get())
+
+    @step()
+    async def handle_llm_input(
+        self,
+        ctx: Context,
+        ev: InputEvent,
+    ) -> ToolCallEvent | StopEvent:
+        chat_history = ev.input
+
+        response = await chat_with_tools(
+            self.llm,
+            self.tools,
+            chat_history,
+        )
+        is_tool_call = isinstance(response, ToolCallResponse)
+        if not is_tool_call:
+            if ctx.data["streaming"]:
+                return StopEvent(result=response)
+            else:
+                full_response = ""
+                async for chunk in response.generator:
+                    full_response += chunk.message.content
+                return StopEvent(result=full_response)
+        return ToolCallEvent(input=response)
+
+    @step()
+    async def handle_tool_calls(self, ctx: Context, ev: ToolCallEvent) -> InputEvent:
+        tool_calls = ev.input.tool_calls
+        tool_call_message = ev.input.tool_call_message
+        self.memory.put(tool_call_message)
+        tool_messages = await call_tools(self.name, self.tools, ctx, tool_calls)
+        self.memory.put_messages(tool_messages)
+        return InputEvent(input=self.memory.get())

templates/components/multiagent/python/app/workflows/tools.py

@@ -0,0 +1,227 @@
+import logging
+import uuid
+from abc import ABC, abstractmethod
+from typing import Any, AsyncGenerator, Callable, Optional
+
+from app.workflows.events import AgentRunEvent, AgentRunEventType
+from llama_index.core.base.llms.types import ChatMessage, ChatResponse, MessageRole
+from llama_index.core.llms.function_calling import FunctionCallingLLM
+from llama_index.core.tools import (
+    BaseTool,
+    FunctionTool,
+    ToolOutput,
+    ToolSelection,
+)
+from llama_index.core.workflow import Context
+from pydantic import BaseModel, ConfigDict
+
+logger = logging.getLogger("uvicorn")
+
+
+class ContextAwareTool(FunctionTool, ABC):
+    @abstractmethod
+    async def acall(self, ctx: Context, input: Any) -> ToolOutput:  # type: ignore
+        pass
+
+
+class ChatWithToolsResponse(BaseModel):
+    """
+    A tool call response from chat_with_tools.
+    """
+
+    tool_calls: Optional[list[ToolSelection]]
+    tool_call_message: Optional[ChatMessage]
+    generator: Optional[AsyncGenerator[ChatResponse | None, None]]
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    def is_calling_different_tools(self) -> bool:
+        tool_names = {tool_call.tool_name for tool_call in self.tool_calls}
+        return len(tool_names) > 1
+
+    def has_tool_calls(self) -> bool:
+        return self.tool_calls is not None and len(self.tool_calls) > 0
+
+    def tool_name(self) -> str:
+        assert self.has_tool_calls()
+        assert not self.is_calling_different_tools()
+        return self.tool_calls[0].tool_name
+
+    async def full_response(self) -> str:
+        assert self.generator is not None
+        full_response = ""
+        async for chunk in self.generator:
+            full_response += chunk.message.content
+        return full_response
+
+
+async def chat_with_tools(  # type: ignore
+    llm: FunctionCallingLLM,
+    tools: list[BaseTool],
+    chat_history: list[ChatMessage],
+) -> ChatWithToolsResponse:
+    """
+    Request LLM to call tools or not.
+    This function doesn't change the memory.
+    """
+    generator = _tool_call_generator(llm, tools, chat_history)
+    is_tool_call = await generator.__anext__()
+    if is_tool_call:
+        # Last chunk is the full response
+        # Wait for the last chunk
+        full_response = None
+        async for chunk in generator:
+            full_response = chunk
+        assert isinstance(full_response, ChatResponse)
+        return ChatWithToolsResponse(
+            tool_calls=llm.get_tool_calls_from_response(full_response),
+            tool_call_message=full_response.message,
+            generator=None,
+        )
+    else:
+        return ChatWithToolsResponse(
+            tool_calls=None,
+            tool_call_message=None,
+            generator=generator,
+        )
+
+
+async def call_tools(
+    ctx: Context,
+    agent_name: str,
+    tools: list[BaseTool],
+    tool_calls: list[ToolSelection],
+    emit_agent_events: bool = True,
+) -> list[ChatMessage]:
+    if len(tool_calls) == 0:
+        return []
+
+    tools_by_name = {tool.metadata.get_name(): tool for tool in tools}
+    if len(tool_calls) == 1:
+        return [
+            await call_tool(
+                ctx,
+                tools_by_name[tool_calls[0].tool_name],
+                tool_calls[0],
+                lambda msg: ctx.write_event_to_stream(
+                    AgentRunEvent(
+                        name=agent_name,
+                        msg=msg,
+                    )
+                ),
+            )
+        ]
+    # Multiple tool calls, show progress
+    tool_msgs: list[ChatMessage] = []
+
+    progress_id = str(uuid.uuid4())
+    total_steps = len(tool_calls)
+    if emit_agent_events:
+        ctx.write_event_to_stream(
+            AgentRunEvent(
+                name=agent_name,
+                msg=f"Making {total_steps} tool calls",
+            )
+        )
+    for i, tool_call in enumerate(tool_calls):
+        tool = tools_by_name.get(tool_call.tool_name)
+        if not tool:
+            tool_msgs.append(
+                ChatMessage(
+                    role=MessageRole.ASSISTANT,
+                    content=f"Tool {tool_call.tool_name} does not exist",
+                )
+            )
+            continue
+        tool_msg = await call_tool(
+            ctx,
+            tool,
+            tool_call,
+            event_emitter=lambda msg: ctx.write_event_to_stream(
+                AgentRunEvent(
+                    name=agent_name,
+                    msg=msg,
+                    event_type=AgentRunEventType.PROGRESS,
+                    data={
+                        "id": progress_id,
+                        "total": total_steps,
+                        "current": i,
+                    },
+                )
+            ),
+        )
+        tool_msgs.append(tool_msg)
+    return tool_msgs
+
+
+async def call_tool(
+    ctx: Context,
+    tool: BaseTool,
+    tool_call: ToolSelection,
+    event_emitter: Optional[Callable[[str], None]],
+) -> ChatMessage:
+    if event_emitter:
+        event_emitter(
+            f"Calling tool {tool_call.tool_name}, {str(tool_call.tool_kwargs)}"
+        )
+    try:
+        if isinstance(tool, ContextAwareTool):
+            if ctx is None:
+                raise ValueError("Context is required for context aware tool")
+            # inject context for calling an context aware tool
+            response = await tool.acall(ctx=ctx, **tool_call.tool_kwargs)
+        else:
+            response = await tool.acall(**tool_call.tool_kwargs)  # type: ignore
+        return ChatMessage(
+            role=MessageRole.TOOL,
+            content=str(response.raw_output),
+            additional_kwargs={
+                "tool_call_id": tool_call.tool_id,
+                "name": tool.metadata.get_name(),
+            },
+        )
+    except Exception as e:
+        logger.error(f"Got error in tool {tool_call.tool_name}: {str(e)}")
+        if event_emitter:
+            event_emitter(f"Got error in tool {tool_call.tool_name}: {str(e)}")
+        return ChatMessage(
+            role=MessageRole.TOOL,
+            content=f"Error: {str(e)}",
+            additional_kwargs={
+                "tool_call_id": tool_call.tool_id,
+                "name": tool.metadata.get_name(),
+            },
+        )
+
+
+async def _tool_call_generator(
+    llm: FunctionCallingLLM,
+    tools: list[BaseTool],
+    chat_history: list[ChatMessage],
+) -> AsyncGenerator[ChatResponse | bool, None]:
+    response_stream = await llm.astream_chat_with_tools(
+        tools,
+        chat_history=chat_history,
+        allow_parallel_tool_calls=False,
+    )
+
+    full_response = None
+    yielded_indicator = False
+    async for chunk in response_stream:
+        if "tool_calls" not in chunk.message.additional_kwargs:
+            # Yield a boolean to indicate whether the response is a tool call
+            if not yielded_indicator:
+                yield False
+                yielded_indicator = True
+
+            # if not a tool call, yield the chunks!
+            yield chunk  # type: ignore
+        elif not yielded_indicator:
+            # Yield the indicator for a tool call
+            yield True
+            yielded_indicator = True
+
+        full_response = chunk
+
+    if full_response:
+        yield full_response  # type: ignore

templates/components/multiagent/typescript/express/chat.controller.ts

@@ -1,36 +1,34 @@
-import { StopEvent } from "@llamaindex/core/workflow";
 import { Message, streamToResponse } from "ai";
 import { Request, Response } from "express";
-import { ChatResponseChunk } from "llamaindex";
+import {
+  convertToChatHistory,
+  retrieveMessageContent,
+} from "./llamaindex/streaming/annotations";
 import { createWorkflow } from "./workflow/factory";
-import { toDataStream, workflowEventsToStreamData } from "./workflow/stream";
+import { createStreamFromWorkflowContext } from "./workflow/stream";
 
 export const chat = async (req: Request, res: Response) => {
   try {
-    const { messages, data }: { messages: Message[]; data?: any } = req.body;
-    const userMessage = messages.pop();
-    if (!messages || !userMessage || userMessage.role !== "user") {
+    const { messages }: { messages: Message[] } = req.body;
+    if (!messages || messages.length === 0) {
       return res.status(400).json({
-        error:
-          "messages are required in the request body and the last message must be from the user",
+        error: "messages are required in the request body",
       });
     }
+    const chatHistory = convertToChatHistory(messages);
+    const userMessageContent = retrieveMessageContent(messages);
 
-    const agent = createWorkflow(messages, data);
-    const result = agent.run<AsyncGenerator<ChatResponseChunk>>(
-      userMessage.content,
-    ) as unknown as Promise<StopEvent<AsyncGenerator<ChatResponseChunk>>>;
+    const workflow = await createWorkflow({ chatHistory });
 
-    // convert the workflow events to a vercel AI stream data object
-    const agentStreamData = await workflowEventsToStreamData(
-      agent.streamEvents(),
-    );
-    // convert the workflow result to a vercel AI content stream
-    const stream = toDataStream(result, {
-      onFinal: () => agentStreamData.close(),
+    const context = workflow.run({
+      message: userMessageContent,
+      streaming: true,
     });
 
-    return streamToResponse(stream, res, {}, agentStreamData);
+    const { stream, dataStream } =
+      await createStreamFromWorkflowContext(context);
+
+    return streamToResponse(stream, res, {}, dataStream);
   } catch (error) {
     console.error("[LlamaIndex]", error);
     return res.status(500).json({

templates/components/multiagent/typescript/nextjs/route.ts

@@ -1,11 +1,14 @@
 import { initObservability } from "@/app/observability";
-import { StopEvent } from "@llamaindex/core/workflow";
-import { Message, StreamingTextResponse } from "ai";
-import { ChatResponseChunk } from "llamaindex";
+import { StreamingTextResponse, type Message } from "ai";
 import { NextRequest, NextResponse } from "next/server";
 import { initSettings } from "./engine/settings";
+import {
+  convertToChatHistory,
+  isValidMessages,
+  retrieveMessageContent,
+} from "./llamaindex/streaming/annotations";
 import { createWorkflow } from "./workflow/factory";
-import { toDataStream, workflowEventsToStreamData } from "./workflow/stream";
+import { createStreamFromWorkflowContext } from "./workflow/stream";
 
 initObservability();
 initSettings();
@@ -16,9 +19,8 @@ export const dynamic = "force-dynamic";
 export async function POST(request: NextRequest) {
   try {
     const body = await request.json();
-    const { messages, data }: { messages: Message[]; data?: any } = body;
-    const userMessage = messages.pop();
-    if (!messages || !userMessage || userMessage.role !== "user") {
+    const { messages }: { messages: Message[]; data?: any } = body;
+    if (!isValidMessages(messages)) {
       return NextResponse.json(
         {
           error:
@@ -28,20 +30,20 @@ export async function POST(request: NextRequest) {
       );
     }
 
-    const agent = createWorkflow(messages, data);
-    // TODO: fix type in agent.run in LITS
-    const result = agent.run<AsyncGenerator<ChatResponseChunk>>(
-      userMessage.content,
-    ) as unknown as Promise<StopEvent<AsyncGenerator<ChatResponseChunk>>>;
-    // convert the workflow events to a vercel AI stream data object
-    const agentStreamData = await workflowEventsToStreamData(
-      agent.streamEvents(),
-    );
-    // convert the workflow result to a vercel AI content stream
-    const stream = toDataStream(result, {
-      onFinal: () => agentStreamData.close(),
+    const chatHistory = convertToChatHistory(messages);
+    const userMessageContent = retrieveMessageContent(messages);
+
+    const workflow = await createWorkflow({ chatHistory });
+
+    const context = workflow.run({
+      message: userMessageContent,
+      streaming: true,
     });
-    return new StreamingTextResponse(stream, {}, agentStreamData);
+    const { stream, dataStream } =
+      await createStreamFromWorkflowContext(context);
+
+    // Return the two streams in one response
+    return new StreamingTextResponse(stream, {}, dataStream);
   } catch (error) {
     console.error("[LlamaIndex]", error);
     return NextResponse.json(

templates/components/multiagent/typescript/workflow/factory.ts

@@ -1,230 +0,0 @@
-import {
-  Context,
-  StartEvent,
-  StopEvent,
-  Workflow,
-  WorkflowEvent,
-} from "@llamaindex/core/workflow";
-import { Message } from "ai";
-import { ChatMessage, ChatResponseChunk, Settings } from "llamaindex";
-import { getAnnotations } from "../llamaindex/streaming/annotations";
-import {
-  createPublisher,
-  createResearcher,
-  createReviewer,
-  createWriter,
-} from "./agents";
-import { AgentInput, AgentRunEvent } from "./type";
-
-const TIMEOUT = 360 * 1000;
-const MAX_ATTEMPTS = 2;
-
-class ResearchEvent extends WorkflowEvent<{ input: string }> {}
-class WriteEvent extends WorkflowEvent<{
-  input: string;
-  isGood: boolean;
-}> {}
-class ReviewEvent extends WorkflowEvent<{ input: string }> {}
-class PublishEvent extends WorkflowEvent<{ input: string }> {}
-
-const prepareChatHistory = (chatHistory: Message[]): ChatMessage[] => {
-  // By default, the chat history only contains the assistant and user messages
-  // all the agents messages are stored in annotation data which is not visible to the LLM
-
-  const MAX_AGENT_MESSAGES = 10;
-  const agentAnnotations = getAnnotations<{ agent: string; text: string }>(
-    chatHistory,
-    { role: "assistant", type: "agent" },
-  ).slice(-MAX_AGENT_MESSAGES);
-
-  const agentMessages = agentAnnotations
-    .map(
-      (annotation) =>
-        `\n<${annotation.data.agent}>\n${annotation.data.text}\n</${annotation.data.agent}>`,
-    )
-    .join("\n");
-
-  const agentContent = agentMessages
-    ? "Here is the previous conversation of agents:\n" + agentMessages
-    : "";
-
-  if (agentContent) {
-    const agentMessage: ChatMessage = {
-      role: "assistant",
-      content: agentContent,
-    };
-    return [
-      ...chatHistory.slice(0, -1),
-      agentMessage,
-      chatHistory.slice(-1)[0],
-    ] as ChatMessage[];
-  }
-  return chatHistory as ChatMessage[];
-};
-
-export const createWorkflow = (messages: Message[], params?: any) => {
-  const chatHistoryWithAgentMessages = prepareChatHistory(messages);
-  const runAgent = async (
-    context: Context,
-    agent: Workflow,
-    input: AgentInput,
-  ) => {
-    const run = agent.run(new StartEvent({ input }));
-    for await (const event of agent.streamEvents()) {
-      if (event.data instanceof AgentRunEvent) {
-        context.writeEventToStream(event.data);
-      }
-    }
-    return await run;
-  };
-
-  const start = async (context: Context, ev: StartEvent) => {
-    context.set("task", ev.data.input);
-
-    const chatHistoryStr = chatHistoryWithAgentMessages
-      .map((msg) => `${msg.role}: ${msg.content}`)
-      .join("\n");
-
-    // Decision-making process
-    const decision = await decideWorkflow(ev.data.input, chatHistoryStr);
-
-    if (decision !== "publish") {
-      return new ResearchEvent({
-        input: `Research for this task: ${ev.data.input}`,
-      });
-    } else {
-      return new PublishEvent({
-        input: `Publish content based on the chat history\n${chatHistoryStr}\n\n and task: ${ev.data.input}`,
-      });
-    }
-  };
-
-  const decideWorkflow = async (task: string, chatHistoryStr: string) => {
-    const llm = Settings.llm;
-
-    const prompt = `You are an expert in decision-making, helping people write and publish blog posts.
-If the user is asking for a file or to publish content, respond with 'publish'.
-If the user requests to write or update a blog post, respond with 'not_publish'.
-
-Here is the chat history:
-${chatHistoryStr}
-
-The current user request is:
-${task}
-
-Given the chat history and the new user request, decide whether to publish based on existing information.
-Decision (respond with either 'not_publish' or 'publish'):`;
-
-    const output = await llm.complete({ prompt: prompt });
-    const decision = output.text.trim().toLowerCase();
-    return decision === "publish" ? "publish" : "research";
-  };
-
-  const research = async (context: Context, ev: ResearchEvent) => {
-    const researcher = await createResearcher(
-      chatHistoryWithAgentMessages,
-      params,
-    );
-    const researchRes = await runAgent(context, researcher, {
-      message: ev.data.input,
-    });
-    const researchResult = researchRes.data.result;
-    return new WriteEvent({
-      input: `Write a blog post given this task: ${context.get("task")} using this research content: ${researchResult}`,
-      isGood: false,
-    });
-  };
-
-  const write = async (context: Context, ev: WriteEvent) => {
-    const writer = createWriter(chatHistoryWithAgentMessages);
-
-    context.set("attempts", context.get("attempts", 0) + 1);
-    const tooManyAttempts = context.get("attempts") > MAX_ATTEMPTS;
-    if (tooManyAttempts) {
-      context.writeEventToStream(
-        new AgentRunEvent({
-          name: "writer",
-          msg: `Too many attempts (${MAX_ATTEMPTS}) to write the blog post. Proceeding with the current version.`,
-        }),
-      );
-    }
-
-    if (ev.data.isGood || tooManyAttempts) {
-      // the blog post is good or too many attempts
-      // stream the final content
-      const result = await runAgent(context, writer, {
-        message: `Based on the reviewer's feedback, refine the post and return only the final version of the post. Here's the current version: ${ev.data.input}`,
-        streaming: true,
-      });
-      return result as unknown as StopEvent<AsyncGenerator<ChatResponseChunk>>;
-    }
-
-    const writeRes = await runAgent(context, writer, {
-      message: ev.data.input,
-    });
-    const writeResult = writeRes.data.result;
-    context.set("result", writeResult); // store the last result
-    return new ReviewEvent({ input: writeResult });
-  };
-
-  const review = async (context: Context, ev: ReviewEvent) => {
-    const reviewer = createReviewer(chatHistoryWithAgentMessages);
-    const reviewRes = await reviewer.run(
-      new StartEvent<AgentInput>({ input: { message: ev.data.input } }),
-    );
-    const reviewResult = reviewRes.data.result;
-    const oldContent = context.get("result");
-    const postIsGood = reviewResult.toLowerCase().includes("post is good");
-    context.writeEventToStream(
-      new AgentRunEvent({
-        name: "reviewer",
-        msg: `The post is ${postIsGood ? "" : "not "}good enough for publishing. Sending back to the writer${
-          postIsGood ? " for publication." : "."
-        }`,
-      }),
-    );
-    if (postIsGood) {
-      return new WriteEvent({
-        input: "",
-        isGood: true,
-      });
-    }
-
-    return new WriteEvent({
-      input: `Improve the writing of a given blog post by using a given review.
-            Blog post:
-            \`\`\`
-            ${oldContent}
-            \`\`\`
-
-            Review:
-            \`\`\`
-            ${reviewResult}
-            \`\`\``,
-      isGood: false,
-    });
-  };
-
-  const publish = async (context: Context, ev: PublishEvent) => {
-    const publisher = await createPublisher(chatHistoryWithAgentMessages);
-
-    const publishResult = await runAgent(context, publisher, {
-      message: `${ev.data.input}`,
-      streaming: true,
-    });
-    return publishResult as unknown as StopEvent<
-      AsyncGenerator<ChatResponseChunk>
-    >;
-  };
-
-  const workflow = new Workflow({ timeout: TIMEOUT, validate: true });
-  workflow.addStep(StartEvent, start, {
-    outputs: [ResearchEvent, PublishEvent],
-  });
-  workflow.addStep(ResearchEvent, research, { outputs: WriteEvent });
-  workflow.addStep(WriteEvent, write, { outputs: [ReviewEvent, StopEvent] });
-  workflow.addStep(ReviewEvent, review, { outputs: WriteEvent });
-  workflow.addStep(PublishEvent, publish, { outputs: StopEvent });
-
-  return workflow;
-};

templates/components/multiagent/typescript/workflow/single-agent.ts

@@ -1,22 +1,21 @@
 import {
-  Context,
+  HandlerContext,
   StartEvent,
   StopEvent,
   Workflow,
   WorkflowEvent,
-} from "@llamaindex/core/workflow";
+} from "@llamaindex/workflow";
 import {
   BaseToolWithCall,
   ChatMemoryBuffer,
   ChatMessage,
-  ChatResponse,
   ChatResponseChunk,
+  QueryEngineTool,
   Settings,
   ToolCall,
   ToolCallLLM,
-  ToolCallLLMMessageOptions,
-  callTool,
 } from "llamaindex";
+import { callTools, chatWithTools } from "./tools";
 import { AgentInput, AgentRunEvent } from "./type";
 
 class InputEvent extends WorkflowEvent<{
@@ -27,11 +26,23 @@ class ToolCallEvent extends WorkflowEvent<{
   toolCalls: ToolCall[];
 }> {}
 
-export class FunctionCallingAgent extends Workflow {
+type FunctionCallingAgentContextData = {
+  streaming: boolean;
+};
+
+export type FunctionCallingAgentInput = AgentInput & {
+  displayName: string;
+};
+
+export class FunctionCallingAgent extends Workflow<
+  FunctionCallingAgentContextData,
+  FunctionCallingAgentInput,
+  string | AsyncGenerator<boolean | ChatResponseChunk<object>>
+> {
   name: string;
   llm: ToolCallLLM;
   memory: ChatMemoryBuffer;
-  tools: BaseToolWithCall[];
+  tools: BaseToolWithCall[] | QueryEngineTool[];
   systemPrompt?: string;
   writeEvents: boolean;
   role?: string;
@@ -53,7 +64,9 @@ export class FunctionCallingAgent extends Workflow {
     });
     this.name = options?.name;
     this.llm = options.llm ?? (Settings.llm as ToolCallLLM);
-    this.checkToolCallSupport();
+    if (!(this.llm instanceof ToolCallLLM)) {
+      throw new Error("LLM is not a ToolCallLLM");
+    }
     this.memory = new ChatMemoryBuffer({
       llm: this.llm,
       chatHistory: options.chatHistory,
@@ -64,173 +77,103 @@ export class FunctionCallingAgent extends Workflow {
     this.role = options?.role;
 
     // add steps
-    this.addStep(StartEvent<AgentInput>, this.prepareChatHistory, {
-      outputs: InputEvent,
-    });
-    this.addStep(InputEvent, this.handleLLMInput, {
-      outputs: [ToolCallEvent, StopEvent],
-    });
-    this.addStep(ToolCallEvent, this.handleToolCalls, {
-      outputs: InputEvent,
-    });
+    this.addStep(
+      {
+        inputs: [StartEvent<AgentInput>],
+        outputs: [InputEvent],
+      },
+      this.prepareChatHistory,
+    );
+    this.addStep(
+      {
+        inputs: [InputEvent],
+        outputs: [ToolCallEvent, StopEvent],
+      },
+      this.handleLLMInput,
+    );
+    this.addStep(
+      {
+        inputs: [ToolCallEvent],
+        outputs: [InputEvent],
+      },
+      this.handleToolCalls,
+    );
   }
 
   private get chatHistory() {
     return this.memory.getMessages();
   }
 
-  private async prepareChatHistory(
-    ctx: Context,
+  prepareChatHistory = async (
+    ctx: HandlerContext<FunctionCallingAgentContextData>,
     ev: StartEvent<AgentInput>,
-  ): Promise<InputEvent> {
-    const { message, streaming } = ev.data.input;
-    ctx.set("streaming", streaming);
+  ): Promise<InputEvent> => {
+    const { message, streaming } = ev.data;
+    ctx.data.streaming = streaming ?? false;
     this.writeEvent(`Start to work on: ${message}`, ctx);
     if (this.systemPrompt) {
       this.memory.put({ role: "system", content: this.systemPrompt });
     }
     this.memory.put({ role: "user", content: message });
     return new InputEvent({ input: this.chatHistory });
-  }
+  };
 
-  private async handleLLMInput(
-    ctx: Context,
+  handleLLMInput = async (
+    ctx: HandlerContext<FunctionCallingAgentContextData>,
     ev: InputEvent,
-  ): Promise<StopEvent<string | AsyncGenerator> | ToolCallEvent> {
-    if (ctx.get("streaming")) {
-      return await this.handleLLMInputStream(ctx, ev);
+  ): Promise<StopEvent<string | AsyncGenerator> | ToolCallEvent> => {
+    const toolCallResponse = await chatWithTools(
+      this.llm,
+      this.tools,
+      this.chatHistory,
+    );
+    if (toolCallResponse.toolCallMessage) {
+      this.memory.put(toolCallResponse.toolCallMessage);
     }
 
-    const result = await this.llm.chat({
-      messages: this.chatHistory,
-      tools: this.tools,
-    });
-    this.memory.put(result.message);
-
-    const toolCalls = this.getToolCallsFromResponse(result);
-    if (toolCalls.length) {
-      return new ToolCallEvent({ toolCalls });
+    if (toolCallResponse.hasToolCall()) {
+      return new ToolCallEvent({ toolCalls: toolCallResponse.toolCalls });
     }
-    this.writeEvent("Finished task", ctx);
-    return new StopEvent({ result: result.message.content.toString() });
-  }
 
-  private async handleLLMInputStream(
-    context: Context,
-    ev: InputEvent,
-  ): Promise<StopEvent<AsyncGenerator> | ToolCallEvent> {
-    const { llm, tools, memory } = this;
-    const llmArgs = { messages: this.chatHistory, tools };
-
-    const responseGenerator = async function* () {
-      const responseStream = await llm.chat({ ...llmArgs, stream: true });
-
-      let fullResponse = null;
-      let yieldedIndicator = false;
-      for await (const chunk of responseStream) {
-        const hasToolCalls = chunk.options && "toolCall" in chunk.options;
-        if (!hasToolCalls) {
-          if (!yieldedIndicator) {
-            yield false;
-            yieldedIndicator = true;
-          }
-          yield chunk;
-        } else if (!yieldedIndicator) {
-          yield true;
-          yieldedIndicator = true;
-        }
-
-        fullResponse = chunk;
+    if (ctx.data.streaming) {
+      if (!toolCallResponse.responseGenerator) {
+        throw new Error("No streaming response");
       }
-
-      if (fullResponse?.options && Object.keys(fullResponse.options).length) {
-        memory.put({
-          role: "assistant",
-          content: "",
-          options: fullResponse.options,
-        });
-        yield fullResponse;
-      }
-    };
-
-    const generator = responseGenerator();
-    const isToolCall = await generator.next();
-    if (isToolCall.value) {
-      const fullResponse = await generator.next();
-      const toolCalls = this.getToolCallsFromResponse(
-        fullResponse.value as ChatResponseChunk<ToolCallLLMMessageOptions>,
-      );
-      return new ToolCallEvent({ toolCalls });
+      return new StopEvent(toolCallResponse.responseGenerator);
     }
 
-    this.writeEvent("Finished task", context);
-    return new StopEvent({ result: generator });
-  }
+    const fullResponse = await toolCallResponse.asFullResponse();
+    this.memory.put(fullResponse);
+    return new StopEvent(fullResponse.content.toString());
+  };
 
-  private async handleToolCalls(
-    ctx: Context,
+  handleToolCalls = async (
+    ctx: HandlerContext<FunctionCallingAgentContextData>,
     ev: ToolCallEvent,
-  ): Promise<InputEvent> {
+  ): Promise<InputEvent> => {
     const { toolCalls } = ev.data;
 
-    const toolMsgs: ChatMessage[] = [];
-
-    for (const call of toolCalls) {
-      const targetTool = this.tools.find(
-        (tool) => tool.metadata.name === call.name,
-      );
-      // TODO: make logger optional in callTool in framework
-      const toolOutput = await callTool(targetTool, call, {
-        log: () => {},
-        error: console.error.bind(console),
-        warn: () => {},
-      });
-      toolMsgs.push({
-        content: JSON.stringify(toolOutput.output),
-        role: "user",
-        options: {
-          toolResult: {
-            result: toolOutput.output,
-            isError: toolOutput.isError,
-            id: call.id,
-          },
-        },
-      });
-    }
+    const toolMsgs = await callTools({
+      tools: this.tools,
+      toolCalls,
+      ctx,
+      agentName: this.name,
+    });
 
     for (const msg of toolMsgs) {
       this.memory.put(msg);
     }
 
     return new InputEvent({ input: this.memory.getMessages() });
-  }
+  };
 
-  private writeEvent(msg: string, context: Context) {
+  writeEvent = (
+    msg: string,
+    ctx: HandlerContext<FunctionCallingAgentContextData>,
+  ) => {
     if (!this.writeEvents) return;
-    context.writeEventToStream({
-      data: new AgentRunEvent({ name: this.name, msg }),
-    });
-  }
-
-  private checkToolCallSupport() {
-    const { supportToolCall } = this.llm as ToolCallLLM;
-    if (!supportToolCall) throw new Error("LLM does not support tool calls");
-  }
-
-  private getToolCallsFromResponse(
-    response:
-      | ChatResponse<ToolCallLLMMessageOptions>
-      | ChatResponseChunk<ToolCallLLMMessageOptions>,
-  ): ToolCall[] {
-    let options;
-    if ("message" in response) {
-      options = response.message.options;
-    } else {
-      options = response.options;
-    }
-    if (options && "toolCall" in options) {
-      return options.toolCall as ToolCall[];
-    }
-    return [];
-  }
+    ctx.sendEvent(
+      new AgentRunEvent({ agent: this.name, text: msg, type: "text" }),
+    );
+  };
 }

templates/components/multiagent/typescript/workflow/stream.ts

@@ -1,65 +1,77 @@
-import { StopEvent } from "@llamaindex/core/workflow";
 import {
-  createCallbacksTransformer,
-  createStreamDataTransformer,
+  StopEvent,
+  WorkflowContext,
+  WorkflowEvent,
+} from "@llamaindex/workflow";
+import {
   StreamData,
+  createStreamDataTransformer,
   trimStartOfStreamHelper,
-  type AIStreamCallbacksAndOptions,
 } from "ai";
 import { ChatResponseChunk } from "llamaindex";
 import { AgentRunEvent } from "./type";
 
-export function toDataStream(
-  result: Promise<StopEvent<AsyncGenerator<ChatResponseChunk>>>,
-  callbacks?: AIStreamCallbacksAndOptions,
-) {
-  return toReadableStream(result)
-    .pipeThrough(createCallbacksTransformer(callbacks))
-    .pipeThrough(createStreamDataTransformer());
-}
-
-function toReadableStream(
-  result: Promise<StopEvent<AsyncGenerator<ChatResponseChunk>>>,
-) {
+export async function createStreamFromWorkflowContext<Input, Output, Context>(
+  context: WorkflowContext<Input, Output, Context>,
+): Promise<{ stream: ReadableStream<string>; dataStream: StreamData }> {
   const trimStartOfStream = trimStartOfStreamHelper();
-  return new ReadableStream<string>({
-    start(controller) {
-      controller.enqueue(""); // Kickstart the stream
+  const dataStream = new StreamData();
+  const encoder = new TextEncoder();
+  let generator: AsyncGenerator<ChatResponseChunk> | undefined;
+
+  const closeStreams = (controller: ReadableStreamDefaultController) => {
+    controller.close();
+    dataStream.close();
+  };
+
+  const mainStream = new ReadableStream({
+    async start(controller) {
+      // Kickstart the stream by sending an empty string
+      controller.enqueue(encoder.encode(""));
     },
-    async pull(controller): Promise<void> {
-      const stopEvent = await result;
-      const generator = stopEvent.data.result;
-      const { value, done } = await generator.next();
+    async pull(controller) {
+      while (!generator) {
+        // get next event from workflow context
+        const { value: event, done } =
+          await context[Symbol.asyncIterator]().next();
+        if (done) {
+          closeStreams(controller);
+          return;
+        }
+        generator = handleEvent(event, dataStream);
+      }
+
+      const { value: chunk, done } = await generator.next();
       if (done) {
-        controller.close();
+        closeStreams(controller);
         return;
       }
-
-      const text = trimStartOfStream(value.delta ?? "");
-      if (text) controller.enqueue(text);
+      const text = trimStartOfStream(chunk.delta ?? "");
+      if (text) {
+        controller.enqueue(encoder.encode(text));
+      }
     },
   });
+
+  return {
+    stream: mainStream.pipeThrough(createStreamDataTransformer()),
+    dataStream,
+  };
 }
 
-export async function workflowEventsToStreamData(
-  events: AsyncIterable<AgentRunEvent>,
-): Promise<StreamData> {
-  const streamData = new StreamData();
-
-  (async () => {
-    for await (const event of events) {
-      if (event instanceof AgentRunEvent) {
-        const { name, msg } = event.data;
-        if ((streamData as any).isClosed) {
-          break;
-        }
-        streamData.appendMessageAnnotation({
-          type: "agent",
-          data: { agent: name, text: msg },
-        });
-      }
-    }
-  })();
-
-  return streamData;
+function handleEvent(
+  event: WorkflowEvent<any>,
+  dataStream: StreamData,
+): AsyncGenerator<ChatResponseChunk> | undefined {
+  // Handle for StopEvent
+  if (event instanceof StopEvent) {
+    return event.data as AsyncGenerator<ChatResponseChunk>;
+  }
+  // Handle for AgentRunEvent
+  if (event instanceof AgentRunEvent) {
+    dataStream.appendMessageAnnotation({
+      type: "agent",
+      data: event.data,
+    });
+  }
 }

templates/components/multiagent/typescript/workflow/tools.ts

@@ -1,54 +1,342 @@
-import fs from "fs/promises";
-import { BaseToolWithCall, QueryEngineTool } from "llamaindex";
-import path from "path";
+import { HandlerContext } from "@llamaindex/workflow";
+import {
+  BaseToolWithCall,
+  callTool,
+  ChatMessage,
+  ChatResponse,
+  ChatResponseChunk,
+  LlamaCloudIndex,
+  PartialToolCall,
+  QueryEngineTool,
+  ToolCall,
+  ToolCallLLM,
+  ToolCallLLMMessageOptions,
+} from "llamaindex";
+import crypto from "node:crypto";
 import { getDataSource } from "../engine";
-import { createTools } from "../engine/tools/index";
+import { AgentRunEvent } from "./type";
 
-export const getQueryEngineTool = async (
-  params?: any,
-): Promise<QueryEngineTool | null> => {
-  const index = await getDataSource(params);
+export const getQueryEngineTools = async (): Promise<
+  QueryEngineTool[] | null
+> => {
+  const topK = process.env.TOP_K ? parseInt(process.env.TOP_K) : undefined;
+
+  const index = await getDataSource();
   if (!index) {
     return null;
   }
+  // index is LlamaCloudIndex use two query engine tools
+  if (index instanceof LlamaCloudIndex) {
+    return [
+      new QueryEngineTool({
+        queryEngine: index.asQueryEngine({
+          similarityTopK: topK,
+          retrieval_mode: "files_via_content",
+        }),
+        metadata: {
+          name: "document_retriever",
+          description: `Document retriever that retrieves entire documents from the corpus.
+  ONLY use for research questions that may require searching over entire research reports.
+  Will be slower and more expensive than chunk-level retrieval but may be necessary.`,
+        },
+      }),
+      new QueryEngineTool({
+        queryEngine: index.asQueryEngine({
+          similarityTopK: topK,
+          retrieval_mode: "chunks",
+        }),
+        metadata: {
+          name: "chunk_retriever",
+          description: `Retrieves a small set of relevant document chunks from the corpus.
+      Use for research questions that want to look up specific facts from the knowledge corpus,
+      and need entire documents.`,
+        },
+      }),
+    ];
+  } else {
+    return [
+      new QueryEngineTool({
+        queryEngine: index.asQueryEngine({
+          similarityTopK: topK,
+        }),
+        metadata: {
+          name: "retriever",
+          description: `Use this tool to retrieve information about the text corpus from the index.`,
+        },
+      }),
+    ];
+  }
+};
 
-  const topK = process.env.TOP_K ? parseInt(process.env.TOP_K) : undefined;
-  return new QueryEngineTool({
-    queryEngine: index.asQueryEngine({
-      similarityTopK: topK,
-    }),
-    metadata: {
-      name: "query_index",
-      description: `Use this tool to retrieve information about the text corpus from the index.`,
+/**
+ * Call multiple tools and return the tool messages
+ */
+export const callTools = async <T>({
+  tools,
+  toolCalls,
+  ctx,
+  agentName,
+  writeEvent = true,
+}: {
+  toolCalls: ToolCall[];
+  tools: BaseToolWithCall[];
+  ctx: HandlerContext<T>;
+  agentName: string;
+  writeEvent?: boolean;
+}): Promise<ChatMessage[]> => {
+  const toolMsgs: ChatMessage[] = [];
+  if (toolCalls.length === 0) {
+    return toolMsgs;
+  }
+  if (toolCalls.length === 1) {
+    const tool = tools.find((tool) => tool.metadata.name === toolCalls[0].name);
+    if (!tool) {
+      throw new Error(`Tool ${toolCalls[0].name} not found`);
+    }
+    return [
+      await callSingleTool(
+        tool,
+        toolCalls[0],
+        writeEvent
+          ? (msg: string) => {
+              ctx.sendEvent(
+                new AgentRunEvent({
+                  agent: agentName,
+                  text: msg,
+                  type: "text",
+                }),
+              );
+            }
+          : undefined,
+      ),
+    ];
+  }
+  // Multiple tool calls, show events in progress
+  const progressId = crypto.randomUUID();
+  const totalSteps = toolCalls.length;
+  let currentStep = 0;
+  for (const toolCall of toolCalls) {
+    const tool = tools.find((tool) => tool.metadata.name === toolCall.name);
+    if (!tool) {
+      throw new Error(`Tool ${toolCall.name} not found`);
+    }
+    const toolMsg = await callSingleTool(tool, toolCall, (msg: string) => {
+      ctx.sendEvent(
+        new AgentRunEvent({
+          agent: agentName,
+          text: msg,
+          type: "progress",
+          data: {
+            id: progressId,
+            total: totalSteps,
+            current: currentStep,
+          },
+        }),
+      );
+      currentStep++;
+    });
+    toolMsgs.push(toolMsg);
+  }
+  return toolMsgs;
+};
+
+export const callSingleTool = async (
+  tool: BaseToolWithCall,
+  toolCall: ToolCall,
+  eventEmitter?: (msg: string) => void,
+): Promise<ChatMessage> => {
+  if (eventEmitter) {
+    eventEmitter(
+      `Calling tool ${toolCall.name} with input: ${JSON.stringify(toolCall.input)}`,
+    );
+  }
+
+  const toolOutput = await callTool(tool, toolCall, {
+    log: () => {},
+    error: (...args: unknown[]) => {
+      console.error(`Tool ${toolCall.name} got error:`, ...args);
+      if (eventEmitter) {
+        eventEmitter(`Tool ${toolCall.name} got error: ${args.join(" ")}`);
+      }
+      return {
+        content: JSON.stringify({
+          error: args.join(" "),
+        }),
+        role: "user",
+        options: {
+          toolResult: {
+            id: toolCall.id,
+            result: JSON.stringify({
+              error: args.join(" "),
+            }),
+            isError: true,
+          },
+        },
+      };
     },
+    warn: () => {},
+  });
+
+  return {
+    content: JSON.stringify(toolOutput.output),
+    role: "user",
+    options: {
+      toolResult: {
+        result: toolOutput.output,
+        isError: toolOutput.isError,
+        id: toolCall.id,
+      },
+    },
+  };
+};
+
+class ChatWithToolsResponse {
+  toolCalls: ToolCall[];
+  toolCallMessage?: ChatMessage;
+  responseGenerator?: AsyncGenerator<ChatResponseChunk>;
+
+  constructor(options: {
+    toolCalls: ToolCall[];
+    toolCallMessage?: ChatMessage;
+    responseGenerator?: AsyncGenerator<ChatResponseChunk>;
+  }) {
+    this.toolCalls = options.toolCalls;
+    this.toolCallMessage = options.toolCallMessage;
+    this.responseGenerator = options.responseGenerator;
+  }
+
+  hasMultipleTools() {
+    const uniqueToolNames = new Set(this.getToolNames());
+    return uniqueToolNames.size > 1;
+  }
+
+  hasToolCall() {
+    return this.toolCalls.length > 0;
+  }
+
+  getToolNames() {
+    return this.toolCalls.map((toolCall) => toolCall.name);
+  }
+
+  async asFullResponse(): Promise<ChatMessage> {
+    if (!this.responseGenerator) {
+      throw new Error("No response generator");
+    }
+    let fullResponse = "";
+    for await (const chunk of this.responseGenerator) {
+      fullResponse += chunk.delta;
+    }
+    return {
+      role: "assistant",
+      content: fullResponse,
+    };
+  }
+}
+
+export const chatWithTools = async (
+  llm: ToolCallLLM,
+  tools: BaseToolWithCall[],
+  messages: ChatMessage[],
+): Promise<ChatWithToolsResponse> => {
+  const responseGenerator = async function* (): AsyncGenerator<
+    boolean | ChatResponseChunk,
+    void,
+    unknown
+  > {
+    const responseStream = await llm.chat({ messages, tools, stream: true });
+
+    let fullResponse = null;
+    let yieldedIndicator = false;
+    const toolCallMap = new Map();
+    for await (const chunk of responseStream) {
+      const hasToolCalls = chunk.options && "toolCall" in chunk.options;
+      if (!hasToolCalls) {
+        if (!yieldedIndicator) {
+          yield false;
+          yieldedIndicator = true;
+        }
+        yield chunk;
+      } else if (!yieldedIndicator) {
+        yield true;
+        yieldedIndicator = true;
+      }
+
+      if (chunk.options && "toolCall" in chunk.options) {
+        for (const toolCall of chunk.options.toolCall as PartialToolCall[]) {
+          if (toolCall.id) {
+            toolCallMap.set(toolCall.id, toolCall);
+          }
+        }
+      }
+
+      if (
+        hasToolCalls &&
+        (chunk.raw as any)?.choices?.[0]?.finish_reason !== null
+      ) {
+        // Update the fullResponse with the tool calls
+        const toolCalls = Array.from(toolCallMap.values());
+        fullResponse = {
+          ...chunk,
+          options: {
+            ...chunk.options,
+            toolCall: toolCalls,
+          },
+        };
+      }
+    }
+
+    if (fullResponse) {
+      yield fullResponse;
+    }
+  };
+
+  const generator = responseGenerator();
+  const isToolCall = await generator.next();
+
+  if (isToolCall.value) {
+    // If it's a tool call, we need to wait for the full response
+    let fullResponse = null;
+    for await (const chunk of generator) {
+      fullResponse = chunk;
+    }
+
+    if (fullResponse) {
+      const responseChunk = fullResponse as ChatResponseChunk;
+      const toolCalls = getToolCallsFromResponse(responseChunk);
+      return new ChatWithToolsResponse({
+        toolCalls,
+        toolCallMessage: {
+          options: responseChunk.options,
+          role: "assistant",
+          content: "",
+        },
+      });
+    } else {
+      throw new Error("Cannot get tool calls from response");
+    }
+  }
+
+  return new ChatWithToolsResponse({
+    toolCalls: [],
+    responseGenerator: generator as AsyncGenerator<ChatResponseChunk>,
   });
 };
 
-export const getAvailableTools = async () => {
-  const configFile = path.join("config", "tools.json");
-  let toolConfig: any;
-  const tools: BaseToolWithCall[] = [];
-  try {
-    toolConfig = JSON.parse(await fs.readFile(configFile, "utf8"));
-  } catch (e) {
-    console.info(`Could not read ${configFile} file. Using no tools.`);
-  }
-  if (toolConfig) {
-    tools.push(...(await createTools(toolConfig)));
-  }
-  const queryEngineTool = await getQueryEngineTool();
-  if (queryEngineTool) {
-    tools.push(queryEngineTool);
+export const getToolCallsFromResponse = (
+  response:
+    | ChatResponse<ToolCallLLMMessageOptions>
+    | ChatResponseChunk<ToolCallLLMMessageOptions>,
+): ToolCall[] => {
+  let options;
+
+  if ("message" in response) {
+    options = response.message.options;
+  } else {
+    options = response.options;
   }
 
-  return tools;
-};
-
-export const lookupTools = async (
-  toolNames: string[],
-): Promise<BaseToolWithCall[]> => {
-  const availableTools = await getAvailableTools();
-  return availableTools.filter((tool) =>
-    toolNames.includes(tool.metadata.name),
-  );
+  if (options && "toolCall" in options) {
+    return options.toolCall as ToolCall[];
+  }
+  return [];
 };

templates/components/multiagent/typescript/workflow/type.ts

@@ -1,11 +1,24 @@
-import { WorkflowEvent } from "@llamaindex/core/workflow";
+import { WorkflowEvent } from "@llamaindex/workflow";
+import { MessageContent } from "llamaindex";
 
 export type AgentInput = {
-  message: string;
+  message: MessageContent;
   streaming?: boolean;
 };
 
+export type AgentRunEventType = "text" | "progress";
+
+export type ProgressEventData = {
+  id: string;
+  total: number;
+  current: number;
+};
+
+export type AgentRunEventData = ProgressEventData;
+
 export class AgentRunEvent extends WorkflowEvent<{
-  name: string;
-  msg: string;
+  agent: string;
+  text: string;
+  type: AgentRunEventType;
+  data?: AgentRunEventData;
 }> {}

templates/components/routers/python/sandbox.py

@@ -16,11 +16,12 @@ import base64
 import logging
 import os
 import uuid
-from typing import Dict, List, Optional, Union
+from dataclasses import asdict
+from typing import Any, Dict, List, Optional, Union
 
 from app.engine.tools.artifact import CodeArtifact
-from app.engine.utils.file_helper import save_file
-from e2b_code_interpreter import CodeInterpreter, Sandbox
+from app.services.file import FileService
+from e2b_code_interpreter import CodeInterpreter, Sandbox  # type: ignore
 from fastapi import APIRouter, HTTPException, Request
 from pydantic import BaseModel
 
@@ -36,7 +37,7 @@ class ExecutionResult(BaseModel):
     template: str
     stdout: List[str]
     stderr: List[str]
-    runtime_error: Optional[Dict[str, Union[str, List[str]]]] = None
+    runtime_error: Optional[Dict[str, Any]] = None
     output_urls: List[Dict[str, str]]
     url: Optional[str]
 
@@ -54,15 +55,27 @@ class ExecutionResult(BaseModel):
         }
 
 
+class FileUpload(BaseModel):
+    id: str
+    name: str
+
+
 @sandbox_router.post("")
 async def create_sandbox(request: Request):
     request_data = await request.json()
+    artifact_data = request_data.get("artifact", None)
+    sandbox_files = artifact_data.get("files", [])
+
+    if not artifact_data:
+        raise HTTPException(
+            status_code=400, detail="Could not create artifact from the request data"
+        )
 
     try:
-        artifact = CodeArtifact(**request_data["artifact"])
+        artifact = CodeArtifact(**artifact_data)
     except Exception:
         logger.error(f"Could not create artifact from request data: {request_data}")
-        return HTTPException(
+        raise HTTPException(
             status_code=400, detail="Could not create artifact from the request data"
         )
 
@@ -94,6 +107,10 @@ async def create_sandbox(request: Request):
                 f"Installed dependencies: {', '.join(artifact.additional_dependencies)} in sandbox {sbx}"
             )
 
+    # Copy files
+    if len(sandbox_files) > 0:
+        _upload_files(sbx, sandbox_files)
+
     # Copy code to disk
     if isinstance(artifact.code, list):
         for file in artifact.code:
@@ -107,11 +124,12 @@ async def create_sandbox(request: Request):
     if artifact.template == "code-interpreter-multilang":
         result = sbx.notebook.exec_cell(artifact.code or "")
         output_urls = _download_cell_results(result.results)
+        runtime_error = asdict(result.error) if result.error else None
         return ExecutionResult(
             template=artifact.template,
             stdout=result.logs.stdout,
             stderr=result.logs.stderr,
-            runtime_error=result.error,
+            runtime_error=runtime_error,
             output_urls=output_urls,
             url=None,
         ).to_response()
@@ -126,6 +144,19 @@ async def create_sandbox(request: Request):
         ).to_response()
 
 
+def _upload_files(
+    sandbox: Union[CodeInterpreter, Sandbox],
+    sandbox_files: List[str] = [],
+) -> None:
+    for file_path in sandbox_files:
+        file_name = os.path.basename(file_path)
+        local_file_path = os.path.join("output", "uploaded", file_name)
+        with open(local_file_path, "rb") as f:
+            content = f.read()
+            sandbox.files.write(file_path, content)
+    return None
+
+
 def _download_cell_results(cell_results: Optional[List]) -> List[Dict[str, str]]:
     """
     To pull results from code interpreter cell and save them to disk for serving
@@ -141,14 +172,18 @@ def _download_cell_results(cell_results: Optional[List]) -> List[Dict[str, str]]
                 data = result[ext]
 
                 if ext in ["png", "svg", "jpeg", "pdf"]:
-                    file_path = f"output/tools/{uuid.uuid4()}.{ext}"
                     base64_data = data
                     buffer = base64.b64decode(base64_data)
-                    file_meta = save_file(content=buffer, file_path=file_path)
+                    file_name = f"{uuid.uuid4()}.{ext}"
+                    file_meta = FileService.save_file(
+                        content=buffer,
+                        file_name=file_name,
+                        save_dir=os.path.join("output", "tools"),
+                    )
                     output.append(
                         {
                             "type": ext,
-                            "filename": file_meta.filename,
+                            "filename": file_meta.name,
                             "url": file_meta.url,
                         }
                     )

templates/components/services/python/file.py

@@ -1,124 +0,0 @@
-import base64
-import mimetypes
-import os
-from io import BytesIO
-from pathlib import Path
-from typing import List, Optional, Tuple
-
-from app.engine.index import IndexConfig, get_index
-from llama_index.core import VectorStoreIndex
-from llama_index.core.ingestion import IngestionPipeline
-from llama_index.core.readers.file.base import (
-    _try_loading_included_file_formats as get_file_loaders_map,
-)
-from llama_index.core.schema import Document
-from llama_index.indices.managed.llama_cloud.base import LlamaCloudIndex
-from llama_index.readers.file import FlatReader
-
-
-def get_llamaparse_parser():
-    from app.engine.loaders import load_configs
-    from app.engine.loaders.file import FileLoaderConfig, llama_parse_parser
-
-    config = load_configs()
-    file_loader_config = FileLoaderConfig(**config["file"])
-    if file_loader_config.use_llama_parse:
-        return llama_parse_parser()
-    else:
-        return None
-
-
-def default_file_loaders_map():
-    default_loaders = get_file_loaders_map()
-    default_loaders[".txt"] = FlatReader
-    return default_loaders
-
-
-class PrivateFileService:
-    PRIVATE_STORE_PATH = "output/uploaded"
-
-    @staticmethod
-    def preprocess_base64_file(base64_content: str) -> Tuple[bytes, str | None]:
-        header, data = base64_content.split(",", 1)
-        mime_type = header.split(";")[0].split(":", 1)[1]
-        extension = mimetypes.guess_extension(mime_type)
-        # File data as bytes
-        return base64.b64decode(data), extension
-
-    @staticmethod
-    def store_and_parse_file(file_name, file_data, extension) -> List[Document]:
-        # Store file to the private directory
-        os.makedirs(PrivateFileService.PRIVATE_STORE_PATH, exist_ok=True)
-        file_path = Path(os.path.join(PrivateFileService.PRIVATE_STORE_PATH, file_name))
-
-        # write file
-        with open(file_path, "wb") as f:
-            f.write(file_data)
-
-        # Load file to documents
-        # If LlamaParse is enabled, use it to parse the file
-        # Otherwise, use the default file loaders
-        reader = get_llamaparse_parser()
-        if reader is None:
-            reader_cls = default_file_loaders_map().get(extension)
-            if reader_cls is None:
-                raise ValueError(f"File extension {extension} is not supported")
-            reader = reader_cls()
-        documents = reader.load_data(file_path)
-        # Add custom metadata
-        for doc in documents:
-            doc.metadata["file_name"] = file_name
-            doc.metadata["private"] = "true"
-        return documents
-
-    @staticmethod
-    def process_file(
-        file_name: str, base64_content: str, params: Optional[dict] = None
-    ) -> List[str]:
-        if params is None:
-            params = {}
-
-        file_data, extension = PrivateFileService.preprocess_base64_file(base64_content)
-
-        # Add the nodes to the index and persist it
-        index_config = IndexConfig(**params)
-        current_index = get_index(index_config)
-
-        # Insert the documents into the index
-        if isinstance(current_index, LlamaCloudIndex):
-            from app.engine.service import LLamaCloudFileService
-
-            project_id = current_index._get_project_id()
-            pipeline_id = current_index._get_pipeline_id()
-            # LlamaCloudIndex is a managed index so we can directly use the files
-            upload_file = (file_name, BytesIO(file_data))
-            return [
-                LLamaCloudFileService.add_file_to_pipeline(
-                    project_id,
-                    pipeline_id,
-                    upload_file,
-                    custom_metadata={
-                        # Set private=true to mark the document as private user docs (required for filtering)
-                        "private": "true",
-                    },
-                )
-            ]
-        else:
-            # First process documents into nodes
-            documents = PrivateFileService.store_and_parse_file(
-                file_name, file_data, extension
-            )
-            pipeline = IngestionPipeline()
-            nodes = pipeline.run(documents=documents)
-
-            # Add the nodes to the index and persist it
-            if current_index is None:
-                current_index = VectorStoreIndex(nodes=nodes)
-            else:
-                current_index.insert_nodes(nodes=nodes)
-            current_index.storage_context.persist(
-                persist_dir=os.environ.get("STORAGE_DIR", "storage")
-            )
-
-            # Return the document ids
-            return [doc.doc_id for doc in documents]