Compare commits

..

7 Commits

Author SHA1 Message Date
Marcus Schiesser 2234d8ff6d chore: update lock 2025-05-09 13:54:38 +07:00
Marcus Schiesser ca2405811b fix: using new workflow with npm by forcing to use the current version of core in llamaindex (and not the one from the old workflow) 2025-05-09 12:11:42 +07:00
Marcus Schiesser 9b2e25a184 fix: Use Uint8Array instead of Buffer for file type messages (works w… (#1921) 2025-05-08 13:19:59 +07:00
github-actions[bot] b29521bf6c Release @llamaindex/google@0.2.6 (#1918)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-05-07 16:31:58 +07:00
Marcus Schiesser 73e25787e7 feat: add gemini-2.5-pro-preview-05-06 (#1917) 2025-05-07 16:18:21 +07:00
Marcus Schiesser 3ce80540fe docs: add workflows documentation and update installation instruction… (#1916) 2025-05-07 15:22:08 +07:00
Marcus Schiesser dbc1ee3089 docs: update installation instructions for LlamaIndex to include Work… (#1915) 2025-05-07 12:31:48 +07:00
34 changed files with 302 additions and 203 deletions
+10
View File
@@ -0,0 +1,10 @@
---
"@llamaindex/anthropic": patch
"@llamaindex/google": patch
"@llamaindex/openai": patch
"@llamaindex/core": patch
"@llamaindex/env": patch
"@llamaindex/examples": patch
---
Use Uint8Array instead of Buffer for file type messages (works with non-NodeJS)
+5
View File
@@ -0,0 +1,5 @@
---
"llamaindex": patch
---
Fix using new workflow with npm by forcing to use the current version of core in llamaindex (and not the one from the old workflow)
@@ -9,10 +9,10 @@ To install llamaindex, run the following command:
npm i llamaindex
```
In most cases, you'll also need an LLM package to use LlamaIndex. For example, to use the OpenAI LLM, you would install the following:
In most cases, you'll also need an LLM package and the Workflow package to use LlamaIndex. For example, to use the OpenAI LLM with agents, you would install the following:
```package-install
npm i @llamaindex/openai
npm i @llamaindex/openai @llamaindex/workflow
```
Go to [LLM APIs](/docs/llamaindex/modules/models/llms) to find out how to use other LLMs.
@@ -6,171 +6,13 @@ A `Workflow` in LlamaIndex is a lightweight, event-driven abstraction used to ch
Workflows are designed to be flexible and can be used to build agents, RAG flows, extraction flows, or anything else you want to implement.
To use workflows install this package:
```package-install
npm i @llamaindex/workflow @llamaindex/openai
npm i @llamaindex/workflow
```
## Getting Started
This package is a stable, production-ready version of our [llama-flow](../../../llamaflow) project.
Let's explore a simple workflow example where a joke is generated and then critiqued and iterated on:
While you can still reference the llama-flow documentation for detailed information about the underlying concepts, we recommend using the `@llamaindex/workflow` package for all new projects to ensure stability and long-term availability.
<include cwd>../../examples/agents/workflow/joke.ts</include>
There are a few moving pieces here, so let's go through this step by step.
### Defining Workflow Events
```typescript
const startEvent = workflowEvent<string>(); // Input topic for joke
const jokeEvent = workflowEvent<{ joke: string }>(); // Intermediate joke
const critiqueEvent = workflowEvent<{ joke: string; critique: string }>(); // Intermediate critique
const resultEvent = workflowEvent<{ joke: string; critique: string }>(); // Final joke + critique
```
Events are defined using the `workflowEvent` function and contain arbitrary data provided as a generic type. In this example, we have four events:
- `startEvent`: Takes a string input (the joke topic)
- `jokeEvent`: Contains an object with a joke property
- `critiqueEvent`: Contains both the joke and its critique, used for the feedback loop
- `resultEvent`: Contains the final joke and critique after any iterations
### Setting up the Workflow with Stateful Middleware
```typescript
const { withState, getContext } = createStatefulMiddleware(() => ({
numIterations: 0,
maxIterations: 3,
}));
const jokeFlow = withState(createWorkflow());
```
Our workflow is implemented using the `createWorkflow()` function, enhanced with the `withState` middleware. This middleware provides shared state across all handlers, which in this case tracks:
- `numIterations`: Counts how many iterations of joke improvement we've done
- `maxIterations`: Sets a limit to prevent infinite loops
This state will be accessible within workflows by using the `getContext().state` function.
### Adding Handlers with Loops
We have three key handlers in our workflow:
1. The first handler processes the `startEvent`, generates an initial joke, and emits a `jokeEvent`:
```typescript
jokeFlow.handle([startEvent], async (event) => {
// Prompt the LLM to write a joke
const prompt = `Write your best joke about ${event.data}. Write the joke between <joke> and </joke> tags.`;
const response = await llm.complete({ prompt });
// Parse the joke from the response
const joke =
response.text.match(/<joke>([\s\S]*?)<\/joke>/)?.[1]?.trim() ??
response.text;
return jokeEvent.with({ joke: joke });
});
```
2. The second handler handles the `jokeEvent`, critiques the joke, and either:
- Emits a `critiqueEvent` if the joke needs improvement
- Emits a `resultEvent` if the joke is good enough
```typescript
jokeFlow.handle([jokeEvent], async (event) => {
// Prompt the LLM to critique the joke
const prompt = `Give a thorough critique of the following joke. If the joke needs improvement, put "IMPROVE" somewhere in the critique: ${event.data.joke}`;
const response = await llm.complete({ prompt });
// If the critique includes "IMPROVE", keep iterating, else, return the result
if (response.text.includes("IMPROVE")) {
return critiqueEvent.with({
joke: event.data.joke,
critique: response.text,
});
}
return resultEvent.with({ joke: event.data.joke, critique: response.text });
});
```
3. The third handler processes the `critiqueEvent`, generates an improved joke based on the critique, and either:
- Loops back to the joke evaluation (if under the iteration limit)
- Emits the final `resultEvent` (if iteration limit reached)
```typescript
jokeFlow.handle([critiqueEvent], async (event) => {
// Keep track of the number of iterations
const state = getContext().state;
state.numIterations++;
// Write a new joke based on the previous joke and critique
const prompt = `Write a new joke based on the following critique and the original joke. Write the joke between <joke> and </joke> tags.\n\nJoke: ${event.data.joke}\n\nCritique: ${event.data.critique}`;
const response = await llm.complete({ prompt });
// Parse the joke from the response
const joke =
response.text.match(/<joke>([\s\S]*?)<\/joke>/)?.[1]?.trim() ??
response.text;
// If we've done less than the max number of iterations, keep iterating
// else, return the result
if (state.numIterations < state.maxIterations) {
return jokeEvent.with({ joke: joke });
}
return resultEvent.with({ joke: joke, critique: event.data.critique });
});
```
### Running the Workflow
```typescript
async function main() {
const { stream, sendEvent } = jokeFlow.createContext();
sendEvent(startEvent.with("pirates"));
let result: { joke: string, critique: string } | undefined;
for await (const event of stream) {
// console.log(event.data); optionally log the event data
if (resultEvent.include(event)) {
result = event.data;
break; // Stop when we get the final result
}
}
console.log(result);
}
```
To run the workflow, we:
1. Create a workflow context with `createContext()`
2. Trigger the initial event with `sendEvent()`
3. Listen to the event stream and process events as they arrive
4. Use `include()` to check if an event is of a specific type
5. Break the loop when we receive our final result
### Using Stream Utilities
The `stream` returned by `createContext` contains utility functions to make working with event streams easier:
```typescript
// Create a workflow context and send the initial event
const { stream, sendEvent } = jokeFlow.createContext();
sendEvent(startEvent.with("pirates"));
// Collect all events until we get a resultEvent
const allEvents = await stream.until(resultEvent).toArray();
// The last event will be the resultEvent
const finalEvent = allEvents.at(-1);
console.log(finalEvent.data); // Output the joke and critique
```
The stream utilities make it easier to work with the asynchronous event flow. In this example, we use:
- `toArray`: Aggregates all events into an array
- `until`: Creates a stream that emits events until a condition is met (in this case, until a resultEvent is received)
You can combine these utilities with other stream operators like `filter` and `map` to create powerful processing pipelines.
## Next Steps
To learn more about workflows, check out [the documentation in the tutorial section](../../../llamaflow).
@@ -8,9 +8,10 @@ We have a comprehensive, step-by-step [guide to building agents in LlamaIndex.TS
In a new folder:
```bash npm2yarn
```package-install
npm init
npm i -D typescript @types/node
npm i @llamaindex/openai @llamaindex/workflow llamaindex zod
```
## Run agent
@@ -20,15 +21,14 @@ Create the file `example.ts`. This code will:
- Create two tools for use by the agent:
- A `sumNumbers` tool that adds two numbers
- A `divideNumbers` tool that divides numbers
-
- Give an example of the data structure we wish to generate
- Prompt the LLM with instructions and the example, plus a sample transcript
<include cwd>../../examples/agent/openai.ts</include>
<include cwd>../../examples/agents/agent/openai.ts</include>
To run the code:
```bash
```package-install
npx tsx example.ts
```
@@ -36,9 +36,18 @@ You should expect output something like:
```
{
content: 'The sum of 5 + 5 is 10. When you divide 10 by 2, you get 5.',
role: 'assistant',
options: {}
result: '5 + 5 is 10. Then, 10 divided by 2 is 5.',
state: {
memory: ChatMemoryBuffer {
chatStore: SimpleChatStore {},
chatStoreKey: 'chat_history',
tokenLimit: 750000
},
scratchpad: [],
currentAgentName: 'Agent',
agents: [ 'Agent' ],
nextAgentName: null
}
}
Done
```
@@ -4,7 +4,7 @@
"basic_agent",
"rag",
"agents",
"../../llamaflow",
"workflows",
"local_llm",
"chatbot",
"structured_data_extraction"
@@ -16,7 +16,7 @@ LlamaIndex uses a two stage method when using an LLM with your data:
1. **indexing stage**: preparing a knowledge base, and
2. **querying stage**: retrieving relevant context from the knowledge to assist the LLM in responding to a question
![](./_static/concepts/rag.jpg)
![](/_static/concepts/rag.jpg)
This process is also known as Retrieval Augmented Generation (RAG).
@@ -28,7 +28,7 @@ Let's explore each stage in detail.
LlamaIndex.TS help you prepare the knowledge base with a suite of data connectors and indexes.
![](./_static/concepts/indexing.jpg)
![](/_static/concepts/indexing.jpg)
[**Data Loaders**](/docs/llamaindex/modules/data/readers):
A data connector (i.e. `Reader`) ingest data from different data sources and data formats into a simple `Document` representation (text and simple metadata).
@@ -54,7 +54,7 @@ LlamaIndex provides composable modules that help you build and integrate RAG pip
These building blocks can be customized to reflect ranking preferences, as well as composed to reason over multiple knowledge bases in a structured way.
![](./_static/concepts/querying.jpg)
![](/_static/concepts/querying.jpg)
#### Building Blocks
@@ -8,9 +8,10 @@ One of the most common use-cases for LlamaIndex is Retrieval-Augmented Generatio
In a new folder, run:
```bash npm2yarn
```package-install
npm init
npm i -D typescript @types/node
npm i llamaindex
```
Then, check out the [installation](/docs/llamaindex/getting_started/installation) steps to install LlamaIndex.TS and prepare an OpenAI key.
@@ -34,7 +35,7 @@ Create a `tsconfig.json` file in the same folder:
Now you can run the code with
```bash
```package-install
npx tsx example.ts
```
@@ -10,9 +10,10 @@ You can use [other LLMs](/docs/llamaindex/modules/models/llms) via their APIs; i
In a new folder:
```bash npm2yarn
```package-install
npm init
npm i -D typescript @types/node
npm i @llamaindex/openai zod
```
## Extract data
@@ -27,7 +28,7 @@ Create the file `example.ts`. This code will:
To run the code:
```bash
```package-install
npx tsx example.ts
```
@@ -0,0 +1,176 @@
---
title: Workflows
---
A `Workflow` in LlamaIndex is a lightweight, event-driven abstraction used to chain together several events. Workflows are made up of `handlers`, with each one responsible for processing specific event types and emitting new events.
Workflows are designed to be flexible and can be used to build agents, RAG flows, extraction flows, or anything else you want to implement.
```package-install
npm i @llamaindex/workflow @llamaindex/openai
```
## Getting Started
Let's explore a simple workflow example where a joke is generated and then critiqued and iterated on:
<include cwd>../../examples/agents/workflow/joke.ts</include>
There are a few moving pieces here, so let's go through this step by step.
### Defining Workflow Events
```typescript
const startEvent = workflowEvent<string>(); // Input topic for joke
const jokeEvent = workflowEvent<{ joke: string }>(); // Intermediate joke
const critiqueEvent = workflowEvent<{ joke: string; critique: string }>(); // Intermediate critique
const resultEvent = workflowEvent<{ joke: string; critique: string }>(); // Final joke + critique
```
Events are defined using the `workflowEvent` function and contain arbitrary data provided as a generic type. In this example, we have four events:
- `startEvent`: Takes a string input (the joke topic)
- `jokeEvent`: Contains an object with a joke property
- `critiqueEvent`: Contains both the joke and its critique, used for the feedback loop
- `resultEvent`: Contains the final joke and critique after any iterations
### Setting up the Workflow with Stateful Middleware
```typescript
const { withState, getContext } = createStatefulMiddleware(() => ({
numIterations: 0,
maxIterations: 3,
}));
const jokeFlow = withState(createWorkflow());
```
Our workflow is implemented using the `createWorkflow()` function, enhanced with the `withState` middleware. This middleware provides shared state across all handlers, which in this case tracks:
- `numIterations`: Counts how many iterations of joke improvement we've done
- `maxIterations`: Sets a limit to prevent infinite loops
This state will be accessible within workflows by using the `getContext().state` function.
### Adding Handlers with Loops
We have three key handlers in our workflow:
1. The first handler processes the `startEvent`, generates an initial joke, and emits a `jokeEvent`:
```typescript
jokeFlow.handle([startEvent], async (event) => {
// Prompt the LLM to write a joke
const prompt = `Write your best joke about ${event.data}. Write the joke between <joke> and </joke> tags.`;
const response = await llm.complete({ prompt });
// Parse the joke from the response
const joke =
response.text.match(/<joke>([\s\S]*?)<\/joke>/)?.[1]?.trim() ??
response.text;
return jokeEvent.with({ joke: joke });
});
```
2. The second handler handles the `jokeEvent`, critiques the joke, and either:
- Emits a `critiqueEvent` if the joke needs improvement
- Emits a `resultEvent` if the joke is good enough
```typescript
jokeFlow.handle([jokeEvent], async (event) => {
// Prompt the LLM to critique the joke
const prompt = `Give a thorough critique of the following joke. If the joke needs improvement, put "IMPROVE" somewhere in the critique: ${event.data.joke}`;
const response = await llm.complete({ prompt });
// If the critique includes "IMPROVE", keep iterating, else, return the result
if (response.text.includes("IMPROVE")) {
return critiqueEvent.with({
joke: event.data.joke,
critique: response.text,
});
}
return resultEvent.with({ joke: event.data.joke, critique: response.text });
});
```
3. The third handler processes the `critiqueEvent`, generates an improved joke based on the critique, and either:
- Loops back to the joke evaluation (if under the iteration limit)
- Emits the final `resultEvent` (if iteration limit reached)
```typescript
jokeFlow.handle([critiqueEvent], async (event) => {
// Keep track of the number of iterations
const state = getContext().state;
state.numIterations++;
// Write a new joke based on the previous joke and critique
const prompt = `Write a new joke based on the following critique and the original joke. Write the joke between <joke> and </joke> tags.\n\nJoke: ${event.data.joke}\n\nCritique: ${event.data.critique}`;
const response = await llm.complete({ prompt });
// Parse the joke from the response
const joke =
response.text.match(/<joke>([\s\S]*?)<\/joke>/)?.[1]?.trim() ??
response.text;
// If we've done less than the max number of iterations, keep iterating
// else, return the result
if (state.numIterations < state.maxIterations) {
return jokeEvent.with({ joke: joke });
}
return resultEvent.with({ joke: joke, critique: event.data.critique });
});
```
### Running the Workflow
```typescript
async function main() {
const { stream, sendEvent } = jokeFlow.createContext();
sendEvent(startEvent.with("pirates"));
let result: { joke: string, critique: string } | undefined;
for await (const event of stream) {
// console.log(event.data); optionally log the event data
if (resultEvent.include(event)) {
result = event.data;
break; // Stop when we get the final result
}
}
console.log(result);
}
```
To run the workflow, we:
1. Create a workflow context with `createContext()`
2. Trigger the initial event with `sendEvent()`
3. Listen to the event stream and process events as they arrive
4. Use `include()` to check if an event is of a specific type
5. Break the loop when we receive our final result
### Using Stream Utilities
The `stream` returned by `createContext` contains utility functions to make working with event streams easier:
```typescript
// Create a workflow context and send the initial event
const { stream, sendEvent } = jokeFlow.createContext();
sendEvent(startEvent.with("pirates"));
// Collect all events until we get a resultEvent
const allEvents = await stream.until(resultEvent).toArray();
// The last event will be the resultEvent
const finalEvent = allEvents.at(-1);
console.log(finalEvent.data); // Output the joke and critique
```
The stream utilities make it easier to work with the asynchronous event flow. In this example, we use:
- `toArray`: Aggregates all events into an array
- `until`: Creates a stream that emits events until a condition is met (in this case, until a resultEvent is received)
You can combine these utilities with other stream operators like `filter` and `map` to create powerful processing pipelines.
## Next Steps
To learn more about workflows, check out [the Workflows documentation](/docs/llamaindex/modules/agents/workflows).
+1 -1
View File
@@ -1,3 +1,3 @@
{
"pages": ["llamaindex", "api"]
"pages": ["llamaindex", "api", "llamaflow"]
}
+1 -1
View File
@@ -26,7 +26,7 @@ const divideNumbers = tool({
async function main() {
const mathAgent = agent({
tools: [sumNumbers, divideNumbers],
llm: openai({ model: "gpt-4o-mini" }),
llm: openai({ model: "gpt-4.1-mini" }),
verbose: false,
});
+1 -1
View File
@@ -25,7 +25,7 @@ async function main() {
},
{
type: "file",
data: fs.readFileSync("./data/manga.pdf"),
data: Uint8Array.from(fs.readFileSync("./data/manga.pdf")),
mimeType: "application/pdf",
},
],
+1 -1
View File
@@ -32,7 +32,7 @@ import fs from "fs";
},
{
type: "file",
data: fs.readFileSync("./data/manga.pdf"),
data: Uint8Array.from(fs.readFileSync("./data/manga.pdf")),
mimeType: "application/pdf",
},
],
+1 -1
View File
@@ -26,7 +26,7 @@ import fs from "fs";
},
{
type: "file",
data: fs.readFileSync("./data/manga.pdf"),
data: new Uint8Array(fs.readFileSync("./data/manga.pdf")),
mimeType: "application/pdf",
},
],
+1 -1
View File
@@ -21,7 +21,7 @@ async function main() {
},
{
type: "file",
data: fs.readFileSync("./data/manga.pdf"),
data: Uint8Array.from(fs.readFileSync("./data/manga.pdf")),
mimeType: "application/pdf",
},
],
+1 -1
View File
@@ -165,7 +165,7 @@ export type MessageContentImageDetail = {
export type MessageContentFileDetail = {
type: "file";
data: Buffer;
data: Uint8Array;
mimeType: string;
};
+2 -1
View File
@@ -6,7 +6,8 @@
import "./global-check.js";
export * from "./als/index.web.js";
export { consoleLogger, emptyLogger, type Logger } from "./logger/index.js";
export * from "./logger/index.js";
export * from "./utils/base64.js";
export { NotSupportCurrentRuntimeClass } from "./utils/shared.js";
export * from "./web-polyfill.js";
if (typeof window === "undefined") {
+2 -1
View File
@@ -5,6 +5,7 @@
*/
export * from "./als/index.non-node.js";
export { consoleLogger, emptyLogger, type Logger } from "./logger/index.js";
export * from "./logger/index.js";
export * from "./node-polyfill.js";
export * from "./utils/base64.js";
export { NotSupportCurrentRuntimeClass } from "./utils/shared.js";
+2 -1
View File
@@ -37,7 +37,8 @@ export function createSHA256(): SHA256 {
export const process = globalThis.process;
export * from "./als/index.node.js";
export { consoleLogger, emptyLogger, type Logger } from "./logger/index.js";
export * from "./logger/index.js";
export * from "./utils/base64.js";
export { CustomEvent, getEnv, setEnvs } from "./utils/index.js";
export { NotSupportCurrentRuntimeClass } from "./utils/shared.js";
export {
+2 -1
View File
@@ -11,9 +11,10 @@ export * from "./als/index.workerd.js";
export { NotSupportCurrentRuntimeClass } from "./utils/shared.js";
export * from "./node-polyfill.js";
export * from "./utils/base64.js";
export function getEnv(name: string): string | undefined {
return INTERNAL_ENV[name];
}
export { consoleLogger, emptyLogger, type Logger } from "./logger/index.js";
export * from "./logger/index.js";
+38
View File
@@ -0,0 +1,38 @@
/**
* Converts a Uint8Array to a base64 string.
* For large arrays, it processes the data in chunks to avoid memory issues.
* Falls back to Buffer if available for better performance.
*
* @param bytes - The Uint8Array to convert
* @returns The base64 string representation
*/
export function uint8ArrayToBase64(bytes: Uint8Array): string {
// Use Buffer if available (Node.js environment)
if (typeof Buffer !== "undefined") {
return Buffer.from(bytes).toString("base64");
}
// For browsers and other environments without Buffer
// Process in chunks for large arrays to avoid memory issues
const CHUNK_SIZE = 32768; // 32KB chunks
let result = "";
// For small arrays, use the built-in btoa function directly
if (bytes.length < CHUNK_SIZE) {
const binary = Array.from(bytes)
.map((byte) => String.fromCharCode(byte))
.join("");
return globalThis.btoa(binary);
}
// For large arrays, process in chunks
for (let i = 0; i < bytes.length; i += CHUNK_SIZE) {
const chunk = bytes.subarray(i, i + CHUNK_SIZE);
const binary = Array.from(chunk)
.map((byte) => String.fromCharCode(byte))
.join("");
result += globalThis.btoa(binary);
}
return result;
}
+1 -1
View File
@@ -21,7 +21,7 @@
],
"dependencies": {
"@llamaindex/cloud": "workspace:*",
"@llamaindex/core": "workspace:*",
"@llamaindex/core": "0.6.3",
"@llamaindex/env": "workspace:*",
"@llamaindex/node-parser": "workspace:*",
"@llamaindex/openai": "workspace:*",
+2 -2
View File
@@ -28,7 +28,7 @@ import type {
} from "@llamaindex/core/llms";
import { ToolCallLLM } from "@llamaindex/core/llms";
import { extractText } from "@llamaindex/core/utils";
import { getEnv } from "@llamaindex/env";
import { getEnv, uint8ArrayToBase64 } from "@llamaindex/env";
import { isDeepEqual } from "remeda";
export class AnthropicSession {
@@ -332,7 +332,7 @@ export class Anthropic extends ToolCallLLM<
source: {
type: "base64" as const,
media_type: content.mimeType,
data: content.data.toString("base64"),
data: uint8ArrayToBase64(content.data),
},
};
}
+6
View File
@@ -1,5 +1,11 @@
# @llamaindex/google
## 0.2.6
### Patch Changes
- 73e2578: Add support for gemini-2.5-pro-preview-05-06
## 0.2.5
### Patch Changes
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "@llamaindex/google",
"description": "Google Adapter for LlamaIndex",
"version": "0.2.5",
"version": "0.2.6",
"type": "module",
"main": "./dist/index.cjs",
"module": "./dist/index.js",
+2
View File
@@ -64,6 +64,7 @@ export const GEMINI_MODEL_INFO_MAP: Record<GEMINI_MODEL, GeminiModelInfo> = {
[GEMINI_MODEL.GEMINI_2_0_FLASH_THINKING_EXP]: { contextWindow: 32768 },
[GEMINI_MODEL.GEMINI_2_0_PRO_EXPERIMENTAL]: { contextWindow: 2 * 10 ** 6 },
[GEMINI_MODEL.GEMINI_2_5_PRO_PREVIEW]: { contextWindow: 10 ** 6 },
[GEMINI_MODEL.GEMINI_2_5_PRO_PREVIEW_LATEST]: { contextWindow: 10 ** 6 },
[GEMINI_MODEL.GEMINI_2_5_FLASH_PREVIEW]: { contextWindow: 10 ** 6 },
};
@@ -82,6 +83,7 @@ export const SUPPORT_TOOL_CALL_MODELS: GEMINI_MODEL[] = [
GEMINI_MODEL.GEMINI_2_0_FLASH,
GEMINI_MODEL.GEMINI_2_0_PRO_EXPERIMENTAL,
GEMINI_MODEL.GEMINI_2_5_PRO_PREVIEW,
GEMINI_MODEL.GEMINI_2_5_PRO_PREVIEW_LATEST,
GEMINI_MODEL.GEMINI_2_5_FLASH_PREVIEW,
];
+1
View File
@@ -74,6 +74,7 @@ export enum GEMINI_MODEL {
GEMINI_2_0_FLASH_THINKING_EXP = "gemini-2.0-flash-thinking-exp-01-21",
GEMINI_2_0_PRO_EXPERIMENTAL = "gemini-2.0-pro-exp-02-05",
GEMINI_2_5_PRO_PREVIEW = "gemini-2.5-pro-preview-03-25",
GEMINI_2_5_PRO_PREVIEW_LATEST = "gemini-2.5-pro-preview-05-06",
GEMINI_2_5_FLASH_PREVIEW = "gemini-2.5-flash-preview-04-17",
}
+1 -1
View File
@@ -289,7 +289,7 @@ export class GeminiHelper {
if (fileContents.length > 0) {
for (const file of fileContents) {
const uploadResponse = await GeminiHelper.uploadFile(
file.data,
Buffer.from(file.data),
file.mimeType,
);
parts.push({
+3 -2
View File
@@ -13,7 +13,7 @@ import {
type ToolCallLLMMessageOptions,
} from "@llamaindex/core/llms";
import { extractText } from "@llamaindex/core/utils";
import { getEnv } from "@llamaindex/env";
import { getEnv, uint8ArrayToBase64 } from "@llamaindex/env";
import { Tokenizers } from "@llamaindex/env/tokenizers";
import type {
AzureClientOptions,
@@ -217,10 +217,11 @@ export class OpenAI extends ToolCallLLM<OpenAIAdditionalChatOptions> {
if (item.mimeType !== "application/pdf") {
throw new Error("Only PDF files are supported");
}
const base64Data = uint8ArrayToBase64(item.data);
return {
type: "file",
file: {
file_data: `data:${item.mimeType};base64,${item.data.toString("base64")}`,
file_data: `data:${item.mimeType};base64,${base64Data}`,
filename: `part-${index}.pdf`,
},
} satisfies ChatCompletionContentPart.File;
+3 -2
View File
@@ -16,7 +16,7 @@ import {
} from "@llamaindex/core/llms";
import type { StoredValue } from "@llamaindex/core/schema";
import { extractText } from "@llamaindex/core/utils";
import { getEnv } from "@llamaindex/env";
import { getEnv, uint8ArrayToBase64 } from "@llamaindex/env";
import {
OpenAI as OpenAILLM,
type AzureClientOptions,
@@ -703,10 +703,11 @@ export class OpenAIResponses extends ToolCallLLM<OpenAIResponsesChatOptions> {
);
}
const base64Data = uint8ArrayToBase64(item.data);
return {
type: "input_file",
filename: `part-${index}.pdf`,
file_data: `data:${item.mimeType};base64,${item.data.toString("base64")}`,
file_data: `data:${item.mimeType};base64,${base64Data}`,
};
}
throw new Error("Unsupported content type");
@@ -205,7 +205,7 @@ describe("OpenAIResponses Unit Tests", () => {
{
type: "file",
mimeType: "image/jpeg",
data: Buffer.from("test image content"),
data: Uint8Array.from(Buffer.from("test image content")),
},
];
// @ts-expect-error accessing private method
+3 -1
View File
@@ -2,7 +2,9 @@ import { describe, expect, test } from "vitest";
import { duckduckgo, type DuckDuckGoToolOutput } from "../src/tools/duckduckgo";
describe("DuckDuckGo Tool", () => {
test("performs search and returns results", async () => {
// Needs to be skipped: duck-duck-scrape@2.2.7 throws an error:
// DDG detected an anomaly in the request, you are likely making requests too quickly.
test.skip("performs search and returns results", async () => {
const searchTool = duckduckgo();
const results = (await searchTool.call({
query: "OpenAI ChatGPT",
+1 -1
View File
@@ -1028,7 +1028,7 @@ importers:
specifier: workspace:*
version: link:../cloud
'@llamaindex/core':
specifier: workspace:*
specifier: 0.6.3
version: link:../core
'@llamaindex/env':
specifier: workspace:*