Compare commits

..

48 Commits

Author SHA1 Message Date
Jerry Liu 68ea59b623 cr 2025-10-18 11:27:39 -07:00
Jerry Liu 41050ae084 cr 2025-10-17 23:54:35 -07:00
Jerry Liu 02009cb249 cr 2025-10-17 23:40:30 -07:00
Jerry Liu 117af53323 cr 2025-10-17 23:12:36 -07:00
github-actions[bot] d0649ece6e chore: version packages (#982) 2025-10-16 16:58:29 -06:00
MartijnLeplae 5d4cabd843 Add ImageNode support in TypeScript (#969) 2025-10-16 16:56:28 -06:00
github-actions[bot] 9070a6ac16 chore: version packages (#981) 2025-10-15 12:01:34 -06:00
Bogdan Gheorghe 4f24f537f6 Add agressive table extraction argument (#980) 2025-10-15 11:57:34 -06:00
github-actions[bot] 8859a203e2 chore: version packages (#977) 2025-10-14 19:03:36 -06:00
dependabot[bot] b091364054 build(deps): bump astral-sh/setup-uv from 6 to 7 (#974) 2025-10-14 19:02:32 -06:00
dependabot[bot] 43b1a013ca build(deps): bump github/codeql-action from 3 to 4 (#973) 2025-10-14 19:02:20 -06:00
Logan f81532e7f2 safest types possible for parse (#976) 2025-10-14 19:02:07 -06:00
github-actions[bot] 986d3987d3 chore: version packages (#965)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-10-14 08:14:49 -06:00
Logan 1bf522311f fix default bbox values (#975) 2025-10-14 07:44:35 -06:00
Preston Carlson 24166dcfc8 Only escape single dollar sign in notebook md (#964)
* Limit escaping to lone dollar signs - preserve double dollar for latex equations

* Updated uv.lock via make lint

* Patch bump

* Unit test for _format_markdown_for_notebook

Test doesn't depend on getting real results/is just testing a string manipulation function, so inserting before other tests. Should move to its own file if we add additional formatting configurations
2025-10-07 08:06:03 -07:00
github-actions[bot] bfb7f3973f chore: version packages (#956)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-10-06 11:15:55 -04:00
dependabot[bot] 979f643c77 build(deps): bump actions/checkout from 4 to 5 (#961) 2025-10-06 09:12:38 -06:00
dependabot[bot] aefd89cf1b build(deps): bump actions/setup-python from 5 to 6 (#960) 2025-10-06 09:12:30 -06:00
dependabot[bot] 8ea2b2c64e build(deps): bump pnpm/action-setup from 3 to 4 (#959) 2025-10-06 09:12:20 -06:00
dependabot[bot] 4a9a2a21d8 build(deps): bump astral-sh/setup-uv from 3 to 6 (#958) 2025-10-06 09:12:08 -06:00
Logan e6a7939206 loosen packaging requirements (#962) 2025-10-06 09:11:57 -06:00
Adrian Lyjak 104a03e829 fix: re-enable js publishing (#963) 2025-10-06 11:10:46 -04:00
Terry Zhao 6e0f2f4ca0 citation can be null (#869)
* citation can be null

* Add changeset

---------

Co-authored-by: Terry Zhao <terryzhao@runllama.ai>
Co-authored-by: Adrian Lyjak <adrianlyjak@gmail.com>
2025-10-04 16:26:11 -04:00
dependabot[bot] 0708d11f8a Bump actions/setup-node from 4 to 5 (#909)
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 5.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-04 16:21:50 -04:00
github-actions[bot] be19185503 chore: version packages (#954)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-10-03 20:14:04 -04:00
Adrian Lyjak 7571b0d6c4 Missed some things again with tag fixes (#955)
guh
2025-10-03 20:12:53 -04:00
Adrian Lyjak ad6734bf80 fixup tagging more better (#953)
* fix: correct private field type in py/package.json to be recognized by pnpm

* use packages more directly, make public

* add bump

* fix crash
2025-10-03 19:53:57 -04:00
github-actions[bot] 9ec2a8322e chore: version packages (#952) 2025-10-03 15:11:14 -06:00
Logan 51011b9f30 fix changeset harder (#951) 2025-10-03 15:09:58 -06:00
Logan 09805f9e15 swap changesets (#949) 2025-10-03 15:06:00 -06:00
Adrian Lyjak 8ced6f6eab fix: explicitly tag. I thought the action did this (#948) 2025-10-03 16:59:41 -04:00
Preston Carlson 081ddeca34 Escaping dollar signs in md output when running in a jupyter notebook (#945) 2025-10-03 14:52:26 -06:00
Adrian Lyjak 2460908789 Disable npm release (#946) 2025-10-03 16:13:16 -04:00
Adrian Lyjak c226d6a54c Fix more bugs in publishing (#944) 2025-10-03 11:16:43 -04:00
Adrian Lyjak 5d4c682eb2 fix: theres just one publish token (#943) 2025-10-03 10:56:10 -04:00
github-actions[bot] f72d3535c8 chore: version packages (#941)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-10-03 10:25:11 -04:00
Adrian Lyjak 1ea09a366e Update llama-cloud dep (#940) 2025-10-03 09:56:56 -04:00
Adrian Lyjak d4bbeb6389 ignore nvmrc (#942)
ignore npmrc
2025-10-03 00:21:32 -04:00
Adrian Lyjak d028397603 version and release via changesets (#849) 2025-10-03 00:08:52 -04:00
Emanuel Ferreira 35ea8476db docs: parse -> classify -> extract (#931) 2025-09-24 18:52:15 -03:00
Logan 3e5f7c4f1e Update parse.md 2025-09-24 11:35:13 -06:00
Adrian Lyjak 9d9b816644 Handle reasoning field conflict (#929)
* Handle reasoning field conflict

* update version to 0.6.69
2025-09-22 11:29:11 -04:00
Adrian Lyjak 83555f76e6 Handle validation errors for agent data retrieval (#928)
* feat: Add untyped agent data retrieval and handling

Introduces methods to retrieve agent data as untyped dictionaries,
handling validation errors gracefully. This allows for more flexible
data access when strict typing is not required or when data may be
malformed.

Co-authored-by: adrian <adrian@runllama.ai>

* Expose raw api result

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-09-22 11:28:49 -04:00
Adrian Lyjak 5edf5f914a Support creating indexes in a specified project_id (#924)
* Support creating indexes in a specified project_id

* Bump
2025-09-18 11:07:07 -04:00
Adrian Lyjak 22e4975cb2 Refactor agent fields in llama_cloud_services (#921) 2025-09-17 15:14:40 -04:00
Peter Rowlands (변기호) bc2f04379b py: bump version to v.0.6.66 (#920) 2025-09-16 19:34:18 +09:00
Peter Rowlands (변기호) f9f951d5d8 parse: expose spreadsheet_force_formula_computation option (#919) 2025-09-16 19:28:03 +09:00
Emmanuel Ferdman 355129fea5 Fix colab broken links (#750)
Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>
2025-09-14 23:10:21 +02:00
70 changed files with 13391 additions and 6878 deletions
+8
View File
@@ -0,0 +1,8 @@
# Changesets
Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
with multi-package repos, or single-package repos to help you version and publish your code. You can
find the full documentation for it [in our repository](https://github.com/changesets/changesets)
We have a quick list of common questions to get you started engaging with this project in
[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)
+11
View File
@@ -0,0 +1,11 @@
{
"$schema": "https://unpkg.com/@changesets/config@3.1.1/schema.json",
"changelog": "@changesets/cli/changelog",
"commit": false,
"fixed": [],
"linked": [],
"access": "restricted",
"baseBranch": "main",
"updateInternalDependencies": "patch",
"ignore": []
}
+1 -1
View File
@@ -27,7 +27,7 @@ jobs:
- uses: actions/checkout@v5
- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
with:
version: ${{ env.UV_VERSION }}
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
- uses: pnpm/action-setup@v4
- name: Setup Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@v5
with:
node-version-file: "ts/llama_cloud_services/.nvmrc"
+2 -2
View File
@@ -30,12 +30,12 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
uses: github/codeql-action/init@v4
with:
languages: python
dependency-caching: true
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
uses: github/codeql-action/analyze@v4
with:
category: "/language:python"
+2 -2
View File
@@ -22,7 +22,7 @@ jobs:
with:
fetch-depth: ${{ github.event_name == 'pull_request' && 2 || 0 }}
- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
with:
version: ${{ env.UV_VERSION }}
@@ -31,7 +31,7 @@ jobs:
- uses: pnpm/action-setup@v4
- name: Setup Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@v5
with:
node-version-file: "ts/llama_cloud_services/.nvmrc"
- name: Install dependencies
-66
View File
@@ -1,66 +0,0 @@
name: Publish Release - Python
on:
push:
tags:
- "v*"
workflow_dispatch:
env:
UV_VERSION: "0.7.20"
jobs:
build-n-publish:
name: Build and publish to PyPI
if: github.repository == 'run-llama/llama_cloud_services'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
version: ${{ env.UV_VERSION }}
- name: Set up Python
run: uv python install
- name: Display Python version
run: python --version
- name: Build
working-directory: py
run: uv build
- name: Test installing built package
shell: bash
working-directory: py
run: |
uv venv
uv pip install dist/*.whl
- name: Publish package
shell: bash
working-directory: py
run: uv publish --token ${{ secrets.LLAMA_PARSE_PYPI_TOKEN }}
- name: Build and publish llama-parse
working-directory: py/llama_parse/
run: |
uv build
uv publish --token ${{ secrets.LLAMA_PARSE_PYPI_TOKEN }}
- name: Create GitHub Release
id: create_release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
with:
tag_name: ${{ github.ref }}
release_name: ${{ github.ref }} - LlamaCloud Services PY
artifacts: "py/**/dist/*"
generateReleaseNotes: true
draft: false
prerelease: false
-52
View File
@@ -1,52 +0,0 @@
name: Publish Release - TypeScript
on:
push:
tags:
- "llama-cloud-services@*"
jobs:
build-and-publish:
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout@v5
- uses: pnpm/action-setup@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version-file: "ts/llama_cloud_services/.nvmrc"
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Run Build
working-directory: ts/llama_cloud_services/
run: pnpm build
- name: Build tarball
run: |
pnpm pack
working-directory: ts/llama_cloud_services
- name: Setup npm authentication
run: echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Release
working-directory: ts/llama_cloud_services
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
run: pnpm publish --access public --no-git-checks
- name: Create release
uses: ncipollo/release-action@v1
with:
artifacts: "ts/llama_cloud_services/llama-cloud-services*.tgz"
name: Release ${{ github.ref_name }} - LlamaCloud Services TS
generateReleaseNotes: true
token: ${{ secrets.GITHUB_TOKEN }}
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
with:
fetch-depth: 0
- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
with:
version: ${{ env.UV_VERSION }}
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
with:
fetch-depth: 0
- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
with:
version: ${{ env.UV_VERSION }}
+1 -1
View File
@@ -24,7 +24,7 @@ jobs:
- uses: actions/checkout@v5
- uses: pnpm/action-setup@v4
- name: Setup Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@v5
with:
node-version-file: "ts/llama_cloud_services/.nvmrc"
- name: Install dependencies
@@ -0,0 +1,61 @@
name: Version Bump and Release
on:
push:
branches:
- main
concurrency: ${{ github.workflow }}-${{ github.ref }}
jobs:
release:
name: Release
runs-on: ubuntu-latest
# Only run on main branch pushes
if: github.ref == 'refs/heads/main'
steps:
- name: Checkout Repo
uses: actions/checkout@v5
- uses: pnpm/action-setup@v4
- name: Setup Node.js
uses: actions/setup-node@v5
with:
node-version: "22"
cache: "pnpm"
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: "3.11"
- name: Install uv
uses: astral-sh/setup-uv@v7
- name: Install dependencies
run: pnpm install
- name: Add auth token to .npmrc file
run: |
cat << EOF >> ".npmrc"
//registry.npmjs.org/:_authToken=$NPM_TOKEN
EOF
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Create Release Pull Request or Publish packages
id: changesets
uses: changesets/action@v1
with:
commit: "chore: version packages"
title: "chore: version packages"
# Custom version script
version: pnpm -w run version
# Custom publish script
publish: pnpm -w run publish
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_TOKEN }}
LLAMA_PARSE_PYPI_TOKEN: ${{ secrets.LLAMA_PARSE_PYPI_TOKEN }}
+1
View File
@@ -9,3 +9,4 @@ __pycache__/
node_modules/
.turbo/
dist/
.npmrc
+1 -1
View File
@@ -29,7 +29,7 @@ repos:
- id: black-jupyter
name: black-src
alias: black
exclude: ".*uv.lock"
exclude: ".*uv.lock|examples/extract/solar_panel_e2e_comparison.ipynb"
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.0.1
hooks:
@@ -7,7 +7,7 @@
"source": [
"# Extraction and Analysis over a Fidelity Multi-Fund Annual Report\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services-demo/blob/main/examples/extract/asset_manager_fund_analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/extract/asset_manager_fund_analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"In this notebook we show you how to create an agentic document workflow over a complex document that contains annual reports for multiple funds - each fund reports financials in a standardized reporting structure, and it's all consolidated in the same document.\n",
"\n",
@@ -7,7 +7,7 @@
"source": [
"# Automotive Equity Research: A Multi-Step Agentic Workflow\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services-demo/blob/main/examples/extract/automotive_sector_analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/extract/automotive_sector_analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"This notebook demonstrates an endtoend agentic workflow using LlamaExtract and the LlamaIndex eventdriven workflow framework for automotive sector analysis.\n",
"\n",
+2 -2
View File
@@ -1035,7 +1035,7 @@
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
@@ -1052,5 +1052,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 4
}
File diff suppressed because it is too large Load Diff
@@ -7,7 +7,7 @@
"source": [
"# Dynamic Section Retrieval with LlamaParse\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services-demo/blob/main/examples/parse/advanced_rag/dynamic_section_retrieval.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/advanced_rag/dynamic_section_retrieval.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"This notebook showcases a concept called \"dynamic section retrieval\".\n",
"\n",
+1 -1
View File
@@ -6,7 +6,7 @@
"source": [
"# Advanced RAG with LlamaParse\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/demo_advanced.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/parse/demo_advanced.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"This notebook is a complete walkthrough for using LlamaParse with advanced indexing/retrieval techniques in LlamaIndex over the Apple 10K Filing. \n",
"\n",
+1 -1
View File
@@ -6,7 +6,7 @@
"source": [
"# RAG with Excel Spreadsheet using LlamaPrase\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/demo_excel.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/demo_excel.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"This notebook shows you using LlamaParse with Excel Spreadsheet.\n",
"\n",
+1 -1
View File
@@ -7,7 +7,7 @@
"source": [
"# Download Charts\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/demo_get_charts.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/demo_get_charts.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"This notebook demonstrates how to download charts from a document using the result object.\n",
"\n",
+4 -4
View File
@@ -6,7 +6,7 @@
"source": [
"# LlamaParse - Fast checking Insurance Contract for Coverage\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/demo_insurance.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/demo_insurance.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"In this notebook we will look at how LlamaParse can be used to extract structured coverage information from an insurance policy.\n",
"\n",
@@ -36,7 +36,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Download an insurance policy fron IRDAI\n",
"## Download an insurance policy from IRDAI\n",
"\n",
"The Insurance Regulatory and Development Authority of India (IRDAI) maintains a great resource: https://policyholder.gov.in/web/guest/non-life-insurance-products where all insurance policies available in India are publicly available for download! Let's download a complex health insurance policy as an example."
]
@@ -228,11 +228,11 @@
" result_type=\"markdown\",\n",
" system_prompt_append=\"\"\"\n",
"This document is an insurance policy.\n",
"When a benefits/coverage/exlusion is describe in the document ammend to it add a text in the follwing benefits string format (where coverage could be an exclusion).\n",
"When a benefits/coverage/exlusion is describe in the document amend to it add a text in the following benefits string format (where coverage could be an exclusion).\n",
"\n",
"For {nameofrisk} and in this condition {whenDoesThecoverageApply} the coverage is {coverageDescription}. \n",
" \n",
"If the document contain a benefits TABLE that describe coverage amounts, do not ouput it as a table, but instead as a list of benefits string.\n",
"If the document contain a benefits TABLE that describe coverage amounts, do not output it as a table, but instead as a list of benefits string.\n",
" \n",
"\"\"\",\n",
").aparse(\"./policy.pdf\")\n",
+1 -1
View File
@@ -7,7 +7,7 @@
"source": [
"# LlamaParse `JobResult` Tour\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/demo_json.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/demo_json.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"The `JobResult` object is the main object returned by the LlamaParse API. It contains all the information about the job, including the parsed data, metadata, and any errors.\n",
"\n",
+1 -1
View File
@@ -9,7 +9,7 @@
"\n",
"LlamaParse supports users to specify a `language` parameter before uploading documents, giving users better OCR capabilities over non-English PDFs, parsing images into more accurate representations.\n",
"\n",
"You can specify 80+ different languages: see this file for a full list of supported languages: https://github.com/run-llama/llama_cloud_services/blob/main/llama_parse/base.py.\n",
"You can specify 80+ different languages: see this file for a full list of supported languages: https://github.com/run-llama/llama_cloud_services/blob/main/py/llama_cloud_services/parse/base.py.\n",
"\n",
"This notebook shows a demo of this in action. \n",
"\n",
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/excel/o1_excel_rag.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/excel/o1_excel_rag.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@@ -740,7 +740,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, these pages aren't going to be that different when parsed, but we can verify which pages triggered auto-made by looking at the [JSON output](https://github.com/run-llama/llama_cloud_services/blob/main/examples/demo_json_tour.ipynb) of LlamaParse:"
"In this example, these pages aren't going to be that different when parsed, but we can verify which pages triggered auto-made by looking at the [JSON output](https://github.com/run-llama/llama_cloud_services/blob/main/examples/parse/demo_json_tour.ipynb) of LlamaParse:"
]
},
{
+8 -1
View File
@@ -5,9 +5,16 @@
"private": true,
"keywords": [],
"author": "",
"scripts": {
"pre-commit-version": "pnpm changeset",
"version": "./scripts/changeset-version.py version",
"publish": "./scripts/changeset-version.py publish --tag"
},
"devDependencies": {
"prettier": "^3.6.2",
"lint-staged": "^15.4.2"
"lint-staged": "^15.4.2",
"@changesets/cli": "^2.29.5",
"changesets": "^1.0.2"
},
"lint-staged": {
"ts/llama_cloud_services/src/**/*.{ts,tsx,js,jsx}": [
+1 -1
View File
@@ -147,7 +147,7 @@ documents = SimpleDirectoryReader(
).load_data()
```
Full documentation for `SimpleDirectoryReader` can be found on the [LlamaIndex Documentation](https://docs.llamaindex.ai/en/stable/module_guides/loading/simpledirectoryreader.html).
Full documentation for `SimpleDirectoryReader` can be found on the [LlamaIndex Documentation](https://developers.llamaindex.ai/python/framework/module_guides/loading/simpledirectoryreader/).
## Examples
+589 -10
View File
File diff suppressed because it is too large Load Diff
+3 -1
View File
@@ -1,2 +1,4 @@
packages:
- "ts/**"
- "ts/*"
- "py"
- "py/*"
+44
View File
@@ -0,0 +1,44 @@
# llama-cloud-services-py
## 0.6.76
### Patch Changes
- 4f24f53: Add aggressive_table_extraction flag in python sdk
## 0.6.75
### Patch Changes
- f81532e: Safest types possible for parse
## 0.6.74
### Patch Changes
- 1bf5223: Fix default bbox values
- 24166dc: Now only escape single dollar signs - preserve double for latex equations
## 0.6.73
### Patch Changes
- e6a7939: Loosen packaging dep requirement
## 0.6.72
### Patch Changes
- ad6734b: Fixup and test versioning
## 0.6.71
### Patch Changes
- 51011b9: Escape dollar signs in jupyter notebooks
## 0.6.70
### Patch Changes
- d028397: Update llama-cloud api version, and integrate with agent data deletion
@@ -1,6 +1,11 @@
import os
from typing import Any, Dict, Generic, List, Optional, Type
from llama_cloud import (
AgentData,
PaginatedResponseAgentData,
PaginatedResponseAggregateGroup,
)
from llama_cloud.client import AsyncLlamaCloud
from tenacity import (
WrappedFn,
@@ -86,7 +91,7 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
client=llama_client,
type=ExtractedPerson,
collection="extracted_people",
agent_url_id="person-extraction-agent"
deployment_name="person-extraction-agent"
)
# Create data
@@ -109,10 +114,12 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
self,
type: Type[AgentDataT],
collection: str = "default",
agent_url_id: Optional[str] = None,
deployment_name: Optional[str] = None,
client: Optional[AsyncLlamaCloud] = None,
token: Optional[str] = None,
base_url: Optional[str] = None,
# deprecated, use deployment_name instead
agent_url_id: Optional[str] = None,
):
"""
Initialize the AsyncAgentDataClient.
@@ -123,11 +130,11 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
collection: Named collection within the agent for organizing data.
Defaults to "default". Collections allow logical separation of
different data types or workflows within the same agent.
agent_url_id: Unique identifier for the agent. This normally appears in the
url of an agent within the llama cloud platform. If not provided,
will attempt to use the LLAMA_DEPLOY_DEPLOYMENT_NAME environment
variable. Data can only be added to an already existing agent in the
platform.
deployment_name: Unique identifier for the agent deployment. This normally
appears in the URL of an agent within the Llama Cloud platform. If not
provided, will attempt to use the LLAMA_DEPLOY_DEPLOYMENT_NAME
environment variable. Data can only be added to an already existing
agent in the platform.
client: AsyncLlamaCloud client instance for API communication. If not provided, will
construct one from the provided api token and base url
token: Llama Cloud API token. Reads from LLAMA_CLOUD_API_KEY if not provided
@@ -135,15 +142,14 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
defaults to https://api.cloud.llamaindex.ai
Raises:
ValueError: If agent_url_id is not provided and the
ValueError: If deployment_name is not provided and the
LLAMA_DEPLOY_DEPLOYMENT_NAME environment variable is not set
Note:
The client automatically applies retry logic to all API calls with
exponential backoff for timeout, connection, and HTTP status errors.
"""
self.agent_url_id = agent_url_id or get_default_agent_id()
self.deployment_name = deployment_name or agent_url_id or get_default_agent_id()
self.collection = collection
if not client:
@@ -156,15 +162,19 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
@agent_data_retry
async def get_item(self, item_id: str) -> TypedAgentData[AgentDataT]:
raw_data = await self.client.beta.get_agent_data(
raw_data = await self.untyped_get_item(item_id)
return TypedAgentData.from_raw(raw_data, self.type)
@agent_data_retry
async def untyped_get_item(self, item_id: str) -> AgentData:
return await self.client.beta.get_agent_data(
item_id=item_id,
)
return TypedAgentData.from_raw(raw_data, validator=self.type)
@agent_data_retry
async def create_item(self, data: AgentDataT) -> TypedAgentData[AgentDataT]:
raw_data = await self.client.beta.create_agent_data(
agent_slug=self.agent_url_id,
deployment_name=self.deployment_name,
collection=self.collection,
data=data.model_dump(),
)
@@ -184,6 +194,21 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
async def delete_item(self, item_id: str) -> None:
await self.client.beta.delete_agent_data(item_id=item_id)
@agent_data_retry
async def delete(
self, filter: Optional[Dict[str, Dict[ComparisonOperator, Any]]] = None
) -> int:
"""
Delete agent data by query, similar to search.
Returns the number of deleted items.
"""
response = await self.client.beta.delete_agent_data_by_query_api_v_1_beta_agent_data_delete_post(
deployment_name=self.deployment_name,
collection=self.collection,
filter=filter,
)
return response.deleted_count
@agent_data_retry
async def search(
self,
@@ -210,9 +235,7 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
offset: Number of items to skip from the beginning. Defaults to 0.
include_total: Whether to include the total count in the response. Defaults to False to improve performance. It's recommended to only request on the first page.
"""
raw = await self.client.beta.search_agent_data_api_v_1_beta_agent_data_search_post(
agent_slug=self.agent_url_id,
collection=self.collection,
raw = await self.untyped_search(
filter=filter,
order_by=order_by,
offset=offset,
@@ -227,6 +250,25 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
total=raw.total_size,
)
@agent_data_retry
async def untyped_search(
self,
filter: Optional[Dict[str, Dict[ComparisonOperator, Any]]] = None,
order_by: Optional[str] = None,
offset: Optional[int] = None,
page_size: Optional[int] = None,
include_total: bool = False,
) -> PaginatedResponseAgentData:
return await self.client.beta.search_agent_data_api_v_1_beta_agent_data_search_post(
deployment_name=self.deployment_name,
collection=self.collection,
filter=filter,
order_by=order_by,
offset=offset,
page_size=page_size,
include_total=include_total,
)
@agent_data_retry
async def aggregate(
self,
@@ -253,8 +295,38 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
offset: Number of groups to skip from the beginning. Defaults to 0.
page_size: Maximum number of groups to return per page.
"""
raw = await self.client.beta.aggregate_agent_data_api_v_1_beta_agent_data_aggregate_post(
agent_slug=self.agent_url_id,
raw = await self.untyped_aggregate(
filter=filter,
group_by=group_by,
count=count,
first=first,
order_by=order_by,
offset=offset,
page_size=page_size,
)
return TypedAggregateGroupItems(
items=[
TypedAggregateGroup.from_raw(grp, validator=self.type)
for grp in raw.items
],
has_more=raw.next_page_token is not None,
total=raw.total_size,
)
@agent_data_retry
async def untyped_aggregate(
self,
filter: Optional[Dict[str, Dict[ComparisonOperator, Any]]] = None,
group_by: Optional[List[str]] = None,
count: Optional[bool] = None,
first: Optional[bool] = None,
order_by: Optional[str] = None,
offset: Optional[int] = None,
page_size: Optional[int] = None,
) -> PaginatedResponseAggregateGroup:
return await self.client.beta.aggregate_agent_data_api_v_1_beta_agent_data_aggregate_post(
deployment_name=self.deployment_name,
collection=self.collection,
page_size=page_size,
filter=filter,
@@ -264,11 +336,3 @@ class AsyncAgentDataClient(Generic[AgentDataT]):
first=first,
offset=offset,
)
return TypedAggregateGroupItems(
items=[
TypedAggregateGroup.from_raw(item, validator=self.type)
for item in raw.items
],
has_more=raw.next_page_token is not None,
total=raw.total_size,
)
@@ -10,7 +10,7 @@ CRUD operations, search capabilities, filtering, and aggregation functionality
for managing agent-generated data at scale.
Key Concepts:
- Agent Slug: Unique identifier for an agent instance
- Deployment Name: Unique identifier for an agent deployment
- Collection: Named grouping of data within an agent (defaults to "default"). Data within a collection should be of the same type.
- Agent Data: Individual structured data records with metadata and timestamps
@@ -26,7 +26,7 @@ Example Usage:
client=async_llama_cloud,
type=Person,
collection="people",
agent_url_id="my-extraction-agent-xyz"
deployment_name="my-extraction-agent-xyz"
)
# Create typed data
@@ -56,7 +56,6 @@ from typing import (
# Type variable for user-defined data models
AgentDataT = TypeVar("AgentDataT", bound=BaseModel)
# Type variable for extracted data (can be dict or Pydantic model)
ExtractedT = TypeVar("ExtractedT", bound=Union[BaseModel, dict])
@@ -78,7 +77,7 @@ class TypedAgentData(BaseModel, Generic[AgentDataT]):
Attributes:
id: Unique identifier for this data record
agent_url_id: Identifier of the agent that created this data
deployment_name: Identifier of the agent deployment that created this data
collection: Named collection within the agent (used for organization)
data: The actual structured data payload (typed as AgentDataT)
created_at: Timestamp when the record was first created
@@ -94,8 +93,8 @@ class TypedAgentData(BaseModel, Generic[AgentDataT]):
"""
id: Optional[str] = Field(description="Unique identifier for this data record")
agent_url_id: str = Field(
description="Identifier of the agent that created this data"
deployment_name: str = Field(
description="Identifier of the agent deployment that created this data"
)
collection: Optional[str] = Field(
description="Named collection within the agent for data organization"
@@ -116,15 +115,15 @@ class TypedAgentData(BaseModel, Generic[AgentDataT]):
Args:
raw_data: Raw agent data from the API
validator: Pydantic model class to validate the data field
Returns:
TypedAgentData instance with validated data
"""
data: AgentDataT = validator.model_validate(raw_data.data)
return cls(
id=raw_data.id,
agent_url_id=raw_data.agent_slug,
deployment_name=raw_data.deployment_name,
collection=raw_data.collection,
data=data,
created_at=raw_data.created_at,
@@ -222,12 +221,16 @@ def parse_extracted_field_metadata(
return {
k: _parse_extracted_field_metadata_recursive(v)
for k, v in field_metadata.items()
if k not in _METADATA_FIELDS_SIBLING_TO_LEAF
and k not in _ADDITIONAL_ROOT_METADATA_FIELDS
if not _is_reasoning_field(k, v) and k not in _ADDITIONAL_ROOT_METADATA_FIELDS
}
_METADATA_FIELDS_SIBLING_TO_LEAF = {"reasoning"}
def _is_reasoning_field(field_name: str, field_value: Any) -> bool:
# There can either be a user specified reasoning field (from the schema), or a reasoning metadata field for the
# dict of values
return field_name == "reasoning" and isinstance(field_value, str)
_ADDITIONAL_ROOT_METADATA_FIELDS = {"error"}
@@ -257,14 +260,12 @@ def _parse_extracted_field_metadata_recursive(
except ValidationError:
pass
additional_fields = {
k: v
for k, v in field_value.items()
if k in _METADATA_FIELDS_SIBLING_TO_LEAF
k: v for k, v in field_value.items() if _is_reasoning_field(k, v)
}
return {
k: _parse_extracted_field_metadata_recursive(v, additional_fields)
for k, v in field_value.items()
if k not in _METADATA_FIELDS_SIBLING_TO_LEAF
if not _is_reasoning_field(k, v)
}
elif isinstance(field_value, list):
return [_parse_extracted_field_metadata_recursive(item) for item in field_value]
+4 -58
View File
@@ -19,14 +19,12 @@ from llama_cloud import (
ExtractAgent as CloudExtractAgent,
ExtractConfig,
ExtractJob,
ExtractJobCreate,
ExtractRun,
File,
FileData,
ExtractMode,
StatusEnum,
ExtractTarget,
LlamaExtractSettings,
PaginatedExtractRunsResponse,
)
from llama_cloud.client import AsyncLlamaCloud
@@ -463,56 +461,6 @@ class ExtractionAgent:
)
)
async def _run_extraction_test(
self,
files: Union[FileInput, List[FileInput]],
extract_settings: LlamaExtractSettings,
) -> Union[ExtractJob, List[ExtractJob]]:
if not isinstance(files, list):
files = [files]
single_file = True
else:
single_file = False
upload_tasks = [self._upload_file(file) for file in files]
with augment_async_errors():
uploaded_files = await run_jobs(
upload_tasks,
workers=self.num_workers,
desc="Uploading files",
show_progress=self.show_progress,
)
async def run_job(file: File) -> ExtractRun:
job_queued = await self._client.llama_extract.run_job_test_user(
job_create=ExtractJobCreate(
extraction_agent_id=self.id,
file_id=file.id,
data_schema_override=self.data_schema,
config_override=self.config,
),
extract_settings=extract_settings,
)
return await self._wait_for_job_result(job_queued.id)
job_tasks = [run_job(file) for file in uploaded_files]
with augment_async_errors():
extract_results = await run_jobs(
job_tasks,
workers=self.num_workers,
desc="Running extraction jobs",
show_progress=self.show_progress,
)
if self._verbose:
for file, job in zip(files, extract_results):
file_repr = (
str(file) if isinstance(file, (str, Path)) else "<bytes/buffer>"
)
print(f"Running extraction for file {file_repr} under job_id {job.id}")
return extract_results[0] if single_file else extract_results
async def queue_extraction(
self,
files: Union[FileInput, List[FileInput]],
@@ -544,12 +492,10 @@ class ExtractionAgent:
job_tasks = [
self._client.llama_extract.run_job(
request=ExtractJobCreate(
extraction_agent_id=self.id,
file_id=file.id,
data_schema_override=self.data_schema,
config_override=self.config,
),
extraction_agent_id=self.id,
file_id=file.id,
data_schema_override=self.data_schema,
config_override=self.config,
)
for file in uploaded_files
]
+14 -12
View File
@@ -489,6 +489,7 @@ class LlamaCloudIndex(BaseManagedIndex):
name: str,
project_name: str = DEFAULT_PROJECT_NAME,
organization_id: Optional[str] = None,
project_id: Optional[str] = None,
api_key: Optional[str] = None,
base_url: Optional[str] = None,
app_url: Optional[str] = None,
@@ -504,15 +505,15 @@ class LlamaCloudIndex(BaseManagedIndex):
app_url = app_url or os.environ.get("LLAMA_CLOUD_APP_URL", DEFAULT_APP_URL)
client = get_client(api_key, base_url, app_url, timeout)
# create project if it doesn't exist
project = client.projects.upsert_project(
organization_id=organization_id, request=ProjectCreate(name=project_name)
)
if project.id is None:
raise ValueError(f"Failed to create/get project {project_name}")
if verbose:
print(f"Created project {project.id} with name {project.name}")
if project_id is None:
# create project if it doesn't exist
project = client.projects.upsert_project(
organization_id=organization_id,
request=ProjectCreate(name=project_name),
)
project_id = project.id
if verbose:
print(f"Created project {project_id} with name {project_name}")
# create pipeline
pipeline_create = PipelineCreate(
@@ -523,7 +524,7 @@ class LlamaCloudIndex(BaseManagedIndex):
llama_parse_parameters=llama_parse_parameters or LlamaParseParameters(),
)
pipeline = client.pipelines.upsert_pipeline(
project_id=project.id, request=pipeline_create
project_id=project_id, request=pipeline_create
)
if pipeline.id is None:
raise ValueError(f"Failed to create/get pipeline {name}")
@@ -532,8 +533,7 @@ class LlamaCloudIndex(BaseManagedIndex):
return cls(
name,
project_name=project.name,
organization_id=project.organization_id,
project_id=project_id,
api_key=api_key,
base_url=base_url,
app_url=app_url,
@@ -606,6 +606,7 @@ class LlamaCloudIndex(BaseManagedIndex):
name: str,
project_name: str = DEFAULT_PROJECT_NAME,
organization_id: Optional[str] = None,
project_id: Optional[str] = None,
api_key: Optional[str] = None,
base_url: Optional[str] = None,
app_url: Optional[str] = None,
@@ -631,6 +632,7 @@ class LlamaCloudIndex(BaseManagedIndex):
verbose=verbose,
embedding_config=embedding_config,
transform_config=transform_config,
project_id=project_id,
)
app_url = app_url or os.environ.get("LLAMA_CLOUD_APP_URL", DEFAULT_APP_URL)
+47 -94
View File
@@ -188,6 +188,10 @@ class LlamaParse(BasePydanticReader):
default=False,
description="If set to true, LlamaParse will try to detect long table and adapt the output.",
)
aggressive_table_extraction: Optional[bool] = Field(
default=False,
description="If set to true, LlamaParse will try to extract tables aggressively, may lead to false positives.",
)
annotate_links: Optional[bool] = Field(
default=False,
description="Annotate links found in the document to extract their URL.",
@@ -420,6 +424,10 @@ class LlamaParse(BasePydanticReader):
default=False,
description="If set to true, the parser will extract sub-tables from the spreadsheet when possible (more than one table per sheet).",
)
spreadsheet_force_formula_computation: Optional[bool] = Field(
default=False,
description="If set to true, the parser will re-compute values for all spreadsheet cells containing formulas.",
)
specialized_chart_parsing_agentic: Optional[bool] = Field(
default=False,
description="If set to true, the parser will use a specialized agentic chart parsing model to extract data from charts. This model is able to understand the chart type and extract the data accordingly.",
@@ -709,6 +717,9 @@ class LlamaParse(BasePydanticReader):
if self.adaptive_long_table:
data["adaptive_long_table"] = self.adaptive_long_table
if self.aggressive_table_extraction:
data["aggressive_table_extraction"] = self.aggressive_table_extraction
if self.annotate_links:
data["annotate_links"] = self.annotate_links
@@ -965,6 +976,11 @@ class LlamaParse(BasePydanticReader):
if self.spreadsheet_extract_sub_tables:
data["spreadsheet_extract_sub_tables"] = self.spreadsheet_extract_sub_tables
if self.spreadsheet_force_formula_computation:
data[
"spreadsheet_force_formula_computation"
] = self.spreadsheet_force_formula_computation
if self.specialized_chart_parsing_agentic:
data[
"specialized_chart_parsing_agentic"
@@ -1073,11 +1089,7 @@ class LlamaParse(BasePydanticReader):
return current_interval # Default fallback
async def _get_job_result(
self,
job_id: str,
result_type: str,
verbose: bool = False,
raise_job_error: bool = True,
self, job_id: str, result_type: str, verbose: bool = False
) -> Dict[str, Any]:
start = time.time()
tries = 0
@@ -1110,25 +1122,7 @@ class LlamaParse(BasePydanticReader):
print(".", end="", flush=True)
current_interval = self._calculate_backoff(current_interval)
else:
if raise_job_error:
raise JobFailedException.from_result(result_json)
else:
error_code = result_json.get("error_code")
error_message = result_json.get("error_message")
error_parts = [f"Job ID: {job_id} failed with status: {status}"]
if error_code:
error_parts.append(f"Error code: {error_code}")
if error_message:
error_parts.append(f"Error message: {error_message}")
error_str = ", ".join(error_parts)
return {
"pages": [],
"job_metadata": {"job_pages": 0},
"error": error_str,
"status": status,
}
raise JobFailedException.from_result(result_json)
except (
httpx.ConnectError,
httpx.ReadError,
@@ -1159,7 +1153,6 @@ class LlamaParse(BasePydanticReader):
fs: Optional[AbstractFileSystem] = None,
result_type: Optional[str] = None,
num_workers: Optional[int] = None,
raise_job_error: bool = True,
) -> List[Tuple[str, Dict[str, Any]]]:
if self.partition_pages is None:
job_results = [
@@ -1168,7 +1161,6 @@ class LlamaParse(BasePydanticReader):
extra_info=extra_info,
fs=fs,
result_type=result_type,
raise_job_error=raise_job_error,
)
]
else:
@@ -1178,7 +1170,6 @@ class LlamaParse(BasePydanticReader):
fs=fs,
result_type=result_type,
num_workers=num_workers,
raise_job_error=raise_job_error,
)
return job_results
@@ -1188,7 +1179,6 @@ class LlamaParse(BasePydanticReader):
extra_info: Optional[dict] = None,
fs: Optional[AbstractFileSystem] = None,
result_type: Optional[str] = None,
raise_job_error: bool = True,
**create_kwargs: Any,
) -> Tuple[str, Dict[str, Any]]:
"""Create one parse job and wait for the result."""
@@ -1198,10 +1188,7 @@ class LlamaParse(BasePydanticReader):
if self.verbose:
print("Started parsing the file under job_id %s" % job_id)
result = await self._get_job_result(
job_id,
result_type or self.result_type.value,
verbose=self.verbose,
raise_job_error=raise_job_error,
job_id, result_type or self.result_type.value, verbose=self.verbose
)
return job_id, result
@@ -1212,7 +1199,6 @@ class LlamaParse(BasePydanticReader):
fs: Optional[AbstractFileSystem] = None,
result_type: Optional[str] = None,
num_workers: Optional[int] = None,
raise_job_error: bool = True,
) -> List[Tuple[str, Dict[str, Any]]]:
"""Partition a file and run separate parse jobs per partition segment."""
assert self.partition_pages is not None
@@ -1227,7 +1213,6 @@ class LlamaParse(BasePydanticReader):
extra_info=extra_info,
fs=fs,
result_type=result_type,
raise_job_error=raise_job_error,
partition_target_pages=target_pages,
)
for target_pages in partition_pages(
@@ -1255,33 +1240,28 @@ class LlamaParse(BasePydanticReader):
size = self.partition_pages
if not size:
break
# Fetch JSON result type first to get accurate pagination data
# and then fetch the user's desired result type if needed
job_id, json_result = await self._parse_one_unpartitioned(
file_path,
extra_info=extra_info,
fs=fs,
result_type=ResultType.JSON.value,
raise_job_error=raise_job_error,
partition_target_pages=f"{total}-{total + size - 1}",
)
if json_result.get("error"):
if results and "NO_DATA_FOUND_IN_FILE" in json_result.get("error", ""):
return results
results.append((job_id, json_result))
return results
result_type = result_type or self.result_type.value
if result_type == ResultType.JSON.value:
job_result = json_result
else:
job_result = await self._get_job_result(
job_id,
result_type,
verbose=self.verbose,
raise_job_error=raise_job_error,
try:
# Fetch JSON result type first to get accurate pagination data
# and then fetch the user's desired result type if needed
job_id, json_result = await self._parse_one_unpartitioned(
file_path,
extra_info=extra_info,
fs=fs,
result_type=ResultType.JSON.value,
partition_target_pages=f"{total}-{total + size - 1}",
)
result_type = result_type or self.result_type.value
if result_type == ResultType.JSON.value:
job_result = json_result
else:
job_result = await self._get_job_result(
job_id, result_type, verbose=self.verbose
)
except JobFailedException as e:
if results and e.error_code == "NO_DATA_FOUND_IN_FILE":
# Expected when we try to read past the end of the file
return results
raise
results.append((job_id, job_result))
if len(json_result["pages"]) < size:
break
@@ -1395,7 +1375,6 @@ class LlamaParse(BasePydanticReader):
extra_info: Optional[dict] = None,
fs: Optional[AbstractFileSystem] = None,
num_workers: Optional[int] = None,
raise_job_error: bool = True,
) -> List[JobResult]:
job_results = await self._parse_one(
file_path,
@@ -1403,7 +1382,6 @@ class LlamaParse(BasePydanticReader):
fs=fs,
result_type=ResultType.JSON.value,
num_workers=num_workers,
raise_job_error=raise_job_error,
)
return [
JobResult(
@@ -1423,7 +1401,6 @@ class LlamaParse(BasePydanticReader):
file_path: Union[List[FileInput], FileInput],
extra_info: Optional[dict] = None,
fs: Optional[AbstractFileSystem] = None,
raise_job_error: bool = True,
) -> Union[List["JobResult"], "JobResult"]:
"""
Parse the file and return a JobResult object instead of Document objects.
@@ -1435,7 +1412,6 @@ class LlamaParse(BasePydanticReader):
file_path: Path to the file to parse. Can be a string, path, bytes, file-like object, or a list of these.
extra_info: Additional metadata to include in the result.
fs: Optional filesystem to use for reading files.
raise_job_error: If True (default), raises JobFailedException when jobs fail. If False, returns JobResult objects with error information.
Returns:
JobResult object or list of JobResult objects if either multiple files were provided or file(s) were partitioned before parsing.
@@ -1451,11 +1427,7 @@ class LlamaParse(BasePydanticReader):
else:
file_name = str(file_path)
result = await self._aparse_one(
file_path,
file_name,
extra_info=extra_info,
fs=fs,
raise_job_error=raise_job_error,
file_path, file_name, extra_info=extra_info, fs=fs
)
return result[0] if len(result) == 1 else result
@@ -1481,7 +1453,6 @@ class LlamaParse(BasePydanticReader):
extra_info=extra_info,
fs=fs,
num_workers=1,
raise_job_error=raise_job_error,
)
for i, f in enumerate(file_path)
],
@@ -1507,7 +1478,6 @@ class LlamaParse(BasePydanticReader):
file_path: Union[List[FileInput], FileInput],
extra_info: Optional[dict] = None,
fs: Optional[AbstractFileSystem] = None,
raise_job_error: bool = True,
) -> Union[List["JobResult"], "JobResult"]:
"""
Parse the file and return a JobResult object instead of Document objects.
@@ -1519,17 +1489,12 @@ class LlamaParse(BasePydanticReader):
file_path: Path to the file to parse. Can be a string, path, bytes, file-like object, or a list of these.
extra_info: Additional metadata to include in the result.
fs: Optional filesystem to use for reading files.
raise_job_error: If True (default), raises JobFailedException when jobs fail. If False, returns JobResult objects with error information.
Returns:
JobResult object or list of JobResult objects if multiple files were provided
"""
try:
return asyncio_run(
self.aparse(
file_path, extra_info, fs=fs, raise_job_error=raise_job_error
)
)
return asyncio_run(self.aparse(file_path, extra_info, fs=fs))
except RuntimeError as e:
if nest_asyncio_err in str(e):
raise RuntimeError(nest_asyncio_msg)
@@ -1796,7 +1761,7 @@ class LlamaParse(BasePydanticReader):
return sub_docs
async def aget_result(
self, job_id: Union[str, List[str]], raise_job_error: bool = True
self, job_id: Union[str, List[str]]
) -> Union[JobResult, List[JobResult]]:
"""
Return JobResult object for previously parsed job(s).
@@ -1805,17 +1770,13 @@ class LlamaParse(BasePydanticReader):
Args:
job_id: Job ID or list of multiple Job IDs to be retrieved.
raise_job_error: If True (default), raises JobFailedException when jobs fail. If False, returns JobResult objects with error information.
Returns:
JobResult object or list of JobResult objects if multiple job IDs were provided.
"""
if isinstance(job_id, str):
result = await self._get_job_result(
job_id,
ResultType.JSON.value,
verbose=self.verbose,
raise_job_error=raise_job_error,
job_id, ResultType.JSON.value, verbose=self.verbose
)
return JobResult(
job_id=job_id,
@@ -1829,12 +1790,7 @@ class LlamaParse(BasePydanticReader):
elif isinstance(job_id, list):
results = []
jobs = [
self._get_job_result(
id_,
ResultType.JSON.value,
verbose=self.verbose,
raise_job_error=raise_job_error,
)
self._get_job_result(id_, ResultType.JSON.value, verbose=self.verbose)
for id_ in job_id
]
results = await run_jobs(
@@ -1859,7 +1815,7 @@ class LlamaParse(BasePydanticReader):
raise ValueError("The input job_id must be a string or a list of strings.")
def get_result(
self, job_id: Union[str, List[str]], raise_job_error: bool = True
self, job_id: Union[str, List[str]]
) -> Union[JobResult, List[JobResult]]:
"""
Return JobResult object for previously parsed job(s).
@@ -1868,15 +1824,12 @@ class LlamaParse(BasePydanticReader):
Args:
job_id: Job ID or list of multiple Job IDs to be retrieved.
raise_job_error: If True (default), raises JobFailedException when jobs fail. If False, returns JobResult objects with error information.
Returns:
JobResult object or list of JobResult objects if multiple job IDs were provided.
"""
try:
return asyncio_run(
self.aget_result(job_id, raise_job_error=raise_job_error)
)
return asyncio_run(self.aget_result(job_id))
except RuntimeError as e:
if nest_asyncio_err in str(e):
raise RuntimeError(nest_asyncio_msg)
+146 -27
View File
@@ -1,17 +1,87 @@
import httpx
import os
import re
from pydantic import BaseModel, Field, SerializeAsAny
from typing import Dict, Any, List, Optional
from pydantic import BaseModel, ConfigDict, Field, SerializeAsAny, model_validator
from typing import Dict, Any, List, Optional, get_origin, get_args
from llama_cloud_services.parse.utils import make_api_request
from llama_cloud_services.parse.utils import (
make_api_request,
is_jupyter,
)
from llama_index.core.async_utils import asyncio_run
from llama_index.core.schema import Document, ImageDocument, ImageNode, TextNode
PAGE_REGEX = r"page[-_](\d+)\.jpg$"
SAFE_MODEL_CONFIGS = ConfigDict(
extra="allow",
validate_assignment=False,
arbitrary_types_allowed=True,
validate_default=False,
)
class JobMetadata(BaseModel):
class SafeBaseModel(BaseModel):
"""Base model that gracefully handles None values from unstable backend responses."""
model_config = SAFE_MODEL_CONFIGS
@model_validator(mode="before")
@classmethod
def coerce_none_to_defaults(cls, data: Any) -> Any:
"""
Replace None values with appropriate defaults based on field type annotations.
This prevents validation errors when the backend returns None for non-optional fields.
"""
if not isinstance(data, dict):
return data
# Process each field that has a None value
result = {}
for key, value in data.items():
if value is not None or key not in cls.model_fields:
result[key] = value
continue
# Value is None and field exists in model
field_info = cls.model_fields[key]
# If field has a default or default_factory, let Pydantic handle it
from pydantic_core import PydanticUndefined
if (
field_info.default is not PydanticUndefined
or field_info.default_factory is not None
):
continue
# Otherwise, provide a sensible default based on the type annotation
annotation = field_info.annotation
origin = get_origin(annotation)
# Handle List types
if origin is list:
result[key] = []
# Handle Dict types
elif origin is dict:
result[key] = {}
# Handle basic types
elif annotation == str or (origin and str in get_args(annotation)):
result[key] = ""
elif annotation == int or (origin and int in get_args(annotation)):
result[key] = 0
elif annotation == float or (origin and float in get_args(annotation)):
result[key] = 0.0
elif annotation == bool or (origin and bool in get_args(annotation)):
result[key] = False
# If we can't determine a safe default, skip (let Pydantic try)
else:
result[key] = value
return result
class JobMetadata(SafeBaseModel):
"""Metadata about the job."""
job_pages: int = Field(default=0, description="The number of pages in the job.")
@@ -24,19 +94,31 @@ class JobMetadata(BaseModel):
)
class BBox(BaseModel):
class BBox(SafeBaseModel):
"""A bounding box."""
x: float = Field(description="The x-coordinate of the bounding box.")
y: float = Field(description="The y-coordinate of the bounding box.")
w: float = Field(description="The width of the bounding box.")
h: float = Field(description="The height of the bounding box.")
x: Optional[float] = Field(
default=None,
description="The x-coordinate of the bounding box.",
)
y: Optional[float] = Field(
default=None,
description="The y-coordinate of the bounding box.",
)
w: Optional[float] = Field(
default=None,
description="The width of the bounding box.",
)
h: Optional[float] = Field(
default=None,
description="The height of the bounding box.",
)
class PageItem(BaseModel):
class PageItem(SafeBaseModel):
"""An item in a page."""
type: str = Field(description="The type of the item.")
type: str = Field(default="", description="The type of the item.")
lvl: Optional[int] = Field(
default=None, description="The level of indentation of the item."
)
@@ -58,10 +140,10 @@ class PageItem(BaseModel):
)
class ImageItem(BaseModel):
class ImageItem(SafeBaseModel):
"""An image in a page."""
name: str = Field(description="The name of the image.")
name: str = Field(default="", description="The name of the image.")
height: Optional[float] = Field(
default=None, description="The height of the image."
)
@@ -81,22 +163,28 @@ class ImageItem(BaseModel):
type: Optional[str] = Field(default=None, description="The type of the image.")
class LayoutItem(BaseModel):
class LayoutItem(SafeBaseModel):
"""The layout of a page."""
image: str = Field(description="The name of the image containing the layout item")
confidence: float = Field(description="The confidence of the layout item.")
label: str = Field(description="The label of the layout item.")
image: str = Field(
default="", description="The name of the image containing the layout item"
)
confidence: float = Field(
default=0.0, description="The confidence of the layout item."
)
label: str = Field(default="", description="The label of the layout item.")
bbox: Optional[BBox] = Field(
default=None, description="The bounding box of the layout item."
)
isLikelyNoise: bool = Field(description="Whether the layout item is likely noise.")
isLikelyNoise: bool = Field(
default=False, description="Whether the layout item is likely noise."
)
class ChartItem(BaseModel):
class ChartItem(SafeBaseModel):
"""A chart in a page."""
name: str = Field(description="The name of the chart.")
name: str = Field(default="", description="The name of the chart.")
x: Optional[float] = Field(
default=None, description="The x-coordinate of the chart."
)
@@ -109,7 +197,7 @@ class ChartItem(BaseModel):
)
class Page(BaseModel):
class Page(SafeBaseModel):
"""A page of the document."""
page: int = Field(default=0, description="The page number.")
@@ -164,7 +252,7 @@ class Page(BaseModel):
)
class JobResult(BaseModel):
class JobResult(SafeBaseModel):
"""The raw JSON result from the LlamaParse API."""
pages: List[Page] = Field(
@@ -258,6 +346,29 @@ class JobResult(BaseModel):
documents = await self.aget_text_documents(split_by_page)
return [TextNode(text=doc.text, metadata=doc.metadata) for doc in documents]
def _format_markdown_for_notebook(self, text: Optional[str]) -> Optional[str]:
"""Format markdown text for Jupyter notebook display by escaping dollar signs."""
if text is None:
return None
def escape_single_dollar_signs(text: str) -> str:
"""Escape single dollar signs in text to prevent Jupyter from interpreting them as LaTeX.
Preserves all strings of dollar signs greater than length 1,
especially preserving double dollar signs ($$) which denote LaTeX equations.
Args:
text: The text to escape
Returns:
Text with single dollar signs escaped
"""
# Replace single $ with \$, but preserve $$
# Use negative lookahead and lookbehind to match $ not preceded or followed by $
return re.sub(r"(?<!\$)\$(?!\$)", r"\$", text)
return escape_single_dollar_signs(text)
def get_markdown_documents(self, split_by_page: bool = False) -> List[Document]:
"""
Get the markdown documents from the job.
@@ -268,17 +379,22 @@ class JobResult(BaseModel):
if split_by_page:
return [
Document(
text=page.md,
text=self._format_markdown_for_notebook(page.md)
if is_jupyter()
else page.md,
metadata={"page_number": page.page, "file_name": self.file_name},
)
for page in self.pages
]
else:
text = self._page_separator.join(
[page.md if page.md is not None else "" for page in self.pages]
)
return [
Document(
text=self._page_separator.join(
[page.md if page.md is not None else "" for page in self.pages]
),
text=self._format_markdown_for_notebook(text)
if is_jupyter()
else text,
metadata={"file_name": self.file_name},
)
]
@@ -328,7 +444,10 @@ class JobResult(BaseModel):
"""
url = f"{self._base_url}/api/v1/parsing/job/{self.job_id}/result/raw/markdown"
response = await make_api_request(self._client, "GET", url)
return response.content.decode("utf-8")
markdown = response.content.decode("utf-8")
return (
self._format_markdown_for_notebook(markdown) if is_jupyter() else markdown
)
def get_text(self) -> str:
"""
+12
View File
@@ -1,3 +1,4 @@
import functools
import httpx
import itertools
import logging
@@ -356,6 +357,17 @@ def partition_pages(
return
@functools.lru_cache(maxsize=1)
def is_jupyter() -> bool:
"""Check if we're running in a Jupyter environment."""
try:
from IPython import get_ipython
return get_ipython().__class__.__name__ == "ZMQInteractiveShell"
except (ImportError, AttributeError):
return False
def extract_tables_from_json_results(
json_results: List[dict], download_path: str
) -> List[str]:
+37
View File
@@ -0,0 +1,37 @@
# llama_parse
## 0.6.76
### Patch Changes
- Updated dependencies [4f24f53]
- llama-cloud-services-py@0.6.76
## 0.6.75
### Patch Changes
- Updated dependencies [f81532e]
- llama-cloud-services-py@0.6.75
## 0.6.74
### Patch Changes
- Updated dependencies [1bf5223]
- Updated dependencies [24166dc]
- llama-cloud-services-py@0.6.74
## 0.6.73
### Patch Changes
- Updated dependencies [e6a7939]
- llama-cloud-services-py@0.6.73
## 0.6.72
### Patch Changes
- Updated dependencies [ad6734b]
- llama-cloud-services-py@0.6.72
+20
View File
@@ -0,0 +1,20 @@
{
"name": "llama_parse",
"version": "0.6.76",
"description": "",
"main": "index.js",
"private": false,
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"dependencies": {
"llama-cloud-services-py": "workspace:*"
},
"keywords": [],
"author": "",
"license": "ISC",
"packageManager": "pnpm@10.11.1",
"devDependencies": {
"changesets": "^1.0.2"
}
}
+2 -2
View File
@@ -11,13 +11,13 @@ dev = [
[project]
name = "llama-parse"
version = "0.6.65"
version = "0.6.76"
description = "Parse files into RAG-Optimized formats."
authors = [{name = "Logan Markewich", email = "logan@llamaindex.ai"}]
requires-python = ">=3.9,<4.0"
readme = "README.md"
license = "MIT"
dependencies = ["llama-cloud-services>=0.6.64"]
dependencies = ["llama-cloud-services>=0.6.76"]
[project.scripts]
llama-parse = "llama_parse.cli.main:parse"
+10
View File
@@ -0,0 +1,10 @@
{
"name": "llama-cloud-services-py",
"version": "0.6.76",
"private": false,
"license": "MIT",
"scripts": {},
"devDependencies": {
"changesets": "^1.0.2"
}
}
+3 -3
View File
@@ -19,7 +19,7 @@ dev = [
[project]
name = "llama-cloud-services"
version = "0.6.65"
version = "0.6.76"
description = "Tailored SDK clients for LlamaCloud services."
authors = [{name = "Logan Markewich", email = "logan@runllama.ai"}]
requires-python = ">=3.9,<4.0"
@@ -27,14 +27,14 @@ readme = "README.md"
license = "MIT"
dependencies = [
"llama-index-core>=0.12.0",
"llama-cloud==0.1.41",
"llama-cloud==0.1.43",
"pydantic>=2.8,!=2.10",
"click>=8.1.7,<9",
"python-dotenv>=1.0.1,<2",
"eval-type-backport>=0.2.0,<0.3 ; python_version < '3.10'",
"platformdirs>=4.3.7,<5",
"tenacity>=8.5.0, <10.0",
"packaging>=25.0"
"packaging>=23.0"
]
[project.scripts]
@@ -68,7 +68,7 @@ async def test_agent_data_crud_operations():
client=client,
type=ExampleData,
collection=f"test-collection-{test_id[:8]}",
agent_url_id=LLAMA_DEPLOY_DEPLOYMENT_NAME,
deployment_name=LLAMA_DEPLOY_DEPLOYMENT_NAME,
)
# Create test data
+1 -28
View File
@@ -1,16 +1,13 @@
import os
import pytest
from llama_cloud_services.extract import LlamaExtract, ExtractionAgent
from time import perf_counter
from llama_cloud_services.extract import LlamaExtract
from collections import namedtuple
import json
import uuid
from llama_cloud.types import (
ExtractConfig,
ExtractMode,
LlamaParseParameters,
LlamaExtractSettings,
)
from tests.extract.util import load_test_dotenv
@@ -122,27 +119,3 @@ def extraction_agent(test_case: BenchmarkTestCase, extractor: LlamaExtract):
# Create new agent
agent = extractor.create_agent(agent_name, schema, config=test_case.config)
yield agent
@pytest.mark.skipif(
"CI" in os.environ or not LLAMA_CLOUD_API_KEY,
reason="LLAMA_CLOUD_API_KEY not set or CI environment not suitable for benchmarking",
)
@pytest.mark.parametrize("test_case", get_test_cases(), ids=lambda x: x.name)
@pytest.mark.asyncio(loop_scope="session")
async def test_extraction(
test_case: BenchmarkTestCase, extraction_agent: ExtractionAgent
) -> None:
start = perf_counter()
result = await extraction_agent._run_extraction_test(
test_case.input_file,
extract_settings=LlamaExtractSettings(
llama_parse_params=LlamaParseParameters(
invalidate_cache=True,
do_not_cache=True,
)
),
)
end = perf_counter()
print(f"Time taken: {end - start} seconds")
print(result)
+1 -1
View File
@@ -7,7 +7,7 @@ from pathlib import Path
def load_test_dotenv():
load_dotenv(Path(__file__).parent.parent.parent / ".env.dev", override=True)
load_dotenv(Path(__file__).parent.parent.parent.parent / ".env.dev", override=True)
def json_subset_match_score(expected: Any, actual: Any) -> float:
+3
View File
@@ -304,6 +304,9 @@ async def test_page_screenshot_retrieval(index_name: str, local_file: str):
not base_url or not api_key, reason="No platform base url or api key set"
)
@pytest.mark.asyncio
@pytest.mark.skip(
reason="Consistently failing with FAILED tests/index/test_index.py::test_page_figure_retrieval - assert 0 > 0 + where 0 = len([])"
)
async def test_page_figure_retrieval(index_name: str, local_figures_file: str):
index = await LlamaCloudIndex.acreate_index(
name=index_name,
-196
View File
@@ -2,7 +2,6 @@ import os
import pytest
import shutil
from typing import Optional, cast
from unittest.mock import patch
from fsspec.implementations.local import LocalFileSystem
from httpx import AsyncClient
@@ -205,10 +204,6 @@ async def test_get_result(markdown_parser: LlamaParse) -> None:
assert len(result.pages) == len(expected.pages)
@pytest.mark.skipif(
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
reason="LLAMA_CLOUD_API_KEY not set",
)
@pytest.mark.asyncio
async def test_parse_audio() -> None:
parser = LlamaParse()
@@ -216,194 +211,3 @@ async def test_parse_audio() -> None:
result = await parser.aparse(filepath)
assert result.job_id is not None
@pytest.mark.asyncio
async def test_error_handling_with_raise_job_error_false() -> None:
"""Test that failed jobs return JobResult objects with error information when raise_job_error=False."""
parser = LlamaParse(api_key="test_key")
# Mock error result with full error information
mock_error_result = {
"pages": [],
"job_metadata": {"job_pages": 0},
"error": "Job ID: test_job_123 failed with status: ERROR, Error code: INVALID_FILE, Error message: File format not supported",
"status": "ERROR",
}
with patch.object(parser, "_create_job", return_value="test_job_123"), patch.object(
parser, "_get_job_result", return_value=mock_error_result
):
# Test aparse with raise_job_error=False
result = await parser.aparse("test_file.txt", raise_job_error=False)
assert isinstance(result, type(result)) # Check it's a JobResult
assert result.job_id == "test_job_123"
assert result.error is not None
assert "ERROR" in result.error
assert "INVALID_FILE" in result.error
assert "File format not supported" in result.error
assert len(result.pages) == 0
# Test parse (synchronous version) with raise_job_error=False
result_sync = parser.parse("test_file.txt", raise_job_error=False)
assert isinstance(result_sync, type(result_sync))
assert result_sync.job_id == "test_job_123"
assert result_sync.error is not None
assert "INVALID_FILE" in result_sync.error
assert "File format not supported" in result_sync.error
@pytest.mark.asyncio
async def test_error_handling_with_raise_job_error_true() -> None:
"""Test that failed jobs raise JobFailedException when raise_job_error=True (default behavior)."""
parser = LlamaParse(api_key="test_key")
# Mock that _get_job_result will raise JobFailedException when raise_job_error=True
from llama_cloud_services.parse.base import JobFailedException
def mock_get_job_result(job_id, result_type, verbose=False, raise_job_error=True):
if raise_job_error:
raise JobFailedException(
"test_job_123",
"ERROR",
error_code="INVALID_FILE",
error_message="File format not supported",
)
else:
return {
"pages": [],
"job_metadata": {"job_pages": 0},
"error": "Job ID: test_job_123 failed with status: ERROR, Error code: INVALID_FILE, Error message: File format not supported",
"status": "ERROR",
}
with patch.object(parser, "_create_job", return_value="test_job_123"), patch.object(
parser, "_get_job_result", side_effect=mock_get_job_result
):
# Test aparse with raise_job_error=True (default) - should raise exception
with pytest.raises(JobFailedException) as exc_info:
await parser.aparse("test_file.txt")
assert exc_info.value.job_id == "test_job_123"
assert exc_info.value.status == "ERROR"
assert exc_info.value.error_code == "INVALID_FILE"
# Test parse (synchronous version) with raise_job_error=True (default) - should raise exception
with pytest.raises(JobFailedException) as exc_info:
parser.parse("test_file.txt")
assert exc_info.value.job_id == "test_job_123"
assert exc_info.value.status == "ERROR"
@pytest.mark.asyncio
async def test_error_handling_with_minimal_fields() -> None:
"""Test error handling when only status is available (no error_code/error_message) with raise_job_error=False."""
parser = LlamaParse(api_key="test_key")
# Mock error result with minimal fields (only what's guaranteed)
mock_minimal_error_result = {
"pages": [],
"job_metadata": {"job_pages": 0},
"error": "Job ID: test_job_456 failed with status: CANCELED",
"status": "CANCELED",
}
with patch.object(parser, "_create_job", return_value="test_job_456"), patch.object(
parser, "_get_job_result", return_value=mock_minimal_error_result
):
# Test aparse with a minimal error response and raise_job_error=False
result = await parser.aparse("test_file.txt", raise_job_error=False)
assert isinstance(result, type(result))
assert result.job_id == "test_job_456"
assert result.error is not None
assert "CANCELED" in result.error
assert len(result.pages) == 0
@pytest.mark.asyncio
async def test_successful_job_still_works() -> None:
"""Test that successful jobs still work as before after error handling changes."""
parser = LlamaParse(api_key="test_key")
# Mock successful result
mock_success_result = {
"pages": [
{
"page": 0,
"text": "Sample text content",
"md": "# Sample markdown content",
"images": [],
"charts": [],
"tables": [],
"layout": [],
"items": [],
"status": "SUCCESS",
"links": [],
"width": 612.0,
"height": 792.0,
}
],
"job_metadata": {"job_pages": 1},
}
with patch.object(
parser, "_create_job", return_value="success_job_456"
), patch.object(parser, "_get_job_result", return_value=mock_success_result):
# Test aparse with a successful job (both with and without raise_job_error parameter)
result = await parser.aparse("test_file.txt")
assert isinstance(result, type(result))
assert result.job_id == "success_job_456"
assert result.error is None # No error for successful jobs
assert len(result.pages) == 1
assert result.pages[0].text == "Sample text content"
# Test with explicit raise_job_error=False (should work the same for successful jobs)
result2 = await parser.aparse("test_file.txt", raise_job_error=False)
assert isinstance(result2, type(result2))
assert result2.job_id == "success_job_456"
assert result2.error is None
assert len(result2.pages) == 1
assert result2.pages[0].text == "Sample text content"
@pytest.mark.asyncio
async def test_get_result_with_raise_job_error_parameter() -> None:
"""Test that get_result method respects the raise_job_error parameter."""
parser = LlamaParse(api_key="test_key")
# Mock error result
mock_error_result = {
"pages": [],
"job_metadata": {"job_pages": 0},
"error": "Job ID: test_job_789 failed with status: ERROR, Error code: TIMEOUT, Error message: Job timed out",
"status": "ERROR",
}
with patch.object(parser, "_get_job_result", return_value=mock_error_result):
# Test aget_result with raise_job_error=False
result = await parser.aget_result("test_job_789", raise_job_error=False)
assert isinstance(result, type(result))
assert result.job_id == "test_job_789"
assert result.error is not None
assert "TIMEOUT" in result.error
assert len(result.pages) == 0
# Test get_result (synchronous version) with raise_job_error=False
result_sync = parser.get_result("test_job_789", raise_job_error=False)
assert isinstance(result_sync, type(result_sync))
assert result_sync.job_id == "test_job_789"
assert result_sync.error is not None
assert "TIMEOUT" in result_sync.error
+34
View File
@@ -6,6 +6,40 @@ from llama_cloud_services import LlamaParse
from llama_cloud_services.parse.types import JobResult
def test_format_parse_result_markdown_for_notebook():
"""Test the _format_markdown_for_notebook function.
Right now, the only work it does is escape single dollar signs."""
result = JobResult(job_id="test", file_name="test.pdf", job_result={})
# Test None input
assert result._format_markdown_for_notebook(None) is None
# Test single dollar sign gets escaped
assert result._format_markdown_for_notebook("This costs $5") == "This costs \\$5"
# Test double dollar signs are preserved (LaTeX equations)
assert (
result._format_markdown_for_notebook("$$x^2 + y^2 = z^2$$")
== "$$x^2 + y^2 = z^2$$"
)
# Test mixed single and double dollar signs
text = "This costs $5, but $$E = mc^2$$ is priceless"
expected = "This costs \\$5, but $$E = mc^2$$ is priceless"
assert result._format_markdown_for_notebook(text) == expected
# Test multiple single dollar signs
assert result._format_markdown_for_notebook("$10 and $20") == "\\$10 and \\$20"
# Test three or more consecutive dollar signs (preserve them)
assert result._format_markdown_for_notebook("$$$") == "$$$"
# Test adjacent dollar signs with text in between
text = "$$inline$$ and $separate"
expected = "$$inline$$ and \\$separate"
assert result._format_markdown_for_notebook(text) == expected
@pytest.fixture
def file_path() -> str:
return "tests/test_files/attention_is_all_you_need.pdf"
@@ -0,0 +1,158 @@
import pytest
from typing import Any, Dict, List, Optional
from pydantic import BaseModel
from datetime import datetime
from llama_cloud.types.agent_data import AgentData
from llama_cloud.types.aggregate_group import AggregateGroup
from llama_cloud_services.beta.agent_data.client import AsyncAgentDataClient
class Person(BaseModel):
name: str
age: int
class FakeBeta:
def __init__(self) -> None:
self._get_item_response: Optional[AgentData] = None
self._search_items: List[AgentData] = []
self._aggregate_items: List[AggregateGroup] = []
self._total_size: Optional[int] = None
self._next_page_token: Optional[str] = None
# Single get
async def get_agent_data(self, item_id: str) -> AgentData:
assert self._get_item_response is not None, "_get_item_response not set"
return self._get_item_response
# Search
async def search_agent_data_api_v_1_beta_agent_data_search_post(
self,
*,
deployment_name: str,
collection: str,
filter: Optional[Dict[str, Any]] = None,
order_by: Optional[str] = None,
offset: Optional[int] = None,
page_size: Optional[int] = None,
include_total: bool = False,
) -> Any:
class Resp:
def __init__(
self,
items: List[AgentData],
total_size: Optional[int],
next_page_token: Optional[str],
) -> None:
self.items = items
self.total_size = total_size
self.next_page_token = next_page_token
return Resp(self._search_items, self._total_size, self._next_page_token)
# Aggregate
async def aggregate_agent_data_api_v_1_beta_agent_data_aggregate_post(
self,
*,
deployment_name: str,
collection: str,
page_size: Optional[int] = None,
filter: Optional[Dict[str, Any]] = None,
order_by: Optional[str] = None,
group_by: Optional[List[str]] = None,
count: Optional[bool] = None,
first: Optional[bool] = None,
offset: Optional[int] = None,
) -> Any:
class Resp:
def __init__(
self,
items: List[AggregateGroup],
total_size: Optional[int],
next_page_token: Optional[str],
) -> None:
self.items = items
self.total_size = total_size
self.next_page_token = next_page_token
return Resp(self._aggregate_items, self._total_size, self._next_page_token)
class FakeClient:
def __init__(self) -> None:
self.beta = FakeBeta()
def make_agent_data(data: Dict[str, Any]) -> AgentData:
return AgentData(
id="id-1",
deployment_name="dep",
collection="col",
data=data,
created_at=datetime.now(),
updated_at=datetime.now(),
)
def make_group(
group_key: Dict[str, Any],
first_item: Optional[Dict[str, Any]],
count: Optional[int] = None,
) -> AggregateGroup:
return AggregateGroup(group_key=group_key, count=count, first_item=first_item)
@pytest.mark.asyncio
async def test_untyped_get_item_valid_to_dict() -> None:
client = FakeClient()
client.beta._get_item_response = make_agent_data({"name": "Alice", "age": 30})
adc = AsyncAgentDataClient(type=Person, client=client, deployment_name="dep")
item = await adc.untyped_get_item("id-1")
assert item.data == {"name": "Alice", "age": 30}
@pytest.mark.asyncio
async def test_untyped_get_item_invalid_retains_dict() -> None:
client = FakeClient()
# age wrong type; will fail validation and should be returned as dict
client.beta._get_item_response = make_agent_data({"name": "Bob", "age": "x"})
adc = AsyncAgentDataClient(type=Person, client=client, deployment_name="dep")
item = await adc.untyped_get_item("id-1")
assert item.data == {"name": "Bob", "age": "x"}
@pytest.mark.asyncio
async def test_untyped_search_mixed_items() -> None:
client = FakeClient()
client.beta._search_items = [
make_agent_data({"name": "Carol", "age": 22}),
make_agent_data({"name": "Dave", "age": "bad"}),
]
client.beta._total_size = 2
adc = AsyncAgentDataClient(type=Person, client=client, deployment_name="dep")
results = await adc.untyped_search(include_total=True)
assert len(results.items) == 2
assert results.items[0].data == {"name": "Carol", "age": 22}
assert results.items[1].data == {"name": "Dave", "age": "bad"}
assert results.total_size == 2
@pytest.mark.asyncio
async def test_untyped_aggregate_first_item_dict() -> None:
client = FakeClient()
client.beta._aggregate_items = [
make_group({"k": 1}, {"name": "Eve", "age": 40}),
make_group({"k": 2}, {"name": "Frank", "age": "bad"}),
]
client.beta._total_size = 2
adc = AsyncAgentDataClient(type=Person, client=client, deployment_name="dep")
results = await adc.untyped_aggregate(group_by=["k"], first=True)
assert len(results.items) == 2
assert results.items[0].first_item == {"name": "Eve", "age": 40}
assert results.items[1].first_item == {"name": "Frank", "age": "bad"}
@@ -38,7 +38,7 @@ def test_typed_agent_data_from_raw():
"""Test TypedAgentData.from_raw class method."""
raw_data = AgentData(
id="456",
agent_slug="extraction-agent",
deployment_name="extraction-agent",
collection="employees",
data={"name": "Jane Smith", "age": 25, "email": "jane@company.com"},
created_at=datetime.now(),
@@ -48,7 +48,7 @@ def test_typed_agent_data_from_raw():
typed_data = TypedAgentData.from_raw(raw_data, Person)
assert typed_data.id == "456"
assert typed_data.agent_url_id == "extraction-agent"
assert typed_data.deployment_name == "extraction-agent"
assert typed_data.collection == "employees"
assert typed_data.data.name == "Jane Smith"
assert typed_data.data.age == 25
@@ -56,10 +56,10 @@ def test_typed_agent_data_from_raw():
def test_typed_agent_data_from_raw_validation_error():
"""Test TypedAgentData.from_raw with invalid data."""
"""Test TypedAgentData.from_raw with invalid data now raises InvalidTypedAgentData."""
raw_data = AgentData(
id="789",
agent_slug="test-agent",
deployment_name="test-agent",
collection="people",
data={"name": "Invalid Person", "age": "not_a_number"}, # Invalid age
created_at=datetime.now(),
@@ -613,3 +613,51 @@ def test_parses_field_metadata_with_error_field():
}
assert parsed.metadata.get("field_errors") == "This is an error"
assert parsed.metadata.get("job_id") == "job-123"
REASONING_IN_SCHEMA = {
"majority_opinion": {
"type": {
"citation": [
{
"page": 4,
"matching_text": "BARRETT, J., delivered the opinion for a unanimous Court.",
},
{"page": 11, "matching_text": "Opinion of the Court"},
],
"parsing_confidence": 1.0,
"extraction_confidence": 0.9999998919950147,
"confidence": 0.9999998919950147,
},
"reasoning": {
"citation": [
{
"page": 15,
"matching_text": "We hold that §5110(b)(1) is not subject to equitable tolling and affirm the judg...",
}
],
"parsing_confidence": 1.0,
"extraction_confidence": 0.414292785946868,
"confidence": 0.414292785946868,
},
},
"reasoning": {
"citation": [
{
"page": 15,
"matching_text": "We hold that §5110(b)(1) is not subject to equitable tolling and affirm the judg...",
}
],
"parsing_confidence": 1.0,
"extraction_confidence": 0.414292785946868,
"confidence": 0.414292785946868,
},
}
def test_field_conflict_in_schema():
extracted = parse_extracted_field_metadata(REASONING_IN_SCHEMA)
assert isinstance(extracted["reasoning"], ExtractedFieldMetadata)
assert isinstance(
extracted["majority_opinion"]["reasoning"], ExtractedFieldMetadata
)
+2 -4
View File
@@ -118,10 +118,8 @@ async def test_extraction_agent_aextract_accepts_llama_file(
dummy_llama_extract_iface = SimpleNamespace()
async def fake_run_job(**kwargs):
# Ensure we are receiving a request with the right file_id
request = kwargs.get("request")
assert hasattr(request, "file_id")
assert request.file_id == llama_file.id
file_id = kwargs.get("file_id")
assert file_id == llama_file.id
return SimpleNamespace(id="job_42")
dummy_llama_extract_iface.run_job = fake_run_job
+143 -4
View File
@@ -1,16 +1,155 @@
import pytest
from unittest.mock import MagicMock, patch
import llama_cloud_services.index.base as base
from llama_cloud import (
PipelineEmbeddingConfig_ManagedOpenaiEmbedding,
Project,
Pipeline,
CloudDocument,
)
from llama_index.core.constants import DEFAULT_PROJECT_NAME
from llama_index.core.indices.managed.base import BaseManagedIndex
from llama_cloud_services.index import (
LlamaCloudIndex,
from llama_index.core.schema import Document
from llama_cloud_services.index import LlamaCloudIndex
# Simple test data as values, not fixtures
TEST_PROJECT = Project(id="proj-123", name="test-project", organization_id="org-123")
EMBEDDING_CONFIG = PipelineEmbeddingConfig_ManagedOpenaiEmbedding(
type="MANAGED_OPENAI_EMBEDDING"
)
TEST_PIPELINE = Pipeline(
id="pipe-456",
name="test-pipeline",
project_id="proj-123",
embedding_config=PipelineEmbeddingConfig_ManagedOpenaiEmbedding(
type="MANAGED_OPENAI_EMBEDDING"
),
)
def test_class():
@pytest.fixture
def mock_client() -> MagicMock:
"""Mock client with sensible defaults."""
client = MagicMock()
client.projects.upsert_project.return_value = Project(
id="default-proj", name=DEFAULT_PROJECT_NAME, organization_id="default-org"
)
client.pipelines.upsert_pipeline.return_value = Pipeline(
id="default-pipe",
name="default",
project_id="default-proj",
embedding_config=EMBEDDING_CONFIG,
)
client.pipelines.upsert_batch_pipeline_documents.return_value = [
CloudDocument(id="doc-1", text="test", metadata={})
]
return client
@pytest.fixture(autouse=True)
def base_patches(mock_client: MagicMock) -> None:
"""Auto-applied patches for all tests."""
with (
patch.object(base, "get_client", return_value=mock_client),
patch.object(
base,
"resolve_project_and_pipeline",
return_value=(TEST_PROJECT, TEST_PIPELINE),
),
patch.object(base.LlamaCloudIndex, "wait_for_completion"),
):
yield
def test_class() -> None:
names_of_base_classes = [b.__name__ for b in LlamaCloudIndex.__mro__]
assert BaseManagedIndex.__name__ in names_of_base_classes
def test_conflicting_index_identifiers():
def test_conflicting_index_identifiers() -> None:
with pytest.raises(ValueError):
LlamaCloudIndex(name="test", pipeline_id="test", index_id="test")
def test_from_documents_uses_provided_project_id(mock_client: MagicMock) -> None:
provided_project_id = "proj-123"
organization_id = "org-abc"
index_name = "my_new_index"
# Override resolve to return project with provided ID
test_project = Project(
id=provided_project_id, name="my_project", organization_id=organization_id
)
test_pipeline = Pipeline(
id="pipe-xyz",
name=index_name,
project_id=provided_project_id,
embedding_config=EMBEDDING_CONFIG,
)
with patch.object(
base, "resolve_project_and_pipeline", return_value=(test_project, test_pipeline)
):
docs = [Document(text="hello")]
index = LlamaCloudIndex.from_documents(
documents=docs,
name=index_name,
project_id=provided_project_id,
)
# Assert - project upsert not called; pipeline uses provided project_id
mock_client.projects.upsert_project.assert_not_called()
assert mock_client.pipelines.upsert_pipeline.call_count == 1
assert (
mock_client.pipelines.upsert_pipeline.call_args.kwargs["project_id"]
== provided_project_id
)
assert index.project.id == provided_project_id
def test_from_documents_upserts_project_when_project_id_missing(
mock_client: MagicMock,
) -> None:
organization_id = "org-xyz"
index_name = "my_new_index"
# Project is created when project_id is not provided
upserted_project = Project(
id="proj-999", name=DEFAULT_PROJECT_NAME, organization_id=organization_id
)
mock_client.projects.upsert_project.return_value = upserted_project
test_pipeline = Pipeline(
id="pipe-xyz",
name=index_name,
project_id=upserted_project.id,
embedding_config=EMBEDDING_CONFIG,
)
with patch.object(
base,
"resolve_project_and_pipeline",
return_value=(upserted_project, test_pipeline),
):
docs = [Document(text="world")]
index = LlamaCloudIndex.from_documents(
documents=docs,
name=index_name,
organization_id=organization_id,
)
# Assert - project was upserted with org id and default project name
mock_client.projects.upsert_project.assert_called_once()
kwargs = mock_client.projects.upsert_project.call_args.kwargs
assert kwargs["organization_id"] == organization_id
assert kwargs["request"].name == DEFAULT_PROJECT_NAME
# Pipeline created under the upserted project id
assert (
mock_client.pipelines.upsert_pipeline.call_args.kwargs["project_id"]
== upserted_project.id
)
assert index.project.id == upserted_project.id
Generated
+7 -7
View File
@@ -1,5 +1,5 @@
version = 1
revision = 2
revision = 3
requires-python = ">=3.9, <4.0"
resolution-markers = [
"python_full_version >= '3.14'",
@@ -1582,21 +1582,21 @@ wheels = [
[[package]]
name = "llama-cloud"
version = "0.1.41"
version = "0.1.43"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "certifi" },
{ name = "httpx" },
{ name = "pydantic" },
]
sdist = { url = "https://files.pythonhosted.org/packages/62/6c/b2e84eebed376aea34c446cab745da5fc4e9dc53309180672299083219d5/llama_cloud-0.1.41.tar.gz", hash = "sha256:dcb741b779e3e740cd64928cfffc8ef70ed0e9bae9ef26acbe1d7e32aa737bdc", size = 109854, upload-time = "2025-09-05T22:45:13.069Z" }
sdist = { url = "https://files.pythonhosted.org/packages/9b/33/33a8bd3a617c071caf450ca2627969f8b28272d0692f122997c10a32247e/llama_cloud-0.1.43.tar.gz", hash = "sha256:00429f05aea515449d90cde91ef3ed3687fcd93e46f6246d08cbea02f9b397a9", size = 112992, upload-time = "2025-10-02T21:55:38.355Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/1e/4d/f0af76b389310840ce3483a92560a152025b0eefe4eee0c81102bf3317e6/llama_cloud-0.1.41-py3-none-any.whl", hash = "sha256:c847f288f0d3f4b23f47345088006deae5f2cf3f223ac1819d4c1531e9aaa13e", size = 307646, upload-time = "2025-09-05T22:45:11.597Z" },
{ url = "https://files.pythonhosted.org/packages/2b/54/559a67542396d5660a71115b29e0160e9dd784e570e1f4ef55ad22bf5b39/llama_cloud-0.1.43-py3-none-any.whl", hash = "sha256:540605d4dd13c6536a3b75cd4d04b211f29b16d17faee9381e3793a651f1dec1", size = 311460, upload-time = "2025-10-02T21:55:37.282Z" },
]
[[package]]
name = "llama-cloud-services"
version = "0.6.65"
version = "0.6.73"
source = { editable = "." }
dependencies = [
{ name = "click", version = "8.1.8", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" },
@@ -1631,9 +1631,9 @@ dev = [
requires-dist = [
{ name = "click", specifier = ">=8.1.7,<9" },
{ name = "eval-type-backport", marker = "python_full_version < '3.10'", specifier = ">=0.2.0,<0.3" },
{ name = "llama-cloud", specifier = "==0.1.41" },
{ name = "llama-cloud", specifier = "==0.1.43" },
{ name = "llama-index-core", specifier = ">=0.12.0" },
{ name = "packaging", specifier = ">=25.0" },
{ name = "packaging", specifier = ">=23.0" },
{ name = "platformdirs", specifier = ">=4.3.7,<5" },
{ name = "pydantic", specifier = ">=2.8,!=2.10" },
{ name = "python-dotenv", specifier = ">=1.0.1,<2" },
+290
View File
@@ -0,0 +1,290 @@
#!/usr/bin/env -S uv run --script
# /// script
# dependencies = ["click", "tomlkit", "packaging"]
# ///
"""
This is a script called by the changeset bot. Normally changeset can do the following things, but this is a mixed ts and python repo, so we need to do some extra things.
There's 2 things this does:
- Versioning: Makes changes that may be committed with the newest version.
- Releasing/Tagging: After versions are changed, we check each package to see if its released, and if not, we release it and tag it.
"""
from dataclasses import dataclass
import json
import os
import subprocess
from pathlib import Path
from typing import Any, List, cast
import urllib.request
import urllib.error
import re
import click
import tomlkit
from packaging.version import Version
def _run_command(
cmd: List[str], cwd: Path | None = None, env: dict[str, str] | None = None
) -> None:
"""Run a command, streaming output to the console, and raise on failure."""
subprocess.run(cmd, check=True, text=True, cwd=cwd or Path.cwd(), env=env)
def _run_and_capture(
cmd: List[str], cwd: Path | None = None, env: dict[str, str] | None = None
) -> str:
"""Run a command and return stdout as text, raising on failure."""
result = subprocess.run(
cmd,
check=True,
text=True,
cwd=cwd or Path.cwd(),
env=env,
capture_output=True,
)
return result.stdout
@dataclass
class Package:
name: str
version: str
path: Path
def python_package_name(self) -> str | None:
if "/py/" in str(self.path) or str(self.path).endswith("/py"):
return self.name.removesuffix("-py")
return None
def _get_pnpm_workspace_packages() -> list[Package]:
"""Return directories for all workspace packages from pnpm list JSON output."""
output = _run_and_capture(["pnpm", "list", "-r", "--depth=-1", "--json"])
data = cast(list[dict[str, Any]], json.loads(output))
packages: list[Package] = [
Package(name=data["name"], version=data["version"], path=Path(data["path"]))
for data in data
]
return packages
def _sync_package_version_with_pyproject(
package_dir: Path, packages: dict[str, Package], js_package_name: str
) -> None:
"""Sync version from package.json to pyproject.toml.
Returns True if pyproject was changed, else False.
"""
pyproject_path = package_dir / "pyproject.toml"
if not pyproject_path.exists():
return
package_version = packages[js_package_name].version
py_doc = tomlkit.parse(pyproject_path.read_text())
by_python_name = {
pkg.python_package_name(): pkg
for pkg in packages.values()
if pkg.python_package_name()
}
current_version = py_doc["project"]["version"]
assert isinstance(current_version, str)
# update workspace dependency strings by replacing the first version after == or >=
deps = py_doc["project"]["dependencies"] or []
changed = False
for i, dep in enumerate(deps):
if not isinstance(dep, str):
continue
pkg = (cast(str, dep).split("==")[0]).split(">=")[0]
if pkg not in by_python_name:
continue
target_version = by_python_name[pkg].version
new_dep = re.sub(
r"(==|>=)\s*([0-9A-Za-z_.+-]+)",
lambda m: m.group(1) + target_version,
dep,
count=1,
)
if new_dep != dep:
deps[i] = new_dep
changed = True
if current_version != package_version:
py_doc["project"]["version"] = package_version
changed = True
if changed:
pyproject_path.write_text(tomlkit.dumps(py_doc))
click.echo(
f"Updated {pyproject_path} version to {package_version} and synced dependency specs"
)
def lock_python_dependencies() -> None:
"""Lock Python dependencies."""
try:
_run_command(["uv", "lock"])
click.echo("Locked Python dependencies")
except subprocess.CalledProcessError as e:
click.echo(f"Warning: Failed to lock Python dependencies: {e}", err=True)
@click.group()
def cli() -> None:
"""Changeset-based version management for llama-cloud-services."""
pass
@cli.command()
def version() -> None:
"""Apply changeset versions, then sync versions for co-located JS/Py packages.
- Runs changesets to bump package.json versions.
- Discovers all workspace packages via pnpm.
- For any directory containing both package.json and pyproject.toml, and with
package.json private: false, set pyproject [project].version to match the JS version.
- If a pyproject is updated, run `uv sync` in that directory to update its lock file.
"""
# Ensure we're at the repo root
os.chdir(Path(__file__).parent.parent)
# First, run changeset version to update all package.json files
_run_command(["npx", "@changesets/cli", "version"])
# Enumerate workspace packages and perform syncs
packages = _get_pnpm_workspace_packages()
version_map = {pkg.name: pkg for pkg in packages}
for pkg in packages:
_sync_package_version_with_pyproject(pkg.path, version_map, pkg.name)
@cli.command()
@click.option("--tag", is_flag=True, help="Tag the packages after publishing")
@click.option("--dry-run", is_flag=True, help="Dry run the publish")
@click.option("--js/--no-js", default=True, help="Publish the js package")
@click.option("--py/--no-py", default=True, help="Publish the py package")
def publish(tag: bool, dry_run: bool, js: bool, py: bool) -> None:
"""Publish all packages."""
# move to the root
os.chdir(Path(__file__).parent.parent)
if js:
if not os.getenv("NPM_TOKEN"):
click.echo("NPM_TOKEN is not set, skipping publish", err=True)
raise click.Abort("No token set")
if py:
if not os.getenv("LLAMA_PARSE_PYPI_TOKEN"):
click.echo("LLAMA_PARSE_PYPI_TOKEN is not set, skipping publish", err=True)
raise click.Abort("No token set")
# not general script. Just checks each of the 2 packages to see if they need to be published.
if js:
maybe_publish_npm(dry_run)
if py:
maybe_publish_pypi(dry_run)
if tag:
if dry_run:
click.echo("Dry run, skipping tag. Would run:")
click.echo(" npx @changesets/cli tag")
click.echo(" git push --tags")
else:
# Let changesets create JS-related tags as usual
_run_command(["npx", "@changesets/cli", "tag"])
_run_command(["git", "push", "--tags"])
def maybe_publish_npm(dry_run: bool) -> None:
"""Publish the ts package if it needs to be published."""
target_dir = Path("ts/llama_cloud_services")
ts_path_package = target_dir / "package.json"
package_json = json.loads(ts_path_package.read_text())
version = package_json["version"]
# Check if this version is already published on npm
result = subprocess.run(
["npm", "view", "llama-cloud-services", "versions", "--json"],
check=True,
capture_output=True,
text=True,
cwd=target_dir,
)
published_versions = json.loads(result.stdout)
if version in published_versions:
click.echo(
f"npm package llama-cloud-services@{version} already published, skipping"
)
return
click.echo(f"Publishing npm package llama-cloud-services@{version}")
# defer to the package.json publish script
if dry_run:
click.echo("Dry run, skipping publish. Would run:")
click.echo(" pnpm run publish")
return
else:
_run_command(["pnpm", "run", "build"], cwd=target_dir)
_run_command(["pnpm", "publish"], cwd=target_dir)
def maybe_publish_pypi(dry_run: bool) -> None:
"""Publish the py packages if they need to be published."""
for pyproject in list(Path("py").glob("*/pyproject.toml")) + [
Path("py/pyproject.toml")
]:
name, version = current_version(pyproject)
if is_published(name, version):
click.echo(f"PyPI package {name}@{version} already published, skipping")
continue
click.echo(f"Publishing PyPI package {name}@{version}")
# Use different tokens for different packages
env = os.environ.copy()
token = os.environ["LLAMA_PARSE_PYPI_TOKEN"]
env["UV_PUBLISH_TOKEN"] = token
if dry_run:
summary = (token[:3] + "***") if len(token) <= 6 else token[:6] + "****"
click.echo(
f"Dry run, skipping publish. Would run with publish token {summary}:"
)
click.echo(" uv build")
click.echo(" uv publish")
else:
_run_command(["uv", "build"], cwd=pyproject.parent)
_run_command(["uv", "publish"], cwd=pyproject.parent, env=env)
def current_version(pyproject: Path) -> tuple[str, str]:
"""Return (package_name, version_str) taken from the given pyproject.toml."""
doc = tomlkit.parse(pyproject.read_text())
name = doc["project"]["name"]
version = str(Version(doc["project"]["version"])) # normalise
return name, version
def is_published(
name: str, version: str, index_url: str = "https://pypi.org/pypi"
) -> bool:
"""
True → `<name>==<version>` exists on the given index
False → package missing *or* version missing
"""
url = f"{index_url.rstrip('/')}/{name}/json"
try:
data = json.load(urllib.request.urlopen(url))
except urllib.error.HTTPError as e: # 404 → package not published at all
if e.code == 404:
return False
raise # any other error should surface
return version in data["releases"] # keys are version strings
if __name__ == "__main__":
cli()
-226
View File
@@ -1,226 +0,0 @@
#!/usr/bin/env -S uv run --script
# /// script
# dependencies = ["click", "tomlkit"]
# ///
import click
import subprocess
import sys
import tomlkit
from pathlib import Path
import json
def get_current_versions() -> tuple[str, str, str, str | None]:
"""Get current versions from both pyproject.toml files and TS package.json."""
# Read main pyproject.toml
main_content = Path("py/pyproject.toml").read_text()
main_doc = tomlkit.parse(main_content)
main_version = main_doc["project"]["version"]
# Read llama_parse/pyproject.toml
llama_parse_content = Path("py/llama_parse/pyproject.toml").read_text()
llama_parse_doc = tomlkit.parse(llama_parse_content)
llama_parse_version = llama_parse_doc["project"]["version"]
# Find llama-cloud-services dependency in the dependencies list
dependency_version = None
for dep in llama_parse_doc["project"]["dependencies"]:
if isinstance(dep, str) and dep.startswith("llama-cloud-services"):
dependency_version = (
dep.split("==")[1]
if "==" in dep
else dep.split(">=")[1]
if ">=" in dep
else None
)
break
# Read TypeScript package.json version via helper
ts_version: str = get_ts_version()
return (
str(main_version),
str(llama_parse_version),
str(dependency_version),
str(ts_version) if ts_version is not None else None,
)
def validate_versions(
main_version: str,
llama_parse_version: str,
dependency_version: str,
) -> list[str]:
"""Validate that versions are consistent and return warnings."""
warnings = []
if main_version != llama_parse_version:
warnings.append(
f"Version mismatch: main={main_version}, llama_parse={llama_parse_version}"
)
# Extract version from dependency string (e.g., ">=0.6.51" -> "0.6.51")
if dependency_version and dependency_version.startswith(">="):
dep_ver = dependency_version[2:]
if dep_ver != main_version:
warnings.append(
f"Dependency version mismatch: dependency={dep_ver}, main={main_version}"
)
return warnings
def set_version(version: str) -> None:
"""Set version across Python projects (no TS change)."""
# Update main pyproject.toml
main_content = Path("py/pyproject.toml").read_text()
main_doc = tomlkit.parse(main_content)
main_doc["project"]["version"] = version
Path("py/pyproject.toml").write_text(tomlkit.dumps(main_doc))
# Update llama_parse/pyproject.toml
llama_parse_content = Path("py/llama_parse/pyproject.toml").read_text()
llama_parse_doc = tomlkit.parse(llama_parse_content)
llama_parse_doc["project"]["version"] = version
for dep_index, dep in enumerate(llama_parse_doc["project"]["dependencies"]):
if isinstance(dep, str) and dep.startswith("llama-cloud-services"):
llama_parse_doc["project"]["dependencies"][
dep_index
] = f"llama-cloud-services>={version}"
break
Path("py/llama_parse/pyproject.toml").write_text(tomlkit.dumps(llama_parse_doc))
click.echo(f"Updated Python versions to {version}")
def get_ts_version() -> str:
"""Read TypeScript package.json version (if present)."""
ts_package_path = Path("ts/llama_cloud_services/package.json")
package_data = json.loads(ts_package_path.read_text())
data = package_data.get("version")
if data is None:
raise RuntimeError("TypeScript package.json version not found")
return data
def set_ts_version(version: str) -> None:
"""Set TypeScript package.json version only."""
ts_package_path = Path("ts/llama_cloud_services/package.json")
package_data = json.loads(ts_package_path.read_text())
package_data["version"] = version
ts_package_path.write_text(json.dumps(package_data, indent=2) + "\n")
click.echo(f"Updated TypeScript package.json version to {version}")
def get_current_branch() -> str:
"""Get the current git branch."""
result = subprocess.run(
["git", "branch", "--show-current"], capture_output=True, text=True, check=True
)
return result.stdout.strip()
def create_if_not_exists(version: str) -> str:
"""Create a git tag and push it."""
current_branch = get_current_branch()
if current_branch != "main":
click.echo(
f"Error: Not on main branch (currently on {current_branch})", err=True
)
sys.exit(1)
tag_name = f"v{version}" if version[0].isdigit() else version
if not tag_exists(tag_name):
# Create tag
subprocess.run(["git", "tag", tag_name], check=True)
click.echo(f"Created tag {tag_name}")
else:
click.echo(f"Tag {tag_name} already exists")
return tag_name
def tag_exists(tag_name: str) -> bool:
"""Check if a git tag exists."""
result = subprocess.run(
["git", "tag", "-l", tag_name], capture_output=True, text=True, check=True
)
return tag_name in result.stdout.strip()
def push_tag(tag_name: str) -> None:
"""Push a git tag."""
subprocess.run(["git", "push", "origin", tag_name], check=True)
click.echo(f"Pushed tag {tag_name}")
@click.group()
def cli() -> None:
"""Version management for llama-cloud-services."""
pass
@cli.command()
def get() -> None:
"""Get current versions and show validation warnings."""
(
main_version,
llama_parse_version,
dependency_version,
ts_version,
) = get_current_versions()
click.echo("Current versions:")
click.echo(f" llama-cloud-services: {main_version}")
click.echo(f" llama-parse: {llama_parse_version}")
click.echo(f" dependency reference: {dependency_version}")
click.echo(f" typescript package: {ts_version}")
warnings = validate_versions(main_version, llama_parse_version, dependency_version)
if warnings:
click.echo("\nValidation warnings:")
for warning in warnings:
click.echo(f" ⚠️ {warning}")
else:
click.echo("\n✅ All versions are consistent")
@cli.command()
@click.argument("version")
@click.option("--js", is_flag=True, help="Update TypeScript package.json only")
def set(version: str, js: bool) -> None:
"""Set version for Python, TypeScript, or both (default: Python only)."""
if js:
set_ts_version(version)
return
else:
set_version(version)
@cli.command()
@click.option(
"--version", help="Version to tag (uses current version if not specified)"
)
@click.option(
"--push",
is_flag=True,
help="Push the tag to the remote repository",
)
@click.option(
"--js",
is_flag=True,
help="tag TypeScript package.json only",
)
def tag(version: str | None = None, push: bool = False, js: bool = False) -> None:
"""Create and push a git tag for the current version."""
if not version:
main_version, _, _, js_version = get_current_versions()
version = f"llama-cloud-services@{js_version}" if js else main_version
tag_name = create_if_not_exists(version)
if push:
push_tag(tag_name)
if __name__ == "__main__":
cli()
+3
View File
@@ -9,10 +9,12 @@ test("LlamaIndex module resolution test", async (t) => {
const index = new LlamaCloudIndex({
name: "test-index",
projectName: "Default",
apiKey: process.env.LLAMA_CLOUD_API_KEY || "test-key",
});
const reader = new LlamaParseReader({
resultType: "markdown",
verbose: false,
apiKey: process.env.LLAMA_CLOUD_API_KEY || "test-key",
});
ok(index !== undefined);
ok(reader !== undefined);
@@ -24,6 +26,7 @@ test("LlamaIndex module resolution test", async (t) => {
const index = new mod.LlamaCloudIndex({
name: "test-index",
projectName: "Default",
apiKey: process.env.LLAMA_CLOUD_API_KEY || "test-key",
});
ok(index !== undefined);
});
+18
View File
@@ -1,5 +1,23 @@
# llama-cloud-services
## 0.3.9
### Patch Changes
- 5d4cabd: Add ImageNode support in TypeScript
## 0.3.8
### Patch Changes
- 6e0f2f4: Agent data extraction citations can be undefined
## 0.3.7
### Patch Changes
- d028397: Update llama-cloud api version, and integrate with agent data deletion
## v0.1.0
First release for `llama-cloud-services`.
File diff suppressed because it is too large Load Diff
+4 -2
View File
@@ -1,9 +1,10 @@
{
"name": "llama-cloud-services",
"version": "0.3.5",
"version": "0.3.9",
"type": "module",
"license": "MIT",
"scripts": {
"get-openapi": "node ./scripts/get-openapi.js",
"generate": "./node_modules/.bin/openapi-ts",
"build": "pnpm run generate && bunchee",
"dev": "bunchee --watch",
@@ -13,7 +14,8 @@
"test": "vitest run --testTimeout=60000",
"test:watch": "vitest --watch",
"test:ui": "vitest --ui",
"test:coverage": "vitest --coverage"
"test:coverage": "vitest --coverage",
"release": "pnpm run build && pnpm publish"
},
"files": [
"openapi.json",
@@ -0,0 +1,21 @@
import fs from 'fs';
async function downloadOpenApiSpec() {
try {
const response = await fetch('https://api.cloud.llamaindex.ai/api/openapi.json');
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const data = await response.json();
fs.writeFileSync('openapi.json', JSON.stringify(data, null, 2));
console.log('Successfully downloaded openapi.json');
} catch (error) {
console.error('Error downloading OpenAPI spec:', error);
process.exit(1);
}
}
downloadOpenApiSpec();
@@ -9,10 +9,16 @@ import { DEFAULT_PROJECT_NAME } from "@llamaindex/core/global";
import type { QueryBundle } from "@llamaindex/core/query-engine";
import { BaseRetriever } from "@llamaindex/core/retriever";
import type { NodeWithScore } from "@llamaindex/core/schema";
import { jsonToNode, ObjectType } from "@llamaindex/core/schema";
import { jsonToNode, ObjectType, ImageNode } from "@llamaindex/core/schema";
import { extractText } from "@llamaindex/core/utils";
import type { ClientParams, CloudConstructorParams } from "./type.js";
import { getPipelineId, initService } from "./utils.js";
import { getPipelineId, getProjectId, initService } from "./utils.js";
import {
type PageScreenshotNodeWithScore,
type PageFigureNodeWithScore,
generateFilePageScreenshotPresignedUrlApiV1FilesIdPageScreenshotsPageIndexPresignedUrlPost,
generateFilePageFigurePresignedUrlApiV1FilesIdPageFiguresPageIndexFigureNamePresignedUrlPost,
} from "./api";
export type CloudRetrieveParams = Omit<
RetrievalParams,
@@ -43,6 +49,95 @@ export class LlamaCloudRetriever extends BaseRetriever {
});
}
private async fetchBase64FromPresignedUrl(url: string): Promise<string> {
const response = await fetch(url);
if (!response.ok) {
throw new Error(
`Failed to fetch media from presigned URL: ${response.status} ${response.statusText}`,
);
}
const buffer = Buffer.from(await response.arrayBuffer());
return buffer.toString("base64");
}
private async pageScreenshotNodesToNodeWithScore(
nodes: PageScreenshotNodeWithScore[] | undefined,
projectId: string,
): Promise<NodeWithScore[]> {
if (!nodes || nodes.length === 0) return [];
const results = await Promise.all(
nodes.map(async (n) => {
const { data: presigned } =
await generateFilePageScreenshotPresignedUrlApiV1FilesIdPageScreenshotsPageIndexPresignedUrlPost(
{
throwOnError: true,
path: {
id: n.node.file_id,
page_index: n.node.page_index,
},
query: {
project_id: projectId,
organization_id: this.organizationId ?? null,
},
},
);
const base64 = await this.fetchBase64FromPresignedUrl(presigned.url);
const imageNode = new ImageNode({
image: base64,
metadata: {
...(n.node.metadata ?? {}),
file_id: n.node.file_id,
page_index: n.node.page_index,
},
});
return { node: imageNode, score: n.score } satisfies NodeWithScore;
}),
);
return results;
}
private async pageFigureNodesToNodeWithScore(
nodes: PageFigureNodeWithScore[] | undefined,
projectId: string,
): Promise<NodeWithScore[]> {
if (!nodes || nodes.length === 0) return [];
const results = await Promise.all(
nodes.map(async (n) => {
const { data: presigned } =
await generateFilePageFigurePresignedUrlApiV1FilesIdPageFiguresPageIndexFigureNamePresignedUrlPost(
{
throwOnError: true,
path: {
id: n.node.file_id,
page_index: n.node.page_index,
figure_name: n.node.figure_name,
},
query: {
project_id: projectId,
organization_id: this.organizationId ?? null,
},
},
);
const base64 = await this.fetchBase64FromPresignedUrl(presigned.url);
const imageNode = new ImageNode({
image: base64,
metadata: {
...(n.node.metadata ?? {}),
file_id: n.node.file_id,
page_index: n.node.page_index,
figure_name: n.node.figure_name,
},
});
return { node: imageNode, score: n.score } satisfies NodeWithScore;
}),
);
return results;
}
// LlamaCloud expects null values for filters, but LlamaIndexTS uses undefined for empty values
// This function converts the undefined values to null
private convertFilter(filters?: MetadataFilters): MetadataFilters | null {
@@ -76,6 +171,35 @@ export class LlamaCloudRetriever extends BaseRetriever {
}
async _retrieve(query: QueryBundle): Promise<NodeWithScore[]> {
// Handle deprecated image retrieval flag
const retrieveImageNodes = (this.retrieveParams as RetrievalParams)
.retrieve_image_nodes;
if (typeof retrieveImageNodes !== "undefined") {
console.warn(
"The `retrieve_image_nodes` parameter is deprecated. Use `retrieve_page_screenshot_nodes` and `retrieve_page_figure_nodes` instead.",
);
}
const retrievePageScreenshotNodes = (this.retrieveParams as RetrievalParams)
.retrieve_page_screenshot_nodes;
const retrievePageFigureNodes = (this.retrieveParams as RetrievalParams)
.retrieve_page_figure_nodes;
if (retrieveImageNodes) {
if (
retrievePageScreenshotNodes === false ||
retrievePageFigureNodes === false
) {
throw new Error(
"If `retrieve_image_nodes` is set to true, both `retrieve_page_screenshot_nodes` and `retrieve_page_figure_nodes` must also be set to true or omitted.",
);
}
(this.retrieveParams as RetrievalParams).retrieve_page_screenshot_nodes =
true;
(this.retrieveParams as RetrievalParams).retrieve_page_figure_nodes =
true;
}
const pipelineId = await getPipelineId(
this.pipelineName,
this.projectName,
@@ -98,6 +222,34 @@ export class LlamaCloudRetriever extends BaseRetriever {
},
});
return this.resultNodesToNodeWithScore(results.retrieval_nodes);
const textNodes = this.resultNodesToNodeWithScore(results.retrieval_nodes);
const needScreenshots = (this.retrieveParams as RetrievalParams)
.retrieve_page_screenshot_nodes;
const needFigures = (this.retrieveParams as RetrievalParams)
.retrieve_page_figure_nodes;
if (!needScreenshots && !needFigures) {
return textNodes;
}
const projectId = await getProjectId(this.projectName, this.organizationId);
const [screenshotNodes, figureNodes] = await Promise.all([
needScreenshots
? this.pageScreenshotNodesToNodeWithScore(
results.image_nodes,
projectId,
)
: Promise.resolve([] as NodeWithScore[]),
needFigures
? this.pageFigureNodesToNodeWithScore(
results.page_figure_nodes,
projectId,
)
: Promise.resolve([] as NodeWithScore[]),
]);
return [...textNodes, ...screenshotNodes, ...figureNodes];
}
}
@@ -4,6 +4,7 @@ import {
aggregateAgentDataApiV1BetaAgentDataAggregatePost,
createAgentDataApiV1BetaAgentDataPost,
deleteAgentDataApiV1BetaAgentDataItemIdDelete,
deleteAgentDataByQueryApiV1BetaAgentDataDeletePost,
getAgentDataApiV1BetaAgentDataItemIdGet,
searchAgentDataApiV1BetaAgentDataSearchPost,
updateAgentDataApiV1BetaAgentDataItemIdPut,
@@ -12,6 +13,7 @@ import {
} from "../../client";
import type {
AggregateAgentDataOptions,
DeleteAgentDataOptions,
SearchAgentDataOptions,
TypedAgentData,
TypedAgentDataItems,
@@ -25,20 +27,23 @@ import type {
export class AgentClient<T = unknown> {
private client: ReturnType<typeof createClient>;
private collection: string;
private agentUrlId: string;
private deploymentName: string;
constructor({
client = defaultClient,
collection = "default",
agentUrlId = "_public",
deploymentName = "_public",
agentUrlId,
}: {
client?: ReturnType<typeof createClient>;
collection?: string;
deploymentName?: string;
// deprecated, use deploymentName instead
agentUrlId?: string;
}) {
this.client = client;
this.collection = collection;
this.agentUrlId = agentUrlId;
this.deploymentName = agentUrlId || deploymentName;
}
/**
@@ -48,7 +53,7 @@ export class AgentClient<T = unknown> {
const response = await createAgentDataApiV1BetaAgentDataPost({
throwOnError: true,
body: {
agent_slug: this.agentUrlId,
deployment_name: this.deploymentName,
collection: this.collection,
data: data as Record<string, unknown>,
},
@@ -109,6 +114,24 @@ export class AgentClient<T = unknown> {
});
}
/**
* Delete all matching agent data, returns the total number of deleted items
*/
async delete(options: DeleteAgentDataOptions): Promise<number> {
const response = await deleteAgentDataByQueryApiV1BetaAgentDataDeletePost({
throwOnError: true,
body: {
deployment_name: this.deploymentName,
...(this.collection !== undefined && {
collection: this.collection,
}),
...(options.filter !== undefined && { filter: options.filter }),
},
client: this.client,
});
return response.data.deleted_count;
}
/**
* Search agent data
*/
@@ -118,7 +141,7 @@ export class AgentClient<T = unknown> {
const response = await searchAgentDataApiV1BetaAgentDataSearchPost({
throwOnError: true,
body: {
agent_slug: this.agentUrlId,
deployment_name: this.deploymentName,
...(this.collection !== undefined && {
collection: this.collection,
}),
@@ -165,7 +188,7 @@ export class AgentClient<T = unknown> {
const response = await aggregateAgentDataApiV1BetaAgentDataAggregatePost({
throwOnError: true,
body: {
agent_slug: this.agentUrlId,
deployment_name: this.deploymentName,
...(this.collection !== undefined && {
collection: this.collection,
}),
@@ -209,7 +232,7 @@ export class AgentClient<T = unknown> {
private transformResponse(data: AgentData): TypedAgentData<T> {
const result: TypedAgentData<T> = {
id: data.id!,
agentUrlId: data.agent_slug,
deploymentName: data.deployment_name,
data: data.data as T,
createdAt: new Date(data.created_at!),
updatedAt: new Date(data.updated_at!),
@@ -250,10 +273,10 @@ export interface AgentDataClientOptions {
/** Base URL for the client */
/** Base URL of the llama cloud api */
baseUrl?: string;
/** If running in an agent runtime, optionally provide the window url to infer the agent url id */
/** If running in an agent runtime, optionally provide the window url to infer the deployment name */
windowUrl?: string;
/** Agent URL ID for the client, if not provided, it will be inferred from the window url, or fall back to "default" */
agentUrlId?: string;
/** Deployment name for the client, if not provided, it will be inferred from the window url, or fall back to "default" */
deploymentName?: string;
/** Collection name for the client, defaults to "default" */
collection?: string;
}
@@ -267,22 +290,25 @@ export function createAgentDataClient<T = unknown>({
client = defaultClient,
windowUrl,
env,
deploymentName,
agentUrlId,
collection = "default",
}: {
client?: ReturnType<typeof createClient>;
windowUrl?: string;
env?: Record<string, string>;
deploymentName?: string;
// deprecated, use deploymentName instead
agentUrlId?: string;
collection?: string;
} = {}): AgentClient<T> {
if (env && !agentUrlId) {
agentUrlId =
if (env && !deploymentName) {
deploymentName =
env.LLAMA_DEPLOY_DEPLOYMENT_NAME ||
env.NEXT_PUBLIC_LLAMA_DEPLOY_DEPLOYMENT_NAME ||
env.VITE_LLAMA_DEPLOY_DEPLOYMENT_NAME;
}
if (windowUrl && !agentUrlId) {
if (windowUrl && !deploymentName) {
try {
const url = new URL(windowUrl);
const path = url.pathname;
@@ -291,17 +317,18 @@ export function createAgentDataClient<T = unknown>({
url.hostname.includes("127.0.0.1");
if (path.startsWith("/deployments/") && !isLocalhost) {
// /deployments/<agent-url-id>/ui/ -> ["", "deployments", "<agent-url-id>", "ui"]
agentUrlId = path.split("/")[2];
deploymentName = path.split("/")[2];
}
} catch (error) {
console.warn(
"Failed to infer agent url id from window url, falling back to default",
"Failed to infer deployment name from window url, falling back to default",
error,
);
}
}
return new AgentClient({
...(deploymentName && { deploymentName }),
...(agentUrlId && { agentUrlId }),
collection,
client,
@@ -38,7 +38,7 @@ export interface ExtractedFieldMetadata {
confidence?: number;
/** The confidence score for the field based on the extracted text only */
extraction_confidence?: number;
citation: FieldCitation[];
citation?: FieldCitation[];
}
export interface FieldCitation {
@@ -87,8 +87,8 @@ export interface ExtractedData<T = unknown> {
export interface TypedAgentData<T = unknown> {
/** The unique ID of the agent data record. */
id: string;
/** The ID of the agent that created the data. */
agentUrlId: string;
/** The deployment name of the agent that created the data. */
deploymentName: string;
/** The collection of the agent data. */
collection?: string;
/** The data of the agent data. Usually an ExtractedData&lt;SomeOtherType&gt; */
@@ -127,6 +127,14 @@ export interface SearchAgentDataOptions {
includeTotal?: boolean;
}
/**
* Options for deleting agent data
*/
export interface DeleteAgentDataOptions {
/** Filter options for the deletion. */
filter?: Record<string, FilterOperation>;
}
/**
* Options for aggregating agent data
*/
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,246 @@
import { describe, it, expect, beforeEach, vi } from "vitest";
import { AgentClient, createAgentDataClient } from "../src/beta/agent/index.js";
import * as sdk from "../src/client/index.js";
describe("AgentClient", () => {
beforeEach(() => {
vi.restoreAllMocks();
});
it("createItem sends correct payload and returns typed data", async () => {
const spy = vi
.spyOn(sdk, "createAgentDataApiV1BetaAgentDataPost")
.mockResolvedValue({
data: {
id: "1",
deployment_name: "dep",
collection: "col",
data: { foo: "bar" },
created_at: "2024-01-01T00:00:00Z",
updated_at: "2024-01-01T00:00:00Z",
},
} as any);
const client = new AgentClient<{ foo: string }>({
deploymentName: "dep",
collection: "col",
});
const result = await client.createItem({ foo: "bar" });
expect(spy).toHaveBeenCalledOnce();
const call = spy.mock.calls[0][0];
expect(call.body.deployment_name).toBe("dep");
expect(call.body.collection).toBe("col");
expect(call.body.data).toEqual({ foo: "bar" });
expect(result.id).toBe("1");
expect(result.deploymentName).toBe("dep");
expect(result.collection).toBe("col");
expect(result.data).toEqual({ foo: "bar" });
expect(result.createdAt).toEqual(new Date("2024-01-01T00:00:00Z"));
expect(result.updatedAt).toEqual(new Date("2024-01-01T00:00:00Z"));
});
it("getItem returns null for 404 errors", async () => {
const spy = vi
.spyOn(sdk, "getAgentDataApiV1BetaAgentDataItemIdGet")
.mockImplementation(async () => {
const err: any = new Error("Not found");
err.response = { status: 404 };
throw err;
});
const client = new AgentClient({ deploymentName: "dep" });
const res = await client.getItem("missing-id");
expect(spy).toHaveBeenCalledOnce();
expect(res).toBeNull();
});
it("updateItem updates and returns typed data", async () => {
const spy = vi
.spyOn(sdk, "updateAgentDataApiV1BetaAgentDataItemIdPut")
.mockResolvedValue({
data: {
id: "123",
deployment_name: "dep",
collection: "col",
data: { foo: "baz" },
created_at: "2024-01-01T00:00:00Z",
updated_at: "2024-01-02T00:00:00Z",
},
} as any);
const client = new AgentClient<{ foo: string }>({
deploymentName: "dep",
collection: "col",
});
const res = await client.updateItem("123", { foo: "baz" });
expect(spy).toHaveBeenCalledOnce();
const call = spy.mock.calls[0][0];
expect(call.path.item_id).toBe("123");
expect(call.body.data).toEqual({ foo: "baz" });
expect(res.id).toBe("123");
expect(res.updatedAt).toEqual(new Date("2024-01-02T00:00:00Z"));
});
it("deleteItem calls delete endpoint with correct path", async () => {
const spy = vi
.spyOn(sdk, "deleteAgentDataApiV1BetaAgentDataItemIdDelete")
.mockResolvedValue({} as any);
const client = new AgentClient({ deploymentName: "dep" });
await client.deleteItem("abc");
expect(spy).toHaveBeenCalledOnce();
expect(spy.mock.calls[0][0].path.item_id).toBe("abc");
});
it("delete by query returns deleted count", async () => {
const spy = vi
.spyOn(sdk, "deleteAgentDataByQueryApiV1BetaAgentDataDeletePost")
.mockResolvedValue({ data: { deleted_count: 7 } } as any);
const client = new AgentClient({
deploymentName: "dep",
collection: "col",
});
const count = await client.delete({
filter: { status: { op: "eq", value: "accepted" } as any },
});
expect(spy).toHaveBeenCalledOnce();
const body = spy.mock.calls[0][0].body;
expect(body.deployment_name).toBe("dep");
expect(body.collection).toBe("col");
expect(count).toBe(7);
});
it("search maps items and optional fields correctly", async () => {
const now = "2024-01-01T00:00:00Z";
const spy = vi
.spyOn(sdk, "searchAgentDataApiV1BetaAgentDataSearchPost")
.mockResolvedValue({
data: {
items: [
{
id: "1",
deployment_name: "dep",
collection: "col",
data: { foo: "bar" },
created_at: now,
updated_at: now,
},
],
total_size: 1,
next_page_token: "next",
},
} as any);
const client = new AgentClient<{ foo: string }>({
deploymentName: "dep",
collection: "col",
});
const result = await client.search({
includeTotal: true,
orderBy: "created_at desc",
pageSize: 1,
offset: 0,
});
expect(spy).toHaveBeenCalledOnce();
const body = spy.mock.calls[0][0].body;
expect(body.deployment_name).toBe("dep");
expect(body.collection).toBe("col");
expect(body.include_total).toBe(true);
expect(body.order_by).toBe("created_at desc");
expect(body.page_size).toBe(1);
expect(body.offset).toBe(0);
expect(result.items).toHaveLength(1);
expect(result.totalSize).toBe(1);
expect(result.nextPageToken).toBe("next");
expect(result.items[0].createdAt).toEqual(new Date(now));
});
it("aggregate maps groups and optional fields correctly", async () => {
const spy = vi
.spyOn(sdk, "aggregateAgentDataApiV1BetaAgentDataAggregatePost")
.mockResolvedValue({
data: {
items: [
{
group_key: { status: "accepted" },
count: 3,
first_item: { foo: "bar" },
},
],
total_size: 1,
next_page_token: "tok",
},
} as any);
const client = new AgentClient<{ foo: string }>({
deploymentName: "dep",
collection: "col",
});
const result = await client.aggregate({
groupBy: ["status"],
count: true,
first: true,
pageSize: 1,
offset: 0,
});
expect(spy).toHaveBeenCalledOnce();
const body = spy.mock.calls[0][0].body;
expect(body.deployment_name).toBe("dep");
expect(body.collection).toBe("col");
expect(body.group_by).toEqual(["status"]);
expect(body.count).toBe(true);
expect(body.first).toBe(true);
expect(body.page_size).toBe(1);
expect(body.offset).toBe(0);
expect(result.items).toHaveLength(1);
expect(result.totalSize).toBe(1);
expect(result.nextPageToken).toBe("tok");
expect(result.items[0].groupKey).toEqual({ status: "accepted" });
expect(result.items[0].count).toBe(3);
expect(result.items[0].firstItem).toEqual({ foo: "bar" });
});
it("createAgentDataClient infers deployment name from env", async () => {
const spy = vi
.spyOn(sdk, "searchAgentDataApiV1BetaAgentDataSearchPost")
.mockResolvedValue({
data: { items: [], total_size: 0 },
} as any);
const client = createAgentDataClient({
env: { LLAMA_DEPLOY_DEPLOYMENT_NAME: "env-dep" },
});
await client.search({});
const body = spy.mock.calls[0][0].body;
expect(body.deployment_name).toBe("env-dep");
});
it("createAgentDataClient infers deployment name from windowUrl (non-local)", async () => {
const spy = vi
.spyOn(sdk, "deleteAgentDataByQueryApiV1BetaAgentDataDeletePost")
.mockResolvedValue({
data: { deleted_count: 0 },
} as any);
const client = createAgentDataClient({
windowUrl: "https://app.llamaindex.ai/deployments/abc/ui/",
});
await client.delete({});
const body = spy.mock.calls[0][0].body;
expect(body.deployment_name).toBe("abc");
});
});