[PR #17] unstructured[minor]: 08 - Refactoring 17 unstructured loaders #24

New Issue

2026-02-16T06:15:42-05:00

yindo commented

2026-02-16 06:15:42 -05:00

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/langchain-unstructured/pull/17
Author: @pprados
Created: 2/27/2025
Status: 🔄 Open

Base: main ← Head: pprados/08-unstructured

📝 Commits (1)

fbf9fc1 Refactor with PDF standard compatibility

📊 Changes

12 files changed (+5295 additions, -2060 deletions)

View changed files

📝 libs/unstructured/README.md (+8 -8)
➕ libs/unstructured/example_docs/hello.pdf (+0 -0)
➕ libs/unstructured/example_docs/layout-parser-paper-password.pdf (+0 -0)
📝 libs/unstructured/langchain_unstructured/__init__.py (+9 -2)
📝 libs/unstructured/langchain_unstructured/document_loaders.py (+657 -50)
📝 libs/unstructured/poetry.lock (+3508 -1949)
📝 libs/unstructured/pyproject.toml (+45 -23)
➕ libs/unstructured/tests/integration_tests/conftest.py (+19 -0)
📝 libs/unstructured/tests/integration_tests/test_document_loaders.py (+72 -28)
➕ libs/unstructured/tests/integration_tests/test_parser.py (+314 -0)
📝 libs/unstructured/tests/unit_tests/test_imports.py (+2 -0)
➕ libs/unstructured/unstructured_pdfloader.ipynb (+661 -0)

📄 Description

In this PR, we propose a migration of the various Unstructured*Loader implementations to the langchain-unstructured package.

Improvements

We’ve made several key improvements:

Each loader is tested with all supported file types (unlike langchain-community)
The result is compared to the same execution using the original version from langchain-community (see test_migration.py)
Each Loader is split into a Loader/Parser to allow usage with GenericLoader
The output format is Markdown-oriented rather than text stream. This includes support for headings (with # prefixes) and tables in either Markdown or HTML format. It’s possible to revert to the original behavior by changing a few parameters.
Headers and footers can be excluded (keep_header_footer=False)
Loaders accept:
- Path objects or strings
- web_url
- IO object
PDF processing complies with the specifications shared across other PDF parsers, and supports the four strategies (auto, fast, hi_res, and ocr_only)
Still compatible with lazy_load()
UnstructuredLoader additionally supports a list of PATHs in file_path. While we don’t consider this very clean (why only this loader? Why no plural? The user could just loop), we replicate the behavior from langchain-community.
Nearly 300 tests validate this PR
The langchain-unstructured dependencies offer the same extras as unstructured (csv, pdf, docx, etc.). This allows specifying a dependency on langchain-unstructured limited to certain file types (langchain-unstructured[pdf]). The previous behavior pulled in all possible formats, resulting in a package too large for environments like AWS Lambda.

With this PR, it will be possible to mark 17 Loader as "deprecated". There will remain 5 dependencies on unstructured in langchain-community.

Once this version is released, we plan to propose a PR to langchain-community to mark all Unstructured*Loader as @deprecated. Any changes to default parameter values will be explained in the comments.

Other dependencies on Unstructured in `langchain-community`

There are not part of unstructured

CHM is not a format directly supported by Unstructured, even though UnstructuredCHMLoader exists. The langchain-community version doesn’t work with the files we tested. We are leaving this loader as-is.
UnstructuredLakeFSLoader
SeleniumURLLoader
S3FileLoader. Use GenericLoader + CloudBlobLoader
UnstructuredHtmlEvaluator

pyproject.toml

unstructured is a framework that can pull in a large number of dependencies, depending on the file formats it needs to process. The framework offers various extras to include only the strictly necessary dependencies, for example: unstructured[pdf,csv].

langchain-unstructured does not currently work this way. It pulls in all dependencies from unstructured, resulting in very large projects that are incompatible with environments that have size limitations, such as AWS Lambda.

The change to pyproject.toml replicates the different extras provided by unstructured and propagates them into langchain-unstructured.

PDF

This is one part of a larger Pull Request (PR) that is too large to be submitted all at once. This specific part focuses on updating the UnstructuredPDFParser and UnstructuredPDFLoader.

For more details, see here

Note

I will not split this PR into multiple smaller PRs, each covering a single loader. That approach would take too much time for zero benefit (I’ve had some bad experiences with it). Either this PR works for you, and I’ll make the requested changes, or you can close it and ignore it. It will then be up to another contributor to migrate the various Unstructured*Loader to this project.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/langchain-unstructured/pull/17 **Author:** [@pprados](https://github.com/pprados) **Created:** 2/27/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `pprados/08-unstructured` --- ### 📝 Commits (1) - [`fbf9fc1`](https://github.com/langchain-ai/langchain-unstructured/commit/fbf9fc14e3e93b6730158414b4dea414d69a38da) Refactor with PDF standard compatibility ### 📊 Changes **12 files changed** (+5295 additions, -2060 deletions) <details> <summary>View changed files</summary> 📝 `libs/unstructured/README.md` (+8 -8) ➕ `libs/unstructured/example_docs/hello.pdf` (+0 -0) ➕ `libs/unstructured/example_docs/layout-parser-paper-password.pdf` (+0 -0) 📝 `libs/unstructured/langchain_unstructured/__init__.py` (+9 -2) 📝 `libs/unstructured/langchain_unstructured/document_loaders.py` (+657 -50) 📝 `libs/unstructured/poetry.lock` (+3508 -1949) 📝 `libs/unstructured/pyproject.toml` (+45 -23) ➕ `libs/unstructured/tests/integration_tests/conftest.py` (+19 -0) 📝 `libs/unstructured/tests/integration_tests/test_document_loaders.py` (+72 -28) ➕ `libs/unstructured/tests/integration_tests/test_parser.py` (+314 -0) 📝 `libs/unstructured/tests/unit_tests/test_imports.py` (+2 -0) ➕ `libs/unstructured/unstructured_pdfloader.ipynb` (+661 -0) </details> ### 📄 Description In this PR, we propose a migration of the various `Unstructured*Loader` implementations to the `langchain-unstructured` package. # Improvements We’ve made several key improvements: - Each loader is tested with all supported file types (unlike `langchain-community`) - The result is compared to the same execution using the original version from `langchain-community` (see `test_migration.py`) - Each `Loader` is split into a `Loader`/`Parser` to allow usage with `GenericLoader` - The output format is **Markdown-oriented** rather than **text stream**. This includes support for headings (with `# ` prefixes) and tables in either Markdown or HTML format. It’s possible to revert to the original behavior by changing a few parameters. - Headers and footers can be excluded (`keep_header_footer=False`) - Loaders accept: - `Path` objects or strings - `web_url` - `IO` object - PDF processing complies with the specifications shared across other PDF parsers, and supports the four strategies (`auto`, `fast`, `hi_res`, and `ocr_only`) - Still compatible with `lazy_load()` - `UnstructuredLoader` additionally supports a list of PATHs in `file_path`. While we don’t consider this very clean (why only this loader? Why no plural? The user could just loop), we replicate the behavior from `langchain-community`. - Nearly 300 tests validate this PR - The `langchain-unstructured` dependencies offer the same extras as `unstructured` (csv, pdf, docx, etc.). This allows specifying a dependency on `langchain-unstructured` limited to certain file types (`langchain-unstructured[pdf]`). The previous behavior pulled in all possible formats, resulting in a package too large for environments like AWS Lambda. With this PR, it will be possible to mark 17 Loader as "deprecated". There will remain 5 dependencies on `unstructured` in `langchain-community`. Once this version is released, we plan to propose a PR to `langchain-community` to mark all `Unstructured*Loader` as `@deprecated`. Any changes to default parameter values will be explained in the comments. # Other dependencies on Unstructured in `langchain-community` There are not part of unstructured - CHM is not a format directly supported by `Unstructured`, even though `UnstructuredCHMLoader` exists. The `langchain-community` version doesn’t work with the files we tested. We are leaving this loader as-is. - `UnstructuredLakeFSLoader` - `SeleniumURLLoader` - `S3FileLoader`. Use `GenericLoader` + `CloudBlobLoader` - `UnstructuredHtmlEvaluator` # pyproject.toml `unstructured` is a framework that can pull in a large number of dependencies, depending on the file formats it needs to process. The framework offers various *extras* to include only the strictly necessary dependencies, for example: `unstructured[pdf,csv]`. `langchain-unstructured` does not currently work this way. It pulls in **all** dependencies from `unstructured`, resulting in very large projects that are incompatible with environments that have size limitations, such as AWS Lambda. The change to `pyproject.toml` replicates the different extras provided by `unstructured` and propagates them into `langchain-unstructured`. # PDF This is one part of a larger Pull Request (PR) that is too large to be submitted all at once. This specific part focuses on updating the `UnstructuredPDFParser` and `UnstructuredPDFLoader`. For more details, see [here](https://github.com/langchain-ai/langchain/pull/28970) # Note I will not split this PR into multiple smaller PRs, each covering a single loader. That approach would take too much time for zero benefit (I’ve had some bad experiences with it). Either this PR works for you, and I’ll make the requested changes, or you can close it and ignore it. It will then be up to another contributor to migrate the various `Unstructured*Loader` to this project. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-02-16 06:15:42 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/langchain-unstructured#24

[PR #17] unstructured[minor]: 08 - Refactoring 17 unstructured loaders #24