neuml/txtai
1
0
Fork 0
mirror of https://github.com/neuml/txtai.git synced 2026-10-12 02:19:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add text extraction with Docling, closes #814. Refactor Textractor pipeline into multiple pipelines, closes #828

Browse Source
This commit is contained in:
davidmezzetti
2024-12-03 14:27:49 -05:00
parent 5fdbc90d24
commit 9efc9ba070
13 changed files with 846 additions and 480 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pipeline/data/filetohtml.md
+74
View File
@@ -0,0 +1,74 @@
# File To HTML
![pipeline](../../images/pipeline.png#only-light)
![pipeline](../../images/pipeline-dark.png#only-dark)
The File To HTML pipeline transforms files to HTML. It supports the following text extraction backends.
## Apache Tika
[Apache Tika](https://tika.apache.org/) detects and extracts metadata and text from over a thousand different file types. See [this link](https://tika.apache.org/2.9.2/formats.html) for a list of supported document formats.
Apache Tika requires [Java](https://en.wikipedia.org/wiki/Java_(programming_language)) to be installed. An alternative to that is starting a separate Apache Tika service via [this Docker Image](https://hub.docker.com/r/apache/tika) and setting these [environment variables](https://github.com/chrismattmann/tika-python?tab=readme-ov-file#environment-variables).
## Docling
[Docling](https://github.com/DS4SD/docling) parses documents and exports them to the desired format with ease and speed. This is a library that has rapidly gained popularity starting in late 2024. Docling excels in parsing formatting elements from PDFs (tables, sections etc).
See [this link](https://github.com/DS4SD/docling?tab=readme-ov-file#features) for a list of supported document formats.
## Example
The following shows a simple example using this pipeline.
```python
from txtai.pipeline import FileToHTML
# Create and run pipeline
html = FileToHTML()
html("/path/to/file")
```
## Configuration-driven example
Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
### config.yml
```yaml
# Create pipeline using lower case class name
filetohtml:
# Run pipeline with workflow
workflow:
html:
tasks:
- action: filetohtml
```
### Run with Workflows
```python
from txtai import Application
# Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("html", ["/path/to/file"]))
```
### Run with API
```bash
CONFIG=config.yml uvicorn "txtai.api:app" &
curl \
-X POST "http://localhost:8000/workflow" \
-H "Content-Type: application/json" \
-d '{"name":"html", "elements":["/path/to/file"]}'
```
## Methods
Python documentation for the pipeline.
### ::: txtai.pipeline.FileToHTML.__init__
### ::: txtai.pipeline.FileToHTML.__call__
docs/pipeline/data/htmltomd.md
+68
View File
@@ -0,0 +1,68 @@
# HTML To Markdown
![pipeline](../../images/pipeline.png#only-light)
![pipeline](../../images/pipeline-dark.png#only-dark)
The HTML To Markdown pipeline transforms HTML to Markdown.
Markdown formatting is applied for headings, blockquotes, lists, code, tables and text. Visual formatting is also included (bold, italic etc).
This pipeline searches for the best node that has relevant text, often found with an `article`, `main` or `body` tag.
The HTML to Markdown pipeline requires the [BeautifulSoup4](https://pypi.org/project/beautifulsoup4/) library to be installed.
## Example
The following shows a simple example using this pipeline.
```python
from txtai.pipeline import HTMLToMarkdown
# Create and run pipeline
md = HTMLToMarkdown()
md("<html><body>This is a test</body></html>")
```
## Configuration-driven example
Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
### config.yml
```yaml
# Create pipeline using lower case class name
htmltomarkdown:
# Run pipeline with workflow
workflow:
markdown:
tasks:
- action: htmltomarkdown
```
### Run with Workflows
```python
from txtai import Application
# Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("markdown", ["<html><body>This is a test</body></html>"]))
```
### Run with API
```bash
CONFIG=config.yml uvicorn "txtai.api:app" &
curl \
-X POST "http://localhost:8000/workflow" \
-H "Content-Type: application/json" \
-d '{"name":"markdown", "elements":["<html><body>This is a test</body></html>"]}'
```
## Methods
Python documentation for the pipeline.
### ::: txtai.pipeline.HTMLToMarkdown.__init__
### ::: txtai.pipeline.HTMLToMarkdown.__call__
docs/pipeline/data/textractor.md
+6 -4
View File
@@ -3,15 +3,17 @@
![pipeline](../../images/pipeline.png#only-light)
![pipeline](../../images/pipeline-dark.png#only-dark)
The Textractor pipeline extracts and splits text from documents. This pipeline uses [Apache Tika](https://github.com/chrismattmann/tika-python) (if Java is available) and [BeautifulSoup4](https://pypi.org/project/beautifulsoup4/). See [this link](https://tika.apache.org/2.9.2/formats.html) for a list of supported document formats.
The Textractor pipeline extracts and splits text from documents.
Each document goes through the following process.
- Content is retrieved if it's not local
- If the document `mime-type` isn't plain text or HTML, it's run through Tika and converted to XHTML
- XHTML is converted to Markdown and returned
- If the document `mime-type` isn't plain text or HTML, it's converted to HTML via the [FiletoHTML](../filetohtml) pipeline
- HTML is converted to Markdown via the [HTMLToMarkdown](../htmltomd) pipeline and returned
Without Apache Tika, this pipeline only supports plain text and HTML. Other document types require Tika and Java to be installed. Another option is to start Apache Tika via [this Docker Image](https://hub.docker.com/r/apache/tika).
The [backend](../filetohtml/#txtai.pipeline.FileToHTML.__init__) parameter sets the FileToHTML pipeline backend. If a backend isn't available, this pipeline assumes input is HTML content and only converts it to Markdown.
See the [FiletoHTML](../filetohtml) and [HTMLToMarkdown](../htmltomd) pipelines to learn more on the dependencies necessary for each of those pipelines.
## Example
docs/pipeline/index.md
+2
View File
@@ -21,6 +21,8 @@ The `LLM` and `RAG` pipelines also have integrations for [llama.cpp](https://git
- [TextToSpeech](audio/texttospeech)
- [Transcription](audio/transcription)
- Data Processing
- [FileToHTML](data/filetohtml)
- [HTMLToMarkdown](data/htmltomd)
- [Segmentation](data/segmentation)
- [Tabular](data/tabular)
- [Text extraction](data/textractor)
mkdocs.yml
+2
View File
@@ -85,6 +85,8 @@ nav:
- Text To Speech: pipeline/audio/texttospeech.md
- Transcription: pipeline/audio/transcription.md
- Data:
- File To HTML: pipeline/data/filetohtml.md
- HTML To Markdown: pipeline/data/htmltomd.md
- Segmentation: pipeline/data/segmentation.md
- Tabular: pipeline/data/tabular.md
- Textractor: pipeline/data/textractor.md
setup.py
+1 -1
View File
@@ -63,7 +63,7 @@ extras["pipeline-audio"] = [
"webrtcvad-wheels>=2.0.14",
]
extras["pipeline-data"] = ["beautifulsoup4>=4.9.3", "nltk>=3.5", "pandas>=1.1.0", "tika>=1.24"]
extras["pipeline-data"] = ["beautifulsoup4>=4.9.3", "docling>=2.8.2", "nltk>=3.5", "pandas>=1.1.0", "tika>=1.24"]
extras["pipeline-image"] = ["imagehash>=4.2.1", "pillow>=7.1.2", "timm>=0.4.12"]
src/python/txtai/pipeline/data/__init__.py
+2
View File
@@ -2,6 +2,8 @@
Segment imports
"""
from .filetohtml import FileToHTML
from .htmltomd import HTMLToMarkdown
from .segmentation import Segmentation
from .tabular import Tabular
from .textractor import Textractor
src/python/txtai/pipeline/data/filetohtml.py
+186
View File
@@ -0,0 +1,186 @@
"""
FileToHTML module
"""
import os
import re
from subprocess import Popen
# Conditional import
try:
from tika import detector, parser
TIKA = True
except ImportError:
TIKA = False
# Conditional import
try:
from docling.document_converter import DocumentConverter
DOCLING = True
except ImportError:
DOCLING = False
from ..base import Pipeline
class FileToHTML(Pipeline):
"""
File to HTML pipeline.
"""
def __init__(self, backend="available"):
"""
Creates a new File to HTML pipeline.
Args:
backend: backend to use to extract content, supports "tika", "docling" or "available" (default) which finds the first available
"""
# Lowercase backend parameter
backend = backend.lower() if backend else None
# Check for available backend
if backend == "available":
backend = "tika" if Tika.available() else "docling" if Docling.available() else None
# Create backend instance
self.backend = Tika() if backend == "tika" else Docling() if backend == "docling" else None
def __call__(self, path):
"""
Converts file at path to HTML. Returns None if no backend is available.
Args:
path: input file path
Returns:
html if a backend is available, otherwise returns None
"""
return self.backend(path) if self.backend else None
class Tika:
"""
File to HTML conversion via Apache Tika.
"""
@staticmethod
def available():
"""
Checks if a Java executable is available and Tika is installed.
Returns:
True if Java is available and Tika is installed, False otherwise
"""
# Get path to Java executable
path = os.environ.get("TIKA_JAVA", "java")
# pylint: disable=R1732,W0702,W1514
# Check if Java binary is available on path
try:
_ = Popen(path, stdout=open(os.devnull, "w"), stderr=open(os.devnull, "w"))
except:
return False
# Return True if Java is available AND Tika is installed
return TIKA
def __init__(self):
"""
Creates a new Tika instance.
"""
if not Tika.available():
raise ImportError('Tika engine is not available - install "pipeline" extra to enable. Also check that Java is available.')
def __call__(self, path):
"""
Parses content to HTML.
Args:
path: file path
Returns:
html
"""
# Skip parsing if input is plain text or HTML
mimetype = detector.from_file(path)
if mimetype in ("text/plain", "text/html", "text/xhtml"):
return None
# Parse content to HTML
parsed = parser.from_file(path, xmlContent=True)
return parsed["content"]
class Docling:
"""
File to HTML conversion via Docling.
"""
@staticmethod
def available():
"""
Checks if Docling is available.
Returns:
True if Docling is available, False otherwise
"""
return DOCLING
def __init__(self):
"""
Creates a new Docling instance.
"""
if not Docling.available():
raise ImportError('Docling engine is not available - install "pipeline" extra to enable')
self.converter = DocumentConverter()
def __call__(self, path):
"""
Parses content to HTML.
Args:
path: file path
Returns:
html
"""
# Skip parsing if input is HTML
if self.ishtml(path):
return None
# Parse content to HTML
html = self.converter.convert(path).document.export_to_html()
# Standardize HTML
return html.replace("</head>", "</head><body>").replace("</html>", "</body></html>").replace("</p>", "</p><p/>")
def ishtml(self, path):
"""
Detects if this file looks like HTML.
Args:
path: file path
Returns:
True if this is HTML
"""
with open(path, "rb") as f:
# Read first 1024 bytes, ignore encoding errors and strip leading/trailing whitespace
content = f.read(1024)
content = content.decode("ascii", errors="ignore").lower().strip()
# Check for HTML
return re.search(r"<!doctype\s+html|<html|<head|<body", content)
src/python/txtai/pipeline/data/htmltomd.py
+411
View File
@@ -0,0 +1,411 @@
"""
HTMLToMarkdown module
"""
import re
# Conditional import
try:
from bs4 import BeautifulSoup, NavigableString
SOUP = True
except ImportError:
SOUP = False
from ..base import Pipeline
class HTMLToMarkdown(Pipeline):
"""
HTML to Markdown pipeline.
Markdown formatting is applied for headings, blockquotes, lists, code, tables and text. Visual formatting is also
included (bold, italic etc).
This pipeline searches for the best node that has relevant text, often found with an article, main or body tag.
"""
def __init__(self, paragraphs=False, sections=False):
"""
Create a new Extract instance.
Args:
paragraphs: True if paragraph parsing enabled, False otherwise
sections: True if section parsing enabled, False otherwise
"""
if not SOUP:
raise ImportError('HTMLToMarkdown pipeline is not available - install "pipeline" extra to enable')
self.paragraphs = paragraphs
self.sections = sections
def __call__(self, html):
"""
Transforms input HTML into Markdown formatted text.
Args:
html: input html
Returns:
markdown formatted text
"""
# HTML Parser
soup = BeautifulSoup(html, features="html.parser")
# Ignore script and style tags
for script in soup.find_all(["script", "style"]):
script.decompose()
# Check for article sections
article = next((x for x in ["article", "main"] if soup.find(x)), None)
# Extract text from each section element
nodes = []
for node in soup.find_all(article if article else "body"):
# Skip article sections without at least 1 paragraph
if not article or node.find("p"):
nodes.append(self.process(node, article))
# Return extracted text, fallback to default text extraction if no nodes found
return "\n".join(self.metadata(soup) + nodes) if nodes else self.default(soup)
def process(self, node, article):
"""
Extracts text from a node. This method applies transforms for headings, blockquotes, lists, code, tables and text.
Page breaks are detected and reflected in the output text as a page break character.
Args:
node: input node
article: True if the main section node is an article
Returns:
node text
"""
if self.isheader(node):
return self.header(node, article)
if node.name in ("blockquote", "q"):
return self.block(node)
if node.name in ("ul", "ol"):
return self.items(node, article)
if node.name in ("code", "pre"):
return self.code(node)
if node.name == "table":
return self.table(node, article)
# Nodes to skip
if node.name in ("aside",) + (() if article else ("header", "footer")):
return ""
# Get page break symbol, if available
page = node.name and node.get("class") and "page" in node.get("class")
# Get node children
children = self.children(node)
# Join elements into text
if self.iscontainer(node, children):
texts = [self.process(node, article) for node in children]
text = "\n".join(text for text in texts if text or not article)
else:
text = self.text(node, article)
# Add page breaks, if section parsing enabled. Otherwise add node text.
return f"{text}\f" if page and self.sections else text
def metadata(self, node):
"""
Builds a metadata section. The metadata section consists of the title and
description fields.
Args:
node: input document node
Returns:
metadata as a list
"""
title = node.find("title")
metadata = [f"**{title.text.strip()}**"] if title and title.text else []
description = node.find("meta", attrs={"name": "description"})
if description and description["content"]:
metadata.append(f"\n*{description['content'].strip()}*")
# Add separator
if metadata:
metadata.append("\f" if self.sections else "\n\n")
return metadata
def default(self, soup):
"""
Default text handler when valid HTML isn't detected.
Args:
soup: BeautifulSoup object
Returns:
text
"""
lines = []
for line in soup.get_text().split("\n"):
# Detect markdown headings and add page breaks
lines.append(f"\f{line}" if self.sections and re.search(r"^#+ ", line) else line)
return "\n".join(lines)
def text(self, node, article):
"""
Text handler. This method flattens a node and it's children to text.
Args:
node: input node
article: True if the main section node is an article
Returns:
node text
"""
# Get node children if available, otherwise use node as item
items = self.children(node)
items = items if items else [node]
# Apply emphasis and link formatting
texts = []
for x in items:
target, text = x if x.name else node, x.text
if text.strip():
if target.name in ("b", "strong"):
text = f"**{text.strip()}** "
elif target.name in ("i", "em"):
text = f"*{text.strip()}* "
elif target.name == "a":
text = f"[{text.strip()}]({target.get('href')}) "
texts.append(text)
# Join text elements
text = "".join(texts)
# Article text processing
text = self.articletext(node, text) if article else text
# Return text, strip leading/trailing whitespace if this is a string only node
text = text if node.name and text else text.strip()
return text
def header(self, node, article):
"""
Header handler. This method transforms a HTML heading into a Markdown formatted heading.
Args:
node: input node
article: True if the main section node is an article
Returns:
heading as markdown
"""
# Get heading level and text
level = "#" * int(node.name[1])
text = self.text(node, article)
# Add section break or newline, if necessary
level = f"\f{level}" if self.sections else f"\n{level}"
# Return formatted header. Remove leading whitespace as it was added before level in step above.
return f"{level} {text.lstrip()}" if text.strip() else ""
def block(self, node):
"""
Blockquote handler. This method transforms a HTML blockquote or q block into a Markdown formatted
blockquote
Args:
node: input node
Returns:
block as markdown
"""
text = "\n".join(f"> {x}" for x in node.text.strip().split("\n"))
return f"{text}\n\n" if self.paragraphs else f"{text}\n"
def items(self, node, article):
"""
List handler. This method transforms a HTML ordered/unordered list into a Markdown formatted list.
Args:
node: input node
article: True if the main section node is an article
Returns:
list as markdown
"""
elements = []
for x, element in enumerate(node.find_all("li")):
# Unordered lists use dashes. Ordered lists use numbers.
prefix = "-" if node.name == "ul" else f"{x + 1}."
# List item text
text = self.process(element, article)
# Add list element
if text:
elements.append(f"{prefix} {text}")
# Join elements together as string
return "\n".join(elements)
def code(self, node):
"""
Code block handler. This method transforms a HTML pre or code block into a Markdown formatted
code block.
Args:
node: input node
Returns:
code as markdown
"""
text = f"```\n{node.text.strip()}\n```"
return f"{text}\n\n" if self.paragraphs else f"{text}\n"
def table(self, node, article):
"""
Table handler. This method transforms a HTML table into a Markdown formatted table.
Args:
node: input node
article: True if the main section node is an article
Returns:
table as markdown
"""
elements, header = [], False
# Process all rows
rows = node.find_all("tr")
for row in rows:
# Get list of columns for row
columns = row.find_all(lambda tag: tag.name in ("th", "td"))
# Add columns with separator
elements.append(f"|{'|'.join(self.process(column, article) for column in columns)}|")
# If there are multiple rows, add header format row
if not header and len(rows) > 1:
elements.append(f"{'|---' * len(columns)}|")
header = True
# Join elements together as string
return "\n".join(elements)
def iscontainer(self, node, children):
"""
Analyzes a node and it's children to determine if this is a container element. A container
element is defined as being a div, body, article or not having any string elements as children.
Args:
node: input node
nodes: input node's children
Returns:
True if this is a container element, False otherwise
"""
return children and (node.name in ("div", "body", "article") or not any(isinstance(x, NavigableString) for x in children))
def children(self, node):
"""
Gets the node children, if available.
Args:
node: input node
Returns:
node children or None if not available
"""
if node.name and node.contents:
# Iterate over children and remove whitespace-only string nodes
return [node for node in node.contents if node.name or node.text.strip()]
return None
def articletext(self, node, text):
"""
Transforms node text using article parsing rules. Article parsing is designed to extract text content from web articles.
It ignores navigation headers and other superfluous elements.
Args:
node: input node
text: current text
Returns:
article text
"""
# List of valid text nodes
valid = ["p", "th", "td", "li", "a", "b", "strong", "i", "em"]
# Check if text is valid article text
text = text if (node.name in valid or self.isheader(node)) and not self.islink(node) else ""
if text:
# Replace non-breaking space plus newline with double newline
text = text.replace("\xa0\n", "\n\n")
# Format paragraph whitespace
if node.name == "p":
text = f"{text.strip()}\n\n" if self.paragraphs else f"{text.strip()}\n"
return text
def isheader(self, node):
"""
Checks if node is a header node.
Args:
node: input node
Returns:
True if node is a header node, False otherwise
"""
return node.name in ["h1", "h2", "h3", "h4", "h5", "h6"]
def islink(self, node):
"""
Checks if node is a link node. This method does not consider links without tables as link nodes.
Args:
node: input node
Returns:
True if node is a link node, False otherwise
"""
# Check if this is a link node or link container
link, parent = False, node
while parent:
if parent.name == "a":
link = True
break
parent = parent.parent
# Return if this node or any parents are a link. Ignore links in table cells.
return link and node.parent.name not in ("th", "td")
src/python/txtai/pipeline/data/textractor.py
+28 -458
View File
@@ -4,22 +4,13 @@ Textractor module
import contextlib
import os
import re
import tempfile
from subprocess import Popen
from urllib.parse import urlparse
from urllib.request import urlopen, Request
# Conditional import
try:
from bs4 import BeautifulSoup, NavigableString
from tika import detector, parser
TIKA = True
except ImportError:
TIKA = False
from .filetohtml import FileToHTML
from .htmltomd import HTMLToMarkdown
from .segmentation import Segmentation
@@ -29,19 +20,28 @@ class Textractor(Segmentation):
"""
def __init__(
self, sentences=False, lines=False, paragraphs=False, minlength=None, join=False, tika=True, sections=False, cleantext=True, headers=None
self,
sentences=False,
lines=False,
paragraphs=False,
minlength=None,
join=False,
sections=False,
cleantext=True,
headers=None,
backend="available",
**kwargs
):
if not TIKA:
raise ImportError('Textractor pipeline is not available - install "pipeline" extra to enable')
super().__init__(sentences, lines, paragraphs, minlength, join, sections, cleantext)
# Determine if Apache Tika (default if Java is available) or Beautiful Soup should be used
# Beautiful Soup only supports HTML, Tika supports a wide variety of file formats.
self.tika = self.checkjava() if tika else False
# Get backend parameter - handle legacy tika flag
backend = "tika" if "tika" in kwargs and kwargs["tika"] else None if "tika" in kwargs else backend
# HTML to Text extractor
self.extract = Extract(self.paragraphs, self.sections)
# File to HTML pipeline
self.html = FileToHTML(backend) if backend else None
# HTML to Markdown pipeline
self.markdown = HTMLToMarkdown(self.paragraphs, self.sections)
# HTTP headers
self.headers = headers if headers else {}
@@ -54,14 +54,17 @@ class Textractor(Segmentation):
# Not a valid file path, treat input as data
html = text
elif self.tika:
# Use Tika if available
elif self.html:
# Use FileToHTML pipeline, if available
# Retrieve remote file, if necessary
path = path if exists else self.download(path)
# Parse content to XHTML
# Parse content to HTML
html = self.html(path)
# FiletoHTML pipeline returns None when input is already HTML
html = html if html else self.retrieve(path)
# Delete temporary file
if not exists:
os.remove(path)
@@ -70,32 +73,8 @@ class Textractor(Segmentation):
# Read data from url/path
html = self.retrieve(path)
# Extract text from HTML
return self.extract(html)
def checkjava(self, path=None):
"""
Checks if a Java executable is available for Tika.
Args:
path: path to java executable
Returns:
True if Java is available, False otherwise
"""
# Get path to java executable if path not set
if not path:
path = os.environ.get("TIKA_JAVA", "java")
# pylint: disable=R1732,W0702,W1514
# Check if java binary is available on path
try:
_ = Popen(path, stdout=open(os.devnull, "w"), stderr=open(os.devnull, "w"))
except:
return False
return True
# HTML to Markdown
return self.markdown(html)
def valid(self, path):
"""
@@ -156,412 +135,3 @@ class Textractor(Segmentation):
# Remote file
with contextlib.closing(urlopen(Request(url, headers=self.headers))) as connection:
return connection.read()
def html(self, path):
"""
Parses content to HTML using Apache Tika.
Args:
path: file path
Returns:
html
"""
# Skip parsing if input is plain text or HTML
mimetype = detector.from_file(path)
if mimetype in ("text/plain", "text/html", "text/xhtml"):
return self.retrieve(path)
# Parse content to XHTML
parsed = parser.from_file(path, xmlContent=True)
return parsed["content"]
class Extract:
"""
HTML to Text extractor. Markdown formatting is applied for headings, blockquotes, lists, code, tables and text.
Visual formatting is also included (bold, italic etc).
"""
def __init__(self, paragraphs, sections):
"""
Create a new Extract instance.
Args:
paragraphs: True if paragraph parsing enabled, False otherwise
sections: True if section parsing enabled, False otherwise
"""
self.paragraphs = paragraphs
self.sections = sections
def __call__(self, html):
"""
Transforms input HTML into Markdown formatted text.
Args:
html: input html
Returns:
markdown formatted text
"""
# HTML Parser
soup = BeautifulSoup(html, features="html.parser")
# Ignore script and style tags
for script in soup.find_all(["script", "style"]):
script.decompose()
# Check for article sections
article = next((x for x in ["article", "main"] if soup.find(x)), None)
# Extract text from each section element
nodes = []
for node in soup.find_all(article if article else "body"):
# Skip article sections without at least 1 paragraph
if not article or node.find("p"):
nodes.append(self.process(node, article))
# Return extracted text, fallback to default text extraction if no nodes found
return "\n".join(self.metadata(soup) + nodes) if nodes else self.default(soup)
def process(self, node, article):
"""
Extracts text from a node. This method applies transforms for headings, blockquotes, lists, code, tables and text.
Page breaks are detected and reflected in the output text as a page break character.
Args:
node: input node
article: True if the main section node is an article
Returns:
node text
"""
if self.isheader(node):
return self.header(node, article)
if node.name in ("blockquote", "q"):
return self.block(node)
if node.name in ("ul", "ol"):
return self.items(node, article)
if node.name in ("code", "pre"):
return self.code(node)
if node.name == "table":
return self.table(node, article)
# Nodes to skip
if node.name in ("aside",) + (() if article else ("header", "footer")):
return ""
# Get page break symbol, if available
page = node.name and node.get("class") and "page" in node.get("class")
# Get node children
children = self.children(node)
# Join elements into text
if self.iscontainer(node, children):
texts = [self.process(node, article) for node in children]
text = "\n".join(text for text in texts if text or not article)
else:
text = self.text(node, article)
# Add page breaks, if section parsing enabled. Otherwise add node text.
return f"{text}\f" if page and self.sections else text
def metadata(self, node):
"""
Builds a metadata section. The metadata section consists of the title and
description fields.
Args:
node: input document node
Returns:
metadata as a list
"""
title = node.find("title")
metadata = [f"**{title.text.strip()}**"] if title and title.text else []
description = node.find("meta", attrs={"name": "description"})
if description and description["content"]:
metadata.append(f"\n*{description['content'].strip()}*")
# Add separator
if metadata:
metadata.append("\f" if self.sections else "\n\n")
return metadata
def default(self, soup):
"""
Default text handler when valid HTML isn't detected.
Args:
soup: BeautifulSoup object
Returns:
text
"""
lines = []
for line in soup.get_text().split("\n"):
# Detect markdown headings and add page breaks
lines.append(f"\f{line}" if self.sections and re.search(r"^#+ ", line) else line)
return "\n".join(lines)
def text(self, node, article):
"""
Text handler. This method flattens a node and it's children to text.
Args:
node: input node
article: True if the main section node is an article
Returns:
node text
"""
# Get node children if available, otherwise use node as item
items = self.children(node)
items = items if items else [node]
# Apply emphasis and link formatting
texts = []
for x in items:
target, text = x if x.name else node, x.text
if text.strip():
if target.name in ("b", "strong"):
text = f"**{text.strip()}** "
elif target.name in ("i", "em"):
text = f"*{text.strip()}* "
elif target.name == "a":
text = f"[{text.strip()}]({target.get('href')}) "
texts.append(text)
# Join text elements
text = "".join(texts)
# Article text processing
text = self.articletext(node, text) if article else text
# Return text, strip leading/trailing whitespace if this is a string only node
text = text if node.name and text else text.strip()
return text
def header(self, node, article):
"""
Header handler. This method transforms a HTML heading into a Markdown formatted heading.
Args:
node: input node
article: True if the main section node is an article
Returns:
heading as markdown
"""
# Get heading level and text
level = "#" * int(node.name[1])
text = self.text(node, article)
# Add section break or newline, if necessary
level = f"\f{level}" if self.sections else f"\n{level}"
# Return formatted header. Remove leading whitespace as it was added before level in step above.
return f"{level} {text.lstrip()}" if text.strip() else ""
def block(self, node):
"""
Blockquote handler. This method transforms a HTML blockquote or q block into a Markdown formatted
blockquote
Args:
node: input node
Returns:
block as markdown
"""
text = "\n".join(f"> {x}" for x in node.text.strip().split("\n"))
return f"{text}\n\n" if self.paragraphs else f"{text}\n"
def items(self, node, article):
"""
List handler. This method transforms a HTML ordered/unordered list into a Markdown formatted list.
Args:
node: input node
article: True if the main section node is an article
Returns:
list as markdown
"""
elements = []
for x, element in enumerate(node.find_all("li")):
# Unordered lists use dashes. Ordered lists use numbers.
prefix = "-" if node.name == "ul" else f"{x + 1}."
# List item text
text = self.process(element, article)
# Add list element
if text:
elements.append(f"{prefix} {text}")
# Join elements together as string
return "\n".join(elements)
def code(self, node):
"""
Code block handler. This method transforms a HTML pre or code block into a Markdown formatted
code block.
Args:
node: input node
Returns:
code as markdown
"""
text = f"```\n{node.text.strip()}\n```"
return f"{text}\n\n" if self.paragraphs else f"{text}\n"
def table(self, node, article):
"""
Table handler. This method transforms a HTML table into a Markdown formatted table.
Args:
node: input node
article: True if the main section node is an article
Returns:
table as markdown
"""
elements, header = [], False
# Process all rows
rows = node.find_all("tr")
for row in rows:
# Get list of columns for row
columns = row.find_all(lambda tag: tag.name in ("th", "td"))
# Add columns with separator
elements.append(f"|{'|'.join(self.process(column, article) for column in columns)}|")
# If there are multiple rows, add header format row
if not header and len(rows) > 1:
elements.append(f"{'|---' * len(columns)}|")
header = True
# Join elements together as string
return "\n".join(elements)
def iscontainer(self, node, children):
"""
Analyzes a node and it's children to determine if this is a container element. A container
element is defined as being a div, body, article or not having any string elements as children.
Args:
node: input node
nodes: input node's children
Returns:
True if this is a container element, False otherwise
"""
return children and (node.name in ("div", "body", "article") or not any(isinstance(x, NavigableString) for x in children))
def children(self, node):
"""
Gets the node children, if available.
Args:
node: input node
Returns:
node children or None if not available
"""
if node.name and node.contents:
# Iterate over children and remove whitespace-only string nodes
return [node for node in node.contents if node.name or node.text.strip()]
return None
def articletext(self, node, text):
"""
Transforms node text using article parsing rules. Article parsing is designed to extract text content from web articles.
It ignores navigation headers and other superfluous elements.
Args:
node: input node
text: current text
Returns:
article text
"""
# List of valid text nodes
valid = ["p", "th", "td", "li", "a", "b", "strong", "i", "em"]
# Check if text is valid article text
text = text if (node.name in valid or self.isheader(node)) and not self.islink(node) else ""
if text:
# Replace non-breaking space plus newline with double newline
text = text.replace("\xa0\n", "\n\n")
# Format paragraph whitespace
if node.name == "p":
text = f"{text.strip()}\n\n" if self.paragraphs else f"{text.strip()}\n"
return text
def isheader(self, node):
"""
Checks if node is a header node.
Args:
node: input node
Returns:
True if node is a header node, False otherwise
"""
return node.name in ["h1", "h2", "h3", "h4", "h5", "h6"]
def islink(self, node):
"""
Checks if node is a link node. This method does not consider links without tables as link nodes.
Args:
node: input node
Returns:
True if node is a link node, False otherwise
"""
# Check if this is a link node or link container
link, parent = False, node
while parent:
if parent.name == "a":
link = True
break
parent = parent.parent
# Return if this node or any parents are a link. Ignore links in table cells.
return link and node.parent.name not in ("th", "td")
test/python/testoptional.py
+13 -4
View File
@@ -22,7 +22,9 @@ class TestOptional(unittest.TestCase):
modules = [
"annoy",
"bs4",
"croniter",
"docling.document_converter",
"duckdb",
"fastapi",
"fasttext",
@@ -184,8 +186,10 @@ class TestOptional(unittest.TestCase):
AudioMixer,
AudioStream,
Caption,
FileToHTML,
HFOnnx,
HFTrainer,
HTMLToMarkdown,
ImageHash,
LiteLLM,
LlamaCpp,
@@ -194,7 +198,6 @@ class TestOptional(unittest.TestCase):
Objects,
Segmentation,
Tabular,
Textractor,
TextToAudio,
TextToSpeech,
Transcription,
@@ -210,12 +213,21 @@ class TestOptional(unittest.TestCase):
with self.assertRaises(ImportError):
Caption()
with self.assertRaises(ImportError):
FileToHTML(backend="docling")
with self.assertRaises(ImportError):
FileToHTML(backend="tika")
with self.assertRaises(ImportError):
HFOnnx()("google/bert_uncased_L-2_H-128_A-2", quantize=True)
with self.assertRaises(ImportError):
HFTrainer()(None, None, lora=True)
with self.assertRaises(ImportError):
HTMLToMarkdown()
with self.assertRaises(ImportError):
ImageHash()
@@ -240,9 +252,6 @@ class TestOptional(unittest.TestCase):
with self.assertRaises(ImportError):
Tabular()
with self.assertRaises(ImportError):
Textractor()
with self.assertRaises(ImportError):
TextToAudio()
test/python/testpipeline/testdata/testfiletohtml.py
Vendored
+24
View File
@@ -0,0 +1,24 @@
"""
FileToHTML module tests
"""
import os
import unittest
from unittest.mock import patch
from txtai.pipeline.data.filetohtml import Tika
class TestFileToHTML(unittest.TestCase):
"""
FileToHTML tests.
"""
@patch.dict(os.environ, {"TIKA_JAVA": "1112444abc"})
def testTika(self):
"""
Test the Tika.available returns False when Java is not available
"""
self.assertFalse(Tika.available())
test/python/testpipeline/testdata/testtextractor.py
Vendored
+29 -13
View File
@@ -1,5 +1,5 @@
"""
Summary module tests
Textractor module tests
"""
import unittest
@@ -15,14 +15,6 @@ class TestTextractor(unittest.TestCase):
Textractor tests.
"""
def testCheckJava(self):
"""
Test the checkjava method
"""
textractor = Textractor()
self.assertFalse(textractor.checkjava("1112444abc"))
def testClean(self):
"""
Test text cleaning method
@@ -46,7 +38,7 @@ class TestTextractor(unittest.TestCase):
"""
# Text input
textractor = Textractor(tika=False)
textractor = Textractor(backend=None)
text = textractor(Utils.PATH + "/tabular.csv")
self.assertEqual(len(text), 125)
@@ -57,6 +49,17 @@ class TestTextractor(unittest.TestCase):
# Check number of sections is as expected
self.assertEqual(len(sections), 2)
def testDocling(self):
"""
Test docling backend
"""
textractor = Textractor(backend="docling")
# Extract text and check for Markdown formatting
text = textractor(Utils.PATH + "/article.pdf")
self.assertTrue("## Introducing txtai" in text)
def testLines(self):
"""
Test extraction to lines
@@ -177,14 +180,27 @@ class TestTextractor(unittest.TestCase):
# Check for table header
self.assertTrue("|---|" in text)
def testTikaFlag(self):
"""
Test legacy tika flag
"""
textractor = Textractor(tika=True)
self.assertIsNotNone(textractor.html)
textractor = Textractor(tika=False)
self.assertIsNone(textractor.html)
def testURL(self):
"""
Test parsing a remote URL
"""
textractor = Textractor()
text = textractor("https://github.com/neuml/txtai")
self.assertTrue("txtai is an all-in-one embeddings database" in text)
# Test parsing URLs for each backend
for backend in ["docling", "tika"]:
textractor = Textractor(backend=backend)
text = textractor("https://github.com/neuml/txtai")
self.assertTrue("txtai is an all-in-one embeddings database" in text)
def assertMarkdown(self, html, expected):
"""