Originally the MarkdownTextSplitter would split tables into chunks
for each row (producing a header + single row) in each chunk. This
change adds an option to join multiple rows into a single chunk.
Fixes: #938
Extracted the logic for handling link closure and image processing into
separate functions. This reduces the complexity of the `splitInline` function,
improving readability and maintainability. Removed unnecessary comments and
linting directives.
Extend the test coverage for the Markdown splitter. The changes include
new tests for code splitting, inline elements, and handling of different
markdown features such as fenced code blocks, hard breaks, emphasis,
strikethrough, images, and links. The changes aim to ensure the robustness
of the Markdown splitter and its ability to handle a variety of markdown
syntaxes.
The changes introduce two new options in the MarkdownTextSplitter struct:
CodeBlocks and ReferenceLinks. These options allow users to specify
whether they want to render code blocks and reference links when
splitting a markdown text.
The CodeBlocks option determines whether indented and fenced code
blocks should be rendered. If set to false, these blocks are ignored
during the splitting process.
The ReferenceLinks option determines whether reference links should be
patched with the URL and title from their definition. By default,
reference definitions are dropped from the output.
These changes provide users with more control over how their markdown
text is split, allowing them to include or exclude specific elements
as needed.
The markdown splitter function 'splitInline' has been significantly
enhanced to handle a variety of inline elements as per the CommonMark
specification. Previously, it simply returned the content of the inline
element. Now, it checks for the presence of children in the inline
element and processes them accordingly.
This change was necessary to support reference links, which have the
following format:
```markdown
[foo][bar]
[bar]: /url "title"
```
When this gets parsed, the URL and title from the definition are
added to each reference link and the definition is dropped from the
document. This means when the document is rendered we end up with
incomplete links.
With this change, we render full links instead of references, so the
example above ends up looking like:
```markdown
[foo](/url "title")
```
Enhance markdown text splitting functionality by adding handlers for
markdown code blocks, fences, and horizontal rules. This change allows
the markdown splitter to properly process and format these elements,
improving the accuracy of the text splitting process. However, long
code blocks and fenced blocks will be split as text and won't be
properly wrapped, which is a known limitation that aligns with the
current behavior of the python langchain.
The markdown splitter was not correctly handling singular markdown tokens
like Hr or Fence. These tokens do not have a corresponding close type,
which was causing issues in the indexOfCloseTag function.
To fix this, a check has been added to return the start index if the
closeType is nil, indicating a singular token. This prevents the function
from incorrectly trying to find a close tag for these tokens and running
past the end of the document.
Pedram Razavi