Daniel Bos 536f16c12a textsplitter: handle singular markdown tokens in markdown_splitter (#467) ...

The markdown splitter was not correctly handling singular markdown tokens
like Hr or Fence. These tokens do not have a corresponding close type,
which was causing issues in the indexOfCloseTag function.

To fix this, a check has been added to return the start index if the
closeType is nil, indicating a singular token. This prevents the function
from incorrectly trying to find a close tag for these tokens and running
past the end of the document.

2023-12-30 03:57:23 -08:00

..

testdata

textsplitter: add markdown header text splitter (#274)

2023-09-27 12:07:05 -07:00

doc.go

Fixed lint issues

2023-06-19 09:57:09 +01:00

markdown_splitter_test.go

fix: golangci-lint v1.55 errors

2023-12-04 22:38:28 -08:00

markdown_splitter.go

textsplitter: handle singular markdown tokens in markdown_splitter (#467)

2023-12-30 03:57:23 -08:00

options.go

textsplitter: add markdown header text splitter (#274)

2023-09-27 12:07:05 -07:00

recursive_character_test.go

fix: golangci-lint v1.55 errors

2023-12-04 22:38:28 -08:00

recursive_character.go

Docs: fix comment typo (#442)

2023-12-20 19:57:37 +00:00

split_documents.go

Docs: fix comment typo (#442)

2023-12-20 19:57:37 +00:00

text_spliter.go

fix: golangci-lint v1.55 errors

2023-12-04 22:38:28 -08:00

token_splitter_test.go

fix: golangci-lint v1.55 errors

2023-12-04 22:38:28 -08:00

token_splitter.go

textsplitter: add markdown header text splitter (#274)

2023-09-27 12:07:05 -07:00