langchain-ai/docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/docs.git synced 2026-07-23 12:55:40 -04:00
Code Issues 490 Packages Projects Releases Wiki Activity

[GH-ISSUE #1175] [langchain]: The default list of RecursiveCharacterTextSplitter should include sentence splitting characters #159

New Issue
Closed
opened 2026-02-17 17:19:18 -05:00 by yindo · 1 comment
yindo commented 2026-02-17 17:19:18 -05:00
Owner
Copy Link
Copy Source

Originally created by @mdrxy on GitHub (Oct 28, 2025).
Original GitHub issue: https://github.com/langchain-ai/docs/issues/1175

Type of issue

issue / bug

Language

Python

Description

Transferring from langchain discussions

It seems that the default list of RecursiveCharacterTextSplitter should include sentence splitting characters (".", "!", "?"). Otherwise the documentation is misleading ("This has the effect of trying to keep all paragraphs (and then sentences, and then words) together as long as possible"). Currently, it does not try to keep sentences together.

https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter

You're absolutely right that the documentation could make this clearer. The default `RecursiveCharacterTextSplitter` does **not** include sentence-level separators like `"."`, `"!"`, or `"?"` in its `separators` list. Instead, it prioritizes structural boundaries (paragraphs, newlines, spaces) for general-purpose use cases.

This design choice was intentional — the default splitter aims to work well across **non-natural language content** (e.g., code, markdown, data tables) where punctuation-based splitting might cause unwanted fragmentation.

If you want sentence-aware behavior (to actually “keep sentences together” as the docs suggest), you can explicitly override the `separators` list, for example:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", ".", "!", "?", " ", ""],
    chunk_size=1000,
    chunk_overlap=100
)

This will preserve sentence-level structure before falling back to finer splits.

That said, your observation about the documentation wording is valid — it could better reflect that sentence-level separators are *not* part of the default configuration. A small docs clarification or PR adding these as optional defaults might help align expectations.
Originally created by @mdrxy on GitHub (Oct 28, 2025). Original GitHub issue: https://github.com/langchain-ai/docs/issues/1175 ### Type of issue issue / bug ### Language Python ### Description Transferring from langchain discussions ``` It seems that the default list of RecursiveCharacterTextSplitter should include sentence splitting characters (".", "!", "?"). Otherwise the documentation is misleading ("This has the effect of trying to keep all paragraphs (and then sentences, and then words) together as long as possible"). Currently, it does not try to keep sentences together. https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter ``` ``` You're absolutely right that the documentation could make this clearer. The default `RecursiveCharacterTextSplitter` does **not** include sentence-level separators like `"."`, `"!"`, or `"?"` in its `separators` list. Instead, it prioritizes structural boundaries (paragraphs, newlines, spaces) for general-purpose use cases. This design choice was intentional — the default splitter aims to work well across **non-natural language content** (e.g., code, markdown, data tables) where punctuation-based splitting might cause unwanted fragmentation. If you want sentence-aware behavior (to actually “keep sentences together” as the docs suggest), you can explicitly override the `separators` list, for example: from langchain.text_splitter import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( separators=["\n\n", "\n", ".", "!", "?", " ", ""], chunk_size=1000, chunk_overlap=100 ) This will preserve sentence-level structure before falling back to finer splits. That said, your observation about the documentation wording is valid — it could better reflect that sentence-level separators are *not* part of the default configuration. A small docs clarification or PR adding these as optional defaults might help align expectations. ```
yindo added the langchaininternal labels 2026-02-17 17:19:18 -05:00
yindo commented 2026-02-17 17:19:18 -05:00
Author
Owner
Copy Link
Copy Source

@akshatsabavat commented on GitHub (Nov 23, 2025):

Hey there, so I had a question regarding the text splitters I choose,
I am working on making a policy ingestion pipeline that will specifically first classify chunks and then move a head to create rules out of it, using some prompts and pydantic objects I defined

Since I am kind of dealing with a lot of natural language documents what would be your reommendation

    recursive_splitter = RecursiveCharacterTextSplitter(
        chunk_size=800,   # max chars per chunk before splitting, bigger chunks better for rules
        chunk_overlap=0,  # No context preservation this is designed for rules
        separators=["\n\n", "\n", "●", ". "]  # splitting at paragraphs, new lines, bullets, sentences
    )

these are my settings right now

thanks a ton, in advance !!

@akshatsabavat commented on GitHub (Nov 23, 2025): Hey there, so I had a question regarding the text splitters I choose, I am working on making a policy ingestion pipeline that will specifically first classify chunks and then move a head to create rules out of it, using some prompts and pydantic objects I defined Since I am kind of dealing with a lot of natural language documents what would be your reommendation recursive_splitter = RecursiveCharacterTextSplitter( chunk_size=800, # max chars per chunk before splitting, bigger chunks better for rules chunk_overlap=0, # No context preservation this is designed for rules separators=["\n\n", "\n", "●", ". "] # splitting at paragraphs, new lines, bullets, sentences ) these are my settings right now thanks a ton, in advance !!
yindo changed title from [langchain]: The default list of `RecursiveCharacterTextSplitter` should include sentence splitting characters to [GH-ISSUE #1175] [langchain]: The default list of `RecursiveCharacterTextSplitter` should include sentence splitting characters 2026-06-05 17:25:22 -04:00
yindo closed this issue 2026-06-05 17:25:23 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
deepagents
docs-infra
external
external
help wanted
integration
internal
internal
javascript
js/ts
langchain
langgraph
langsmith
needs-title
pull-request
Mirrored from GitHub Pull Request
 python
reference
No Label internal langchain
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/docs#159
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?