langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Add the Index Shards and Replicas Parameter Configuration for Elasticsearch Vector Database #20362

New Issue
Closed
opened 2026-02-21 20:07:01 -05:00 by yindo · 4 comments
yindo commented 2026-02-21 20:07:01 -05:00
Owner
Copy Link
Copy Source

Originally created by @canghaizhiyue on GitHub (Nov 17, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I want to use an ES cluster as my vector database, but I currently haven't figured out how to configure the shards and replicas for the index. I've only seen the default configuration of 1 shard and 1 replica. Are there any related configurations available?

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @canghaizhiyue on GitHub (Nov 17, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. I want to use an ES cluster as my vector database, but I currently haven't figured out how to configure the shards and replicas for the index. I've only seen the default configuration of 1 shard and 1 replica. Are there any related configurations available? ### 2. Additional context or comments _No response_ ### 3. Can you help us with this feature? - [ ] I am interested in contributing to this feature.
yindo closed this issue 2026-02-21 20:07:01 -05:00
yindo commented 2026-02-21 20:07:02 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 17, 2025):

Hi @canghaizhiyue! I'm Dosu and I’m helping the dify team.

Dify does not currently support configuring Elasticsearch index shards and replicas for vector database usage. The index creation logic in the codebase always uses Elasticsearch's defaults (1 shard, 1 replica) and does not expose any way to customize these settings through environment variables or configuration. The relevant implementation is in the ElasticSearchVector.create_collection method, which only sets up the index mappings and does not pass shard or replica parameters to the Elasticsearch API. There is an open issue tracking this feature request, but no merged pull requests or workarounds are available yet. If you need custom shard/replica settings, you would need to modify the code to include these parameters when creating the index, or manually pre-create the index in your Elasticsearch cluster with your desired settings before Dify tries to use it.
See code reference

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Nov 17, 2025): <!-- Greeting --> Hi @canghaizhiyue! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify team. <!-- Answer --> Dify does not currently support configuring Elasticsearch index shards and replicas for vector database usage. The index creation logic in the codebase always uses Elasticsearch's defaults (1 shard, 1 replica) and does not expose any way to customize these settings through environment variables or configuration. The relevant implementation is in the `ElasticSearchVector.create_collection` method, which only sets up the index mappings and does not pass shard or replica parameters to the Elasticsearch API. There is an open issue tracking this feature request, but no merged pull requests or workarounds are available yet. If you need custom shard/replica settings, you would need to modify the code to include these parameters when creating the index, or manually pre-create the index in your Elasticsearch cluster with your desired settings before Dify tries to use it. [See code reference](https://github.com/langgenius/dify/blob/4a89403566c85eff76a959bc62be76f1e8af5ce6/api/core/rag/datasource/vdb/elasticsearch/elasticsearch_vector.py#L73-L358) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28276)
yindo commented 2026-02-21 20:07:02 -05:00
Author
Owner
Copy Link
Copy Source

@canghaizhiyue commented on GitHub (Nov 17, 2025):

Hi @canghaizhiyue! I'm Dosu and I’m helping the dify team.

Dify does not currently support configuring Elasticsearch index shards and replicas for vector database usage. The index creation logic in the codebase always uses Elasticsearch's defaults (1 shard, 1 replica) and does not expose any way to customize these settings through environment variables or configuration. The relevant implementation is in the ElasticSearchVector.create_collection method, which only sets up the index mappings and does not pass shard or replica parameters to the Elasticsearch API. There is an open issue tracking this feature request, but no merged pull requests or workarounds are available yet. If you need custom shard/replica settings, you would need to modify the code to include these parameters when creating the index, or manually pre-create the index in your Elasticsearch cluster with your desired settings before Dify tries to use it. See code reference

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

OK，3Q for your reply!

@canghaizhiyue commented on GitHub (Nov 17, 2025): > Hi [@canghaizhiyue](https://github.com/canghaizhiyue)! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify team. > > Dify does not currently support configuring Elasticsearch index shards and replicas for vector database usage. The index creation logic in the codebase always uses Elasticsearch's defaults (1 shard, 1 replica) and does not expose any way to customize these settings through environment variables or configuration. The relevant implementation is in the `ElasticSearchVector.create_collection` method, which only sets up the index mappings and does not pass shard or replica parameters to the Elasticsearch API. There is an open issue tracking this feature request, but no merged pull requests or workarounds are available yet. If you need custom shard/replica settings, you would need to modify the code to include these parameters when creating the index, or manually pre-create the index in your Elasticsearch cluster with your desired settings before Dify tries to use it. [See code reference](https://github.com/langgenius/dify/blob/4a89403566c85eff76a959bc62be76f1e8af5ce6/api/core/rag/datasource/vdb/elasticsearch/elasticsearch_vector.py#L73-L358) > > _To reply, just mention [@dosu](https://go.dosu.dev/dosubot)._ > > How did I do? [Good](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2fbd7917-7bb0-491f-9868-464bcde26abe?feedback_type=other)  [![Chat with Dosu](https://camo.githubusercontent.com/7c571478962a6b8a5d47cac74b6824c75e4f6731bba52377e6007f9150462923/68747470733a2f2f646f73752e6465762f646f73752d636861742d62616467652e737667)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github) [![Join Discord](https://camo.githubusercontent.com/7d2066700925db1e370d956834f9ea0e5bc92cecbe84963fbd2a363e45091b4e/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6a6f696e2d3538363546323f6c6f676f3d646973636f7264266c6f676f436f6c6f723d7768697465266c6162656c3d)](https://go.dosu.dev/discord-bot) [![Share on X](https://camo.githubusercontent.com/a8bd7f3fcf3f5e3bd124eee9ae50ae2ac17b40971726665ac2121d48f9aee155/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f582d73686172652d626c61636b)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28276) OK，3Q for your reply!
yindo commented 2026-02-21 20:07:02 -05:00
Author
Owner
Copy Link
Copy Source

@canghaizhiyue commented on GitHub (Dec 1, 2025):

@dosu According to your suggestion, I have made the following modifications to meet the relevant requirements：
1、modify the .env file，add the configs as this:

ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=difytest
# add shards and replicas config
ELASTICSEARCH_SHARDS=3
ELASTICSEARCH_REPLICAS=2

2、modify the api/configs/middleware/vdb/elasitcsearch_config.py file as this:

......

ELASTICSEARCH_PASSWORD: Optional[str] = Field(
    description="Password for authenticating with Elasticsearch (default is 'elastic')",
    default="elastic",
)
“””add the following features”“”
ELASTICSEARCH_SHARDS: PositiveInt = Field(
    description="Number of shards for the Elasticsearch Vector Database (default is 1)",
    default=1,
)
ELASTICSEARCH_REPLICAS: NonNegativeInt = Field(
    description="Number of replicas for the Elasticsearch Vector Database (default is 1)",
    default=1,
)

3、modify the api/core/rag/datasource/vdb/elasticsearch/elasticsearch_vector.py file as this:

.......
  **from configs import dify_config**
 .......
 class ElasticSearchConfig(BaseModel):
  ......
   password: str
   shards: int   
   replicas: int
 .....
 class ElasticSearchVector(BaseVector):
   def __init__(self, index_name: str, config: ElasticSearchConfig, attributes: list):
       super().__init__(index_name.lower())
       self._client_config = config
       self._client = self._init_client(config)
       .......
   def create_collection(
       self,
       embeddings: list[list[float]],
       metadatas: Optional[list[dict[Any, Any]]] = None,
       index_params: Optional[dict] = None,
   ):
       lock_name = f"vector_indexing_lock_{self._collection_name}"
       ......

        settings={
                 "number_of_shards": self._client_config.shards,
                 "number_of_replicas": self._client_config.replicas
         }
        self._client.indices.create(index=self._collection_name, mappings=mappings ,settings=settings)
......
class ElasticSearchVectorFactory(AbstractVectorFactory):
      ......
       return ElasticSearchVector(
           index_name=collection_name,
           config=ElasticSearchConfig(
               host=dify_config.ELASTICSEARCH_HOST or "",
               port=dify_config.ELASTICSEARCH_PORT or 9200,
               username=dify_config.ELASTICSEARCH_USERNAME or "",
               password=dify_config.ELASTICSEARCH_PASSWORD or "",
               shards=dify_config.ELASTICSEARCH_SHARDS,
               replicas=dify_config.ELASTICSEARCH_REPLICAS,
           ),
           attributes=[],
       )
@canghaizhiyue commented on GitHub (Dec 1, 2025): @dosu According to your suggestion, I have made the following modifications to meet the relevant requirements： 1、modify the .env file，add the configs as this: ``` ELASTICSEARCH_USERNAME=elastic ELASTICSEARCH_PASSWORD=difytest # add shards and replicas config ELASTICSEARCH_SHARDS=3 ELASTICSEARCH_REPLICAS=2 ``` 2、modify the _api/configs/middleware/vdb/elasitcsearch_config.py_ file as this: ...... ELASTICSEARCH_PASSWORD: Optional[str] = Field( description="Password for authenticating with Elasticsearch (default is 'elastic')", default="elastic", ) “””add the following features”“” ELASTICSEARCH_SHARDS: PositiveInt = Field( description="Number of shards for the Elasticsearch Vector Database (default is 1)", default=1, ) ELASTICSEARCH_REPLICAS: NonNegativeInt = Field( description="Number of replicas for the Elasticsearch Vector Database (default is 1)", default=1, ) 3、modify the api/core/rag/datasource/vdb/elasticsearch/elasticsearch_vector.py file as this: ``` ....... **from configs import dify_config** ....... class ElasticSearchConfig(BaseModel): ...... password: str shards: int replicas: int ..... class ElasticSearchVector(BaseVector): def __init__(self, index_name: str, config: ElasticSearchConfig, attributes: list): super().__init__(index_name.lower()) self._client_config = config self._client = self._init_client(config) ....... def create_collection( self, embeddings: list[list[float]], metadatas: Optional[list[dict[Any, Any]]] = None, index_params: Optional[dict] = None, ): lock_name = f"vector_indexing_lock_{self._collection_name}" ...... settings={ "number_of_shards": self._client_config.shards, "number_of_replicas": self._client_config.replicas } self._client.indices.create(index=self._collection_name, mappings=mappings ,settings=settings) ...... class ElasticSearchVectorFactory(AbstractVectorFactory): ...... return ElasticSearchVector( index_name=collection_name, config=ElasticSearchConfig( host=dify_config.ELASTICSEARCH_HOST or "", port=dify_config.ELASTICSEARCH_PORT or 9200, username=dify_config.ELASTICSEARCH_USERNAME or "", password=dify_config.ELASTICSEARCH_PASSWORD or "", shards=dify_config.ELASTICSEARCH_SHARDS, replicas=dify_config.ELASTICSEARCH_REPLICAS, ), attributes=[], ) ```
yindo commented 2026-02-21 20:07:02 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 1, 2025):

Your changes are on the right track for enabling shard and replica configuration in Dify's Elasticsearch vector integration. Defining ELASTICSEARCH_SHARDS and ELASTICSEARCH_REPLICAS in the config and wiring them through to index creation is necessary for this feature to work as expected.

A couple of suggestions for completeness and robustness:

  • Make sure the new config fields are added to all relevant config classes (including any used for Elastic Cloud, if applicable).
  • Update the Japan-specific vector implementation (ElasticSearchJaVector) to also support shard and replica settings for consistency across all Elasticsearch integrations citation.
  • Validate that your environment variables are correctly loaded and mapped to the config fields, and consider adding input validation for edge cases (e.g., shards > 0, replicas ≥ 0).

If you follow these steps, your implementation should work as intended and be maintainable with future updates.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Dec 1, 2025): <!-- Answer --> Your changes are on the right track for enabling shard and replica configuration in Dify's Elasticsearch vector integration. Defining `ELASTICSEARCH_SHARDS` and `ELASTICSEARCH_REPLICAS` in the config and wiring them through to index creation is necessary for this feature to work as expected. A couple of suggestions for completeness and robustness: - Make sure the new config fields are added to all relevant config classes (including any used for Elastic Cloud, if applicable). - Update the Japan-specific vector implementation (`ElasticSearchJaVector`) to also support shard and replica settings for consistency across all Elasticsearch integrations [citation](https://github.com/langgenius/dify/blob/d162f7e5ef0db74d3396239c82e6283732f043ae/api/core/rag/datasource/vdb/elasticsearch/elasticsearch_ja_vector.py#L21-L104). - Validate that your environment variables are correctly loaded and mapped to the config fields, and consider adding input validation for edge cases (e.g., shards > 0, replicas ≥ 0). If you follow these steps, your implementation should work as intended and be maintainable with future updates. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/5f302a82-6247-46e1-b214-bb0acd400c5d?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/5f302a82-6247-46e1-b214-bb0acd400c5d?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/5f302a82-6247-46e1-b214-bb0acd400c5d?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/5f302a82-6247-46e1-b214-bb0acd400c5d?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/5f302a82-6247-46e1-b214-bb0acd400c5d?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/5f302a82-6247-46e1-b214-bb0acd400c5d?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/5f302a82-6247-46e1-b214-bb0acd400c5d?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=Your%20changes%20are%20on%20the%20right%20track%20for%20enabling%20shard%20and%20replica%20configuration%20in%20Dify%27s%20Elasticsearch%20vector%20integration.%20Defining%20%60ELASTICSEARCH_SHARDS%60%20and%20%60ELASTICSEARCH_REPLICAS%60%20in%20the%20config%20and%20wiring%20them%20through%20to%20index%20creation%20is%20necessary%20for%20this%20feature%20to%20work%20as%20expected.%20%0A%0AA%20couple%20of%20suggestions%20for%20completeness%20and%20robustness%3A%0A-%20Make%20sure%20the%20new%20config%20fields%20are%20added%20to%20all%20relevant%20config%20classes%20%28including%20any%20used%20for%20Elastic%20Cloud%2C%20if%20applicable%29.%0A-%20Update%20the%20Japan-specific%20vector%20implementation%20%28%60ElasticSearchJaVector%60%29%20to%20also%20support%20shard%20and%20replica%20settings%20for%20consistency%20across%20all%20Elasticsearch%20integrations%C2%A0%5Bcitation%5D%28https%3A//github.com/langgenius/dify/blob/d162f7e5ef0db74d3396239c82e6283732f043ae/api/core/rag/datasource/vdb/elasticsearch/elasticsearch_ja_vector.py%23L21-L104%29.%0A-%20Validate%20that%20your%20environment%20variables%20are%20correctly%20loaded%20and%20mapped%20to%20the%20config%20fields%2C%20and%20consider%20adding%20input%20validation%20for%20edge%20cases%20%28e.g.%2C%20shards%20%3E%200%2C%20replicas%20%E2%89%A5%200%29.%0A%0AIf%20you%20follow%20these%20steps%2C%20your%20implementation%20should%20work%20as%20intended%20and%20be%20maintainable%20with%20future%20updates.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28276)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#20362
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?