langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Migration files delete vector tables with pgvector in PostgreSQL #7289

New Issue
Closed
opened 2026-02-21 18:19:50 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 18:19:50 -05:00
Owner
Copy Link
Copy Source

Originally created by @faye1225 on GitHub (Dec 19, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.14.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When deploying Dify with pgvector as the vector store, the vector tables are stored with names starting with embedding_vector_index_ in PostgreSQL. However, when generating migration files, the new migration will delete all these vector tables.

There are two potential solutions to address this issue:

1. Store the base data in one database and store the vectors in another database.
2. a line of code to the migration file to include an exception for these vector tables:

/api/migrations/env.py

.....
def include_name(name, type_, parent_names):
    if type_ == "table":
        # takin command:忽略所有 embedding_vector_index 开头的表
        return not name.startswith('embedding_vector_index_')
    return True

...


def run_migrations_online():
...
    with connectable.connect() as connection:
        context.configure(
            connection=connection,
            target_metadata=get_metadata(),
            process_revision_directives=process_revision_directives,
            include_object=include_object,
            include_name=include_name, // add this
            **current_app.extensions['migrate'].configure_args
        )


....

✔️ Expected Behavior

Migration files should not delete vector tables, or they should provide a way to manage these tables properly in the migration process.

❌ Actual Behavior

No response

Originally created by @faye1225 on GitHub (Dec 19, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.14.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce When deploying Dify with pgvector as the vector store, the vector tables are stored with names starting with **embedding_vector_index_** in PostgreSQL. However, when generating migration files, the new migration will delete all these vector tables. There are two potential solutions to address this issue: **1. Store the base data in one database and store the vectors in another database.** **2. a line of code to the migration file to include an exception for these vector tables:** `/api/migrations/env.py` ```python ..... def include_name(name, type_, parent_names): if type_ == "table": # takin command:忽略所有 embedding_vector_index 开头的表 return not name.startswith('embedding_vector_index_') return True ... def run_migrations_online(): ... with connectable.connect() as connection: context.configure( connection=connection, target_metadata=get_metadata(), process_revision_directives=process_revision_directives, include_object=include_object, include_name=include_name, // add this **current_app.extensions['migrate'].configure_args ) .... ``` ### ✔️ Expected Behavior Migration files should not delete vector tables, or they should provide a way to manage these tables properly in the migration process. ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 18:19:50 -05:00
yindo closed this issue 2026-02-21 18:19:50 -05:00
yindo commented 2026-02-21 18:19:50 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 19, 2024):

To prevent migration files from deleting vector tables with names starting with embedding_vector_index_ in PostgreSQL, you can implement a custom function in the env.py file to exclude these tables from migrations. The include_name function you've mentioned is a suitable approach. Here's how it works:

  1. Custom Function: The include_name function checks the name of each table during the migration process. If the table name starts with embedding_vector_index_, it returns False, effectively excluding these tables from being affected by the migration.

  2. Integration in env.py: You need to ensure that this function is integrated into the migration context configuration. This is done by adding include_name=include_name to the context.configure call in the run_migrations_online function.

Here's a brief overview of the code snippet you provided:

def include_name(name, type_, parent_names):
    if type_ == "table":
        # Ignore all tables starting with 'embedding_vector_index_'
        return not name.startswith('embedding_vector_index_')
    return True

def run_migrations_online():
    ...
    with connectable.connect() as connection:
        context.configure(
            connection=connection,
            target_metadata=get_metadata(),
            process_revision_directives=process_revision_directives,
            include_object=include_object,
            include_name=include_name,  # Add this line
            **current_app.extensions['migrate'].configure_args
        )
    ...

By implementing this change, you ensure that the migration process respects the presence of these vector tables and does not delete them inadvertently. This approach allows you to manage your vector tables effectively without needing to separate them into a different database.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Dec 19, 2024): <!-- Answer --> To prevent migration files from deleting vector tables with names starting with `embedding_vector_index_` in PostgreSQL, you can implement a custom function in the `env.py` file to exclude these tables from migrations. The `include_name` function you've mentioned is a suitable approach. Here's how it works: 1. **Custom Function**: The `include_name` function checks the name of each table during the migration process. If the table name starts with `embedding_vector_index_`, it returns `False`, effectively excluding these tables from being affected by the migration. 2. **Integration in `env.py`**: You need to ensure that this function is integrated into the migration context configuration. This is done by adding `include_name=include_name` to the `context.configure` call in the `run_migrations_online` function. Here's a brief overview of the code snippet you provided: ```python def include_name(name, type_, parent_names): if type_ == "table": # Ignore all tables starting with 'embedding_vector_index_' return not name.startswith('embedding_vector_index_') return True def run_migrations_online(): ... with connectable.connect() as connection: context.configure( connection=connection, target_metadata=get_metadata(), process_revision_directives=process_revision_directives, include_object=include_object, include_name=include_name, # Add this line **current_app.extensions['migrate'].configure_args ) ... ``` By implementing this change, you ensure that the migration process respects the presence of these vector tables and does not delete them inadvertently. This approach allows you to manage your vector tables effectively without needing to separate them into a different database. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/93ed3057-d94b-4122-abcf-64baa8c1fc8f?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/93ed3057-d94b-4122-abcf-64baa8c1fc8f?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/93ed3057-d94b-4122-abcf-64baa8c1fc8f?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/93ed3057-d94b-4122-abcf-64baa8c1fc8f?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/93ed3057-d94b-4122-abcf-64baa8c1fc8f?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/93ed3057-d94b-4122-abcf-64baa8c1fc8f?feedback_type=other)</sup>
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7289
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?