vector database collection_name too long #8934

Closed
opened 2026-02-21 18:28:42 -05:00 by yindo · 2 comments
Owner

Originally created by @leonzy on GitHub (Mar 3, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

2025-03-03 12:24:51.062 ERROR [Dummy-15] [app.py:875] - Exception on /console/api/datasets/0eee4051-7c8f-4e81-a4ee-6d745a9c0d86/documents [POST]
Traceback (most recent call last):
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(40)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper
resp = resource(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view
return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
resp = meth(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/api/controllers/console/wraps.py", line 147, in decorated
return view(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/api/libs/login.py", line 94, in decorated_view
return current_app.ensure_sync(func)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/controllers/console/wraps.py", line 27, in decorated
return view(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 696, in wrapper
resp = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/app/api/controllers/console/wraps.py", line 86, in decorated
return view(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/api/controllers/console/datasets/datasets_document.py", line 287, in post
documents, batch = DocumentService.save_document_with_dataset_id(dataset, knowledge_config, current_user)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/services/dataset_service.py", line 824, in save_document_with_dataset_id
dataset_collection_binding = DatasetCollectionBindingService.get_dataset_collection_binding(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/services/dataset_service.py", line 2085, in get_dataset_collection_binding
db.session.commit()
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit
return self._proxied.commit()
^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2028, in commit
trans.commit(_to_root=True)
File "", line 2, in commit
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
ret_value = fn(self, *arg, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1313, in commit
self._prepare_impl()
File "", line 2, in _prepare_impl
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
ret_value = fn(self, *arg, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1288, in _prepare_impl
self.session.flush()
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
self._flush(objects)
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4487, in _flush
with util.safe_reraise():
^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in exit
raise exc_value.with_traceback(exc_tb)
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush
flush_context.execute()
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
rec.execute(self)
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
util.preloaded.orm_persistence.save_obj(
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
_emit_insert_statements(
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
result = connection.execute(
^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
return meth(
^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
self._handle_dbapi_exception(
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(40)

[SQL: INSERT INTO dataset_collection_bindings (provider_name, model_name, type, collection_name) VALUES (%(provider_name)s, %(model_name)s, %(type)s, %(collection_name)s) RETURNING dataset_collection_bindings.id, dataset_collection_bindings.created_at]
[parameters: {'provider_name': 'langgenius/openai_api_compatible/openai_api_compatible', 'model_name': 'embedding_v1', 'type': 'dataset', 'collection_name': 'Vector_index_fd0dddf3_c4a1_40eb_a5ae_e92ac9cd8932_Node'}]
(Background on this error at: https://sqlalche.me/e/20/9h9h
import a pdf file, you will see an error message. The collection_name has 55 charactors which is longer than the required charactor varing(40)

✔️ Expected Behavior

the collection_name is too long, you should truncate it into 40 charactors

Actual Behavior

No response

Originally created by @leonzy on GitHub (Mar 3, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 2025-03-03 12:24:51.062 ERROR [Dummy-15] [app.py:875] - Exception on /console/api/datasets/0eee4051-7c8f-4e81-a4ee-6d745a9c0d86/documents [POST] Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context self.dialect.do_execute( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute cursor.execute(statement, parameters) psycopg2.errors.StringDataRightTruncation: value too long for type character varying(40) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper resp = resource(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request resp = meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/api/controllers/console/wraps.py", line 147, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/api/libs/login.py", line 94, in decorated_view return current_app.ensure_sync(func)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/controllers/console/wraps.py", line 27, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 696, in wrapper resp = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/app/api/controllers/console/wraps.py", line 86, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/api/controllers/console/datasets/datasets_document.py", line 287, in post documents, batch = DocumentService.save_document_with_dataset_id(dataset, knowledge_config, current_user) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/services/dataset_service.py", line 824, in save_document_with_dataset_id dataset_collection_binding = DatasetCollectionBindingService.get_dataset_collection_binding( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/services/dataset_service.py", line 2085, in get_dataset_collection_binding db.session.commit() File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit return self._proxied.commit() ^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2028, in commit trans.commit(_to_root=True) File "<string>", line 2, in commit File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go ret_value = fn(self, *arg, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1313, in commit self._prepare_impl() File "<string>", line 2, in _prepare_impl File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go ret_value = fn(self, *arg, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1288, in _prepare_impl self.session.flush() File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4352, in flush self._flush(objects) File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4487, in _flush with util.safe_reraise(): ^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__ raise exc_value.with_traceback(exc_tb) File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush flush_context.execute() File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute rec.execute(self) File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute util.preloaded.orm_persistence.save_obj( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj _emit_insert_statements( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements result = connection.execute( ^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1418, in execute return meth( ^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection return connection._execute_clauseelement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context return self._exec_single_context( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context self._handle_dbapi_exception( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context self.dialect.do_execute( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.DataError: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(40) [SQL: INSERT INTO dataset_collection_bindings (provider_name, model_name, type, collection_name) VALUES (%(provider_name)s, %(model_name)s, %(type)s, %(collection_name)s) RETURNING dataset_collection_bindings.id, dataset_collection_bindings.created_at] [parameters: {'provider_name': 'langgenius/openai_api_compatible/openai_api_compatible', 'model_name': 'embedding_v1', 'type': 'dataset', 'collection_name': 'Vector_index_fd0dddf3_c4a1_40eb_a5ae_e92ac9cd8932_Node'}] (Background on this error at: https://sqlalche.me/e/20/9h9h import a pdf file, you will see an error message. The collection_name has 55 charactors which is longer than the required charactor varing(40) ### ✔️ Expected Behavior the collection_name is too long, you should truncate it into 40 charactors ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug👻 feat:rag labels 2026-02-21 18:28:42 -05:00
yindo closed this issue 2026-02-21 18:28:42 -05:00
Author
Owner

@dosubot[bot] commented on GitHub (Mar 3, 2025):

To resolve the issue of the collection_name exceeding the 40-character limit in the vector database, you should truncate the collection_name to ensure it does not exceed this limit. You can modify the code where the collection_name is set to ensure it is truncated to a maximum of 40 characters before it is used. Here's an example of how you might handle this:

class PGVectoRSFactory(AbstractVectorFactory):
    def init_vector(self, dataset: Dataset, attributes: list, embeddings: Embeddings) -> PGVectoRS:
        if dataset.index_struct_dict:
            class_prefix: str = dataset.index_struct_dict["vector_store"]["class_prefix"]
            collection_name = class_prefix.lower()
        else:
            dataset_id = dataset.id
            collection_name = Dataset.gen_collection_name_by_id(dataset_id).lower()
            dataset.index_struct = json.dumps(self.gen_index_struct_dict(VectorType.PGVECTO_RS, collection_name))
        
        # Ensure collection_name does not exceed 40 characters
        collection_name = collection_name[:40]

        dim = len(embeddings.embed_query("pgvecto_rs"))

        return PGVectoRS(
            collection_name=collection_name,
            config=PgvectoRSConfig(
                host=dify_config.PGVECTO_RS_HOST or "localhost",
                port=dify_config.PGVECTO_RS_PORT or 5432,
                user=dify_config.PGVECTO_RS_USER or "postgres",
                password=dify_config.PGVECTO_RS_PASSWORD or "",
                database=dify_config.PGVECTO_RS_DATABASE or "postgres",
            ),
            dim=dim,
        )

This code snippet ensures that the collection_name is truncated to a maximum of 40 characters before it is used to initialize the PGVectoRS instance [1].

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 3, 2025): <!-- Answer --> To resolve the issue of the `collection_name` exceeding the 40-character limit in the vector database, you should truncate the `collection_name` to ensure it does not exceed this limit. You can modify the code where the `collection_name` is set to ensure it is truncated to a maximum of 40 characters before it is used. Here's an example of how you might handle this: ```python class PGVectoRSFactory(AbstractVectorFactory): def init_vector(self, dataset: Dataset, attributes: list, embeddings: Embeddings) -> PGVectoRS: if dataset.index_struct_dict: class_prefix: str = dataset.index_struct_dict["vector_store"]["class_prefix"] collection_name = class_prefix.lower() else: dataset_id = dataset.id collection_name = Dataset.gen_collection_name_by_id(dataset_id).lower() dataset.index_struct = json.dumps(self.gen_index_struct_dict(VectorType.PGVECTO_RS, collection_name)) # Ensure collection_name does not exceed 40 characters collection_name = collection_name[:40] dim = len(embeddings.embed_query("pgvecto_rs")) return PGVectoRS( collection_name=collection_name, config=PgvectoRSConfig( host=dify_config.PGVECTO_RS_HOST or "localhost", port=dify_config.PGVECTO_RS_PORT or 5432, user=dify_config.PGVECTO_RS_USER or "postgres", password=dify_config.PGVECTO_RS_PASSWORD or "", database=dify_config.PGVECTO_RS_DATABASE or "postgres", ), dim=dim, ) ``` This code snippet ensures that the `collection_name` is truncated to a maximum of 40 characters before it is used to initialize the `PGVectoRS` instance <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/rag/datasource/vdb/pgvecto_rs/pgvecto_rs.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/1a0b3f0d-d442-4184-b025-12a7d73aaa33?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/1a0b3f0d-d442-4184-b025-12a7d73aaa33?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/1a0b3f0d-d442-4184-b025-12a7d73aaa33?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/1a0b3f0d-d442-4184-b025-12a7d73aaa33?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/1a0b3f0d-d442-4184-b025-12a7d73aaa33?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/1a0b3f0d-d442-4184-b025-12a7d73aaa33?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/1a0b3f0d-d442-4184-b025-12a7d73aaa33?feedback_type=other)</sup>
Author
Owner

@crazywoola commented on GitHub (Mar 3, 2025):

fix: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(40) Fixes #14593 (#14597)

@crazywoola commented on GitHub (Mar 3, 2025): fix: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(40) Fixes #14593 (#14597)
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#8934