langchain-ai/langchain-postgres
1
0
Fork 0
mirror of https://github.com/langchain-ai/langchain-postgres.git synced 2026-07-16 01:33:18 -04:00
Code Issues 85 Packages Projects Releases 11 Wiki Activity

Error while indexing #1

New Issue
Open
opened 2026-02-16 05:16:00 -05:00 by yindo · 4 comments
yindo commented 2026-02-16 05:16:00 -05:00
Owner
Copy Link
Copy Source

Originally created by @Sachin-Bhat on GitHub (Apr 19, 2024).

Hello,

I was getting the following error while indexing my embeddings onto PGVector. It did not throw such errors when I was using the old langchain-community implementation. Wondering if the database schema that created with the new implementation had some changes. Any clarification on how to go about this would be very helpful.

[SQL: INSERT INTO langchain_pg_embedding (id, collection_id, embedding, document, cmetadata) VALUES (%(id_m0)s::VARCHAR, %(collection_id_m0)s::UUID, %(embedding_m0)s, %(document_m0)s::VARCHAR, %(cmetadata_m0)s::JSONB) ON CONFLICT (id) DO UPDATE SET embedding = excluded.embedding, document = excluded.document, cmetadata = excluded.cmetadata][parameters: {'id_m0': 'eb11c383-782a-491d-925d-fce35340e169', 'collection_id_m0': UUID('bc218a7a-3b99-4685-aabf-4800bd6fe6f0'), 'embedding_m0': '[0.0012261948,-0.0032268283,-0.021794429,0.028127616,0.025112469,0.00040851647,-0.010263896,-0.00914182,0.035575997,0.07880689,0.00234741,0.033373147 ... (12471 characters truncated) ... 018,-0.03224419,0.011916031,0.005445165,0.004877243,-0.06553473,-0.051298827,0.0001616641,0.07010564,0.040146906,0.021629214,-0.01095917,0.010195057]', 'document_m0': "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur vitae ex in ante fermentum tincidunt. Donec et velit molestie, faucibus sapien vel, viverra nunc. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Praesent rhoncus lacinia semper. Vestibulum non magna ac est egestas tempor. Suspendisse tempor mi nec vestibulum egestas. Vivamus a efficitur nisi, eget molestie nibh. Praesent at lorem lacus. Vivamus id nisl efficitur, faucibus justo nec, varius arcu. Praesent elementum sem euismod finibus commodo. In imperdiet tincidunt metus eget suscipit. Curabitur consectetur mi vitae sapien placerat mattis. Phasellus pellentesque ante id felis congue, tempus. \n", 'cmetadata_m0': Jsonb({'chunk_index': 5, 'id': 'eb11c383- ... (4083 chars))}](Background on this error at: https://sqlalche.me/e/20/f405)

Cheers,
Sachin

Originally created by @Sachin-Bhat on GitHub (Apr 19, 2024). Hello, I was getting the following error while indexing my embeddings onto PGVector. It did not throw such errors when I was using the old langchain-community implementation. Wondering if the database schema that created with the new implementation had some changes. Any clarification on how to go about this would be very helpful. ``` [SQL: INSERT INTO langchain_pg_embedding (id, collection_id, embedding, document, cmetadata) VALUES (%(id_m0)s::VARCHAR, %(collection_id_m0)s::UUID, %(embedding_m0)s, %(document_m0)s::VARCHAR, %(cmetadata_m0)s::JSONB) ON CONFLICT (id) DO UPDATE SET embedding = excluded.embedding, document = excluded.document, cmetadata = excluded.cmetadata][parameters: {'id_m0': 'eb11c383-782a-491d-925d-fce35340e169', 'collection_id_m0': UUID('bc218a7a-3b99-4685-aabf-4800bd6fe6f0'), 'embedding_m0': '[0.0012261948,-0.0032268283,-0.021794429,0.028127616,0.025112469,0.00040851647,-0.010263896,-0.00914182,0.035575997,0.07880689,0.00234741,0.033373147 ... (12471 characters truncated) ... 018,-0.03224419,0.011916031,0.005445165,0.004877243,-0.06553473,-0.051298827,0.0001616641,0.07010564,0.040146906,0.021629214,-0.01095917,0.010195057]', 'document_m0': "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur vitae ex in ante fermentum tincidunt. Donec et velit molestie, faucibus sapien vel, viverra nunc. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Praesent rhoncus lacinia semper. Vestibulum non magna ac est egestas tempor. Suspendisse tempor mi nec vestibulum egestas. Vivamus a efficitur nisi, eget molestie nibh. Praesent at lorem lacus. Vivamus id nisl efficitur, faucibus justo nec, varius arcu. Praesent elementum sem euismod finibus commodo. In imperdiet tincidunt metus eget suscipit. Curabitur consectetur mi vitae sapien placerat mattis. Phasellus pellentesque ante id felis congue, tempus. \n", 'cmetadata_m0': Jsonb({'chunk_index': 5, 'id': 'eb11c383- ... (4083 chars))}](Background on this error at: https://sqlalche.me/e/20/f405) ``` Cheers, Sachin
yindo commented 2026-02-16 05:16:01 -05:00
Author
Owner
Copy Link
Copy Source

@eyurtsev commented on GitHub (Apr 22, 2024):

I changed the schema due to issues with the community implementation.

Easiest solution is to re-create the schema if you don't mind re-indexing the data.

If you don't want to re-index the data, connect to the postgres instance and you'll need to write a bit of SQL to make the migration.

If I'm not mistaken the main change in the schema:

  1. Rename custom_id to id in langchain_pg_embedding
  2. JSON column was changed to JSONB
@eyurtsev commented on GitHub (Apr 22, 2024): I changed the schema due to issues with the community implementation. Easiest solution is to re-create the schema if you don't mind re-indexing the data. If you don't want to re-index the data, connect to the postgres instance and you'll need to write a bit of SQL to make the migration. If I'm not mistaken the main change in the schema: 1) Rename custom_id to id in langchain_pg_embedding 2) JSON column was changed to JSONB
yindo commented 2026-02-16 05:16:01 -05:00
Author
Owner
Copy Link
Copy Source

@Sachin-Bhat commented on GitHub (Apr 23, 2024):

Hey @eyurtsev,

Tried writing the necessary SQL to update the schema as specified by you. However, when I try to index documents, I get the following error:

Error indexing document with id 000218462 and chunk_id e49fa716-422a-430a-b633-7d0901ed45d3: (psycopg.errors.InvalidColumnReference) there is no unique or exclusion constraint matching the ON CONFLICT specification

Cheers,
Sachin

@Sachin-Bhat commented on GitHub (Apr 23, 2024): Hey @eyurtsev, Tried writing the necessary SQL to update the schema as specified by you. However, when I try to index documents, I get the following error: ``` Error indexing document with id 000218462 and chunk_id e49fa716-422a-430a-b633-7d0901ed45d3: (psycopg.errors.InvalidColumnReference) there is no unique or exclusion constraint matching the ON CONFLICT specification ``` Cheers, Sachin
yindo commented 2026-02-16 05:16:01 -05:00
Author
Owner
Copy Link
Copy Source

@eyurtsev commented on GitHub (Apr 23, 2024):

You need something like this to add a uniqueness constraint. This fixes a bug that is present in the community implementation and will make sure that you won't have duplicated content if you're indexing by id

ALTER TABLE [your_table_name]
ADD CONSTRAINT id_unique UNIQUE (id);

Double check the commands / that it matches what you have in the database. Make a backup if you need before running it etc.

@eyurtsev commented on GitHub (Apr 23, 2024): You need something like this to add a uniqueness constraint. This fixes a bug that is present in the community implementation and will make sure that you won't have duplicated content if you're indexing by id ```sql ALTER TABLE [your_table_name] ADD CONSTRAINT id_unique UNIQUE (id); ``` Double check the commands / that it matches what you have in the database. Make a backup if you need before running it etc.
yindo commented 2026-02-16 05:16:01 -05:00
Author
Owner
Copy Link
Copy Source

@Sachin-Bhat commented on GitHub (Apr 24, 2024):

Hey @eyurtsev,

I have tried to replicate the langchain-community indexed documents table schema to match that of langchain-postgres as closely as possible. I do still get some errors:

Error indexing document with id 000134955 and chunk_id 67cf6d54-7885-47ab-a42f-91367c070fd7: (psycopg2.errors.NotNullViolation) null value in column "uuid" of relation "langchain_pg_embedding" violates not-null constraint
DETAIL:  Failing row contains (058cabb5-a81a-4fee-b77c-3a3acdc04454, [0.019579852,-0.03575338,-0.01300123,0.0077682347,0.020762963,-0..., lorem ipsum ..., {"id": "67cf6d54-7885-47ab-a42f-91367c070fd7", "Ack": "", "CveId..., 67cf6d54-7885-47ab-a42f-91367c070fd7, null).

[SQL: INSERT INTO langchain_pg_embedding (id, collection_id, embedding, document, cmetadata) VALUES (%(id_m0)s, %(collection_id_m0)s::UUID, %(embedding_m0)s, %(document_m0)s, %(cmetadata_m0)s) ON CONFLICT (id) DO UPDATE SET embedding = excluded.embedding, document = excluded.document, cmetadata = excluded.cmetadata]
[parameters: {'id_m0': '67cf6d54-7885-47ab-a42f-91367c070fd7', 'collection_id_m0': UUID('058cabb5-a81a-4fee-b77c-3a3acdc04454'), 'embedding_m0': '[0.019579852,-0.03575338,-0.01300123,0.0077682347,0.020762963,-0.018955793,0.016758585,0.015887503,0.014431365,0.043164082,-0.05083481,0.036741473,0. ... (12432 characters truncated) ... .040121794,-0.0029317772,0.023233198,-0.013586285,-0.043216087,-0.03837963,0.033075128,0.036013406,0.034115225,-0.023662237,-0.03458327,0.0066761314]', 'document_m0': 'lorem ipsum sit amet.\n', 'cmetadata_m0': 'metadata items'}]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

Any help here would be much appreciated.

Cheers,
Sachin

@Sachin-Bhat commented on GitHub (Apr 24, 2024): Hey @eyurtsev, I have tried to replicate the `langchain-community` indexed documents table schema to match that of `langchain-postgres` as closely as possible. I do still get some errors: ``` Error indexing document with id 000134955 and chunk_id 67cf6d54-7885-47ab-a42f-91367c070fd7: (psycopg2.errors.NotNullViolation) null value in column "uuid" of relation "langchain_pg_embedding" violates not-null constraint DETAIL: Failing row contains (058cabb5-a81a-4fee-b77c-3a3acdc04454, [0.019579852,-0.03575338,-0.01300123,0.0077682347,0.020762963,-0..., lorem ipsum ..., {"id": "67cf6d54-7885-47ab-a42f-91367c070fd7", "Ack": "", "CveId..., 67cf6d54-7885-47ab-a42f-91367c070fd7, null). [SQL: INSERT INTO langchain_pg_embedding (id, collection_id, embedding, document, cmetadata) VALUES (%(id_m0)s, %(collection_id_m0)s::UUID, %(embedding_m0)s, %(document_m0)s, %(cmetadata_m0)s) ON CONFLICT (id) DO UPDATE SET embedding = excluded.embedding, document = excluded.document, cmetadata = excluded.cmetadata] [parameters: {'id_m0': '67cf6d54-7885-47ab-a42f-91367c070fd7', 'collection_id_m0': UUID('058cabb5-a81a-4fee-b77c-3a3acdc04454'), 'embedding_m0': '[0.019579852,-0.03575338,-0.01300123,0.0077682347,0.020762963,-0.018955793,0.016758585,0.015887503,0.014431365,0.043164082,-0.05083481,0.036741473,0. ... (12432 characters truncated) ... .040121794,-0.0029317772,0.023233198,-0.013586285,-0.043216087,-0.03837963,0.033075128,0.036013406,0.034115225,-0.023662237,-0.03458327,0.0066761314]', 'document_m0': 'lorem ipsum sit amet.\n', 'cmetadata_m0': 'metadata items'}] (Background on this error at: https://sqlalche.me/e/20/gkpj) ``` Any help here would be much appreciated. Cheers, Sachin
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langchain-postgres#1
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?