Metadata field not properly deserialized when using async_mode=True with PGVector #46

New Issue

2026-02-16T05:16:17-05:00

yindo commented

2026-02-16 05:16:17 -05:00

Originally created by @shamspias on GitHub (Oct 8, 2024).

When using the PGVector class with async_mode=True, the metadata field of the Document objects returned from query methods (e.g., asimilarity_search_with_score_by_vector) is not deserialized into a Python dict. Instead, it remains as a Fragment object or another non-dict type. This causes a ValidationError when the Document class expects metadata to be a dictionary.

To Reproduce

Steps to reproduce the behavior:

Initialize PGVector with async_mode=True and use_jsonb=True.
Add documents to the vector store with metadata.
Perform an asynchronous similarity search, e.g., asimilarity_search or asimilarity_search_with_score_by_vector.
Observe that the returned Document objects have metadata fields that are not dictionaries.

Expected behavior

The metadata field of the returned Document objects should be properly deserialized into Python dictionaries, matching the behavior when async_mode=False.

Actual behavior

When async_mode=True, the metadata field is a Fragment object (from asyncpg), leading to errors when the code expects a dict.

Error message

ValidationError: 1 validation error for Document
metadata
  Input should be a valid dictionary [type=dict_type, input_value=Fragment(buf=b'{"user_id": "ahmed"}'), input_type=Fragment]

Environment:

langchain_postgres version: 0.0.12
Python version: 10,11,12
Database: PostgreSQL with pgvector extension
Async driver: asyncpg

Additional context

This issue arises because asyncpg returns JSONB fields as Record or Fragment objects, which are not automatically deserialized into Python dictionaries by SQLAlchemy when using asynchronous sessions.

Code to Reproduce

Ensure that the required connection details like connection_string, collection_name, and embedding_model are securely provided when testing the code.

from langchain_postgres.vectorstores import PGVector

# Setup the connection to PGVector
connection_string = 'your_connection_string_here'
collection_name = 'your_collection_name_here'
embedding_model = 'your_embedding_model_here'

# Initialize PGVector with the necessary parameters
vstore = PGVector(
    connection=connection_string,
    collection_name=collection_name,
    embeddings=embedding_model,
    use_jsonb=True,
    pre_delete_collection=False,
    async_mode=True  # Set to True to reproduce the issue
)

# Add a document with metadata
vstore.add_document({"user_id": "ahmed"}, metadata={"data": "example"})

# Perform an asynchronous similarity search
result = vstore.asimilarity_search_with_score_by_vector()
print(result.metadata)  # The issue: metadata is not returned as a dictionary

Proposed Solution

Modify the _results_to_docs_and_scores method in the PGVector class to ensure that the metadata field is correctly converted into a dictionary before creating the Document objects.

Related Issues:
#118

Originally created by @shamspias on GitHub (Oct 8, 2024). When using the `PGVector` class with `async_mode=True`, the `metadata` field of the `Document` objects returned from query methods (e.g., `asimilarity_search_with_score_by_vector`) is not deserialized into a Python `dict`. Instead, it remains as a `Fragment` object or another non-dict type. This causes a `ValidationError` when the `Document` class expects `metadata` to be a dictionary. **To Reproduce** Steps to reproduce the behavior: 1. Initialize `PGVector` with `async_mode=True` and `use_jsonb=True`. 2. Add documents to the vector store with metadata. 3. Perform an asynchronous similarity search, e.g., `asimilarity_search` or `asimilarity_search_with_score_by_vector`. 4. Observe that the returned `Document` objects have `metadata` fields that are not dictionaries. **Expected behavior** The `metadata` field of the returned `Document` objects should be properly deserialized into Python dictionaries, matching the behavior when `async_mode=False`. **Actual behavior** When `async_mode=True`, the `metadata` field is a `Fragment` object (from `asyncpg`), leading to errors when the code expects a `dict`. **Error message** ``` ValidationError: 1 validation error for Document metadata Input should be a valid dictionary [type=dict_type, input_value=Fragment(buf=b'{"user_id": "ahmed"}'), input_type=Fragment] ``` **Environment:** - `langchain_postgres` version: 0.0.12 - Python version: 10,11,12 - Database: PostgreSQL with `pgvector` extension - Async driver: `asyncpg` **Additional context** This issue arises because `asyncpg` returns JSONB fields as `Record` or `Fragment` objects, which are not automatically deserialized into Python dictionaries by SQLAlchemy when using asynchronous sessions. **Code to Reproduce** Ensure that the required connection details like `connection_string`, `collection_name`, and `embedding_model` are securely provided when testing the code. ```python from langchain_postgres.vectorstores import PGVector # Setup the connection to PGVector connection_string = 'your_connection_string_here' collection_name = 'your_collection_name_here' embedding_model = 'your_embedding_model_here' # Initialize PGVector with the necessary parameters vstore = PGVector( connection=connection_string, collection_name=collection_name, embeddings=embedding_model, use_jsonb=True, pre_delete_collection=False, async_mode=True # Set to True to reproduce the issue ) # Add a document with metadata vstore.add_document({"user_id": "ahmed"}, metadata={"data": "example"}) # Perform an asynchronous similarity search result = vstore.asimilarity_search_with_score_by_vector() print(result.metadata) # The issue: metadata is not returned as a dictionary ``` **Proposed Solution** Modify the `_results_to_docs_and_scores` method in the `PGVector` class to ensure that the `metadata` field is correctly converted into a dictionary before creating the `Document` objects. Related Issues: #118

yindo commented

2026-02-16 05:16:17 -05:00

@simadimonyan commented on GitHub (Nov 11, 2024):

Did you fix it? I have related issue: https://github.com/langchain-ai/langchain/issues/28029

@simadimonyan commented on GitHub (Nov 11, 2024): Did you fix it? I have related issue: https://github.com/langchain-ai/langchain/issues/28029

yindo commented

2026-02-16 05:16:17 -05:00

@shamspias commented on GitHub (Nov 28, 2024):

Did you fix it? I have related issue: langchain-ai/langchain#28029

already fixed in
https://github.com/langchain-ai/langchain-postgres/pull/125

@shamspias commented on GitHub (Nov 28, 2024): > Did you fix it? I have related issue: [langchain-ai/langchain#28029](https://github.com/langchain-ai/langchain/issues/28029) already fixed in https://github.com/langchain-ai/langchain-postgres/pull/125

yindo commented

2026-02-16 05:16:17 -05:00

@eyurtsev commented on GitHub (Feb 7, 2025):

I can see other users are encountering this issue, but we don't have a way to reproduce currently.

One thing I'm noticing is that the this is using asyncpg as the driver, but the package is only meant to work with psycopg3 right now.

A unit test that shows how to reproduce is required

@eyurtsev commented on GitHub (Feb 7, 2025): I can see other users are encountering this issue, but we don't have a way to reproduce currently. One thing I'm noticing is that the this is using asyncpg as the driver, but the package is only meant to work with psycopg3 right now. A unit test that shows how to reproduce is required

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/langchain-postgres#46