langchain-ai/langchain-postgres
1
0
Fork 0
mirror of https://github.com/langchain-ai/langchain-postgres.git synced 2026-07-16 01:33:18 -04:00
Code Issues 85 Packages Projects Releases 11 Wiki Activity

AsyncPGVectorStore Metadata Filtering Not Working #91

New Issue
Closed
opened 2026-02-16 05:16:31 -05:00 by yindo · 7 comments
yindo commented 2026-02-16 05:16:31 -05:00
Owner
Copy Link
Copy Source

Originally created by @bharatht19 on GitHub (Aug 31, 2025).

Originally assigned to: @dishaprakash on GitHub.

I am seeing the below error while using meta data filtering as below .Please suggest.

Code

async def _build_chain():
    store = await get_vector_store()
    FILTER = {'source': {"$eq":"data/handbooks/employee_handbook_2025.docx"}}
    
    retriever = store.as_retriever(
    search_kwargs={
        "filter": FILTER
    }
)

DB Table

CREATE TABLE IF NOT EXISTS langchain_pg_embedding (
langchain_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), -- Auto-generate UUIDs
content text, -- Raw text chunk
embedding vector(1536), -- Vector dimension matches embedding model
langchain_metadata jsonb -- Metadata
);

Error:

cka-app | sqlalchemy.exc.ProgrammingError: (sqlalchemy.dialects.postgresql.asyncpg.ProgrammingError) <class 'asyncpg.exceptions.UndefinedColumnError'>: column "source" does not exist
cka-app | [SQL: SELECT "langchain_id", "content", "embedding", "langchain_metadata", cosine_distance("embedding", $1) as distance
cka-app | FROM "public"."langchain_pg_embedding" WHERE source = $2 ORDER BY "embedding" <=> $1 LIMIT $3;
cka-app | ]
cka-app | [parameters: ('[0.019000347703695297, 0.008494899608194828, -0.0014854423934593797, -0.008188657462596893, -0.03637627884745598, -0.04138267785310745, -0.0131018515 ... (33892 characters truncated) ... 6108424216508865, -0.023460835218429565, 0.022315755486488342, 0.001256592688150704, 0.0203052069991827, 0.017788691446185112, -0.020238632336258888]', 'data/handbooks/employee_handbook_2025.docx', 4)]
cka-app | (Background on this error at: https://sqlalche.me/e/20/f405)

Originally created by @bharatht19 on GitHub (Aug 31, 2025). Originally assigned to: @dishaprakash on GitHub. I am seeing the below error while using meta data filtering as below .Please suggest. **Code** ``` async def _build_chain(): store = await get_vector_store() FILTER = {'source': {"$eq":"data/handbooks/employee_handbook_2025.docx"}} retriever = store.as_retriever( search_kwargs={ "filter": FILTER } ) ``` **DB Table** CREATE TABLE IF NOT EXISTS langchain_pg_embedding ( langchain_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), -- Auto-generate UUIDs content text, -- Raw text chunk embedding vector(1536), -- Vector dimension matches embedding model langchain_metadata jsonb -- Metadata ); **Error:** cka-app | sqlalchemy.exc.ProgrammingError: (sqlalchemy.dialects.postgresql.asyncpg.ProgrammingError) <class 'asyncpg.exceptions.UndefinedColumnError'>: column "source" does not exist cka-app | [SQL: SELECT "langchain_id", "content", "embedding", "langchain_metadata", cosine_distance("embedding", $1) as distance cka-app | FROM "public"."langchain_pg_embedding" WHERE source = $2 ORDER BY "embedding" <=> $1 LIMIT $3; cka-app | ] cka-app | [parameters: ('[0.019000347703695297, 0.008494899608194828, -0.0014854423934593797, -0.008188657462596893, -0.03637627884745598, -0.04138267785310745, -0.0131018515 ... (33892 characters truncated) ... 6108424216508865, -0.023460835218429565, 0.022315755486488342, 0.001256592688150704, 0.0203052069991827, 0.017788691446185112, -0.020238632336258888]', 'data/handbooks/employee_handbook_2025.docx', 4)] cka-app | (Background on this error at: https://sqlalche.me/e/20/f405)
yindo added the enhancement label 2026-02-16 05:16:31 -05:00
yindo closed this issue 2026-02-16 05:16:31 -05:00
yindo commented 2026-02-16 05:16:31 -05:00
Author
Owner
Copy Link
Copy Source

@averikitsch commented on GitHub (Sep 4, 2025):

Hi @bharatht19 can you share the code behind function get_vector_store()?

@averikitsch commented on GitHub (Sep 4, 2025): Hi @bharatht19 can you share the code behind function `get_vector_store()`?
yindo commented 2026-02-16 05:16:31 -05:00
Author
Owner
Copy Link
Copy Source

@bharatht19 commented on GitHub (Sep 5, 2025):

@averikitsch - below is the code

import os
from langchain_openai import OpenAIEmbeddings
from langchain_postgres.v2.engine import PGEngine
from langchain_postgres.v2.async_vectorstore import AsyncPGVectorStore

PG_CONN_STR = os.getenv("DATABASE_URL")

PG_ENGINE = PGEngine.from_connection_string(PG_CONN_STR)

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")


async def get_vector_store()->AsyncPGVectorStore:
    return await AsyncPGVectorStore.create(
        engine=PG_ENGINE,
        embedding_service=embeddings,
        table_name="langchain_pg_embedding",
        metadata_json_column="langchain_metadata"
    )
@bharatht19 commented on GitHub (Sep 5, 2025): @averikitsch - below is the code ``` import os from langchain_openai import OpenAIEmbeddings from langchain_postgres.v2.engine import PGEngine from langchain_postgres.v2.async_vectorstore import AsyncPGVectorStore PG_CONN_STR = os.getenv("DATABASE_URL") PG_ENGINE = PGEngine.from_connection_string(PG_CONN_STR) embeddings = OpenAIEmbeddings(model="text-embedding-3-small") async def get_vector_store()->AsyncPGVectorStore: return await AsyncPGVectorStore.create( engine=PG_ENGINE, embedding_service=embeddings, table_name="langchain_pg_embedding", metadata_json_column="langchain_metadata" ) ```
yindo commented 2026-02-16 05:16:31 -05:00
Author
Owner
Copy Link
Copy Source

@bharatht19 commented on GitHub (Sep 7, 2025):

@averikitsch Did you get a chance to look in to this.This is stopping me from using metadata in my RAG pipeline. Is this a bug?

@bharatht19 commented on GitHub (Sep 7, 2025): @averikitsch Did you get a chance to look in to this.This is stopping me from using metadata in my RAG pipeline. Is this a bug?
yindo commented 2026-02-16 05:16:31 -05:00
Author
Owner
Copy Link
Copy Source

@dishaprakash commented on GitHub (Sep 10, 2025):

Hi @bharatht19
PGVectorStore does not support filtering metadata stored in a JSONB column. For filtering to work, each metadata field must have its own separate column. This approach is designed to make filtering quicker and efficient in Postgres.

As a workaround, you can

  • Mention the metadata column during the VectorStore initialization before adding the data and make sure the JSON key's value is being added into the metadata column OR

  • Use SQL to ALTER your already existing table by adding a new column for the JSON key you want to filter and copying the data into it.
    Please see heading Search for documents with JSON Filter in this how-to guide.

Thank you for opening this issue, we will update the PGVectorStore documentation to reflect the same!

@dishaprakash commented on GitHub (Sep 10, 2025): Hi @bharatht19 PGVectorStore does not support filtering metadata stored in a JSONB column. For filtering to work, each metadata field must have its own separate column. This approach is designed to make filtering quicker and efficient in Postgres. As a workaround, you can - Mention the metadata column during the VectorStore initialization before adding the data and make sure the JSON key's value is being added into the metadata column OR - Use SQL to ALTER your already existing table by adding a new column for the JSON key you want to filter and copying the data into it. Please see heading `Search for documents with JSON Filter` in this [how-to guide](https://github.com/googleapis/langchain-google-alloydb-pg-python/blob/main/docs/vector_store.ipynb). Thank you for opening this issue, we will update the PGVectorStore documentation to reflect the same!
yindo commented 2026-02-16 05:16:31 -05:00
Author
Owner
Copy Link
Copy Source

@mrodriguez2 commented on GitHub (Sep 12, 2025):

What's the reasoning behind removing the JSON filtering in the new PGVectorStore class? It's a pretty breaking change with any current Langchain implementation utilizing PGVector.

Langchain4j's PGVector implementation, for example, supports metadataStorageConfig with the following parameters:

  • COLUMN_PER_KEY
  • COMBINED_JSON
  • COMBINED_JSONB

Maybe this would be a better approach?

@mrodriguez2 commented on GitHub (Sep 12, 2025): What's the reasoning behind removing the JSON filtering in the new PGVectorStore class? It's a pretty breaking change with any current Langchain implementation utilizing PGVector. Langchain4j's PGVector [implementation](https://docs.langchain4j.dev/integrations/embedding-stores/pgvector), for example, supports metadataStorageConfig with the following parameters: - COLUMN_PER_KEY - COMBINED_JSON - COMBINED_JSONB Maybe this would be a better approach?
yindo commented 2026-02-16 05:16:31 -05:00
Author
Owner
Copy Link
Copy Source

@dishaprakash commented on GitHub (Sep 17, 2025):

Our current design only permits filtering on the dedicated metadata columns, not on the JSONB field itself. This is a deliberate choice for performance, as filtering on standard columns are much more efficient in a PostgreSQL database. We recommend converting any filterable metadata from a JSONB field to a standard column.

This issue will be marked as a feature request for JSONB filtering in PGVectorStore

@dishaprakash commented on GitHub (Sep 17, 2025): Our current design only permits filtering on the dedicated metadata columns, not on the JSONB field itself. This is a deliberate choice for performance, as filtering on standard columns are much more efficient in a PostgreSQL database. We recommend converting any filterable metadata from a JSONB field to a standard column. This issue will be marked as a feature request for JSONB filtering in PGVectorStore
yindo commented 2026-02-16 05:16:31 -05:00
Author
Owner
Copy Link
Copy Source

@rmlekus commented on GitHub (Nov 14, 2025):

@dishaprakash The flexibility and (in combination with JSONB indices created to optimize query execution plans for the older PGVector Databases) I did not see any performance impediments with a lot of use cases:

See also my comment in https://github.com/langchain-ai/langchain-postgres/issues/266#issuecomment-3534068632

IMHO with giving up the support for the 2-table layout with JSONB query support, this project looses a lot of attractiveness and advantages in comparison to other vector store solutions.

It might be some effort, but supporting both embedding/collection mapping solutions (selecting with a parameter dynamically when creating a new collection) might be a way to keep the flexibility of the older DDL mapping schema in parallel to the simplified maintenance for users who do not need 10**N collections to be dynamically generated and to be used in combined (SQL) queries with other database tables.?

@rmlekus commented on GitHub (Nov 14, 2025): @dishaprakash The flexibility and (in combination with JSONB indices created to optimize query execution plans for the older PGVector Databases) I did not see any performance impediments with a lot of use cases: See also my comment in https://github.com/langchain-ai/langchain-postgres/issues/266#issuecomment-3534068632 IMHO with giving up the support for the 2-table layout with JSONB query support, this project looses a lot of attractiveness and advantages in comparison to other vector store solutions. It might be some effort, but supporting both embedding/collection mapping solutions (selecting with a parameter dynamically when creating a new collection) might be a way to keep the flexibility of the older DDL mapping schema in parallel to the simplified maintenance for users who do not need 10**N collections to be dynamically generated and to be used in combined (SQL) queries with other database tables.?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
pull-request
Mirrored from GitHub Pull Request
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langchain-postgres#91
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?