[PR #272] [MERGED] Feature: Support metadata based filtering for delete operations #273

New Issue

Closed

opened 2026-02-16 05:17:12 -05:00 by yindo · 0 comments

yindo commented 2026-02-16 05:17:12 -05:00

Owner

Copy Link

Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/langchain-postgres/pull/272
Author: @yukiharada1228
Created: 12/17/2025
Status: ✅ Merged
Merged: 1/21/2026
Merged by: @dishaprakash

Base: main ← Head: support-metadata-based-filtering-for-delete-operations

---

📝 Commits (9)

• c2c8add Update README.md
• 711714d Merge branch 'langchain-ai:main' into main
• 09a38de Add metadata-based filtering support for delete operations
• 1e076de Update documentation and tests for metadata-based filtering in delete operations
• ec11d4b Add filter parameter and documentation to delete method in AsyncPGVectorStore
• 882c0d7 Add metadata filter examples to delete operations in notebook
• b0b5e20 Format code: adjust line breaks and simplify notebook examples
• e6ac1be Merge branch 'main' into support-metadata-based-filtering-for-delete-operations
• 8d1aa1e Merge branch 'main' into support-metadata-based-filtering-for-delete-operations

📊 Changes

6 files changed (+479 additions, -18 deletions)

View changed files

📝 examples/pg_vectorstore_how_to.ipynb (+62 -11)
📝 langchain_postgres/v2/async_vectorstore.py (+72 -4)
📝 langchain_postgres/v2/vectorstores.py (+50 -2)
📝 tests/unit_tests/v2/test_async_pg_vectorstore.py (+204 -0)
📝 tests/unit_tests/v2/test_pg_vectorstore.py (+78 -0)
📝 uv.lock (+13 -1)

📄 Description

Add metadata-based filtering support for delete operations

Summary

This PR adds support for metadata-based filtering in delete() and adelete() methods, enabling bulk deletion of documents based on metadata criteria rather than just by IDs.

Motivation

Currently, the delete methods only support deletion by document IDs, which is limiting for common use cases:

• Bulk deletions: Users often need to delete groups of documents based on metadata (e.g., all documents from a specific source, time period, or category)
• Data lifecycle management: Remove documents based on status, expiration dates, or other metadata flags
• Cleanup operations: Delete documents matching specific criteria without knowing their IDs

Other vector stores (Chroma, Pinecone, Weaviate) already support metadata-based deletion, and the infrastructure for metadata filtering already exists in this codebase via the _create_filter_clause() method.

Changes

Modified Files

1. langchain_postgres/v2/async_vectorstore.py

   • Enhanced adelete() method to accept optional filter parameter
   • Supports deletion by IDs, filter, or both (combined with AND logic)
   • Leverages existing _create_filter_clause() for consistent filter syntax
   • Updated docstring with comprehensive examples and important limitations

2. langchain_postgres/v2/vectorstores.py

   • Updated both adelete() and delete() methods to accept filter parameter
   • Sync wrapper properly passes filter to async implementation
   • Updated docstrings with examples and limitations

3. Test files

   • Added 7 new test cases covering various filtering scenarios
   • Tests for simple filters, operators, complex filters, combined ID+filter, and edge cases
   • All tests use metadata_columns to ensure proper filtering behavior

Usage Examples

Setup: Define metadata columns

# First, create vectorstore with metadata columns
vectorstore = await AsyncPGVectorStore.create(
    engine,
    embedding_service=embeddings_service,
    table_name="my_documents",
    metadata_columns=["source", "category", "year", "status"],  # Define filterable fields
)

Delete by metadata filter only

# Delete all documents from a specific source
await vectorstore.adelete(filter={"source": "documentation"})

# Delete documents with numeric comparisons
await vectorstore.adelete(filter={"year": {"$lt": 2020}})

# Delete with complex filters
await vectorstore.adelete(
    filter={"$and": [{"category": "obsolete"}, {"status": "archived"}]}
)

Delete by IDs only (existing behavior)

await vectorstore.adelete(ids=["id1", "id2", "id3"])

Delete by both IDs and filter (must match both criteria)

# Only deletes documents that match BOTH the ID list AND the filter
await vectorstore.adelete(
    ids=["id1", "id2", "id3"],
    filter={"status": "archived"}
)

Sync methods work identically

# Sync version
vectorstore.delete(filter={"source": "deprecated"})

Filter Syntax

The filter parameter supports the same rich filtering syntax as similarity_search():

• Equality: {"field": "value"}
• Comparison operators: {"field": {"$lt": 100}} ($eq, $ne, $lt, $lte, $gt, $gte)
• List operators: {"field": {"$in": [1, 2, 3]}} ($in, $nin)
• Text operators: {"field": {"$like": "pattern%"}} ($like, $ilike)
• Logical operators: {"$and": [...]}, {"$or": [...]}, {"$not": {...}}
• Existence: {"field": {"$exists": True}}
• Range: {"field": {"$between": [10, 20]}}

⚠️ Important Limitation

Filters only work on fields defined in metadata_columns, not on fields stored in metadata_json_column.

This is consistent with how similarity_search() filtering works. To use metadata-based deletion, you must define the metadata fields as actual database columns when creating the vectorstore:

# ✅ Correct: Define metadata columns
vectorstore = await AsyncPGVectorStore.create(
    engine,
    embedding_service=embeddings_service,
    table_name="my_table",
    metadata_columns=["source", "category", "year"],  # These fields can be filtered
)

# Now you can filter on these columns
await vectorstore.adelete(filter={"source": "documentation"})

# ❌ Won't work: Fields only in metadata_json_column cannot be filtered
vectorstore = await AsyncPGVectorStore.create(
    engine,
    embedding_service=embeddings_service,
    table_name="my_table",
    # No metadata_columns defined - all metadata goes to JSON column
)

# This will fail - "source" is not a database column
await vectorstore.adelete(filter={"source": "documentation"})

Fields stored only in metadata_json_column cannot be used in filters. This design choice provides better query performance and leverages PostgreSQL's native indexing capabilities.

Implementation Details

• Backward compatible: Existing code using adelete(ids=[...]) continues to work unchanged
• SQL injection safe: Uses parameterized queries via existing _create_filter_clause() method
• Consistent behavior: Filter syntax matches similarity_search() for consistency
• Performance: Generates efficient SQL DELETE statements with WHERE clauses on indexed columns
• Type safe: Full type hints and passes mypy strict checking

Test Coverage

Added comprehensive test coverage (all tests passing):

• ✅ test_adelete_with_filter: Basic metadata filter deletion
• ✅ test_adelete_with_filter_and_operator: Deletion with comparison operators
• ✅ test_adelete_with_complex_filter: Complex filters with logical operators
• ✅ test_adelete_with_filter_and_ids: Combined ID and filter deletion
• ✅ test_adelete_with_filter_no_matches: Graceful handling of no matches
• ✅ test_adelete_with_filter (sync): Async method in sync wrapper
• ✅ test_delete_with_filter (sync): Sync method filtering
• ✅ test_adelete: Existing tests continue to pass

All tests follow existing patterns and integrate with the current test suite.

Breaking Changes

None. This is a backward-compatible enhancement:

• All existing code continues to work unchanged
• New filter parameter is optional
• Default behavior (no parameters) remains the same

Checklist

• Implementation complete for async methods
• Implementation complete for sync methods
• Comprehensive test coverage added (7 tests, all passing)
• Code passes ruff linting
• Code passes mypy type checking
• Docstrings updated with examples and limitations
• Backward compatible with existing code
• Follows existing codebase patterns

Related Issues

Closes #271

Additional Notes

Why metadata_columns are required for filtering

This implementation reuses the robust _create_filter_clause() method that's already extensively tested for search operations. The method generates SQL WHERE clauses that operate on actual database columns, which provides:

1. Better performance: Direct column filtering is faster than JSON field extraction
2. Index support: Metadata columns can be indexed for even better performance
3. Type safety: Database column types ensure type correctness
4. Consistency: Same behavior as similarity_search() filtering

This design is consistent with the existing filtering implementation and aligns with how other parts of the codebase handle metadata filtering.

---

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/langchain-postgres/pull/272 **Author:** [@yukiharada1228](https://github.com/yukiharada1228) **Created:** 12/17/2025 **Status:** ✅ Merged **Merged:** 1/21/2026 **Merged by:** [@dishaprakash](https://github.com/dishaprakash) **Base:** `main` ← **Head:** `support-metadata-based-filtering-for-delete-operations` --- ### 📝 Commits (9) - [`c2c8add`](https://github.com/langchain-ai/langchain-postgres/commit/c2c8add875c7a1500971eb7189ed0cfd9992420b) Update README.md - [`711714d`](https://github.com/langchain-ai/langchain-postgres/commit/711714d9738e38ccb6d9bb61703fa24bcfba79c8) Merge branch 'langchain-ai:main' into main - [`09a38de`](https://github.com/langchain-ai/langchain-postgres/commit/09a38de7580b4e19c6cd5f00d651b4d4d8e81f23) Add metadata-based filtering support for delete operations - [`1e076de`](https://github.com/langchain-ai/langchain-postgres/commit/1e076de660609db4edd3dc237c5b928dff11f987) Update documentation and tests for metadata-based filtering in delete operations - [`ec11d4b`](https://github.com/langchain-ai/langchain-postgres/commit/ec11d4b2d06346199b336dc1a74d7584fd09f838) Add filter parameter and documentation to delete method in AsyncPGVectorStore - [`882c0d7`](https://github.com/langchain-ai/langchain-postgres/commit/882c0d7512e62a5252327ceb8aacb0316aca1711) Add metadata filter examples to delete operations in notebook - [`b0b5e20`](https://github.com/langchain-ai/langchain-postgres/commit/b0b5e203f3115633b789895515356c1f4fbdd647) Format code: adjust line breaks and simplify notebook examples - [`e6ac1be`](https://github.com/langchain-ai/langchain-postgres/commit/e6ac1be3034bb9ecf5859cd859f4ce515eed3eaf) Merge branch 'main' into support-metadata-based-filtering-for-delete-operations - [`8d1aa1e`](https://github.com/langchain-ai/langchain-postgres/commit/8d1aa1ec8b00a79a3ed8d6b7ce813e2c620fbafc) Merge branch 'main' into support-metadata-based-filtering-for-delete-operations ### 📊 Changes **6 files changed** (+479 additions, -18 deletions) <details> <summary>View changed files</summary> 📝 `examples/pg_vectorstore_how_to.ipynb` (+62 -11) 📝 `langchain_postgres/v2/async_vectorstore.py` (+72 -4) 📝 `langchain_postgres/v2/vectorstores.py` (+50 -2) 📝 `tests/unit_tests/v2/test_async_pg_vectorstore.py` (+204 -0) 📝 `tests/unit_tests/v2/test_pg_vectorstore.py` (+78 -0) 📝 `uv.lock` (+13 -1) </details> ### 📄 Description # Add metadata-based filtering support for delete operations ## Summary This PR adds support for metadata-based filtering in `delete()` and `adelete()` methods, enabling bulk deletion of documents based on metadata criteria rather than just by IDs. ## Motivation Currently, the delete methods only support deletion by document IDs, which is limiting for common use cases: - **Bulk deletions**: Users often need to delete groups of documents based on metadata (e.g., all documents from a specific source, time period, or category) - **Data lifecycle management**: Remove documents based on status, expiration dates, or other metadata flags - **Cleanup operations**: Delete documents matching specific criteria without knowing their IDs Other vector stores (Chroma, Pinecone, Weaviate) already support metadata-based deletion, and the infrastructure for metadata filtering already exists in this codebase via the `_create_filter_clause()` method. ## Changes ### Modified Files 1. **`langchain_postgres/v2/async_vectorstore.py`** - Enhanced `adelete()` method to accept optional `filter` parameter - Supports deletion by IDs, filter, or both (combined with AND logic) - Leverages existing `_create_filter_clause()` for consistent filter syntax - Updated docstring with comprehensive examples and important limitations 2. **`langchain_postgres/v2/vectorstores.py`** - Updated both `adelete()` and `delete()` methods to accept `filter` parameter - Sync wrapper properly passes filter to async implementation - Updated docstrings with examples and limitations 3. **Test files** - Added 7 new test cases covering various filtering scenarios - Tests for simple filters, operators, complex filters, combined ID+filter, and edge cases - All tests use `metadata_columns` to ensure proper filtering behavior ## Usage Examples ### Setup: Define metadata columns ```python # First, create vectorstore with metadata columns vectorstore = await AsyncPGVectorStore.create( engine, embedding_service=embeddings_service, table_name="my_documents", metadata_columns=["source", "category", "year", "status"], # Define filterable fields ) ``` ### Delete by metadata filter only ```python # Delete all documents from a specific source await vectorstore.adelete(filter={"source": "documentation"}) # Delete documents with numeric comparisons await vectorstore.adelete(filter={"year": {"$lt": 2020}}) # Delete with complex filters await vectorstore.adelete( filter={"$and": [{"category": "obsolete"}, {"status": "archived"}]} ) ``` ### Delete by IDs only (existing behavior) ```python await vectorstore.adelete(ids=["id1", "id2", "id3"]) ``` ### Delete by both IDs and filter (must match both criteria) ```python # Only deletes documents that match BOTH the ID list AND the filter await vectorstore.adelete( ids=["id1", "id2", "id3"], filter={"status": "archived"} ) ``` ### Sync methods work identically ```python # Sync version vectorstore.delete(filter={"source": "deprecated"}) ``` ## Filter Syntax The `filter` parameter supports the same rich filtering syntax as `similarity_search()`: - **Equality**: `{"field": "value"}` - **Comparison operators**: `{"field": {"$lt": 100}}` ($eq, $ne, $lt, $lte, $gt, $gte) - **List operators**: `{"field": {"$in": [1, 2, 3]}}` ($in, $nin) - **Text operators**: `{"field": {"$like": "pattern%"}}` ($like, $ilike) - **Logical operators**: `{"$and": [...]}`, `{"$or": [...]}`, `{"$not": {...}}` - **Existence**: `{"field": {"$exists": True}}` - **Range**: `{"field": {"$between": [10, 20]}}` ### ⚠️ Important Limitation **Filters only work on fields defined in `metadata_columns`, not on fields stored in `metadata_json_column`.** This is consistent with how `similarity_search()` filtering works. To use metadata-based deletion, you must define the metadata fields as actual database columns when creating the vectorstore: ```python # ✅ Correct: Define metadata columns vectorstore = await AsyncPGVectorStore.create( engine, embedding_service=embeddings_service, table_name="my_table", metadata_columns=["source", "category", "year"], # These fields can be filtered ) # Now you can filter on these columns await vectorstore.adelete(filter={"source": "documentation"}) ``` ```python # ❌ Won't work: Fields only in metadata_json_column cannot be filtered vectorstore = await AsyncPGVectorStore.create( engine, embedding_service=embeddings_service, table_name="my_table", # No metadata_columns defined - all metadata goes to JSON column ) # This will fail - "source" is not a database column await vectorstore.adelete(filter={"source": "documentation"}) ``` Fields stored only in `metadata_json_column` cannot be used in filters. This design choice provides better query performance and leverages PostgreSQL's native indexing capabilities. ## Implementation Details - **Backward compatible**: Existing code using `adelete(ids=[...])` continues to work unchanged - **SQL injection safe**: Uses parameterized queries via existing `_create_filter_clause()` method - **Consistent behavior**: Filter syntax matches `similarity_search()` for consistency - **Performance**: Generates efficient SQL DELETE statements with WHERE clauses on indexed columns - **Type safe**: Full type hints and passes mypy strict checking ## Test Coverage Added comprehensive test coverage (all tests passing): - ✅ `test_adelete_with_filter`: Basic metadata filter deletion - ✅ `test_adelete_with_filter_and_operator`: Deletion with comparison operators - ✅ `test_adelete_with_complex_filter`: Complex filters with logical operators - ✅ `test_adelete_with_filter_and_ids`: Combined ID and filter deletion - ✅ `test_adelete_with_filter_no_matches`: Graceful handling of no matches - ✅ `test_adelete_with_filter` (sync): Async method in sync wrapper - ✅ `test_delete_with_filter` (sync): Sync method filtering - ✅ `test_adelete`: Existing tests continue to pass All tests follow existing patterns and integrate with the current test suite. ## Breaking Changes None. This is a backward-compatible enhancement: - All existing code continues to work unchanged - New `filter` parameter is optional - Default behavior (no parameters) remains the same ## Checklist - [x] Implementation complete for async methods - [x] Implementation complete for sync methods - [x] Comprehensive test coverage added (7 tests, all passing) - [x] Code passes `ruff` linting - [x] Code passes `mypy` type checking - [x] Docstrings updated with examples and limitations - [x] Backward compatible with existing code - [x] Follows existing codebase patterns ## Related Issues Closes #271 ## Additional Notes ### Why metadata_columns are required for filtering This implementation reuses the robust `_create_filter_clause()` method that's already extensively tested for search operations. The method generates SQL WHERE clauses that operate on actual database columns, which provides: 1. **Better performance**: Direct column filtering is faster than JSON field extraction 2. **Index support**: Metadata columns can be indexed for even better performance 3. **Type safety**: Database column types ensure type correctness 4. **Consistency**: Same behavior as `similarity_search()` filtering This design is consistent with the existing filtering implementation and aligns with how other parts of the codebase handle metadata filtering. --- _{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

yindo added the pull-request label 2026-02-16 05:17:12 -05:00

yindo closed this issue 2026-02-16 05:17:12 -05:00

yindo referenced this issue 2026-02-16 05:17:12 -05:00

[PR #273] chore: add deprecation warning to PGVector #275

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

langster/harden-dependabot-workflows

v0.3

dep-warning

upstream_add

private-variable

eugene/remove_deprecation_for_now

eugene/tutorial_rework

eugene/update_poetry_lock-file

eyurtsev-patch-1

eugene/document_inedxer

eugene/relax_langgraph

eugene/langchain_postgres

langchain-postgres==0.0.17

langchain-postgres==0.0.16

langchain-postgres==0.0.15

v0.0.14

v0.0.14rc1

v0.0.13

v0.0.12

v0.0.11

v0.0.10

v0.0.9

v0.0.8

v0.0.7

v0.0.6

v0.0.5

v0.0.4

v0.0.3

v0.0.2

v0.0.1

Labels

Clear labels

bug

enhancement

help wanted

pull-request

Mirrored from GitHub Pull Request

No Label pull-request

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

yindo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/langchain-postgres#273