mirror of
https://github.com/langchain-ai/langchain-postgres.git
synced 2026-07-16 09:54:28 -04:00
Yuki Harada
4738301643
# Add metadata-based filtering support for delete operations
## Summary
This PR adds support for metadata-based filtering in `delete()` and
`adelete()` methods, enabling bulk deletion of documents based on
metadata criteria rather than just by IDs.
## Motivation
Currently, the delete methods only support deletion by document IDs,
which is limiting for common use cases:
- **Bulk deletions**: Users often need to delete groups of documents
based on metadata (e.g., all documents from a specific source, time
period, or category)
- **Data lifecycle management**: Remove documents based on status,
expiration dates, or other metadata flags
- **Cleanup operations**: Delete documents matching specific criteria
without knowing their IDs
Other vector stores (Chroma, Pinecone, Weaviate) already support
metadata-based deletion, and the infrastructure for metadata filtering
already exists in this codebase via the `_create_filter_clause()`
method.
## Changes
### Modified Files
1. **`langchain_postgres/v2/async_vectorstore.py`**
- Enhanced `adelete()` method to accept optional `filter` parameter
- Supports deletion by IDs, filter, or both (combined with AND logic)
- Leverages existing `_create_filter_clause()` for consistent filter
syntax
- Updated docstring with comprehensive examples and important
limitations
2. **`langchain_postgres/v2/vectorstores.py`**
- Updated both `adelete()` and `delete()` methods to accept `filter`
parameter
- Sync wrapper properly passes filter to async implementation
- Updated docstrings with examples and limitations
3. **Test files**
- Added 7 new test cases covering various filtering scenarios
- Tests for simple filters, operators, complex filters, combined
ID+filter, and edge cases
- All tests use `metadata_columns` to ensure proper filtering behavior
## Usage Examples
### Setup: Define metadata columns
```python
# First, create vectorstore with metadata columns
vectorstore = await AsyncPGVectorStore.create(
engine,
embedding_service=embeddings_service,
table_name="my_documents",
metadata_columns=["source", "category", "year", "status"], # Define filterable fields
)
```
### Delete by metadata filter only
```python
# Delete all documents from a specific source
await vectorstore.adelete(filter={"source": "documentation"})
# Delete documents with numeric comparisons
await vectorstore.adelete(filter={"year": {"$lt": 2020}})
# Delete with complex filters
await vectorstore.adelete(
filter={"$and": [{"category": "obsolete"}, {"status": "archived"}]}
)
```
### Delete by IDs only (existing behavior)
```python
await vectorstore.adelete(ids=["id1", "id2", "id3"])
```
### Delete by both IDs and filter (must match both criteria)
```python
# Only deletes documents that match BOTH the ID list AND the filter
await vectorstore.adelete(
ids=["id1", "id2", "id3"],
filter={"status": "archived"}
)
```
### Sync methods work identically
```python
# Sync version
vectorstore.delete(filter={"source": "deprecated"})
```
## Filter Syntax
The `filter` parameter supports the same rich filtering syntax as
`similarity_search()`:
- **Equality**: `{"field": "value"}`
- **Comparison operators**: `{"field": {"$lt": 100}}` ($eq, $ne, $lt,
$lte, $gt, $gte)
- **List operators**: `{"field": {"$in": [1, 2, 3]}}` ($in, $nin)
- **Text operators**: `{"field": {"$like": "pattern%"}}` ($like, $ilike)
- **Logical operators**: `{"$and": [...]}`, `{"$or": [...]}`, `{"$not":
{...}}`
- **Existence**: `{"field": {"$exists": True}}`
- **Range**: `{"field": {"$between": [10, 20]}}`
### ⚠️ Important Limitation
**Filters only work on fields defined in `metadata_columns`, not on
fields stored in `metadata_json_column`.**
This is consistent with how `similarity_search()` filtering works. To
use metadata-based deletion, you must define the metadata fields as
actual database columns when creating the vectorstore:
```python
# ✅ Correct: Define metadata columns
vectorstore = await AsyncPGVectorStore.create(
engine,
embedding_service=embeddings_service,
table_name="my_table",
metadata_columns=["source", "category", "year"], # These fields can be filtered
)
# Now you can filter on these columns
await vectorstore.adelete(filter={"source": "documentation"})
```
```python
# ❌ Won't work: Fields only in metadata_json_column cannot be filtered
vectorstore = await AsyncPGVectorStore.create(
engine,
embedding_service=embeddings_service,
table_name="my_table",
# No metadata_columns defined - all metadata goes to JSON column
)
# This will fail - "source" is not a database column
await vectorstore.adelete(filter={"source": "documentation"})
```
Fields stored only in `metadata_json_column` cannot be used in filters.
This design choice provides better query performance and leverages
PostgreSQL's native indexing capabilities.
## Implementation Details
- **Backward compatible**: Existing code using `adelete(ids=[...])`
continues to work unchanged
- **SQL injection safe**: Uses parameterized queries via existing
`_create_filter_clause()` method
- **Consistent behavior**: Filter syntax matches `similarity_search()`
for consistency
- **Performance**: Generates efficient SQL DELETE statements with WHERE
clauses on indexed columns
- **Type safe**: Full type hints and passes mypy strict checking
## Test Coverage
Added comprehensive test coverage (all tests passing):
- ✅ `test_adelete_with_filter`: Basic metadata filter deletion
- ✅ `test_adelete_with_filter_and_operator`: Deletion with comparison
operators
- ✅ `test_adelete_with_complex_filter`: Complex filters with logical
operators
- ✅ `test_adelete_with_filter_and_ids`: Combined ID and filter deletion
- ✅ `test_adelete_with_filter_no_matches`: Graceful handling of no
matches
- ✅ `test_adelete_with_filter` (sync): Async method in sync wrapper
- ✅ `test_delete_with_filter` (sync): Sync method filtering
- ✅ `test_adelete`: Existing tests continue to pass
All tests follow existing patterns and integrate with the current test
suite.
## Breaking Changes
None. This is a backward-compatible enhancement:
- All existing code continues to work unchanged
- New `filter` parameter is optional
- Default behavior (no parameters) remains the same
## Checklist
- [x] Implementation complete for async methods
- [x] Implementation complete for sync methods
- [x] Comprehensive test coverage added (7 tests, all passing)
- [x] Code passes `ruff` linting
- [x] Code passes `mypy` type checking
- [x] Docstrings updated with examples and limitations
- [x] Backward compatible with existing code
- [x] Follows existing codebase patterns
## Related Issues
Closes #271
## Additional Notes
### Why metadata_columns are required for filtering
This implementation reuses the robust `_create_filter_clause()` method
that's already extensively tested for search operations. The method
generates SQL WHERE clauses that operate on actual database columns,
which provides:
1. **Better performance**: Direct column filtering is faster than JSON
field extraction
2. **Index support**: Metadata columns can be indexed for even better
performance
3. **Type safety**: Database column types ensure type correctness
4. **Consistency**: Same behavior as `similarity_search()` filtering
This design is consistent with the existing filtering implementation and
aligns with how other parts of the codebase handle metadata filtering.
---------
Co-authored-by: dishaprakash <57954147+dishaprakash@users.noreply.github.com>