mirror of https://github.com/langchain-ai/datafusion.git synced 2026-07-19 15:24:21 -04:00

Files

T

Adrian Garcia Badaracco 0cf45cae9c Refactor cache APIs to support ordering information (#19597 )

## Which issue does this PR close?

Part of #19433

## Rationale for this change

In preparation for ordering inference from Parquet metadata, the cache
system needs refactoring to:
1. Support storing ordering information alongside statistics
2. Simplify the `CacheAccessor` trait by removing the `Extra` associated
type and `*_with_extra` methods
3. Move validation logic into typed wrapper structs with explicit
`is_valid_for()` methods

## What changes are included in this PR?

### Simplify `CacheAccessor` trait

**Before:**
```rust
pub trait CacheAccessor<K, V>: Send + Sync {
    type Extra: Clone;
    fn get(&self, k: &K) -> Option<V>;
    fn get_with_extra(&self, k: &K, e: &Self::Extra) -> Option<V>;
    fn put(&self, key: &K, value: V) -> Option<V>;
    fn put_with_extra(&self, key: &K, value: V, e: &Self::Extra) -> Option<V>;
    // ... other methods
}
```

**After:**
```rust
pub trait CacheAccessor<K, V>: Send + Sync {
    fn get(&self, key: &K) -> Option<V>;
    fn put(&self, key: &K, value: V) -> Option<V>;
    // ... other methods (no Extra type, no *_with_extra methods)
}
```

### Introduce typed wrapper structs for cached values

Instead of passing validation metadata separately via `Extra`, embed it
in the cached value type:

- **`CachedFileMetadata`** - contains `meta: ObjectMeta`, `statistics:
Arc<Statistics>`, `ordering: Option<LexOrdering>`
- **`CachedFileList`** - contains `files: Arc<Vec<ObjectMeta>>` with
`filter_by_prefix()` helper
- **`CachedFileMetadataEntry`** - contains `meta: ObjectMeta`,
`file_metadata: Arc<dyn FileMetadata>`

Each wrapper has an `is_valid_for(&ObjectMeta)` method that checks if
the cached entry is still valid (size and last_modified match).

### New validation pattern

The typical usage pattern changes from:
```rust
// Before: validation hidden in get_with_extra
if let Some(stats) = cache.get_with_extra(&path, &object_meta) {
    // use stats
}
```

To:
```rust
// After: explicit validation
if let Some(cached) = cache.get(&path) {
    if cached.is_valid_for(&object_meta) {
        // use cached.statistics
    }
}
```

### Add ordering support

- `CachedFileMetadata` has new `ordering: Option<LexOrdering>` field
- `FileStatisticsCacheEntry` has new `has_ordering: bool` field for
introspection

## Are these changes tested?

Yes, existing cache tests pass plus new tests for ordering support.

## Are there any user-facing changes?

Breaking change to cache traits. Users with custom cache implementations
will need to:
1. Update `CacheAccessor` impl to remove `Extra` type and `*_with_extra`
methods
2. Update cached value types to the new wrappers (`CachedFileMetadata`,
etc.)
3. Update callsites to use the new validation pattern with
`is_valid_for()`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

2026-01-08 17:56:41 +00:00

examples

Update 4 crates to rust 2024 edition (#19357 )

2025-12-16 20:04:38 +00:00

src

Refactor cache APIs to support ordering information (#19597 )

2026-01-08 17:56:41 +00:00

tests

chore: bump testcontainers-modules to 0.14 and remove testcontainers dep (#19620 )

2026-01-03 19:31:21 +00:00

Cargo.toml

chore: bump testcontainers-modules to 0.14 and remove testcontainers dep (#19620 )