From f9c05a949d1a0dc8e16dee28ca9605d38611faeb Mon Sep 17 00:00:00 2001 From: Andrew Gallant Date: Fri, 24 Jul 2026 07:38:17 -0400 Subject: [PATCH] index: remove incorrect README --- crates/index/README.md | 101 +---------------------------------------- 1 file changed, 1 insertion(+), 100 deletions(-) diff --git a/crates/index/README.md b/crates/index/README.md index db965990..00d7bdd4 100644 --- a/crates/index/README.md +++ b/crates/index/README.md @@ -1,100 +1 @@ -grep-index -========== - -`grep-index` is an experimental persistent byte-trigram index for ripgrep. -It never claims that a file matches. It returns files that cannot be ruled -out, and ripgrep still performs the real search over those candidates. - -The design keeps the useful pieces of Nakala's PLAN.md while delegating the -embedded-database work to redb: - -- writes publish immutable logical segments; -- each segment assigns dense, one-based `u32` DocIDs in lexicographic path - order; -- a monotonic `fst::Map` provides both path-to-DocID and DocID-to-path - lookups; -- redb stores contiguous delta-varint posting-list blobs, metadata, the - active segment catalog and compact tombstone bitmaps; -- an upsert atomically tombstones older path versions and publishes a new - segment; -- compaction rewrites live documents into one tombstone-free segment. - -Only raw byte trigrams are indexed. Document contents and positions are not -stored. - -## API sketch - -```rust -use grep_index::{FileMetadata, Index, Query}; - -let index = Index::create("corpus.index.redb")?; -index.upsert("src/lib.rs", contents, FileMetadata::default())?; - -// The normal ripgrep-facing path: no metadata decoding. -let paths = index.candidate_paths(&Query::literal(b"needle"))?; - -// Metadata is available when a caller wants post-index filtering/sorting. -let candidates = index.search_with(&Query::literal(b"needle"), |meta| { - !meta.hidden && meta.size < 10_000_000 -})?; -``` - -`IndexWriter` batches many files into one segment. `Index::index_directory` -walks a tree and commits bounded-size segments. The free `build_index` -function preserves the prototype's one-shot directory API and publishes its -initial database with a no-clobber hard link. - -Logical compaction is for faster searches. A separate storage-compaction -method asks redb to reclaim unused pages when the caller can afford a slower -physical rewrite. - -`Index::open_read_only` uses redb 4's read-only handle for multi-process -candidate readers. Multiple read-only handles can coexist, but redb does not -allow them to overlap a writable `Database` handle. - -## Storage notes - -Paths are stored as raw platform path bytes instead of UTF-8 strings. On -Unix this preserves non-UTF-8 paths; on Windows it stores UTF-16 code units. -DocIDs are assigned only after paths are sorted, which is the invariant -required by `fst::raw::Fst::get_key_into` for efficient reverse lookup. - -Posting values have a count prefix followed by delta-varint encoded sorted -`u32` DocIDs. Query conjunctions decode the smallest lists first and use a -galloping intersection when list sizes are highly skewed. The implementation -is entirely safe Rust; mmap and custom unsafe codecs are intentionally left -for benchmark-driven follow-up work. - -Files with a UTF-16 byte-order mark contribute trigrams from both their raw -bytes and ripgrep's default UTF-8 transcoding. This preserves conservative -candidate selection for both default searches and `--encoding none`; the -extra terms can only create false positives. - -File metadata currently records size, the time at which the file was indexed, -optional modified/created/accessed timestamps, hidden, read-only, executable -and symlink flags. Metadata has a versioned binary encoding independent of -posting lists. - -An index can also persist an absolute corpus root for resolving its stored -paths and a minimal set of absolute traversal scopes for ordinary-search -fallback. `build_index` sets both automatically; callers constructing an -`Index` directly can use `Index::set_root` and `Index::add_scopes`. - -## Current boundaries - -- `index_directory` is append/upsert-oriented. It updates files it sees, but - does not infer deletes for files that disappeared from the tree; call - `Index::delete` for explicit removals. -- Directory walking intentionally has only small built-in policies: it can - skip binary files, hidden entries and `.git`, but it is not ripgrep's full - ignore walker yet. -- Search currently materializes result vectors. The path-only API avoids - metadata work, but a streaming visitor is a likely next step for enormous - candidate sets. -- Logical compaction currently materializes its live remap and postings. - A streaming, bounded-memory merge like the full Nakala plan is a future - optimization. -- This crate uses redb 4.1 and Rust 1.97. Earlier experimental index formats - are rebuild-only. - -Dual-licensed under MIT or the [UNLICENSE](http://unlicense.org). +WIP