Antonio S. MusumeciandClaude Sonnet 5
8eed256747
Reinstate the splice read/write/move paths removed in 84592a9f, now
gated behind runtime options with negotiated pipe sizing and fallback
logging.
- Transport: msgbuf-piped splice receive, fd-referenced splice reply
(vmsplice + splice), splice-move pipe-bounce, and restored
fuse_buf_copy/fuse_buf_flags plumbing for copy fallbacks
- Negotiation: new pipe-max-size option (auto-raised at INIT when
possible; the sysctl read/write is factored into src/pipe_max_size.cpp
so fuse_init.cpp and option_parser.cpp share one implementation);
fuse-msg-size clamped to what the pipe can hold; splice is disabled
at INIT with a warning if the sysctl can't fit even the minimum,
retracting the FUSE_CAP_SPLICE_* bits already advertised to the
kernel in that case
- Read replies and splice-move are gated at 128 KiB (below that the
syscalls cost more than the copy they avoid); every fallback path
degrades to the copy path and logs once per process via SysLog
- msgbuf pipe cache is process-global and mutex-guarded (mutex_lockguard)
so pipes can move between read and worker threads under
process-thread mode
- API simplified after review: collapsed the three per-axis options
(splice-read/write/move) into a single splice boolean (default on);
no-splice-read/write/move and the bare splice-read/write/move
spellings stay registered as accepted no-ops so existing mount
options don't fail the mount, but they do NOT alter the splice
knob - upgraders with no-splice-* in their config get splice
enabled and must set splice=false explicitly (see
deprecated_options.md)
- Fixed several correctness issues found in parallel review: a
dropped write_buf registration that silently corrupted writes, a
dirty-pipe reuse bug in msgbuf_free, an iov double-send in
fuse_reply_data's splice-move fallback, and pipe-capacity math that
assumed a 4096-byte page size
- Modernized the transport/negotiation code to this codebase's C++
style: RAII (unique_ptr/DEFER) instead of malloc/free and manual fd
close(), std::from_chars/to_chars instead of stoul/snprintf,
_ReplyPipe's pipe ops as methods instead of free functions closing
over a thread_local global, and vendored/libfuse/lib/buffer.cpp (a
near-verbatim port of upstream's buffer.c) brought in line with the
rest of the fork
- Documented splice and pipe-max-size options (mkdocs, FAQ,
deprecated-options list); added tools/bench-splice.sh
- Benchmarked on tmpfs: seq_write +21%, seq_read +52%, small_write
+5%, small_read +12%; validated with a 1h soak (~271 GiB
hash-verified I/O, zero integrity failures), a live-mount round-trip
of I/O above the splice threshold, and the existing unit/smoke
suites at their pre-existing baseline
Co-Authored-By: Claude Sonnet 5 <[email protected]>
mergerfs - a featureful union filesystem
mergerfs is a FUSE based union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to mhddfs, unionfs, aufs, DrivePool, etc.
Features
- Logically combine numerous filesystems/paths into a single mount point (JBOFS: Just a Bunch of FileSystems)
- Combine paths of the same or different filesystems
- Ability to add or remove filesystems/paths without impacting the rest of the data
- Unaffected by individual filesystem failure
- Configurable file selection and creation placement
- File IO passthrough for near native IO performance (where supported)
- Works with filesystems of any size
- Works with filesystems of almost any type
- Ignore read-only filesystems when creating files
- Hard links
- Hard link copy-on-write / CoW
- Runtime configurable
- Support for extended attributes (xattrs)
- Support for file attributes (chattr)
- Support for POSIX ACLs
Non-features
- Read/write overlay on top of read-only filesystem like OverlayFS
- File whiteout
- RAID like parity calculation (see SnapRAID)
- Redundancy
- Splitting of files across branches
How it works
mergerfs logically merges multiple filesystem paths together. Not block devices, not filesystem mounts, just paths. It acts as a proxy to the underlying filesystem paths. Combining the behaviors of some functions and being a selector for others.
When the contents of a directory are requested mergerfs combines the list of files from each directory, deduplicating entries, and returns that list.
When a file or directory is created a policy is first run to determine which branch will be selected for the creation.
For functions which change attributes or remove the file the behavior may be applied to all instances found.
The way in which mergerfs behaves is controlled by the config/options/settings. More specifically by policies.
Visualization
A + B = C
/disk1 /disk2 /merged
| | |
+-- /dir1 +-- /dir1 +-- /dir1
| | | | | |
| +-- file1 | | | +-- file1
| | +-- file2 | +-- file2
| | +-- file3 | +-- file3
| | |
+-- /dir2 | +-- /dir2
| | | | |
| +-- file4 | | +-- file4
| | |
| +-- /dir3 +-- /dir3
| | | | |
| | +-- file5 | +-- file5
| | |
+-- file6 | +-- file6
+-- file7 +-- file7 +-- file7
QuickStart
https://trapexit.github.io/mergerfs/latest/quickstart/
Documentation
https://trapexit.github.io/mergerfs
Support
https://trapexit.github.io/mergerfs/latest/support/
Sponsorship and Donations
https://github.com/trapexit/support
Development and support of a project like mergerfs requires a significant amount of time and effort. The software is released under the very liberal ISC license and is therefore free to use for personal or commercial uses.
If you are a non-commercial user and find mergerfs and its support valuable and would like to support the project financially it would be very much appreciated.
If you are using mergerfs commercially please consider sponsoring the project to ensure it continues to be maintained and receive updates. If custom features are needed feel free to contact me directly.