snapraid/AGENTS.md

SnapRAID Development Guide

This document provides guidance for agentic coding agents working on the SnapRAID codebase.

Project Structure

Core Source Files (cmdline/)

The CLI is organized into focused C modules:

Modules	Purpose
main.c, snapraid.h / snapraid.c	Program entry points, option parsing, and CLI command dispatch
state.h / state.c	Global state (snapraid_state), config file parsing, and content file read/write
elem.h / elem.c	Core data model (files, links, dirs, disks, parity, filters) and filter logic
scan.c, import.h / import.c	Multi-threaded directory scanner and file-move hash importer
search.h / search.c, locate.h / locate.c	Path pattern matching and file lookup / path location commands
io.h / io.c	Parallel read-ahead and write-behind I/O queue management
parity.h / parity.c	Parity file operations supporting split parity files
handle.h / handle.c	File handles abstraction for data disk I/O with POSIX fadvise settings
bw.h / bw.c	Bandwidth token-bucket rate limiter
stream.h / stream.c	Buffered stream abstraction with CRC32 integrity checks
sync.c, scrub.c, check.c, status.c	CLI commands for core operations (synchronize, integrity scrub, check, health status)
pool.c, dup.c, list.c, touch.c, rehash.c	CLI commands for pooling layouts, finding duplicates, listing files, updating tracking/hashes
device.c, dry.c	Disk device geometry/UUID querying, and dry-run execution simulation
murmur3.c, spooky2.c, museair.c (and tests)	Non-cryptographic hash implementations (MurmurHash3, SpookyHash, MuseAir) and tests
support.h / support.c, app.h	Locks, logging, thread wrappers, global constants
str.h / str.c	Path/string utilities
memory.h / memory.c	Safe memory allocation and tracking helpers
util.h / util.c	Low-level utilities (CRC32 calculations, numeric formatting, hash helpers)
unixapp.c, mingwapp.c	Platform-specific implementations of system info, process management, and services
thermal.h / thermal.c	Temperature monitoring to prevent hardware overheating
selftest.c, speed.c	Self-test command (test) and CPU RAID parity performance benchmark (-T)
mktest.c, mkstream.c	Standalone test utility tools for recovery and stream validation

OS Abstraction Files (os/)

The platform abstraction layer is separated into os/. Note that os/ acts as a library layer, so having unused functions (e.g., platform helper functions not currently invoked by the main CLI application) is expected.

Module	Purpose
os.h	Core OS-independent interface declarations (syslog, signals, execution, threads, mutexes)
portable.h	Platform detection macros and compatibility definitions
unix.c/h	UNIX implementation of OS abstraction layer (fork, exec, signals, thread primitives)
mingw.c/h	Windows implementation of OS abstraction layer (process creation, registry, thread primitives)

Key Data Structures

Struct	Defined in	Description
snapraid_state	state.h	Global program state: all disks, parity levels, filter list, options, hash kind, progress
snapraid_option	state.h	Runtime options (force flags, sort order, IO advise mode, bandwidth limit, etc.)
snapraid_disk	elem.h	One data disk: name, mount dir, device id, UUID, file/link/dir hash tables and lists, parity extent tree
snapraid_file	elem.h	A tracked file: sub-path, size, mtime, inode, block vector
snapraid_block	elem.h	One parity block: state (BLK/CHG/REP/DELETED/EMPTY) + hash
snapraid_extent	elem.h	Contiguous run of file blocks mapped to contiguous parity positions
snapraid_filter	elem.h	One include/exclude rule: pattern, root scope, is_disk, is_abs, is_dir, direction
snapraid_parity	elem.h	One parity level: up to SPLIT_MAX=8 split files, total/free blocks
snapraid_io	io.h	Parallel I/O controller: buffer pool, worker threads for each disk
snapraid_worker	io.h	One I/O thread: reads or writes one disk at a time using a task ring buffer

Block States

Constant	Value	Meaning
BLOCK_STATE_EMPTY	0	No file mapped here; parity position is zero-filled
BLOCK_STATE_BLK	1	File block with valid hash and up-to-date parity
BLOCK_STATE_CHG	2	File block changed; hash may be old (pre-sync); parity not yet updated
BLOCK_STATE_REP	3	File block hashed (from copy heuristic); parity not yet updated
BLOCK_STATE_DELETED	4	File deleted; block retains old hash for parity recovery; parity not yet zeroed
BLOCK_STATE_REBUILD	5	File block with valid hash; parity untrusted, rebuild in place without reallocation

Crash-Recovery Invariants

SnapRAID is designed to recover from an interruption, including SIGKILL, at any point. It achieves this by reading recovery data from multiple snapshots, checking file metadata when reading, and validating data with hashes. Account for this design before treating a non-atomic transition or mixed snapshot lifecycle as an integrity defect.

Build System

• configure.ac: Autoconf script (detects systemd vs BSD init)
• Makefile.am: Source file lists, dependencies, install hooks (including rules for generating documentation)
• uncrustify.cfg: Code formatting rules for main application source files
• linux.cfg: Code formatting rules for raid/ directory and its subdirectories
• Run make doc to regenerate all manual pages (*.1) and text manuals (*.txt)
• Always use parallel compilation with make -j$(nproc) instead of plain make
• Create temporary files and directories under /tmp/snapraid/.
• Both Linux and Windows builds are out-of-tree in dedicated subdirectories (linux/ and windows/) in the project directory, keeping the root directory clean:
  • Linux build: mkdir -p linux && (cd linux && ../configure && make -j$(nproc)) Subsequent Linux rebuilds: (cd linux && make -j$(nproc))
  • Windows x64 cross-compilation: mkdir -p windows && (cd windows && ../configure.windows-x64 && make -j$(nproc)) Subsequent Windows rebuilds: (cd windows && make -j$(nproc))
• The Windows build needs to be tested only when modifying Windows-specific code (e.g., os/mingw.*, cmdline/mingwapp.c, or Windows-specific #ifdef paths)

Code Style Guidelines

Design Simplicity

• Simplicity First: Always prefer the simplest and cleanest implementation that fully satisfies the demonstrated requirements. Do not introduce extra state, abstractions, conditions, or defensive handling unless a concrete requirement or verified failure mode needs them. When replacing existing logic, preserve its narrow established semantics instead of broadening it through inferred state from unrelated subsystems.
• Smallest Necessary Change Harmless existing behavior or artifacts do not justify refactoring. Always make the smallest change necessary to satisfy the requirement.

Code Style

• Language: The codebase uses C99 standard (not C11/C++)
• Format: Enforce via uncrustify -c uncrustify.cfg --no-backup *.c *.h (or uncrustify -c linux.cfg --no-backup *.c *.h for raid/ directory and subdirectories)
• Naming: Snake_case for functions, UPPER_CASE for macros/constants
• Indentation: Tabs for indentation, no alignment (existing codebase style)
• Comments: Use /** ... */ for multiline top comments preceding declarations and /* ... */ for multiline comments within code. Use /* first letter lowercase */ for single-line inline notes.
• Critical Comments: Always add a comment at non-obvious critical points, especially around data-integrity invariants, crash recovery, fallback behavior, concurrency, and fatal versus best-effort error handling. Explain why the logic is required, not merely what the code does.
• Headers: All .h files have include guards (#ifndef __NAME_H)
• Preferences: Use 0 instead of NULL and '\0'
• Preferences: Use prefix ++variable and --variable instead of postfix variable++ and variable-- where both are equivalent
• Safety Checks: Avoid adding safety checks for conditions that never happen
• Type Casts: Avoid type casts unless strictly necessary for compilation with -Werror and unavoidable by other mean
• Stack Over Heap: Prefer stack-allocated fixed-size buffers over dynamic memory allocation (malloc/free) whenever the upper bound is small, fixed, and known at compile time.
• Commit Messages: Every time a change is done, a single line commit description should be provided for that change
• Git Commits: Never commit changes to git.

Error Handling & Memory Management

• Return codes: 0 means no-error; < 0 means error; positive for specific results
• Logging: Use log_fatal(), log_tag(), msg_info(), msg_progress(), msg_verbose() from support.h
• Tag Escaping: When emitting structured tags with log_tag(), always wrap file paths, directory paths, link targets, disk names, filter patterns, and arbitrary string fields with esc_tag() to prevent colons (e.g., Windows drive letters C:), backslashes, or newlines from breaking colon-delimited tag parsing.
• Allocation: Always use custom malloc_nofail(), calloc_nofail(), nalloc_nofail(), or strdup_nofail() wrappers (which handle OOM by aborting)
• Path Manipulation: Never use standard unsafe string functions (strcpy, strcat, strcmp) on paths. Always use the platform-correct, size-bounded wrappers in support.h (e.g., pathcpy(), pathcat(), pathcmp(), pathslash()).
• TommyDS Iteration: For tommy_list, iterate using: tommy_node* node = list.head; while (node) { void* obj = node->data; ... node = node->next; } Or use callback iteration helpers: tommy_list_foreach(), tommy_list_foreach_arg(), tommy_hashdyn_foreach(), etc.
• Cleanup: Use goto pattern for cleanup in complex functions when appropriate
• Fatal paths: Use exit(EXIT_FAILURE) for unrecoverable errors; os_abort() for internal inconsistencies
• Exit Code & Logging Invariants:
  • log_fatal(): Any invocation of log_fatal() represents an unrecoverable failure and must force a non-zero exit code upon termination.
  • log_error(): Whenever log_error() is called with a hardware error (is_hw(err)), it must force a non-zero exit code upon termination.

Threading

• Compile-time guard: #if HAVE_THREAD wraps all thread-specific code
• Mutexes: Always use the custom thread_mutex_init/destroy/lock/unlock() wrappers
• IO workers: The snapraid_io / snapraid_worker infrastructure handles parallel disk I/O via a pool of reader and writer threads

Testing Changes

• Never run make check because it's too expansive
• Run ./linux/snapraid test for the regression test
• The test and selftest suites (in cmdline/selftest.c and ./linux/snapraid test) must be strictly memory-only without creating files or performing filesystem I/O
• If running the speed benchmark (./linux/snapraid -T), always use --speed-test-period 100 (e.g. ./linux/snapraid -T --speed-test-period 100) to run it faster
• The Windows build needs to be tested only when modifying Windows-specific code
• Verify error paths and recovery scenarios (the test suite covers aborted sync, UUID changes, disk failures)