Merge pull request #10179 from ThomasWaldmann/kernel-defaults-10160

pick chunker scan kernel defaults per platform

docs/changes.rst

@@ -236,13 +236,17 @@ New features:
 
     - AVX-512 / AVX2 on x86-64 (Intel / AMD), NEON on aarch64, plus a portable
       blockwise one; all bit-identical to the sequential loop
-    - the sequential loop is what runs unless one of BORG_FASTCDC_KERNEL /
-      BORG_BUZHASH64_KERNEL / BORG_AES_CHUNKER_KERNEL selects another kernel;
-      which one is fastest depends on the CPU *and* the compiler, so nothing
-      is chosen automatically
+    - which one is fastest depends on the CPU *and* the compiler, so the
+      default is what benchmarking found: NEON on aarch64, the sequential loop
+      on x86-64 (the compiler folds the rolling hash update into a single
+      instruction there), blockwise elsewhere, #10160
+    - BORG_FASTCDC_KERNEL / BORG_BUZHASH64_KERNEL / BORG_AES_CHUNKER_KERNEL
+      override that; a kernel this build or CPU can not run is an error, never
+      a silent fallback
   - toeplitz-aes, rabin-aes, goldilocks-aes: fingerprinting-resistant chunkers
     (UHF-then-PRF), with direct AES hardware acceleration (AES-NI or
-    VAES/AVX-512) or via OpenSSL, #9987, #10043
+    VAES/AVX-512) or via OpenSSL, #9987, #10043. Here wider is simply faster,
+    so the default is the best path the build and the CPU offer, #10160
   - zero-copy fill and lazy buffer compaction optimizations
   - log the chunker and its scan kernel at debug level
 - compression: support zstd's negative ("fast") levels, ``zstd,-1`` .. ``zstd,-128``, #9950.

docs/usage/general/environment.rst.inc

@@ -198,15 +198,18 @@ General:
         it uses more cpu time in total to reduce the wallclock time. Set it to 1 if you would
         rather have the smaller archive, or if borg has to share the cpu with other work.
     BORG_FASTCDC_KERNEL / BORG_BUZHASH64_KERNEL
-        Select the scan kernel the ``fastcdc`` / ``buzhash64`` chunker uses (default:
-        ``scalar``, the plain sequential loop). Accepted values are ``avx512``, ``avx2``,
-        ``neon``, ``blockwise`` and ``scalar``.
+        Select the scan kernel the ``fastcdc`` / ``buzhash64`` chunker uses. Accepted values
+        are ``avx512``, ``avx2``, ``neon``, ``blockwise`` and ``scalar``.
+        The default is whichever benchmarked fastest for the architecture: ``neon`` on
+        aarch64, and ``scalar`` (the plain sequential loop) on x86-64, where the compiler
+        folds the rolling hash update into a single instruction and thereby beats the vector
+        kernels. Other architectures get ``blockwise``, the portable multi-lane C kernel.
         All kernels chunk identically - same cut points, same chunk ids - and differ only in
         speed, so this is safe to change at any time, also for an existing repository.
         Which kernel is fastest is not predictable from the instruction set: it depends on the
         cpu and on the compiler that built borg, and the sequential loop wins on some machines.
-        Nothing is selected automatically, so measure on your own hardware with
-        ``borg benchmark cpu --chunking`` before setting these.
+        Measure on your own hardware with ``borg benchmark cpu --chunking`` before overriding
+        the default.
         ``avx512`` and ``avx2`` exist only on x86-64, ``neon`` only on aarch64, and only if the
         compiler that built borg supported them; ``scalar`` and ``blockwise`` are portable C
         and always available.
@@ -215,11 +218,14 @@ General:
         ``borg create --debug`` logs the chunker and the kernel it was created with.
     BORG_AES_CHUNKER_KERNEL
         Select the scan kernel used by the AES based chunkers - one variable for all three of
-        ``toeplitz-aes``, ``rabin-aes`` and ``goldilocks-aes`` (default: ``evp``, the portable
-        OpenSSL path). Accepted values are ``vaes``, ``aes-ni``, ``aes-arm64`` and ``evp``.
+        ``toeplitz-aes``, ``rabin-aes`` and ``goldilocks-aes``. Accepted values are ``vaes``,
+        ``aes-ni``, ``aes-arm64`` and ``evp``.
+        Unlike the chunker kernels above, wider is simply faster here, so the default is the
+        best path this build and cpu offer: ``vaes``, else ``aes-ni`` on x86-64, ``aes-arm64``
+        on aarch64, and ``evp`` (the portable OpenSSL path) where there is no AES hardware
+        path.
         As with the chunker kernels above, all of them chunk identically and differ only in
-        speed, nothing is selected automatically, and a kernel that can not run here is an
-        error rather than a silent fallback.
+        speed, and a kernel that can not run here is an error rather than a silent fallback.
         ``vaes`` and ``aes-ni`` exist only on x86-64, ``aes-arm64`` only on aarch64.
         ``vaes`` additionally needs a compiler that knows it (gcc >= 11 / clang >= 14), so a
         cpu supporting VAES is not by itself enough to have that kernel available.

src/borg/archiver/help_cmd.py

@@ -783,15 +783,18 @@ class HelpMixIn:
                 Single-threaded can even be faster on data zstd races through anyway, e.g.
                 already-compressed/incompressible data or long-repeat data like VM images.
             BORG_FASTCDC_KERNEL / BORG_BUZHASH64_KERNEL
-                Select the scan kernel the ``fastcdc`` / ``buzhash64`` chunker uses (default:
-                ``scalar``, the plain sequential loop). Accepted values are ``avx512``, ``avx2``,
-                ``neon``, ``blockwise`` and ``scalar``.
+                Select the scan kernel the ``fastcdc`` / ``buzhash64`` chunker uses. Accepted values
+                are ``avx512``, ``avx2``, ``neon``, ``blockwise`` and ``scalar``.
+                The default is whichever benchmarked fastest for the architecture: ``neon`` on
+                aarch64, and ``scalar`` (the plain sequential loop) on x86-64, where the compiler
+                folds the rolling hash update into a single instruction and thereby beats the vector
+                kernels. Other architectures get ``blockwise``, the portable multi-lane C kernel.
                 All kernels chunk identically - same cut points, same chunk ids - and differ only in
                 speed, so this is safe to change at any time, also for an existing repository.
                 Which kernel is fastest is not predictable from the instruction set: it depends on the
                 cpu and on the compiler that built borg, and the sequential loop wins on some machines.
-                Nothing is selected automatically, so measure on your own hardware with
-                ``borg benchmark cpu --chunking`` before setting these.
+                Measure on your own hardware with ``borg benchmark cpu --chunking`` before overriding
+                the default.
                 ``avx512`` and ``avx2`` exist only on x86-64, ``neon`` only on aarch64, and only if the
                 compiler that built borg supported them; ``scalar`` and ``blockwise`` are portable C
                 and always available.
@@ -800,11 +803,14 @@ class HelpMixIn:
                 ``borg create --debug`` logs the chunker and the kernel it was created with.
             BORG_AES_CHUNKER_KERNEL
                 Select the scan kernel used by the AES based chunkers - one variable for all three of
-                ``toeplitz-aes``, ``rabin-aes`` and ``goldilocks-aes`` (default: ``evp``, the portable
-                OpenSSL path). Accepted values are ``vaes``, ``aes-ni``, ``aes-arm64`` and ``evp``.
+                ``toeplitz-aes``, ``rabin-aes`` and ``goldilocks-aes``. Accepted values are ``vaes``,
+                ``aes-ni``, ``aes-arm64`` and ``evp``.
+                Unlike the chunker kernels above, wider is simply faster here, so the default is the
+                best path this build and cpu offer: ``vaes``, else ``aes-ni`` on x86-64, ``aes-arm64``
+                on aarch64, and ``evp`` (the portable OpenSSL path) where there is no AES hardware
+                path.
                 As with the chunker kernels above, all of them chunk identically and differ only in
-                speed, nothing is selected automatically, and a kernel that can not run here is an
-                error rather than a silent fallback.
+                speed, and a kernel that can not run here is an error rather than a silent fallback.
                 ``vaes`` and ``aes-ni`` exist only on x86-64, ``aes-arm64`` only on aarch64.
                 ``vaes`` additionally needs a compiler that knows it (gcc >= 11 / clang >= 14), so a
                 cpu supporting VAES is not by itself enough to have that kernel available.

src/borg/chunkers/buzhash64.pyx

@@ -21,7 +21,7 @@ cdef extern from "buzhash64_impl.h":
     const char *bz64_kernel_name(int kernel)
     int bz64_kernel_select(const char *name, int *out_id)
     const char *bz64_kernel_names()
-    int BZ_K_SCALAR
+    int bz64_kernel_default()
 
 from .kernel_env import kernel_error, requested_kernel
 
@@ -109,14 +109,14 @@ cdef uint64_t _buzhash64_update(uint64_t sum, unsigned char remove, unsigned cha
 cdef int _select_kernel() except -1:
     """Resolve BORG_BUZHASH64_KERNEL to a kernel id, raising if it cannot be honoured.
 
-    Unset means the simplest implementation, not the fastest one: nothing here
-    guesses which kernel a given CPU and compiler make fastest.
+    Unset means the kernel that measured fastest on this platform, see
+    bz64_kernel_default().
     """
-    cdef int kid = BZ_K_SCALAR
+    cdef int kid
     cdef int rc
     want = requested_kernel("BORG_BUZHASH64_KERNEL")
     if want is None:
-        return BZ_K_SCALAR
+        return bz64_kernel_default()
     rc = bz64_kernel_select(want.encode("ascii"), &kid)
     if rc != 0:
         raise kernel_error("BORG_BUZHASH64_KERNEL", want, rc,
@@ -180,8 +180,8 @@ cdef class ChunkerBuzHash64(ChunkerBase):
     def kernel(self):
         """Which scan kernel this chunker uses: 'neon', 'avx512', 'avx2', 'blockwise' or 'scalar'.
 
-        'scalar' unless BORG_BUZHASH64_KERNEL names another one, in which case
-        this is always that one - creating the chunker fails otherwise.
+        The platform default unless BORG_BUZHASH64_KERNEL names another one, in
+        which case this is always that one - creating the chunker fails otherwise.
         """
         return (<bytes>bz64_kernel_name(self.kernel_id)).decode("ascii")
 

src/borg/chunkers/buzhash64_impl.c

@@ -407,6 +407,22 @@ int bz64_kernel_select(const char *name, int *out_id)
     return BZ_KSEL_UNKNOWN;
 }
 
+int bz64_kernel_default(void)
+{
+#if defined(__aarch64__)
+    /* Same picture as for fastcdc: NEON is baseline on aarch64 and wins, so
+     * this only falls back if the build lacks it. */
+    int kid;
+    if (bz64_kernel_select("neon", &kid) == BZ_KSEL_OK)
+        return kid;
+    return BZ_K_BLOCKWISE;
+#elif defined(__x86_64__) || defined(_M_X64)
+    return BZ_K_SCALAR;
+#else
+    return BZ_K_BLOCKWISE;
+#endif
+}
+
 /* --- dispatch ----------------------------------------------------------- */
 
 size_t bz64_scan(const uint64_t *table, const uint64_t *table_rot,

src/borg/chunkers/buzhash64_impl.h

@@ -23,11 +23,10 @@
  * (p_add = p_rem + window_size); both must have n readable bytes.
  * kernel is one of BZ_K_*.
  * All kernels return bit-identical results. */
-/* Scan kernel ids; see fastcdc_impl.h for the rationale. There is no
- * automatic selection - the caller says which kernel to run. */
+/* Scan kernel ids; see fastcdc_impl.h for the rationale. */
 #define BZ_K_SCALAR 0    /* sequential reference loop */
 #define BZ_K_BLOCKWISE 1 /* portable 8-lane C */
-#define BZ_K_VECTOR 2    /* neon / avx2 (neon is not auto-selected) */
+#define BZ_K_VECTOR 2    /* the platform's vector kernel: neon or avx2 */
 #define BZ_K_VECTOR512 3 /* avx512 */
 
 /* Results of bz64_kernel_select(). */
@@ -42,6 +41,10 @@ int bz64_kernel_select(const char *name, int *out_id);
 /* Comma-separated list of the kernel names this build accepts. */
 const char *bz64_kernel_names(void);
 
+/* The kernel to run when the caller did not ask for a specific one, see
+ * bz64_kernel_default() in buzhash64_impl.c for what is chosen where. */
+int bz64_kernel_default(void);
+
 size_t bz64_scan(const uint64_t *table, const uint64_t *table_rot,
                  const uint8_t *p_rem, const uint8_t *p_add,
                  size_t n, uint64_t *sum, uint64_t mask, int kernel);

src/borg/chunkers/fastcdc.pyx

@@ -17,7 +17,7 @@ cdef extern from "fastcdc_impl.h":
     const char *fc_kernel_name(int kernel)
     int fc_kernel_select(const char *name, int *out_id)
     const char *fc_kernel_names()
-    int FC_K_SCALAR
+    int fc_kernel_default()
 
 from .kernel_env import kernel_error, requested_kernel
 
@@ -40,8 +40,9 @@ from .kernel_env import kernel_error, requested_kernel
 # The inner scan runs in a C kernel (fastcdc_impl.c) with SIMD implementations (NEON on
 # aarch64, AVX-512 or AVX2 on x86-64, blockwise scalar elsewhere) that are bit-identical to
 # the plain sequential Gear loop: every byte position is tested, cuts and hash state are
-# exactly the same, only faster. The sequential loop is what runs by default; the others
-# are selected via BORG_FASTCDC_KERNEL, see kernel_env.py and the .kernel property.
+# exactly the same, only faster. Which one runs by default is decided per platform by
+# fc_kernel_default(); BORG_FASTCDC_KERNEL overrides that, see kernel_env.py and the
+# .kernel property.
 
 
 @cython.boundscheck(False)
@@ -67,14 +68,14 @@ cdef uint64_t* fastcdc_init_gear(bytes key) except NULL:
 cdef int _select_kernel() except -1:
     """Resolve BORG_FASTCDC_KERNEL to a kernel id, raising if it cannot be honoured.
 
-    Unset means the simplest implementation, not the fastest one: nothing here
-    guesses which kernel a given CPU and compiler make fastest.
+    Unset means the kernel that measured fastest on this platform, see
+    fc_kernel_default().
     """
-    cdef int kid = FC_K_SCALAR
+    cdef int kid
     cdef int rc
     want = requested_kernel("BORG_FASTCDC_KERNEL")
     if want is None:
-        return FC_K_SCALAR
+        return fc_kernel_default()
     rc = fc_kernel_select(want.encode("ascii"), &kid)
     if rc != 0:
         raise kernel_error("BORG_FASTCDC_KERNEL", want, rc,
@@ -111,8 +112,8 @@ cdef class ChunkerFastCDC(ChunkerBase):
     def kernel(self):
         """Which scan kernel this chunker uses: 'neon', 'avx512', 'avx2', 'blockwise' or 'scalar'.
 
-        'scalar' unless BORG_FASTCDC_KERNEL names another one, in which case
-        this is always that one - creating the chunker fails otherwise.
+        The platform default unless BORG_FASTCDC_KERNEL names another one, in
+        which case this is always that one - creating the chunker fails otherwise.
         """
         return (<bytes>fc_kernel_name(self.kernel_id)).decode("ascii")
 

src/borg/chunkers/fastcdc_impl.c

@@ -399,6 +399,26 @@ int fc_kernel_select(const char *name, int *out_id)
     return FC_KSEL_UNKNOWN;
 }
 
+int fc_kernel_default(void)
+{
+#if defined(__aarch64__)
+    /* NEON beats the sequential loop clearly (2.1x on an Apple M3). It is
+     * baseline on aarch64, so this only falls back if the build lacks it. */
+    int kid;
+    if (fc_kernel_select("neon", &kid) == FC_KSEL_OK)
+        return kid;
+    return FC_K_BLOCKWISE;
+#elif defined(__x86_64__) || defined(_M_X64)
+    /* The compiler folds the Gear update into a single instruction here, so
+     * the sequential loop beat AVX2 and AVX-512 on every x86-64 CPU that was
+     * benchmarked for #10160 - an Atom N2800 and a Core i5-3570K just as much
+     * as a Ryzen 9 9900X. */
+    return FC_K_SCALAR;
+#else
+    return FC_K_BLOCKWISE;
+#endif
+}
+
 /* --- dispatch ----------------------------------------------------------- */
 
 int64_t fc_scan(const uint64_t *gear, const uint8_t *p, size_t n, uint64_t *fp, uint64_t mask, int kernel)

src/borg/chunkers/fastcdc_impl.h

@@ -14,13 +14,12 @@
 /* Scan kernel ids. Which names map onto them depends on the build: "neon"
  * exists only on aarch64, "avx2"/"avx512" only on x86-64.
  *
- * There is no automatic selection: the caller says which kernel to run, and
- * that is what runs. FC_K_SCALAR, the plain sequential loop, is id 0 and the
- * default everywhere - it is the simplest implementation and the one the
- * others are checked against. Which of the rest is fastest turned out not to
- * be predictable from the instruction set (on an Apple M3 the NEON kernel
- * beats the sequential loop 2.1x; with gcc on a Zen 4 the sequential loop
- * beats AVX-512 by 1.7x), so choosing one is left to whoever measured. */
+ * FC_K_SCALAR, the plain sequential loop, is id 0 and the one the others are
+ * checked against. Which kernel is fastest is not predictable from the
+ * instruction set (on an Apple M3 the NEON kernel beats the sequential loop
+ * 2.1x; with gcc on a Zen 4 the sequential loop beats AVX-512 by 1.7x), so
+ * fc_kernel_default() encodes what was measured per platform rather than
+ * "the widest vectors this CPU has". */
 #define FC_K_SCALAR 0    /* sequential reference loop */
 #define FC_K_BLOCKWISE 1 /* portable 8-lane C */
 #define FC_K_VECTOR 2    /* the platform's vector kernel: neon or avx2 */
@@ -42,6 +41,10 @@ int fc_kernel_select(const char *name, int *out_id);
  * messages. Names a CPU cannot run are still listed. */
 const char *fc_kernel_names(void);
 
+/* The kernel to run when the caller did not ask for a specific one, see
+ * fc_kernel_default() in fastcdc_impl.c for what is chosen where. */
+int fc_kernel_default(void);
+
 /* Scan up to n positions: for i = 0..n-1 advance fp = (fp << 1) + gear[p[i]]
  * and test (fp & mask) == 0.
  * Returns the first i that matched (fp is left at position i), or -1 if none

src/borg/chunkers/goldilocks_aes.pyx

@@ -48,6 +48,7 @@ cdef extern from "goldilocks_aes_impl.h":
     GL_CTX *gl_new(const uint64_t *tables, uint64_t k1, uint64_t k2, const uint8_t *aes_key, int kernel)
     int phte_kernel_select(const char *name, int *out_id)
     const char *phte_kernel_names()
+    int phte_kernel_default()
     int PHTE_K_EVP
     int PHTE_K_HW
     void gl_free(GL_CTX *ctx)
@@ -117,12 +118,15 @@ cdef int _select_kernel() except -1:
 
     One selector for all three AES chunkers: they share phte_scan.h, so the
     available paths never differ between them.
+
+    Unset means the fastest path this build and CPU support, see
+    phte_kernel_default().
     """
-    cdef int kid = PHTE_K_EVP
+    cdef int kid
     cdef int rc
     want = requested_kernel("BORG_AES_CHUNKER_KERNEL")
     if want is None:
-        return PHTE_K_EVP
+        return phte_kernel_default()
     rc = phte_kernel_select(want.encode("ascii"), &kid)
     if rc != 0:
         raise kernel_error("BORG_AES_CHUNKER_KERNEL", want, rc,

src/borg/chunkers/kernel_env.py

@@ -1,16 +1,15 @@
 """Shared parsing for the BORG_*_KERNEL scan-kernel selection env vars.
 
-Each chunker runs the simplest implementation - the plain sequential loop, or
-the portable OpenSSL path for the AES chunkers - unless one of these env vars
-names a different one. There is no automatic selection: which kernel is
-fastest turned out not to be predictable from the instruction set (an Apple M3
-runs NEON 2.1x faster than the sequential loop; gcc on a Zen 4 runs the
-sequential loop 1.7x faster than AVX-512), so nothing guesses on the user's
-behalf.
+Unset, each chunker runs the kernel that measured fastest on this platform
+(*_kernel_default() on the C side). Which one that is is not predictable from
+the instruction set - an Apple M3 runs NEON 2.1x faster than the sequential
+loop, while gcc on a Zen 4 runs the sequential loop 1.7x faster than AVX-512 -
+so the defaults encode benchmark results, not "the widest vectors available".
 
-A value is a demand, not a preference: if that kernel cannot run here, chunker
-creation raises ValueError rather than quietly falling back, because a silent
-fallback turns a benchmark or a test into a measurement of something else.
+A value here is a demand, not a preference: if that kernel cannot run here,
+chunker creation raises ValueError rather than quietly falling back to the
+default, because a silent fallback turns a benchmark or a test into a
+measurement of something else.
 
 The C side resolves names, since which names exist depends on the build and
 the CPU; this module only turns its verdict into an exception.

src/borg/chunkers/phte_core.h

@@ -202,6 +202,21 @@ int phte_kernel_select(const char *name, int *out_id)
     return PHTE_KSEL_UNKNOWN;
 }
 
+int phte_kernel_default(void)
+{
+    int kid;
+    (void)kid;
+#if (defined(__x86_64__) || defined(_M_X64)) && (defined(__GNUC__) || defined(__clang__))
+    if (phte_kernel_select("vaes", &kid) == PHTE_KSEL_OK)
+        return kid;
+#endif
+#if PHTE_HAVE_HW
+    if (phte_kernel_select(PHTE_KIND_HW, &kid) == PHTE_KSEL_OK)
+        return kid;
+#endif
+    return PHTE_K_EVP;
+}
+
 /* --- context base management ------------------------------------------- */
 
 /* Expand the AES key, select the scan path and set up the OpenSSL context.

src/borg/chunkers/phte_kernel.h

@@ -13,10 +13,9 @@
  * on x86-64, and the 128-bit hardware path is spelled "aes-ni" on x86-64 and
  * "aes-arm64" on aarch64.
  *
- * There is no automatic selection: the caller says which path to run.
- * PHTE_K_EVP, the portable OpenSSL batch path, is id 0 and the default - it
- * is the simplest implementation and the one the others are checked
- * against. */
+ * PHTE_K_EVP, the portable OpenSSL batch path, is id 0 and the one the others
+ * are checked against; phte_kernel_default() picks the fastest path this
+ * build and CPU can actually run. */
 #define PHTE_K_EVP 0   /* portable OpenSSL EVP batch path */
 #define PHTE_K_HW 1    /* 128-bit AES instructions: aes-ni / aes-arm64 */
 #define PHTE_K_HW512 2 /* VAES/AVX-512, 4 AES blocks per instruction */
@@ -37,4 +36,9 @@ int phte_kernel_select(const char *name, int *out_id);
  * messages. Names a CPU cannot run are still listed. */
 const char *phte_kernel_names(void);
 
+/* The scan path to run when the caller did not ask for a specific one: the
+ * widest AES path this build and CPU support, down to the portable EVP one.
+ * Unlike the rolling-hash kernels, wider is simply faster here. */
+int phte_kernel_default(void);
+
 #endif /* BORG_PHTE_KERNEL_H */

src/borg/chunkers/rabin_aes.pyx

@@ -60,6 +60,7 @@ cdef extern from "rabin_aes_impl.h":
     RA_CTX *ra_new(const uint64_t *tables, const uint8_t *aes_key, int kernel)
     int phte_kernel_select(const char *name, int *out_id)
     const char *phte_kernel_names()
+    int phte_kernel_default()
     int PHTE_K_EVP
     void ra_free(RA_CTX *ctx)
     const char *ra_kind(const RA_CTX *ctx)
@@ -186,12 +187,15 @@ cdef int _select_kernel() except -1:
 
     One selector for all three AES chunkers: they share phte_scan.h, so the
     available paths never differ between them.
+
+    Unset means the fastest path this build and CPU support, see
+    phte_kernel_default().
     """
-    cdef int kid = PHTE_K_EVP
+    cdef int kid
     cdef int rc
     want = requested_kernel("BORG_AES_CHUNKER_KERNEL")
     if want is None:
-        return PHTE_K_EVP
+        return phte_kernel_default()
     rc = phte_kernel_select(want.encode("ascii"), &kid)
     if rc != 0:
         raise kernel_error("BORG_AES_CHUNKER_KERNEL", want, rc,

src/borg/chunkers/toeplitz_aes.pyx

@@ -53,6 +53,7 @@ cdef extern from "toeplitz_aes_impl.h":
     TP_CTX *tp_new(const uint64_t *tables, const uint8_t *aes_key, int kernel)
     int phte_kernel_select(const char *name, int *out_id)
     const char *phte_kernel_names()
+    int phte_kernel_default()
     int PHTE_K_EVP
     void tp_free(TP_CTX *ctx)
     const char *tp_kind(const TP_CTX *ctx)
@@ -118,12 +119,15 @@ cdef int _select_kernel() except -1:
 
     One selector for all three AES chunkers: they share phte_scan.h, so the
     available paths never differ between them.
+
+    Unset means the fastest path this build and CPU support, see
+    phte_kernel_default().
     """
-    cdef int kid = PHTE_K_EVP
+    cdef int kid
     cdef int rc
     want = requested_kernel("BORG_AES_CHUNKER_KERNEL")
     if want is None:
-        return PHTE_K_EVP
+        return phte_kernel_default()
     rc = phte_kernel_select(want.encode("ascii"), &kid)
     if rc != 0:
         raise kernel_error("BORG_AES_CHUNKER_KERNEL", want, rc,

src/borg/testsuite/chunkers/fastcdc_test.py

@@ -1,7 +1,9 @@
 from hashlib import sha256
 from io import BytesIO
 import os
+import platform
 import random
+import sys
 
 import pytest
 
@@ -196,7 +198,7 @@ def test_fastcdc_kernels_identical(kernel, monkeypatch):
     # Every scan kernel this platform accepts must produce identical cut points.
     # Kernels this build/CPU cannot run are skipped, so the same test covers
     # whatever tier the machine happens to have - including kernels that exist
-    # but are not auto-selected, which nothing else would exercise.
+    # but are not the default here, which nothing else would exercise.
     data = os.urandom(4 * 1024 * 1024)
 
     def sizes(chunker):
@@ -220,6 +222,51 @@ def test_fastcdc_portable_kernel_available(kernel, monkeypatch):
     assert ChunkerFastCDC(key0, 10, 16, 14, 2).kernel == kernel
 
 
+# What an unset BORG_*_KERNEL must resolve to, per architecture, most preferred
+# first: the first entry this build and CPU can actually run is the default.
+# Mirrors fc_kernel_default() / bz64_kernel_default() (fastcdc_impl.c,
+# buzhash64_impl.c) and phte_kernel_default() (phte_core.h).
+ROLLING_HASH_DEFAULTS = {"x86_64": ["scalar"], "aarch64": ["neon", "blockwise"]}
+AES_DEFAULTS = {"x86_64": ["vaes", "aes-ni", "evp"], "aarch64": ["aes-arm64", "evp"]}
+
+
+def default_kernel_arch():
+    """Which key of those tables applies here.
+
+    uname spells the same architecture differently per OS - x86-64 is "amd64"
+    on the BSDs and "i86pc" on illumos - while the C side just asks the
+    compiler for __x86_64__ / __aarch64__.
+    """
+    machine = platform.machine().lower()
+    if machine in ("x86_64", "amd64") or (machine == "i86pc" and sys.maxsize > 2**32):
+        return "x86_64"
+    if machine in ("aarch64", "arm64"):
+        return "aarch64"
+    return machine
+
+
+def expected_default_kernel(envvar, make, key, monkeypatch):
+    """The kernel <envvar> unset must give here: the first of this architecture's
+    preference order that this build and CPU can run.
+
+    Determined by asking for each candidate explicitly, so this checks the
+    fallback chain rather than just repeating whatever the default resolved to.
+    """
+    machine = default_kernel_arch()
+    if envvar == "BORG_AES_CHUNKER_KERNEL":
+        preference = AES_DEFAULTS.get(machine, ["evp"])
+    else:
+        preference = ROLLING_HASH_DEFAULTS.get(machine, ["blockwise"])
+    for kernel in preference:
+        monkeypatch.setenv(envvar, kernel)
+        try:
+            make(key)
+        except ValueError:
+            continue  # not in this build, or this CPU can not run it
+        return kernel
+    raise AssertionError(f"none of {preference} is usable for {envvar} on {machine}")
+
+
 @pytest.mark.parametrize("envvar", ["BORG_FASTCDC_KERNEL", "BORG_BUZHASH64_KERNEL", "BORG_AES_CHUNKER_KERNEL"])
 def test_kernel_env_rejects_unusable(envvar, monkeypatch):
     # A kernel that cannot run here must fail loudly instead of silently
@@ -227,10 +274,10 @@ def test_kernel_env_rejects_unusable(envvar, monkeypatch):
     # means to pin one kernel, into a measurement of a different one.
     from ...chunkers import ChunkerBuzHash64, ChunkerToeplitzAES
 
-    make, default = {
-        "BORG_FASTCDC_KERNEL": (lambda k: ChunkerFastCDC(k, 10, 16, 14, 2), "scalar"),
-        "BORG_BUZHASH64_KERNEL": (lambda k: ChunkerBuzHash64(k, 10, 16, 14, 4095, 2), "scalar"),
-        "BORG_AES_CHUNKER_KERNEL": (lambda k: ChunkerToeplitzAES(k, 10, 16, 14, 2), "evp"),
+    make = {
+        "BORG_FASTCDC_KERNEL": lambda k: ChunkerFastCDC(k, 10, 16, 14, 2),
+        "BORG_BUZHASH64_KERNEL": lambda k: ChunkerBuzHash64(k, 10, 16, 14, 4095, 2),
+        "BORG_AES_CHUNKER_KERNEL": lambda k: ChunkerToeplitzAES(k, 10, 16, 14, 2),
     }[envvar]
     key0 = hex_to_bin("ad9f89095817f0566337dc9ee292fcd59b70f054a8200151f1df5f21704824da")
 
@@ -238,12 +285,14 @@ def test_kernel_env_rejects_unusable(envvar, monkeypatch):
     with pytest.raises(ValueError, match="no-such-kernel"):
         make(key0)
 
-    # there is no automatic selection, so "auto" is not a kernel name either
+    # the default is picked without naming it, so "auto" is not a kernel name either
     monkeypatch.setenv(envvar, "auto")
     with pytest.raises(ValueError, match="auto"):
         make(key0)
 
-    # unset means the simplest implementation, on every platform
+    # unset means the kernel measured fastest on this architecture, falling back
+    # where this build or this CPU does not have it
+    default = expected_default_kernel(envvar, make, key0, monkeypatch)
     monkeypatch.delenv(envvar)
     assert make(key0).kernel == default