Niko1221/Strata
1
0 Fork 0
mirror of https://github.com/Niko1221/Strata.git synced 2026-11-07 14:21:06 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

Dense dequant: the wide kernels by default on sm_80 too (bitwise)

Browse Source 
Measured on a CMP 170HX (GA100, 70 SMs): the prompt path dequantizes the dense Q4_K/Q5_K/Q6_K/IQ4_XS weights every
chunk. With the wide kernels (STRATA_DEQUANT_WIDE=1, the sm_61 / sm_86 default): GPU time of a 3,259-token prompt
857 -> 792 ms; prompts 3.3K +6.3..+7.2%, 13K +4.7%, 52K +3.9%; greedy output identical (11 of 11 prompts).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
linxuhaoandClaude Opus 5.5 committed 2026-10-22 20:52:26 +00:00
1 parent 011c9e600a
commit 48dc535e77
1 file changed
+4 -2
Download Patch File Download Diff File Expand all files Collapse all files
src/kernels/cuda/dequant_bf16.cu
+4 -2
View File
@@ -285,14 +285,16 @@ bool dequant_wide_on() {
static const int env = [] { const char* v = std::getenv("STRATA_DEQUANT_WIDE"); return v != nullptr && v[0] != '\0' ? (std::atoi(v) != 0 ? 1 : 0) : -1; }();
if (g_dequant_wide >= 0) return g_dequant_wide != 0;
if (env >= 0) return env != 0;
static int per_dev[64]; // 0 unknown, 1 no, 2 yes: sm_61 (measured on a Tesla P40) and sm_86 (RTX 3060 / A4000, 0.1.45: short-prompt TTFT -12..-17%, bitwise)
// 0 unknown, 1 no, 2 yes: sm_61 (measured on a Tesla P40), sm_86 (RTX 3060 / A4000, 0.1.45: short-prompt TTFT -12..-17%,
// bitwise) and sm_80 (CMP 170HX: a 3,259-token prompt 857 -> 792 ms of GPU time, prompts +4-7%, bitwise)
static int per_dev[64];
int dev = 0;
if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) { (void) cudaGetLastError(); return false; }
if (!per_dev[dev]) {
int major = 0, minor = 0;
if (cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, dev) != cudaSuccess ||
cudaDeviceGetAttribute(&minor, cudaDevAttrComputeCapabilityMinor, dev) != cudaSuccess) { (void) cudaGetLastError(); return false; }
per_dev[dev] = (major == 6 && minor == 1) || (major == 8 && minor == 6) ? 2 : 1;
per_dev[dev] = (major == 6 && minor == 1) || (major == 8 && (minor == 6 || minor == 0)) ? 2 : 1;
}
return per_dev[dev] == 2;
}