mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2026-10-12 02:15:47 +00:00
Daniël de Kok
c6071749db
Custom masks are padded to the shape `[batch_size, max_len, max_len]`. However, flashinfer expects an unpadded mask of the shape `[sum(q_len[i] * k_len[i] for i in range(batch_size)]`. This change unpads the custom mask (currently only used by Gemma 3) to this shape (assuming q_len == k_len, since we only use the custom mask during prefill).