Skip to content

Add GFNI bit-transpose path - #182

Open
stdpain wants to merge 1 commit into
kiyo-masui:masterfrom
stdpain:feature/gfni-bit-transpose
Open

Add GFNI bit-transpose path #182
stdpain wants to merge 1 commit into
kiyo-masui:masterfrom
stdpain:feature/gfni-bit-transpose

Conversation

@stdpain

@stdpain stdpain commented Jul 23, 2026

Copy link
Copy Markdown

The AVX512 bit transpose (bshuf_trans_bit_byte / bshuf_shuffle_bit_eightelem) peels the 8 bit-planes out of each 64-byte block with an 8-pass loop of vpcmpb-to-mask + vpsllw. On CPUs with GFNI a single vgf2p8affineqb transposes the 8x8 bit-matrix of every 64-bit lane, and vpermb (AVX512VBMI) regroups the bytes -- ~4 ops in place of ~24.

A new USEGFNI path (gated on USEAVX512 && GFNI && AVX512VBMI; every AVX512-class CPU exposing GFNI also exposes VBMI) provides drop-in, byte-identical replacements wired into bshuf_trans_bit_elem_AVX512 and bshuf_untrans_bit_elem_AVX512. Output is bit-for-bit unchanged, so the compressed format is preserved. using_GFNI() is exposed alongside the existing using_* accessors (C and Python).

Validated on Zen4 (transform is vendor-independent):

  • GFNI transpose byte-identical to TRANS_BIT_8X8 (all 64 basis vectors + random);
  • full round trip for elem_size in {1,2,4,8,16}, aligned and unaligned sizes;
  • compressed output byte-identical (same checksum) to the non-GFNI build.

Bit-transpose kernel: AVX2 ~5, AVX512 ~8, GFNI ~33-55 GB/s (~4-6x over the AVX512 loop; AVX2->AVX512 is only ~1.5x -- same algorithm, wider registers). End-to-end bshuf_decompress_lz4 improves ~1.3-1.7x, largest on high-entropy / floating-point data where the transpose dominates decode time.

Performance

Measured on AMD Zen4 (Ryzen 9 8945HX). The transform is vendor-independent and
applies equally to Ice Lake / Sapphire Rapids.

Bit-transpose kernel (the stage this PR changes)

Throughput of the isolated bit transpose, all three code paths doing the same work:

size AVX2 AVX512 GFNI GFNI vs AVX512
8 KB 6.5 GB/s 9.7 GB/s 55 GB/s 5.7×
64 KB 5.5 GB/s 8.5 GB/s 38 GB/s 4.5×
256 KB 5.3 GB/s 8.2 GB/s 38 GB/s 4.7×
1 MB 5.2 GB/s 7.8 GB/s 33 GB/s 4.2×

The AVX2 → AVX512 step is only ~1.5× (same 8-pass algorithm, wider
registers); GFNI wins ~4–6× by changing the algorithm, not the width.

End-to-end bshuf_decompress_lz4 (elem_size = 8)

The bit transpose is 43–84% of decode time (highest on incompressible data,
where LZ4 is close to memcpy), so the kernel speedup carries through:

data ratio AVX512 GFNI speedup
high entropy 0.995 4.98 GB/s 8.64 GB/s 1.73×
float (noisy) 1.21 4.38 GB/s 6.97 GB/s 1.59×
low entropy 7.6 3.64 GB/s 5.35 GB/s 1.47×
sorted ids 20.9 2.75 GB/s 3.62 GB/s 1.32×

The AVX512 bit transpose (bshuf_trans_bit_byte / bshuf_shuffle_bit_eightelem)
peels the 8 bit-planes out of each 64-byte block with an 8-pass loop of
vpcmpb-to-mask + vpsllw. On CPUs with GFNI a single vgf2p8affineqb transposes
the 8x8 bit-matrix of every 64-bit lane, and vpermb (AVX512VBMI) regroups the
bytes -- ~4 ops in place of ~24.

A new USEGFNI path (gated on USEAVX512 && __GFNI__ && __AVX512VBMI__; every
AVX512-class CPU exposing GFNI also exposes VBMI) provides drop-in,
byte-identical replacements wired into bshuf_trans_bit_elem_AVX512 and
bshuf_untrans_bit_elem_AVX512. Output is bit-for-bit unchanged, so the
compressed format is preserved. using_GFNI() is exposed alongside the existing
using_* accessors (C and Python).

Validated on Zen4 (transform is vendor-independent):
- GFNI transpose byte-identical to TRANS_BIT_8X8 (all 64 basis vectors + random);
- full round trip for elem_size in {1,2,4,8,16}, aligned and unaligned sizes;
- compressed output byte-identical (same checksum) to the non-GFNI build.

Bit-transpose kernel: AVX2 ~5, AVX512 ~8, GFNI ~33-55 GB/s (~4-6x over the
AVX512 loop; AVX2->AVX512 is only ~1.5x -- same algorithm, wider registers).
End-to-end bshuf_decompress_lz4 improves ~1.3-1.7x, largest on high-entropy /
floating-point data where the transpose dominates decode time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant