Add GFNI bit-transpose path - #182
Open
stdpain wants to merge 1 commit into
Open
Conversation
The AVX512 bit transpose (bshuf_trans_bit_byte / bshuf_shuffle_bit_eightelem)
peels the 8 bit-planes out of each 64-byte block with an 8-pass loop of
vpcmpb-to-mask + vpsllw. On CPUs with GFNI a single vgf2p8affineqb transposes
the 8x8 bit-matrix of every 64-bit lane, and vpermb (AVX512VBMI) regroups the
bytes -- ~4 ops in place of ~24.
A new USEGFNI path (gated on USEAVX512 && __GFNI__ && __AVX512VBMI__; every
AVX512-class CPU exposing GFNI also exposes VBMI) provides drop-in,
byte-identical replacements wired into bshuf_trans_bit_elem_AVX512 and
bshuf_untrans_bit_elem_AVX512. Output is bit-for-bit unchanged, so the
compressed format is preserved. using_GFNI() is exposed alongside the existing
using_* accessors (C and Python).
Validated on Zen4 (transform is vendor-independent):
- GFNI transpose byte-identical to TRANS_BIT_8X8 (all 64 basis vectors + random);
- full round trip for elem_size in {1,2,4,8,16}, aligned and unaligned sizes;
- compressed output byte-identical (same checksum) to the non-GFNI build.
Bit-transpose kernel: AVX2 ~5, AVX512 ~8, GFNI ~33-55 GB/s (~4-6x over the
AVX512 loop; AVX2->AVX512 is only ~1.5x -- same algorithm, wider registers).
End-to-end bshuf_decompress_lz4 improves ~1.3-1.7x, largest on high-entropy /
floating-point data where the transpose dominates decode time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The AVX512 bit transpose (bshuf_trans_bit_byte / bshuf_shuffle_bit_eightelem) peels the 8 bit-planes out of each 64-byte block with an 8-pass loop of vpcmpb-to-mask + vpsllw. On CPUs with GFNI a single vgf2p8affineqb transposes the 8x8 bit-matrix of every 64-bit lane, and vpermb (AVX512VBMI) regroups the bytes -- ~4 ops in place of ~24.
A new USEGFNI path (gated on USEAVX512 && GFNI && AVX512VBMI; every AVX512-class CPU exposing GFNI also exposes VBMI) provides drop-in, byte-identical replacements wired into bshuf_trans_bit_elem_AVX512 and bshuf_untrans_bit_elem_AVX512. Output is bit-for-bit unchanged, so the compressed format is preserved. using_GFNI() is exposed alongside the existing using_* accessors (C and Python).
Validated on Zen4 (transform is vendor-independent):
Bit-transpose kernel: AVX2 ~5, AVX512 ~8, GFNI ~33-55 GB/s (~4-6x over the AVX512 loop; AVX2->AVX512 is only ~1.5x -- same algorithm, wider registers). End-to-end bshuf_decompress_lz4 improves ~1.3-1.7x, largest on high-entropy / floating-point data where the transpose dominates decode time.
Performance
Measured on AMD Zen4 (Ryzen 9 8945HX). The transform is vendor-independent and
applies equally to Ice Lake / Sapphire Rapids.
Bit-transpose kernel (the stage this PR changes)
Throughput of the isolated bit transpose, all three code paths doing the same work:
The
AVX2 → AVX512step is only ~1.5× (same 8-pass algorithm, widerregisters); GFNI wins ~4–6× by changing the algorithm, not the width.
End-to-end
bshuf_decompress_lz4(elem_size = 8)The bit transpose is 43–84% of decode time (highest on incompressible data,
where LZ4 is close to
memcpy), so the kernel speedup carries through: