serialize.rs

A simple bitpacking serializer for Rust.

This is a port of the C++ serialize library, bit-for-bit wire compatible with it and with the Go port (goserialize): a golden wire format test pins the exact bytes, copied verbatim from the C++ test suite, and on every push and pull request CI builds the real C++ library and verifies head-to-head that both implementations write byte-identical data and decode each other's output. Packets written by any of the three libraries decode in the others. Zero dependencies, no unsafe code, MBSL.

Values are packed with exactly the number of bits they need: a bool takes 1 bit, an integer in [0,31] takes 5 bits. Write one serialize function and it handles write, read and measure — the stream type is a generic parameter, so the branches are resolved at compile time, exactly like the C++ library's templated serialize methods:

use serialize::{Stream, WriteStream, ReadStream, Result};

struct Packet {
    position: i32,
    health: i32,
    alive: bool,
}

impl Packet {
    fn serialize<S: Stream>(&mut self, stream: &mut S) -> Result {
        stream.serialize_int(&mut self.position, -1000, 1000)?;
        stream.serialize_int(&mut self.health, 0, 100)?;
        stream.serialize_bool(&mut self.alive)?;
        Ok(())
    }
}

See examples/packet.rs for a fuller example with nested objects, variable length arrays and measuring.

Reading untrusted data

The read path is the trust boundary. Every read is bounds checked and range validated at runtime and fails with an Error instead of panicking — malicious packet data never panics. The ? operator aborts the entire serialize function on the first error, so a value that controls a loop (a count, a length) is always validated before it drives anything. This is the Rust rendering of the C++ library's early-return serialize macros and the Go port's sticky errors, and it is the reason serialize methods take &mut values and return Result.

The write path is trusted, like the C++ library: correctness is checked with debug assertions, and in release it is the caller's responsibility — size buffers conservatively or pre-measure with MeasureStream (its estimate is guaranteed conservative). Writing past the end of a buffer panics via the slice bounds check rather than being undefined behavior.

Panics are reserved for API misuse: bits out of [1,32]/[1,64], min >= max, a write buffer that is not a multiple of 8 bytes.

Buffer contracts

• Write buffers must be a multiple of 8 bytes. The writer flushes 64 bit words to memory (half as many flushes as a 32 bit design). Bytes past the written data are only ever written as zeros.
• Give the reader slack for full speed. The reader loads 64 bit windows at byte granularity. ReadStream::new(buffer, bytes) takes the full buffer plus the packet length: when the buffer extends at least 8 bytes past the packet data, every load stays on the branchless fast path (the same trick the Go port plays with slice capacity). Without slack, loads near the end fall back to a guarded copy — correct, just slower.

Differences from the C++ library

• Errors instead of return false: serialize functions return Result and propagate with ?. No macros needed.
• serialize_string operates on String and validates UTF-8 on read (C++ strings are raw bytes, so only valid UTF-8 interoperates). serialize_wide_string matches the wchar_t wire format (32 bits per code point) and validates code points on read.
• The stream context is &dyn Any instead of void*. There is no allocator pointer — Rust serialize functions can carry whatever state they need.
• Buffer sizes and bit counts are u64 internally, matching the C++ library's 64 bit bookkeeping (buffers past 256 MB round trip; the test suite proves it).

Performance

cargo bench runs benches/throughput.rs, a direct port of the C++ library's bench.cpp with identical methodology (same mixed bit-width table, same packet, same LCG-varied fields, escape barriers, best of five trials). Apple M3 Ultra, single core, Rust 1.97 vs clang -O3:

	serialize.rs	C++ serialize
bitpacker write	2.4 GB/s	5.8 GB/s
bitpacker read	2.6 GB/s	7.9 GB/s
stream write	4.2 GB/s (92 M pkt/s)	2.1 GB/s (47 M pkt/s)
stream read	18.7 GB/s (410 M pkt/s)	6.5 GB/s (144 M pkt/s)
stream measure	1352 M pkt/s	805 M pkt/s

The stream rows are the numbers that matter — that is the API. With the serialize function monomorphized, every field's bit width is a compile-time constant, the asserts and masks fold away, and the branchless reader's no-dependency design lets the CPU overlap the whole decode: the Rust stream path comes out 2-3x faster than the C++ build on the same machine. The raw bitpacker rows use runtime-variable bit widths from a table — the worst case for the safe path, where the per-call validation and bounds-checked loads that replace the C++ library's unchecked memory access cost about 2.5x (this crate is forbid(unsafe_code); that's the trade, measured).

Tests

cargo test                                   # the C++ suite, ported, plus differential tests
cargo test --release -- --include-ignored    # includes the 320 MB large buffer test
cargo clippy --all-targets -- -D warnings    # pedantic, configured via [lints] in Cargo.toml
cargo fmt --check
cargo +nightly miri test                     # the whole suite under Miri
cargo +nightly fuzz run hostile_read         # libFuzzer (also: round_trip)

The test suite mirrors serialize.h test-for-test, including the adversarial cases (out-of-range values smuggled into bit headroom, full-range integers, NaN handling, >2^31 relative gaps) and the golden wire format test. tests/differential.rs adds a deterministic differential write→read round trip and a hostile read pass, and fuzz/ carries the same two passes as real libFuzzer targets, mirroring the C++ library's fuzz harness.

CI runs the test matrix on Linux/macOS/Windows (debug and release), pedantic clippy, rustfmt, rustdoc, an MSRV (1.85) check, the whole suite under Miri, 60 seconds of each fuzz target, a zero-dependency guard, a big-endian s390x run under qemu, and cargo semver-checks on pull requests. The crate is #![forbid(unsafe_code)], enforced by the compiler.

License

Más Bandwidth Source License (MBSL) — BSD 3-Clause plus one credit clause, described under Crediting below.

Note that this is not the same licence as the C++ serialize library, which remains BSD 3-Clause. The two are separate projects with separate licences.

Crediting

This library is licensed under the Más Bandwidth Source License (MBSL), which is BSD 3-Clause plus one clause: products that incorporate it must include this credit in their product credits, or in their documentation:

serialize.rs by Glenn Fiedler and Rowan Claude

Free to use, source open, credit required. Fair credit keeps open source honest.