Add realtime_decoding_demo example: drive the delivered decoding server from an FPGA or a QPU kernel#703
Open
cketcham2333 wants to merge 51 commits into
Open
Add realtime_decoding_demo example: drive the delivered decoding server from an FPGA or a QPU kernel#703cketcham2333 wants to merge 51 commits into
cketcham2333 wants to merge 51 commits into
Conversation
…ng-server test Extend hsb_fpga_decoding_server_test.sh to drive the nv-qldpc Relay BP decoder through the decoding server's gpu_roce transport -- the self-relaunching device-graph scheduler (GpuRoceTransceiver): enqueue_syndromes / get_corrections / reset_decoder execute as DEVICE_CALLs on the GPU and the captured RelayBP decode graph fires device-side (fire-and-forget + tail-self-relaunch) when a window completes. This replaces hololink_qldpc_graph_decoder_bridge for the decoding-server flow; the old bridge test is untouched. hsb_fpga_decoding_server_test.sh: - --transport cpu_roce|gpu_roce, defaulted from the decoder profile (pymatching -> cpu_roce HOST_CALL path, nv-qldpc-decoder -> gpu_roce), plus --gpu, --proprietary-archive, --nv-qldpc-plugin. - Data generation (surface_code-4-yaml, fresh each run) adds --use-relay-bp for the nv-qldpc profile and injects `transport: gpu_roce` into the generated YAML: DecoderServer selects its transceiver from the per-decoder transport key, the generator omits non-default optionals, and the default (cpu_roce) resolves to the not-yet-implemented CpuRoceTransceiver stub. - gpu_roce server launch: Hololink parameters go via HOLOLINK_* env (the server's gpu_roce mode ignores the cpu_roce CLI flags), remote QP converted to decimal ($((qp)) -- GpuRoceConfig::from_env parses base-10 only), CUDA_MODULE_LOADING=EAGER as with the old bridge launcher, and readiness keyed on "QEC_DECODING_SERVER_READY gpu_roce" (the transceiver prints the QP/RKey/Buffer handshake before that line, without the bridge banner). - Build phase: builds gpu_roce_transceiver (HSB) and cudaq-realtime-bridge-hololink (cuda-quantum), wires CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE / GPU_ROCE_TRANSCEIVER_LIB / CUDAQ_REALTIME_BRIDGE_HOLOLINK_LIBRARY into the cudaqx configure, clears the CMake cache (find_library NOTFOUND staleness), and symlinks the nv-qldpc plugin into build/lib/decoder-plugins. All gated on the proprietary archive's presence so pymatching-only rigs build unchanged. The plugin symlink is also created opportunistically on --build-less runs. decoding-server-cqr/CMakeLists.txt (bug fix, separable): - CUDAQ_GPU_ROCE_AVAILABLE was set as CACHE INTERNAL on success, but the plain `set(... FALSE)` at the top of the file shadows the cache under CMP0126 (NEW), so the compile-definition gate for cudaq-qec-decoding-server never fired: the library was silently built WITHOUT gpu_roce (make_transport threw "CUDAQ_GPU_ROCE_AVAILABLE is not set" at runtime) while the sibling decoding_server target -- which reads the cache -- reported "gpu_roce transport enabled". Set the normal variable TRUE alongside the cache entry. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
With CUDAQ_GPU_ROCE_AVAILABLE now actually reaching the library target (the CMP0126 normal-vs-cache shadow fix), cudaq-qec-decoding-server links libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so -- both of which reference the CUDA Driver API (cu*). Consumers of the static lib (test_decoding_server_core, test_decoders_yaml, decoding_server) inherit those dependencies, and on driverless build machines (CI containers: CUDA toolkit present, no GPU driver, no libcuda.so.1) their links fail with dozens of `undefined reference to cuStreamCreate` etc. -- the failure seen on all four QEC Build-and-test CI jobs. Link CUDA::cuda_driver alongside CUDA::cudart in the gpu_roce block: the CMake target resolves to the real driver where one is installed and to the toolkit's stubs (lib/stubs/libcuda.so) in driverless environments. This matches the existing precedent in unittests/utils (playback and the old bridge already link CUDA::cuda_driver). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
With CUDAQ_GPU_ROCE_AVAILABLE actually reaching the library target (the CMP0126 fix), every consumer of cudaq-qec-decoding-server (test_decoding_server_core, test_decoders_yaml, the CQR plugin) inherited runtime dependencies on libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so, which require libcuda.so.1 at LOAD time. On driverless CI machines (CUDA toolkit + HSB + DOCA present, no GPU driver) all four QEC Build-and-test jobs failed: gtest_discover_tests executes the test binaries at build time and the loader cannot satisfy libcuda.so.1 -- link-time driver stubs satisfy ld but not the runtime loader. Restructure so the core library carries no GPU RoCE dependencies while keeping full CI build coverage of the HSB path: - New optional component cudaq-qec-decoding-server-gpuroce holds GpuRoceTransceiver.cpp plus GpuRoceFactory.cpp, the strong definition of the factory (cudaqx_qec_make_gpu_roce_transceiver) that DecodingServer.cpp now references weakly. All DOCA / hololink / CUDA-driver link deps move to this component. - make_transport(gpu_roce) calls the factory when it is linked in and throws a clear "GPU RoCE support is not linked into this binary" error otherwise; the ifdef and the direct GpuRoceTransceiver construction are gone from the core. - The scheduler hookup no longer dynamic_casts to the concrete type: ITransceiver grows a launch_device_scheduler(void*) virtual (default: no device scheduler), overridden by GpuRoceTransceiver to forward to launch_scheduler(). The config-driven constructor keys the hookup on the transport enum instead. - decoding_server links the component WHOLE_ARCHIVE in its proprietary-archive-gated gpu_roce block -- a weak reference does not pull archive members. - A link canary (cudaq-qec-decoding-server-gpuroce-linkcheck; never executed, not installed, not a test) preserves CI link validation of the component: a static archive alone never resolves symbols, and the one real consumer (the daemon's gpu_roce block) is gated on the proprietary cudevice archive, which CI does not provision. The canary forces full link resolution against the hololink / DOCA libraries with the driver stubs -- the same check that used to happen incidentally via the test binaries. - decoding_server's QEC_HAVE_GPU_ROCE_TRANSPORT gate is split from the proprietary-archive gate: the gpu_roce CLI branch references only the (now dependency-free) core DecodingServer API, so it compiles and links whenever HSB/DOCA are present -- CI now builds that branch too. The component + cudevice whole-archive link stays behind the archive gate, so the CI daemon remains loadable on driverless runners. Branch [2a] wraps server startup in try/catch, turning what used to be std::terminate on any gpu_roce bring-up failure into an "ERROR: gpu_roce startup failed: ..." message with exit 1. Net CI coverage: GpuRoceTransceiver.cpp compiles against the HSB/DOCA headers (component builds in `all`), its link resolves (canary), and the daemon's gpu_roce CLI branch compiles and links (split gate). No binary CI executes carries a libcuda.so.1 dependency. On dev rigs the daemon keeps full gpu_roce. As a bonus, the CQR plugin .so is loadable on driverless machines again (it too had inherited the deps). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The server now contains no transport-specific code. --transport=<name> resolves to libcudaq-realtime-bridge-<name>.so next to the CUDA-Q realtime libraries (udp and cpu_roce ship there, incl. the TCP rendezvous and hsb_fpga QP exchange that previously lived inline here); --transport=/path/to/lib.so loads a partner's out-of-tree provider verbatim with no changes to this server. Unrecognized CLI arguments are forwarded to the provider's create(), so the existing flags (--port/--num-slots/--slot-size/--device/--local-ip/--qp_config/ --peer-ip/--remote-qp/--frame-size) keep working. Dispatch moves from a hand-rolled thread around cudaq_host_ring_dispatch_loop to the libcudaq-realtime dispatcher object (create/set_ringbuffer/set_function_table/set_control/start), the supported surface after cuda-quantum PR 4869; ring geometry comes from the provider's geometry query and readiness from its endpoint query (READY line format unchanged: port= hoisted first for the sscanf in test_decoding_server.cpp). gpu_roce is untouched: it predates the provider interface and takes a different dispatch shape (device-side scheduler; PR 670 is active in that code). CMake: the server no longer links the udp/cpu_roce transports, the host-dispatch archive, or ibverbs -- providers are runtime artifacts. Requires a CUDA-Q realtime install with bridge interface v2 (endpoint info + ring geometry) and the provider libraries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The Hololink transceiver bring-up moves behind the CUDA-Q realtime transport-provider interface: the constructor loads the provider (built-in libcudaq-realtime-bridge-hololink.so, or the library named by CUDAQ_REALTIME_BRIDGE_LIB) and adopts its RING_BUFFER context, geometry, and endpoint identity (v2 queries, with config-derived and banner fallbacks for a v1 provider). launch_scheduler()'s device-graph scheduler wiring is unchanged; the provider's launch() now owns the Hololink monitor thread, and disconnect()/destroy() replace direct hololink_close/destroy calls. Consequence: cudaq-qec-decoding-server no longer links Hololink / DOCA / HSB / ibverbs at all -- CUDAQ_GPU_ROCE_AVAILABLE now gates only on the CUDA-Q realtime headers, libcudaq-realtime.so, and the CUDA toolkit, so the gpu_roce path compiles on machines with none of the RDMA stack installed and fails loudly at runtime when the provider .so is absent. (Also applies PR 670's CMP0126 cache-shadowing fix so the compile definition actually fires.) Runtime validation on HSB rigs still required: this box has no DOCA / HSB, so the provider .so cannot be built or exercised here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
# Conflicts: # libs/qec/lib/realtime/decoding-server-cqr/CMakeLists.txt # libs/qec/tools/decoding-server/CMakeLists.txt Signed-off-by: Ben Howe <bhowe@nvidia.com>
…roce Follow-up to the bridge-provider refactor + PR 670 merge, closing the naming leaks that made the device path look like (and require the vocabulary of) a specific wire: - GpuRoceTransceiver/GpuRoceConfig -> DeviceGraphTransceiver/-Config (files, class, weak factory symbol cudaqx_qec_make_device_graph_transceiver, component cudaq-qec-decoding-server-device-graph, defines CUDAQ_QEC_DEVICE_GRAPH_AVAILABLE / QEC_HAVE_DEVICE_GRAPH_DISPATCH). It never was a transport: it is the GPU device-graph dispatch engine riding whatever bridge provider is loaded. - Per-decoder YAML key is now `dispatch: host|device_graph` (DecoderDispatch), naming the dispatch SHAPE instead of a wire. The legacy `transport: cpu_roce|gpu_roce` key and spellings remain input-only aliases. Fixes a subtle parser bug found while testing: the alias mapping must use two-arg mapOptional, or an absent legacy key resets the parsed dispatch value to the default. - Config env vars are QEC_DEVICE_GRAPH_* with the HOLOLINK_* spellings as fallback (values are forwarded to whatever provider is loaded, which need not be Hololink; existing orchestration scripts keep working). - decoding_server routes to the device-graph path from the YAML's declared dispatch shape, not from --transport=gpu_roce (kept as a legacy alias). The CLI/YAML mismatch that used to silently select a stub transceiver is now a loud, specific error, and a non-legacy --transport value selects the provider for the device path the same way it does for the host path. Validated: udp two-process suite 5/5, core 5/5; dispatch: device_graph, legacy transport: gpu_roce + HOLOLINK_* env, and the mismatch error all exercised end-to-end (device path fails at provider load on this box, as expected without the hololink provider .so). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…split Everything here is new in this development cycle with no deployed consumers, so the transition aliases go: - YAML: the per-decoder legacy key `transport: cpu_roce|gpu_roce` and the wire-named enum spellings are gone; `dispatch: host|device_graph` is the only form. An old config now fails at parse time with "unknown key 'transport'" (and the server reports YAML parse errors cleanly instead of terminating on the exception). - CLI: `--transport=gpu_roce` is gone; --transport only ever names the wire/provider, and the device-graph path is selected solely by the config's dispatch key (default provider: built-in hololink). READY sentinel for that path is now `QEC_DECODING_SERVER_READY device_graph`. - Env: QEC_DEVICE_GRAPH_* only; the HOLOLINK_* fallback spellings are removed. - DeviceGraphTransceiver now REQUIRES a bridge-interface-v2 provider (ring geometry + endpoint identity queries); the v1-provider fallback paths are gone. - CpuRoceTransceiver (a stub whose constructor always threw) is deleted; DecoderDispatch::host in the standalone DecodingServer path throws a direct, accurate error instead. - hsb_fpga_decoding_server_test.sh updated to the new spellings (dispatch: device_graph injection, QEC_DEVICE_GRAPH_* env, no --transport for the device path, new READY sentinel). Validated: udp two-process suite 5/5, core 5/5; device path reached via dispatch: device_graph with QEC_DEVICE_GRAPH_* env; removed spellings all fail loudly (unknown YAML key; HOLOLINK_* env ignored with a clear missing-variable error). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…n plan) Records the bridge-provider refactor end to end so it can be reproduced from the document alone: problem statement, after-architecture with the device-link vs dlopen design rules, the v2 bridge interface contract and the three providers, the server CLI/YAML/env contracts, the dispatch-vs- wire split, the ordered reproduction plan across both repos, validation gates (including the known pre-existing failure), and recorded follow-ups (rig validation, upstreaming, per-decoder rings, fan-in, vp_id). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…r_id The QEC device wrappers now route every decoding RPC through cudaq::device_call's device-id overload with device_id == decoder_id. The CUDA-Q device_call runtime keys sessions, rings, and dispatchers by device id, so each decoder gets its own ring buffer and dispatcher: no shared ring, no head-of-line blocking between decoders, per-decoder backpressure. The payload keeps decoder_id as a cross-check (and for future multi-source fan-in). The CQR service plugin needs no change for the host path: per-device sessions share the same HOST_CALL handlers and the registry still routes by payload decoder_id. New app example surface_code-5-per-decoder-rings (+ test script, registered in ctest): two pymatching decoders decoded over two rings by one kernel, verified by (1) per-decoder corrections + dispatch count, (2) two per-device session-init lines in the runtime log, and (3) the per-device channel override (CUDAQ_DEVICE_CALL_CHANNEL= host_dispatch,1=device_dispatch) steering device 1 to the GPU channel while device 0 stays on its host ring. Pairs with cuda-quantum commit '[device-call] Per-device sessions on demand + per-device channel spec' (lazy per-device init; channel spec syntax; external channels keep one shared endpoint). Design doc per_decoder_rings_design.md records the topology, the CPU-ring + GPU-ring channel matrix, the recipe for the plugin's Gpu-mode session (the remaining piece for simultaneous CPU+GPU rings), and follow-ups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The server now opens ONE provider instance per decoder in the YAML -- own endpoint, own ring buffer, own dispatcher -- completing the two-process form of the one-ring-per-decoder topology (the product path; the earlier in-process demo only proved the runtime mechanism). Wire contract additions (single-decoder output unchanged apart from a ring0 token): - READY: 'QEC_DECODING_SERVER_READY port=<P0> transport=udp ring0=<P0> ring1=<P1> ...' -- leading port stays ring 0's for existing single-endpoint consumers, which keep working (all traffic lands on ring 0, payload-demuxed). - Shutdown: one 'QEC_DECODING_SERVER_RING decoder=<id> dispatched=<n>' line per ring, sampled AFTER dispatcher_stop (the loop flushes its stats counter on exit). New test DecodingServerTwoProcess.TwoProcessPerDecoderRings: two decoders, two udp endpoints, one caller wiring each decoder's device_call session to its own ring via device-scoped channel args (udp-port=<P0> udp-port.1=<P1>); asserts correct corrections AND >=3 dispatches on EACH ring -- proving per-ring traffic, not one shared wire. Full suite 6/6. Pairs with cuda-quantum commits 'Register bridge handles created through the cached-provider path' (second bridge instance on one provider was unusable) and 'Per-device external channel endpoints'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
At run time the server is N independent 1-ring consumers constructed by
one setup loop: every decoder gets one bridge provider instance and one
ring (same CQR RPC wire format), and the decoder's dispatch: shape
selects the ring's consumer -- a host dispatcher-object thread, or the
CUDAQ device-graph scheduler. Host decoders and a device_graph decoder
now run simultaneously in one server; an all-device_graph config still
takes the standalone DecodingServer (HSB) path.
- DeviceGraphRingConsumer: the device-graph scheduler as a ring
CONSUMER (3 proprietary DEVICE_CALL entries + the decoder's captured
decode graph over a ring it does not own), extracted from
DeviceGraphTransceiver (which now delegates to it). Exposed through
a weak C ABI so server builds without the proprietary component link
and fail loudly at runtime instead.
- SessionRegistry accepts mixed dispatch shapes; the single-transceiver
DecodingServer paths still require uniformity (required_dispatch()
throws on mixed).
- CQR plugin exports cudaqx_qec_decoding_server_graph_resources(id) so
the server can wire the scheduler to a decoder hosted behind the
plugin (graphs are captured at session creation, before READY).
- Decoder YAML gains an optional per-decoder transport override
('transport: "udp --pinned-rings"', or 'hololink' for the builtin
provider slot). One EXTERNAL provider library per process (loader
limitation) is diagnosed with a clear error.
Validated locally: mixed config brings up ring0 (host) + ring1 (pinned
udp), READY publishes both rings, the non-graph decoder fails loudly at
the graph-capture requirement with clean teardown; full two-process
suite 6/6, per-decoder-rings example green. Positive decode on the
device ring needs a graph-capable decoder built against this tree's
ABI (see design doc follow-ups).
Pairs with cuda-quantum commits 'UDP transport external-rings API +
provider --pinned-rings' and 'Do not re-export transport-archive
symbols from the runtime' (ODR fix this work flushed out).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Replaces the per-decoder transport override added with the mixed-
dispatch server: the wire is deployment configuration and lives OUTSIDE
the decoders list. Transports differ between rings only by dispatch
shape (a device_graph ring must be GPU-pollable), so the top-level YAML
transport: section carries provider/args plus one shape-keyed
device_graph: override -- decoder entries carry no transport
information and stay portable across environments.
transport:
provider: udp
device_graph:
args: [--pinned-rings]
Per-ring precedence: shape override > section provider/args >
--transport CLI default; an explicit --transport still overrides the
section's provider for one-off experiments.
Validated: mixed config brings up both rings from the YAML alone (no
transport CLI); full two-process suite 6/6; per-decoder-rings example
and device_call tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
… + WSL2 wall The sc4-cqr external channel guard wires per-decoder device_call sessions to their own rings via QEC_DECODING_SERVER_PORT_<id> -> udp-port.<id>= scoped args. Design doc records the positive-validation campaign: nv-qldpc rebuilt against this ABI (cuda-qx scratch merge), two-process per-decoder-ring CONTROL run decodes correctly with two host nv-qldpc rings; the mixed run's GPU scheduler serves DEVICE_CALLs over pinned-udp rings end to end (raw RPC datagram probes: reset + full-window enqueue round-trip status=0); the remaining decode-graph firing step is blocked by a platform limitation isolated with a minimal CUDA probe -- device-side cudaStreamGraphFireAndForget launches silently never execute on this WSL2 stack -- so the final step runs on the rig. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…aunch The wedge is not a silent no-op: launching any kernel referencing device-side cudaGraphLaunch fails with cudaErrorNotSupported, visible only via cudaGetLastError() immediately after the launch (sync reports success). Documents the 10-second gating check for rig/CI machines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…s exonerated Rig data showed the original probe failing on target hardware too, which exposed the probe's API-contract violation: fire-and-forget device graph launch is only legal from a kernel inside a device-launched graph; a plain trigger kernel correctly gets cudaErrorNotSupported everywhere. The corrected two-graph probe (trigger inside a device-launched parent) PASSES on WSL2, native SASS and forced-JIT alike, and artifact arch coverage is ruled out. The wedge is a real pipeline issue, locally reproducible; next suspects recorded (triggered-graph instantiation flags/upload, tail relaunch, decode graph execution), plus the missing device-side error surfacing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Three-way split with a sharpened boundary: - transport_provider_design.md now owns ALL deployment contracts in one place: alongside the provider ABI, section 4.2 gains the top-level YAML transport section (shape-keyed override + precedence), the multi-ring READY tokens and per-ring shutdown lines, the N-ring server flow, and the caller-side per-ring endpoint contract (key.<id>= scoped args, channel spec). Stale follow-ups refreshed. - per_decoder_rings_design.md slims to pure topology (rings, consumers, dispatch shapes, mixed server), pointing at the provider doc for contracts; the validation campaign narrative is replaced by a short status section. - per_decoder_rings_validation_notes.md (new) is the explicit campaign log: what is validated, the open decode-graph wedge with ordered suspects, the probe methodology including the invalid-probe lesson, the proprietary-rebuild recipe, and the build/runtime gotchas. No code changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Add transport_provider_design.md section 4.3 with complete minimal recipes for the nine deployment shapes covered by the 4.2 contracts (udp loopback, per-decoder rings, cpu_roce rendezvous + hsb_fpga, mixed dispatch local/HSB, all-device_graph, partner drop-in, in-process channels). Renumber the two-axis-split section to 4.4 and cross-reference the cookbook from per_decoder_rings_design.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Root cause of the device-graph scheduler wedge: cooperative-launch co-residency starvation. The decode graph was captured with reserved_sms=0, sizing its cooperative grid for every SM; with the persistent dispatch graph resident, the device-side fire-and-forget launch queued forever and deadlocked at grid.sync(), stalling the tail self-relaunch. DecodingSession now captures with reserved_sms=1 (QEC_DECODE_GRAPH_RESERVED_SMS overrides, e.g. on rigs where Hololink RX/TX kernels are also resident). Same bug class, second instance (documented, plugin-side fix pending): HOST-path nv-qldpc decode launches unreserved cooperative kernels and hangs when a scheduler is resident -- in mixed deployments put a CPU decoder on host rings, or plumb reservation into the host decode. DeviceGraphRingConsumer logs the new trigger diagnostics (cudaq_dispatch_get_trigger_debug) at shutdown, and dispatched() now uses an async copy on a non-blocking stream -- a legacy default-stream cudaMemcpy synchronizes against the persistent scheduler graph and self-deadlocks. Proof (two-process, pinned-udp device ring, WSL2 laptop): sc4 app, 10 shots at p_spam=0.08, multi_error_lut host ring + nv-qldpc RelayBP device ring: trigger rc=0, fires=12 == tail_relaunches, both rings dispatched=72, decoder[1] corrections=2 logical_errors=0/10, clean teardown. Full regression suites stay green. Pairs with cuda-quantum commit 'Surface the device-side trigger-launch result'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…ss limit Pairs with cuda-quantum '[realtime] Key the bridge-provider loader by library name/path'. The server now passes each ring's resolved provider library to cudaq_bridge_create_from_library directly: no more CUDAQ_REALTIME_BRIDGE_LIB setenv side-channel for the per-ring path, no 'hololink is the builtin slot' special case, and rings may mix provider libraries freely -- the corresponding startup error and design-doc limitation are removed. DeviceGraphTransceiver likewise names its default provider library (hololink) as a plain string, with CUDAQ_REALTIME_BRIDGE_LIB still honored as the replacement override. Also note in --help that the installed cudaq-realtime is the source of truth for available providers and their arguments; the enumerated names and args are examples. Validated: two-process suite 4/4, per-decoder-rings app test, and the full mixed device-graph E2E (trigger rc=0, fires=10=tail_relaunches, both rings dispatched=60, real corrections on both decoders). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
decoding_server.cpp: - Set transport_from_cli when --transport is parsed: the documented "explicit --transport overrides the YAML transport section" precedence was dead code (the section's provider always won). - The all-device_graph path now resolves its provider with the same precedence as the per-ring loop (CLI > section device_graph override > section provider > built-in default) instead of consulting only --transport != udp. - The v1-provider endpoint fallback advertises the ring's own resolved provider, not the process default. - Guard --timeout and the provider port token against non-numeric input (clean error instead of an uncaught std::stoi/stoul exception). - Include DeviceGraphRingConsumer.h so the weak C-ABI redeclarations are compile-checked against the canonical prototypes. CQR components: - Consolidate the SM-reservation knob on the documented device-graph env family: DecodingSession reads QEC_DEVICE_GRAPH_RESERVED_SMS (was the undocumented QEC_DECODE_GRAPH_RESERVED_SMS), and the transceiver's parsed-but-unused reserved_sms field is removed. - Delete dead helpers left behind when the scheduler wiring moved into DeviceGraphRingConsumer (alloc_pinned_mapped, populate_device_call, GPU_CUDA_CHECK) -- they produced -Wunused-function noise. - DeviceGraphRingConsumer.h self-compiles with the component macro off (the C-ABI block's includes were inside the #ifdef). - Free the function table, shutdown flag, and stats allocations when the scheduler-stream create fails (the one error path without a cleanup ladder). - Fix a stale HOLOLINK_FRAME_SIZE error message (QEC_DEVICE_GRAPH_FRAME_SIZE) and a stale SessionRegistry comment; correct the READY-line doc: the leading port= token belongs to the first decoder listed in the YAML, not necessarily decoder id 0. Validated: two-process suite 4/4, per-decoder-rings app test, and the mixed device-graph E2E (trigger rc=0, fires=10=tail_relaunches, both rings dispatched=60, correct RelayBP decodes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
… path The wire's identity lives in the YAML transport section; --transport now applies only to configs that intentionally leave the wire unspecified (one YAML reused across wires, selected per launch -- the pattern every two-process test uses). A config that names a provider combined with an explicit --transport is rejected at startup instead of resolved by a precedence rule, in both the mixed per-ring path and the standalone all-device_graph path. Help text and design docs updated to match. New coverage for the YAML path, which previously had none: - DecoderYAMLTest.TransportSectionAndMixedDispatch: round-trips the transport section (provider/args + device_graph shape override) with a host + device_graph decoder mix, and pins the exact YAML key spelling against a literal document. - DecodingServerTwoProcess.TwoProcessHostDispatchYamlTransportSection: full two-process decode where the server is launched WITHOUT --transport and the YAML names the provider and its args. - DecodingServerTwoProcess.TransportCliConflictsWithYamlSection: the conflict is rejected before READY with a nonzero exit and a pointed error message. - app_examples.surface_code-4-yaml-mixed-dispatch: the flagship mixed host+device_graph two-process flow as a gated ctest (registered when the server links the device-graph component; skips at runtime without a GPU or the nv-qldpc plugin). Asserts per-decoder rings on the READY line, per-decoder results in the app, a healthy scheduler (trigger rc=0, fires == tail_relaunches > 0), and traffic on both rings. ServerProcess (test harness) learns to launch without the CLI transport args, to fold stderr into the captured output, to report the child's exit code, and to bail out on EOF instead of burning the READY timeout. 29/29 in the decoding-server/YAML/session/app regression sweep, including the new tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The after-diagram now reflects the one-ring-per-decoder consumers and the YAML transport section (--transport as fallback, conflict rejected). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
…ridge branch Main landed the squashed PR 670 plus NVIDIA#656/NVIDIA#673/NVIDIA#674/NVIDIA#678/NVIDIA#680/NVIDIA#683/NVIDIA#688/ NVIDIA#690/NVIDIA#691/NVIDIA#692. Resolution keeps this branch's design and ports main's content into it: - Naming: main's GpuRoce{Transceiver,Factory,LinkCheck} map 1:1 onto this branch's DeviceGraph* files (the de-transport-ing follow-up requested in the PR 670 review). Main's post-review fixes are ported into the renamed files: ring-size overflow + host-page-size alignment validation, gpu_id resolved from the decoder's cuda_device_id instead of an env var, the factory taking pinned_cuda_device, and the linkcheck exercising the new factory signature. - Schema: per-decoder 'dispatch: host|device_graph' plus the top-level transport section stay; main's per-decoder 'transport:' key does not return. cuda_device_id is adopted as a per-decoder placement knob (config struct + YAML trait), and the hsb script now injects 'dispatch: device_graph' + cuda_device_id and drives the server with QEC_DEVICE_GRAPH_* env and the device_graph READY sentinel. - Features adopted from main: decoder CUDA-device pinning (worker-thread pin via promise, graph-capture pin, hardware_guards.h), per-session decode counters + print_session_stats + QEC_DECODING_SERVER_STATS server-side stats printing, DecodingServer ctor exception safety, virtualized realtime decoder API (NVIDIA#674), CMP0126 fix, CUDA::cudart on the core lib, and the cudevice-archive dedup in the server link. - DecodingSession combines main's set_graph_capture_device with this branch's reserved-SMs decode-graph capture (QEC_DEVICE_GRAPH_RESERVED_SMS). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Addresses PR 682 review feedback (the standalone transceiver was still hololink-shaped): - The transceiver no longer parses or formats wire-specific rendezvous tokens. It publishes the provider's interface-v2 endpoint description verbatim as one line -- 'QEC_DECODING_SERVER_ENDPOINT <key=value ...>' -- replacing the bespoke QP Number / RKey / Buffer Addr lines, and the qp/rkey/buffer_addr members and accessors are gone. The orchestration layer scrapes whatever tokens its wire uses (the HSB script now pulls qp=/rkey=/buffer_addr= from the endpoint line for the playback tool; a socket-style wire would pull port=). - Provider arguments are generic. The named QEC_DEVICE_GRAPH_* knobs (DEVICE, PEER_IP, REMOTE_QP, FRAME_SIZE, PAGE_SIZE, NUM_PAGES) each emit their --flag= only when set -- the constructor no longer requires HSB env, and no hololink-shaped defaults are forced on a foreign provider. --gpu= is the one always-passed argument (the decoder's cuda_device_id, not an env knob). QEC_DEVICE_GRAPH_PROVIDER_ARGS is a free-form pass-through appended verbatim after the named args, and the server's all-device_graph path fills it from the YAML transport section (section args + device_graph override), so a partner provider is configured entirely from the YAML. The ring-size / host-page alignment validation now gates on the geometry knobs being provided; authoritative geometry still comes from the provider's v2 query. Compile- and regression-verified locally (two-process suite, DecoderYAMLTest, per-decoder-rings and mixed-dispatch app tests, 26/26); the standalone path's endpoint handshake needs one HSB rig run of the updated script. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
…an FPGA or a QPU kernel
A shipped example that compiles against the installed SDK and exercises the
delivered decoding_server from two syndrome sources, plus its docs/CI
registration and an FPGA RX-ring sizing fix in the unittests driver.
The example (docs/sphinx/examples/qec/realtime_decoding_demo/):
- surface_code_realtime_decoding.cpp (a slimmed surface_code-4-yaml: the
Ising-bundle and syndrome-replay machinery removed) is built two ways by a
standalone CMakeLists.txt using installed headers/libs only -- a generator
(writes the decoder config, and for the FPGA source the playback syndrome
file) and the lowered -cqr kernel (-frealtime-lowering).
- run_realtime_decoding.sh drives either source with pymatching / trt_decoder
/ nv-qldpc-decoder:
--source qpu-kernel the lowered kernel streams syndromes to the server
over UDP (no hardware);
--source fpga the delivered hololink_fpga_syndrome_playback streams
syndromes from a real FPGA over RoCE (cpu_roce for
CPU decoders, gpu_roce for the nv-qldpc device path).
- Pass/fail matches the existing surface-code tests. The qpu-kernel path
applies the same checks as the surface_code-4-yaml-test.sh ctest driver:
hard decoder-failure greps, the "Number of corrections decoder found:"
completion proof, a residual logical-error ceiling of num_shots/50 (min 1),
the server-owned-decoders report, and a server dispatch-count floor of
num_shots * (num_rounds + 3). The fpga path passes on the playback tool's
per-shot correction verification, same as hsb_fpga_decoding_server_test.sh.
- The gpu_roce RX ring runs at the HSB QP's 64-entry receive WQE depth (the
receiver pre-posts one WQE per ring slot; deeper rings drop frames under
load), with a preflight of DOCA's host-page alignment that fails fast with
the stride constraint on 16K/64K-page hosts. The server gets --timeout=300
(total-runtime failsafe, matching the in-tree external-server tests).
Docs + CI:
- New narrative page examples_rst/qec/realtime_decoding_demo.rst
(literalincludes the source), registered in the QEC examples toctree; the
FPGA predecoder + Relay BP pages point at the example for end-to-end runs.
- conf.py.in excludes examples/**/README.md from sphinx (example READMEs are
for the shipped tree / GitHub; the '.md' source suffix would otherwise
ingest them as toctree-orphan pages).
- lib_qec.yaml: build-and-test (CPU) builds the two example binaries against
the installed SDK (enforcing the installed-headers-only rule) and runs the
qpu-kernel pymatching decode over UDP loopback -- a real decode, no
hardware -- archiving the SDK install + example build for the GPU job.
gpu-test runs the example's trt_decoder qpu-kernel decode over UDP reusing
those artifacts (+ pip install onnx). This is the only CI coverage of
trt_decoder THROUGH the decoding server: the TRTDecoder unit tests exercise
the decoder standalone, and the app_examples happy-path trt test is
configured out because its onnx gate is evaluated before onnx is
pip-installed. Gated on TensorRT availability (skips arm64-cu12).
- unittests/utils/CMakeLists.txt installs hololink_fpga_syndrome_playback
alongside decoding_server (already hololink-gated; a no-op in the vanilla
release) so the example can resolve it from the install prefix.
hsb_fpga_decoding_server_test.sh -- fix HSB FPGA RX-ring sizing (two bugs,
both rooted in the HSB QP's fixed 64 receive WQEs; the receiver pre-posts one
WQE per ring slot, so a ring with MORE slots than WQEs drops frames):
1. cpu_roce modulus desync (regression from NVIDIA#683 / 2cdc9df): NVIDIA#683 added
PLAYBACK_NUM_PAGES=512 as the playback --num-pages to size the gpu_roce
ring, but run_playback is SHARED with cpu_roce, whose server RX ring is 64
slots. The FPGA writes frame rid to slot (rid % num-pages), so frames
rid>=64 landed outside the server's registered MR and the test starved
after ~10 shots (DISPATCHED=64). Fix: program the playback ring modulus
from the server ring depth per transport, and clamp both to the WQE depth
so the invariant is structural.
2. gpu_roce over-sized ring: the ring was 512 slots but the DOCA receiver only
pre-posts 64 WQEs, dropping frames under load. Cap it at the WQE depth
(clamp+warn on an over-sized --gpu-roce-num-pages), and preflight DOCA's
host-page alignment (64 x 384 B covers 4K/8K-page kernels; 16K/64K-page
hosts fail fast with the --page-size constraint spelled out).
The emulator does not parse --num-pages (its RDMA-write modulus is the SIF
target the playback programs), so the dead arg, the PLAYBACK_NUM_PAGES
variable, and the --playback-num-pages option are removed.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Replace the example source with the surface_code-1.cpp from NVIDIA#685 (new syndrome format, decoder_context_from_memory_circuit, seeded runs) and adapt the driver script, build, docs, and CI to its interface. The example copy differs from the in-tree file only in the header comment, the usage line, and the --save_syndrome capture, which now emits true-width groups (the packed bytes are byte-padded; the FPGA playback tool sets each replayed frame's num_syndromes from the group's line count, and the decoder's per-shot measurement capacity is the exact sum, so padded groups overran it). - run_realtime_decoding.sh: --p-cnot replaces --p-spam (two-qubit depolarizing on the CNOTs, default 0.001); new --seed (default 42 -- runs and pass/fail counts are reproducible); config/kernel phases use --save_dem / --load_dem; the kernel needs only QEC_DECODING_SERVER_PORT (the app brings up the udp channel itself in its QEC_APP_CQR build). Decoders: pymatching, nv-qldpc-decoder, multi_error_lut (trt_decoder is not supported by the new app and is dropped; sliding_window exists in the app but is not exposed -- over device calls it needs matching updates to the proprietary cudevice archive). - Pass/fail criteria keep the surface-code test checks (hard decoder-failure greps, corrections-line completion proof, num_shots/50 residual ceiling, num_shots*(num_rounds+3) server dispatch floor) and adopt the two-process test's server-ownership proof: the app's in-process dispatch count must be 0. - CMakeLists.txt: -DQEC_APP_CQR; link cudaq-qec-realtime-decoding + cudaq-qec-realtime-decoding-server-cqr (exports the dispatch-count symbol). - Docs: decoder tables/text updated (3 decoders, --p-cnot/--seed, criteria); the FPGA-predecoder page's pointer at this example is removed (trt dropped). - CI (lib_qec.yaml): the gpu-test trt step and its artifact plumbing are removed (trt dropped); the CPU job's pymatching decode now runs the seeded 200-shot default, making the ceiling check deterministic (the 50-shot unseeded run could statistically exceed a ceiling of 1 -- the amd64-12.6 failure on the previous run). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
cketcham2333
marked this pull request as ready for review
July 17, 2026 02:21
bmhowe23
reviewed
Jul 17, 2026
bmhowe23
left a comment
bmhowe23
left a comment
Collaborator
There was a problem hiding this comment.
This is looking good...very descriptive and very thorough. Just a few comments for tonight.
Comment on lines
+6
to
+9
| This page describes a C++ example that you compile against the *installed* | ||
| CUDA-QX SDK. The decoding server and the FPGA playback tool it drives are | ||
| **deliverables** — prebuilt and installed alongside the SDK, not built from | ||
| this example. |
Collaborator
There was a problem hiding this comment.
For the user docs, I think it is implied that they are dealing with installed deliverables. I think we could get the point across better by saying which Docker container they should be using (which is admittedly TBD) and provide the full path to the where the relevant files are inside that Docker container.
| # CPU host-call path (pymatching/multi_error_lut) or the | ||
| # GPU (nv-qldpc). This is the portable, hardware-free mode. | ||
| # | ||
| # --source fpga The delivered hololink_fpga_syndrome_playback tool |
Collaborator
There was a problem hiding this comment.
Noting to self: we need to get this option hooked up to automated testing, even if it isn't tied in with GitHub's CI.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
eliotheinrich
left a comment
eliotheinrich
left a comment
Collaborator
There was a problem hiding this comment.
Looks good; just a few clean-up comments
Signed-off-by: Ben Howe <bhowe@nvidia.com> # Conflicts: # libs/qec/include/cudaq/qec/realtime/decoding_config.h # libs/qec/lib/realtime/decoding-server-cqr/CMakeLists.txt # libs/qec/lib/realtime/decoding-server-cqr/DecodingServer.cpp # libs/qec/lib/realtime/decoding-server-cqr/GpuRoceTransceiver.cpp
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
…' into decoding_server_examples
File: libs/qec/lib/realtime/decoding-server-cqr/DeviceGraphTransceiver.cpp
The transceiver configures its bridge provider by passing an argv-style
vector to cudaq_bridge_create_from_library(), but built that vector with the
first real option (--device=<nic>) at position 0. The provider's
parse_bridge_args follows the C argv convention and starts parsing at
argv[1], so --device was silently skipped on every start and the bridge fell
back to its compiled-in default device ("rocep1s0f0"). All remaining options
(--peer-ip, --remote-qp, --gpu, --page-size, --num-pages, --payload-size)
sat at index >= 1 and parsed correctly, so only the device was wrong and
nothing reported it.
On rigs where the default happens to name the cabled NIC the bug is
invisible. Where it does not, device_graph startup fails with
"gpu_roce_transceiver: Cannot find GID for RoCE v2": the default device's
GID table has only link-local entries, never the IPv4-mapped RoCE v2 GID the
transceiver searches for (confirmed via HSB debug logs showing "Opening DOCA
device with name: rocep1s0f0" while QEC_DEVICE_GRAPH_DEVICE named the cabled
port).
Prepend a program-name placeholder so the vector honors the argv[0]
convention and --device parses. The parser side is left alone: starting at
argv[1] is correct for its other callers, which pass a genuine argv[0].
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Files: docs/sphinx/examples/qec/realtime_decoding_demo/run_realtime_decoding.sh
docs/sphinx/examples/qec/realtime_decoding_demo/README.md
docs/sphinx/examples_rst/qec/realtime_decoding_demo.rst
libs/qec/unittests/utils/hsb_fpga_decoding_server_test.sh
Adapt the example driver, its docs, and the hsb unittests script to the
merged transport-agnostic decoding server (bridge providers, one ring per
decoder, per-decoder dispatch). The example source needs no changes.
run_realtime_decoding.sh:
- Replace --transport with the server's two independent knobs, both derived
automatically: --wire udp|cpu_roce|hololink (the bridge-provider library
carrying syndromes into the server) and --dispatch host|device_graph (the
engine consuming each decoder's ring). Combinations this example has not
wired up are rejected with the reason: the qpu-kernel source is udp-only
for now (kernel-over-cpu_roce is planned follow-up work), device_graph
over udp (pinned shared rings) is not exposed, and device_graph serves
only nv-qldpc-decoder over the hololink wire. fpga + nv-qldpc +
--dispatch host is newly supported (GPU decode via host call over the
cpu_roce wire).
- Config injection writes the per-decoder `dispatch: device_graph` key
(replacing the retired `transport: gpu_roce`), still with cuda_device_id,
and verifies both landed.
- The device_graph server launch uses the QEC_DEVICE_GRAPH_* environment
(was HOLOLINK_*), drops the retired --transport=gpu_roce flag, and waits
for the "QEC_DECODING_SERVER_READY device_graph" sentinel.
- The RDMA handshake scrape reads the qp=/rkey=/buffer_addr= tokens from
whichever line carries them: the READY line on host dispatch (the bridge
provider's endpoint info rides it) or the device-graph transceiver's
dedicated QEC_DECODING_SERVER_ENDPOINT line. Replaces the retired
three-line "QP Number:/RKey:/Buffer Addr:" scrape.
- Pass/fail criteria additionally assert the new per-decoder ring counter
(QEC_DECODING_SERVER_RING decoder=0 dispatched > 0).
- The RoCE wire's provider token is passed hyphenated (cpu-roce): the server
resolves "libcudaq-realtime-bridge-<token>.so" literally and the built
provider soname is hyphenated.
hsb_fpga_decoding_server_test.sh:
- Pass the hyphenated cpu-roce provider token on the server command line.
- Scrape the handshake tokens from whichever line carries buffer_addr= (the
unconditional ENDPOINT-line wait broke the cpu_roce path, where the tokens
ride the READY line).
- Resolve the server's realtime lib dir via CUDAQ_REALTIME_LIB_DIR or the
conventional realtime build dirs, so the server dlopens the bridge-provider
API matching .cudaq_version instead of a stale build.
Docs (README + realtime_decoding_demo.rst): describe the wire/dispatch model
and the --wire/--dispatch overrides; the decoder table uses host /
device_graph dispatch vocabulary; the FPGA run section lists all three
decoder invocations explicitly.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
File: libs/qec/lib/realtime/decoding-server-cqr/DeviceGraphTransceiver.cpp
The transceiver configures its bridge provider by passing an argv-style
vector to cudaq_bridge_create_from_library(), but built that vector with the
first real option (--device=<nic>) at position 0. The provider's
parse_bridge_args follows the C argv convention and starts parsing at
argv[1], so --device was silently skipped on every start and the bridge fell
back to its compiled-in default device ("rocep1s0f0"). All remaining options
(--peer-ip, --remote-qp, --gpu, --page-size, --num-pages, --payload-size)
sat at index >= 1 and parsed correctly, so only the device was wrong and
nothing reported it.
On rigs where the default happens to name the cabled NIC the bug is
invisible. Where it does not, device_graph startup fails with
"gpu_roce_transceiver: Cannot find GID for RoCE v2": the default device's
GID table has only link-local entries, never the IPv4-mapped RoCE v2 GID the
transceiver searches for (confirmed via HSB debug logs showing "Opening DOCA
device with name: rocep1s0f0" while QEC_DEVICE_GRAPH_DEVICE named the cabled
port).
Prepend a program-name placeholder so the vector honors the argv[0]
convention and --device parses. The parser side is left alone: starting at
argv[1] is correct for its other callers, which pass a genuine argv[0].
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com> # Conflicts: # libs/qec/lib/realtime/decoding-server-cqr/DecodingServer.cpp # libs/qec/lib/realtime/decoding-server-cqr/DecodingSession.cpp
Files: docs/sphinx/examples/qec/realtime_decoding_demo/run_realtime_decoding.sh
docs/sphinx/examples/qec/realtime_decoding_demo/README.md
docs/sphinx/examples_rst/qec/realtime_decoding_demo.rst
The qpu-kernel source can now drive the server over the cpu_roce wire
(--wire cpu_roce) with any of the three decoders, in addition to the
default udp wire, which is unchanged. No C++ changes: the example
binary's QEC_APP_CQR channel bring-up already carried the cpu_roce
client path; the script now selects it.
run_realtime_decoding.sh:
- The combination gate admits qpu-kernel + cpu_roce + host; qpu-kernel +
hololink and qpu-kernel device_graph dispatch stay rejected with their
reasons.
- RDMA topology comes from the same four env vars as every in-tree
cpu_roce test (CUDA-Q's CpuRoceChannelTester and the QEC two-process
test): CUDAQ_CPU_ROCE_TEST_{CHANNEL,DAEMON}_{DEVICE,IP}. The device
vars are required (the error lists the machine's IB devices); the IPs
default to 10.0.0.1 / 10.0.0.2.
- --setup-network now also covers this pair: per side it sweeps the IP
off any other interface (the duplicate-IP routing trap), configures
the port (reusing setup_port), and waits for the matching RoCE v2 GID
(the kernel populates it asynchronously after ip addr add). No ARP
seeding: the RDMA endpoints cross the TCP rendezvous.
- The cpu_roce server launch adds --device/--local-ip and pins the ring
geometry explicitly (--num-slots=8 --slot-size=256): the geometry is
part of the wire contract -- the kernel's channel writes straight into
the server's rings and hardcodes 8 x 256.
- The kernel invocation additionally carries
QEC_DECODING_SERVER_TRANSPORT=cpu_roce plus the channel-side topology
env; the READY scrape is tightened to the READY line (for cpu_roce the
port= token is the TCP rendezvous port).
- Pass/fail criteria are unchanged -- they are wire-agnostic.
Docs: README + rst gain the cpu_roce invocations for all three decoders
and the topology env-var block; the decoder table gets the udp/cpu_roce
wire vocabulary.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
…' into decoding_server_examples
18 tasks
…example New files: docs/sphinx/examples/qec/realtime_decoding_demo/surface_code_ising_realtime_decoding.cpp docs/sphinx/examples/qec/realtime_decoding_demo/gen_dsparse_from_memory_circuit.py Updated: docs/sphinx/examples/qec/realtime_decoding_demo/CMakeLists.txt docs/sphinx/examples/qec/realtime_decoding_demo/run_realtime_decoding.sh docs/sphinx/examples/qec/realtime_decoding_demo/README.md docs/sphinx/examples_rst/qec/realtime_decoding_demo.rst The example gains an opt-in fourth decoder profile: --decoder trt_decoder, the published Ising neural-network predecoder (a TensorRT engine built from its ONNX export) chained into a PyMatching global decoder, served by the same delivered decoding_server over all three shipped paths -- qpu-kernel over udp, qpu-kernel over cpu_roce, and the FPGA over cpu_roce (host dispatch on every path). The profile runs its own example source, a trimmed port of the in-tree surface_code-4-yaml app, because the Ising artifacts are bound to that lineage's conventions: orientation XV, per-round measurement layout [X-ancillas..., Z-ancillas...], and SPAM noise at the model's trained operating point (d=7, T=7, basis Z, p_spam=0.01). The port keeps the kernels and enqueue schedule byte-identical to the lineage (the artifact directory's D_sparse.txt maps Ising detector rows onto exactly that live measurement stream) and adds what the example needs on top: the cpu_roce device-call channel bring-up, a --seed flag, the in-process dispatch-count self-check, and --ising-artifacts-dir replacing the older --ising_bundle/--onnx_path pair (ONNX resolves to <dir>/model.onnx). The artifacts (six files: model.onnx, H_csr.bin, O_csr.bin, priors.bin, metadata.txt, D_sparse.txt) never ship with CUDA-QX: the weights are a gated Hugging Face download prepared through the NVIDIA/Ising-Decoding repository, and D_sparse.txt is generated against this example's own printed CNOT schedules with the gen_dsparse_from_memory_circuit.py copy shipped alongside (--ising-repo now required). The docs carry the full preparation recipe. An absent or incomplete artifact directory (or a missing TensorRT decoder plugin in the install prefix) makes the profile SKIP (exit 77), naming each missing file -- so CI and artifact-less machines are unaffected; the profile never runs in CI. run_realtime_decoding.sh: trt_decoder joins the decoder whitelist with its own binary pair selection; the profile pins d=7/rounds=7 and rejects conflicting explicit --distance/--num-rounds/--p-cnot (SPAM noise comes from the new --p-spam, default 0.01); the generated config is identity- checked (type + onnx_load_path); the server READY wait is extended for the TensorRT engine build; on the FPGA source the shot default is 56 -- the 512-frame playback BRAM ceiling at 9 frames/shot (8 syndrome slices + 1 corrections frame), the same fill-the-BRAM convention as the d3 default 85 -- and --spacing defaults to 5 ms for this profile: the 9-frame bursts overrun the server's 64-slot RX ring (~7 shots of buffer vs a measured ~35 us decode round trip) at the usual 10 us, corrupting the stream around shot 8-18. Also fixed here (bites every cpu_roce profile after a reboot): --setup-network's cpu_roce path now seeds static neighbor entries between the channel and daemon ports, the same-host analogue of the FPGA path's seed_fpga_neighbor. Same-host port pairs never ARP for each other's address (the kernel routes local IPs via lo and never probes the wire), so the RoCE path resolution failed with ibv_modify_qp(RTR) errno=110 once the neighbor state from the original bring-up was lost. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
melody-ren
reviewed
Jul 22, 2026
Collaborator
There was a problem hiding this comment.
I don't think you meant to commit this .gitignore?
| "cudaq": { | ||
| "repository": "NVIDIA/cuda-quantum", | ||
| "ref": "5f0ee5be3bf5cb33afb544d9f01af77b1d50b26e" | ||
| "repository": "bmhowe23/cuda-quantum", |
Collaborator
There was a problem hiding this comment.
Is this just a leftover while you're experimenting?
Collaborator
There was a problem hiding this comment.
I think this PR should be marked as draft. It can't merge until #682 does (or something like it). That will also reduce the size of this PR since this PR currently contains both embedded in it.
Collaborator
run_realtime_decoding.sh: the nv-qldpc profiles cannot drain the server's 64-slot RX ring at the 10 us default playback spacing (measured on-rig: 171/510 frames captured on host dispatch, ~15/85 shots verified on device_graph, both looking like a broken FPGA/scheduler). Auto-pace them like the trt profile already does -- 5 ms on host dispatch, 100 us on device_graph (both measured-good) -- with an explicit --spacing always winning. Documented in the README and Sphinx page. hsb_fpga_decoding_server_test.sh: the 10.0.0.1 bridge-ip default is the emulate-loop convention; applied silently in real-FPGA mode it puts the server NIC on the wrong subnet and the only symptom is a hololink control-plane read_uint32 timeout that masquerades as dead hardware. Default the bridge IP by mode (10.0.0.1 emulate / 192.168.0.1 FPGA) and reject an explicit bridge-ip outside the FPGA's /24 with an error that names the failure mode. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
cketcham2333
force-pushed
the
decoding_server_examples
branch
from
July 24, 2026 14:31
367b1cf to
65a6820
Compare
…mples Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a shipped example (
docs/sphinx/examples/qec/realtime_decoding_demo/) thatcompiles against the installed SDK — installed headers/libs only — and exercises
the delivered
decoding_serverend-to-end, plus its docs/CI registration and anFPGA RX-ring sizing fix in the unittests driver.
The example source is the refactored
surface_code-1.cppfrom #685 (new syndromeformat,
decoder_context_from_memory_circuit, seeded runs), copied with threeadaptations: the header comment, the usage line, and a true-width
--save_syndromecapture so the FPGA playback tool replays exact per-round measurement counts.
The example
One source, built two ways: a generator (decoder config + FPGA syndrome file) and
the lowered
-cqrkernel (-frealtime-lowering -DQEC_APP_CQR). One script, twosyndrome sources, three decoders (
pymatching,multi_error_lut,nv-qldpc-decoder):--source qpu-kernel— the lowered kernel streams syndromes to the server overUDP (no hardware;
QEC_DECODING_SERVER_PORTroutes the device_calls).--source fpga— the delivered playback tool streams syndromes from a realFPGA over RoCE (
cpu_roce/gpu_roce).Runs are seeded (
--seed, default 42), so results are reproducible. Pass/failmatches the existing surface-code tests: decoder-failure greps, the completion
proof, a
num_shots/50residual logical-error ceiling, an in-process dispatchcount of 0 (the two-process test's proof the decode stayed in the server), and a
num_shots*(num_rounds+3)server dispatch floor.CI
lib_qec.yamlbuilds the example against the installed SDK (enforcing theinstalled-headers-only rule) and runs a seeded 200-shot qpu-kernel pymatching
decode over UDP loopback on the CPU job — a real decode, no hardware, and
deterministic pass/fail.
FPGA ring fix (
hsb_fpga_decoding_server_test.sh)Fixes a #683 regression: the playback ring modulus (
--num-pages 512) no longermatches the cpu_roce server's 64-slot RX ring, starving the test after ~10 shots.
Both transports now run a ring equal to the HSB QP's 64-entry receive WQE depth
(deeper rings drop frames), with a fail-fast preflight of DOCA's host-page
alignment on 16K/64K-page hosts.