pstop: consolidated working branch (restructure re-land + README + CI bumps) - #58
Closed
iliabaranov wants to merge 163 commits into
Closed
pstop: consolidated working branch (restructure re-land + README + CI bumps)#58iliabaranov wants to merge 163 commits into
iliabaranov wants to merge 163 commits into
Conversation
Squash of the reverted #53 restructure (65273ed) and the parked PR #57 README fix (mermaid diagram back, monorepo-correct layout table, latest status folded in). Re-landed on a working branch, NOT main: main stays on the pre-restructure tree until the downstream breakage that forced the revert (c35bd95) is identified and fixed here first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Squash of dependabot PRs #54/#55/#56: actions/checkout 4->7, actions/upload-artifact 4->7, actions/setup-python 6->7. The "pstop_c coverage" failures on those PRs were NOT the bumps: dependabot-triggered runs get no repo secrets, so BULLSEYE_LICENSE_KEY was empty and the installer died. Guard the job to skip for dependabot[bot] instead of failing red. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
This was referenced Jul 23, 2026
…ents Mirror of the machine side's many-remote support: the remote now heartbeats up to DCS_PSTOP_MAX_MACHINES (4) machines as fully independent sessions. NO changes to the certification-track pstop_c library — each session simply owns its own pstop_c protocol_data_t. firmware/main/main.c (the auditable safety core): - Per-machine session array: own socket (local ports 8891..8894, so egress binding stays per peer type — Tailscale peers source-bind to the VPN IP, LAN peers don't; the 2026-07-20 fail-open lesson holds per machine), own bond/counter state, own reply-loss watchdog. - The blocking bond_with_retry() is gone: bonding is a non-blocking per-session state machine driven inside the 10 Hz tick (one BOND in flight, 5 s retry, drain-before-bond preserved), so one unreachable machine never stalls heartbeats to the others. - Lockstep now covers N encodings per tick: each core samples its E-stop loop ONCE, then encodes one message per bonded machine; the comparator memcmps every pair and a mismatch on ANY slot silences ALL sessions (lockstep integrity is a device property). - Replies demuxed per-session socket with source-address validation; select() bounds the reply wait regardless of machine count. - Transport priority-health: true on any reply; the repeated unhealthy kick (needed to force WG past a zombie keypair) fires on the old 5 s cadence only when EVERY configured machine is silent — a partially healthy device never churns the shared transport. - STOP and the arming press-and-release broadcast to all bonded machines; each machine still enforces its own min_stop_ms veto. dcs_support: - NVS ps_peers blob (byte-serialized, versioned) with migration from the legacy ps_ip/ps_port pair into slot 0; slot 0 mirrors back to the legacy keys so a firmware rollback keeps its primary machine. - POST /api/pstop_peers?slot=N&ip&port[&id] / ?slot=N&clear=1 — applies live within one tick; /api/pstop_peer still works (= slot 0) so existing fleet tooling is untouched. - /state.json: pstop_machines[4] with per-session cfg/endpoint/state/ counters; legacy pstop_* scalars stay as aggregates (worst-of last_msg, summed counters). JSON_CAP 2304 -> 3072. - LED ring: divides evenly among configured slots in slot order from LED 1 (one machine = whole ring, unchanged look); per-segment colours as before; device-level states (lockstep purple, OTA, locate) still override the whole ring. First cut — display design expected to evolve. tools/pstop_multi_machine_test.py: HIL harness — N unmodified machine_app_runner instances on this host, chip pointed at them via the API, asserts session isolation (kill one machine -> only that session re-bonds, others' replies keep climbing), recovery, slot clear, and zero lockstep mismatches throughout. Restores the bench unit's original peer afterwards. Builds clean under IDF 5.5; host runner untouched and building; MISRA delta on main.c limited to already-registered deviation classes. HIL validation on real hardware still pending. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
A steady green or red ring is indistinguishable from a frozen display. Overlay the boot spinner's comet shape as a subtle brightness wave on OK and STOP segments: head at 160% of base with a fading tail, far pixels at 85%, chasing the whole ring at the boot-spinner cadence so multiple machine segments share one coherent wave. Unreachable pulse, bond blue, and the device-level overrides are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Design intent: the MACHINE decides the update rate its safety case needs. pstop_c already supports this end to end — remote_details_t .heartbeat_ms (machine.toml default_heartbeat_ms / per-operator heartbeat_ms) is enforced by machine_check_heartbeats AND advertised to the operator in every reply's heartbeat_timeout field, bond ack included (machine.c). The gap was only on our remote, which ignored the field and always transmitted at the raw 10 Hz tick. Remote (main.c): each session adopts the machine's advertised window and transmits every heartbeat_timeout/2 (2x margin so jitter never costs the machine a whole window), clamped to [100 ms tick, 1000 ms]. Sends are decimated ticks; the per-session counter advances only on transmitting ticks so the machine sees contiguous counters (protocol.c tolerates gaps only up to max_lost_messages+1). The E-stop loop sampling stays at the fixed 10 Hz lockstep tick via a dedicated roll counter — the safety input cadence never slows down. The reply-loss watchdog scales with the adopted period (4 missed replies, floored at the validated 1500 ms). machine.toml: documents that heartbeat_ms is the machine-owned rate and retunes the remote with no remote-side configuration. state.json pstop_machines[] gains hb_ms (the window each session has adopted). No pstop_c changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Bench feedback: 85% base was too subtle. Dim the non-comet pixels to 55% of base and lengthen the tail (170/135/105/80/65 -> 55), so the wave reads clearly at a glance while the state colour stays dominant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
The adopted heartbeat window was applied correctly (HIL: 2 Hz sends against a 1000 ms machine window) but the atomic store for the pstop_machines[].hb_ms field was missing, so /state.json reported 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Every line now states its direction from the machine's perspective so two-node tests read unambiguously: - RX <- remote 0x... (what a pstop sent us; verbose adds cnt/rcnt/crc) - TX -> remote 0x... (our reply; verbose adds counter + advertised hb) - *** ROBOT STATUS / ARMED *** for the actuation-relevant transitions - STATE ... (trigger: RX <msg> from remote 0x...) for the arming cycle Startup banner names the role, machine_id, and the log grammar. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Root cause of remotes showing offline/idle on the fleet console while fully healthy on the data plane (all 3 chips froze at one control-server stream drop 2026-07-24T03:52Z and stayed control-plane-offline ~17 h; diagnosed jointly with the fleet-side session): 1. When control ends the streaming map request but the TCP+Noise session stays up, GOAWAY / RST_STREAM(5) / END_STREAM / closing trailers were silently skipped — the chip never re-issued the MapRequest, and Tailscale marks a node online only while that stream is open. 2. The 120 s control watchdog was fed by ANY received frame, so the ACKs to our own H2 PINGs (stream 0) kept it fed forever. This is the surviving sibling of the 9cffb97 half-open fix: that one stopped SENT pings from feeding the watchdog; received PING ACKs still masked a dead map stream. Now: explicit stream termination returns connection-lost (immediate reconnect), and only stream-5 traffic feeds the watchdog. Control's keepalive MapResponses (~60 s, we request KeepAlive=true) feed a healthy stream with 2x margin against the 120 s watchdog; a dead one trips it and self-heals within 2 min instead of never. Fleet-console side (deriving "online" from check-in age + not evicting check-in-fresh devices from the live poll) is tracked in the pstop-fleet repo, not here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Documents the two external connections (WS2812 ring on GPIO17, DPST NC E-stop loops on GPIO39/40 and GPIO41/42) with the standard wire colors: red 5V, black GND, green LED data, white pole 1, yellow pole 2. Includes the fail-safe rationale (open loop = STOP) and the LED-1 rotation calibration pointer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Chained edges and mixed subgraph ranks made mermaid route wires through the middle of the boxes. One node per switch terminal, plain undirected links, TB rank inside each box. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
…mption Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Search links rather than pinned listings (they rot); the load-bearing part is the specs, especially the E-stop needing 2NC contacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Links open in a new window. The NKK FF0126BBCAEA01 matches the STEP model already in this folder. M1.6 x 5 x 2.5 mm inserts anchor the ESP32 board to the base. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
- LICENSES/ carries the canonical texts (CERN-OHL-P-2.0 from the CERN OHL repository, CC-BY-4.0 from creativecommons.org). - hardware/README.md gains a Licensing section: original design files under CERN-OHL-P-2.0, original docs/images under CC-BY-4.0, and a manifest naming each vendor reference file's owner and origin link (Waveshare schematic/photo, NKK FF01 E-stop, GlobTek USB-C jack). Two reference models have no identifiable origin (esp32.step, LED.step) — flagged in the to-do list for identify-or-redraw. - Root README states the license split; OSHWA checklist rows 1, 2, 4, 6, 9, 10 move to Met. - Removed the unused LED lens STEP (not part of this design, unclear origin). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
…sp32.step The board mock-up is xro's Printables model under CC BY-NC-SA 4.0 — attributed, with a note that the NC term conflicts with the OSHW definition and commercial use, so it is queued for a redraw. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
…ck-up New board fit model drawn in-house (CERN-OHL-P-2.0 with the rest of the design); casing.FCStd updated to reference it. The xro Printables mock-up and its NC-license caveat are gone — every file in hardware/ is now openly licensed or vendor-attributed. Last remaining to-do is assembly instructions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
The design has no custom electronics; hardware/README.md now lists the freely accessible datasheets for every active component (ESP32-S3, W5500, carrier schematic, WS2812B, NKK FF01), all links verified live. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
The Waveshare silkscreen labels pins IOnn, not GPIOnn — relabel every doc pin reference so an assembler never has to translate. Firmware code identifiers are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b
… fix DU-1 (SR-R-09 GPIO re-verify) and DU-3 (SR-R-03 object-code diversity) marked CLOSED; production networking + firmware HIL + SR-H-04b frozen-clock reflected as done. New section 8 enumerates what actually remains, split by blocker: needs-user (FMEDA part numbers / demand rate / proof-test interval; OD-1), blocked-on-HW/people (machn HW clock validation, pstop->main PR), and lower-priority engineering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…C-14 gcov elsewhere User-confirmed. Two tools by design, measuring disjoint code: the pre-qualified pstop_c library keeps its Bullseye coverage/CI as-is; every other in-scope codebase uses the free GCC-14 gcov/gcovr flow (line+branch+MC/DC) already wired into scripts/coverage.sh and CI. Recorded in COVERAGE.md policy note + OPEN_ITEMS decisions log; OD-1 closed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
Closes the deferred machn-side clock-guard HIL. The live machine node machn-01d77a1c shipped 6e7492f (the commit right before the guard); built HEAD, verified the guard strings in the ELF, OTA-deployed 3dcd08d, and validated on real silicon: clean boot, no false clock-fault trip over ~5.5 min, dual-core lockstep agreement, and a full soft-remote arm->RUN (0.53s min-delay)->STOP->re-arm->safe-STOP exercising the real series relays with feedback. Genuine esp_timer-freeze detection stays fault-injection- proven on the remote analog (347de6f) + host clock_guard tests (freeze is not injectable on unmodified silicon). Record in docs/safety/MACHN_CLOCK_GUARD_HIL.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…re-rate data Synthesized from two IEC 61508 advisor passes (data-sourcing + method/defensibility), tailored to pstop (1oo2D, ESP32-S3, series relays). Covers: the two hardware gates (architectural SFF vs quantitative PFH) and why pstop is fragile at the Type-B SFF-90% boundary not PFH; generic-rate sources usable without a vendor (SN 29500/IEC 61709, exida SERH, FMD-2016, FIDES/MIL-217); worked pi-factor conversion (ESP32 die 60->2320 FIT); claiming DC from IEC 61508-2 Annex A/C for each pstop diagnostic; the ISO 13849 B10d relay cross-check; beta via Annex D with no phantom diversity; the assumptions register; sensitivity analysis; and prior-use/proven-in-use (Route 2H). Highest-leverage pstop action: relay diversity (~halves PFH since PFH~=beta*lambda_DU). Advisor caveats on proprietary table cut-offs preserved. Linked from OPEN_ITEMS OD-2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…4%->99% line) Adds two gtest suites and a loopback HTTP stub to reach the ROS 2 machine paths the inline-transition test_lifecycle cannot: test_node_runtime (16 tests) — spins a SingleThreadedExecutor to exercise publish_tick, every diagnostics summary branch (OK/WARN/ERROR, driven off canned /state.json), on_set_parameters accept/reject, the ~/configure_machine service (apply/reject/backend-refused), and the on_error / on_shutdown / activate-start-failure transitions. test_hardware_backend (13 tests) — drives http_get/http_post/write_cb, the poll loop, start/stop and configure() against an in-process loopback HTTP stub (test/http_stub.hpp) and a closed port. No live device: the stub stands in for the ESP32 machn on 127.0.0.1, a closed port covers the unreachable fail-safe path, and basic-auth + non-2xx cases are exercised too. machine_bridge_node.cpp 43.3%->98.4% line / 27.7%->49.8% branch; hardware_backend.cpp 24.6%->98.6% line / 22.4%->59.5% branch; both now 100% function. software_backend.cpp rises 65.0%->75.0% as a side effect (the node tests run its machine loop live). Residual gaps are network/protocol-only (pstop_c UDP callbacks, curl OOM guards) or unreachable guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
Updates COVERAGE.md §1 + the ROS2 per-module table (dated 2026-08-02, re-measured with gcovr 8.6): hand-written ROS 2 now ~89% line (624/703) over 6 suites / 56 tests, up from ~53% / 4 suites / 27 tests. Per module: node 43.3->98.4% line, hardware 24.6->98.6% line, software 65.0->75.0% line; remaining gaps documented honestly (UDP-remote-only callbacks, curl OOM guards, unreachable guards, curl/rclcpp branch fan-out). The new executor + HTTP-seam tests discharge two P1 test-gaps, so TRACEABILITY.md promotes SR-M-01 (validate_timing reject-loosening, now exercised at unit/param/ service levels) and SR-M-03 (unreachable /state.json -> UNSTABLE + diagnostics) from Unverified-gap to Verified. Headline moves 25/39 -> 27/39 with a test and 13/39 -> 15/39 strictly verified; SR-M area, breakdown table, footnote and the P1-2/P1-3 register rows updated to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
test_node_runtime + test_hardware_backend (loopback HTTP stub, no live robot): machine_bridge_node 43->98% line, hardware_backend 25->99% line, 27->56 tests green. SR-M-01/SR-M-03 promoted Unverified-gap -> Verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
Apply the repo pre-commit standard (polymath-cpp = clang-format + cpplint) to the three files added this session: clang-format reflow + a NOLINT(build/include_subdir) on the test-local http_stub.hpp include (matching the repo NOLINT convention). All polymath-cpp/copyright/cmake/general hooks pass; 56/56 functional gtests still green. Markdown normalization is left deferred: every committed doc uses "-" list markers and none use "+", so running pymarkdown on only the touched docs would make them the sole inconsistent files — that reformat stays a repo-wide maintainer item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…ecorded Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…ppendices Per reviewer feedback the guide was hard to parse from constant acronym use. Every acronym is now spelled out at first occurrence (FMEDA, SFF, PFH, SIL, HFT, 1oo2D, DC, DD/DU, beta/lambda, B10d/MTTFd, CCF, the data sources, etc.), a "how to read" pointer was added up top, and three appendices added: A (acronym glossary grouped by theme, each with a few sentences of explanation), B (symbols & notation: lambda buckets, beta, pi-factors, Sigma, theta, E_a/k, t_CE, chi-squared, units), and C (the harder concepts in plain language: random vs systematic, the two hardware gates, Type A/B, demand mode, safe/dangerous & detected/undetected, diagnostics vs proof test, common cause, reference conditions, force-guided contacts, proven-in-use, sensitivity analysis). Technical content unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
… counter) Counts control-plane (coord long-poll) connects; ml_reconnects = connects-1, so 0 = never dropped since boot and each increment is a soft DERP/control-plane re-establishment that a periodic poll could otherwise miss if it self-recovers between samples. Surgical: a volatile counter on the microlink struct incremented at the single CONNECTED-transition site in ml_coord.c, a getter, and one field in the shared dcs_support state.json builder (covers both remote and machine). No state-machine or timing change. Added ahead of the 6h final-verification soak. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
All reachable units (machn + PSTOP54/PSTOP42/DUT) OTA-updated to clean build 95a948b and soaked 6h11m (2864 samples). Result: zero reboots, zero fault resets, zero lockstep/xcheck/gpio/relay faults, zero heap leaks, zero unreachable samples on all four; DERP-relayed DUT held its link throughout. Live safety link PSTOP54<->machn: bonded every sample, 110,687 replies at 99.88%. Only signals: 2 machn control-plane reconnects (both in the first 14min settling; the new ml_reconnects field caught them) + ~19 brief self-healing rebonds over 6h. One logged anomaly total (a transient RECONNECTING at 00:49, self-healed). rst_hist WDT/USB entries are pre-soak. Report: docs/SOAK_TEST_2026-08-03.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
- Adjust message version to 0x01 - Adding two more requirements tests
… SPI margin
USB-NCM (the observed nuisance-STOP source): the safety send dropped a frame on a
transient TinyUSB NCM ERR_IF (all IN NTBs momentarily in-flight, errno -1) and waited
a full 200ms tick, so a run of refusals opened a >1.2s gap -> machine heartbeat STOP +
operator re-arm. Two complementary fixes (per IEC-researched proposal):
P1: CONFIG_TINYUSB_NCM_IN_NTB_BUFFS_COUNT 2->4 (both projects) — widens the IN NTB
free-list the refusal draws from (+6.4KB DRAM).
P2: sess_sendto() now retries an ERR_IF up to 3x, yielding vTaskDelay(1) between so
the lower-prio (5) TinyUSB task can free an NTB; the resend lands in the SAME
200ms tick, no gap. Gated on errno==-1 ONLY — route/ENOMEM (a genuinely dead
link) still fails on the first attempt, so dead-link detection latency is
UNCHANGED. New pstop_sf_txdrv_recovered counter in /state.json proves how many
transients the retry absorbed.
W5500 Ethernet (flawless in soak; proactive margin only): SPI input_delay_ns=20 for
sampling-edge margin, and LWIP_DHCP_RESTORE_LAST_IP for fast IP reacquire after a link
bounce. Deferred (need HW fault-injection to test, documented for follow-up): W5500
health-watchdog + reset ladder, instant-demote-on-link-down, TinyUSB task-prio bump,
TX mutex, static IP. Both firmwares build clean; no safety-logic change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
… fa8b442 validated USB-NCM fix validated on PSTOP54 (40min): 0 unrecovered txdrv drops, 5 transient ERR_IF absorbed by the retry, 0 steady-state rebonds (vs 17 drops + ~9 rebonds in the pre-fix 6h soak), 100% reply rate. Armed safety-function regression on the DUT via HIL relays: arm->RUN->press->STOP->re-arm all correct, no spurious STOP. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…+ v1 revalidation Transport fix rebased cleanly onto v1 (6276bf0). Combined v1+fix = 25c419f, revalidated on the wire: bond re-forms on far-Hamming codewords (~100% reply rate), armed DUT test (arm/STOP/re-arm) all correct, 15-min all-4 stability clean (0 reboots, 0 faults, 0 steady-state rebonds, USB retry active). Fleet updated: 25c419f published for both pstop_remote and machn_machine (first machn build on the fleet); all reachable units migrated to v1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
On a fresh production flash NVS is blank, so dcs_nvs_read_pstop_peers() takes the legacy-migration fallback and unconditionally marked slot 0 configured=true, filling ip/port from the legacy readers. Those readers substitute a hardcoded DEFAULT (10.42.0.1:8890) when the ps_ip key is absent, so a brand-new fleet-only remote (no pstop machine configured, only the pstop-fleet priority-peer) came up with a fabricated, unreachable machine slot -> the safety ring counted ncfg=1 and painted it yellow-flash (UNREACHABLE) instead of white (IDLE). Probe for the real legacy ps_ip key and migrate slot 0 only when it actually exists; otherwise leave the peer table zeroed (no machine configured). Genuine legacy installs still migrate; a real configured machine (present ps_peers blob) is unaffected and still flashes yellow when unreachable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
Blank-NVS fallback fabricated a default machine (10.42.0.1:8890) with configured=true; now migrates slot 0 only when the legacy ps_ip key actually exists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…lock_guard) Two CI failures introduced by earlier changes, now fixed: 1. Firmware build / DU-3 diversity guard: John Hinke's far-Hamming codewords (OK=0x55/STOP=0x92, 6276bf0) made core-1's boolean verdict compile to a branch+movi of the codeword immediates instead of moveqz/movnez, so the check's grep no longer matched even though the two cores are still genuinely diverse (core-0 table-load via k_estop_msg vs core-1 direct immediates). Updated the core-1 pattern to match cmov OR direct codeword-immediate materialisation, deriving the codeword values from pstop_msg.h so it survives future codeword changes; a genuine collapse (core-1 folded into the table) still fails. Verified PASS locally. 2. coverage host stage: machine_app_runner.c uses clock_guard_* (SR-H-04b) but the coverage build did not compile/link clock_guard.c -> undefined references -> collect2 ld error. Added clock_guard.c to the instrumented compile. Host binary now links clean. Not touched: the standing repo-wide pre-commit formatting debt (separate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…n / copyright / cpplint) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
pstop_c moved to v1 (PSTOP_VERSION 0x01; codewords OK=0x55/STOP=0x92/BOND=0xAD/ UNBOND=0x6A, 6276bf0), but pstop_multi_remote_test.py and stop_reset_battery.py still sent the old v0 values (0/1/2/3, version 0x00), so they could not bond against a v1 machine_app_runner. In CI this failed the coverage workflow's host stage (rc=1, host coverage dropped to 45.8%). Bumped both tools to the v1 version and codewords (+ NAMES maps); the constants are used symbolically elsewhere so no other change was needed. Verified: pstop_multi_remote_test 34/34, test_config_floor 7/7 against a freshly-built v1 runner. (test_config_floor.py is protocol-agnostic — config validation only — so it needed no change.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…le/Internet) Decided scheme (ring-only; onboard status LED untouched): an UNREACHABLE segment now blinks AMBER instead of a generic yellow flash, and the blink COUNT names the deepest broken network layer so a tech sees WHERE the break is: 1 blink = bonded peer down (Internet + Tailscale OK -> go to the machine) 2 blinks = Tailscale/control-plane down (ml_state != CONNECTED) 3 blinks = no Internet (dcs_net_inet_down) Device-level probes pick the count, shared by every unreachable segment. A REACHABLE machine still shows its real safety colour (same-LAN-direct needs no Internet). Amber (r>g) is distinct from red STOP and yellow; safety colours (purple mismatch, red STOP, green OK, blue bond, white idle) are unchanged and always win. Builds clean (remote+machn). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
Closes the deferred-P1 "unrecoverable W5500 SPI/PHY wedge" gap (ESP-IDF #11845, #12058, esp32.com "W5500 reset-timeout"): the W5500 can black-hole with NO ETHERNET_EVENT_DISCONNECTED and the netif still UP with a valid lease, so the 1 Hz route supervisor never demotes Ethernet and the link stays wedged until a human power-cycles. The system already fails safe (machine STOPs on heartbeat silence); this adds SELF-HEALING. Additive-only, in dcs_eth.c plus telemetry plumbing. Does NOT touch the safety comparator/send loop or the fail-safe-by-silence semantics. Detection (either signal, must persist 3 ticks @1 Hz): A. read-only esp_eth_ioctl(ETH_CMD_READ_PHY_REG on PHYCFGR) != ESP_OK — a wedged/timed-out SPI bus surfaces here. B. black-hole: driver link UP + valid lease + Ethernet is the active route, yet no pstop reply from ANY peer for 8s AND the uplink gateway is likewise silent (gateway cross-check discriminates a local W5500 wedge from a merely-unreachable peer — reuses existing freshness signals g_dcs_pstop_last_reply_ms + dcs_net_liveness). Recovery ladder (escalates only if the prior rung fails to restore link + an SPI-probe pass within 8s), under a new lifecycle mutex: 1. esp_eth_stop/start — clears a stuck driver FSM/link timer. 2. full driver+netif rebuild via the existing dcs_eth_start() path (driver_install re-runs phy->reset_hw + mac->init SW-reset+VERSIONR verify + phy->init — re-establishes SPI from scratch). 3. datasheet-correct hardware RST pulse on GPIO9 (assert >=T_RC 500us [1ms used], wait >=T_PL 1ms [5ms used] for PLL lock) THEN the same rebuild — last resort for a PLL/logic wedge the IDF's built-in 100us reset_hw pulse is too short to clear. Board: RSTn->GPIO9 via 20R series R (R22), no external RC/pull-up, so the GPIO drives it cleanly. On confirmation Ethernet is demoted at once (force link/lease false; rung-1 stop brings the netif down so the supervisor fails over to USB-NCM/WiFi). Runs on a non-safety PSRAM-stack task (prio 3); after a full ladder it holds a 60s cooldown so it can never thrash. /state.json gains: eth_recoveries, eth_rec_r1/r2/r3 (per-rung successes), eth_rec_reason (1=SPI, 2=black-hole), eth_spi_err. Builds clean for both firmware/ (remote) and machn/ (machine) — shared dcs_eth.c, no new warnings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
The ros2_build.yml leg was red on two real problems (validated + fixed against the exact jazzy CI container, full colcon build+test = 130 tests, 0 failures): - protective_stop_machine ament lint: flake8 I100 import order + pep257 D213 in machine_bridge.launch.py, and uncrustify code-width on 13 C/C++ files (whitespace/wrapping only — token streams unchanged, no logic touched). - archive/ packages (kept building per David a0.. "phased out" note, NOT ignored): protective_stop_remote exited 5 = NO TESTS RAN -> added tests_require=[pytest] so colcon runs pytest; protective_stop_node launch tests could cross-talk under parallel colcon (a neighbour leaked stop=True) -> pinned distinct ROS_DOMAIN_IDs. Workflow: jazzy is first + REQUIRED; humble/lyrical run informational (continue-on-error). rosdep-skip-keys for the Pi-only keys python3-rpi.gpio / python3-roslibpy (nav2_lifecycle_manager is NOT skipped — it resolves on jazzy and is required by the node launch tests). Robustness: ros2/ moved into the pre-commit global exclude alongside pstop_c/ — polymath clang-format/black and the ROS 2 ament linters (uncrustify/flake8) were double-formatting the same files and disagreeing, so exactly one owns ros2/ (ament, the ROS 2 standard). New scripts/ros2_lint.sh reproduces the ament lints locally via the jazzy container; wired as an opt-in pre-push hook (never on commit, no Docker imposed on the normal flow). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
The informational humble leg failed only on ament_uncrustify: humble ships uncrustify 0.72.0, jazzy ships 0.78.1, and 0.72 is stricter about multi-line call/lambda wrapping (first arg on its own indented line, shallower brace-body indent) plus one `/*is_stop_only=*/ false` comment-spacing case. 0.72s output is a MUTUAL FIXED POINT — jazzy 0.78 accepts it unchanged — so reformatting the 8 affected files with humble uncrustify makes both distros clean with no shared config needed. Whitespace-only: token streams byte-identical (verified), so the compiled node + its 130 tests are unaffected. humble/jazzy both: No problems found. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
Contributor
Author
|
Superseded by #63 — a clean single-commit squash of this branch onto main (identical tree, all CI green). Closing this draft. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Working branch consolidating everything that was in flight when main was reverted (c35bd95):
pstop_c coveragenow skips for dependabot-triggered runs, whose missing repo secrets (emptyBULLSEYE_LICENSE_KEY) caused the red X on the dependabot PRsDraft on purpose: do not merge until the downstream breakage that forced the revert is identified and fixed here.
🤖 Generated with Claude Code
https://claude.ai/code/session_01JRZvaANcTGNuMjJN2gVf8b