This repository holds one focused experiment that decides whether a proposed control-theory paper is worth pursuing. If you are an AI or a person who just opened this repo, read this file top to bottom and you will understand exactly what is being tested and why.
A safety controller keeps a robot inside safe limits by assuming "my model could be wrong by at most this much." That cushion is usually set large and conservative. The proposed paper wants to shrink the cushion automatically when recent data proves the model is accurate — letting the robot act less conservatively without becoming unsafe.
The twist the paper is built on: a well-tuned controller keeps the robot calm and on-target, which means the robot stops doing the maneuvers that would reveal whether the model is actually right. So "everything looks fine" can be false confidence — the data is quiet because control is good, not because the model is correct. The paper proposes an external admissibility test that only allows the cushion to shrink when the recent data is genuinely informative about the specific quantity the safety limit depends on.
There is already a strong 2022 paper — Barrier Bayesian Linear Regression (BBLR), Brunke, Zhou & Schoellig, arXiv:2204.03801 — that learns the model error right at the safety boundary and tracks a Bayesian confidence in it. The standard property of a Bayesian estimate is that its predicted uncertainty grows automatically when the data is uninformative.
If BBLR's own posterior already widens (and therefore refuses to shrink the cushion) in exactly the "false confidence" situation the paper is about, then the proposed external test is redundant and the paper is dead as framed.
This repo's single job: find out whether that is true.
A simulated second-order system (q, v) has a safety limit q <= q_safe and an
unknown term that is silent near the operating point and grows toward the
wall (delta(q) = theta * phi(q), with phi(q) ≈ 0 near q=0). A PD
controller regulates the system; when it holds q near 0 it never excites the
unknown term — that is the feedback-blindness.
We run BBLR alongside the proposed external admissibility test across five scenarios and measure, at every step:
- Is BBLR confidently wrong? (its predicted std is small, yet the true error falls outside its own claimed interval). This is the dangerous event.
- Does the external test flag "don't trust / don't tighten" before that? (lead time ≥ 0), and does it stay open in genuinely informative regimes (so it isn't a useless always-block)?
| ID | Setup | What it probes |
|---|---|---|
| S1 | well-excited, correct features | sanity: BBLR should stay calibrated |
| S2 | blind training then excursion, correct features | the strongest "BBLR already handles it" case |
| S3 | blind, misspecified (linear) features | can BBLR be confidently wrong? |
| S4 | blind, local RBF features | does Bayesian OOD inflation save BBLR? |
| S5 | excite, contract posterior, then jump the dynamics | nonstationarity |
Run python -m bblr_test.run_experiment. The driver prints one of:
- GO — at least one blind regime where BBLR is confidently wrong and the external test warns with non-negative lead time. The paper survives, but its honest scope is whatever regimes produced "SURVIVES".
- REDUNDANT — BBLR self-protects everywhere (never confidently wrong). The proposed test adds nothing → do not write this paper as framed.
- NO-GO — BBLR is confidently wrong somewhere but the test fails to warn in time → the proposed mechanism does not work as built; redesign first.
See docs/decision_criteria.md for the exact logic.
The committed run lands on GO, but narrowly and honestly:
- S2 (blind, correct features) → REDUNDANT. With a correctly specified model, BBLR's posterior stays wide and it refuses to tighten on its own. Against well-specified BBLR, the proposed test is redundant — the original reviewer concern is confirmed.
- S3, S4, S5 → SURVIVES. BBLR becomes confidently wrong under feature misspecification (linear and local-RBF) and under abrupt dynamics change, and the external test flags early while staying open in the excited regimes (flag fraction ≈ 0.24 when calibrated vs ≈ 0.79 when blind).
Interpretation: the publishable paper is not "we detect when barrier data is uninformative" (BBLR does that under its own assumptions). It is the narrower, honest claim: "Detecting false posterior concentration before online robust-bound contraction — when BBLR's barrier uncertainty is untrustworthy under feedback (misspecification, nonstationarity)."
pip install -r requirements.txt
python -m bblr_test.run_experiment # prints verdict, writes results/
python -m pytest tests/ -q # smoke testsNo GPU, no hardware, no external data. Pure simulation — which is required here, because deciding "BBLR is confidently wrong" needs ground-truth model error, and you only have that in simulation. (Hardware would hide the very thing we're testing; it belongs only in a later journal-strengthening pass.)
bblr_test/ the simulation package
plant.py true dynamics + barrier-activated unknown term
features.py correct / misspecified / local-RBF feature maps
bblr.py Barrier Bayesian Linear Regression (the incumbent)
controller.py PD regulator (+ optional probing)
admissibility.py the PROPOSED external test (info gate + anytime monitor)
metrics.py confidently-wrong detection, lead time, verdict logic
scenarios.py the five scenarios + single-run simulator
run_experiment.py driver: runs all, prints decision, writes results/
docs/ thesis, decision criteria, and full research context
tests/ smoke tests
results/ committed metrics JSON + per-scenario plots
This experiment is the gate for a larger research program. The complete
literature audit, nearest-prior-art ledgers, threshold options, and the
four-week protocol that this Week-0 test sits in front of are in
docs/context/. Start with
docs/context/bblr_redundancy_assessment.md.