Permea BBB Audit

Do blood–brain barrier peptide predictors measure penetration, or their benchmarks?

A construct-validity audit of the standard BBB-peptide benchmark.

An open, fixed evaluation harness — reproduce every number on a laptop.

---

TL;DR

Published predictors of BBB-penetrating peptides report strong numbers on one shared benchmark (B3Pred Dataset_3). This repository re-evaluates that benchmark under identity-controlled splits and maps its data provenance — and finds that much of the reported performance reflects how the benchmark was built, not BBB penetration:

• Near-duplicate leakage inflates scores across the train/test split.
• Frozen ESM-2 embeddings give no advantage over simply counting amino acids, once the comparison is fair.
• Negatives are random proteins, not validated non-penetrators.
• The "independent" archives are not independent — their positive sequences heavily overlap.

We do not ship a new predictor. We ship an honest evaluation harness so any BBB / delivery-peptide predictor can be checked the same way — and everything below is reproducible from public sources on CPU alone.

Long-term aim: a standing, CASP-style honest-evaluation commons for sequence-based delivery prediction. This audit is the first step.

---

The finding, in four pictures

1 · Similarity-memorization leakage (P1)

Move from a random split (Condition A) to an identity-controlled split (Condition B) and discrimination drops — and the drop lives in the representation itself, not just the classifier. Across the 18 primary random-forest cells, 16 exclude zero.

2 · The leakage lives in the near-duplicate minority

Only ~2.5–5.1% of sequences fall into non-singleton identity clusters — but they hold ~25–50% of all positives, and essentially all of the random-split optimism concentrates there.

3 · Representation inflation (P2)

Model-matched (random forest vs. random forest), frozen ESM-2 embeddings never beat 20-dimensional amino-acid composition on this benchmark — and a 150M model is no better than a 35M one, across mean-, max-, and CLS-pool readouts.

4 · The archives are not independent

The two principal positive archives overlap ~54% verbatim and ~82–88% at near-duplicate identity, and B3Pred's positives derive from B3Pdb by construction — so a clean source-family holdout is structurally impossible here.

P3 (negatives). The negative class is random Swiss-Prot protein, not validated non-penetrators — length alone separates the two classes at ROC ≈ 0.64. So the task is partly "curated peptide pool vs. random protein."

Is the benchmark then worthless? No. Under the corrected, identity-controlled evaluation the baselines still produce a stable, well-calibrated ranking (~9× top-5% enrichment survives the correction). The audit is diagnostic, not nihilistic: it corrects what the scores mean, it doesn't burn the benchmark down.

---

The pillars at a glance

	Question	Finding
P1 — Leakage	Do near-duplicates straddle the split?	Yes. Discrimination drops under identity control; 16/18 cells exclude zero.
P2 — Inflation	Do frozen PLM embeddings beat counting amino acids?	No. ESM-2 ≈ composition, model-matched; 150M ≈ 35M.
P3 — Negatives	Are the negatives real non-penetrators?	No. Random Swiss-Prot; length alone gives ROC ≈ 0.64.
Overlap	Are the public archives independent?	No. ~54% verbatim / ~82–88% near-duplicate overlap.

Benchmark audited: B3Pred Dataset_3 — 2,959 peptides (269 positive / 2,690 negative), peptide lengths ≈ 5–30 aa. Canonical file identified by SHA-256 2e8d3f…8abf.

---

What's in this repository

• data/ — regenerate.py (rebuilds the benchmark from its public source) and manifest.sha256 (the checksum it verifies against). No row-level data.
• harness/ — the evaluation engine: Condition A vs. B, cluster-level bootstrap confidence intervals, the label-permutation control, Benjamini–Hochberg FDR, and enrichment.
• configs/ — thresholds, folds, and hyperparameters.
• results/ — all aggregate outputs (metrics with confidence intervals). No row-level data.
• figures/ — figure regeneration (the canonical figures shown above).
• supplementary/ — Supplementary Methods & Tables.

The ESM-2 embedding cache is not stored in this repo (to keep it lightweight): run.sh fetches it as a separate Zenodo asset, or you can regenerate it with the embed script.

Aggregate-only by design. No row-level sequences, per-row predictions, or embedding matrices are redistributed. The benchmark is rebuilt-then-verified from its public source, never re-hosted.

---

Reproduce it yourself

Everything reproduces on commodity hardware — CPU only, in minutes.

git clone https://github.com/Permea-lab/permea-bbb-audit
cd permea-bbb-audit
pip install -r requirements.txt

bash run.sh       # regenerate → verify SHA-256 → fetch embedding cache → evaluate → figures

What run.sh does:

flowchart LR
    A["Public source<br/>(Zenodo DOI, MIT)"] -->|regenerate| B["Canonical dataset<br/>verify SHA-256"]
    B --> C["Identity clustering<br/>MMseqs2 + alignment"]
    F["ESM-2 embedding cache<br/>(fetched from Zenodo)"] --> D
    C --> D["Condition A vs B<br/>evaluation harness"]
    D --> E["Aggregate results<br/>+ figures"]

Loading

• No GPU needed — run.sh fetches the precomputed ESM-2 embedding cache (a separate Zenodo asset), so a laptop reproduces every reported number on CPU.
• The dataset is rebuilt from a permanent public source (Zenodo 10.5281/zenodo.20093418, MIT License) and verified against the published SHA-256. Nothing curated is re-hosted.
• The MMseqs2 binary ships under tools/mmseqs/bin and must be on your PATH for the clustering step.

Want to audit your own predictor? Point the harness at your scores and run the same identity-controlled, model-matched protocol — that's what it's for.

---

What we claim — and what we don't

We claim: (i) an identity-controlled re-evaluation of the standard BBB-peptide benchmark with paired bootstrap confidence intervals; (ii) a shared-source provenance table + cross-dataset overlap map; and (iii) an open, fixed evaluation harness.

We do not claim: a new predictor or state-of-the-art; that BBB prediction is impossible; that ESM-2 is useless in general (only that it adds nothing on this benchmark, fairly compared); or any wet-lab, transport-mechanism, or clinical result. All findings are computational evaluation findings.

---

Citation

If you use this harness or its results:

@software{permea_bbb_audit_2026,
  author    = {Kim, Albert Heekwan and Papon, Rémi},
  title     = {Permea BBB Audit: a construct-validity audit of BBB-peptide benchmarks},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.21134112},
  url       = {https://github.com/Permea-lab/permea-bbb-audit}
}

• Concept DOI (always resolves to the latest release): 10.5281/zenodo.21134112
• Preprint: bioRxiv — link added on posting (BIORXIV/2026/736171)
• Machine-readable metadata: CITATION.cff

---

License

• Code / harness: MIT — see LICENSE.
• Manuscript & figures: CC-BY 4.0.
• Source dataset: B3Pred Dataset_3, MIT License (Kumar et al., 2021; Zenodo 10.5281/zenodo.20093418) — regenerated, not redistributed.

---

Albert Heekwan Kim · Rémi Papon — Permea Labs Correspondence: a.kim@permea.us

_{Permea Labs · open infrastructure for sequence-first biological delivery prediction}