This repository reproduces every number and figure in the paper How Much Human Label Variation Does Formal Semantic Structure Explain? Group-Level Effects and Item-Level Ceilings in NLI (under anonymous review). It contains the complete analysis code, a pinned data-fetching script, and an English audit trail of the project's preregistrations, deviations, and negative results (AUDIT.md).
In brief, the paper measures how much of human label variation in NLI is explained by formal semantic structure, and answers with bounds. A preregistered group-level boundary exists: hypotheses that are not purely upward monotone show higher annotator entropy (binary Cliff's delta -0.284), and this survives two registered alternative explanations (operator presence alone, and length or parsing complexity). At the item level the same structure explains little: 3.3 to 3.6 percent of entropy variance, median-split AUC 0.606, and three high-powered null contrasts show the composition of the variation does not differ across the boundary. All results are conditioned on ChaosNLI's low-agreement item selection.
Python 3.13.9. Install the pinned dependencies and the spaCy model:
pip install -r requirements.txt
python -m spacy download en_core_web_trfNo GPU is required; all published numbers were computed on CPU.
The four datasets are not redistributed here. Fetch them from their original sources, pinned to the exact versions the published results used and verified by sha256:
./fetch_data.sh| Dataset | Source | Pinned version | License | Items used |
|---|---|---|---|---|
| ChaosNLI | github.com/easonnie/ChaosNLI (zip link in its README) | v1.0 | "Creative Commons-Non Commercial 4.0" (verbatim from its bundled README.txt) | SNLI 1,514 + MNLI-m 1,599 = 3,113 (alphaNLI subset excluded: abductive format) |
| VariErr NLI | github.com/mainlp/VariErr-NLI | commit b2cbdd3 | no license file upstream | 500 |
| LiTEx | github.com/mainlp/litex | commit 13877cd | Apache 2.0 | varierr subset (498 of the three-way overlap) |
| MED | github.com/verypluming/MED | commit 860ba36 | CC BY-SA 4.0 | 5,382 |
Note: the released MED.tsv class counts (upward 1,818 / downward 3,272) differ from the MED paper's Table 5 (1,820 / 3,270) by two items. The analysis uses the released file; the discrepancy is disclosed in the paper.
Run each script as a module from the repository root, in this order (later scripts read the derived files of earlier ones):
| # | Command | Output (data/derived/) | Backs |
|---|---|---|---|
| 1 | python -m analysis.measure_overlap |
overlap_ids.json | dataset overlap, sample construction |
| 2 | python -m analysis.validate_med |
med_validation.json | tagger validation (0.883 edit-site, 0.807 sentence-level) |
| 3 | python -m analysis.tag_chaosnli |
chaosnli_profiles.jsonl (+ meta) | operator profiles, 3,113 items |
| 4 | python -m analysis.compute_outcomes |
chaosnli_outcomes.jsonl (+ meta) | outcome variables (entropy, margin) |
| 5 | python -m analysis.derive_power_rule |
stdout only | preregistered power rule (SESOI 0.33) |
| 6 | python -m analysis.compare_monotonicity |
phase22_results.json | main group comparison |
| 7 | python -m analysis.crossval_phase3 |
phase3_results.json | triangulation contrasts C1 to C3 |
| 8 | python -m analysis.compute_complexity |
chaosnli_complexity.jsonl (+ meta) | length and tree-depth covariates |
| 9 | python -m analysis.robustness_phase2r |
phase2r_results.json | robustness R-A to R-E |
| 10 | python -m analysis.robustness_groupc |
groupc_results.json | robustness C-R1 to C-R5 |
| 11 | python -m analysis.plot_core_signal |
figures/*.pdf, *.png | Figure 1 |
Every stochastic step uses the fixed seed 20260706 (bootstrap resampling and cross-validation folds); the tagger and parsers are deterministic. Steps 3 and 8 parse all items with en_core_web_trf and take about 70 seconds each on CPU; step 6 takes about 43 seconds (10,000 bootstrap resamples); the remaining statistics scripts finish in seconds.
Each script asserts its own input integrity (row counts, id uniqueness, value ranges) and refuses to write outside data/derived/. Implementation discretion is documented in each script's docstring.
The full dated log, including what had been seen at each registration point, is in AUDIT.md. The short version:
- 2026-07-04. Datasets collected and verified (counts and checksums).
- 2026-07-05. Overlap cutoffs preregistered, then measured: the planned e-SNLI cross-validation path had overlap 0 and was replaced by the LiTEx varierr subset (498 items, all MNLI). Tagger v0.1 failed its preregistered validation cutoff on MED; the metric was realigned to the edit site (a construct correction, disclosed, with the old metric still reported); v0.2 failed again (a recorded negative result); v0.3 passed all registered cutoffs (0.883 overall).
- 2026-07-06. Main comparison preregistered and run: upward hypotheses show lower entropy (delta -0.271 vs downward, -0.292 vs non-monotone). Triangulation preregistered and run: all three contrasts null at power 0.99+. Robustness block preregistered and run: signal survives operator presence and complexity controls, but item-level variance explained is 3.3 to 3.6 percent and AUC 0.606. The project's framing was demoted from an item-level map to a group-level boundary accordingly, and the planned extension module was dropped by its registered start condition.
- 2026-07-07. The project directory was accidentally deleted and fully restored the same day; all regenerated effect sizes matched the log exactly. Git commit objects after the second preregistration commit were lost, so later registrations rest on timestamped session transcripts. Details in AUDIT.md. Group C robustness preregistered and run; paper drafted.
- 2026-07-10 to 07-11. External review patches (text only), full citation verification, end-to-end reverification of all 27 quoted numbers, LaTeX conversion.
The code is MIT licensed (see LICENSE). The datasets keep their own licenses (table above) and are deliberately not redistributed. The paper is under anonymous review; citation information will be added here after de-anonymization.