Feat/cell fate estgel - #67
Open
blueee04 wants to merge 5 commits into
Open
Conversation
Reframes the task from whole-graph WT-vs-RNAi (which EPIC cannot support —
the files are almost all wild-type reporter strains) to node-level cell-fate
prediction, which the invariant C. elegans lineage labels for free.
- src/cell_fate.py: 13-class tissue taxonomy + CellFateMap. Terminal cells are
classified from the canonical WormAtlas descriptions shipped in DevoWorm's
cell-phenotype-lineage-data.xlsx; progenitors get the consensus fate of their
lineage descendants with a purity score ('mixed' below 60% agreement).
- scripts/build_cell_fate.py: generates Dataset/fate/cell_fate.csv
(2,769 cells, 2 unknown) with method/purity/n_leaves provenance columns.
- src/estgel_node.py: ESTGELNodeClassifier reusing the EAM -> DRL -> DNL block.
Runs the recurrence strided across the FULL developmental time and pools each
cell over its own alive steps, so late-born cells (~70% of them) get real
embeddings. Includes per-feature standardization (EPIC ships raw values; blot
reaches 1.6e5, which otherwise explodes the head and diverges training).
- scripts/train_fate.py: by-embryo split, macro-F1 + per-class recall reporting,
optional --resume / --cosine.
The cell NAME is never a model feature — fate is predicted from position, size,
expression and graph structure only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The EAM attention weights saturate near 1.0 and so carry little signal, so importance is measured by perturbing inputs instead of reading attention. - scripts/saliency_fate.py: feature occlusion (mean-impute each input and measure the confidence drop), graph ablation (drop every cell-cell edge), and gradient saliency. - scripts/extract_fate_explainability.py: per-cell fate predictions plus a tissue x tissue contact / attention map for one embryo. Key result: removing all cell-cell edges flips 43% of fate predictions and drops accuracy sharply — the graph structure causally drives fate, which is the ESTGEL premise. Feature-wise, position dominates, cell size distinctively drives intestine and pharynx, and blot (reporter fluorescence) contributes ~nothing, as expected since each embryo carries a different reporter. Checkpoint (35 epochs): val accuracy 0.635, macro-F1 0.343. Per-class recall pharynx 0.80, hypodermis 0.72, muscle 0.72, neuron 0.62, intestine 0.60; the four <1% classes stay near zero under unweighted training. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the old WT-vs-RNAi/attention dashboard, which was built on the superseded task and loaded three.js/Chart.js from CDNs. dashboard/index.html is self-contained (no external requests, data inlined) and hand-rolls its 3D canvas: - embryo cells colored by predicted tissue fate, rotatable and zoomable - plays development forward across 30 interpolated frames; cells emerge from their parent at division rather than popping in - per-frame cell-cell contact edges, brightness scaled by prediction confidence, red rings on misclassified cells (both toggleable) - click-to-inspect: predicted fate, confidence, true fate, parent, birth time - explainability panels: graph ablation, feature occlusion, tissue contact map, training curve, per-class recall Regenerate for any embryo with: python scripts/export_dashboard_data.py --embryo <name>.npz --n-frames 30 which rebuilds the data blob and injects it into fate_dashboard.template.html. Removes dashboard/app.js, style.css, embryo_data.json and explainability_report.json (all superseded). PROGRESS.md records the two bugs that were blocking training (missing feature normalization and the temporal-coverage bug) and the final results. DATASET_MEMO.md documents why the EPIC WT-vs-RNAi target was degenerate and compares candidate datasets (WDDD/SSBD, CShaper, Digital Development) for the mutation dimension. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scripts/export_dashboard_data.py runs the trained fate model on one embryo and writes the dashboard data blob (per-cell fate, confidence, 3D positions over time, lineage parents, per-frame co-alive contact edges), then injects it into the template to build the self-contained dashboard/index.html. Without it the dashboard could not be regenerated. src/embryo_labels.py was imported by the already-tracked scripts/train_estgel.py but had never been committed, so a fresh clone failed on import. Committing it as-is to fix the clone; note its filename-keyword heuristic is what produced the degenerate 254-vs-6 WT/RNAi labels described in DATASET_MEMO.md, and it is not used by the cell-fate pipeline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.