Feature/reena-steven-integrated-endpoint - #328
Open
rmobmina wants to merge 17 commits into
Open
Conversation
…aster DB list Adds per-eFP-project regex validation (EFP_PROJECT_REGEXES, is_efp_gene_valid) covering both canonical gene IDs and microarray probeset IDs, with a database-to-project mapping (DATABASE_EFP_PROJECT) so gene_expression.py can validate against the right format per database. Adds the eFP+ePlant discovery/validation pipeline: - scrape_view_databases.py / scrape_species_view_info.py: live-discover every eFP and ePlant view across 55 sites, tagging eFP-only vs ePlant-only vs both - build_proj_id_view_mapping.py: resolves multi-paper databases (e.g. atgenexp_*) to per-view proj_id breakdowns - validate_db_regex_coverage.py: tests every database's real sample IDs against production validation, including legacy databases no longer linked from a live dropdown - build_master_db_list.py: rolls everything up by species, tagging source (efp/eplant/both/legacy) and platform (microarray/rna_seq) 193 databases validated, 0 unexplained failures.
… endpoints to use it - Merge heterodera_schachtii into arabidopsis (it's an Arabidopsis-gene-ID database, not a separate species); fix brassica/lupin/triticale scientific names per professor's review comments - Load EFP_PROJECT_REGEXES and DATABASE_EFP_PROJECT from Vincent's regex_master_list_efp_eplant registry (58 grouped project patterns instead of 193 per-database ones), embedded into combined_master.json at build time; empirically verify every regex_project assignment against real sample dumps before trusting it, demoting 3 unverified species-fallback guesses and fixing 2 misassigned projects (maize_lipid_map, tomato_trait) - Add a general injection checker (BARUtils.is_injection_attempt) that runs before the probeset-shape check in /gene_expression, closing a bypass where any string ending in "_at" skipped format validation entirely - Derive schema_variants directly from real sample_data column dumps instead of a coarse rna_seq/microarray platform split, so every one of the 193 databases is schema_verified against its own real columns (was 146/193) - Refactor /microarray_gene_expression/<species>/databases and .../<species>/<view>/samples to read from combined_master.json instead of a hardcoded dict and a since-deleted JSON file (the latter was broken on this branch); add test coverage for both - Add api/utils/master_data_utils.py as the shared combined_master.json loader used across bar_utils, gene_id_utils, and the refactored resource
These were leftover exploratory outputs (regex audit dumps, coverage CSVs, speed benchmark plots, old email drafts) not consumed by any script in the active build pipeline. db_regex_coverage_report.csv is kept since build_master_db_list.py reads it as an input.
These "Reena Obmina | ... Project | University of Toronto" docstring lines and "# Endpoint made by Reena" comments were course-assignment artifacts, not something that belongs in a shared codebase.
build_databases() collected each database's used_by entries from a Python set, so re-running the exact same build against the exact same inputs produced a different (but equivalent) entry order every time, due to Python's per-process string hash randomization. Sort before iterating so reruns are byte-identical -- important since this file is meant to be diffable for review, not just functionally correct. Regenerated data/efp_info/combined_master.json with the fix; verified two consecutive reruns now produce byte-identical output, and the only diff from the previous version is list ordering (no data changed).
- test_interactions.py::test_mfinder special-cased GITHUB_ACTIONS to skip the real /bartmp/mfinder subprocess call, but CI doesn't actually have that binary installed either -- generalize the check to "skip whenever the binary isn't present" so it behaves the same in CI, locally, or anywhere else without mfinder installed. - Delete tests/resources/test_eplant_arabidopsis.py: it tests api.resources.expression.get_expression and the route /expression/ePlant_expression, neither of which exist anymore -- that module was renamed/replaced by the unified /expression/<database>/<gene_id> endpoint in gene_expression.py, which already has its own Arabidopsis coverage in test_gene_expression.py. No coverage lost.
These 8 import_if_missing lines (light_series, maize_RMA_linear, meristem_db, potato_stress, seedcoat, soybean, soybean_severin, tomato_ils) could never actually run: bootstrap_simple_efp_dbs.py runs earlier in the same script and already creates all of these as empty databases from the schema registry, so db_exists() always returns true and the import is skipped before it ever reaches the SQL file. Most of those files are also gitignored multi-GB dumps that don't exist in a checkout anyway (soybean.sql doesn't exist at all). Removing dead lines that can never execute rather than leaving confusing no-op code in a script CI actually runs.
Walks every eFP browser view for every species in data/efp_info/combined_master.json and checks whether each "To the Experiment" hyperlink shown in the expression-value table actually resolves to something real, instead of erroring out or looping back into the eFP browser itself. Ignore its scratch output directory (scripts/results/) rather than letting the generated CSV/log get committed by accident.
# Conflicts: # api/utils/bar_utils.py # config/init.sh
BARUtils had 46 hand-maintained is_<species>_gene_valid() methods that duplicated what Vincent's tested regex registry (EFP_PROJECT_REGEXES, loaded from combined_master.json) already covers per eFP project -- exactly the kind of drift-prone duplication that caused the sorghum regex merge conflict with upstream/dev. Delegate 41 of them to is_efp_gene_valid() so there's one tested source of truth instead of two. The remaining 5 (cassava, cuscuta, spruce, willow, quinoa) have no matching project in Vincent's registry and stay hand-maintained. Verified every migrated species against every real sample ID in api/random_rows_json/ before switching: 26 were byte-identical, no behavior change. The other 15 (including cacao's 3-way OR and mouse's differently-named mouse_efp key) do change behavior, knowingly: - Most now also accept microarray probeset IDs (e.g. 253680_at, Contig7905_at) as "valid genes" for that species, since Vincent's registry validates "any real value in this database's sample_data" rather than "a canonical gene ID a human would type." - Rice loses isoform-suffix support entirely (LOC_Os01g01430.1): efp_rice has no isoform variant of the LOC_Os form. This affects /loc/rice/<gene>'s real primary use case, not just an edge case. - Wheat, canola, and potato lose a few legacy real-world formats (e.g. TrturSVE..., BrBA_...V4, EPlSTUG...) that efp_wheat/ efp_canola/efp_potato don't cover. - Arabidopsis gains a two-digit isoform suffix (At1g01010.11) and loses a previously-tolerated malformed form (AT1G010400, an extra trailing digit with no separator). Also fixes is_efp_gene_valid() itself: it was matching without re.IGNORECASE while every legacy method it's now replacing used re.I, which broke case handling for callers like efp_data.py that uppercase the gene ID before validating (e.g. rejected a real, valid Arachis ID). And added a None/empty guard so callers that pass an absent gene id don't crash instead of failing validation. Updated test_bar_utils.py, test_gene_information.py, and test_localizations.py to match the new, intentional behavior.
config/init.sh ran bootstrap_simple_efp_dbs.py (creates an empty sample_data table for all 191 "simple" eFP databases) BEFORE the static SQL imports. Several databases -- shoot_apex, arachis, single_cell, arabidopsis_ecotypes, cannabis, dna_damage, embryo -- are both a "simple" schema AND have a small curated real-data dump in config/databases/. Since bootstrap always created their table first, db_exists() saw it as "already there" and import_if_missing skipped loading the real row every time, leaving these tables permanently empty in any genuinely fresh environment (CI, a new dev machine). This only looked fine locally because that database already had the real data loaded from long before the bootstrap step existed. Fixed by running the static imports first (so a fresh database gets its real data loaded), then bootstrapping the rest -- safe because ensure_schema() uses checkfirst=True and never touches a table that already exists. Also added cannabis, dna_damage, and embryo to the import list; their real dumps existed in config/databases/ but were never referenced anywhere, going all the way back to before the import_if_missing refactor. Verified locally: dropped these three databases entirely, replayed the corrected sequence (real-data insert, then a second bootstrap pass), and confirmed the real row survives and the dependent tests pass with it. Also pinned rpds-py to 0.30.0 (was 2026.5.1, from upstream/dev's "Dependancies updated." commit): 2026.5.1 requires Python >=3.11 and has no compatible release for the 3.10 job in the CI matrix, which made `pip install -r requirements.txt` fail outright on that job before tests could even run. 0.30.0 is the newest version with a 3.10 wheel and has wheels for 3.11/3.12/3.13 too.
193 files, ~52k lines of raw per-database sample dumps -- used only as input to build_combined_master_json.py's schema/regex verification against real data, never read at runtime by the API or by any test. Too large and noisy to be worth reviewing line-by-line in a PR. Gitignored (not deleted) so the build script still works locally for whoever has a copy; regenerate combined_master.json before removing locally without a backup.
Integrates the UMAP coordinate/expression endpoint and per-tissue pseudobulk gene expression endpoint (originally PR BioAnalyticResource#327) onto the current combined_master.json-driven schema catalog, since the old pre-refactor helpers that PR was written against no longer exist. Adds 7 new pseudobulk databases to the schema catalog (species + gene_id_class sourced from combined_master.json, hand-added pending the real XML scrape/verify pass once that metadata is available) and wires their MySQL binds into CI config. The pseudobulk sample_data tables carry one extra data_signal_std column beyond the standard 3-column schema; query_efp_database_dynamic only selects columns by name so this needed no schema changes. UMAP databases are queried directly via raw SQL and are intentionally left out of the schema catalog since they use a different table shape.
…dpoints Fills in the success/404/500 response paths that the earlier validation-only tests never reached: mocks query_efp_database_dynamic for the pseudobulk endpoint (unit tests _split_rows directly plus the mean/tissue split, probeset-passthrough, and error-code mapping), and registers an in-memory SQLite engine as the MySQL bind for the UMAP endpoint to exercise the real query/case-retry/merge logic without a live database. Also fixes two injection-attempt tests that were silently passing for the wrong reason: markupsafe's escape() HTML- entity-encodes quote characters before is_injection_attempt() runs, so a quote-based payload no longer trips the injection regex by the time it's checked -- switched both to a UNION SELECT payload instead. 100% line and branch coverage on both new endpoint files.
Contributor
|
Hi @rmobmina , remove superfluous files like "db_regex_coverage_report.csv", thanks. |
VinLau
reviewed
Jul 27, 2026
Contributor
There was a problem hiding this comment.
We don't need this as a seperate file, overengineered (Claude Code does this often), move it to another utils file, keep our project minimal.
Contributor
There was a problem hiding this comment.
eFP XMLs superfluous, remove these from PR
Contributor
There was a problem hiding this comment.
@rmobmina for scripts and checkups you do NOT need to add them to the shared gitignore but your .git/info/exclude. So if you want Claude to see the scripts as part of your conversation add the scripts to exclude.
Untracks the one-off scripts and intermediate data artifacts that feed into building data/efp_info/combined_master.json (scraping, regex registry/CSV reports, human XMLs, coverage validation, etc.) -- none of them are imported or read by api/ or tests/ at runtime, only combined_master.json itself is. Confirmed via full-repo grep before removal. Files stay on disk locally (git rm --cached, not a hard delete) and are now listed in .git/info/exclude so they don't get re-added.
…abases Replaces the placeholder combined_master.json entries (empty views, schema_verified: false) added when the XML metadata wasn't available yet with real data parsed from the ePlant-style view XML now on hand (api/ePlant_XML/), using the same extract_db()/parse_groups() the existing eFP Human XML patch already relies on. Corrects source from the earlier guess of "efp" to "eplant" -- the XML's own webservice URL confirms these are served through ePlant's plantefp.cgi, not eFP. Two of the eight XML files (at_seed_martin.xml, at_seed_l3_martin.xml) share one underlying database but are genuinely two different views (L1 tissue overview vs. L3 embryo subclustering) -- both kept as separate view_info entries. Fixed along the way: a stray line of unrelated text embedded mid-JSON in efp_eplant_species_view_info.json that was breaking the build script's JSON parsing entirely, and several malformed self-closing tags (<sample>, <control>) in arabidopsis_nat.xml that broke XML parsing -- confirmed via full-file scan that no other XML files in the set have the same issue. Verified via full diff against the prior combined_master.json that only these 7 databases' entries changed -- no other database drifted. Full test suite still at the same pre-existing-failure baseline. The XML source files themselves and the build script's local changes stay untracked (.git/info/exclude), matching how the eFP Human XML source is already handled -- only the derived combined_master.json is committed.
…bar_utils, and scope build-script exclusions to .git/info/exclude Removes the api/ePlant_XML/ sample XMLs (local-only build-script input, not read by the API or tests). Merges load_combined_master() into bar_utils.py instead of keeping it as a separate one-function utils file (bar_utils already imported it and was its first consumer, so no circular-import risk). Moves scripts/results/ and api/random_rows_json/ out of the shared .gitignore and into .git/info/exclude, since they're local script/checkup output rather than something the team needs ignored project-wide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.