Unified worker sizing, resilient bulk OSM/NLD downloads, faster branch processing - #12
Open
supathdhitalGEO wants to merge 2 commits into
Open
Unified worker sizing, resilient bulk OSM/NLD downloads, faster branch processing#12supathdhitalGEO wants to merge 2 commits into
supathdhitalGEO wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallelization
fimbox._workers: one place that answers "how many workers can this machineactually run?" Auto-sizes to
min(cpu, RAM / per_worker), honours an explicit1for serial debugging, clamps optimistic requests, and never spawns more workers than
tasks. Overridable via
FIMBOX_WORKERS/FIMBOX_RAM_PER_WORKER.n_workersnow defaults toNone("size to the machine") across branch processing,FIM generation, and every rating-curve calibration step — previously hardcoded
1/4._dask.pysizing now delegates to the shared resolver, son_workers=64on a laptoplands on what the laptop can feed instead of thrashing swap.
n_workers × tile_workers,capped) with a tile cache, instead of a pool per footprint.
Bulk OSM download
_OverpassClientfor roads and bridges: probes all mirrors in parallel upfront, permanently drops dead hosts for the session, and treats an Overpass
remark(query timed out inside an HTTP 200) as a failure rather than partial success.
mirrors, and quadrant sub-splitting for tiles that fail — this is what makes large
AOIs work.
allow_partial=Trueto override).
(no list columns / duplicate-cased columns that used to break GPKG writes).
NLD download
warns instead of silently returning nothing.
NLDQueryErrorkeeps "failed download" distinct from "this AOI has no levees" —a missing levee burn quietly changes the HAND surface. Layers are attempted
independently and failures reported loudly.
Performance
_d8.downstream_index, replacingdeque-basedtraversal in flow accumulation and gage-catchment labelling.