Skip to content

Unified worker sizing, resilient bulk OSM/NLD downloads, faster branch processing - #12

Open
supathdhitalGEO wants to merge 2 commits into
mainfrom
dev-sd
Open

Unified worker sizing, resilient bulk OSM/NLD downloads, faster branch processing#12
supathdhitalGEO wants to merge 2 commits into
mainfrom
dev-sd

Conversation

@supathdhitalGEO

Copy link
Copy Markdown
Member

Parallelization

  • New fimbox._workers: one place that answers "how many workers can this machine
    actually run?"
    Auto-sizes to min(cpu, RAM / per_worker), honours an explicit 1
    for serial debugging, clamps optimistic requests, and never spawns more workers than
    tasks. Overridable via FIMBOX_WORKERS / FIMBOX_RAM_PER_WORKER.
  • n_workers now defaults to None ("size to the machine") across branch processing,
    FIM generation, and every rating-curve calibration step — previously hardcoded 1/4.
  • _dask.py sizing now delegates to the shared resolver, so n_workers=64 on a laptop
    lands on what the laptop can feed instead of thrashing swap.
  • Bridge LiDAR fetching shares a single bounded thread pool (n_workers × tile_workers,
    capped) with a tile cache, instead of a pool per footprint.

Bulk OSM download

  • Shared _OverpassClient for roads and bridges: probes all mirrors in parallel up
    front, permanently drops dead hosts for the session, and treats an Overpass remark
    (query timed out inside an HTTP 200) as a failure rather than partial success.
  • AOI tiling with off-polygon tiles dropped before any request, round-robin across live
    mirrors, and quadrant sub-splitting for tiles that fail — this is what makes large
    AOIs work.
  • Raises instead of writing a file that looks complete but has holes (allow_partial=True
    to override).
  • Bridges now come straight from Overpass rather than osmnx, with a curated tag schema
    (no list columns / duplicate-cased columns that used to break GPKG writes).

NLD download

  • Layer ids resolved by name at runtime (fallback to 15/16), so a USACE renumbering
    warns instead of silently returning nothing.
  • New NLDQueryError keeps "failed download" distinct from "this AOI has no levees"
    a missing levee burn quietly changes the HAND surface. Layers are attempted
    independently and failures reported loudly.
  • Levee Z-geometry path now reuses the paged, shrink-on-failure fetch.

Performance

  • Numba-JIT D8 routing shared via _d8.downstream_index, replacing deque-based
    traversal in flow accumulation and gage-catchment labelling.
  • SRC filtering precomputes reach ladders once instead of per-key.

@supathdhitalGEO supathdhitalGEO added enhancement New feature or request bug Something isn't working labels Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant