Skip to content

Add pip-installable wheels for Linux and macOS - #151

Open
HugoFara wants to merge 2 commits into
forefireAPI:masterfrom
HugoFara:feat/python-wheels
Open

Add pip-installable wheels for Linux and macOS#151
HugoFara wants to merge 2 commits into
forefireAPI:masterfrom
HugoFara:feat/python-wheels

Conversation

@HugoFara

@HugoFara HugoFara commented Jul 28, 2026

Copy link
Copy Markdown

Closes #149.

Makes pip install forefire work. Wheels for Linux and macOS ship the pyforefire module, the forefire command-line interpreter, and a vendored NetCDF. No NetCDF install, no NETCDF_HOME, no prior CMake build.

pip install forefire
forefire -v
python -c "import pyforefire; print(pyforefire.ForeFire())"

The PyPI distribution is named forefire, as the issue asked. The import name stays pyforefire, so existing scripts keep working.

The three obstacles from @antonio-leblanc's reply

1. setup.py assumed a pre-built libforefireL.

Replaced with scikit-build-core, which drives the existing CMakeLists.txt. The wheel build now compiles the C++ core itself. Packaging moved to the repository root, because an sdist rooted at bindings/python/ cannot reach ../../src. bindings/python/setup.py and bindings/python/pyproject.toml are deleted.

2. NetCDF vendoring.

On Linux, tools/devops/install-netcdf-manylinux.sh installs NetCDF C and HDF5 from EPEL inside the manylinux_2_28 container. It then builds netcdf-cxx4 from source with a pinned checksum, since no RHEL-family package provides the C++4 API. auditwheel copies the result into the wheel. macOS uses Homebrew plus delocate.

3. The MPI axis (#102).

Wheel builds set FOREFIRE_ENABLE_MPI=OFF explicitly. A published wheel can no longer depend on whatever happened to be installed on the build machine. Users who need coupling build from source with FOREFIRE_ENABLE_MPI=ON. This is now documented rather than implicit.

CMake changes

The build is option-driven. Defaults preserve the previous behaviour for ordinary source builds:

Option Source build Wheel
FOREFIRE_ENABLE_MPI ON OFF
FOREFIRE_NATIVE_ARCH ON OFF
FOREFIRE_BUILD_PYTHON OFF ON
FOREFIRE_STATIC_CORE OFF ON
FOREFIRE_BUILD_TOOLS ON OFF
FOREFIRE_CHECK_LFS ON OFF

Also changed:

  • NetCDF is located with find_path and find_library, replacing bare link_libraries(netcdf netcdf_c++4). This is what makes the macOS wheel possible. Homebrew names the library libnetcdf-cxx4, not libnetcdf_c++4, and /opt/homebrew is not on the default search path on Apple Silicon. A missing NetCDF now fails at configure time with install instructions, instead of failing at link time.
  • The version is parsed from src/include/Version.h. project(VERSION), CPack and the wheel version now track the file tools/increment-version.sh edits. project() previously said 1.0 while the code said v2.3.0.
  • Four duplicated if(NETCDF_STATIC_LIBS)/if(MPI_FOUND) link blocks collapse into one forefire_netcdf interface target.
  • Writes into the source tree (bin/, lib/), the LFS check and CPack are skipped under SKBUILD only. Native builds are unaffected.

Two portability bugs fixed along the way

  • -march=native was unconditional. Any binary built on one machine and run on an older CPU dies with SIGILL. That includes the published Docker image, so the Dockerfile now passes -DFOREFIRE_NATIVE_ARCH=OFF. Native builds keep the flag on by default.
  • The Dockerfile built its wheel from ./bindings/python, which is no longer a project. It now builds from the root and takes the forefire binary from that single wheel, rather than copying a second one.

CI

New .github/workflows/wheels.yml, running on pull requests and pushes to master:

  • wheels via cibuildwheel on ubuntu-latest (x86_64), ubuntu-24.04-arm (aarch64, native runner, no QEMU), macos-14 (arm64) and macos-13 (x86_64), for CPython 3.9 to 3.13.
  • sdist, compiled and smoke-tested against Ubuntu's NetCDF packages. This is the path a Windows or musl user takes.
  • publish to PyPI on GitHub release, via trusted publishing.

Every wheel is smoke-tested before upload. tests/python/test_wheel.py imports the module, runs an isotropic 1000 s simulation, and asserts the front grew. forefire -v then checks the shipped interpreter.

Verification

Built in a real manylinux_2_28_x86_64 container with podman, then installed and tested on a Fedora 44 host with no NetCDF installed at all. That second step is the meaningful one: the test machine is not the build machine, and
glibc differs (2.43 against 2.28).

Check Result
Wheel size 8.8 MB, full NetCDF/HDF5 stack in forefire.libs/
ldd on the extension Only librt and libresolv from the system
Smoke test 122 fire nodes
forefire -v from any directory v2.3.0
Repository example tests/runff/run.ff Exit 0, valid GeoJSON MultiPolygon

That last row runs a full Rothermel simulation over the 4 MB tests/runff/data.nc, so the vendored HDF5 and NetCDF are genuinely exercised rather than merely loaded.

The plain cmake -S . -B build path is unchanged: shared lib/libforefireL.so, bin/forefire and bin/ANN_test, -march=native still on, LFS check still running. The native binary and the wheel produce byte-identical output on run.ff. The sdist is 402 KB and contains CMakeLists.txt, which .gitignore had been silently excluding.

Two bugs were caught by that local run rather than by review, which is why the smoke test now asserts on them:

  1. Linking the core as a static archive dropped every self-registering propagation model. addLayer("propagation", "Iso", ...) printed model Iso is not recognized, and the next call segfaulted. Fixed with an OBJECT library. tests/python/test_wheel.py now fails loudly instead of crashing.
  2. The static core needed Threads::Threads linked explicitly. The shared library had been providing it implicitly.

macOS wheels are not verified locally, I may do it later on. Both the Linux and macOS jobs run forefire -v against the installed wheel. If delocate does not relocate the interpreter binary the way auditwheel does, CI fails rather than publishing something broken. Worth watching on the first run.

Before this can publish

The forefire project on PyPI needs to exist, I'll leave it to you. Until then the workflow builds and tests wheels, and the publish job never runs. It is gated on github.event_name == 'release'.

Packaging choices worth a look

The root pyproject.toml replaces a much smaller bindings/python/pyproject.toml. Some values in it are forced by the toolchain, others are policy. The policy ones:

Choice Was Now Reason
Minimum Python >=3.8 >=3.9 3.8 is end of life, and cibuildwheel 4 no longer builds it. Keeping 3.8 means pinning an older cibuildwheel.
Version source hardcoded 2025.2 parsed from src/include/Version.h, currently 2.3.0 Stops the two drifting apart. It also moves the published series from CalVer to SemVer.
pybind11 runtime dependency build dependency Only needed to compile. Nothing imports it at runtime.
License metadata { file = "LICENSE" } "GPL-3.0-or-later" PEP 639 SPDX. Matches the LICENSE text, which is GPLv3 with the "any later version" clause. Change it if you mean GPL-3.0-only.
OS classifiers OS Independent Linux and MacOS X The wheels are POSIX only.
Homepage the GitHub repo forefire.univ-corse.fr Documentation and Issues links added alongside it.
sdist contents not applicable tests/, docs/, paper/ excluded, smoke test kept Keeps the sdist at 402 KB rather than carrying roughly 60 MB of LFS fixtures. An sdist therefore cannot run the full suite.
Wheel targets not applicable PyPy, musllinux, i686 and win32 skipped No usable NetCDF C++4 on any of them.
macOS floor not applicable MACOSX_DEPLOYMENT_TARGET = 13.0 Cannot predate the Homebrew bottles being linked. Drops macOS 12 and older.

Two suggestions rather than changes:

  • No Development Status classifier is set. 5 - Production/Stable would fit v2.3.0 and the JOSS paper, if you want the maturity advertised.
  • cibuildwheel will build CPython 3.14, but the classifiers stop at 3.13. Either add the 3.14 classifier or skip 3.14 in the skip list.

Notes for reviewers

  • Windows is out of scope. There is no reasonable prebuilt netcdf_c++4 for it, and the sources use <sys/time.h> and popen, so it needs a porting effort rather than a packaging one. Windows users fall back to the sdist.
  • bindings/python/README.md is rewritten around pip install. It is what PyPI renders as the project page.

HugoFara added 2 commits July 28, 2026 17:38
`pip install forefire` now works, providing both the pyforefire module and
the forefire command-line interpreter with NetCDF vendored in. Closes forefireAPI#149.

Packaging moves to the repository root and is driven by scikit-build-core,
which runs the existing CMake build instead of assuming a pre-built
libforefireL: an sdist rooted at bindings/python cannot reach ../../src.
bindings/python/{setup.py,pyproject.toml} are removed.

CMakeLists.txt becomes option-driven, keeping today's behaviour as the
default for source builds while wheels get portable settings:

  FOREFIRE_ENABLE_MPI, FOREFIRE_NATIVE_ARCH, FOREFIRE_BUILD_PYTHON,
  FOREFIRE_STATIC_CORE, FOREFIRE_BUILD_TOOLS, FOREFIRE_CHECK_LFS

Wheels disable MPI so a published artifact never depends on what happened
to be installed on the builder (see forefireAPI#102), and disable -march=native, which
was previously unconditional and made any redistributed binary — including
the published Docker image — crash with SIGILL on older CPUs.

NetCDF is now located with find_path/find_library behind a single
forefire_netcdf interface target rather than bare link_libraries. This is
what makes a macOS wheel possible: Homebrew names the library
libnetcdf-cxx4, not libnetcdf_c++4, and /opt/homebrew is off the default
search path. Missing NetCDF now fails at configure time with install hints.

The core is an OBJECT library, not STATIC, when linked directly into the
artifacts: propagation and flux models register themselves from static
initialisers that nothing references by name, and a static archive lets the
linker discard them, leaving the model table empty at runtime.

Other changes:
- version is parsed from src/include/Version.h, so project(VERSION), CPack
  and the wheel all track the file tools/increment-version.sh edits
- .gitignore no longer ignores CMakeLists.txt, which had been dropping the
  build file from the sdist
- Dockerfile builds its wheel from the root and installs the interpreter
  from that one wheel instead of copying a second binary
- tests/python/test_wheel.py smoke-tests an installed wheel; every wheel is
  tested before upload
Project maturity is a maintainer's call, not something packaging should
assert. Left unset rather than guessed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python Wheels

1 participant