Self-contained and unique run artifacts

[MJohnson459]

OATS can run against the new pixi docker images but the process for comparing them is ad-hoc. This PR is the first of a series that will give us historical context to changes and allow comparison between values.

Results directory

The main change is that we now store the results in a uniquely keyed results directory:

results/runs/<odm revision>/<image key>/<timestamp>/

Where the ODM revision is the exact git sha followed by the image id and finally the timestamp. This follows the logic of an image has exactly one revision, and a timestamp (run) is from exactly one image. This means repeat runs get grouped together, and different images from the same sha are also grouped.

At the top level there is a manifest that records exactly what was run - it's fairly simple right now but we could expand it (i.e. max memory usage).

results
└── runs
    ├── 32cfaacfe15a              # e.g. master
    │   └── 2da6e064d04a          # image id
    │       └── 20260724T175938Z  # when it ran
    │           ├── tests               # Full run outputs
    │           ├── oats_manifest.tsv   # Generated by bats
    │           ├── reports             # junit reports
    │           └── run_manifest.json   # Complete info on what ran
    ├── a7a8f85f032c              # e.g. some branch
    │   └── b31abdf18a3b
    │       └── 20260722T085052Z
    │           ├── tests
    │           ├── oats_manifest.tsv
    │           ├── reports
    │           └── run_manifest.json
    └── unknown                   #  pre 3.6.1 release don't have git hashes so "unknown" here
        └── 56be7b87a5ef          # but they do have image ids
            └── 20260724T205317Z
                ├── tests
                ├── oats_manifest.tsv
                ├── reports
                └── run_manifest.json

I wouldn't worry too much about the actual directory structure, the main thing is that each run directory is self-contained so you could zip up the 20260724T175938Z directory and it would be complete - what ran, when, results, etc. The next stage sends this to s3-like storage so the actual structure only matters locally, we can easily tweak it if its not right.

Output

Here is the output from a small run. It should be fairly similar to before but with a results summary.

Note: I killed the sheffield_park_3 run so we had a fail, it would normally pass

➜ ./run small

[1/2] latest / brighton  (free mem: 11.3GB)
-------------------------------------------------------------------------------------------------------------------------
1..6
ok 1 default with dsm and dtm # in 203140 ms
ok 2 fast orthophoto # in 47461 ms
ok 3 planar with 3d tiles # in 211192 ms
ok 4 sampling # in 206494 ms
ok 5 GeoTIFF alignment # in 182879 ms
ok 6 LAZ alignment # in 136994 ms
    finished in 989s

[2/2] latest / sheffield_park_3  (free mem: 15.1GB)
-------------------------------------------------------------------------------------------------------------------------
1..1
^CERROR: command `bats-format-junit --base-path tests/build/sheffield_park_3_latest.bats` failed with status 130
# bats warning: Executed 0 instead of expected 1 tests
    finished in 397s

=========================================================================================================================
 OATS SUMMARY
=========================================================================================================================
       DATASET                      TIME
 PASS  brighton                     989s
 FAIL  sheffield_park_3             397s

 Total: 2   Passed: 1   Failed: 1   Skipped: 0

 FAILURES
 sheffield_park_3: a test failed before ODM ran (see the bats output above)

 JUnit reports: results/runs/32cfaacfe15a/2da6e064d04a/20260727T113509Z/reports/

 Run manifest: results/runs/32cfaacfe15a/2da6e064d04a/20260727T113509Z/run_manifest.json

Next steps

1. Add a CI job to run this with the ability for it to be triggered from ODM
2. Save the results to an S3 compatible storage (i.e. Garage)
3. Have a quick and dirty static webpage to compare two outputs (this might eventually be replaced by the viewer)

I thought about moving to python but honestly it would be just as complicated but with more steps.

[MJohnson459]

Examples of the file outputs:
oats_manifest.tsv
run_manifest.json

junits:
brighton_latest.xml
sheffield_park_3_latest.xml