Add deterministic multi-worker evaluation and seeded failure replay by Victorbares · Pull Request #533 · Emerge-Lab/PufferDrive

Victorbares · 2026-07-22T15:55:52Z

Eval PR:

Removes previous eval code and eval managers (notably EGL renderer and WOSAC eval will need a subsequent PR)
Add seeding for deterministic replay.
Works with multiprocessing.
Works with mid-training evals, and nightly runs and other stuff

For example you can do the following:

Eval 1000 scenarios on CARLA and render the ones with an offroad

puffer eval puffer_drive carla_fast load_model_path=my_model.pt eval.render-filter=offroad

This will first run a evaluation on 1000 episodes, generate a CSV and then use the csv to know which episodes to render and thanks to seeding will replay them and render them.

…on sdc crash in sdc_only mode)

…c log, freeze on episode end

…option

…interactive replay

…ng and rendering processes

…aining evaluation logic

- Simplified file handling in `make_client` function in `render.h`. - Removed deprecated `compare_episode_metrics.py` script. - Added `benchmark_comparison.py` to aggregate benchmark results. - Created `run_all_eval.sh` for streamlined evaluation of experiments. - Introduced `run_all_latest_eval.py` to execute evaluations on the latest models. - Implemented `run_failure_scenarios.py` to rerun failed scenarios based on metrics. - Updated `launch_nightly_best.sh` script comments for clarity. - Modified `nightly_report.py` to reflect new evaluation metrics. - Enhanced unit tests in `test_drive_config.py` to validate configuration files. - Added new test in `test_eval_zlib.py` to ensure agent capacity constraints are enforced.

…and consistency

…ers and removing the PID assert

…evaluation functions

…s path

…r_eval

… components

eugenevinitsky · 2026-07-31T00:28:28Z

-  smoke-replay:
-    name: Smoke Replay HTML
-    runs-on: ubuntu-latest
-    env:
-      PUFFER_CPU: 1
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      - name: Setup Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-
-      - name: Free up disk space
-        run: |
-          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc
-          sudo apt-get clean
-          sudo rm -rf ~/.cache/pip /tmp/* /var/tmp/*
-
-      - name: Install pufferlib
-        env:
-          TMPDIR: ${{ runner.temp }}/build
-          PIP_NO_CACHE_DIR: 1
-        run: |
-          sudo apt-get update && sudo apt-get install -y build-essential cmake
-          python -m pip install -U pip pytest
-          pip install -e . --no-cache-dir
-          python setup.py build_ext --inplace --force
-
-      - name: Replay HTML test
-        run: python -m pytest -v tests/smoke_tests/test_validation_replay_html.py
-        timeout-minutes: 20
-


I understand deleting this, but why not have a replacement test for the htmls?

I deleted a lot of tests to focus only on testing the 4 most important mechanisms of the PR:

multiprocessed eval dispatches map correctly and gather the correct number of scenarios

Seeding gets you exactly the same numbers for all metrics

Running a mid-training eval doesn't affect training

Capturing replays work (complex interaction between C and Python)

My idea was to focus on the most complex mechanisms that rely on both C and Python code, and could be broken by a coming PR that touches drive.h or binding.c, and thus avoid having 2k lines of tests for things that are "less complex"

But I'm very open to advice on it, if you think we should have tests for every mechanism because it's the best way to make the codebase robust for the future, we'll add them !

eugenevinitsky · 2026-07-31T00:30:04Z

+    num_scenarios: 1000
+    num_maps: 8
+    max_agents_per_env: 50
+    scenario_length: 6000


3600? Why 6000?

I think that's the setup Valentin used for CARLA eval.
The benchmarks in benchmark.yaml are a first draft, open to changing them to more appropriate values.

@eugenevinitsky I think a separate PR should fix the values in this file, and thiese values should come from a discussion between all of us to align our eval objectives

eugenevinitsky · 2026-07-31T00:30:27Z

+  - name: womd_multi
+    seed: 42
+    simulation_mode: replay
+    num_scenarios: 1000
+    num_maps: 1000
+    max_agents_per_env: 128
+    scenario_length: 91
+    control_mode: control_vehicles
+    map_dir: pufferlib/resources/drive/binaries/womd
+


do these binaries actually exist?

They're not part of the github repo, but it would be up to people to download them and put them at the correct location, maybe we can rename womd to womd_valid

eugenevinitsky · 2026-07-31T00:31:16Z

  seed: 42
+  # Null disables evaluation during training.
+  evaluation_interval_epochs: null
+  evaluation_benchmarks: carla_fast


how do we ask for multiple evals? a list?

Also, why does this arg not live under eval?

Also, why does this arg not live under eval?

Made this choice because it's dedicated to mid-training evals, so I felt like it should be configured alongside other training args

how do we ask for multiple evals? a list?

Puttting: evaluation_benchmarks: carla_fast,womd_single,nuplan_multi for example, Victor is adding a comment to explain it

eugenevinitsky · 2026-07-31T01:01:00Z

+  output_name: null
+  # Capture and render every scenario during the standard benchmark pass.
+  render_scenarios: false
+  # Render scenarios where any configured metric is positive; use all for every failure type; null disables it.


all for every failure type suggests that there is an alternative to all or null?

Exact, the comments aren't clear:

render_filter can accept any list metrics from the csv (even non-sense like seed), and select rows where at least one of the selected metrics is positive.

For example: render_filter=offroad,at_fault_collisions would render every offroad and every at-fault collision.

render_filter=all is a shortcut for the 3 infractions: offroad+collision+traffic_light, it could be renamed all_infractions for example.

Ideally we wanted to have it working for any sets of requests like "reward>0.2 AND lane_align<-0.1" but it would have make the PR a bit messy so we tried to find a compromise here.

eugenevinitsky · 2026-07-31T01:01:08Z

+  max_rendered_failures: null
+  # Existing episode metrics CSV to replay without running the benchmark first.
+  failure_replay_csv: null
+  capture_observations: false


what does this do?

The goal is to be able to replay failures from a csv directly without executing the bench before.
Like this command :

puffer eval puffer_drive carla \ load_model_path=experiments/mimolette/models/model_puffer_drive_003815.pt \ eval.failure_replay_csv=experiments/mimolette/eval/carla/<timestamp>/episode_metrics.csv \ eval.render_filter=all \ eval.max_rendered_failures=10

eugenevinitsky · 2026-07-31T01:01:16Z

+  # Existing episode metrics CSV to replay without running the benchmark first.
+  failure_replay_csv: null
+  capture_observations: false
+  observation_replay_wave_size: 16


what is a wave size?

Observation capture uses substantial RAM, especially for CARLA scenarios with 6,000 timesteps. To limit peak memory usage, failure scenarios are processed in sequential waves.
observation_replay_wave_size sets the maximum number of scenarios whose observations are captured concurrently.

This is active when eval.capture_observations=true

May be rename "chunk_size" instead of "wave_size" ?

eugenevinitsky · 2026-07-31T01:01:26Z

+  failure_replay_csv: null
+  capture_observations: false
+  observation_replay_wave_size: 16
+  observation_replay_writer_count: 4


what is this number?

That's also something to limit RAM Usage during zlib serialization.
eval.observation_replay_writer_count controls how many replay files are serialized and compressed concurrently after observation capture.
This is also only active when eval.capture_observations=true

eugenevinitsky · 2026-07-31T01:19:23Z

-        PyList_SetItem(agent_offsets, env_count, PyLong_FromLong(total_agent_count));
-        total_agent_count += env->active_agent_count;
-        env_count++;
+        // In eval, keep whole scenes: a scene that would overflow the buffer is


a little confused by this comment, what is a partial scene?

Issue is, each worker has a budget of agents it can handle (e.g. num_agents=1000), and we want to fill it with scenarios.

If for examples the maps all have 400 agents (like huge replay maps) we should only assign the two first maps i.e. 800 agents, and not try to fit the 3rd map by throwing away the 200 exceeding agents.

So like the last map is never complete (except the very rare case where your maps add up perfectly to your num_agents), and we prefer to have like 0 padding than a non-complete ("partial") scene.

I agree the comment could be clearer

eugenevinitsky · 2026-07-31T01:29:25Z

Okay working through this bit by bit but my first concern is that a lot of eval logic has migrated into eval.py so if you wanted to later change the evaluator, you would have to hunt down the corresponding changes in drive.py. It feels like breaking some abstraction barriers? I'm raising the issue but I don't have a solution yet

riccardosavorgnan · 2026-07-31T17:32:18Z

[non-blocking] are all scripts in scripts/eval now dead code? should we remove them?

WaelDLZ and others added 30 commits July 6, 2026 15:58

Step 0: DELETE EVERYTHING EVAL RELATED - SORRY

b024795

Step 0:adapt eh code so it still passes tests

12c9e68

Step 0: also delete the docs

c61cf91

Step 1: Restore multiprocessed eval

fc22792

Step 2: Per-episode logging (for evals with early reset, for example …

ee1b1cd

…on sdc crash in sdc_only mode)

a tiny refacto of pufferl.py

48b8c77

Step 3: seeding and reusability

5df58db

replay scenarios based on condition

485e989

Step 3: Seeding + Replay

dd993de

Remove Episode Summary: There was something better to do --> adapt ve…

fa0b0b7

…c log, freeze on episode end

Force same num_agents_per_batch for seed replay on GPU

9b688ec

Pad replay inference to preserve agents per batch

8f5389f

Capture eval episodes as standard zlib replays

3232cdf

Render saved eval replays as HTML

3269859

Add scenario progress tracking to eval rollout function

f4a3ed5

Update README and config for eval replays; add observation rendering …

db2c7e0

…option

Refactor replay capture logic and remove unused schema validation in …

ec69615

…interactive replay

STEP 5

f19c116

Increase number of evaluation agents to 512 and optimize replay writi…

a6fef93

…ng and rendering processes

Less Workers for OBS rendering. Refacto

2d4c799

Add validation for training evaluation configuration and implement tr…

4f00c7a

…aining evaluation logic

Merge remote-tracking branch 'emergelab/3.0' into vb/better_eval

26b120e

Enhance evaluation configuration and simplify replay validation logic

cdb190a

Add goal configuration parameters to benchmark evaluation

7f43e93

Optim : Add CSV render only, compile policy

985a95f

Enhance benchmark evaluation and visualization logic

d0bde82

Reduce pr diff

86a4e02

Refactor evaluation configurations and update YAML files for clarity …

b684609

…and consistency

Reduce PR diff2

1da087d

vbares and others added 4 commits July 29, 2026 18:49

Add drive eval golden

fa80ab3

WIP Goldens eval

fb923fd

clean eval tests: Remove all tests

53506e5

Proper eval tests on the 4 most important features

1bc5288

WaelDLZ marked this pull request as ready for review July 29, 2026 19:48

WaelDLZ and others added 8 commits July 29, 2026 22:06

clean eval tests: make the test CI-compatible by reducing to two work…

36b8100

…ers and removing the PID assert

Refactor evaluation logic: modularize replay rendering and benchmark …

5c9cd62

…evaluation functions

Change folder position

d4e638c

Refactor imports in evaluation modules to use the new evaluation_util…

cad5f85

…s path

Merge commit '4624078f20a6ff5236f75a215b8e367de1d82a30' into vb/bette…

ccae7cb

…r_eval

Fix typo in lane index assignment in compute_metrics function

8492577

Update golden JSON files with corrected evaluation metrics and reward…

8eeecf6

… components

Refactor evaluation documentation for clarity and conciseness

90e9b59

eugenevinitsky reviewed Jul 31, 2026

View reviewed changes

Comment thread notebooks/01_observations.py

eugenevinitsky reviewed Jul 31, 2026

View reviewed changes

Comment thread pufferlib/config/evaluation/benchmark.yaml Outdated

eugenevinitsky reviewed Jul 31, 2026

View reviewed changes

vbares added 4 commits July 31, 2026 10:52

Update evaluation configuration for clarity and functionality

cb541e0

update folder for map_dir wod-motion_val

175f66d

rename all to all_infractions

e4f1c94

Add max_agents_per_env configuration and improve evaluation handling

4671bc7

Uh oh!

Conversation

Victorbares commented Jul 22, 2026 • edited by WaelDLZ Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eugenevinitsky commented Jul 31, 2026

Uh oh!

riccardosavorgnan commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Victorbares commented Jul 22, 2026 •

edited by WaelDLZ

Loading