When running the whole test suite on the same machine, same software env (container), one GPU requested, we see these run times
- 16 cores: 31 min
- 1 core: 16 min
In the 16 core case, some test(s) use all cores, some don't (observe e.g. via htop). I haven't investigated which ones do, but for those, my first guess is that maybe multiple data loader threads are stepping on each others toes and slow things down.
When running the whole test suite on the same machine, same software env (container), one GPU requested, we see these run times
In the 16 core case, some test(s) use all cores, some don't (observe e.g. via
htop). I haven't investigated which ones do, but for those, my first guess is that maybe multiple data loader threads are stepping on each others toes and slow things down.