Falied to acquire lock on file

[onnoeberhard]

Hi, I am having some trouble saving a checkpoint. My code is very simple:

PyTreeCheckpointer().save(dir_ / 'actor_params', controller._actor_params)

and works fine on my local machine. However, when running on an HPC cluster, I get the following error:

ValueError: FAILED_PRECONDITION: Error writing local file "/home/oeberhard/laurel/dat/runs/test/working_directories/1/actor_params.orbax-checkpoint-tmp-1695894959581296/_sharding": Failed to acquire lock on file: /home/oeberhard/laurel/dat/runs/test/working_directories/1/actor_params.orbax-checkpoint-tmp-1695894959581296/_sharding.__lock [OS error: Function not implemented] [source locations='tensorstore/kvstore/file/file_key_value_store.cc:676\ntensorstore/kvstore/kvstore.cc:268']

There are no access problems i.e. checkpointing with flax.training.checkpoints works perfectly. Is there maybe a way to disable the locking? Thanks!

[niketkumar]

This issue might be related to https://github.com/google/tensorstore. Can you please try their discussion forum?

[huohuoxi]

I have the same problem on hpc server. Have you fixed it?

[niketkumar]

Please consider reporting it to https://github.com/google/tensorstore/issues directly.

[laramiel]

Tensorstore release v0.1.64 includes fallback locking; let us know if that works for your case.