Pickle: Mesh points at wrong series after deserializing

[pordyna]

Describe the bug
When serializing and deserializing a series object in python and doing the same for a MeshRecord (or MeshRecordComponent ?) both will successfully deserialize. But, although calling load_chunk on the deserialized mesh will work, calling flush on the deserialized series won't actually load the data.
There is no proper error message, but the buffer will contain random data. When serializing with some parallelization libraries like joblib workers can fail with messages like:

Fatal Python error: gilstate_tss_set: failed to set current tstate (TSS)
Python runtime state: finalizing (tstate=0x00000000017f9e18)

To Reproduce
uv run deserialize_bug.py
deserialize_bug.py

Expected behavior
Since the MeshRecordComponent and MeshRecord both have add_pickle both methods in the example should work

Software Environment
See attached file

[pordyna]

@franzpoeschel this is not really urgent, just pinging in case you haven't seen it yet.

[franzpoeschel]

@franzpoeschel this is not really urgent, just pinging in case you haven't seen it yet.

Ah thanks, I had forgotten about it
Will check

[franzpoeschel]

I see that something is going wrong, but I have not yet understood why
Will continue

[franzpoeschel]

Okaayyy, I understand why this happens, but this might get ugly
Our serialization/deserialization strategy is to save only the filename + the location of the object within that file (e.g. /400/meshes/rho), and then upon deserialization init a new Series and point it to the right object again.
Consequence: After deserializing, series, it and mesh are three entirely different openPMD Series objects. Meaning that series.flush() won't do things that you enqueued into mesh.

You can work around this by adding a line mesh.series_flush(), but the pitfall remains.. No clue how to easily solve that at the moment

[pordyna]

If there is no easy solution maybe you should think about removing pickle capabilities from everything except for series. And just force the user to rebuild everything from series.

[franzpoeschel]

If there is no easy solution maybe you should think about removing pickle capabilities from everything except for series. And just force the user to rebuild everything from series.

That would be the last resort, and we will have to consider it if there is no other solution
As this would break a lot of production use cases, I would like to avoid it
My current idea would be to deserialize additionally an ID for the Series, such that readers will know which Series objects to associate with each other. Will have to try if this can work