Create a Python icechunk Storage from obstore

[kylebarron]

This implements conversion from an Obstore S3Store. I figure I should get feedback on this overall before expanding this to support GCS and Azure.

This works by using obstore's pickle mechanism to extract the config from a non-statically linked instance of object-store. The configs don't line up cleanly between the two libraries.

And might I add that object-store's config is vastly broader than the config options you support, so any specialized config settings that a user might have defined in obstore will be silently lost on conversion to an icechunk store. We could at least print a warning? Or should we error?

• This is pointing to a git tag of pyo3-object_store because it requires some changes to the public API.
  • The target pyo3-object_store branch is also reverted to pyo3 0.27. Icechunk will need to be updated to pyo3 0.28 before this can be merged.
• I don't know how to interact with your implementation of refreshable credentials.

I don't know how to use icechunk so I don't have a way to test this PR myself, though I'm sure someone from devseed could help out.

I have not yet used AI in the creation of this PR.

Closes #1632

[kylebarron]

FWIW uv run maturin dev --uv succeeds locally.

Maybe @vladidobro you'd be interested in testing this because you created #1632

[li-em]

  * The target pyo3-object_store branch is also reverted to pyo3 0.27. Icechunk will need to be updated to pyo3 0.28 before this can be merged.

Not released yet, but now main is pyo3 0.29

[kylebarron]

pyo3-object-store is already updated for pyo3 0.29; but I'll wait for some feedback on this design before updating the PR

[TomNicholas]

I don't have much feedback about the low-level design, but a function like this (or equivalent mapping in obstore to icechunk) would allow saving a few LoC in every VirtualiZarr Icechunk example.