Skip to content

Implement Arrow-to-Frame Conversion (Deserialization) for Arrow Backend - #987

Merged
jmcarcell merged 14 commits into
AIDASoft:masterfrom
arnavdham:feature/TabletoFrame
Aug 4, 2026
Merged

Implement Arrow-to-Frame Conversion (Deserialization) for Arrow Backend#987
jmcarcell merged 14 commits into
AIDASoft:masterfrom
arnavdham:feature/TabletoFrame

Conversation

@arnavdham

@arnavdham arnavdham commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

This PR implements the deserialization capability for the Apache Arrow backend, completing the round-trip workflow. It adds support for reconstructing PODIO Frame objects from a single-row slice of an Apache Arrow Table.

Key Features & Changes

  • convertTableToFrame API: Introduced a public utility to read and reconstruct a Frame from a specific row of an Arrow Table.
  • Auto-generated Readers: Updated cpp_generator.py and generator templates (ArrowMapper.cc.jinja2) to generate deserialization/reader functions for all concrete data model types.
  • Complex Type Support: Handles reconstruction of collections, subset collections, vector members, relations/references, and Link Collections.
  • Generic Parameters: Reconstructs GenericParameters from the frame_parameters struct column in the Arrow Table.

Testing

  • Added comprehensive unit tests in test_arrow_converter.cpp covering:
  • Full round-trip (Frame $\rightarrow$ Table $\rightarrow$ Frame) checks.
  • Multi-row tables and non-zero row index offset mapping.
  • Verification of reconstructed relations, subsets, and parameter maps.

Fixes: #986

BEGINRELEASENOTES

  • Add support for Arrow-to-Frame conversion (deserialization) for the Apache Arrow backend to complete the round-trip workflow.
  • Introduce the convertTableToFrame API to reconstruct podio::Frame objects from a specific row of an Arrow Table.
  • Add support for reconstructing complex types (collections, vector members, relations, subsets) and GenericParameters.

ENDRELEASENOTES

@arnavdham

Copy link
Copy Markdown
Contributor Author

@jmcarcell Please take a look.

Comment thread tests/unittests/test_arrow_converter.cpp Outdated
Comment thread tests/unittests/test_arrow_converter.cpp Outdated
Comment thread include/podio/utilities/ArrowConverterRegistry.h Outdated
Comment thread include/podio/utilities/ArrowFrameConverter.h Outdated
Comment thread src/ArrowConverterRegistry.cc Outdated
Comment thread src/ArrowFrameData.cc
Comment thread tests/unittests/test_arrow_converter.cpp Outdated
Comment thread src/ArrowFrameData.cc
Comment thread src/ArrowFrameData.cc Outdated
Comment thread python/templates/ArrowMapper.cc.jinja2 Outdated
Comment thread tests/unittests/test_arrow_converter.cpp Outdated
@jmcarcell
jmcarcell force-pushed the feature/TabletoFrame branch from 934c79d to 90ccc9d Compare August 3, 2026 18:49
@jmcarcell
jmcarcell enabled auto-merge (squash) August 3, 2026 19:31
@jmcarcell

Copy link
Copy Markdown
Member

Can you format the PR to have the text between BEGINRELEASENOTES and ENDRELEASENOTES like in other PRs? We have a script that reads those and puts them in a doc file and commits it when tagging. I could do it myself, but it would be good if you do that for this and other PRs.

@jmcarcell
jmcarcell disabled auto-merge August 3, 2026 19:40
@arnavdham

Copy link
Copy Markdown
Contributor Author

Can you format the PR to have the text between BEGINRELEASENOTES and ENDRELEASENOTES like in other PRs? We have a script that reads those and puts them in a doc file and commits it when tagging. I could do it myself, but it would be good if you do that for this and other PRs.

Done for this PR. Will follow this for any further PRs as well. Do you want me to do this for already merged PRs as well?

@jmcarcell

Copy link
Copy Markdown
Member

Yes, please, I may have forgotten to say so in other PRs.

@jmcarcell
jmcarcell merged commit 32b44e8 into AIDASoft:master Aug 4, 2026
32 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Arrow Table-to-Frame Conversion (Deserialization) in PODIO

2 participants