Feature Request: Native iteration over zero-copy memory buffers (PEP 3118 / PyArrow) for structured data

[lijose]

Hi team,

First, thank you for the incredible work on mypyc and librt.vecs. I have been benchmarking it for heavy data pipelines, and the performance for flat primitives (like vec[i64]) is spectacular—matching C++ SIMD speeds.

I am reaching out to ask: Is there any existing or planned support for mypyc to natively iterate over zero-copy memory buffers via the Python Buffer Protocol (PEP 3118) or the Arrow C-Data Interface?

The Context & Limitation

While vec is brilliant for flat arrays, real-world data pipelines rely heavily on structured data (tuples, dicts) and strings. Because vec requires a packed binary representation, it rightfully rejects types like Tuple and Dict, forcing a fallback to standard Python List objects.

When attempting to bridge this gap using PyArrow's columnar memory (StructArray), mypyc currently treats Arrow arrays as Any (partially due to PyArrow currently lacking complete py.typed stubs). As a result, compiling a loop over an Arrow array forces mypyc to fall back to the generic CPython API (PyObject_GetIter), causing massive boxing/unboxing overhead that is actually slower than standard Python.

Minimal Reproducible Example

Here is a minimal benchmark demonstrating the gap between mypyc's optimized flat loops, the fallback for structured data, and PyArrow's native C++ compute.

1. core_ops.py (Compiled with mypyc --ignore-missing-imports core_ops.py)

import pyarrow as pa
import pyarrow.compute as pc
from typing import List, Tuple, Any
from mypy_extensions import i64
from librt.vecs import vec

# 1. FLAT PRIMITIVES (Blazing fast via vec)
def loop_vec(data: vec[i64]) -> i64:
    total: i64 = 0
    for x in data:
        total += x
    return total

# 2. PYARROW LOOP FALLBACK (Slow due to type erasure and object boxing)
def loop_arrow_fallback(data: Any) -> Any:
    total = 0
    for x in data:
        total += x.as_py()
    return total

# 3. STRUCTURED DATA FALLBACK (vec rejects Tuple, forced to use List)
def process_list_tuple(data: List[Tuple[int, float]]) -> float:
    total: float = 0.0
    for row in data:
        total += row[1]
    return total

# 4. PYARROW NATIVE COMPUTE (The target performance for structured data)
def process_arrow_tuple(data: pa.StructArray) -> float:
    return pc.sum(data.field("f1")).as_py()

2. benchmark.py (Driver)

import time
import pyarrow as pa
from librt.vecs import vec
from mypy_extensions import i64
from core_ops import (
    loop_vec, loop_arrow_fallback, 
    process_list_tuple, process_arrow_tuple
)

size = 1_000_000

# Data prep
flat_data = list(range(size))
v_flat = vec[i64](flat_data)
a_flat = pa.array(flat_data)

tuple_data = [(i, float(i)) for i in range(size)]
a_tuples = pa.StructArray.from_arrays(
    [pa.array([t[0] for t in tuple_data]), pa.array([t[1] for t in tuple_data])],
    names=["f0", "f1"]
)

# 1. Iteration Benchmark
start = time.perf_counter()
loop_vec(v_flat)
print(f"librt vec[i64] loop:   {time.perf_counter() - start:.4f}s")

start = time.perf_counter()
loop_arrow_fallback(a_flat)
print(f"PyArrow manual loop:   {time.perf_counter() - start:.4f}s")

# 2. Structured Data Benchmark
start = time.perf_counter()
process_list_tuple(tuple_data)
print(f"Mypyc List[Tuple]:     {time.perf_counter() - start:.4f}s")

start = time.perf_counter()
process_arrow_tuple(a_tuples)
print(f"PyArrow Struct compute:{time.perf_counter() - start:.4f}s")

My Output (Python 3.14):

librt vec[i64] loop:   0.0010s
PyArrow manual loop:   0.0984s
Mypyc List[Tuple]:     0.0060s
PyArrow Struct compute:0.0003s

The Proposal

I completely understand that mypyc cannot hardcode logic for every third-party library. However, if mypyc could natively support iterating over objects that implement the Python Buffer Protocol (PEP 3118), users could write fast, AOT-compiled C-loops that interact directly with PyArrow, NumPy, or Pandas contiguous memory buffers.

This would allow users to leverage mypyc for custom, branch-heavy ETL logic on structured data without incurring the massive CPython API boxing penalties or requiring the mypyc team to reinvent complex columnar structures inside librt.vecs.

Would love to hear your thoughts on if this aligns with the compiler's roadmap!