[BUG] extern "C" FFI: u16/u8 stack-passed arguments use 2-byte store (s16i) instead of 4-byte store (s32i), violating Xtensa windowed ABI specification

[vintcessun]

Summary

When calling extern "C" functions from Rust on Xtensa ESP32-S3,
u16 arguments passed on the stack (i.e., the 7th argument and beyond)
are stored using s16i (16-bit store, 2 bytes). However, the Xtensa
windowed ABI specification explicitly requires all stack-passed
arguments to occupy a minimum of 4 bytes (1 word):

"All arguments consist of an integral number of 4-byte words. Thus,
the minimum argument size is one word. Integer values smaller than a
word (that is, char and short) are stored in the least significant
portion of the argument word, with the upper bits set to zero for
unsigned values or sign-extended for signed values."
— Xtensa ISA Reference Manual

This causes all subsequent stack arguments to be read from wrong offsets
by the callee, resulting in silent data corruption.

Background

This was discovered while developing
ember-esp-nn, a no_std
Rust wrapper around Espressif's
esp-nn optimized neural network
library for ESP32-S3. The library exposes C functions with many
uint16_t parameters (e.g., image dimensions, strides, padding),
several of which are stack-passed beyond the 6th argument.

I tried this code

// Binding for esp_nn_max_pool_s8_esp32s3 as generated by bindgen
// from the esp-nn C header for target xtensa-esp32s3-none-elf
unsafe extern "C" {
    pub fn esp_nn_max_pool_s8_esp32s3(
        input: *const i8,
        input_wd: u16,
        input_ht: u16,
        output: *mut i8,
        output_wd: u16,
        output_ht: u16,
        stride_wd: u16,   // 7th arg — stack-passed
        stride_ht: u16,   // 8th arg — stack-passed
        filter_wd: u16,   // 9th arg — stack-passed
        filter_ht: u16,   // 10th arg — stack-passed
        pad_wd: u16,      // 11th arg — stack-passed
        pad_ht: u16,      // 12th arg — stack-passed
        activation_min: i32,
        activation_max: i32,
        channels: u16,    // 15th arg — stack-passed
    );
}

// Calling with 16-byte aligned buffers (required by esp32s3 asm):
#[repr(C, align(16))]
struct Buf<const N: usize>([i8; N]);

let input = Buf::<64>(core::array::from_fn(|i| i as i8));
let mut output = Buf::<64>([0i8; 64]);

unsafe {
    esp_nn_max_pool_s8_esp32s3(
        input.0.as_ptr(),
        4, 4,        // input_wd, input_ht
        output.0.as_mut_ptr(),
        4, 4,        // output_wd, output_ht
        1, 1,        // stride_wd, stride_ht
        1, 1,        // filter_wd, filter_ht
        0, 0,        // pad_wd, pad_ht
        -128, 127,   // activation_min, activation_max
        4u16,        // channels
    );
}
// With 1x1 filter and no padding, output MUST equal input exactly.
assert_eq!(&output.0[..], &input.0[..]);

I expected to see this happen

With a 1×1 filter and no padding, esp_nn_max_pool_s8_esp32s3 is an
identity operation — output must equal input. The same call from C
(via ESP-IDF) produces correct results.

Expected output [0, 1, 2, 3, 4, 5, ...] matching the input.

Instead, this happened

The output is all -128 (INT8_MIN), indicating the C function received
completely wrong parameter values and did no useful work.

Inspecting the generated assembly with --emit=asm reveals the root
cause:

Rust-generated assembly:

s16i    a8, a1, 20    # stride_wd  — only 2 bytes written!
s16i    a8, a1, 16    # stride_ht  — only 2 bytes written!
s16i    a8, a1, 12    # filter_wd
s16i    a8, a1, 8     # filter_ht
s16i    a8, a1, 4     # pad_wd
s16i    a8, a1, 0     # channels   ← lands at offset 20, callee expects offset 32
callx8  a8

GCC-generated assembly (xtensa-esp32s3-elf-gcc):

s32i    a8, a1, 20    # stride_wd  — 4 bytes, ABI-correct
s32i    a8, a1, 16    # stride_ht
s32i    a8, a1, 12    # filter_wd
s32i    a8, a1, 8     # filter_ht
s32i    a8, a1, 4     # pad_wd
s32i    a8, a1, 0     # channels   ← correctly at offset 32

Rust uses s16i (2-byte store) where the ABI requires s32i (4-byte
store), causing every subsequent stack argument to be at the wrong
offset.

Workaround: Changing the Rust binding to use u32 instead of u16
for all stack-passed parameters fixes the issue — the function then
receives correct values and produces correct output.

Meta

rustc 1.95.0-nightly (95e5bda86 2026-04-15) (1.95.0.0)
binary: rustc
commit-hash: 95e5bda868c960c607597bc03ed9e8f0ad26226d
commit-date: 2026-04-15
host: x86_64-pc-windows-msvc
release: 1.95.0-nightly
LLVM version: 21.1.3

Target: xtensa-esp32s3-none-elf

Backtrace

N/A — this is a silent data corruption bug, no panic or backtrace.
The bug is observable by comparing the output of the C function called
from Rust vs. called from C with identical inputs.