Skip to content

feat: chunked file transfer protocol - #369

Open
llogen wants to merge 1 commit into
mainfrom
feat/chunked-file-transfer
Open

feat: chunked file transfer protocol#369
llogen wants to merge 1 commit into
mainfrom
feat/chunked-file-transfer

Conversation

@llogen

@llogen llogen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Chunked file transfer

Files move between client and agent in 1 MiB acknowledged chunks instead of one protobuf message carrying the whole payload. The old design held the entire file in memory on both ends, and while that message was in flight nothing else on the Run stream moved — module output and stdin included. Chunks interleave, so memory stays bounded at one chunk per side and logs keep flowing during a transfer.

Protocol. Four new messages in RunRequest/RunResponse: FileMetadata, FileTransferRequest (carrying a Direction), FileChunk, FileTransferResponse. Fields 3 and 4 are reserved rather than reused, so an old peer fails to parse instead of silently misreading a chunk as a whole file. Agent and client must be upgraded together — a deliberate wire break.

Module API. RequestFile and SendFile now take a context.Context, and SendFile takes the file size. SendFile returns before the bytes are sent, so the session takes ownership of the reader and closes it on completion — on a nil return only. file, flash, flash-emulate and dummy are updated.

Limits. A transfer is bounded by that context rather than by a constant. The sensible ceiling is hardware-dependent — a Pi driving slow SPI flash and a rack server differ — so it belongs with the module that knows it, via context.WithTimeout, matching how modules already bound subprocesses. An agent-wide default can layer onto the same mechanism later.

Flow control. One chunk outstanding at a time, acknowledged only after the module has consumed it. Real end-to-end backpressure: the client cannot outrun the hardware.

Safety. The client writes only to paths named in the command arguments; anything else is refused and the agent drops the transfer.

CI. New protobuf job: buf lint, buf format, and a generated-code drift check.

Testing. 22 transfer tests under -race, most written against a reproduced failure and re-checked to fail with the fix reverted. All use in-process fakes, so nothing has crossed real hardware yet — the FWCI suite on the tester closes that gap.

@llogen
llogen marked this pull request as draft July 27, 2026 06:33
@llogen
llogen force-pushed the feat/chunked-file-transfer branch from ea122c3 to 54cdde7 Compare July 27, 2026 08:00
@llogen
llogen marked this pull request as ready for review July 27, 2026 08:37
Files move between client and agent in bounded, individually
acknowledged chunks, so large transfers stream reliably within the
run.

Signed-off-by: llogen <christoph.lange@blindspot.software>
@llogen
llogen force-pushed the feat/chunked-file-transfer branch from 54cdde7 to 8babdbe Compare July 30, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant