feat: chunked file transfer protocol - #369
Open
llogen wants to merge 1 commit into
Open
Conversation
llogen
marked this pull request as draft
July 27, 2026 06:33
llogen
force-pushed
the
feat/chunked-file-transfer
branch
from
July 27, 2026 08:00
ea122c3 to
54cdde7
Compare
llogen
marked this pull request as ready for review
July 27, 2026 08:37
Files move between client and agent in bounded, individually acknowledged chunks, so large transfers stream reliably within the run. Signed-off-by: llogen <christoph.lange@blindspot.software>
llogen
force-pushed
the
feat/chunked-file-transfer
branch
from
July 30, 2026 08:33
54cdde7 to
8babdbe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chunked file transfer
Files move between client and agent in 1 MiB acknowledged chunks instead of one protobuf message carrying the whole payload. The old design held the entire file in memory on both ends, and while that message was in flight nothing else on the
Runstream moved — module output and stdin included. Chunks interleave, so memory stays bounded at one chunk per side and logs keep flowing during a transfer.Protocol. Four new messages in
RunRequest/RunResponse:FileMetadata,FileTransferRequest(carrying aDirection),FileChunk,FileTransferResponse. Fields 3 and 4 are reserved rather than reused, so an old peer fails to parse instead of silently misreading a chunk as a whole file. Agent and client must be upgraded together — a deliberate wire break.Module API.
RequestFileandSendFilenow take acontext.Context, andSendFiletakes the file size.SendFilereturns before the bytes are sent, so the session takes ownership of the reader and closes it on completion — on a nil return only.file,flash,flash-emulateanddummyare updated.Limits. A transfer is bounded by that context rather than by a constant. The sensible ceiling is hardware-dependent — a Pi driving slow SPI flash and a rack server differ — so it belongs with the module that knows it, via
context.WithTimeout, matching how modules already bound subprocesses. An agent-wide default can layer onto the same mechanism later.Flow control. One chunk outstanding at a time, acknowledged only after the module has consumed it. Real end-to-end backpressure: the client cannot outrun the hardware.
Safety. The client writes only to paths named in the command arguments; anything else is refused and the agent drops the transfer.
CI. New
protobufjob:buf lint,buf format, and a generated-code drift check.Testing. 22 transfer tests under
-race, most written against a reproduced failure and re-checked to fail with the fix reverted. All use in-process fakes, so nothing has crossed real hardware yet — the FWCI suite on the tester closes that gap.