AgentSession: reply interrupted before playout start wedges _currentSpeech forever (1.5.3)

[Raysharr]

Describe the bug

In @livekit/agents 1.5.3, when a pipeline reply is interrupted before its playout has started, AgentActivity.mainTask can hang forever on the interrupted speech handle's generation future. _currentSpeech is never cleared, so the agent goes permanently silent — no further replies are ever forwarded to the user until an external mechanism (e.g. an idle/duration guard) tears down the session.

This looks like a deeper regression in the same area as #836 / #1124 / #1085. The waitIfNotInterrupted([generation]) guard proposed in #1124 (and now present in current mainTask) does correctly return early on interrupt — but the very next branch performs a second, unguarded wait on the same generation future, and that second wait is the one that actually deadlocks.

Root cause (traced against 1.5.3 dist/voice/agent_activity.js)

mainTask (~line 1424):

this._currentSpeech = speechHandle;
speechHandle._authorizeGeneration();
const generation = speechHandle._waitForGeneration();
await speechHandle.waitIfNotInterrupted([generation]);   // returns immediately on interrupt — OK
if (speechHandle.interrupted && speechHandle._tasks.length > 0) {
  await ThrowsPromise.race([generation, abortFuture.await]);   // ← hangs here
}
this._currentSpeech = void 0;

generation only resolves via _markGenerationDone() / _markDone() on the speech handle, which only happens once the reply task's own task-done callback fires. abortFuture only resolves when the whole activity is torn down (not per-speech).

The reply task itself gets stuck one level deeper, inside forwardSegment (~line 2156 in the same file):

const interruptedPlaybackEv = await raceWithAbort(
  audioOutput.waitForPlayout(),
  replyAbortController.signal
);

audioOutput.waitForPlayout() waits for a playback-finished event that will never fire, because playout never started. The only other way out of this await is the replyAbortController.signal aborting — but that controller's own .abort() call happens later in the same function (the "Aborting all pipeline reply tasks due to interruption" path), in code that is only reached after the segment loop this await sits in returns. The segment loop can't return, because it's blocked on this same await. It's a self-referential wait: A blocks on B, and B is only triggered by code that runs after A returns.

Net effect: the reply task never calls _markGenerationDone(), so its done-callback never calls _markDone(), so generation in mainTask never resolves, so _currentSpeech stays stuck on the wedged handle forever.

To reproduce

1. Build an agent flow where a reply is generated from an activity/agent-handoff onEnter (i.e. generateReply() fired programmatically, not directly as a response to the turn that just finished) — e.g. an agent handoff in a multi-agent voice workflow.
2. Use VAD-based turn detection.
3. Have the user's speech continue into the handoff window — a mid-utterance pause exceeding minSilence is enough to split one utterance into two turns, with the second turn landing right as the new reply is created but before its TTS audio has started playing.
4. The new-turn interrupt hits the reply before playout starts. Observe: the agent never speaks again for the rest of the session.

In a synthetic voice test harness driving this exact flow (scripted utterance with a natural mid-sentence pause landing in the handoff window), the trigger reproduced deterministically on repeated runs.

Observed log signature

• "speech interrupted, new user turn detected" fires repeatedly, always pointing at the same speech id, across many subsequent user turns.
• Each subsequent turn creates a new speech handle and LLM inference completes for it, but it never plays — mainTask never advances past the wedged handle to authorize the next one.
• No "Aborting all pipeline reply tasks due to interruption" line ever appears for the wedged speech id.
• The session only ends via an external mechanism (idle/duration guard, or user disconnect) — it never recovers on its own.

Environment

• @livekit/agents 1.5.3 (Node.js / TypeScript)
• Voice pipeline: STT + LLM + TTS with Silero VAD turn detection (not STT-endpointing based)
• Appears provider-independent — the bug is in the pipeline state machine, not tied to a specific STT/LLM/TTS backend

Workaround

We're running a downstream patch on mainTask's interrupted-branch wait: bound it with a timeout in addition to generation/abortFuture, and on timeout, force-abort the reply task's own tasks (via speechHandle._tasks + the SDK's own task-cancellation helper, which drives the same abort controller raceWithAbort in forwardSegment is waiting on), plus a _markDone() safety net if the tasks don't close in time. This reliably restores the agent's voice within seconds of the interrupt and has shown zero recurrence of the hang signature across repeated runs of the reproduction above.

Related issues

• mainTask hangs forever when interrupting a speech task which is on the queue #836 — mainTask hangs forever when interrupting a speech task on the queue (same general area, earlier stage of the pipeline)
• Speech handle stuck in limbo after tool call interruption — agent enters unrecoverable silence loop #1124 (closed) — proposed the waitIfNotInterrupted([_waitForGeneration()]) guard that is now present in mainTask, but that only covers the first wait; this issue is about the second, unguarded wait immediately after it
• _waitForGeneration hangs when speech task is canceled before play out #842 — open PR for mainTask hangs forever when interrupting a speech task which is on the queue #836
• performTextForwarding never created when speech handle is interrupted before scheduling #1085 (closed) — different manifestation of interruption-before-scheduling in the same reply pipeline

Happy to share more log excerpts or a minimal repro harness if useful.

[Kyle-shechtman]

@toubatbrian Can we get eyes on this? This is a causing a lot of issues for us in prod.

[toubatbrian]

@Kyle-shechtman I opened a PR and verified the fix locally: #2127.

Could you try this branch and see whether it resolves the issue on your end? The PR may take some time to review and release, and I don’t want that to block you in the meantime.

[Kyle-shechtman]

@toubatbrian You are the man! We will take a look

[toubatbrian]

We were not able to reproduce this live on current main with a plain LiveKit room audio output — six runs of a probe agent built to your description, no wedge in any of them. Your root-cause trace is nevertheless accurate, the deadlock is real at the mechanism level, and #2127 is going forward on that basis, so nothing here blocks the fix. What we are missing is one detail about your setup, because the evidence now points at the audio output as the deciding factor. Details below so you can tell us where our reconstruction diverges from yours.

Your analysis holds up

The self-referential wait you identified — the interrupted speech handle waiting on work that only it can unblock, so _currentSpeech is never cleared — is exactly what #2127 targets, by porting Python's INTERRUPTION_TIMEOUT watchdog into SpeechHandle._cancel(). That much is pinned down as a red-to-green on a clean checkout: agents/src/voice/interrupt_before_playout_deadlock.test.ts fails on clean origin/main, where the follow-on reply never settles within 15 s, and passes on the PR branch with the session recovering in about 5.2 s.

What did not reproduce is reaching it through the pipeline you describe

We built a probe agent to your description: a front-desk agent whose tool returns llm.handoff() to a billing agent whose onEnter calls generateReply(), turnDetection: 'vad' with Silero minSilenceDuration: 250 and fixed endpointing minDelay: 200, and real models throughout (deepgram/nova-3, openai/gpt-4.1-mini, inworld/inworld-tts-2). We drove it with a scripted five-turn client whose second turn commits inside the handoff reply's pre-playout window, matching the mid-utterance pause that splits one utterance across the handoff in your report. We ran it in two configurations: the natural window, and one where the TTS first frame was held for 3000 ms to widen the pre-playout window well beyond what timing luck would give us.

Six valid runs, no wedge in any of them. Every run went on to produce three or four subsequent playouts.

Run	Interrupt to abort line	Wedged
main-D1 (log below)	35 ms	no
Five further runs	0.8 s, 1.2 s, 1.4 s, 1.8 s, 1.9 s	no

The cleanest of them, main-D1, has the interrupt landing 1.39 s after the reply's TTS segment was created and before any frame reached the output:

[12:33:30.092] === PROBE billing onEnter: calling generateReply
[12:33:30.092] Creating speech handle speech_id: "speech_2ececaaf-0f3"
[12:33:30.690] Task.runTask: task performTTSInference started
[12:33:32.079] speech interrupted, new user turn detected speech id: "speech_2ececaaf-0f3"
[12:33:32.114] Aborting all pipeline reply tasks due to interruption speech_id: "speech_2ececaaf-0f3"
[12:33:32.115] firstFrameFut cancelled before first frame

firstFrameFut cancelled before first frame confirms we were genuinely in the pre-playout window — playout never started for that reply. But none of your three signature conditions held: the repeated speech interrupted, new user turn detected lines always named distinct speech ids rather than the same one, the Aborting all pipeline reply tasks due to interruption line always appeared (35 ms after the interrupt here), and later replies played normally.

Why a room output appears to settle on its own

AudioOutput.waitForPlayout() waits only until playbackFinishedCount reaches the segment count captured at the time of the call, so with zero frames captured it returns immediately — which is the main-D1 case above. If frames had been captured, cancelAndWait(forwardTasks) runs forwardAudio's finally, which calls flush() and then clearBuffer(); ParticipantAudioOutput.flush() starts waitForPlayoutTask, and clearBuffer() resolves the interruptedFuture that task races, which produces onPlaybackFinished. Both sub-windows therefore settle with a room output, which is consistent with what we observed.

The hypothesis, and the question

#2127's regression test does not use a room output. It uses a stand-in sink whose flush() withholds the playback-finished event for a segment the remote never started and whose clearBuffer() is a no-op — standing in, per the test's own docstring, for the class of sinks whose playback-finished event is not the pipeline's to produce: remote avatar outputs (DataStreamAudioOutput and every avatar plugin built on it) and any user-supplied AudioOutput subclass. That is where the defect looks reachable, and it is not where the plain room pipeline described in the issue text sits.

We want to be careful about how far that goes: we have not verified that an avatar or DataStreamAudioOutput sink reproduces this live. That was not tested, and hand-rolling another stand-in sink would be construction rather than evidence. It is also worth noting you were on 1.5.3 while our testing was on current main.

So, the question that would settle this: what audio output is your session using — a plain LiveKit room output, an avatar plugin or DataStreamAudioOutput, or a custom AudioOutput subclass? And secondarily, since you reproduced deterministically in a harness: is the sink in that harness the same one your production sessions use? Synthetic harnesses often substitute a simpler or a mock sink, and given the above, that substitution would be load-bearing rather than incidental.

If your answer is a room output, then we have a gap in our reconstruction and would like to find it, and any detail about how your harness commits the second turn would help. Either way, #2127 proceeds.

[toubatbrian]

Reproduced this live and confirmed the mechanism, so I can close the loop on my earlier question about your audio output.

Your description was accurate. The reply that is interrupted before its audio finishes leaves the speech scheduler parked on that handle forever, and every later turn is queued but never authorized — the agent stops responding for the rest of the session rather than recovering.

The ingredient that decides whether you hit it is the audio output. Against a plain LiveKit room output I could not reproduce it in six attempts, because that path resolves the pending futures itself when the forwarding loop unwinds. Against a sink that cannot confirm a segment the remote never began playing — remote avatar outputs, DataStreamAudioOutput, or any custom AudioOutput — it reproduces on the first barge-in. In an instrumented run the scheduler parked and never released, and three spoken follow-ups afterwards produced no reaction at all.

One detail worth knowing if you are writing your own sink: a single missed playback-finished report is not confined to one turn. waitForPlayout() waits for the finished count to catch up to the captured count, so one missing report leaves the counter permanently one behind and every subsequent barge-in degrades too.

#2127 fixes it by porting python's INTERRUPTION_TIMEOUT watchdog: an interrupted speech that has not finished within 5 s has its tasks cancelled and is marked done, which releases the scheduler. In the same instrumented scenario each barge-in then recovered after a bounded ~5 s instead of muting the session permanently.

[Raysharr]

Thanks for chasing this down, and for the write-up — far more than we expected from a drive-by report. Glad #2127 landed.

To answer your question, and our earlier framing was too simple here: not a bare room output. There's no avatar plugin, no DataStreamAudioOutput and no custom AudioOutput subclass anywhere in our code — we just call session.start({ agent, room }) — but we pass record: true, and AgentSession then replaces output.audio with RecorderIO's RecorderAudioOutput chained in front of the room output (agent_session.js:351 on 1.5.3). So the sink forwardSegment actually talks to is a chained recorder sink, which is a third category next to the two you tested.

One caveat on that: recording is enabled in our config and the wiring predates the report by months, but I can't prove from the artifacts we kept that it was on in the specific runs that wedged. The rest of the setup is as described — production is SIP inbound, and the harness that reproduced it deterministically isn't a mock either: real LiveKit Cloud room, real WebRTC participant, full STT/LLM/TTS pipeline, same sink as production.

If the chained-recorder path is worth pinning down, we're happy to re-run the harness against main with recording on and our mainTask patch off, and send whatever the logs show — as a separate issue rather than reviving this one. Either way we'll drop our patch once #2127 ships. Thanks again for the quick turnaround.