diff --git a/docs/realtime/history.md b/docs/realtime/history.md index 049ff520d1..1f50255d3e 100644 --- a/docs/realtime/history.md +++ b/docs/realtime/history.md @@ -161,6 +161,12 @@ delivered to your app yet, such as a queued [`PartEndEvent`][pydantic_ai.message waiting in a [`stream_transcripts()`][pydantic_ai.realtime.RealtimeSession.stream_transcripts] view, keep their part's audio until they are consumed. +On OpenAI, Azure OpenAI, and xAI, retained audio is kept in the history +[`all_messages()`][pydantic_ai.realtime.RealtimeSession.all_messages] returns. The conversation a +tool sees as [`RunContext.messages`][pydantic_ai.tools.RunContext.messages], and the input recorded on +each response's [span](observability.md), carry transcripts without that audio for now; the events and +the response a span records keep it. + Input retention follows provider-reported boundaries rather than locally trimming speech. OpenAI, Azure OpenAI, and xAI normally retain microphone input between reported speech-end boundaries. Gemini does not report those boundaries, so it retains input between response completions. Either @@ -200,14 +206,33 @@ sends one frame per second — so for camera and screen streams, use both delibe Input transcription defaults to `'auto'`; see [Input transcription](audio.md#input-transcription) and each provider page for configuration. Transcripts are recorded with the user turn they describe, -even when they arrive after that turn's response or overlap the following turn. A turn the user -starts while the model is still answering, whether they [barge in](turns.md#barge-in) or push to talk -over it, is recorded after that answer. Such a turn joins history once the provider ends the answer it -cut off, or after a few seconds if the provider never does. In that fallback the turn is recorded where -history stands, so it lands before the answer it interrupted, and ahead of anything sent with -[`send()`][pydantic_ai.realtime.RealtimeSession.send] while that answer was still in flight. If a reported speech -segment never receives a transcript, the session still records its retained audio or a content-less -`SpeechPart` when the session closes. +even when they arrive after that turn's response or overlap the following turn. + +On OpenAI, Azure OpenAI, and xAI, history follows the order of the provider's own conversation: + +- A spoken turn sits where the provider added it, and text, images, and tool results where they reached + it. A turn the user started while the model was still answering, but which the provider only committed + after that answer ended, is recorded after the answer. +- A message appears in `all_messages()` once everything before it is final. A reply waits for the + transcript of the spoken turn before it, which can arrive after the reply itself is done, for up to 30 + seconds after that; past that, the turn is recorded with the transcript it has so far. +- Nothing is inserted ahead of messages already returned, so each snapshot starts with the one before it. + The one exception is a tool's return, which always directly follows the response that called it: a tool + that finishes after later messages were recorded has its return inserted ahead of them. + +- Assistant messages from `gpt-realtime-2` models carry their `phase` (`'commentary'` on the way to a tool + call, or `'final_answer'`) as `'phase'` in the part's `provider_details`, the way a standard OpenAI run + records [text phases](../models/openai.md#text-phases). + +On Gemini Live and GPT-Live, a turn the user starts while the model is still answering, whether they +[barge in](turns.md#barge-in) or push to talk over it, is recorded after that answer. Such a turn joins +history once the provider ends the answer it cut off, or after a few seconds if the provider never does. +In that fallback the turn is recorded where history stands, so it lands before the answer it interrupted, +and ahead of anything sent with [`send()`][pydantic_ai.realtime.RealtimeSession.send] while that answer was +still in flight. + +If a reported speech segment never receives a transcript, the session still records its retained audio or +a content-less `SpeechPart`, at the latest when the session closes. With transcription disabled: @@ -217,8 +242,10 @@ With transcription disabled: the speech between them: the silence an always-on microphone streams between utterances is no turn. Gemini Live reports none, so a turn there runs to the response that answers it, and audio sent after the last response is recorded as one more turn when the session closes; -- with [push-to-talk](turns.md#push-to-talk), each `commit_audio()` after sending audio is a turn; a - commit with no audio since the last one records nothing; +- with [push-to-talk](turns.md#push-to-talk), each `commit_audio()` after sending audio is a turn once the + provider has it (on xAI, commits made while a reply is still under way go out together after it, as one turn, + and one still held back when the session closes records nothing); a commit with no audio since the last one + records nothing; - content-less parts preserve the local turn boundary but contribute no words to a text handoff and are skipped when seeding another realtime session; - transcript-less assistant audio cannot be handed off or seeded on any provider. diff --git a/docs/realtime/observability.md b/docs/realtime/observability.md index b6f0ea005e..c940153243 100644 --- a/docs/realtime/observability.md +++ b/docs/realtime/observability.md @@ -91,7 +91,8 @@ to the normal content-redaction setting. Like a [classic agent run span](../logf it reports only what the session itself spent: a total carried in with `usage=` and a delegated run's tokens still count toward `session.usage`, but not toward the span, so agent run spans can be added up without counting anything twice. Nested provider-response spans have the OpenTelemetry name -`chat {model}` but display as `response {model}` in Logfire. `execute_tool` spans represent tools; +`chat {model}` but display as `response {model}` in Logfire. On OpenAI, Azure OpenAI, and xAI, a response span's input +messages don't include [retained audio](history.md#retaining-audio) for now; its output does. `execute_tool` spans represent tools; a delegated run adds its own `invoke_agent` span inside `execute_tool`. `model turn complete` and `interrupt` spans mark those boundaries. A tool round can produce several response spans within one turn. diff --git a/pydantic_ai_slim/pydantic_ai/realtime/_core.py b/pydantic_ai_slim/pydantic_ai/realtime/_core.py index a75a87f6ab..4f92137ed3 100644 --- a/pydantic_ai_slim/pydantic_ai/realtime/_core.py +++ b/pydantic_ai_slim/pydantic_ai/realtime/_core.py @@ -23,7 +23,9 @@ sits directly after the response that called it, as request-response APIs requir from __future__ import annotations as _annotations -from collections.abc import Callable, Iterator, Sequence +from bisect import insort +from collections import OrderedDict, deque +from collections.abc import Callable, Iterable, Iterator, Sequence from dataclasses import dataclass, field, replace from typing import Any, Literal, TypeAlias, assert_never @@ -56,6 +58,7 @@ from ._lifecycle import ( InputId, InputLost, LifecycleEvent, + OutputItemDetails, ResponseEnded, ResponseRequestRefused, ResponseStarted, @@ -64,6 +67,7 @@ from ._lifecycle import ( UserTurnEnded, UserTurnStarted, ) +from ._retained_audio import RetainedAudioBudget from ._utils import accumulate_transcript, pcm_to_wav, user_transcript_update from .codec import ( AudioDelta, @@ -114,6 +118,11 @@ class AudioSent: data: bytes +@dataclass(frozen=True, kw_only=True) +class AudioUnsent: + """The input audio last reported as sent never went out: it is taken back, as much of it as is still there.""" + + @dataclass(frozen=True, kw_only=True) class AudioCleared: """The caller discarded the buffered input audio.""" @@ -152,6 +161,16 @@ class ExchangeAbandoned: """ +@dataclass(frozen=True, kw_only=True) +class TranscriptOverdue: + """A spoken turn waited too long for its transcript: it is recorded with what it has. + + See `transcript_holding_history`. + """ + + turn_id: str + + @dataclass(frozen=True, kw_only=True) class ReceiveEnded: """The session stopped reading the connection: nothing more will be said, so nothing is owed any more.""" @@ -166,11 +185,13 @@ Command: TypeAlias = ( InputSent | InputWithdrawn | AudioSent + | AudioUnsent | AudioCleared | ToolReturned | ToolCallRefused | Interrupted | ExchangeAbandoned + | TranscriptOverdue | ReceiveEnded | Closed ) @@ -191,6 +212,8 @@ class _Response: open_part_item: str | None = None open_transcript: str = '' open_audio: bytearray = field(default_factory=bytearray) + item_details: dict[str, dict[str, Any]] = field(default_factory=dict[str, dict[str, Any]]) + """What the provider said about each item of the output, for the part the item makes.""" usage: RequestUsage = field(default_factory=RequestUsage) usage_details: dict[str, Any] | None = None usage_finish_reason: FinishReason | None = None @@ -200,6 +223,8 @@ class _Response: message: ModelResponse | None = None """What history records for it, built once it ended (and `None` if it ended with nothing to record).""" tool_calls: list[str] = field(default_factory=list[str]) + placed_at: int = 0 + """Its position among the entities that joined the conversation (see `SessionCore._place`).""" @dataclass(eq=False) @@ -215,17 +240,23 @@ class _UserTurn: audio, and with it the turn, ends at the speech end.""" message: ModelRequest | None = None """Built once the turn is final: transcribed (or known never to be) and joined to the conversation.""" + placed_at: int = 0 + """Its position among the entities that joined the conversation (see `SessionCore._place`).""" @dataclass(eq=False) class _Input: id: InputId request: ModelRequest - withdrawn: bool = False _Entry: TypeAlias = _Response | _UserTurn | _Input + +def _has_audio(message: ModelMessage) -> bool: + return any(isinstance(part, SpeechPart) and part.audio is not None for part in message.parts) + + _Obligation = Literal['pending', 'answered', 'refused', 'lost', 'void'] @@ -254,9 +285,11 @@ class SessionCore: input_transcription_enabled: bool = True, retain_input_audio: bool = False, retain_output_audio: bool = False, + retain_audio_max_seconds: float | None = None, input_sample_rate: int = 24000, output_sample_rate: int = 24000, seeded: Sequence[ModelMessage] = (), + usage: RunUsage | None = None, ) -> None: self._model_name = model_name self._provider_name = provider_name @@ -270,13 +303,19 @@ class SessionCore: self._input_rate = input_sample_rate self._output_rate = output_sample_rate self._seeded = list(seeded) - self.usage = RunUsage() - """Token usage and requests, as the provider reported them for this session.""" + self.usage = usage if usage is not None else RunUsage() + """Token usage and requests, as the provider reported them for this session, added to what it held.""" + self.requests = 0 + """The requests counted into `usage` here.""" self._placed: list[_Entry] = [] """Entities in the order they joined the provider's conversation.""" self._responses: dict[str, _Response] = {} self._turns: dict[str, _UserTurn] = {} + self._replayed_items: set[str] = set() + """Item ids of a resumed conversation's replay, whose content the core already has.""" + self._replayed_calls: set[str] = set() + """Tool call ids of that replay (ids of another kind, so kept apart: an item id can recur as a call id).""" self._inputs: dict[InputId, _Input] = {} """The inputs history records (not audio chunks, say, or a bare request for a response), by id.""" self._unplaced: dict[InputId, _Input] = {} @@ -291,8 +330,25 @@ class SessionCore: self._settled_calls: set[str] = set() """Calls that settled: their result is recorded (sent or not), or the provider cancelled them.""" self._input_audio = bytearray() + # What `retain_audio_max_seconds` bounds, as `retain_images_max` bounds images: the audio built into history, + # the audio of turns waiting to be recorded, and what is still coming in (`_input_audio`, `open_audio`). + self._audio_budget = RetainedAudioBudget( + retain_audio_max_seconds, input_sample_rate=input_sample_rate, output_sample_rate=output_sample_rate + ) + self._waiting_turn_audio: OrderedDict[str, _UserTurn] = OrderedDict() + """Turns with audio cut for them but not recorded yet, oldest first.""" + self._waiting_turn_audio_bytes = 0 + self._open_audio: dict[str, _Response] = {} + """Responses with output audio coming in for a part, oldest first.""" + self._open_audio_bytes = 0 + self._recorded_audio: deque[_Response | _UserTurn] = deque() + """Recorded entities whose message still has audio, in the order they joined the conversation: what eviction + takes audio from first, oldest first, without passing over anything that has none or isn't recorded yet.""" + self._unsent_audio: tuple[_UserTurn | None, int] | None = None + """The latest input audio chunk, until the next: the turn it was cut for since (or `None` while it is in the + buffer), and its length, so `AudioUnsent` can take it back if its send fails.""" + self._placements = 0 self._user_speaking = False - self._speaking_item: str | None = None self._speech_segmented = False """Whether the provider draws speech-end boundaries, so the retained audio is cut into turns at them.""" self._closed = False @@ -313,12 +369,28 @@ class SessionCore: self._response_content(item) elif isinstance(item, ToolCall): self._tool_call(item) + elif isinstance(item, RealtimeSessionReconnectEvent): + # The new connection hears none of what was being said on the old one: a turn still spoken is over, + # with the audio sent for it. + for turn_id in [turn_id for turn_id, turn in self._turns.items() if turn.speaking]: + self._speech_ended(turn_id) + self._user_speaking = False + elif isinstance(item, ConversationItemCreated): + if item.replayed: + # A resumed conversation's replay of what history already has (xAI): nothing in it is new. + if item.item_id is not None: + self._replayed_items.add(item.item_id) + if item.tool_call_id is not None: + self._replayed_calls.add(item.tool_call_id) + elif isinstance(item, OutputItemDetails): + if (response := self._open_response(item.response_id)) is not None: + response.item_details[item.item_id] = item.provider_details elif isinstance(item, SessionUsage): self._usage(item) elif isinstance(item, UserTurnStarted): self._turn(item.turn_id) elif isinstance(item, UserTurnEnded): - self._end_turn(item.turn_id) + self._end_turn(item.turn_id, still_speaking=item.still_speaking) elif isinstance(item, UserTurnDiscarded): self._discard_turn(item.turn_id) elif isinstance(item, InputTranscript): @@ -328,7 +400,6 @@ class SessionCore: self._transcribed(turn, failed=True) elif isinstance(item, RealtimeInputSpeechStartEvent): self._user_speaking = True - self._speaking_item = item.item_id elif isinstance(item, RealtimeInputSpeechEndEvent): self._speech_ended(item.item_id) elif isinstance(item, InputAdded): @@ -347,10 +418,8 @@ class SessionCore: RealtimeOutputSpeechStartEvent, RealtimeOutputSpeechEndEvent, RealtimeResponseInterruptedEvent, - RealtimeSessionReconnectEvent, RealtimeSessionErrorEvent, ConversationCreated, - ConversationItemCreated, PartStartEvent, PartEndEvent, ), @@ -364,11 +433,8 @@ class SessionCore: self._input_sent(command) elif isinstance(command, InputWithdrawn): self._withdraw(command.input_ids) - elif isinstance(command, AudioSent): - if self._retain_input: - self._input_audio.extend(command.data) - elif isinstance(command, AudioCleared): - self._input_audio.clear() + elif isinstance(command, (AudioSent, AudioUnsent, AudioCleared)): + self._input_audio_command(command) elif isinstance(command, ToolReturned): self._returns[command.tool_call_id] = command.request self._settled_calls.add(command.tool_call_id) @@ -380,13 +446,50 @@ class SessionCore: elif isinstance(command, ExchangeAbandoned): self._abandoned.update((token.kind, token.key) for token in self.wait_tokens()) self._epoch += 1 + elif isinstance(command, TranscriptOverdue): + self._end_transcriptions([command.turn_id]) elif isinstance(command, ReceiveEnded): - self._receive_ended() + self._receiving = False + # Nothing more will be read: not the rest of a response under way, nor of a turn that joined. + self._lose_open_responses() + self._end_transcriptions([turn_id for turn_id, turn in self._turns.items() if turn.ended]) elif isinstance(command, Closed): self._close() else: assert_never(command) + def _input_audio_command(self, command: AudioSent | AudioUnsent | AudioCleared) -> None: + if isinstance(command, AudioSent): + if self._retain_input: + self._input_audio.extend(command.data) + self._unsent_audio = (None, len(command.data)) + self._bound_retained_audio() + elif isinstance(command, AudioUnsent): + self._take_back_audio() + else: + self._unsent_audio = None + self._input_audio.clear() + # A turn the user was saying that hadn't joined the conversation never will now. + for turn_id in [turn_id for turn_id, turn in self._turns.items() if not turn.ended]: + self._drop_turn(turn_id) + + def _take_back_audio(self) -> None: + """Take back the latest input audio chunk, whose send failed: from the buffer, or the turn it was cut for. + + By length, as the session takes it back from its own buffer: what of it the budget already trimmed, or + that went into history with its turn, is no longer there to take back. + """ + if self._unsent_audio is None: + return + turn, length = self._unsent_audio + self._unsent_audio = None + if turn is None: + del self._input_audio[len(self._input_audio) - min(length, len(self._input_audio)) :] + elif turn.id in self._waiting_turn_audio and turn.audio: + length = min(length, len(turn.audio)) + turn.audio = turn.audio[: len(turn.audio) - length] + self._waiting_turn_audio_bytes -= length + def _input_sent(self, command: InputSent) -> None: if command.request is not None: self._inputs[command.input_id] = self._unplaced[command.input_id] = _Input( @@ -397,11 +500,10 @@ class SessionCore: if command.tool_call_id is not None: self._result_input[command.tool_call_id] = command.input_id - def _receive_ended(self) -> None: - self._receiving = False - for turn in self._turns.values(): - if turn.ended: - # Nothing more of a turn that joined will be read: not the rest of it, nor its transcript. + def _end_transcriptions(self, turn_ids: Iterable[str]) -> None: + """Record turns that joined the conversation with the transcript they have, waiting for no more of it.""" + for turn_id in turn_ids: + if (turn := self._turns.get(turn_id)) is not None and turn.ended: turn.speaking = False self._transcribed(turn, failed=False) @@ -416,7 +518,7 @@ class SessionCore: self._place_input(input_id) response = _Response(event.response_id, event.answers, self._model_name()) self._responses[event.response_id] = response - self._placed.append(response) + self._place(response) for input_id in event.answers: if self._obligations.get(input_id) == 'pending': self._obligations[input_id] = 'answered' @@ -424,13 +526,16 @@ class SessionCore: def _response_content(self, event: AudioDelta | OutputTranscript) -> None: response = self._open_response(event.response_id) - if response is None: + if response is None or (event.item_id is not None and event.item_id in self._replayed_items): return output_text = isinstance(event, OutputTranscript) and event.output_text part = self._open_part(response, output_text=output_text, item_id=event.item_id) if isinstance(event, AudioDelta): if self._retain_output: response.open_audio.extend(event.data) + self._open_audio[response.id] = response + self._open_audio_bytes += len(event.data) + self._bound_retained_audio() return response.open_transcript, _ = accumulate_transcript(response.open_transcript, event.text) response.open_part = ( @@ -450,13 +555,21 @@ class SessionCore: part = None elif item_id is not None and response.open_part_item is None: response.open_part_item = item_id + part = response.open_part = self._with_item_details(response, part, item_id) if part is None: part = TextPart(content='') if output_text else SpeechPart(speaker='assistant', transcript='') - response.open_part = part + part = response.open_part = self._with_item_details(response, part, item_id) response.open_part_item = item_id response.open_transcript = '' return part + def _with_item_details( + self, response: _Response, part: SpeechPart | TextPart, item_id: str | None + ) -> SpeechPart | TextPart: + if item_id is None or (details := response.item_details.get(item_id)) is None: + return part + return replace(part, provider_name=self._provider_name, provider_details=details) + def _close_part(self, response: _Response) -> None: part = response.open_part if part is None: @@ -466,16 +579,25 @@ class SessionCore: part = replace(part, transcript=None) if self._retain_output and response.open_audio: wav = pcm_to_wav(bytes(response.open_audio), self._output_rate) - part = replace(part, audio=BinaryContent(data=wav, media_type=_WAV_MEDIA_TYPE)) + audio = BinaryContent(data=wav, media_type=_WAV_MEDIA_TYPE) + part = replace(part, audio=self._audio_budget.track(audio, len(response.open_audio), output=True)) response.parts.append(part) response.open_part = None response.open_part_item = None response.open_transcript = '' + self._open_audio_bytes -= len(response.open_audio) + self._open_audio.pop(response.id, None) response.open_audio.clear() def _tool_call(self, event: ToolCall) -> None: response = self._open_response(event.response_id) - if response is None: + if ( + response is None + or event.tool_call_id in self._call_response + or event.tool_call_id in self._replayed_calls + or (event.item_id is not None and event.item_id in self._replayed_items) + ): + # Repeated (a resumed conversation replaying it, say): the call is already recorded, once. return self._close_part(response) response.parts.append(ToolCallPart(tool_name=event.tool_name, args=event.args, tool_call_id=event.tool_call_id)) @@ -492,11 +614,12 @@ class SessionCore: ] def _usage(self, event: SessionUsage) -> None: - self.usage.incr(event.usage) # usage-attribution: the core's own tally + self.usage.incr(event.usage) # usage-attribution: the session owns its spans if not event.response_scoped: return if not self._responses_are_requests: - self.usage.requests += 1 # usage-attribution: the core's own tally + self.usage.requests += 1 # usage-attribution: the session owns its spans; `wrap_run` opens none + self.requests += 1 response = self._responses.get(event.provider_response_id) if event.provider_response_id else None if response is None or response.status is not None: # Usage for a response already recorded counts toward the session, never toward its message. @@ -545,9 +668,13 @@ class SessionCore: provider_cost = message.usage.cost fill_response_cost(message) if provider_cost is None: - self.usage.incr(RequestUsage(cost=message.usage.cost)) # usage-attribution: the core's own tally - self.usage.requests += int(self._responses_are_requests) # usage-attribution: the core's own tally + self.usage.incr(RequestUsage(cost=message.usage.cost)) # usage-attribution: the session owns its spans + self.usage.requests += int(self._responses_are_requests) # usage-attribution: the session owns its spans + self.requests += int(self._responses_are_requests) response.message = message + # The message holds the parts now: the core keeps no other reference to their audio. + response.parts = [] + self._record_audio(response, message) def _open_response(self, response_id: str | None) -> _Response | None: """The open response content names; content naming none (Azure Voice Live's text) goes to the only one open.""" @@ -573,29 +700,42 @@ class SessionCore: self._speech_segmented = True if item_id is None or (turn := self._turns.get(item_id)) is None: return - if self._retain_input and self._input_audio: - if turn.audio is None: - turn.audio = bytes(self._input_audio) - self._input_audio.clear() + if self._retain_input and self._input_audio and turn.audio is None: + self._cut_turn_audio(turn) if turn.speaking: turn.speaking = False self._turn_said(turn) - def _end_turn(self, turn_id: str) -> None: + def _end_turn(self, turn_id: str, *, still_speaking: bool) -> None: turn = self._turn(turn_id) if turn.ended: return turn.ended = True - self._placed.append(turn) - if self._user_speaking and turn_id == self._speaking_item: + self._place(turn) + if still_speaking: turn.speaking = True return if turn.audio is None and self._retain_input and self._input_audio: # Committed by hand, with no speech end to cut it at: the whole buffer is this turn's. - turn.audio = bytes(self._input_audio) - self._input_audio.clear() + self._cut_turn_audio(turn) self._turn_said(turn) + def _cut_turn_audio(self, turn: _UserTurn) -> None: + """Give `turn` the input audio retained since the last cut: it waits with the turn until it is recorded.""" + turn.audio = bytes(self._input_audio) + self._input_audio.clear() + self._waiting_turn_audio[turn.id] = turn + self._waiting_turn_audio_bytes += len(turn.audio) + if self._unsent_audio is not None and self._unsent_audio[0] is None: + self._unsent_audio = (turn, self._unsent_audio[1]) + + def _release_turn_audio(self, turn: _UserTurn) -> None: + if self._waiting_turn_audio.pop(turn.id, None) is not None and turn.audio: + self._waiting_turn_audio_bytes -= len(turn.audio) + + def _drop_turn(self, turn_id: str) -> None: + self._release_turn_audio(self._turns.pop(turn_id)) + def _turn_said(self, turn: _UserTurn) -> None: """All of a joined turn's audio is in: it is final once transcribed, or at once if nothing transcribes it.""" if not self._transcription: @@ -611,7 +751,7 @@ class SessionCore: turn.speaking = False self._transcribed(turn, failed=False) else: - del self._turns[turn_id] + self._drop_turn(turn_id) def _input_transcript(self, event: InputTranscript) -> None: # Only a spoken turn the connection reported transcribes into history: not, say, the empty audio item @@ -633,26 +773,35 @@ class SessionCore: self._build_turn(turn) def _build_turn(self, turn: _UserTurn) -> None: - audio = ( - BinaryContent(data=pcm_to_wav(turn.audio, self._input_rate), media_type=_WAV_MEDIA_TYPE) - if turn.audio - else None - ) + self._release_turn_audio(turn) + turn.message = self._turn_request(turn, retained=True) + # The message holds the audio now (as WAV): the PCM goes, empty rather than `None` so nothing cuts it again. + turn.audio = b'' + self._record_audio(turn, turn.message) + + def _turn_request(self, turn: _UserTurn, *, retained: bool = False) -> ModelRequest: + """The request `turn` records; counted against `retain_audio_max_seconds` when `retained` in history.""" + audio: BinaryContent | None = None + if turn.audio: + audio = BinaryContent(data=pcm_to_wav(turn.audio, self._input_rate), media_type=_WAV_MEDIA_TYPE) + if retained: + audio = self._audio_budget.track(audio, len(turn.audio), output=False) part = SpeechPart(speaker='user', transcript=turn.transcript.strip() or None, audio=audio) request = ModelRequest(parts=[part]) fill_run_metadata(request, run_id=self._run_id, conversation_id=self._conversation_id) - turn.message = request + return request # --- inputs and obligations -------------------------------------------------------------------- def _place_input(self, input_id: InputId) -> None: if (input_ := self._unplaced.pop(input_id, None)) is not None: - self._placed.append(input_) + self._place(input_) def _withdraw(self, input_ids: Sequence[InputId]) -> None: for input_id in input_ids: - if (input_ := self._inputs.get(input_id)) is not None: - input_.withdrawn = True + # Dropped altogether, so what it carried (an evicted image, say) isn't kept alive for nothing. + if (input_ := self._inputs.pop(input_id, None)) is not None and self._unplaced.pop(input_id, None) is None: + self._placed.remove(input_) self._settle(input_ids, 'void') def _settle(self, input_ids: Sequence[InputId], outcome: _Obligation) -> None: @@ -660,17 +809,93 @@ class SessionCore: if self._obligations.get(input_id) == 'pending': self._obligations[input_id] = outcome + # --- retained audio ------------------------------------------------------------------------------- + + def _bound_retained_audio(self) -> None: + """Evict the oldest retained audio until what the core retains fits `retain_audio_max_seconds`. + + As the session does: audio recorded in history goes first, oldest first, each part keeping its transcript; + then the audio of turns waiting to be recorded, oldest first; and only then what is still coming in, which + keeps its most recent audio. A part not recorded yet is only evicted once it is. Only the local copy is + dropped: the provider received all of it. + """ + budget = self._audio_budget + excess = budget.excess( + untracked_input=self._waiting_turn_audio_bytes + len(self._input_audio), + untracked_output=self._open_audio_bytes, + ) + if excess > 0: + excess = self._evict_recorded_audio(excess) + while excess > 0 and self._waiting_turn_audio: + _, turn = self._waiting_turn_audio.popitem(last=False) + evicted = len(turn.audio or b'') + self._waiting_turn_audio_bytes -= evicted + # Empty rather than `None`, so no later cut gives the turn audio again. + turn.audio = b'' + excess -= budget.weight(evicted, output=False) + excess = budget.trim(self._input_audio, excess, output=False) + for response in list(self._open_audio.values()): + if excess <= 0: + break + before = len(response.open_audio) + excess = budget.trim(response.open_audio, excess, output=True) + self._open_audio_bytes -= before - len(response.open_audio) + if not response.open_audio: + # Emptied: audio that comes in for it later is the newest, so it goes to the back. + del self._open_audio[response.id] + + def _evict_recorded_audio(self, excess: int) -> int: + """Strip the audio of recorded messages, oldest first, until `excess` is freed; the excess left.""" + budget = self._audio_budget + while excess > 0 and self._recorded_audio: + entry = self._recorded_audio[0] + if isinstance(entry, _Response): + assert entry.message is not None + stripped, freed = budget.strip(entry.message, excess) + assert isinstance(stripped, ModelResponse) + entry.message = stripped + else: + assert entry.message is not None + stripped, freed = budget.strip(entry.message, excess) + assert isinstance(stripped, ModelRequest) + entry.message = stripped + excess -= freed + if not freed or not _has_audio(stripped): + self._recorded_audio.popleft() + return excess + + def _place(self, entry: _Entry) -> None: + """`entry` joins the conversation here.""" + if not isinstance(entry, _Input): + self._placements += 1 + entry.placed_at = self._placements + self._placed.append(entry) + + def _record_audio(self, entry: _Response | _UserTurn, message: ModelMessage) -> None: + """`entry` is recorded as `message`: if that has audio, it is evicted in conversation order.""" + if not _has_audio(message): + return + queue = self._recorded_audio + if not queue or queue[-1].placed_at < entry.placed_at: + queue.append(entry) + else: + # Recorded after something that joined the conversation later (a turn whose transcript came late). + insort(queue, entry, key=lambda recorded: recorded.placed_at) + + def _lose_open_responses(self) -> None: + for response in [response for response in self._responses.values() if response.status is None]: + self._end_response(ResponseEnded(response_id=response.id, status='lost')) + def _close(self) -> None: if self._closed: return self._closed = True - for response in [response for response in self._responses.values() if response.status is None]: - self._end_response(ResponseEnded(response_id=response.id, status='lost')) + self._lose_open_responses() for turn in self._turns.values(): if turn.message is None: if not turn.ended: turn.ended = True - self._placed.append(turn) + self._place(turn) turn.speaking = False self._transcribed(turn, failed=False) for input_id in list(self._unplaced): @@ -680,6 +905,16 @@ class SessionCore: self._obligations[input_id] = 'void' self._settled_calls.update(self._call_response) + def set_conversation_id(self, conversation_id: str) -> None: + """The conversation id the session resolved late: what the core records from now on, and has recorded, carries it.""" + self._conversation_id = conversation_id + recorded: list[ModelMessage] = [*(input_.request for input_ in self._inputs.values()), *self._returns.values()] + for entry in self._placed: + if not isinstance(entry, _Input) and entry.message is not None: + recorded.append(entry.message) + for message in recorded: + message.conversation_id = message.conversation_id or conversation_id + # --- what the session reads --------------------------------------------------------------------- def all_messages(self) -> list[ModelMessage]: @@ -690,20 +925,64 @@ class SessionCore: return list(self._visible()) def _visible(self) -> Iterator[ModelMessage]: - for entry in self._placed: + for entry in self._placed[: self._held_from()]: if isinstance(entry, _Response): - if entry.status is None: - return if entry.message is not None: yield entry.message yield from self._returns_of(entry) elif isinstance(entry, _UserTurn): - if entry.message is None: - return + assert entry.message is not None yield entry.message - elif not entry.withdrawn: + else: yield entry.request + def replayable_messages(self) -> list[ModelMessage]: + """The provider's conversation as far as it went, for a reconnect to replay on a new connection. + + Unlike `all_messages()`, this holds nothing back: a spoken turn still waiting for its transcript is in it + with what it has, and so is everything after it. A response under way isn't: the old connection takes it + along. Inputs sent that the provider hadn't acknowledged yet come last, as sent: the old connection may + never have delivered them, and the reply the reconnect asks for again answers them. + """ + messages: list[ModelMessage] = list(self._seeded) + for entry in self._placed: + if isinstance(entry, _Response): + if entry.message is not None: + messages.append(entry.message) + messages.extend(self._returns_of(entry)) + elif isinstance(entry, _UserTurn): + messages.append(entry.message or self._turn_request(entry)) + else: + messages.append(entry.request) + messages.extend(input_.request for input_ in self._unplaced.values()) + return messages + + def _held_from(self) -> int: + """Where history is held back: the first entity not final yet (a response under way, a turn untranscribed).""" + for index, entry in enumerate(self._placed): + if (isinstance(entry, _Response) and entry.status is None) or ( + isinstance(entry, _UserTurn) and entry.message is None + ): + return index + return len(self._placed) + + def transcript_holding_history(self) -> str | None: + """The spoken turn waiting for its transcript that holds back a message recorded after it, if one does.""" + held_from = self._held_from() + if held_from == len(self._placed): + return None + turn = self._placed[held_from] + if not isinstance(turn, _UserTurn) or turn.speaking: + # A response under way ends on its own, and a turn still spoken with the speech. + return None + later_final = any( + (isinstance(entry, _Response) and entry.message is not None) + or (isinstance(entry, _UserTurn) and entry.message is not None) + or isinstance(entry, _Input) + for entry in self._placed[held_from + 1 :] + ) + return turn.id if later_final else None + def _returns_of(self, response: _Response) -> Iterator[ModelRequest]: for call_id in response.tool_calls: if (request := self._returns.get(call_id)) is not None: @@ -730,8 +1009,9 @@ class SessionCore: def still_owed(self, tokens: frozenset[Owed]) -> frozenset[Owed]: """Follow each token to what it led to, keeping only what is still to come. - An answered input leads to the response that answers it; a finished response to its tool calls, which - the model answers once it has their results; a call whose result went out to that result's input. + An answered input leads to the response that answers it; a finished response to its tool calls, which the + model answers once it has their results; a call whose result went out to that result's input. A finished + response can still be held back from history (see `transcript_holding_history`): that is not waited for. Once nothing more is read, nothing more can come, and what was owed when the exchange was abandoned (see `ExchangeAbandoned`) is owed no longer. """ diff --git a/pydantic_ai_slim/pydantic_ai/realtime/_lifecycle.py b/pydantic_ai_slim/pydantic_ai/realtime/_lifecycle.py index 6f7b2759c2..7c4bbe065f 100644 --- a/pydantic_ai_slim/pydantic_ai/realtime/_lifecycle.py +++ b/pydantic_ai_slim/pydantic_ai/realtime/_lifecycle.py @@ -13,7 +13,9 @@ states them outright, with an id on every entity: - a spoken user turn is bracketed by `UserTurnStarted` and `UserTurnEnded` (or `UserTurnDiscarded`); - `InputAdded` marks where an input joined the provider's conversation, so history can follow the provider's order; -- `InputLost` and `ResponseRequestRefused` settle the inputs no response will ever answer. +- `InputLost` and `ResponseRequestRefused` settle the inputs no response will ever answer; +- `OutputItemDetails` carries what the provider says about an item of a response's output, for the part it + makes. An input is identified the way `InputRejected.input_index` identifies it: by its position among every `send()` call made on the connection. @@ -91,6 +93,9 @@ class UserTurnEnded: """A spoken turn joined the provider's conversation, at this point in its order.""" turn_id: str + still_speaking: bool = False + """Whether it joined while the user is still saying it (xAI adds its item at speech start): its audio, and with + it the turn, then ends with the speech.""" @dataclass(frozen=True, kw_only=True) @@ -128,6 +133,19 @@ class ResponseRequestRefused: input_ids: tuple[InputId, ...] +@dataclass(frozen=True, kw_only=True) +class OutputItemDetails: + """What the provider says about one item of a response's output, to keep on the part that item makes. + + Like OpenAI's `phase`, which tells an assistant message the model meant as commentary on its way to a tool + call apart from its final answer. + """ + + response_id: str + item_id: str + provider_details: dict[str, Any] + + LifecycleEvent = TypeAliasType( 'LifecycleEvent', ResponseStarted @@ -137,7 +155,8 @@ LifecycleEvent = TypeAliasType( | UserTurnDiscarded | InputAdded | InputLost - | ResponseRequestRefused, + | ResponseRequestRefused + | OutputItemDetails, ) """The events a connection on the second version of the lifecycle contract yields besides the codec events.""" @@ -150,6 +169,7 @@ LIFECYCLE_EVENT_TYPES = ( InputAdded, InputLost, ResponseRequestRefused, + OutputItemDetails, ) """The `LifecycleEvent` variants, for `isinstance` checks.""" diff --git a/pydantic_ai_slim/pydantic_ai/realtime/_openai_lifecycle.py b/pydantic_ai_slim/pydantic_ai/realtime/_openai_lifecycle.py index d118034e2f..cf8a4c2df3 100644 --- a/pydantic_ai_slim/pydantic_ai/realtime/_openai_lifecycle.py +++ b/pydantic_ai_slim/pydantic_ai/realtime/_openai_lifecycle.py @@ -33,6 +33,7 @@ from ._lifecycle import ( InputId, InputLost, LifecycleEvent, + OutputItemDetails, ResponseEnded, ResponseRequestRefused, ResponseStarted, @@ -62,6 +63,17 @@ def _is_final_transcription(data: dict[str, Any]) -> bool: return status is None or status == 'completed' +def _output_item_details(data: dict[str, Any]) -> list[LifecycleEvent]: + """The `phase` of an assistant message a response adds (`commentary` or `final_answer`), for its part.""" + item, response_id = data.get('item'), data.get('response_id') + if not is_str_dict(item) or not isinstance(response_id, str) or not response_id: + return [] + item_id, phase = item.get('id'), item.get('phase') + if not isinstance(item_id, str) or not item_id or not isinstance(phase, str) or not phase: + return [] + return [OutputItemDetails(response_id=response_id, item_id=item_id, provider_details={'phase': phase})] + + def frame_response_id(event_type: str | None, data: dict[str, Any]) -> str | None: """The id of the response a server frame is about, if it is about one.""" if event_type in ('response.created', 'response.done'): @@ -273,6 +285,8 @@ class OpenAILifecycle: return [] if event_type in _CONVERSATION_ITEM_ADDED_FRAMES: return self.item_added(data) + if event_type == 'response.output_item.added': + return _output_item_details(data) if event_type == 'error': return self.error(data) return [] @@ -321,7 +335,7 @@ class OpenAILifecycle: self._speaking.pop(item_id, None) return events - def _place(self, item_id: str) -> list[LifecycleEvent]: + def _place(self, item_id: str, *, still_speaking: bool = False) -> list[LifecycleEvent]: """The spoken turn joins the conversation (once), whatever says so first.""" if item_id in self._committed: return [] @@ -333,7 +347,7 @@ class OpenAILifecycle: return [] # Push-to-talk reports no speech start: the commit both starts and ends the turn. events: list[LifecycleEvent] = [] if item_id in self._speaking else [UserTurnStarted(turn_id=item_id)] - events.append(UserTurnEnded(turn_id=item_id)) + events.append(UserTurnEnded(turn_id=item_id, still_speaking=still_speaking)) self._unclaimed_turn = item_id if self._transcribes: self._untranscribed.add(item_id) @@ -359,7 +373,7 @@ class OpenAILifecycle: elif item.id is not None and item.id in self._speaking: # A spoken turn joins the conversation when its item is added. That can be before the user has # stopped speaking: xAI adds it at speech start, and starts responding before the commit. - return self._place(item.id) + return self._place(item.id, still_speaking=True) elif not is_user_message_item(item): # The model's output, or a spoken turn already committed. return [] diff --git a/pydantic_ai_slim/pydantic_ai/realtime/_retained_audio.py b/pydantic_ai_slim/pydantic_ai/realtime/_retained_audio.py index b5ccfc745f..489b792903 100644 --- a/pydantic_ai_slim/pydantic_ai/realtime/_retained_audio.py +++ b/pydantic_ai_slim/pydantic_ai/realtime/_retained_audio.py @@ -70,17 +70,17 @@ class RetainedAudioBudget: - self._max_weight ) - def strip(self, message: ModelMessage, excess: int) -> tuple[ModelMessage, int]: + def strip(self, message: ModelMessage, excess: int | None = None) -> tuple[ModelMessage, int]: """A copy of `message` without the audio of its tracked speech parts, until `excess` is freed, and the weight freed. - Returns `message` itself, and `0`, when it has no tracked audio. A copy rather than an in-place edit, so a + With `excess=None`, all of its tracked audio goes. Returns `message` itself, and `0`, when it has no tracked audio. A copy rather than an in-place edit, so a snapshot of history already handed out doesn't change; the caller swaps it in. """ parts = list(message.parts) freed = 0 for index, part in enumerate(parts): if ( - freed < excess + (excess is None or freed < excess) and isinstance(part, SpeechPart) and part.audio is not None and (weight := self._tracked.pop(id(part.audio), None)) is not None diff --git a/pydantic_ai_slim/pydantic_ai/realtime/_session.py b/pydantic_ai_slim/pydantic_ai/realtime/_session.py index be79401373..bfbb73332f 100644 --- a/pydantic_ai_slim/pydantic_ai/realtime/_session.py +++ b/pydantic_ai_slim/pydantic_ai/realtime/_session.py @@ -4,14 +4,12 @@ from __future__ import annotations as _annotations import asyncio import dataclasses -import difflib import weakref from collections import OrderedDict, deque from collections.abc import AsyncGenerator, AsyncIterable, AsyncIterator, Callable, Sequence from copy import copy from dataclasses import dataclass, field, replace from itertools import takewhile -from pprint import pformat from time import time_ns from types import TracebackType from typing import TYPE_CHECKING, Any, Literal, Never, TypeAlias, TypeVar, assert_never, cast, overload @@ -54,7 +52,6 @@ from ..messages import ( FunctionToolCallEvent, FunctionToolResultEvent, ModelMessage, - ModelMessagesTypeAdapter, ModelRequest, ModelRequestPart, ModelResponse, @@ -89,15 +86,19 @@ from ..usage import RequestUsage, RunUsage, UsageLimits from ._core import ( AudioCleared, AudioSent, + AudioUnsent, Closed, + CoreInput, ExchangeAbandoned, InputSent, InputWithdrawn, Interrupted, + Owed, ReceiveEnded, SessionCore, ToolCallRefused, ToolReturned, + TranscriptOverdue, ) from ._instrumentation import ( SessionInstrumentation, @@ -346,6 +347,12 @@ _MAX_TRUNCATABLE_AUDIO_ITEMS = 32 # a second of the user starting to speak; past this, the turn is recorded where history stands instead of # staying out of `all_messages()` for as long as a provider keeps talking. _BARGE_IN_TURN_HOLD_SECONDS = 5.0 +# How long a spoken turn that joined the conversation may hold back history recorded after it while it waits for +# its transcript, with the session core: the turn is never inserted ahead of what was recorded since. Counted from +# when something after it is ready (normally the reply). Transcripts can arrive after the reply and the next turn +# (recorded: push-to-talk on gpt-realtime and gpt-realtime-2.1), so this is generous; past it, the turn is recorded +# with what it has, so a transcript that never comes can't hide the rest of the conversation for good. +_TRANSCRIPT_HOLD_SECONDS = 30.0 #: How long closing waits for the provider to end its session and report the usage it only reports then. #: Not waited for at all when the session is closing because it was cancelled. _END_SESSION_TIMEOUT = 2.0 @@ -359,12 +366,14 @@ _SESSION_DELTA_QUEUE_SIZE = 512 # bound a session nothing iterates keeps every one of them for as long as it runs. The same 512 buys a # late iterator around a hundred turns of structure, comfortably more history than the deltas it keeps. _SESSION_STRUCTURAL_QUEUE_SIZE = 512 -_CoreMode: TypeAlias = Literal['legacy', 'shadow'] -_CORE_MODE: _CoreMode = 'legacy' -"""Which session core a connection on version 2 of the lifecycle contract runs (see `_core.py`). +_CoreMode: TypeAlias = Literal['legacy', 'core'] +_CORE_MODE: _CoreMode = 'core' +"""Which session core keeps the history, usage, and reply waits of a connection on version 2 of the lifecycle +contract (see `_core.py`). -`'shadow'` runs the new core alongside the current one, which stays authoritative, and records where the -two disagree (`RealtimeSession._shadow_divergences`). Private, for the refactor's parity tests. +`'core'` makes the new core authoritative for them; this session still turns the connection's events into the +ones it streams, runs the tools, and owns the spans. `'legacy'` keeps all of it here, as connections on version 1 +always do. Private: a way back while the new core settles in. """ _TapItem = TypeVar('_TapItem') _Tap = TypeVar('_Tap') @@ -653,19 +662,6 @@ def _pending_message_text(pending: PendingMessage) -> str: return '\n\n'.join(texts) -def _comparable(messages: Sequence[ModelMessage]) -> list[Any]: - """Messages as plain data without their timestamps, which two cores building the same message differ in.""" - - def strip(value: Any) -> Any: - if isinstance(value, dict): - return {key: strip(item) for key, item in cast('dict[str, Any]', value).items() if key != 'timestamp'} - if isinstance(value, list): - return [strip(item) for item in cast('list[Any]', value)] - return value - - return strip(ModelMessagesTypeAdapter.dump_python(list(messages), mode='json')) - - class _RealtimePendingMessages(PendingMessageQueue): """A `RunContext.enqueue` queue that validates content and wakes the live session for delivery.""" @@ -910,6 +906,10 @@ class RealtimeSession: self._pending_response_usage = RequestUsage() self._response_limit_checked = False self._pending_response_requests = 0 + # The requests this session recorded. What it reserves against `request_limit` is released when it + # records them, while the core counts them into `usage.requests` as their frames arrive, a moment + # earlier: the projection counts each by this session's clock, so none is counted twice in between. + self._requests_recorded = 0 # Whether a response has actually begun, cleared at the exchange boundary that # `RealtimeTurnCompleteEvent` marks. A tool-calling turn spans several responses, so the # per-response flags above would read as "done" in the gaps. Reads pair with @@ -977,6 +977,9 @@ class RealtimeSession: # the watchdog that records them anyway if it never is (see `_BARGE_IN_TURN_HOLD_SECONDS`). self._held_user_turns: list[ModelRequest] = [] self._held_user_turn_watchdog: asyncio.TimerHandle | None = None + # With the session core: the timer of each spoken turn waiting for its transcript (see + # `_TRANSCRIPT_HOLD_SECONDS`). + self._transcript_watchdogs: dict[str, asyncio.TimerHandle] = {} # Whether audio was sent since the last `commit_audio()` or `clear_audio()`: committing an empty # buffer is no user turn. self._audio_uncommitted = False @@ -1083,11 +1086,10 @@ class RealtimeSession: self._traceparent_value: str | None = None self._result: AgentRunResult[str] | None = None - # The new core, run in shadow of this one on a connection that identifies its responses, turns, - # and inputs (see `_CORE_MODE`). + # The new core, which keeps history, usage, and reply waits on a connection that identifies its + # responses, turns, and inputs (see `_CORE_MODE`). self._core: SessionCore | None = None - self._shadow_divergences: list[str] = [] - if _CORE_MODE == 'shadow' and connection._lifecycle_version == 2: # pyright: ignore[reportPrivateUsage] + if _CORE_MODE == 'core' and connection._lifecycle_version == 2: # pyright: ignore[reportPrivateUsage] requested_model = self._model_name self._core = SessionCore( model_name=lambda: connection.model_name or requested_model, @@ -1099,9 +1101,11 @@ class RealtimeSession: input_transcription_enabled=self._input_transcription_enabled, retain_input_audio=self._retain_input, retain_output_audio=self._retain_output, + retain_audio_max_seconds=retain_audio_max_seconds, input_sample_rate=self.audio_input_sample_rate, output_sample_rate=self.audio_output_sample_rate, seeded=self._seeded, + usage=self.usage, ) async def __aenter__(self) -> RealtimeSession: @@ -1114,7 +1118,9 @@ class RealtimeSession: # Offer the conversation for replay, so a provider that keeps no state across sessions can # carry the call through a reconnect instead of resuming with amnesia. Gated on seeding # support because that is the mechanism, and a no-op where the provider resumes natively. - self._connection.set_message_history(self.all_messages) + self._connection.set_message_history( + self.all_messages if self._core is None else self._core.replayable_messages + ) self._connection._set_audio_commit_listener(self._place_held_commit) # pyright: ignore[reportPrivateUsage] self._session_instrumentation.start_session_span() @@ -1187,9 +1193,7 @@ class RealtimeSession: # recorded as interrupted, and every still-running tool call gets a cancelled return. The # returned events are discarded — the stream is closing and has no consumer left. self._finalize_lost_state() - if self._core is not None: - self._core.apply(Closed()) - self._compare_shadow(self._core) + self._apply_core(Closed()) self._teardown = asyncio.create_task(self._finish_teardown()) elif asyncio.current_task() in self._background_tasks: # A tool closing the session is cancelled by the teardown, at the wait below of its own @@ -1300,6 +1304,9 @@ class RealtimeSession: ) self._loop = None self._stop_held_user_turn_watchdog() + for watchdog in self._transcript_watchdogs.values(): + watchdog.cancel() + self._transcript_watchdogs.clear() # Do not hide the caller's own exception, but make sure every receive-side failure has one # delivery point even when iteration stopped early or was never started. Stored rather than @@ -1606,6 +1613,10 @@ class RealtimeSession: await session.wait_for_reply() ``` + The reply can show up in [`all_messages()`][pydantic_ai.realtime.RealtimeSession.all_messages] later than + it ends (by up to 30 seconds), where history waits for the transcript of a spoken turn before it (see + [History](../realtime/history.md#transcription-and-history-edge-cases)). + Waiting for the model to stop *generating* is not the same as waiting for the speaker to stop *playing*: pair it with [`wait_for_playback()`][pydantic_ai.realtime.RealtimeSession.wait_for_playback] before opening @@ -1613,14 +1624,23 @@ class RealtimeSession: """ self._ensure_streamable() self._start_pump() + # What the model owes at this moment, followed to the end: a reply asked for later is not waited for. + owed = self._core.wait_tokens() if self._core is not None else None while True: # Cleared before the check, so a boundary reached between the check and the wait still # wakes us rather than leaving this parked until the turn after it. self._exchange_progress.clear() - if not self._reply_outstanding(): + if owed is not None: + if self._closed or self._pump_finished or not self._core_still_owes(owed): + return + elif not self._reply_outstanding(): return await self._exchange_progress.wait() + def _core_still_owes(self, owed: frozenset[Owed]) -> bool: + assert self._core is not None + return bool(self._core.still_owed(owed)) + def _release_response_reservation(self) -> None: """Give back a reservation for a response that will never arrive, waking any waiter. @@ -1730,10 +1750,22 @@ class RealtimeSession: to `retain_images_every_n`, bounded by `retain_images_max`, and audio retained by `audio_retention` is bounded by `retain_audio_max_seconds`. """ + if self._core is not None: + return self._core.all_messages() + return [*self._seeded, *self._history] + + def _own_messages(self) -> list[ModelMessage]: + """The history this session itself builds as it streams its events, which has a response by its tool calls. + + What a tool's `RunContext.messages` and a chat span's input show: the core's history (`all_messages()`) + records a response only once it is over. + """ return [*self._seeded, *self._history] def new_messages(self) -> list[ModelMessage]: """A snapshot of the messages created during this session (excluding the seeded history).""" + if self._core is not None: + return self._core.new_messages() return list(self._history) @property @@ -1758,6 +1790,8 @@ class RealtimeSession: # its own messages. The seeded history is the caller's, and is left as it was given. for message in self._history: message.conversation_id = message.conversation_id or self._conversation_id + if self._core is not None: + self._core.set_conversation_id(self._conversation_id) # The session span was opened before this id existed too, and has to agree for the # session to correlate with the text run that continues it. self._session_instrumentation.set_conversation_id(self._conversation_id) @@ -1994,9 +2028,8 @@ class RealtimeSession: def _evict_image(self, request: ModelRequest) -> None: """Drop the oldest retained image from history, to keep within `retain_images_max`.""" self._remove_sent_request(request) - if self._core is not None: - evicted = [index for index, sent in self._input_requests.items() if sent is request] - self._core.apply(InputWithdrawn(input_ids=tuple(evicted))) + evicted = [index for index, sent in self._input_requests.items() if sent is request] + self._apply_core(InputWithdrawn(input_ids=tuple(evicted))) def _remove_sent_request(self, request: ModelRequest) -> None: """Remove a recorded request: a failed network send takes it back, the image cap evicts it.""" @@ -2041,20 +2074,7 @@ class RealtimeSession: user_turn_was_active = self._user_turn_active audio_was_uncommitted = self._audio_uncommitted self._held_commit.has_audio = True - # Without input transcription, a provider that reports speech boundaries opens each turn itself, - # at speech start. Audio alone is then no turn: an always-on microphone streams silence between - # utterances, and taking it for one would record a phantom turn per response and one at close. - audio_opens_turn = self._input_transcription_enabled or not self._provider_segments_input - if audio_opens_turn and not self._anonymous_user_turn_awaiting_answer: - if (turn := self._user_turns.get(None)) is not None and turn.speech_ended: - for event in self._finalize_user(): - self._publish_taps(event) - self._queue_put(event) - if not self._user_turn_active: - # Audio starting is the earliest sign of a user turn, and the only one on a provider that - # reports no speech boundaries, so it's where the turn's place in history is reserved. - self._open_user_turn_anchor() - self._user_turn_active = True + self._open_user_turn_for_audio() previous_length: int | None = None if self._retain_input: # Buffer the raw input so the finalized user turn can retain it. A per-item speech-stopped @@ -2071,8 +2091,6 @@ class RealtimeSession: self._input_audio.extend(data) self._audio_uncommitted = True await self._send_frame(BinaryAudio(data=data, media_type='audio/pcm')) - if self._core is not None: - self._core.apply(AudioSent(data=bytes(data))) except BaseException as e: self._user_turn_active = user_turn_was_active self._audio_uncommitted = audio_was_uncommitted @@ -2093,6 +2111,23 @@ class RealtimeSession: if self._retain_input: self._bound_retained_audio() + def _open_user_turn_for_audio(self) -> None: + """Audio is going out: it opens a user turn, if audio opens turns on this connection and none is open.""" + # Without input transcription, a provider that reports speech boundaries opens each turn itself, + # at speech start. Audio alone is then no turn: an always-on microphone streams silence between + # utterances, and taking it for one would record a phantom turn per response and one at close. + audio_opens_turn = self._input_transcription_enabled or not self._provider_segments_input + if audio_opens_turn and not self._anonymous_user_turn_awaiting_answer: + if (turn := self._user_turns.get(None)) is not None and turn.speech_ended: + for event in self._finalize_user(): + self._publish_taps(event) + self._queue_put(event) + if not self._user_turn_active: + # Audio starting is the earliest sign of a user turn, and the only one on a provider that + # reports no speech boundaries, so it's where the turn's place in history is reserved. + self._open_user_turn_anchor() + self._user_turn_active = True + async def commit_audio(self) -> None: """Commit buffered input audio as a user turn (manual turn-taking / push-to-talk).""" self._ensure_can_send() @@ -2113,6 +2148,7 @@ class RealtimeSession: self._user_turn_active = True for event in self._finalize_untranscribed_user(): self._queue_put(event) + self._drop_audio_the_core_keeps() async def clear_audio(self) -> None: """Discard buffered, uncommitted input audio.""" @@ -2120,8 +2156,7 @@ class RealtimeSession: self._require_media_ownership('clear_audio') self._require_capability('supports_manual_turn_control', method='clear_audio', feature='manual turn-taking') await self._send_frame(ClearAudio()) - if self._core is not None: - self._core.apply(AudioCleared()) + self._apply_core(AudioCleared()) self._audio_uncommitted = False # Drop the locally retained copy too (with `audio_retention='input_audio'`/`'all'`), or the discarded # audio would still be attached to the next finalized user turn. @@ -2215,8 +2250,7 @@ class RealtimeSession: # Only a response still being generated records the position: a finished one keeps what history # recorded, and the position must not land on the next one instead. self._pending_interrupted_at_ms = played_ms if self._response_in_flight else None - if self._core is not None: - self._core.apply(Interrupted(played_ms=played_ms)) + self._apply_core(Interrupted(played_ms=played_ms)) # Mark the barge-in in the trace. When the caller supplied `played_ms` (the ms of output audio # actually played before truncating), record it so a reader can see how far the response got before # the user cut in; it's dropped when absent (a cancel without truncation). @@ -2278,8 +2312,7 @@ class RealtimeSession: # Only a response still being generated can record the position: history is append-only, so a # finished one keeps what it recorded, and the position must not land on the next one instead. self._pending_interrupted_at_ms = played_ms if self._response_in_flight else None - if self._core is not None: - self._core.apply(Interrupted(played_ms=played_ms)) + self._apply_core(Interrupted(played_ms=played_ms)) self._session_instrumentation.record_lifecycle('interrupt', played_ms=played_ms) return True @@ -2375,6 +2408,7 @@ class RealtimeSession: # which is a caller that asked to send. async with self._send_lock: first_input = self._inputs_sent + audio_sent_to_core = False try: for position, content in enumerate(contents): # Numbered before the call, and whether or not it raises, matching how @@ -2384,20 +2418,25 @@ class RealtimeSession: self._inputs_sent += 1 if position == 0 and request is not None: self._input_requests[input_index] = request - if self._core is not None: - self._core.apply( - InputSent( - input_id=input_index, - request=request if position == 0 else None, - solicits=reply_asked and isinstance(content, (str, CreateResponse, ToolResult)), - tool_call_id=content.tool_call_id if isinstance(content, ToolResult) else None, - ) + self._apply_core( + InputSent( + input_id=input_index, + request=request if position == 0 else None, + solicits=reply_asked and isinstance(content, (str, CreateResponse, ToolResult)), + tool_call_id=content.tool_call_id if isinstance(content, ToolResult) else None, ) + ) + if isinstance(content, BinaryAudio): + # Under the lock, so the core takes audio in the order it goes out and a failed send + # takes back its own chunk; before the send, as the pump can end the turn meanwhile. + self._apply_core(AudioSent(data=bytes(content.data))) + audio_sent_to_core = True await self._connection.send(content) except BaseException as e: - if self._core is not None: - # The caller takes back what it sent, so the shadow core does too. - self._core.apply(InputWithdrawn(input_ids=tuple(range(first_input, self._inputs_sent)))) + # The caller takes back what it sent, so the core does too. + self._apply_core(InputWithdrawn(input_ids=tuple(range(first_input, self._inputs_sent)))) + if audio_sent_to_core: + self._apply_core(AudioUnsent()) if not isinstance(e, self._connection.transport_errors): raise # A send that fails because the link is gone is the same failure the receive side @@ -2447,8 +2486,7 @@ class RealtimeSession: """Park a background failure for iteration or close, ending receive-only views if nobody is iterating.""" self._parked_errors.append(error) self._queue_put(error) - if self._core is not None: - self._core.apply(ExchangeAbandoned()) + self._apply_core(ExchangeAbandoned()) if not self._response_limit_checked or any(isinstance(part, ToolCallPart) for part in self._response_parts): # A tool that raised, or a usage limit tripped by the request its result would make, stops # the model from getting that result, so the exchange waiting on it is over: a caller in @@ -2590,6 +2628,20 @@ class RealtimeSession: self._response_parts.append(part) return [PartEndEvent(index=index, part=part)] + def _count_recorded_response(self, response: ModelResponse) -> None: + """Price a response this session records, and count it as a request unless usage reports them.""" + provider_reported_cost = response.usage.cost + fill_response_cost(response) + requests = int(self._responses_are_requests) + self._requests_recorded += requests + if self._core is not None: + return # The core counts it, into this same `RunUsage`. + if provider_reported_cost is None: + # Tokens were added as `SessionUsage` events arrived; only add the price calculated + # at this response boundary. A provider-reported cost arrived in those events too. + self.usage.incr(RequestUsage(cost=response.usage.cost)) # usage-attribution: the session owns its spans + self.usage.requests += requests # usage-attribution: the session owns its spans; `wrap_run` opens none + def _finalize_response( self, *, @@ -2603,7 +2655,7 @@ class RealtimeSession: """Finalize the current assistant response's parts into a `ModelResponse` in history.""" response: ModelResponse | None = None # The chat span's input is the history the response replied to, captured before we append it. - input_messages = self.all_messages() + input_messages = self._own_messages() # Native tool parts (web grounding / code execution) lead the response (call+return, then # speech), matching the classic `GoogleModel`, which prepends them ahead of the assistant's text. parts = [*self._native_tool_parts, *self._response_parts] @@ -2668,21 +2720,13 @@ class RealtimeSession: state='interrupted' if interrupted else 'complete', ) fill_run_metadata(response, run_id=self._run_id, conversation_id=self._conversation_id) - provider_reported_cost = response.usage.cost - fill_response_cost(response) - if provider_reported_cost is None: - # Tokens were added as `SessionUsage` events arrived; only add the price calculated - # at this response boundary. A provider-reported cost arrived in those events too. - self.usage.incr(RequestUsage(cost=response.usage.cost)) # usage-attribution: the session owns its spans + self._count_recorded_response(response) self._history.append(response) self._resolve_in_flight_user_turn_anchors(response) if not any(isinstance(part, ToolCallPart) for part in parts): # The model has said what it had to say (a tool call means its answer is still to come), # so audio from here on can be the user's next turn again. self._anonymous_user_turn_awaiting_answer = False - # Counted here unless the profile says the requests are reported with usage instead. - requests = int(self._responses_are_requests) - self.usage.requests += requests # usage-attribution: the session owns its spans; `wrap_run` opens none self._tool_run_step += 1 self._close_tool_batch() for part in parts: @@ -2699,6 +2743,7 @@ class RealtimeSession: self._pending_sent_requests = [] self._stop_held_user_turn_watchdog() self._session_instrumentation.end_chat_span(input_messages, response) + self._drop_audio_the_core_keeps() self._response_parts = [] self._native_tool_parts = [] self._pending_response_usage = RequestUsage() @@ -2718,6 +2763,7 @@ class RealtimeSession: self._remove_sent_request(request) self._history.append(request) self._held_user_turns = [] + self._drop_audio_the_core_keeps() def _stop_held_user_turn_watchdog(self) -> None: if self._held_user_turn_watchdog is not None: @@ -2921,8 +2967,7 @@ class RealtimeSession: request_parts.append(UserPromptPart(content=content)) request = self._new_request(request_parts) self._insert_tool_return(call_part, request) - if self._core is not None: - self._core.apply(ToolReturned(tool_call_id=call_part.tool_call_id, request=request)) + self._apply_core(ToolReturned(tool_call_id=call_part.tool_call_id, request=request)) return [FunctionToolResultEvent(part=result_part, content=content)] def _insert_tool_return(self, call_part: ToolCallPart, request: ModelRequest) -> None: @@ -3275,6 +3320,24 @@ class RealtimeSession: excess = budget.trim(self._input_audio, excess, output=False) budget.trim(self._output_audio, excess, output=True) + def _drop_audio_the_core_keeps(self) -> None: + """Let go of the retained audio this session's own record holds, when the core keeps history. + + The core's history is the one `all_messages()` returns, with retained audio bounded by + `retain_audio_max_seconds`; this session's own record (what tools see as `RunContext.messages`, and chat + spans as their input) keeps only the transcripts, so the audio isn't held twice. The parts streamed as + events, and the responses chat spans record, are built before this and keep their audio. + """ + if self._core is None or not self._audio_budget.tracked_parts: + return + index = self._audio_eviction_start + while index < len(self._history): + stripped, freed = self._audio_budget.strip(self._history[index]) + if freed: + self._replace_recorded_message(index, stripped) + index += 1 + self._audio_eviction_start = index + def _replace_recorded_message(self, index: int, message: ModelMessage) -> None: """Swap a message in history for an updated copy, carrying over the user-turn anchors that point at it. @@ -3668,7 +3731,7 @@ class RealtimeSession: # nothing) no matter how long the call has been going. Update in place, so contexts # already handed out — `replace()` below keeps the same list object — see the update # too. - ctx.messages[:] = self.all_messages() + ctx.messages[:] = self._own_messages() # A run step here is one model turn: `_tool_run_step` increments when a # `ModelResponse` is finalized, so this re-prepares the *local* manager once per # turn — refreshing prepare hooks, availability filters, and retry state exactly @@ -3874,10 +3937,20 @@ class RealtimeSession: """Check `request_limit` against one more response than those finalized, reserved and in flight.""" if self._usage_limits is not None and self._responses_are_requests: in_flight = 1 if self._response_limit_checked else 0 - projected = dataclasses.replace( - self.usage, requests=self.usage.requests + self._pending_response_requests + in_flight + self._usage_limits.check_before_request( + self._usage_as_recorded(ahead=self._pending_response_requests + in_flight) ) - self._usage_limits.check_before_request(projected) + + def _usage_as_recorded(self, *, ahead: int = 0) -> RunUsage: + """`usage`, with its requests counted as this session records them, plus `ahead` more. + + The core counts a response into `usage.requests` as its terminal frame arrives, a moment before this + session records it and releases what it reserved for it: counted by this session's clock instead, no + request is counted twice in between. + """ + counted_by_core = self._core.requests if self._core is not None else self._requests_recorded + requests = self.usage.requests - counted_by_core + self._requests_recorded + ahead + return dataclasses.replace(self.usage, requests=requests) def _begin_response(self) -> None: """Take the reservation for the response that's starting, or make the check now if it has none. @@ -3891,7 +3964,7 @@ class RealtimeSession: if self._pending_response_requests: self._pending_response_requests -= 1 elif self._usage_limits is not None and self._responses_are_requests: - self._usage_limits.check_before_request(self.usage) + self._usage_limits.check_before_request(self._usage_as_recorded()) self._response_limit_checked = True self._response_active = True # A response that is starting is not the one whose terminal is still to come. Gemini finalizes a @@ -3923,11 +3996,13 @@ class RealtimeSession: self._reported_context_window_used = event.context_window_used if event.response_scoped: self._begin_response() - if not self._responses_are_requests: + if not self._responses_are_requests and self._core is None: # Each report is a request the model has already made: it is recorded in full, and the - # one past the limit ends the session below, once it is. + # one past the limit ends the session below, once it is. (The core counts it otherwise.) self.usage.requests += 1 # usage-attribution: the session owns its spans; `wrap_run` opens none - self.usage.incr(event.usage) # usage-attribution: the session owns its spans; `wrap_run` opens none + if self._core is None: + # (The core counts it otherwise, into this same `RunUsage`.) + self.usage.incr(event.usage) # usage-attribution: the session owns its spans; `wrap_run` opens none self._span_usage.incr(event.usage) # usage-attribution: what the session span reports if event.response_scoped: # Measured before accumulating: a tool-call response is finalized by the accumulation @@ -4056,8 +4131,7 @@ class RealtimeSession: await self._dispatch_tool_call(event) except UsageLimitExceeded: # Tripped before the call joined the response, so it never runs or reaches history. - if self._core is not None: - self._core.apply(ToolCallRefused(tool_call_id=event.tool_call_id)) + self._apply_core(ToolCallRefused(tool_call_id=event.tool_call_id)) raise return False return await self._handle_non_tool_pump_event(event) @@ -4222,7 +4296,7 @@ class RealtimeSession: """Drain the connection into the session queue under the explicit session-span context.""" token = otel_context.attach(context) if context is not None else None try: - events = self._connection if self._core is None else self._shadowed_events(self._core) + events = self._connection if self._core is None else self._core_events(self._core) async for event in events: if merged := self._connection._take_merged_response_requests(): # pyright: ignore[reportPrivateUsage] # Requests the connection answered with a response it was already asking for: none of @@ -4232,12 +4306,12 @@ class RealtimeSession: if await self._handle_pump_event(event): return # a usage limit tripped: stop reading the upstream self._count_closed_tool_batch_replies() + self._drop_audio_the_core_keeps() except Exception as e: self._pump_error = e finally: self._pump_finished = True - if self._core is not None: - self._core.apply(ReceiveEnded()) + self._apply_core(ReceiveEnded()) self._exchange_progress.set() if not self._closed: self._finish_taps() @@ -4245,36 +4319,41 @@ class RealtimeSession: if token is not None: otel_context.detach(token) - async def _shadowed_events(self, core: SessionCore) -> AsyncIterator[RealtimeCodecEvent]: - """The connection's codec events, feeding the shadow core its lifecycle stream on the way.""" + def _apply_core(self, item: CoreInput) -> None: + """Hand the core an event or a command, if it runs, and wake whoever waits on what it owes.""" + if self._core is None: + return + self._core.apply(item) + if isinstance(item, AudioSent): + # A microphone chunk: it neither settles anything owed nor changes what history holds back. + return + self._core_advanced(self._core) + + def _core_advanced(self, core: SessionCore) -> None: + """Wake whoever waits on what the core owes, and bound how long a missing transcript holds history back.""" + self._exchange_progress.set() + if (turn_id := core.transcript_holding_history()) is not None and turn_id not in self._transcript_watchdogs: + self._transcript_watchdogs[turn_id] = asyncio.get_running_loop().call_later( + _TRANSCRIPT_HOLD_SECONDS, self._transcript_overdue, turn_id + ) + + def _transcript_overdue(self, turn_id: str) -> None: + del self._transcript_watchdogs[turn_id] + self._apply_core(TranscriptOverdue(turn_id=turn_id)) + + async def _core_events(self, core: SessionCore) -> AsyncIterator[RealtimeCodecEvent]: + """The connection's codec events, feeding the core its lifecycle stream on the way.""" async for frame in self._connection._tagged_frames(): # pyright: ignore[reportPrivateUsage] # pragma: no branch # Applied a whole frame at a time, before this session handles any of it: a consumer reacting - # to one of its events (and closing, say) must find the shadow core as far along as this one. + # to one of its events (and reading the history, say) must find the core as far along as this one. for event, stale in frame: if not stale: core.apply(event) + self._core_advanced(core) for event, _ in frame: if not isinstance(event, LIFECYCLE_EVENT_TYPES): yield event - def _compare_shadow(self, core: SessionCore) -> None: - """Record where the shadow core's history or usage disagrees with this session's.""" - legacy, shadow = _comparable(self.all_messages()), _comparable(core.all_messages()) - if legacy != shadow: - diff = difflib.unified_diff( - pformat(legacy, width=120).splitlines(), - pformat(shadow, width=120).splitlines(), - 'legacy', - 'shadow', - lineterm='', - ) - self._shadow_divergences.append('history differs:\n' + '\n'.join(diff)) - # Everything but the tool calls, which the session counts as it runs them, not the core. - for usage_field in dataclasses.fields(self.usage): - name = usage_field.name - if name != 'tool_calls' and (theirs := getattr(core.usage, name)) != (ours := getattr(self.usage, name)): - self._shadow_divergences.append(f'usage.{name} differs: legacy {ours}, shadow {theirs}') - def _ensure_streamable(self) -> None: if not self._entered: raise UserError('Enter the realtime session with `async with` before streaming it.') diff --git a/tests/conftest.py b/tests/conftest.py index ac0f1e1f47..da77b19aa5 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -109,10 +109,6 @@ def pytest_configure(config: pytest.Config) -> None: 'markers', 'realtime_ws_hold_open: keep a replay WebSocket open after its last recorded frame', ) - config.addinivalue_line( - 'markers', - 'shadow_divergence(reason): the realtime session cores are known to disagree on this trace; reason required', - ) if TYPE_CHECKING: diff --git a/tests/realtime/conftest.py b/tests/realtime/conftest.py index a14e75b608..c514805026 100644 --- a/tests/realtime/conftest.py +++ b/tests/realtime/conftest.py @@ -13,9 +13,6 @@ from typing import TYPE_CHECKING, Any, cast import pytest from cassetter import RawRequest, RawResponse -from pydantic_ai.realtime import _session as realtime_session # pyright: ignore[reportPrivateUsage] -from pydantic_ai.realtime._core import SessionCore - from .. import cassette_hooks from ..conftest import sanitize_filename, try_import from .ws_cassettes import ProviderName, RealtimeCassette, patched_ws_connect, realtime_cassette_plan @@ -166,31 +163,6 @@ def vcr_config(vcr_config: dict[str, Any]) -> dict[str, Any]: } -@pytest.fixture(autouse=True) -def _shadow_core(request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch) -> Iterator[None]: - """Run the new session core in shadow of the current one, and fail on anything the two disagree about. - - Every session over a connection that identifies its responses, turns, and inputs (the OpenAI protocol) - builds its history twice; they must match when it closes. A test whose trace the current core gets - wrong in a way the new one fixes says so with `@pytest.mark.shadow_divergence(reason)`. - """ - monkeypatch.setattr(realtime_session, '_CORE_MODE', 'shadow') - divergences: list[str] = [] - compare = realtime_session.RealtimeSession._compare_shadow # pyright: ignore[reportPrivateUsage] - - def recorded(session: realtime_session.RealtimeSession, core: SessionCore) -> None: - compare(session, core) - divergences.extend(session._shadow_divergences) # pyright: ignore[reportPrivateUsage] - - monkeypatch.setattr(realtime_session.RealtimeSession, '_compare_shadow', recorded) - yield - node = cast('pytest.Item', request.node) # pyright: ignore[reportUnknownMemberType] - if node.get_closest_marker('shadow_divergence') is None: - assert not divergences, '\n'.join(divergences) - else: - assert divergences, 'the session cores agree on this trace now: drop its `shadow_divergence` mark' - - @pytest.fixture(autouse=True) def _realtime_api_keys(monkeypatch: pytest.MonkeyPatch) -> None: """Provide placeholder API keys so realtime models can resolve their default providers offline. diff --git a/tests/realtime/simulation/_conformance.py b/tests/realtime/simulation/_conformance.py index eae7257aae..674f1631cf 100644 --- a/tests/realtime/simulation/_conformance.py +++ b/tests/realtime/simulation/_conformance.py @@ -48,6 +48,7 @@ from pydantic_ai.realtime._lifecycle import ( LIFECYCLE_EVENT_TYPES, InputAdded, LifecycleEvent, + OutputItemDetails, ResponseEnded, ResponseStarted, UserTurnDiscarded, @@ -169,7 +170,22 @@ class LifecycleChecker: issue('lifecycle.end_without_start', f'{event.response_id!r} ended without starting') self._open_responses.discard(event.response_id) self._responses_ended.add(event.response_id) - elif isinstance(event, UserTurnStarted): + elif isinstance(event, (UserTurnStarted, UserTurnEnded, UserTurnDiscarded)): + self._feed_turn(event, issue) + elif isinstance(event, InputAdded): + if event.input_id in self._added_inputs: + issue('lifecycle.input_added_twice', f'input {event.input_id} joined the conversation twice') + self._added_inputs.add(event.input_id) + elif isinstance(event, OutputItemDetails): + if event.response_id not in self._open_responses: + issue('lifecycle.content_outside_response', f'output item details for {event.response_id!r} outside it') + else: + self._settle(event.input_ids, issue) + + def _feed_turn( + self, event: UserTurnStarted | UserTurnEnded | UserTurnDiscarded, issue: Callable[[str, str], None] + ) -> None: + if isinstance(event, UserTurnStarted): if event.turn_id in self._turns: issue('lifecycle.turn_started_twice', f'spoken turn {event.turn_id!r} started twice') self._turns.add(event.turn_id) @@ -179,18 +195,12 @@ class LifecycleChecker: issue('lifecycle.turn_end_without_start', f'spoken turn {event.turn_id!r} ended without starting') self._open_turns.discard(event.turn_id) self._joined_turns.add(event.turn_id) - elif isinstance(event, UserTurnDiscarded): + else: # A turn can be discarded once it joined, too: it just gets no more audio. if event.turn_id not in self._open_turns and event.turn_id not in self._joined_turns: issue('lifecycle.turn_end_without_start', f'spoken turn {event.turn_id!r} discarded without starting') self._open_turns.discard(event.turn_id) self._joined_turns.discard(event.turn_id) - elif isinstance(event, InputAdded): - if event.input_id in self._added_inputs: - issue('lifecycle.input_added_twice', f'input {event.input_id} joined the conversation twice') - self._added_inputs.add(event.input_id) - else: - self._settle(event.input_ids, issue) def _settle(self, input_ids: tuple[int, ...], issue: Callable[[str, str], None]) -> None: # Checked one at a time, so an event naming an input twice settles it twice too. diff --git a/tests/realtime/simulation/_findings.py b/tests/realtime/simulation/_findings.py index 80a4091dd8..697d6158ec 100644 --- a/tests/realtime/simulation/_findings.py +++ b/tests/realtime/simulation/_findings.py @@ -76,24 +76,9 @@ def _inserted_user_speech(sim: Simulation, violation: InvariantViolation) -> boo ALL = frozenset({'openai', 'azure', 'xai', 'gemini', 'gpt-live'}) OPENAI_PROTOCOL = frozenset({'openai', 'azure', 'xai'}) - - -def _tool_results_request_refused(sim: Simulation, violation: InvariantViolation) -> bool: - return any(input_.kind == 'tool_output' and input_.refused_read is not None for input_ in sim.truth.inputs) - - -REFUSED_TOOL_RESULTS_REQUEST = Finding( - id='SIM-12', - title=( - 'a response request for tool results that the provider refuses keeps its reply reservation (a refused request ' - 'for a user turn releases it), so `wait_for_reply()` hangs' - ), - tracked_by='reply reservations released with a refused request (#8765 did not cover it); found by this simulator', - evidence='simulated', - codes=frozenset({'wait.hang'}), - providers=OPENAI_PROTOCOL, - matches=_tool_results_request_refused, -) +LEGACY_CORE = ALL - OPENAI_PROTOCOL +"""The providers whose sessions still run the current session core: the new one (`_core.py`) fixed the finding for +the others.""" def _late_cancel(sim: Simulation, violation: InvariantViolation) -> bool: @@ -122,7 +107,7 @@ ANCHORED_USER_TURNS = Finding( tracked_by='history projected from an append-only event log, so snapshots are prefixes of later ones', evidence='live-stress', codes=frozenset({'history.inserted'}), - providers=ALL, + providers=LEGACY_CORE, matches=_inserted_user_speech, ) @@ -188,22 +173,6 @@ LOST_RESPONSE_RESERVATION = Finding( ), ) -LOST_UNSTARTED_REQUEST = Finding( - id='SIM-3', - title=( - 'a reconnect re-asks for the requests deferred behind a lost, not-yet-started response, but not for that ' - 'response itself, so its reservation leaks and `wait_for_reply()` hangs' - ), - tracked_by='an outbox that replays every unanswered request after a reconnect; found by this simulator', - evidence='simulated', - codes=frozenset({'wait.hang'}), - providers=OPENAI_PROTOCOL, - matches=lambda sim, violation: ( - getattr(sim, 'deferred_requests', 0) > 0 - and any(response.lost and response.started_read is None for response in sim.truth.responses.values()) - ), -) - def _receive_loop_send_failed(sim: Simulation, violation: InvariantViolation) -> bool: network = getattr(getattr(sim, 'server', None), 'network', None) @@ -262,25 +231,20 @@ SENT_BEFORE_REPLY_STARTED = Finding( ), evidence='recorded', codes=frozenset({'history.order'}), - providers=ALL, + providers=LEGACY_CORE, matches=_sent_before_reply_content, ) def _spoken_before_reply(sim: Simulation, violation: InvariantViolation) -> bool: - """A spoken turn committed after a response ended, which the user started before that response was over. - - Either server VAD heard them start before it ended, or their voiced audio started streaming before any of its - content arrived. - """ + """A spoken turn committed after a response ended, which the user started before that response was over: their + voiced audio started streaming before any of its content arrived.""" input_ = sim.truth.input(violation.context.get('input', '')) response = sim.truth.responses.get(violation.context.get('response', '')) if input_ is None or response is None or input_.kind != 'speech' or response.seq_end is None: return False - if input_.seq <= response.seq_end: + if input_.seq <= response.seq_end: # pragma: lax no cover (the other way round) return False - if (started := sim.truth.speech_started.get(input_.key)) is not None and started <= response.seq_end: - return True # Server VAD heard the user start before the response ended. content_read, ended, cancelled = response.content_read, response.seq_end, response.status == 'cancelled' @@ -317,34 +281,11 @@ SPEAKING_ORDER = Finding( ), evidence='recorded', codes=frozenset({'history.order'}), - providers=ALL, + providers=LEGACY_CORE, matches=_spoken_before_reply, ) -def _committed_by_hand_under_server_vad(sim: Simulation, violation: InvariantViolation) -> bool: - """A spoken turn that reached the server before a response started, with a manual commit in the trace.""" - input_ = sim.truth.input(violation.context.get('input', '')) - response = sim.truth.responses.get(violation.context.get('response', '')) - if input_ is None or response is None or input_.kind != 'speech': - return False - return input_.seq < response.seq_start and any(operation.name == 'commit_audio' for operation in sim.operations) - - -COMMITTED_BY_HAND_UNDER_SERVER_VAD = Finding( - id='OR9', - title=( - 'a spoken turn committed by hand while server VAD is on is filed after the reply to a turn VAD committed ' - 'later (the rest of OR9: push-to-talk, barge-in, and transcription off were fixed by #8764)' - ), - tracked_by='#8764 (follow-up)', - evidence='live-stress', - codes=frozenset({'history.order'}), - providers=OPENAI_PROTOCOL, - matches=_committed_by_hand_under_server_vad, -) - - def _waited_before_reply_content(sim: Simulation, violation: InvariantViolation) -> bool: """The wait began after the client read that a response started, but before any of its content.""" response = sim.truth.responses.get(violation.context.get('response', '')) @@ -363,7 +304,7 @@ WAIT_BEFORE_REPLY_CONTENT = Finding( tracked_by='an explicit response-started event from the adapters opens the exchange; found by this simulator', evidence='recorded', codes=frozenset({'wait.early'}), - providers=ALL, + providers=LEGACY_CORE, matches=_waited_before_reply_content, ) @@ -421,14 +362,11 @@ RESERVATION_TAKEN_BY_OTHER_RESPONSE = Finding( KNOWN_FINDINGS: list[Finding] = [ WAIT_BEFORE_REPLY_CONTENT, RESERVATION_TAKEN_BY_OTHER_RESPONSE, - REFUSED_TOOL_RESULTS_REQUEST, ANCHORED_USER_TURNS, REPEATED_TERMINAL, LATE_CANCEL_DROPS_CONTENT, SENT_BEFORE_REPLY_STARTED, SPEAKING_ORDER, - COMMITTED_BY_HAND_UNDER_SERVER_VAD, - LOST_UNSTARTED_REQUEST, RECEIVE_LOOP_SEND_FAILURE, ] """Checked in order: the more specific findings for a code come before the more general ones.""" @@ -637,32 +575,6 @@ LIVE_RAW_CLOSE_ERROR = Finding( ) -def _refused_while_speaking(sim: Simulation, violation: InvariantViolation) -> bool: - input_ = sim.truth.input(violation.context.get('input', '')) - return ( - input_ is not None - and input_.kind == 'speech' - and any(other.rejected and other.refused_read is not None for other in sim.truth.inputs) - ) - - -REFUSED_CONTEXT_MISFILES_SPEECH = Finding( - id='SIM-14', - title=( - 'context sent while the user is speaking (before the session read `speech_started`) and refused by the ' - 'provider takes the spoken turn with it: the turn is filed after its own reply' - ), - tracked_by=( - 'history ordered by what the provider saw, not by the send a turn was anchored to: the session falls back ' - 'to appending a turn whose anchor was withdrawn; found by this simulator' - ), - evidence='simulated', - codes=frozenset({'history.order'}), - providers=OPENAI_PROTOCOL, - matches=_refused_while_speaking, -) - - def _parked_error(sim: Simulation, violation: InvariantViolation) -> bool: from pydantic_ai.exceptions import UnexpectedModelBehavior, UsageLimitExceeded @@ -688,28 +600,6 @@ PARKED_ERROR_LEAVES_REQUEST_OWED = Finding( ) -def _speech_cleared_after_barge_in(sim: Simulation, violation: InvariantViolation) -> bool: - started = sim.truth.speech_started - return any(key not in sim.truth.speech_committed for key in started) and any( - operation.name == 'clear_audio' for operation in sim.operations - ) - - -CLEARED_BARGE_IN = Finding( - id='SIM-16', - title=( - 'a request deferred behind a response that server VAD then cut off is dropped for the barge-in, but if ' - 'the app clears the buffered speech (`clear_audio`) no spoken turn follows, and the dropped request keeps ' - 'its reservation, so `wait_for_reply()` hangs' - ), - tracked_by='reply reservations released with the request the barge-in dropped; found by this simulator after #8765', - evidence='recorded', - codes=frozenset({'wait.hang'}), - providers=OPENAI_PROTOCOL, - matches=_speech_cleared_after_barge_in, -) - - def _pair_answered_with_the_turn_that_cut_it_off(sim: Simulation) -> bool: """Two calls whose turn a user turn (typed or spoken) cut off, answered by one response together with that turn.""" truth = sim.truth @@ -777,46 +667,33 @@ EXTENDED_THINKING_PARALLEL_CALLS = Finding( ) -BARGE_IN_ON_A_TOOL_ROUND = Finding( - id='SIM-20', - title=( - 'server VAD cuts off a response that called a tool while a request is deferred behind it: the request is ' - 'dropped for the barge-in but keeps its reservation, so `wait_for_reply()` hangs (the part of OR3 #8765 left)' - ), - tracked_by='reply reservations released with the request the barge-in dropped; found by this simulator', - evidence='recorded', - codes=frozenset({'wait.hang'}), - providers=OPENAI_PROTOCOL, - matches=lambda sim, violation: ( - getattr(sim, 'deferred_requests', 0) > 0 - and any(response.tool_calls and response.status == 'cancelled' for response in sim.truth.responses.values()) - ), -) - - def _refusal_around_a_reconnect(sim: Simulation) -> bool: """A refusal the next connection loss followed with no response started in between: nothing released its request.""" truth = sim.truth - for input_ in truth.inputs: - if input_.refused_at is None: - continue - refused = input_.refused_at + + def unreleased(refused: int) -> bool: loss = next((loss for loss in truth.connection_losses if loss > refused), None) - if loss is not None and not any(refused < response.seq_start < loss for response in truth.responses.values()): - return True - return False + return loss is not None and not any( + refused < response.seq_start < loss for response in truth.responses.values() + ) + + return any(input_.refused_at is not None and unreleased(input_.refused_at) for input_ in truth.inputs) LOST_REFUSAL = Finding( id='SIM-21', title=( 'a request for a response the provider refuses around a reconnect (the refusal lost with the connection, or ' - 'read just before it drops) leaves a reservation neither re-asked nor released, so `wait_for_reply()` hangs' + 'read just before it drops) leaves a reservation neither re-asked nor released, so `wait_for_reply()` hangs: ' + 'xAI resumes the conversation, so the connection still expects the reply there' + ), + tracked_by=( + 'a reconnect resolving the reply obligations its connection lost, also where the provider resumes the ' + 'conversation (fixed by the session core elsewhere); found by this simulator' ), - tracked_by='a reconnect resolves the reply obligations its connection lost; found by this simulator', evidence='simulated', codes=frozenset({'wait.hang'}), - providers=OPENAI_PROTOCOL, + providers=frozenset({'xai'}), matches=lambda sim, violation: _refusal_around_a_reconnect(sim), ) @@ -848,39 +725,6 @@ TERMINAL_DISCARDED_WITH_THE_CONNECTION = Finding( ) -def _barged_in_without_a_vad_reply(sim: Simulation) -> bool: - """Server VAD, set not to answer, cut off a response while a request was deferred behind it (or a tool result awaited its reply).""" - options = getattr(sim, 'openai', None) - return ( - options is not None - and options.turn_detection == 'server_vad' - and options.vad_interrupts - and not options.vad_responds - and ( - getattr(sim, 'deferred_requests', 0) > 0 - or any(input_.kind == 'tool_output' and input_.answered_by is None for input_ in sim.truth.inputs) - ) - and any(response.status == 'cancelled' for response in sim.truth.responses.values()) - # The user's turn is still there for VAD to answer (a cleared one is SIM-16's). - and any(sim.truth.input(key) is not None for key in sim.truth.speech_started) - ) - - -BARGE_IN_WITHOUT_A_VAD_REPLY = Finding( - id='SIM-32', - title=( - "with server VAD's `create_response` off, a request deferred behind a response the user barged in on (or the " - "reply to that response's tool results) is dropped as if VAD would answer the turn, but nothing does, so " - '`wait_for_reply()` hangs' - ), - tracked_by='a request dropped for a barge-in only when server VAD answers the turn; found by this simulator', - evidence='simulated', - codes=frozenset({'wait.hang'}), - providers=frozenset({'openai', 'azure'}), - matches=lambda sim, violation: _barged_in_without_a_vad_reply(sim), -) - - def _turn_without_its_transcript(sim: Simulation) -> bool: """With transcription on, a spoken turn's transcript never reached the client before the session ended or dropped.""" ended = sim.close_requested is not None or sim.receive_ended or bool(sim.truth.connection_losses) @@ -904,27 +748,6 @@ def _hand_commit_unanswered_at_the_end(sim: Simulation) -> bool: ) -def _server_vad(sim: Simulation) -> bool: - """Whether the simulated OpenAI-protocol session runs server VAD (xAI push-to-talk reports speech too).""" - options = getattr(sim, 'openai', None) - return options is not None and options.turn_detection == 'server_vad' - - -def _clear_after_an_unread_vad_commit(sim: Simulation) -> bool: - """Without transcription, the app cleared the buffer after server VAD committed a turn, before reading that it had.""" - if not _server_vad(sim) or getattr(sim, 'openai').transcription: - return False - truth = sim.truth - clears = [operation.issued for operation in sim.operations if operation.name == 'clear_audio'] - # Turns server VAD committed (not the app), and when the client read that it had. - vad_turns = [ - (started, input_.committed_read) - for key, started in truth.speech_started.items() - if key in truth.speech_committed and (input_ := truth.input(key)) is not None and not input_.committed_by_client - ] - return any(started < issued and (read is None or issued < read) for started, read in vad_turns for issued in clears) - - def _push_to_talk_audio_after_a_repeated_terminal(sim: Simulation) -> bool: options = getattr(sim, 'openai', None) return ( @@ -940,9 +763,8 @@ PUSH_TO_TALK_AUDIO_AFTER_A_REPEATED_TERMINAL = Finding( id='SIM-36', title=( 'on xAI push-to-talk, audio sent after a `response.done` the server repeated is held behind a reply that ' - 'already ended: the current session loses the turn, and the new core (in shadow) leaves `wait_for_reply()` ' - 'hanging, or holds back the reply after it (a repeated terminal is a robustness fault no provider was ' - 'recorded sending, see 8801)' + 'already ended: the session core leaves `wait_for_reply()` hanging, or holds back the reply after it (a ' + 'repeated terminal is a robustness fault no provider was recorded sending, see 8801)' ), tracked_by="#9070's held audio released by the response that ended, not by the next terminal; found by this simulator", evidence='simulated', @@ -952,27 +774,12 @@ PUSH_TO_TALK_AUDIO_AFTER_A_REPEATED_TERMINAL = Finding( ) -CLEAR_AFTER_AN_UNREAD_VAD_COMMIT = Finding( - id='SIM-35', - title=( - "without input transcription, a `clear_audio()` made after server VAD committed the user's turn, but before " - 'the session read that it had, drops the turn: the provider keeps it, and answers it, but history never ' - 'records it' - ), - tracked_by='user turns recorded from the provider committing them; found by this simulator', - evidence='simulated', - codes=frozenset({'history.turn_missing'}), - providers=OPENAI_PROTOCOL, - matches=lambda sim, violation: _clear_after_an_unread_vad_commit(sim), -) - - HAND_COMMIT_UNANSWERED_AT_THE_END = Finding( id='SIM-34', title=( 'a spoken turn `commit_audio()` committed is lost when the connection drops, or the session closes or ends (on ' - 'a failed send, an exceeded limit), before its reply is over: always in the new session core (in shadow), ' - 'and in the current session on xAI, whose held commit goes out with the request' + "a failed send, an exceeded limit), before its reply is over (the session core's own: the current one loses " + 'it only on xAI, whose held commit goes out with the request)' ), tracked_by='user turns recorded from the provider committing them; found by this simulator on #9422', evidence='simulated', @@ -982,107 +789,21 @@ HAND_COMMIT_UNANSWERED_AT_THE_END = Finding( ) -def _audio_after_a_clear(sim: Simulation) -> bool: - """Microphone audio was on its way at the same time as a clear or commit the app made (the two raced).""" - - def span(operation: Operation) -> tuple[int, float]: - return operation.issued, operation.completed if operation.completed is not None else float('inf') - - audio = [span(operation) for operation in sim.operations if operation.name == 'send_audio'] - buffer_ops = [span(operation) for operation in sim.operations if operation.name in ('clear_audio', 'commit_audio')] - return any(start < end_b and start_b < end for start, end in audio for start_b, end_b in buffer_ops) - - -AUDIO_AFTER_A_CLEAR = Finding( - id='SIM-26', - title=( - 'audio still streaming from the microphone when the app calls `clear_audio()` or `commit_audio()` lands after ' - 'it, and the turn a later `commit_audio()` commits on the provider is never recorded: the session thinks the ' - 'buffer is empty' - ), - tracked_by='user turns recorded from the provider committing them; found by this simulator', - evidence='simulated', - codes=frozenset({'history.turn_missing'}), - providers=OPENAI_PROTOCOL, - matches=lambda sim, violation: _audio_after_a_clear(sim), -) - - -def _hand_commit_under_server_vad(sim: Simulation, violation: InvariantViolation) -> bool: - if ( - not _server_vad(sim) - or not sim.truth.speech_started - or not any(operation.name == 'commit_audio' for operation in sim.operations) - ): - return False - if violation.code != 'history.order': - return True - input_ = sim.truth.input(violation.context.get('input', '')) - return input_ is not None and input_.committed_by_client - - -HAND_COMMIT_UNDER_SERVER_VAD = Finding( - id='SIM-27', - title=( - 'a `commit_audio()` while server VAD is hearing the user commits what was buffered as a turn of its own, and ' - 'VAD commits another when the user stops, but the session records only one of the two; on xAI, which adds ' - 'the VAD turn when it hears speech start, the hand-committed one is recorded where its audio began, before ' - 'a reply that ended before it was committed' - ), - tracked_by='user turns recorded from the provider committing them (the rest of OR9); found by this simulator', - evidence='simulated', - codes=frozenset({'history.turn_missing', 'history.order'}), - providers=OPENAI_PROTOCOL, - matches=lambda sim, violation: _hand_commit_under_server_vad(sim, violation), -) - - TURN_LOST_AT_CLOSE = Finding( id='SIM-25', title=( 'a spoken turn committed with input transcription on, whose transcript never arrives (the session closes or ' - 'ends on an error, or the connection drops, first), is never recorded: it waits for a transcript that never comes (surfaced by ' - 'Macroscope for closing after `commit_audio()`). The new session core (in shadow) also holds back the replies ' - 'after it, so a reply after a reconnect is missing too (and is counted as a request it does not record)' + 'ends on an error, or the connection drops, first), is never recorded: it waits for a transcript that never ' + 'comes (surfaced by Macroscope for closing after `commit_audio()`)' ), tracked_by='user turns recorded from the provider committing them, not from their transcript; found reviewing #9070', evidence='recorded', - codes=frozenset({'history.turn_missing', 'response.missing', 'usage.requests'}), + codes=frozenset({'history.turn_missing'}), providers=OPENAI_PROTOCOL, matches=lambda sim, violation: _turn_without_its_transcript(sim), ) -def _sent_during_a_lost_reply(sim: Simulation, violation: InvariantViolation) -> bool: - """Everything the re-dialed conversation lacks is a user input sent while a reply the drop cut off was in flight.""" - lost = [response for response in sim.truth.responses.values() if response.lost] - - def held_back(fingerprint: str) -> bool: - role, _, key = fingerprint.partition(':') - input_ = sim.truth.input(key) - return ( - role == 'user' - and input_ is not None - and any(response.connection == input_.connection and response.seq_start < input_.seq for response in lost) - ) - - return all(held_back(fingerprint) for fingerprint in violation.context['missing']) - - -INPUT_HELD_BEHIND_A_LOST_REPLY = Finding( - id='SIM-28', - title=( - 'an input sent while a reply is in flight, which history holds back until that reply is final, is left out of ' - 'the replay when a drop cuts the reply off: history records it, but the re-dialed conversation never gets it' - ), - tracked_by='replaying history only once the turn the drop cut off is settled; found by this simulator', - evidence='recorded', - codes=frozenset({'history.not_restored'}), - providers=OPENAI_PROTOCOL, - matches=_sent_during_a_lost_reply, -) - - def _send_failed_across_reconnect(sim: Simulation, violation: InvariantViolation) -> bool: from pydantic_ai.realtime import RealtimeError @@ -1096,24 +817,6 @@ def _send_failed_across_reconnect(sim: Simulation, violation: InvariantViolation ) -def _request_commit_spoken_over(sim: Simulation) -> bool: - """xAI push-to-talk without transcription: the user spoke again after a request for a response committed a turn.""" - options = getattr(sim, 'openai', None) - # (A send that asks for a response sends a request too.) - requests = [ - operation.issued - for operation in sim.operations - if operation.name in ('create_response', 'send_text', 'send_image_respond') - ] - audio = [operation.issued for operation in sim.operations if operation.name == 'send_audio'] - return ( - options is not None - and options.turn_detection == 'manual' - and not options.transcription - and any(request < sent for request in requests for sent in audio) - ) - - def _request_deferred_behind_an_answer_to_speech(sim: Simulation) -> bool: """xAI push-to-talk: `create_response()` was called once a reply answering a spoken turn had started.""" options = getattr(sim, 'openai', None) @@ -1151,40 +854,6 @@ DEFERRED_REQUEST_DROPPED_AFTER_SPEECH = Finding( ) -def _xai_push_to_talk_speech(sim: Simulation, violation: InvariantViolation) -> bool: - options = getattr(sim, 'openai', None) - input_ = sim.truth.input(violation.context.get('input', '')) - return options is not None and options.turn_detection == 'manual' and input_ is not None and input_.kind == 'speech' - - -PUSH_TO_TALK_ORDER = Finding( - id='SIM-31', - title=( - "on xAI push-to-talk, a spoken turn the session held back behind a reply (#9070's held audio and commit) is " - 'recorded on the wrong side of a reply: history does not follow the order xAI got them in' - ), - tracked_by="history in the provider's conversation order (the refactor); found gating #9070", - evidence='simulated', - codes=frozenset({'history.order'}), - providers=frozenset({'xai'}), - matches=_xai_push_to_talk_speech, -) - - -PUSH_TO_TALK_TURN_SPOKEN_OVER = Finding( - id='SIM-29', - title=( - 'on xAI push-to-talk without input transcription, the turn a request for a response commits is never recorded ' - 'when the user speaks again during its reply (which stops the reply with no `response.done`)' - ), - tracked_by='user turns recorded from the provider committing them; found gating #9070', - evidence='simulated', - codes=frozenset({'history.turn_missing'}), - providers=frozenset({'xai'}), - matches=lambda sim, violation: _request_commit_spoken_over(sim), -) - - NON_AUDIO_SEND_DURING_RECONNECT = Finding( id='G3b', title=( @@ -1217,21 +886,25 @@ FRAME_CUT_OFF_BY_CLOSE = Finding( def _provider_reply_before_any_echo(sim: Simulation, violation: InvariantViolation) -> bool: - """A response the provider started on its own was read after the input went out, before any of ours was.""" + """A response the provider started on its own was read after the input went out, before any of ours was. + + When the violation names a response (`history.order`), it is that one. + """ key = violation.context.get('input') - if not isinstance(key, str) or (input_ := sim.truth.input(key)) is None: + if not isinstance(key, str) or (input_ := sim.truth.input(key)) is None: # pragma: lax no cover return False issued = min((operation.issued for operation in sim.operations if operation.key == key), default=input_.seq) responses = sim.truth.responses.values() first_echo = min( (r.started_read for r in responses if r.trigger == 'create' and r.started_read is not None), default=None ) + named = _context_responses(sim, violation) return any( response.trigger != 'create' and response.started_read is not None and response.started_read > issued and (first_echo is None or response.started_read < first_echo) - for response in responses + for response in (responses if named is None else named) ) @@ -1254,27 +927,55 @@ PROVIDER_REPLY_BEFORE_ANY_ECHO = Finding( ) +CLEARED_TURN_LOSES_ITS_TRANSCRIPT = Finding( + id='SIM-37', + title=( + 'clearing the input audio while the user is still speaking ends a spoken turn xAI already added (it adds ' + 'one at speech start): the turn is recorded at once with what transcript it had, so the transcript xAI ' + 'still sends for it is dropped' + ), + tracked_by=( + 'an audio clear that settles only a turn not yet in the conversation, leaving one that joined to wait for ' + 'its transcript; found by review of the session core' + ), + evidence='simulated', + codes=frozenset({'history.transcript_lost'}), + providers=frozenset({'xai'}), + matches=lambda sim, violation: violation.context.get('input') in sim.truth.speech_cleared, +) + + +UNCOMMITTED_STOP_RECORDED_AS_A_TURN = Finding( + id='SIM-38', + title=( + 'a spoken turn joins the conversation at `speech_stopped`, which server VAD normally commits at once; if it ' + "doesn't (the user goes on, and a later start's item is committed instead), the turn waits for a " + 'transcript that never comes, holding back what follows it for the transcript wait, and is then recorded ' + 'as an empty user turn the provider never had' + ), + tracked_by=( + 'placing a spoken turn at its commit wherever the provider commits it separately; found by review of the ' + 'session core' + ), + evidence='simulated', + codes=frozenset({'history.phantom_turn'}), + providers=frozenset({'openai', 'azure'}), + # No more phantom turns than stops taken back: each of those makes at most one. + matches=lambda sim, violation: violation.context.get('extra', 0) <= len(sim.truth.speech_stopped_uncommitted), +) + + KNOWN_FINDINGS.extend( [ FRAME_CUT_OFF_BY_CLOSE, NON_AUDIO_SEND_DURING_RECONNECT, TURN_LOST_AT_CLOSE, - INPUT_HELD_BEHIND_A_LOST_REPLY, - PUSH_TO_TALK_TURN_SPOKEN_OVER, - CLEAR_AFTER_AN_UNREAD_VAD_COMMIT, PUSH_TO_TALK_AUDIO_AFTER_A_REPEATED_TERMINAL, HAND_COMMIT_UNANSWERED_AT_THE_END, - PUSH_TO_TALK_ORDER, DEFERRED_REQUEST_DROPPED_AFTER_SPEECH, - AUDIO_AFTER_A_CLEAR, - HAND_COMMIT_UNDER_SERVER_VAD, - BARGE_IN_WITHOUT_A_VAD_REPLY, LOST_REFUSAL, TERMINAL_DISCARDED_WITH_THE_CONNECTION, - BARGE_IN_ON_A_TOOL_ROUND, - REFUSED_CONTEXT_MISFILES_SPEECH, PARKED_ERROR_LEAVES_REQUEST_OWED, - CLEARED_BARGE_IN, EXTENDED_THINKING_PARALLEL_CALLS, LIVE_REPLY_SPLIT_BY_TOOL_ROUND, ASYNC_BATCH_OF_THREE, @@ -1289,6 +990,8 @@ KNOWN_FINDINGS.extend( # The general reservation leaks last: a more specific finding explains a hang better. LOST_RESPONSE_RESERVATION, PROVIDER_REPLY_BEFORE_ANY_ECHO, + CLEARED_TURN_LOSES_ITS_TRANSCRIPT, + UNCOMMITTED_STOP_RECORDED_AS_A_TURN, ] ) diff --git a/tests/realtime/simulation/_invariants.py b/tests/realtime/simulation/_invariants.py index e35fad59fa..e79808666f 100644 --- a/tests/realtime/simulation/_invariants.py +++ b/tests/realtime/simulation/_invariants.py @@ -83,10 +83,8 @@ from pydantic_ai.messages import ( UserPromptPart, ) from pydantic_ai.realtime import RealtimeError, RealtimeSession -from pydantic_ai.realtime._core import Owed, SessionCore from pydantic_ai.realtime._lifecycle import LifecycleEvent from pydantic_ai.realtime.codec import RealtimeCodecEvent -from pydantic_ai.usage import RunUsage from ._conformance import ConformanceIssue, LifecycleChecker from ._truth import TruthInput, TruthResponse, response_numbers @@ -204,13 +202,9 @@ class Checker: self._judged_truncations = 0 self._consumer_error_judged = False self._codec_issues: list[ConformanceIssue] = [] - self.shadow: ShadowChecker | None = None - """The same invariants on the new session core running in shadow of this one, when it runs one.""" def attach(self, session: RealtimeSession) -> None: """Observe the codec events the connection yields, on their way into the session.""" - if (core := session._core) is not None: # pyright: ignore[reportPrivateUsage] - self.shadow = ShadowChecker(self.sim, core) handle = session._handle_pump_event # pyright: ignore[reportPrivateUsage] async def observed(event: RealtimeCodecEvent) -> bool: @@ -249,22 +243,6 @@ class Checker: # --- every step ------------------------------------------------------------------------------- - def messages(self) -> list[ModelMessage]: - """The history the invariants judge.""" - session = self.sim.session - assert session is not None - return session.all_messages() - - def new_messages(self) -> list[ModelMessage]: - session = self.sim.session - assert session is not None - return session.new_messages() - - def usage(self) -> RunUsage: - session = self.sim.session - assert session is not None - return session.usage - def check_step(self) -> None: session = self.sim.session assert session is not None @@ -276,17 +254,12 @@ class Checker: ) self._check_errors() self._check_simulator_frames() - self._check_history() + messages = session.all_messages() + self._check_history_stability(messages) + self._check_response_identity(messages) waiters = [w for w in self.sim.waiters if w.returned is not None and w.index not in self._judged_waiters] self._judged_waiters.update(waiter.index for waiter in waiters) self.report('wait.early', [violation for waiter in waiters for violation in self._early_return(waiter)]) - if self.shadow is not None: - self.shadow.check_step() - - def _check_history(self) -> None: - messages = self.messages() - self._check_history_stability(messages) - self._check_response_identity(messages) def _check_simulator_frames(self) -> None: """The simulated server must only send frames the real connection can parse; anything else is a harness bug.""" @@ -317,7 +290,7 @@ class Checker: unexpected.append((f'iterating the session raised {error!r}', {'operation': 'iterate'})) self.report('api.unexpected_error', unexpected) - def _removal_allowed(self, message: ModelMessage) -> bool: + def _removal_allowed(self, message: ModelMessage) -> bool: # pragma: lax no cover (the session core never removes) """A refused input taken back, or the request of a send that failed.""" truth = self.sim.truth failed_sends = {operation.key for operation in self.sim.operations if operation.error is not None} @@ -510,18 +483,7 @@ class Checker: self.check_step() session = sim.session assert session is not None - self._check_history_at_rest() - self._check_playback(self.messages()) - self._check_wire() - if not session.closed: - # Also once the event stream has ended: `wait_for_reply()` must never outlive the session's ability to reply. - self._check_sends_survived() - self._check_wait_liveness() - if self.shadow is not None: - self.shadow.check_at_rest() - - def _check_history_at_rest(self) -> None: - messages = self.messages() + messages = session.all_messages() self._check_tool_pairing(messages) self._check_tool_round_order(messages) self._check_order(messages) @@ -529,11 +491,62 @@ class Checker: self._check_spoken_turns(messages) self._check_restored(messages) self._check_usage(messages) + self._check_playback(messages) roundtrip = ModelMessagesTypeAdapter.validate_json(ModelMessagesTypeAdapter.dump_json(messages)) self.report( 'history.roundtrip', [('history did not survive a serialization round trip', {})] if roundtrip != messages else [], ) + self._check_transcripts(messages) + self._check_wire() + if not session.closed: + # Also once the event stream has ended: `wait_for_reply()` must never outlive the session's ability to reply. + self._check_sends_survived() + self._check_wait_liveness() + + def _check_transcripts(self, messages: list[ModelMessage]) -> None: + """A spoken turn's transcript the session read before it was asked to close is in history, and history has + no spoken turn the provider never had.""" + sim = self.sim + truth = sim.truth + if truth.models_every_spoken_turn: + spoken = sum(input_.kind == 'speech' for input_ in truth.inputs) + if sim.close_requested is not None: + # Closing records what the user was still saying, though the provider never committed it. + inputs = {input_.key for input_ in truth.inputs} + spoken += sum( + key not in inputs # (xAI adds the turn at speech start: it's counted above.) + and key not in truth.speech_committed + and key not in truth.speech_stopped_uncommitted + # A clear the session never heard back about (its send failed) leaves the turn as it was. + and (key not in truth.speech_cleared or key not in truth.speech_cleared_read) + for key in truth.speech_started + ) + recorded_turns = sum(is_user_speech_request(message) for message in messages) + self.report( + 'history.phantom_turn', + [ + ( + f'history has {recorded_turns} spoken turns, but the provider had {spoken}', + {'extra': recorded_turns - spoken}, + ) + ] + if recorded_turns > spoken + else [], + ) + if sim.receive_ended and sim.close_requested is None: + # The session ended on its own: what it holds back for a response it will never read the end of is + # recorded when it closes. + return + recorded = {key for message in messages if isinstance(message, ModelRequest) for key in request_keys(message)} + self.report( + 'history.transcript_lost', + [ + (f"the transcript of spoken turn {key!r} was read but isn't in history", {'input': key}) + for key, read in sim.truth.transcripts_read.items() + if key not in recorded and (sim.close_requested is None or read < sim.close_requested) + ], + ) def _check_tool_round_order(self, messages: list[ModelMessage]) -> None: truth = self.sim.truth @@ -752,6 +765,8 @@ class Checker: def _check_usage(self, messages: list[ModelMessage]) -> None: sim = self.sim + session = sim.session + assert session is not None if sim.receive_ended and sim.close_requested is None: # The session ended on its own (an exceeded usage limit, a lost connection): whatever it read # after that point was never going to be accounted. @@ -761,7 +776,7 @@ class Checker: sum(tokens[0] for tokens in truth.usage_read.values()), sum(tokens[1] for tokens in truth.usage_read.values()), ) - usage = self.usage() + usage = session.usage recorded = (usage.input_tokens, usage.output_tokens) # Closing stops the session mid-frame: a report the connection had read ahead may go unaccounted, but # nothing may ever be counted twice. @@ -791,7 +806,7 @@ class Checker: # A model that reports its requests with usage (GPT-Live's delegated backend) counts those instead. expected = sim.expected_requests() if expected is None: - expected = sum(isinstance(message, ModelResponse) for message in self.new_messages()) + expected = sum(isinstance(message, ModelResponse) for message in session.new_messages()) self.report( 'usage.requests', [(f'usage.requests is {usage.requests}, but {expected} were made', {})] @@ -903,124 +918,3 @@ class Checker: if stuck else [], ) - - -SHADOW_PENDING: frozenset[str] = frozenset( - { - 'SIM-1', - 'SIM-4', - 'SIM-10', - 'SIM-15', - 'SIM-22', - 'SIM-23', - 'SIM-24', - 'SIM-25', - 'SIM-27', - 'SIM-28', - 'SIM-30', - 'SIM-34', - 'SIM-35', - 'SIM-36', - '8763c #3', - } -) -"""Known findings the new session core does not fix yet, which the shadow checks tolerate like the others. - -They are the session's and the connection's to fix, not the core's, since the core only sees what the -connection reports: - -- `SIM-1`: a reply lost with a dropped connection is not asked for again by the reconnect; -- `SIM-4`, `SIM-23`: the connection drops a whole frame when a request it sends from its receive loop fails, - or is cancelled by a close, so the core never sees that response's usage or terminal either; -- `SIM-10`: the connection drops what a response said after a cancel that reached the server too late; -- `SIM-15`: a tool that raises leaves the session unable to get the replies still owed after it: the - connection holds a later request behind the calling response, which the provider never ends; -- `SIM-22`: a terminal read as the connection drops is discarded with it; -- `SIM-24` (accepted): until the server echoes request metadata, the connection infers which response answers - a request, and can take one server VAD started for it. -- `SIM-25`: a spoken turn committed with transcription on is recorded only once its transcript comes, so one - whose transcript never does (the session closes, ends, or drops first) is lost; the core records it the same way; -- `SIM-28`: a reconnect replays history as it stands at the re-dial, without an input still held back behind - the reply the drop cut off; -- `SIM-30`: on xAI push-to-talk, the connection sends a deferred request without the clear xAI needs to answer it; -- `SIM-27`: a hand commit under server VAD splits the user's turn in two, and only one is recorded; -- `SIM-34`: unlike the rest, mostly the core's own: it loses a turn `commit_audio()` committed when the connection - drops, or the session closes or ends, before its reply is over (the current session records it, except on xAI); -- `SIM-35`: without transcription, a clear made after server VAD committed a turn, before the session read the - commit, drops that turn; -- `SIM-36`: on xAI push-to-talk, audio sent after a repeated `response.done` is held behind a reply that ended; -- `8763c #3`: a response already under way takes a turn's reservation; the core still does this where server VAD - is set not to answer (`create_response: False`). -""" - - -class ShadowChecker(Checker): - """The session invariants, judged on the new core running in shadow of the session. - - History, usage, and waits come from the core instead of the session, so every check says what the core - would get right or wrong in its place. Its violations are reported as `shadow.`, and only the - findings in `SHADOW_PENDING` are tolerated: whatever else the session gets wrong, the core must not. - Its waits mirror the session's: each `wait_for_reply()` the trace starts takes what the core owed at that - moment, and returns once the core says all of it came. - """ - - def __init__(self, sim: Simulation, core: SessionCore) -> None: - super().__init__(sim, strict=sim.strict, enforce=sim.enforce) - self.core = core - self.waits: dict[int, tuple[Waiter, frozenset[Owed]]] = {} - - def messages(self) -> list[ModelMessage]: - return self.core.all_messages() - - def new_messages(self) -> list[ModelMessage]: - return self.core.new_messages() - - def usage(self) -> RunUsage: - return self.core.usage - - def report(self, code: str, violations: Iterable[Violation]) -> None: - from ._findings import matching_findings - from ._simulation import FindingReproduced, InvariantViolation - - for detail, context in violations: - violation = InvariantViolation(f'shadow.{code}', detail, self.sim.trace, context) - findings = [f.id for f in matching_findings(self.sim, InvariantViolation(code, detail, [], context))] - violation.findings = findings - if self.enforce.intersection(findings): - # A pinned scenario for a finding the core has (and the current session may not). - raise FindingReproduced(f'shadow.{code}', detail, self.sim.trace, context, findings=findings) - pending = [finding for finding in findings if finding in SHADOW_PENDING] - if self.strict or not pending: # pragma: no cover (only when the core breaks an invariant) - raise violation - self.known_hits.append((pending[0], f'shadow.{code}')) - - def waiter_started(self, waiter: Waiter) -> None: - from ._simulation import Waiter - - mirror = Waiter(index=waiter.index, started=waiter.started, snapshot=waiter.snapshot) - self.waits[waiter.index] = (mirror, self.core.wait_tokens()) - - def check_step(self) -> None: - self._check_history() - judged: list[Waiter] = [] - for mirror, tokens in self.waits.values(): - if mirror.returned is None and not self.core.still_owed(tokens): - mirror.returned = self.sim.truth.tick() - judged.append(mirror) - self.report('wait.early', [violation for waiter in judged for violation in self._early_return(waiter)]) - - def check_at_rest(self) -> None: - self.check_step() - self._check_history_at_rest() - session = self.sim.session - assert session is not None - if not session.closed: - stuck = [mirror.index for mirror, _ in self.waits.values() if mirror.returned is None] - if self.core.reply_outstanding(): - stuck.append(len(self.sim.waiters)) - self.report( - 'wait.hang', - [(f'wait_for_reply() {stuck} still waiting with nothing left to wait for', {'waiters': stuck})] - if stuck - else [], - ) diff --git a/tests/realtime/simulation/_openai_server.py b/tests/realtime/simulation/_openai_server.py index 8270f4e53b..7fe58d3a7e 100644 --- a/tests/realtime/simulation/_openai_server.py +++ b/tests/realtime/simulation/_openai_server.py @@ -152,10 +152,13 @@ class OpenAIServer: def __init__(self, *, dialect: Dialect = 'openai', model: str = 'gpt-realtime') -> None: self.dialect = dialect self.model = model - self.truth = GroundTruth() + self.truth = GroundTruth(models_every_spoken_turn=True) self.network = Network(self) self.sessions: list[ServerSession] = [] self._armed_rejections: list[Literal['content', 'response']] = [] + # The turn each `input_audio_buffer.cleared` cut off (if any), until the client reads the frame, by the + # connection it comes back on. + self._clears_unread: dict[int, list[str | None]] = {} self._next_item = 1 # Server event id of an `error` frame -> the inputs it refused, resolved when the client reads it. self._refusals: dict[str, list[str]] = {} @@ -239,6 +242,13 @@ class OpenAIServer: response.started_read = now if response is not None and response.content_read is None and frame_type.startswith(_CONTENT_FRAME_PREFIXES): response.content_read = now + if frame_type == 'input_audio_buffer.cleared' and ( + clears := self._clears_unread.get(self._session_for(socket).index) + ): + if (cleared := clears.pop(0)) is not None: + self.truth.speech_cleared_read.add(cleared) + if frame_type == 'conversation.item.input_audio_transcription.completed': + self.truth.transcripts_read.setdefault(str(frame.get('transcript')), now) if frame_type == 'response.function_call_arguments.done' and ( call := self.truth.tool_calls.get(frame.get('call_id', '')) ): @@ -419,6 +429,13 @@ class OpenAIServer: # (Only a clear between a commit the session sent and its reply's first word.) self._finish_active(session, 'cancelled') # A clear right after a commit cancels its reply. session.audio_ms = 0 + self._clears_unread.setdefault(session.index, []).append(session.speaking) + if (key := session.speaking) is not None: + self.truth.speech_cleared.add(key) + if self.dialect == 'xai' and session.transcription: + # A guess: xAI already added the turn's item when it heard speech start, and still + # transcribes what it had of it (no recording clears the buffer mid-speech on xAI). + session.pending_transcripts.append(key) session.speaking = None self._emit(session, {'type': 'input_audio_buffer.cleared'}) @@ -873,13 +890,20 @@ class OpenAIServer: self._finish_active(session, 'cancelled', reason='turn_detected', late=late) return key - def speech_stop(self) -> str: - """Server VAD hears the user stop: the audio is committed as a user turn, and answered if configured to.""" + def speech_stop(self, *, commit: bool = True) -> str: + """Server VAD hears the user stop: the audio is committed as a user turn, and answered if configured to. + + `commit=False` is a stop server VAD takes back: the user goes on, and the next start opens the turn + the audio is committed as (a guess at semantic VAD's pauses; no recording has one). + """ session = self.session assert session is not None and session.speaking is not None key, session.speaking = session.speaking, None item_id = f'item_{key}' self._emit(session, {'type': 'input_audio_buffer.speech_stopped', 'item_id': item_id, 'audio_end_ms': 1000}) + if not commit: + self.truth.speech_stopped_uncommitted.add(key) + return key session.audio_ms = 0 self.truth.speech_committed.add(key) if self.dialect != 'xai': diff --git a/tests/realtime/simulation/_openai_simulation.py b/tests/realtime/simulation/_openai_simulation.py index 0db9d8e57c..ebaecb2fec 100644 --- a/tests/realtime/simulation/_openai_simulation.py +++ b/tests/realtime/simulation/_openai_simulation.py @@ -198,8 +198,8 @@ class OpenAISimulation(Simulation): self._after_server(deliver, ticks) @step - def speech_stop(self, deliver: bool = True, ticks: int | None = None) -> None: - self.server.speech_stop() + def speech_stop(self, deliver: bool = True, ticks: int | None = None, commit: bool = True) -> None: + self.server.speech_stop(commit=commit) self._after_server(deliver, ticks) @step @@ -358,9 +358,11 @@ class OpenAIMachine(ManualTurnMachine): # pragma: lax no cover (driven only by self.run(lambda: self.openai_sim.speech_start(late=late, deliver=deliver, ticks=ticks)) @precondition(lambda self: (session := self.server_session()) is not None and session.speaking is not None) - @rule(deliver=st.booleans(), ticks=TICKS) - def speech_stop(self, deliver: bool, ticks: int | None) -> None: - self.run(lambda: self.openai_sim.speech_stop(deliver=deliver, ticks=ticks)) + @rule(deliver=st.booleans(), ticks=TICKS, commit=st.booleans()) + def speech_stop(self, deliver: bool, ticks: int | None, commit: bool) -> None: + # xAI adds a turn's item when it hears speech start, so a stop it takes back is not modeled there. + commit = commit or self.openai_sim.openai.dialect == 'xai' + self.run(lambda: self.openai_sim.speech_stop(deliver=deliver, ticks=ticks, commit=commit)) @precondition(lambda self: (session := self.server_session()) is not None and bool(session.pending_transcripts)) @rule(fail=st.booleans(), deliver=st.booleans(), ticks=TICKS) diff --git a/tests/realtime/simulation/_simulation.py b/tests/realtime/simulation/_simulation.py index 703c0f2521..cf166691a8 100644 --- a/tests/realtime/simulation/_simulation.py +++ b/tests/realtime/simulation/_simulation.py @@ -521,8 +521,6 @@ class Simulation(ABC): snapshot=frozenset(op.key for op in self.operations if op.done and op.key is not None and op.error is None), ) self.waiters.append(waiter) - if (shadow := self.checker.shadow) is not None: - shadow.waiter_started(waiter) async def wait() -> None: try: diff --git a/tests/realtime/simulation/_truth.py b/tests/realtime/simulation/_truth.py index 6cf128a2ec..84a34f8e57 100644 --- a/tests/realtime/simulation/_truth.py +++ b/tests/realtime/simulation/_truth.py @@ -150,6 +150,16 @@ class GroundTruth: """When server VAD heard each spoken user turn start, on the shared clock.""" speech_committed: set[str] = field(default_factory=set[str]) """The spoken user turns whose audio was committed (xAI adds a turn's item before it commits it).""" + speech_cleared: set[str] = field(default_factory=set[str]) + """Spoken user turns whose buffered audio was cleared while the user was still saying them.""" + speech_cleared_read: set[str] = field(default_factory=set[str]) + """Those whose `input_audio_buffer.cleared` the client read.""" + speech_stopped_uncommitted: set[str] = field(default_factory=set[str]) + """Spoken user turns server VAD heard stop without committing them (the user went on, as one turn).""" + transcripts_read: dict[str, int] = field(default_factory=dict[str, int]) + """When the client read each spoken user turn's completed transcript.""" + models_every_spoken_turn: bool = False + """Whether every spoken user turn the provider has is an input here, so history can have no other.""" word_seq: dict[str, int] = field(default_factory=dict[str, int]) """When the server generated each word a response said, on the shared clock.""" word_read: dict[str, int] = field(default_factory=dict[str, int]) diff --git a/tests/realtime/simulation/conftest.py b/tests/realtime/simulation/conftest.py deleted file mode 100644 index f61dd160b3..0000000000 --- a/tests/realtime/simulation/conftest.py +++ /dev/null @@ -1,17 +0,0 @@ -"""The simulator judges the new session core with its own invariants, rather than against the current one.""" - -from __future__ import annotations as _annotations - -import pytest - -from pydantic_ai.realtime import _session as realtime_session # pyright: ignore[reportPrivateUsage] - - -@pytest.fixture(autouse=True) -def _shadow_core(monkeypatch: pytest.MonkeyPatch) -> None: - """Run the new core in shadow, and let the simulator's invariants judge it (see `ShadowChecker`). - - Overrides the realtime suite's fixture of the same name, which compares the two cores' histories: the - current core's known findings make them disagree on traces the simulator is built to find. - """ - monkeypatch.setattr(realtime_session, '_CORE_MODE', 'shadow') diff --git a/tests/realtime/simulation/test_conformance.py b/tests/realtime/simulation/test_conformance.py index 041780dac9..95d60a75e9 100644 --- a/tests/realtime/simulation/test_conformance.py +++ b/tests/realtime/simulation/test_conformance.py @@ -23,6 +23,7 @@ with try_import() as imports_successful: InputAdded, InputLost, LifecycleEvent, + OutputItemDetails, ResponseEnded, ResponseRequestRefused, ResponseStarted, @@ -207,6 +208,10 @@ def test_lifecycle_contract_rules() -> None: ) == snapshot(['lifecycle.turn_end_without_start']) assert feed_lifecycle(turn, turn) == snapshot(['lifecycle.turn_started_twice', 'lifecycle.turn_unended_at_close']) assert feed_lifecycle(UserTurnEnded(turn_id='item_u2')) == snapshot(['lifecycle.turn_end_without_start']) + details = OutputItemDetails(response_id='resp_1', item_id='item_a1', provider_details={'phase': 'commentary'}) + assert feed_lifecycle(start, details, end, details, inputs_sent=1) == snapshot( + ['lifecycle.content_outside_response'] + ) async def test_recorded_abnormal_close_is_replayed_as_one() -> None: diff --git a/tests/realtime/simulation/test_simulation.py b/tests/realtime/simulation/test_simulation.py index 00466e10eb..4d21dd6b63 100644 --- a/tests/realtime/simulation/test_simulation.py +++ b/tests/realtime/simulation/test_simulation.py @@ -158,9 +158,8 @@ def test_known_late_cancel_drops_a_finished_reply() -> None: reproduce('SIM-10', OpenAISimulation(openai=OpenAIOptions(turn_detection='manual')), scenario) -@known('SIM-11') -def test_known_turn_spoken_before_a_reply_filed_before_it() -> None: - """The user started talking, the model answered a typed turn, and only then was the spoken turn committed.""" +def test_turn_spoken_before_a_reply_but_committed_after_it_is_filed_after_it() -> None: + """The user started talking, the model answered a typed turn, and only then was the spoken turn committed (SIM-11, fixed by the session core).""" def scenario(sim: OpenAISimulation) -> None: sim.send_audio() @@ -169,13 +168,15 @@ def test_known_turn_spoken_before_a_reply_filed_before_it() -> None: sim.commit_audio() sim.settle() - reproduce('SIM-11', OpenAISimulation(openai=OpenAIOptions(turn_detection='manual')), scenario) + run_clean(OpenAISimulation(openai=OpenAIOptions(turn_detection='manual')), scenario) -@known('SIM-11') -def test_known_second_commit_takes_the_turn_of_earlier_audio() -> None: - """Audio streamed before each of two replies, committed as one turn after them, then a second turn committed: - the session gives the second turn the place of the audio before the second reply.""" +def test_second_commit_keeps_the_turn_of_earlier_audio_in_place() -> None: + """SIM-11, fixed by the session core. + + Audio streamed before each of two replies, committed as one turn after them, then a second turn committed: + the session gives the second turn the place of the audio before the second reply. + """ def scenario(sim: OpenAISimulation) -> None: sim.send_audio() @@ -189,7 +190,7 @@ def test_known_second_commit_takes_the_turn_of_earlier_audio() -> None: sim.commit_audio() sim.settle() - reproduce('SIM-11', OpenAISimulation(openai=OpenAIOptions(turn_detection='manual')), scenario) + run_clean(OpenAISimulation(openai=OpenAIOptions(turn_detection='manual')), scenario) @known('SIM-11') @@ -206,8 +207,9 @@ def test_known_live_utterance_spoken_before_a_reply_filed_before_it() -> None: reproduce('SIM-11', LiveSimulation(), scenario) -@known('SIM-12') -def test_known_refused_tool_results_request_leaves_wait_hanging() -> None: +def test_refused_tool_results_request_ends_the_wait() -> None: + """SIM-12, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_text() sim.call_tool() @@ -215,12 +217,11 @@ def test_known_refused_tool_results_request_leaves_wait_hanging() -> None: sim.finish() sim.settle() - reproduce('SIM-12', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) -@known('OR9') -def test_known_turn_committed_by_hand_under_server_vad_filed_late() -> None: - """Server VAD hears the user start; the app commits the buffer by hand before VAD commits the rest.""" +def test_turn_committed_by_hand_under_server_vad_is_filed_where_the_provider_put_it() -> None: + """Server VAD hears the user start; the app commits the buffer by hand before VAD commits the rest (OR9, fixed by the session core).""" def scenario(sim: OpenAISimulation) -> None: sim.send_audio() @@ -228,7 +229,7 @@ def test_known_turn_committed_by_hand_under_server_vad_filed_late() -> None: sim.commit_audio() sim.settle() - reproduce('OR9', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) @known('SIM-13') @@ -279,9 +280,8 @@ def test_known_gemini_async_result_cuts_into_speech() -> None: reproduce('8760', async_gemini(), scenario) -@known('SIM-11') -def test_known_turn_heard_before_a_cancelled_reply_ended_filed_before_it() -> None: - """Server VAD hears the user start while a reply the app cancelled (after it called a tool) is still ending.""" +def test_turn_heard_before_a_cancelled_reply_ended_is_filed_after_it() -> None: + """Server VAD hears the user start while a reply the app cancelled (after it called a tool) is still ending (SIM-11, fixed by the session core).""" def scenario(sim: OpenAISimulation) -> None: sim.send_text() @@ -291,7 +291,7 @@ def test_known_turn_heard_before_a_cancelled_reply_ended_filed_before_it() -> No sim.speech_start(deliver=False) sim.settle() - reproduce('SIM-11', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) @known('SIM-1') @@ -312,19 +312,21 @@ def test_known_gemini_tool_batch_answer_lost_to_a_drop() -> None: reproduce('SIM-1', GeminiSimulation(), scenario) -@known('SIM-14') -def test_known_refused_context_misfiles_the_spoken_turn() -> None: +def test_refused_context_leaves_the_spoken_turn_in_place() -> None: + """SIM-14, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.reject_next('content') sim.speech_start(deliver=False) sim.send_image() sim.settle() - reproduce('SIM-14', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) -@known('SIM-15') -def test_known_raising_tool_leaves_a_deferred_request_owed() -> None: +def test_raising_tool_ends_the_wait_for_a_deferred_request() -> None: + """SIM-15, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.create_response() sim.create_response() @@ -332,22 +334,24 @@ def test_known_raising_tool_leaves_a_deferred_request_owed() -> None: sim.finish_tool(outcome='error') sim.settle() - reproduce('SIM-15', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) -@known('SIM-15') -def test_known_tool_result_over_request_limit_leaves_wait_hanging() -> None: +def test_tool_result_over_request_limit_ends_the_wait() -> None: + """SIM-15, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_image(respond=True) sim.call_tool(deliver=False) sim.send_image(respond=True) sim.settle() - reproduce('SIM-15', OpenAISimulation(options=SessionOptions(request_limit=2)), scenario) + run_clean(OpenAISimulation(options=SessionOptions(request_limit=2)), scenario) -@known('SIM-16') -def test_known_cleared_barge_in_keeps_the_dropped_request() -> None: +def test_cleared_barge_in_releases_the_dropped_request() -> None: + """SIM-16, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_audio() sim.create_response() @@ -356,7 +360,7 @@ def test_known_cleared_barge_in_keeps_the_dropped_request() -> None: sim.clear_audio() sim.settle() - reproduce('SIM-16', OpenAISimulation(openai=OpenAIOptions(transcription=False)), scenario) + run_clean(OpenAISimulation(openai=OpenAIOptions(transcription=False)), scenario) @known('SIM-17') @@ -431,8 +435,9 @@ def test_known_typed_turn_during_reconnect_fails() -> None: reproduce('G3b', OpenAISimulation(), scenario) -@known('SIM-20') -def test_known_barge_in_on_a_tool_round_keeps_the_deferred_request() -> None: +def test_barge_in_on_a_tool_round_releases_the_deferred_request() -> None: + """SIM-20, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_text() sim.call_tool() @@ -441,7 +446,7 @@ def test_known_barge_in_on_a_tool_round_keeps_the_deferred_request() -> None: sim.speech_start(deliver=False) sim.settle() - reproduce('SIM-20', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) @known('8801') @@ -459,8 +464,9 @@ def test_known_late_terminal_of_a_barged_in_reply_lands_on_the_next() -> None: reproduce('8801', OpenAISimulation(), scenario) -@known('SIM-21') -def test_known_request_whose_refusal_is_lost_keeps_its_reservation() -> None: +def test_request_whose_refusal_is_lost_is_settled_by_the_reconnect() -> None: + """SIM-21, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.reject_next('response') sim.create_response() @@ -468,7 +474,7 @@ def test_known_request_whose_refusal_is_lost_keeps_its_reservation() -> None: sim.drop() sim.settle() - reproduce('SIM-21', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) @known('SIM-23') @@ -522,8 +528,9 @@ def test_known_failed_deferred_create_after_a_refusal_keeps_its_reservation() -> reproduce('SIM-4', OpenAISimulation(), scenario) -@known('SIM-32') -def test_known_barge_in_drops_a_request_vad_will_not_answer() -> None: +def test_barge_in_keeps_a_request_vad_will_not_answer() -> None: + """SIM-32, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.create_response() sim.create_response() @@ -531,12 +538,14 @@ def test_known_barge_in_drops_a_request_vad_will_not_answer() -> None: sim.speech_start(deliver=False) sim.settle() - reproduce('SIM-32', OpenAISimulation(openai=OpenAIOptions(vad_responds=False, transcription=False)), scenario) + run_clean(OpenAISimulation(openai=OpenAIOptions(vad_responds=False, transcription=False)), scenario) -@known('SIM-11') -def test_known_turn_spoken_while_a_cancelled_reply_ends_filed_before_it() -> None: - """The user starts talking after the app cancelled a reply, before the provider has ended it.""" +def test_turn_spoken_while_a_cancelled_reply_ends_is_filed_after_it() -> None: + """SIM-11, fixed by the session core. + + The user starts talking after the app cancelled a reply, before the provider has ended it. + """ def scenario(sim: OpenAISimulation) -> None: sim.send_image(respond=True) @@ -547,7 +556,7 @@ def test_known_turn_spoken_while_a_cancelled_reply_ends_filed_before_it() -> Non sim.commit_audio() sim.settle() - reproduce('SIM-11', OpenAISimulation(openai=OpenAIOptions(turn_detection='manual', dialect='azure')), scenario) + run_clean(OpenAISimulation(openai=OpenAIOptions(turn_detection='manual', dialect='azure')), scenario) @known('SIM-33') @@ -575,16 +584,16 @@ def test_known_turn_committed_before_closing_is_lost() -> None: reproduce('SIM-25', OpenAISimulation(openai=OpenAIOptions(turn_detection='manual', transcription=True)), scenario) -@known('SIM-26') -def test_known_turn_committed_after_a_racing_clear_is_lost() -> None: +def test_turn_committed_after_a_racing_clear_is_recorded() -> None: + """SIM-26, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_audio(chunks=3, ticks=0) sim.clear_audio() sim.commit_audio() sim.settle() - reproduce( - 'SIM-26', + run_clean( OpenAISimulation( options=SessionOptions(latency=True), openai=OpenAIOptions(turn_detection='manual', transcription=False) ), @@ -592,16 +601,16 @@ def test_known_turn_committed_after_a_racing_clear_is_lost() -> None: ) -@known('SIM-27') -def test_known_hand_commit_under_server_vad_loses_a_turn() -> None: +def test_hand_commit_under_server_vad_keeps_both_turns() -> None: + """SIM-27, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_audio() sim.speech_start() sim.commit_audio() sim.settle() - reproduce( - 'SIM-27', + run_clean( OpenAISimulation( openai=OpenAIOptions(dialect='azure', transcription=False, vad_interrupts=False, vad_responds=False) ), @@ -609,8 +618,9 @@ def test_known_hand_commit_under_server_vad_loses_a_turn() -> None: ) -@known('SIM-28') -def test_known_input_held_behind_a_lost_reply_is_not_replayed() -> None: +def test_input_held_behind_a_lost_reply_is_replayed() -> None: + """SIM-28, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_text() sim.speak() @@ -620,7 +630,7 @@ def test_known_input_held_behind_a_lost_reply_is_not_replayed() -> None: sim.send_text() sim.settle() - reproduce('SIM-28', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) @known('SIM-34') @@ -634,8 +644,9 @@ def test_known_core_loses_a_hand_commit_unanswered_at_a_drop() -> None: reproduce('SIM-34', OpenAISimulation(openai=OpenAIOptions(turn_detection='manual', transcription=False)), scenario) -@known('SIM-35') -def test_known_clear_after_an_unread_vad_commit_drops_the_turn() -> None: +def test_clear_after_an_unread_vad_commit_keeps_the_turn() -> None: + """SIM-35, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_audio() sim.speech_start() @@ -643,7 +654,7 @@ def test_known_clear_after_an_unread_vad_commit_drops_the_turn() -> None: sim.clear_audio() sim.settle() - reproduce('SIM-35', OpenAISimulation(openai=OpenAIOptions(transcription=False)), scenario) + run_clean(OpenAISimulation(openai=OpenAIOptions(transcription=False)), scenario) @known('SIM-36') @@ -665,16 +676,16 @@ def test_known_xai_push_to_talk_audio_after_a_repeated_terminal_is_held() -> Non ) -@known('SIM-29') -def test_known_xai_push_to_talk_turn_spoken_over_is_lost() -> None: +def test_xai_push_to_talk_turn_spoken_over_is_recorded() -> None: + """SIM-29, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_audio() sim.create_response() sim.send_audio() sim.settle() - reproduce( - 'SIM-29', + run_clean( OpenAISimulation(openai=OpenAIOptions(dialect='xai', turn_detection='manual', transcription=False)), scenario, ) @@ -695,9 +706,11 @@ def test_known_xai_push_to_talk_request_deferred_behind_an_answer_is_dropped() - ) -@known('SIM-31') -def test_known_xai_push_to_talk_turn_spoken_during_a_reply_filed_before_it() -> None: - """xAI push-to-talk: the user talks during the reply to a held commit; the request after it commits that turn.""" +def test_xai_push_to_talk_turn_spoken_during_a_reply_is_filed_after_it() -> None: + """SIM-31, fixed by the session core. + + xAI push-to-talk: the user talks during the reply to a held commit; the request after it commits that turn. + """ def scenario(sim: OpenAISimulation) -> None: sim.send_audio() @@ -708,15 +721,15 @@ def test_known_xai_push_to_talk_turn_spoken_during_a_reply_filed_before_it() -> sim.create_response() sim.settle() - reproduce( - 'SIM-31', + run_clean( OpenAISimulation(openai=OpenAIOptions(dialect='xai', turn_detection='manual', transcription=True)), scenario, ) -@known('E') -def test_known_late_transcript_inserted_into_recorded_history() -> None: +def test_late_transcript_holds_back_the_reply_instead_of_being_inserted() -> None: + """E, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_audio() sim.speech_start() @@ -725,7 +738,7 @@ def test_known_late_transcript_inserted_into_recorded_history() -> None: sim.finish() sim.transcribe() - reproduce('E', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) @known('8801') @@ -741,9 +754,8 @@ def test_known_repeated_terminal_recorded_as_a_new_response() -> None: reproduce('8801', OpenAISimulation(), scenario) -@known('SIM-1') -def test_known_reply_lost_to_a_drop_keeps_its_reservation_openai() -> None: - """The response had started (`response.created` read) when the socket dropped, so it is not asked for again.""" +def test_started_reply_lost_to_a_drop_is_settled_openai() -> None: + """The response had started (`response.created` read) when the socket dropped, so it is not asked for again (SIM-1, fixed by the session core).""" def scenario(sim: OpenAISimulation) -> None: sim.send_text() @@ -751,7 +763,109 @@ def test_known_reply_lost_to_a_drop_keeps_its_reservation_openai() -> None: sim.drop() sim.settle() - reproduce('SIM-1', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) + + +@known('SIM-1') +def test_known_server_vad_reply_lost_to_a_drop_holds_back_the_next() -> None: + """A server-VAD reply that called a tool is lost with the connection; the reply asked for after the reconnect + is never answered: the provider is still owed the lost call's output.""" + + def scenario(sim: OpenAISimulation) -> None: + sim.send_audio() + sim.speech_start(deliver=False) + sim.fail_next_send(fault='lost') + sim.speech_stop(deliver=False) + sim.call_tool(deliver=False) + sim.settle() + sim.create_response() + sim.settle() + + reproduce('SIM-1', OpenAISimulation(openai=OpenAIOptions(transcription=False)), scenario) + + +@known('SIM-15') +def test_known_raising_tool_leaves_a_later_request_owed_xai() -> None: + """A tool raises; the request sent after that is held behind the calling response, which never ends.""" + + def scenario(sim: OpenAISimulation) -> None: + sim.create_response() + sim.call_tool() + sim.finish_tool(outcome='error') + sim.create_response() + sim.settle() + + reproduce('SIM-15', OpenAISimulation(openai=OpenAIOptions(transcription=False, dialect='xai')), scenario) + + +@known('SIM-37') +def test_known_turn_cleared_while_spoken_loses_its_transcript_xai() -> None: + def scenario(sim: OpenAISimulation) -> None: + sim.send_audio() + sim.speech_start() + sim.clear_audio() + sim.settle() + + reproduce('SIM-37', OpenAISimulation(openai=OpenAIOptions(dialect='xai')), scenario) + + +@known('SIM-38') +def test_known_stop_server_vad_takes_back_holds_history() -> None: + def scenario(sim: OpenAISimulation) -> None: + sim.send_audio() + sim.speech_start() + sim.speech_stop(commit=False) + sim.speech_start() + sim.speech_stop() + sim.settle() + + reproduce('SIM-38', OpenAISimulation(), scenario) + + +@known('E') +def test_known_gemini_spoken_turn_inserted_into_recorded_history() -> None: + def scenario(sim: GeminiSimulation) -> None: + sim.send_audio() + sim.send_text(respond=False) + sim.user_speaks(finished=False, deliver=False) + sim.settle() + + reproduce('E', GeminiSimulation(), scenario) + + +@known('SIM-11') +def test_known_gemini_turn_spoken_before_a_reply_filed_before_it() -> None: + def scenario(sim: GeminiSimulation) -> None: + sim.send_text() + sim.send_audio() + sim.settle() + sim.user_speaks(finished=False, deliver=False) + sim.settle() + + reproduce('SIM-11', GeminiSimulation(), scenario) + + +@known('SIM-2a') +def test_known_gemini_turn_sent_during_a_tool_round_recorded_ahead_of_it() -> None: + def scenario(sim: GeminiSimulation) -> None: + sim.send_text() + sim.call_tools(deliver=False) + sim.send_text(respond=False) + sim.settle() + + reproduce('SIM-2a', GeminiSimulation(), scenario) + + +@known('SIM-21') +def test_known_request_whose_refusal_is_lost_keeps_its_reservation_xai() -> None: + def scenario(sim: OpenAISimulation) -> None: + sim.reject_next('response') + sim.create_response() + sim.send_text() + sim.drop() + sim.settle() + + reproduce('SIM-21', OpenAISimulation(openai=OpenAIOptions(dialect='xai')), scenario) def test_gemini_typed_turn_lost_to_a_drop_is_settled() -> None: @@ -764,9 +878,8 @@ def test_gemini_typed_turn_lost_to_a_drop_is_settled() -> None: run_clean(GeminiSimulation(), scenario) -@known('SIM-2a') -def test_known_turn_sent_before_reply_content_recorded_ahead_of_it() -> None: - """The first reply ended empty before the second turn was sent, but the client hadn't read it yet.""" +def test_turn_sent_after_a_reply_ended_unread_is_recorded_after_it() -> None: + """The first reply ended empty before the second turn was sent, but the client hadn't read it yet (SIM-2a, fixed by the session core).""" def scenario(sim: OpenAISimulation) -> None: sim.send_text() @@ -774,18 +887,19 @@ def test_known_turn_sent_before_reply_content_recorded_ahead_of_it() -> None: sim.send_text(respond=False) sim.settle() - reproduce('SIM-2a', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) -@known('SIM-2b') -def test_known_wait_returns_before_a_started_vad_reply() -> None: +def test_wait_waits_for_a_started_vad_reply() -> None: + """SIM-2b, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_audio() sim.speech_start() sim.speech_stop() sim.wait_for_reply() - reproduce('SIM-2b', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) @known('SIM-2b') @@ -797,15 +911,16 @@ def test_known_wait_returns_before_a_delegated_reply() -> None: reproduce('SIM-2b', LiveSimulation(), scenario) -@known('SIM-3') -def test_known_reconnect_does_not_ask_again_for_an_unstarted_reply() -> None: +def test_reconnect_asks_again_for_an_unstarted_reply() -> None: + """SIM-3, fixed by the session core.""" + def scenario(sim: OpenAISimulation) -> None: sim.send_text() sim.send_text() sim.drop() sim.settle() - reproduce('SIM-3', OpenAISimulation(), scenario) + run_clean(OpenAISimulation(), scenario) @known('SIM-4') @@ -1191,20 +1306,23 @@ def test_baseline_server_vad_answers_each_turn_heard_during_a_reply() -> None: ) -def test_scenario_xai_hand_commit_while_vad_hears_the_user_is_misplaced() -> None: - """SIM-27 on xAI: the hand-committed turn is recorded where its audio began, before a reply that ended first.""" - with OpenAISimulation(strict=False, openai=OpenAIOptions(dialect='xai', transcription=True)) as s: - s.create_response() - s.send_audio() - s.settle() - s.speech_start(deliver=False) - s.commit_audio() - s.settle() - assert ('SIM-27', 'history.order') in s.checker.known_hits +def test_xai_hand_commit_while_vad_hears_the_user_is_filed_in_place() -> None: + """SIM-27 on xAI, fixed by the session core: the hand-committed turn goes where xAI committed it, after the reply + that ended first.""" + + def scenario(sim: OpenAISimulation) -> None: + sim.create_response() + sim.send_audio() + sim.settle() + sim.speech_start(deliver=False) + sim.commit_audio() + sim.settle() + + run_clean(OpenAISimulation(openai=OpenAIOptions(dialect='xai', transcription=True)), scenario) -def test_scenario_core_holds_a_reply_behind_a_turn_without_its_transcript() -> None: - """SIM-25 in the new core: the reply after a reconnect waits behind the untranscribed turn, so it goes missing.""" +def test_scenario_turn_without_its_transcript_lost_at_a_reconnect() -> None: + """SIM-25 in the session core: the untranscribed turn is lost at the reconnect, though the reply after it is not.""" with OpenAISimulation(strict=False, openai=OpenAIOptions(dialect='xai', transcription=True)) as s: s.create_response() s.send_audio() @@ -1216,8 +1334,7 @@ def test_scenario_core_holds_a_reply_behind_a_turn_without_its_transcript() -> N s.settle() s.create_response() s.settle() - assert s.checker.shadow is not None - assert ('SIM-25', 'shadow.response.missing') in s.checker.shadow.known_hits + assert ('SIM-25', 'history.turn_missing') in s.checker.known_hits def test_scenario_late_cancel_drops_a_tool_call_and_ends_the_wait() -> None: @@ -1575,20 +1692,23 @@ def test_scenario_gemini_async_speech_in_flight_at_the_result_is_accepted() -> N assert '8760-accepted' in {finding_id for finding_id, _ in s.checker.known_hits} -def test_scenario_tool_result_awaiting_its_reply_when_barged_in_on_without_a_vad_reply() -> None: - """SIM-32 also drops the reply a cancelled response's tool results were waiting for.""" - sim = OpenAISimulation( - strict=False, - openai=OpenAIOptions(dialect='azure', transcription=False, vad_interrupts=True, vad_responds=False), +def test_tool_result_awaiting_its_reply_when_barged_in_on_without_a_vad_reply() -> None: + """SIM-32, fixed by the session core: the reply a cancelled response's tool results wait for is still owed.""" + + def scenario(sim: OpenAISimulation) -> None: + sim.send_image(respond=True) + sim.call_tool() + sim.tick(ticks=0) + sim.tick(ticks=0) + sim.speech_start(deliver=False) + sim.settle() + + run_clean( + OpenAISimulation( + openai=OpenAIOptions(dialect='azure', transcription=False, vad_interrupts=True, vad_responds=False) + ), + scenario, ) - with sim as s: - s.send_image(respond=True) - s.call_tool() - s.tick(ticks=0) - s.tick(ticks=0) - s.speech_start(deliver=False) - s.settle() - assert ('SIM-32', 'wait.hang') in s.checker.known_hits def test_scenario_gemini_tool_result_sent_on_a_dropped_connection() -> None: @@ -1631,8 +1751,7 @@ def test_scenario_a_provider_reply_before_any_metadata_echo_is_accepted() -> Non s.finish(deliver=False) s.send_text(respond=True) s.settle() - assert s.checker.shadow is not None - assert ('SIM-24', 'shadow.history.order') in s.checker.shadow.known_hits + assert ('SIM-24', 'history.order') in s.checker.known_hits def test_scenario_a_call_the_session_refuses_over_a_limit_is_left_out() -> None: @@ -1841,15 +1960,3 @@ def test_xai_server_vad_answers_every_turn() -> None: vad: dict[str, Any] = {'type': 'server_vad', 'create_response': False} assert not ServerSession(index=0, socket=socket, turn_detection=vad).create_response assert ServerSession(index=0, socket=socket, turn_detection=vad, answers_every_turn=True).create_response - - -def _run_to_the_end(finding_id: str, sim: Simulation, scenario: Callable[[Any], object]) -> None: - """Run a pinned scenario with its finding tolerated, so the shadow core is judged to the end of it.""" - run_tolerant(sim, scenario) - - -@pytest.mark.parametrize('scenario', sorted(name for name in dict(globals()) if name.startswith('test_known_'))) -def test_the_core_gets_the_known_findings_right(scenario: str, monkeypatch: pytest.MonkeyPatch) -> None: - """Every pinned scenario again, run to its end: the shadow core breaks no invariant outside `SHADOW_PENDING`.""" - monkeypatch.setitem(globals(), 'reproduce', _run_to_the_end) - globals()[scenario]() diff --git a/tests/realtime/test_openai.py b/tests/realtime/test_openai.py index 172c5d16ca..65b39027bb 100644 --- a/tests/realtime/test_openai.py +++ b/tests/realtime/test_openai.py @@ -40,6 +40,7 @@ from pydantic_ai.messages import ( ModelResponse, NativeToolCallPart, NativeToolReturnPart, + PartEndEvent, RealtimeSessionErrorEvent, RetryPromptPart, SpeechPart, @@ -66,8 +67,10 @@ from pydantic_ai.realtime import ( RealtimeOutputSpeechStartEvent, RealtimeSession, RealtimeSessionReconnectEvent, + RealtimeTurnCompleteEvent, WebRTCSession, ) +from pydantic_ai.realtime._instrumentation import SessionInstrumentation from pydantic_ai.realtime._lifecycle import UserTurnEnded, UserTurnStarted from pydantic_ai.realtime._openai_protocol import ( RealtimeHandshakeError, @@ -100,7 +103,7 @@ from pydantic_ai.realtime.profiles import merge_realtime_profile from pydantic_ai.realtime.xai import map_conversation_event as _map_conversation_wire_event from pydantic_ai.settings import ThinkingLevel, ToolChoice, ToolOrOutput from pydantic_ai.tools import ToolDefinition -from pydantic_ai.usage import RequestUsage +from pydantic_ai.usage import RequestUsage, UsageLimits from ..conftest import IsDatetime, try_import from .test_session import FakeRealtimeModel, make_tool_manager @@ -4880,7 +4883,6 @@ async def test_image_history_cap_evicts_the_oldest_image_the_provider_added() -> ] -@pytest.mark.shadow_divergence(reason='SIM-2a: turns sent after the reply started are filed after it, not before') @pytest.mark.anyio async def test_text_turns_queued_behind_a_reply_are_answered_once_and_waited_for_once() -> None: """Turns sent while a reply is in flight share one deferred response, and `wait_for_reply()` returns after it. @@ -5062,9 +5064,6 @@ async def test_single_tool_call_frames_and_timing_are_unchanged_by_batching(stat await session.wait_for_reply() -@pytest.mark.shadow_divergence( - reason="SIM-2a: a turn sent after the batch's answer was requested is filed after that answer" -) @pytest.mark.anyio async def test_tool_batch_response_create_counts_as_one_request() -> None: """A batch's `response.create` is one request, however many outputs it follows. @@ -5128,3 +5127,372 @@ async def test_tool_batch_response_create_counts_as_one_request() -> None: ws.push(frame) with anyio.fail_after(5): await waiting + + +@pytest.mark.anyio +async def test_a_reply_waits_for_the_transcript_of_the_turn_before_it_for_a_while( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """History holds the reply back until the spoken turn before it is transcribed, but not for longer than that hold.""" + monkeypatch.setattr('pydantic_ai.realtime._session._TRANSCRIPT_HOLD_SECONDS', 0.05) + ws = _QueuedWebSocket() + connection = OpenAIRealtimeConnection(ws) # type: ignore[arg-type] + session = RealtimeSession( + connection, model=FakeRealtimeModel(connection, system='openai'), tool_manager=make_tool_manager() + ) + async with session: + frames: list[dict[str, Any]] = [ + {'type': 'input_audio_buffer.speech_started', 'item_id': 'item_u1', 'audio_start_ms': 0}, + {'type': 'input_audio_buffer.speech_stopped', 'item_id': 'item_u1', 'audio_end_ms': 500}, + {'type': 'input_audio_buffer.committed', 'item_id': 'item_u1', 'previous_item_id': None}, + { + 'type': 'response.created', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'in_progress', 'output': []}, + }, + { + 'type': 'response.output_audio_transcript.delta', + 'response_id': 'resp_1', + 'item_id': 'item_a1', + 'delta': 'Hello.', + }, + { + 'type': 'response.done', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'completed', 'output': []}, + }, + ] + for frame in frames: + ws.push(frame) + async for event in session: # pragma: no branch + if isinstance(event, RealtimeTurnCompleteEvent): + break + # The reply is done, but held back behind the turn before it, until the hold runs out. + assert session.all_messages() == [] + with anyio.fail_after(5): + while not session.all_messages(): + await asyncio.sleep(0.01) + # The transcript coming after all changes nothing already recorded. + ws.push( + { + 'type': 'conversation.item.input_audio_transcription.completed', + 'item_id': 'item_u1', + 'content_index': 0, + 'transcript': 'Hi.', + } + ) + for _ in range(20): + await asyncio.sleep(0) + assert [[getattr(part, 'transcript', None) for part in message.parts] for message in session.all_messages()] == [ + [None], + ['Hello.'], + ] + + +@pytest.mark.anyio +async def test_a_server_vad_reply_at_the_request_limit_is_not_counted_twice() -> None: + """The core counts the reply as its terminal frame arrives; the limit check it makes then mustn't count it again.""" + ws = _QueuedWebSocket() + connection = OpenAIRealtimeConnection(ws) # type: ignore[arg-type] + session = RealtimeSession( + connection, + model=FakeRealtimeModel(connection, system='openai'), + tool_manager=make_tool_manager(), + usage_limits=UsageLimits(request_limit=1), + ) + async with session: + frames: list[dict[str, Any]] = [ + {'type': 'input_audio_buffer.speech_started', 'item_id': 'item_u1', 'audio_start_ms': 0}, + {'type': 'input_audio_buffer.speech_stopped', 'item_id': 'item_u1', 'audio_end_ms': 500}, + {'type': 'input_audio_buffer.committed', 'item_id': 'item_u1', 'previous_item_id': None}, + { + 'type': 'response.created', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'in_progress', 'output': []}, + }, + { + 'type': 'response.done', + 'response': { + 'id': 'resp_1', + 'object': 'realtime.response', + 'status': 'completed', + 'output': [], + 'usage': {'input_tokens': 5, 'output_tokens': 0, 'total_tokens': 5}, + }, + }, + ] + for frame in frames: + ws.push(frame) + await session.wait_for_reply() + for _ in range(20): + await asyncio.sleep(0) + assert session.usage.requests == 1 + + +@pytest.mark.anyio +async def test_a_chat_spans_input_is_the_conversation_ahead_of_its_response(monkeypatch: pytest.MonkeyPatch) -> None: + """The core records a response as its terminal arrives, before the span ends: the span's input must not hold it.""" + ended: list[tuple[list[ModelMessage], ModelResponse | None]] = [] + end_chat_span = SessionInstrumentation.end_chat_span + + def record( + self: SessionInstrumentation, input_messages: list[ModelMessage], response: ModelResponse | None + ) -> None: + ended.append((input_messages, response)) + end_chat_span(self, input_messages, response) + + monkeypatch.setattr(SessionInstrumentation, 'end_chat_span', record) + ws = _QueuedWebSocket() + connection = OpenAIRealtimeConnection(ws) # type: ignore[arg-type] + session = RealtimeSession( + connection, model=FakeRealtimeModel(connection, system='openai'), tool_manager=make_tool_manager() + ) + async with session: + await session.send('Hi.') + frames: list[dict[str, Any]] = [ + { + 'type': 'response.created', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'in_progress', 'output': []}, + }, + { + 'type': 'response.output_audio_transcript.delta', + 'response_id': 'resp_1', + 'item_id': 'item_a1', + 'delta': 'Hello.', + }, + { + 'type': 'response.done', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'completed', 'output': []}, + }, + ] + for frame in frames: + ws.push(frame) + await session.wait_for_reply() + for _ in range(20): + await asyncio.sleep(0) + [(input_messages, response)] = ended + assert response is not None + assert [type(message).__name__ for message in input_messages] == ['ModelRequest'] + + +class _DropAfterFrames(FakeWebSocket): + """Yields its frames after the handshake, then drops abnormally.""" + + async def __aiter__(self) -> AsyncIterator[Any]: + while self._incoming: + yield self._incoming.pop(0) + raise rt_openai.websockets.ConnectionClosed(None, None) + + +@pytest.mark.parametrize('transcript', ['Weather?', None]) +async def test_reconnect_replays_a_reply_held_back_behind_a_missing_transcript( + monkeypatch: pytest.MonkeyPatch, transcript: str | None +) -> None: + """History holds the reply back for the spoken turn's transcript, but the new socket must still get both. + + A turn with no transcript at all has no words to replay (replay leaves media behind), so it goes out as the + marker that says the user spoke. + """ + frames: list[dict[str, Any]] = [ + {'type': 'input_audio_buffer.speech_started', 'item_id': 'item_u1', 'audio_start_ms': 0}, + {'type': 'input_audio_buffer.speech_stopped', 'item_id': 'item_u1', 'audio_end_ms': 500}, + {'type': 'input_audio_buffer.committed', 'item_id': 'item_u1', 'previous_item_id': None}, + *( + [{'type': 'conversation.item.input_audio_transcription.delta', 'item_id': 'item_u1', 'delta': transcript}] + if transcript + else [] + ), + { + 'type': 'response.created', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'in_progress', 'output': []}, + }, + { + 'type': 'response.output_audio_transcript.delta', + 'response_id': 'resp_1', + 'item_id': 'item_a1', + 'delta': 'Sunny.', + }, + { + 'type': 'response.done', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'completed', 'output': []}, + }, + ] + dropped = _DropAfterFrames([_created(), _updated(), *map(json.dumps, frames)]) + fresh = FakeWebSocket([_created(), _updated()]) + monkeypatch.setattr(rt_openai.websockets, 'connect', _ConnectSequence([dropped, fresh])) + model = OpenAIRealtimeModel( + 'gpt-realtime', + provider=OpenAIProvider(api_key='k'), + settings={'reconnect': {'base_delay': 0.0, 'max_attempts': 1}}, + ) + + async with Agent().realtime(model).session() as session: + with anyio.fail_after(5): + async for event in session: # pragma: no branch + if isinstance(event, RealtimeSessionReconnectEvent): + break + + replayed = [ + json.loads(frame)['item'] for frame in fresh.sent if json.loads(frame)['type'] == 'conversation.item.create' + ] + assert [ + (item['role'], item['content'][0].get('text') or item['content'][0].get('transcript')) for item in replayed + ] == [ + ('user', transcript or '[The user spoke; no transcript is available.]'), + ('assistant', 'Sunny.'), + ] + + +@pytest.mark.anyio +async def test_a_conversation_id_resolved_late_reaches_the_core_history() -> None: + """`RealtimeSession.conversation` mints the id late; the replies the session core recorded before carry it too.""" + ws = _QueuedWebSocket() + connection = OpenAIRealtimeConnection(ws) # type: ignore[arg-type] + session = RealtimeSession( + connection, model=FakeRealtimeModel(connection, system='openai'), tool_manager=make_tool_manager() + ) + frames: list[dict[str, Any]] = [ + { + 'type': 'response.created', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'in_progress', 'output': []}, + }, + { + 'type': 'response.output_audio_transcript.delta', + 'response_id': 'resp_1', + 'item_id': 'item_a1', + 'delta': 'Hello.', + }, + { + 'type': 'response.done', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'completed', 'output': []}, + }, + ] + async with session: + await session.send('Hi.') + for frame in frames: + ws.push(frame) + await session.wait_for_reply() + conversation = session.conversation + + assert conversation.conversation_id is not None + assert [message.conversation_id for message in conversation.messages] == [conversation.conversation_id] * 2 + + +@pytest.mark.anyio +async def test_audio_that_fails_to_go_out_is_taken_back_from_the_core_too() -> None: + class _FailingAudio(_QueuedWebSocket): + async def send(self, data: str) -> None: + raise OSError('gone') # (the audio is all this session sends) + + ws = _FailingAudio() + connection = OpenAIRealtimeConnection(ws) # type: ignore[arg-type] + session = RealtimeSession( + connection, + model=FakeRealtimeModel(connection, system='openai'), + tool_manager=make_tool_manager(), + audio_retention='input_audio', + ) + async with session: + with pytest.raises(RealtimeError): + await session.send_audio(b'\x01\x00' * 100) + core = session._core # pyright: ignore[reportPrivateUsage] + assert core is not None + assert not core._input_audio # pyright: ignore[reportPrivateUsage] + + +@pytest.mark.anyio +async def test_a_failed_send_takes_back_its_own_audio_while_another_waits_to_go_out() -> None: + """Two chunks sent at once go out one after the other: the first failing takes back its own audio, not the second's.""" + + class _FirstAudioFails(_QueuedWebSocket): + def __init__(self) -> None: + super().__init__() + self.sending = asyncio.Event() + self.fail = asyncio.Event() + self.sends = 0 + + async def send(self, data: str) -> None: + self.sends += 1 + if self.sends == 1: + self.sending.set() + await self.fail.wait() + raise OSError('gone') + + ws = _FirstAudioFails() + connection = OpenAIRealtimeConnection(ws) # type: ignore[arg-type] + session = RealtimeSession( + connection, + model=FakeRealtimeModel(connection, system='openai'), + tool_manager=make_tool_manager(), + audio_retention='input_audio', + ) + first, second = b'\x01\x00' * 100, b'\x02\x00' * 50 + async with session: + first_send = asyncio.create_task(session.send_audio(first)) + await ws.sending.wait() + second_send = asyncio.create_task(session.send_audio(second)) + await asyncio.sleep(0) + ws.fail.set() + with pytest.raises(RealtimeError): + await first_send + await second_send + core = session._core # pyright: ignore[reportPrivateUsage] + assert core is not None + assert bytes(core._input_audio) == second # pyright: ignore[reportPrivateUsage] + + +@pytest.mark.anyio +async def test_retained_audio_is_kept_once_when_the_core_keeps_history() -> None: + """The session core's history keeps retained audio; the session's own record (what tools see) lets its copy go. + + The part streamed as an event keeps its audio, so recorded audio is bounded by `retain_audio_max_seconds` once + rather than twice. + """ + ws = _QueuedWebSocket() + connection = OpenAIRealtimeConnection(ws) # type: ignore[arg-type] + session = RealtimeSession( + connection, + model=FakeRealtimeModel(connection, system='openai'), + tool_manager=make_tool_manager(), + audio_retention='output_audio', + ) + frames: list[dict[str, Any]] = [ + { + 'type': 'response.created', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'in_progress', 'output': []}, + }, + { + 'type': 'response.output_audio.delta', + 'response_id': 'resp_1', + 'item_id': 'item_a1', + 'delta': base64.b64encode(b'\x01\x00' * 100).decode('ascii'), + }, + { + 'type': 'response.output_audio_transcript.delta', + 'response_id': 'resp_1', + 'item_id': 'item_a1', + 'delta': 'Hello.', + }, + { + 'type': 'response.done', + 'response': {'id': 'resp_1', 'object': 'realtime.response', 'status': 'completed', 'output': []}, + }, + ] + async with session: + await session.send('Hi.') + for frame in frames: + ws.push(frame) + streamed: list[SpeechPart] = [] + async for event in session: # pragma: no branch + if isinstance(event, PartEndEvent) and isinstance(event.part, SpeechPart): + streamed.append(event.part) + if isinstance(event, RealtimeTurnCompleteEvent): + break + legacy = session._own_messages() # pyright: ignore[reportPrivateUsage] + history = session.all_messages() + + def speech(messages: list[ModelMessage]) -> list[SpeechPart]: + return [part for message in messages for part in message.parts if isinstance(part, SpeechPart)] + + assert [part.transcript for part in speech(history)] == ['Hello.'] + assert all(part.audio is not None for part in speech(history)) + assert [part.transcript for part in speech(legacy)] == ['Hello.'] + assert all(part.audio is None for part in speech(legacy)) + assert [part.audio is not None for part in streamed] == [True] diff --git a/tests/realtime/test_openai_lifecycle.py b/tests/realtime/test_openai_lifecycle.py index cc7c078b2b..5299eec5e6 100644 --- a/tests/realtime/test_openai_lifecycle.py +++ b/tests/realtime/test_openai_lifecycle.py @@ -19,6 +19,7 @@ from pydantic_ai.realtime._lifecycle import ( InputAdded, InputLost, LifecycleEvent, + OutputItemDetails, ResponseEnded, ResponseRequestRefused, ResponseStarted, @@ -737,7 +738,7 @@ async def test_xai_places_a_spoken_turn_when_it_adds_its_item() -> None: [ 'RealtimeInputSpeechStartEvent', UserTurnStarted(turn_id='item_u1'), - UserTurnEnded(turn_id='item_u1'), + UserTurnEnded(turn_id='item_u1', still_speaking=True), ResponseStarted(response_id='resp_1', user_turn_id='item_u1'), UserTurnDiscarded(turn_id='item_u1'), 'RealtimeInputSpeechEndEvent', @@ -827,6 +828,32 @@ async def test_a_reconnect_loses_a_request_the_connection_no_longer_counts_as_ac assert [event for event in events if isinstance(event, InputLost)] == snapshot([InputLost(input_ids=(1,))]) +async def test_an_assistant_messages_phase_is_kept_for_its_part() -> None: + """gpt-realtime-2.x labels each assistant message `commentary` or `final_answer`.""" + + def item_added(item: object) -> dict[str, Any]: + return {'type': 'response.output_item.added', 'response_id': 'resp_1', 'item': item} + + message: dict[str, Any] = {'type': 'message', 'role': 'assistant', 'content': []} + stream = Stream( + created('resp_1'), + item_added({**message, 'id': 'item_a1', 'phase': 'commentary'}), + item_added({**message, 'id': 'item_a2'}), + item_added({**message, 'phase': 'final_answer'}), + item_added('not an item'), + { + 'type': 'response.output_item.added', + 'response_id': '', + 'item': {**message, 'id': 'item_a3', 'phase': 'final_answer'}, + }, + done('resp_1'), + ) + events = [event for event in await stream.rest() if isinstance(event, OutputItemDetails)] + assert events == snapshot( + [OutputItemDetails(response_id='resp_1', item_id='item_a1', provider_details={'phase': 'commentary'})] + ) + + async def test_what_we_sent_ahead_of_our_commit_joins_the_conversation_before_its_turn() -> None: """The provider handles frames in order: inputs not acknowledged yet when our commit went out came first.""" stream = Stream( diff --git a/tests/realtime/test_parity_ws.py b/tests/realtime/test_parity_ws.py index 6258154ad3..6b448040a4 100644 --- a/tests/realtime/test_parity_ws.py +++ b/tests/realtime/test_parity_ws.py @@ -84,6 +84,8 @@ class RealtimeParityCase: Where it does, the spoken scenario keeps a silent microphone running so the clock can run out, the way a real call would. """ + labels_phase: bool = False + """Whether the model labels each assistant message `commentary` or `final_answer` (its `phase`).""" # Adding a supported model generation is one row. Gateway routes intentionally have their own rows: @@ -99,6 +101,7 @@ REALTIME_PARITY_CASES = [ supports_manual_turn_control=True, supports_interruption=True, supports_native_tools=False, + labels_phase=True, ), RealtimeParityCase( id='openai-previous', @@ -333,6 +336,13 @@ async def test_text_tool_round_parity( assert isinstance(final_part, (SpeechPart, TextPart)) answer = final_part.transcript if isinstance(final_part, SpeechPart) else final_part.content assert answer is not None and 'fog' in answer.lower() + if case.labels_phase: + # Kept on the part, under the key the Responses API reads it from when the history is handed off. + assert [part.provider_details for part in messages[1].parts if isinstance(part, TextPart)] == [ + {'phase': 'commentary'} + ] + assert final_part.provider_details == {'phase': 'final_answer'} + assert final_part.provider_name == 'openai' assert session.usage.requests >= 1 assert session.usage.input_tokens >= 0 assert session.usage.output_tokens >= 0 @@ -386,11 +396,6 @@ async def test_audio_tool_round_parity( model like GPT-Live can run at all. Both must produce the same history. """ case, provider, cassette = parity_ws_cassette - if case.model_kind == 'openai': - # OpenAI's server VAD cancels an empty reply when the user goes on speaking, and commits the - # second turn after that reply ended. The shadow core files the turn after it, in the provider's - # order; the current core files it where the user started speaking (SIM-11). - request.applymarker(pytest.mark.shadow_divergence(reason='SIM-11: spoken turn in provider order')) model = _model(case, provider) profile = model.profile assert profile.get('synthesizes_turn_boundary', False) is case.synthesizes_turn_boundary diff --git a/tests/realtime/test_session_core.py b/tests/realtime/test_session_core.py index 0a9d20cadd..1fafe322ec 100644 --- a/tests/realtime/test_session_core.py +++ b/tests/realtime/test_session_core.py @@ -8,9 +8,12 @@ that follows a tool round through to its answer. from __future__ import annotations as _annotations +from collections import OrderedDict +from collections.abc import Iterator from decimal import Decimal from typing import Any +import pytest from inline_snapshot import snapshot from pydantic_ai.messages import ( @@ -20,6 +23,7 @@ from pydantic_ai.messages import ( RealtimeInputSpeechEndEvent, RealtimeInputSpeechStartEvent, RealtimeInputTranscriptionErrorEvent, + RealtimeSessionReconnectEvent, SpeechPart, TextPart, ToolCallPart, @@ -29,6 +33,7 @@ from pydantic_ai.messages import ( from pydantic_ai.realtime._core import ( AudioCleared, AudioSent, + AudioUnsent, Closed, CoreInput, ExchangeAbandoned, @@ -40,10 +45,12 @@ from pydantic_ai.realtime._core import ( SessionCore, ToolCallRefused, ToolReturned, + TranscriptOverdue, ) from pydantic_ai.realtime._lifecycle import ( InputAdded, InputLost, + OutputItemDetails, ResponseEnded, ResponseRequestRefused, ResponseStarted, @@ -51,8 +58,10 @@ from pydantic_ai.realtime._lifecycle import ( UserTurnEnded, UserTurnStarted, ) +from pydantic_ai.realtime._retained_audio import RetainedAudioBudget from pydantic_ai.realtime.codec import ( AudioDelta, + ConversationItemCreated, InputRejected, InputTranscript, OutputTranscript, @@ -326,7 +335,7 @@ def test_a_turn_that_joins_while_it_is_still_spoken_ends_with_the_speech() -> No AudioSent(data=b'\x01\x00'), UserTurnStarted(turn_id='u1'), RealtimeInputSpeechStartEvent(item_id='u1'), - UserTurnEnded(turn_id='u1'), + UserTurnEnded(turn_id='u1', still_speaking=True), AudioSent(data=b'\x02\x00' * 4), ) assert session_core.all_messages() == [] @@ -341,11 +350,11 @@ def test_a_turn_that_joins_while_it_is_still_spoken_ends_with_the_speech() -> No session_core, RealtimeInputSpeechStartEvent(item_id='u2'), UserTurnStarted(turn_id='u2'), - UserTurnEnded(turn_id='u2'), + UserTurnEnded(turn_id='u2', still_speaking=True), UserTurnDiscarded(turn_id='u2'), RealtimeInputSpeechStartEvent(item_id='u3'), UserTurnStarted(turn_id='u3'), - UserTurnEnded(turn_id='u3'), + UserTurnEnded(turn_id='u3', still_speaking=True), Closed(), ) assert summary(session_core.all_messages()) == snapshot(['{user:None+audio}', '{user:None}', '{user:None}']) @@ -471,3 +480,628 @@ def test_a_turn_whose_transcript_can_no_longer_be_read_ends_with_what_it_has() - assert session_core.all_messages() == [] feed(session_core, ReceiveEnded()) assert summary(session_core.all_messages()) == snapshot(['{user:Good}', 'r1 [assistant:Hm.] complete stop']) + + +def test_an_output_items_details_go_on_its_part() -> None: + """OpenAI's `phase`, for one: commentary on the way to a tool call, and the final answer after.""" + session_core = feed( + core(), + started('r1'), + OutputItemDetails(response_id='r1', item_id='i1', provider_details={'phase': 'commentary'}), + OutputItemDetails(response_id='r1', item_id='i2', provider_details={'phase': 'final_answer'}), + OutputItemDetails(response_id='unknown', item_id='i3', provider_details={'phase': 'commentary'}), + said('r1', 'Let me check.', item_id='i1', output_text=True), + said('r1', 'Sunny.'), + said('r1', ' Warm, too.', item_id='i2'), + said('r1', 'No details.', item_id='i4'), + ended('r1'), + ) + [response] = session_core.all_messages() + assert isinstance(response, ModelResponse) + assert [(part.provider_name, part.provider_details) for part in response.parts] == snapshot( + [('openai', {'phase': 'commentary'}), ('openai', {'phase': 'final_answer'}), (None, None)] + ) + + +def test_a_turn_whose_transcript_is_overdue_is_recorded_with_what_it_has() -> None: + session_core = feed( + core(), + UserTurnStarted(turn_id='u1'), + UserTurnEnded(turn_id='u1'), + InputTranscript('Partly', item_id='u1'), + started('r1'), + said('r1', 'Hm.'), + ended('r1'), + ) + assert session_core.transcript_holding_history() == 'u1' + assert session_core.all_messages() == [] + feed(session_core, TranscriptOverdue(turn_id='u1'), TranscriptOverdue(turn_id='unknown')) + assert session_core.transcript_holding_history() is None + # The transcript, if it does come after all, changes nothing recorded. + feed(session_core, InputTranscript(' said.', item_id='u1', is_final=True)) + assert summary(session_core.all_messages()) == snapshot(['{user:Partly}', 'r1 [assistant:Hm.] complete stop']) + + +def test_only_a_turn_with_something_ready_after_it_holds_history() -> None: + session_core = feed(core(), UserTurnStarted(turn_id='u1'), UserTurnEnded(turn_id='u1'), started('r1')) + # Nothing after the turn is ready yet: there is nothing to hold back. + assert session_core.transcript_holding_history() is None + feed(session_core, ended('r1', 'lost')) + assert session_core.transcript_holding_history() is None # (lost before it said anything: not recorded) + feed(session_core, InputSent(input_id=0, request=text_request('Hi.')), InputAdded(input_id=0)) + assert session_core.transcript_holding_history() == 'u1' + # Nor does a response under way, or a turn still spoken, ahead of it. + other = feed(core(), started('r1'), InputSent(input_id=0, request=text_request('Hi.')), InputAdded(input_id=0)) + assert other.transcript_holding_history() is None + spoken = feed( + core(), + UserTurnStarted(turn_id='u1'), + UserTurnEnded(turn_id='u1', still_speaking=True), + InputSent(input_id=0, request=text_request('Hi.')), + InputAdded(input_id=0), + ) + assert spoken.transcript_holding_history() is None + + +def test_clearing_the_audio_drops_a_turn_that_had_not_joined() -> None: + session_core = feed( + core(input_transcription_enabled=False), + UserTurnStarted(turn_id='u1'), + UserTurnEnded(turn_id='u1', still_speaking=True), + UserTurnStarted(turn_id='u2'), + AudioCleared(), + Closed(), + ) + # The turn that joined stays; the one still being said is gone, so closing records nothing for it. + assert summary(session_core.all_messages()) == snapshot(['{user:None}']) + + +def test_a_response_under_way_when_reading_ends_is_recorded_as_cut_off() -> None: + session_core = feed(core(), started('r1'), said('r1', 'Partly'), ReceiveEnded()) + assert summary(session_core.all_messages()) == snapshot(['r1 [assistant:Partly] interrupted None']) + + +def test_a_withdrawn_input_is_let_go() -> None: + """An evicted image, say: nothing keeps what it carried alive, placed or not.""" + session_core = feed( + core(), + InputSent(input_id=0, request=text_request('Placed.')), + InputAdded(input_id=0), + InputSent(input_id=1, request=text_request('Not yet.')), + InputWithdrawn(input_ids=(0, 1, 2)), + InputAdded(input_id=1), + ) + assert session_core.all_messages() == [] + assert not session_core._inputs and not session_core._unplaced # pyright: ignore[reportPrivateUsage] + assert session_core._placed == [] # pyright: ignore[reportPrivateUsage] + + +def test_a_repeated_or_replayed_call_is_recorded_once() -> None: + """xAI's resumption can repeat a call of a response still open, and replay items history already has.""" + call = ToolCall('call_1', tool_name='lookup', args='{}', response_id='r1') + session_core = feed( + core(), + started('r1'), + call, + call, + ConversationItemCreated(item_id='item_old', tool_call_id='call_old', replayed=True), + ConversationItemCreated(tool_call_id='call_older', replayed=True), + ConversationItemCreated(item_id='item_new'), + ToolCall('call_old', tool_name='lookup', args='{}', response_id='r1'), + said('r1', 'Replayed.', item_id='item_old'), + ended('r1'), + ) + assert summary(session_core.all_messages()) == snapshot(['r1 [call:call_1] complete stop']) + + +def test_a_new_call_whose_id_matches_a_replayed_item_is_recorded() -> None: + """Item ids and call ids are different kinds of id: one replayed as an item says nothing about a call.""" + session_core = feed( + core(), + ConversationItemCreated(item_id='shared', replayed=True), + started('r1'), + ToolCall('shared', tool_name='lookup', args='{}', response_id='r1'), + ended('r1'), + ) + assert summary(session_core.all_messages()) == snapshot(['r1 [call:shared] complete stop']) + + +def test_a_turn_still_spoken_when_the_connection_is_replaced_is_over() -> None: + """xAI added it at speech start; the speech end was lost with the old connection, and the new one won't hear it.""" + session_core = feed( + core(input_transcription_enabled=False, retain_input_audio=True), + RealtimeInputSpeechStartEvent(item_id='u1'), + UserTurnStarted(turn_id='u1'), + UserTurnEnded(turn_id='u1', still_speaking=True), + AudioSent(data=b'\x01\x00'), + RealtimeSessionReconnectEvent(state_restored=True), + RealtimeSessionReconnectEvent(state_restored=True), + started('r1'), + said('r1', 'Hm.'), + ended('r1'), + UserTurnStarted(turn_id='u2'), + UserTurnEnded(turn_id='u2'), + ) + # It keeps the audio sent for it, which the next turn doesn't inherit. + assert summary(session_core.all_messages()) == snapshot( + ['{user:None+audio}', 'r1 [assistant:Hm.] complete stop', '{user:None}'] + ) + + +# --- retained audio, bounded by `retain_audio_max_seconds` (as `RealtimeSession` bounds it) --------------- + + +def _tenth_of_a_second(value: int) -> bytes: + """A tenth of a second of PCM16 at 24 kHz, every byte `value`, so each turn's audio is recognizable.""" + return bytes([value]) * 4800 + + +def _retained(session_core: SessionCore) -> list[tuple[str, str | None, bool]]: + """Each recorded speech part as what was said, and whether its audio is still retained.""" + return [ + (part.speaker, part.transcript, part.audio is not None) + for message in session_core.all_messages() + for part in message.parts + if isinstance(part, SpeechPart) + ] + + +def _answered_turn(turn: int) -> list[CoreInput]: + """A spoken question, transcribed at once, and its spoken answer: a tenth of a second of audio each.""" + turn_id, response_id = f'u{turn}', f'r{turn}' + return [ + AudioSent(data=_tenth_of_a_second(turn)), + RealtimeInputSpeechStartEvent(item_id=turn_id), + UserTurnStarted(turn_id=turn_id), + RealtimeInputSpeechEndEvent(item_id=turn_id), + UserTurnEnded(turn_id=turn_id), + InputTranscript(f'Question {turn}.', item_id=turn_id, is_final=True), + started(response_id), + AudioDelta(_tenth_of_a_second(100 + turn), response_id=response_id, item_id=f'a{turn}'), + said(response_id, f'Answer {turn}.', item_id=f'a{turn}'), + ended(response_id), + ] + + +def _budget_core(max_seconds: float | None, **kwargs: Any) -> SessionCore: + return core(retain_input_audio=True, retain_output_audio=True, retain_audio_max_seconds=max_seconds, **kwargs) + + +def test_retained_audio_keeps_the_latest_output_audio() -> None: + session_core = feed( + core(retain_output_audio=True, retain_audio_max_seconds=0.02), + started('r1'), + *(AudioDelta(bytes([index]) * 480, response_id='r1') for index in range(5)), + said('r1', 'A long answer.'), + ended('r1'), + ) + [response] = session_core.all_messages() + [part] = response.parts + assert isinstance(part, SpeechPart) and part.audio is not None + # 20 ms at 24 kHz is 960 bytes: the last two of the five deltas, behind a WAV header. + assert part.audio.data[44:] == bytes([3]) * 480 + bytes([4]) * 480 + + +def test_retained_audio_keeps_the_latest_audio_of_a_turn_still_coming_in() -> None: + session_core = feed( + core(input_transcription_enabled=False, retain_input_audio=True, retain_audio_max_seconds=0.01), + *(AudioSent(data=bytes([index]) * 240) for index in range(5)), + UserTurnStarted(turn_id='u1'), + UserTurnEnded(turn_id='u1'), + ) + [request] = session_core.all_messages() + [part] = request.parts + assert isinstance(part, SpeechPart) and part.audio is not None + assert part.audio.data[44:] == bytes([3]) * 240 + bytes([4]) * 240 + + +def test_retained_audio_evicts_the_oldest_and_keeps_transcripts_and_snapshots() -> None: + session_core = feed(_budget_core(0.3), *_answered_turn(0)) + snapshot_after_first = session_core.all_messages() + feed(session_core, *_answered_turn(1), *_answered_turn(2)) + + # Three tenths of a second fit: the latest three, whichever side said them. + assert _retained(session_core) == [ + ('user', 'Question 0.', False), + ('assistant', 'Answer 0.', False), + ('user', 'Question 1.', False), + ('assistant', 'Answer 1.', True), + ('user', 'Question 2.', True), + ('assistant', 'Answer 2.', True), + ] + # A snapshot taken before an eviction doesn't change. + assert [ + part.audio is not None + for message in snapshot_after_first + for part in message.parts + if isinstance(part, SpeechPart) + ] == [True, True] + + +def test_retained_audio_counts_turns_waiting_for_their_transcript_and_evicts_the_oldest() -> None: + def spoken(turn: int) -> list[CoreInput]: + return [ + AudioSent(data=_tenth_of_a_second(turn)), + RealtimeInputSpeechStartEvent(item_id=f'u{turn}'), + UserTurnStarted(turn_id=f'u{turn}'), + RealtimeInputSpeechEndEvent(item_id=f'u{turn}'), + UserTurnEnded(turn_id=f'u{turn}'), + ] + + session_core = feed( + core(retain_input_audio=True, retain_audio_max_seconds=0.15), + *spoken(0), + *spoken(1), + # A repeated speech end cuts nothing new, for the evicted turn or the other one. + AudioSent(data=bytes(960)), + RealtimeInputSpeechEndEvent(item_id='u0'), + InputTranscript('First.', item_id='u0', is_final=True), + InputTranscript('Second.', item_id='u1', is_final=True), + ) + assert _retained(session_core) == [('user', 'First.', False), ('user', 'Second.', True)] + + +def test_retained_audio_drops_a_cleared_or_discarded_turns_waiting_audio() -> None: + session_core = feed( + core(retain_input_audio=True, retain_audio_max_seconds=0.15), + AudioSent(data=_tenth_of_a_second(0)), + RealtimeInputSpeechStartEvent(item_id='u0'), + UserTurnStarted(turn_id='u0'), + RealtimeInputSpeechEndEvent(item_id='u0'), + UserTurnDiscarded(turn_id='u0'), + AudioSent(data=_tenth_of_a_second(1)), + RealtimeInputSpeechStartEvent(item_id='u1'), + UserTurnStarted(turn_id='u1'), + RealtimeInputSpeechEndEvent(item_id='u1'), + AudioCleared(), + # Neither counts any more, so this one fits beside nothing else. + AudioSent(data=_tenth_of_a_second(2)), + UserTurnStarted(turn_id='u2'), + UserTurnEnded(turn_id='u2'), + InputTranscript('Third.', item_id='u2', is_final=True), + ) + assert _retained(session_core) == [('user', 'Third.', True)] + + +class _CountingTurns(OrderedDict[str, Any]): + """Counts every walk over the turns waiting with audio, which checking the budget on each chunk must not need.""" + + walks = 0 + + def __iter__(self) -> Iterator[str]: + type(self).walks += 1 + return super().__iter__() + + +def test_retained_audio_check_does_not_walk_waiting_turns() -> None: + session_core = _budget_core(1) + session_core._waiting_turn_audio = _CountingTurns() # pyright: ignore[reportPrivateUsage] + for turn in range(20): + feed( + session_core, + AudioSent(data=_tenth_of_a_second(turn)), + RealtimeInputSpeechStartEvent(item_id=f'u{turn}'), + UserTurnStarted(turn_id=f'u{turn}'), + RealtimeInputSpeechEndEvent(item_id=f'u{turn}'), + UserTurnEnded(turn_id=f'u{turn}'), + ) + feed(session_core, *(AudioSent(data=bytes([index]) * 480) for index in range(10))) + feed(session_core, *(InputTranscript(f'Q{turn}.', item_id=f'u{turn}', is_final=True) for turn in range(20))) + + assert _CountingTurns.walks == 0 + _ = list(session_core._waiting_turn_audio) # pyright: ignore[reportPrivateUsage] + assert _CountingTurns.walks == 1, 'the counter sees a walk' + # A second holds the last nine waiting turns beside the 0.1 s streamed after them. + assert [audio for _, _, audio in _retained(session_core)] == [turn >= 11 for turn in range(20)] + + +def test_retained_audio_passes_over_history_once_across_many_turns(monkeypatch: pytest.MonkeyPatch) -> None: + """Each eviction resumes where the last one stopped, so many short turns cost work linear in their number.""" + strips = 0 + strip = RetainedAudioBudget.strip + + def counting_strip(self: RetainedAudioBudget, message: ModelMessage, excess: int) -> tuple[ModelMessage, int]: + nonlocal strips + strips += 1 + return strip(self, message, excess) + + monkeypatch.setattr(RetainedAudioBudget, 'strip', counting_strip) + session_core = feed(_budget_core(0.2), *(item for turn in range(40) for item in _answered_turn(turn))) + + messages = session_core.all_messages() + assert len(messages) == 80 + assert [audio for _, _, audio in _retained(session_core)] == [False] * 78 + [True] * 2 + # Starting every pass from the beginning of history would be about 80 * 80. + assert strips <= 3 * len(messages) + + +def test_retained_audio_comes_back_for_a_message_recorded_after_the_eviction_passed_it( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A response under way isn't skipped for good: once it is recorded, its audio is evicted too.""" + strips = 0 + strip = RetainedAudioBudget.strip + + def counting_strip(self: RetainedAudioBudget, message: ModelMessage, excess: int) -> tuple[ModelMessage, int]: + nonlocal strips + strips += 1 + return strip(self, message, excess) + + session_core = feed(_budget_core(0.2), *_answered_turn(0)) + monkeypatch.setattr(RetainedAudioBudget, 'strip', counting_strip) + feed( + session_core, + started('r1'), + AudioDelta(_tenth_of_a_second(101) * 2, response_id='r1', item_id='a1a'), + # A second part closes the first, which the budget then tracks while the response is still under way. + AudioDelta(b'\x01\x00', response_id='r1', item_id='a1b'), + *(AudioSent(data=bytes(480)) for _ in range(30)), + ) + # The chunks streamed meanwhile don't search history again each time: each recorded message is looked at + # about once (the last one evicted from again, once). + assert strips <= 3 + feed(session_core, ended('r1'), *_answered_turn(2)) + assert [audio for _, _, audio in _retained(session_core)] == [False, False, False, False, True, True] + + +def test_retained_audio_still_evicts_after_an_earlier_message_leaves_history() -> None: + """An input withdrawn from ahead of where eviction resumes doesn't make it skip the next answer's audio.""" + + def answer(turn: int) -> list[CoreInput]: + response_id = f'r{turn}' + return [ + started(response_id), + AudioDelta(_tenth_of_a_second(100 + turn), response_id=response_id, item_id=f'a{turn}'), + ended(response_id), + ] + + session_core = feed( + core(retain_output_audio=True, retain_audio_max_seconds=0.1), + InputSent(input_id=0, request=text_request('An image, say.')), + InputAdded(input_id=0), + *answer(0), + *answer(1), + InputWithdrawn(input_ids=(0,)), + *answer(2), + ) + assert [audio for _, _, audio in _retained(session_core)] == [False, False, True] + + +def test_retained_audio_unbounded_keeps_it_all() -> None: + session_core = feed(_budget_core(None), *(item for turn in range(3) for item in _answered_turn(turn))) + assert all(audio for _, _, audio in _retained(session_core)) + + +def test_a_conversation_id_resolved_late_reaches_what_the_core_recorded() -> None: + session_core = feed( + core(), + *_answered_turn(0), + InputSent(input_id=0, request=text_request('Typed.')), + InputAdded(input_id=0), + started('r9'), + ToolCall('call_1', tool_name='lookup', args='{}', response_id='r9'), + ended('r9'), + ToolReturned( + tool_call_id='call_1', + request=ModelRequest(parts=[ToolReturnPart(tool_name='lookup', content='ok', tool_call_id='call_1')]), + ), + # Still being said, so not recorded yet. + UserTurnStarted(turn_id='u9'), + UserTurnEnded(turn_id='u9'), + ) + session_core.set_conversation_id('c1') + feed(session_core, InputTranscript('Late.', item_id='u9', is_final=True), *_answered_turn(1)) + assert {message.conversation_id for message in session_core.all_messages()} == {'c1'} + + +def test_input_audio_that_never_went_out_is_taken_back() -> None: + session_core = feed( + core(input_transcription_enabled=False, retain_input_audio=True), + AudioSent(data=b'\x01\x00'), + AudioSent(data=b'\x02\x00'), + AudioUnsent(), + # Only the latest chunk is taken back, once. + AudioUnsent(), + UserTurnStarted(turn_id='u1'), + UserTurnEnded(turn_id='u1'), + ) + [request] = session_core.all_messages() + [part] = request.parts + assert isinstance(part, SpeechPart) and part.audio is not None + assert part.audio.data[44:] == b'\x01\x00' + + +def test_a_reconnect_replays_a_held_turns_audio_without_counting_it() -> None: + """The replay builds the held turn's request with its audio, which isn't retained history, so isn't counted.""" + session_core = feed( + core(retain_input_audio=True, retain_audio_max_seconds=0.1), + AudioSent(data=_tenth_of_a_second(1)), + RealtimeInputSpeechStartEvent(item_id='u1'), + UserTurnStarted(turn_id='u1'), + RealtimeInputSpeechEndEvent(item_id='u1'), + UserTurnEnded(turn_id='u1'), + ) + [request] = session_core.replayable_messages() + [part] = request.parts + assert isinstance(part, SpeechPart) and part.audio is not None + assert session_core._audio_budget.tracked_parts == 0 # pyright: ignore[reportPrivateUsage] + + +def test_retained_audio_eviction_passes_over_what_is_not_recorded() -> None: + """A response lost before it said anything records nothing, and a turn waiting for its transcript is come back to.""" + session_core = feed( + _budget_core(0.2), + started('r_lost'), + ended('r_lost', 'lost'), + AudioSent(data=_tenth_of_a_second(5)), + RealtimeInputSpeechStartEvent(item_id='u5'), + UserTurnStarted(turn_id='u5'), + RealtimeInputSpeechEndEvent(item_id='u5'), + UserTurnEnded(turn_id='u5'), + started('r6'), + AudioDelta(_tenth_of_a_second(106), response_id='r6', item_id='a6'), + said('r6', 'Answer 6.', item_id='a6'), + ended('r6'), + # Over budget: the eviction passes the lost response, waits on the turn, and evicts the answer after it. + AudioSent(data=_tenth_of_a_second(7)), + InputTranscript('Question 5.', item_id='u5', is_final=True), + ) + assert _retained(session_core) == [('user', 'Question 5.', True), ('assistant', 'Answer 6.', False)] + # The turn, recorded now, is next in line: its audio is the oldest left. + assert list(session_core._recorded_audio) == [session_core._turns['u5']] # pyright: ignore[reportPrivateUsage] + + +def test_input_audio_cut_for_a_turn_before_its_send_failed_is_taken_back_from_the_turn() -> None: + session_core = feed( + core(retain_input_audio=True), + AudioSent(data=b'\x01\x00'), + AudioSent(data=b'\x02\x00'), + RealtimeInputSpeechStartEvent(item_id='u1'), + UserTurnStarted(turn_id='u1'), + # The speech end cuts the turn's audio while the last chunk is still on its way, and its send then fails. + RealtimeInputSpeechEndEvent(item_id='u1'), + AudioUnsent(), + UserTurnEnded(turn_id='u1'), + InputTranscript('Hi.', item_id='u1', is_final=True), + ) + [request] = session_core.all_messages() + [part] = request.parts + assert isinstance(part, SpeechPart) and part.audio is not None + assert part.audio.data[44:] == b'\x01\x00' + + +def test_retained_audio_trims_the_oldest_audio_coming_in_first() -> None: + """A response whose incoming audio was emptied counts as newest when more comes in for it.""" + session_core = feed( + core(retain_output_audio=True, retain_audio_max_seconds=0.1), + started('r1'), + started('r2'), + AudioDelta(_tenth_of_a_second(1), response_id='r1', item_id='a1'), + AudioDelta(_tenth_of_a_second(2), response_id='r2', item_id='a2'), + AudioDelta(_tenth_of_a_second(3), response_id='r1', item_id='a1'), + said('r1', 'One.', item_id='a1'), + said('r2', 'Two.', item_id='a2'), + ended('r1'), + ended('r2'), + ) + by_transcript = { + part.transcript: part.audio.data[44:] if part.audio is not None else None + for message in session_core.all_messages() + for part in message.parts + if isinstance(part, SpeechPart) + } + # r1's first tenth went first, then r2's (older than r1's second): only r1's newest audio fits. + assert by_transcript == {'One.': _tenth_of_a_second(3), 'Two.': None} + + +def test_retained_audio_evicted_is_let_go_by_the_core_too() -> None: + """Evicted audio isn't held anywhere else in the core: not as a turn's PCM, nor in a response's parts.""" + session_core = feed(_budget_core(0.2), *(item for turn in range(5) for item in _answered_turn(turn))) + assert [audio for _, _, audio in _retained(session_core)] == [False] * 8 + [True] * 2 + turns = session_core._turns.values() # pyright: ignore[reportPrivateUsage] + responses = session_core._responses.values() # pyright: ignore[reportPrivateUsage] + assert all(not turn.audio for turn in turns) + assert all(not response.parts for response in responses) + # What is left: the last question and answer, in history alone. + assert session_core._audio_budget.tracked_parts == 2 # pyright: ignore[reportPrivateUsage] + assert len(session_core._recorded_audio) == 2 # pyright: ignore[reportPrivateUsage] + + +def test_a_failed_send_of_audio_trimmed_by_the_budget_leaves_nothing_of_it() -> None: + """A chunk longer than the whole budget is trimmed before its send fails: what is left of it is taken back.""" + session_core = feed( + core(input_transcription_enabled=False, retain_input_audio=True, retain_audio_max_seconds=0.1), + AudioSent(data=_tenth_of_a_second(1) * 2), + AudioUnsent(), + AudioSent(data=b'\x05\x00'), + UserTurnStarted(turn_id='u1'), + UserTurnEnded(turn_id='u1'), + ) + [request] = session_core.all_messages() + [part] = request.parts + assert isinstance(part, SpeechPart) and part.audio is not None + assert part.audio.data[44:] == b'\x05\x00' + + +def test_audio_taken_back_from_a_turn_no_longer_waiting_leaves_it_alone() -> None: + """Once its turn is recorded, the core doesn't change the message to take the audio back. + + A known gap, not the intent (the current session core has it too): a turn recorded while the send of its last + chunk is still under way keeps that chunk's audio if the send then fails. + """ + session_core = feed( + core(input_transcription_enabled=False, retain_input_audio=True), + AudioSent(data=b'\x01\x00'), + UserTurnStarted(turn_id='u1'), + UserTurnEnded(turn_id='u1'), + AudioUnsent(), + ) + [request] = session_core.all_messages() + [part] = request.parts + assert isinstance(part, SpeechPart) and part.audio is not None + + +def test_retained_audio_eviction_does_not_walk_what_waits_or_has_no_audio(monkeypatch: pytest.MonkeyPatch) -> None: + """A turn waiting for its transcript ahead of many recorded messages doesn't make each chunk walk them again. + + Counts the strips, so the work of passing over messages; taking from the front of the queue is constant + work as it is a deque. + """ + strips = 0 + strip = RetainedAudioBudget.strip + + def counting_strip(self: RetainedAudioBudget, message: ModelMessage, excess: int) -> tuple[ModelMessage, int]: + nonlocal strips + strips += 1 + return strip(self, message, excess) + + session_core = feed( + _budget_core(0.3), + AudioSent(data=_tenth_of_a_second(9)), + RealtimeInputSpeechStartEvent(item_id='held'), + UserTurnStarted(turn_id='held'), + RealtimeInputSpeechEndEvent(item_id='held'), + UserTurnEnded(turn_id='held'), + *(item for turn in range(20) for item in _answered_turn(turn)), + ) + monkeypatch.setattr(RetainedAudioBudget, 'strip', counting_strip) + feed(session_core, *(AudioSent(data=bytes(480)) for _ in range(50))) + # Each chunk strips a message that still has audio, at most: it never passes over those that don't. + assert strips <= 50 + + +def test_retained_audio_eviction_keeps_a_message_that_still_has_audio_in_line() -> None: + """A message with several parts that loses only its oldest part's audio stays first in line for the next eviction.""" + session_core = feed( + _budget_core(0.2), + started('r1'), + AudioDelta(_tenth_of_a_second(1), response_id='r1', item_id='a1'), + said('r1', 'One.', item_id='a1'), + AudioDelta(_tenth_of_a_second(2), response_id='r1', item_id='a2'), + said('r1', 'Two.', item_id='a2'), + ended('r1'), + AudioSent(data=_tenth_of_a_second(3)), + ) + assert [(transcript, audio) for _, transcript, audio in _retained(session_core)] == [ + ('One.', False), + ('Two.', True), + ] + assert list(session_core._recorded_audio) == [session_core._responses['r1']] # pyright: ignore[reportPrivateUsage] + + +def test_a_turn_recorded_after_the_reply_to_it_is_evicted_before_that_reply() -> None: + session_core = feed( + _budget_core(1), + AudioSent(data=_tenth_of_a_second(1)), + RealtimeInputSpeechStartEvent(item_id='u1'), + UserTurnStarted(turn_id='u1'), + RealtimeInputSpeechEndEvent(item_id='u1'), + UserTurnEnded(turn_id='u1'), + started('r1'), + AudioDelta(_tenth_of_a_second(101), response_id='r1', item_id='a1'), + said('r1', 'Answer.', item_id='a1'), + ended('r1'), + InputTranscript('Question.', item_id='u1', is_final=True), + ) + turn, response = session_core._turns['u1'], session_core._responses['r1'] # pyright: ignore[reportPrivateUsage] + assert list(session_core._recorded_audio) == [turn, response] # pyright: ignore[reportPrivateUsage] diff --git a/tests/realtime/test_xai.py b/tests/realtime/test_xai.py index 8ef41b0ba7..21fb2e5fa1 100644 --- a/tests/realtime/test_xai.py +++ b/tests/realtime/test_xai.py @@ -40,6 +40,7 @@ from pydantic_ai.native_tools import WebSearchTool from pydantic_ai.realtime import ( RealtimeModelProfile, RealtimeSessionReconnectEvent, + _session as realtime_session, # pyright: ignore[reportPrivateUsage] ) from pydantic_ai.realtime.codec import ( AudioDelta, @@ -662,6 +663,16 @@ class _DropAfterFrames(FakeWebSocket): raise rt_xai.websockets.ConnectionClosed(None, None) +class _AnsweringAfterSend(_DropAfterFrames): + """Like `_DropAfterFrames`, but reads nothing past the handshake until the session asked for a response.""" + + async def __aiter__(self) -> AsyncIterator[Any]: + while not any('"response.create"' in frame for frame in self.sent): + await asyncio.sleep(0) + async for frame in super().__aiter__(): # pragma: no branch (it ends by dropping the connection) + yield frame + + class _RecordingConnect: """Stand-in for `websockets.connect` that hands out sockets in order and records closes.""" @@ -924,17 +935,11 @@ async def test_max_duration_error_is_not_reconnected(monkeypatch: pytest.MonkeyP assert connect.urls == ['wss://api.x.ai/v1/realtime?model=grok-voice-latest'] -@pytest.mark.shadow_divergence( - reason=( - 'synthetic frames: the transcript and the terminal name different responses, and the input is ' - 'acknowledged before it is sent' - ) -) async def test_reconnect_replay_burst_is_deduplicated_from_session_history( monkeypatch: pytest.MonkeyPatch, ) -> None: """Resumed items are suppressed even when xAI assigns new IDs to the replayed copies.""" - dropped = _DropAfterFrames( + dropped = _AnsweringAfterSend( [ _created(), _conversation_created(), @@ -942,12 +947,14 @@ async def test_reconnect_replay_burst_is_deduplicated_from_session_history( json.dumps( { 'type': 'conversation.item.added', - 'item': {'id': 'item-user', 'type': 'message', 'role': 'user'}, + 'item': {'id': 'pydantic_ai_item_0', 'type': 'message', 'role': 'user'}, } ), + json.dumps({'type': 'response.created', 'response': {'id': 'response-1', 'status': 'in_progress'}}), json.dumps( { 'type': 'response.output_audio_transcript.done', + 'response_id': 'response-1', 'item_id': 'item-assistant', 'transcript': 'Hello back.', } @@ -977,6 +984,7 @@ async def test_reconnect_replay_burst_is_deduplicated_from_session_history( json.dumps( { 'type': 'response.output_audio_transcript.done', + 'response_id': 'response-1', 'item_id': 'replayed-item-assistant', 'transcript': 'Hello back.', } @@ -1910,13 +1918,17 @@ def _spoken_reply(response_id: str, pcm: bytes, transcript: str) -> list[str]: ] -@pytest.mark.shadow_divergence('the new session core does not apply `retain_audio_max_seconds` yet') -async def test_retained_audio_eviction_keeps_a_committed_turn_in_its_place(monkeypatch: pytest.MonkeyPatch) -> None: +@pytest.mark.parametrize('core_mode', ['core', 'legacy']) +async def test_retained_audio_eviction_keeps_a_committed_turn_in_its_place( + monkeypatch: pytest.MonkeyPatch, core_mode: str +) -> None: """A spoken turn goes after what its commit followed, even once the retained-audio budget evicted that answer's audio. The second answer arrives before the second turn's transcript and evicts the first answer's audio, which the - second commit was placed after. + second commit was placed after. Also run on the current core (`_CORE_MODE = 'legacy'`), whose own record keeps + retained audio only there: in core mode it lets that audio go as soon as it records it. """ + monkeypatch.setattr(realtime_session, '_CORE_MODE', core_mode) tenth_of_a_second = b'\x10\x27' * 2400 ws = _PhasedWebSocket( [_created(), _updated()], @@ -1968,16 +1980,7 @@ async def test_retained_audio_eviction_keeps_a_committed_turn_in_its_place(monke [ pytest.param(True, 1, True, id='transcribed'), pytest.param(False, 1, True, id='untranscribed'), - pytest.param( - False, - 2, - True, - id='untranscribed-twice', - marks=pytest.mark.shadow_divergence( - reason='the connection folds the two held commits into one, which xAI makes one turn; the new core ' - 'records that one' - ), - ), + pytest.param(False, 2, True, id='untranscribed-twice'), pytest.param(True, 1, False, id='transcribed-no-text'), ], ) @@ -2038,16 +2041,14 @@ async def test_commit_held_behind_a_reply_goes_after_what_was_sent_meanwhile( 'user: What is two plus two?', 'assistant speech: Four.', *(['user: Then this.', 'user: Some context.'] if text else []), - *[spoken] * commits, + # Commits held together go out as one, which xAI makes one turn. + spoken, 'assistant speech: Answer.', ] -@pytest.mark.shadow_divergence( - reason='the commit never went out, so xAI made no turn of it; the new core records only turns the provider made' -) -async def test_untranscribed_turn_with_a_held_commit_is_recorded_on_close(monkeypatch: pytest.MonkeyPatch) -> None: - """A spoken turn whose commit never went out still ends up in history when the session closes.""" +async def test_untranscribed_turn_with_a_held_commit_is_not_recorded_on_close(monkeypatch: pytest.MonkeyPatch) -> None: + """A commit that never went out made no turn in xAI's conversation, so history has none either.""" monkeypatch.setattr(rt_xai.websockets, 'connect', _RecordingConnect([_PhasedWebSocket([_created(), _updated()])])) settings = rt_xai.XaiRealtimeModelSettings(turn_detection=False, input_transcription_model=None) @@ -2056,7 +2057,7 @@ async def test_untranscribed_turn_with_a_held_commit_is_recorded_on_close(monkey await session.commit_audio() assert session.all_messages() == [] - assert _turns(session.all_messages()) == ['user speech: None'] + assert _turns(session.all_messages()) == [] async def test_request_replayed_after_a_reconnect_sends_no_clear() -> None: @@ -2203,27 +2204,35 @@ async def test_speech_for_audio_already_committed_is_not_held_for_the_next_commi assert turns == ['user speech: First.', 'assistant speech: One.', 'user speech: Second.', 'assistant speech: Two.'] -@pytest.mark.shadow_divergence( - reason=( - 'synthetic frames: xAI reports speech in audio the connection still holds back, and adds no item at speech ' - 'start (recorded: `test_xai_ws/test_push_to_talk_replies_only_when_asked`)' - ) -) async def test_turns_of_a_commit_held_behind_a_reply_keep_their_order(monkeypatch: pytest.MonkeyPatch) -> None: - """Two utterances xAI transcribed before their held commit went out are recorded in the order spoken.""" + """Two utterances in a commit held until the reply ends are recorded in the order spoken, after the text sent + while it was held, which reached xAI first.""" + + def added(item_id: str) -> str: + return json.dumps( + { + 'type': 'conversation.item.added', + 'item': {'id': item_id, 'type': 'message', 'role': 'user', 'content': []}, + } + ) def utterance(item_id: str, transcript: str) -> list[str]: return [ json.dumps({'type': 'input_audio_buffer.speech_started', 'item_id': item_id, 'audio_start_ms': 0}), + added(item_id), _user_transcript(item_id, transcript), + json.dumps({'type': 'input_audio_buffer.speech_stopped', 'item_id': item_id, 'audio_end_ms': 500}), ] ws = _PhasedWebSocket( [_created(), _updated()], - [_response_frame('response.created', 'r0'), _transcript_delta('r0')], - [*utterance('item-u1', 'One.'), *utterance('item-u2', 'Two.')], + [added('pydantic_ai_item_0'), _response_frame('response.created', 'r0'), _transcript_delta('r0')], [ + added('pydantic_ai_item_3'), _response_frame('response.done', 'r0'), + # xAI hears the speech only once the held audio goes out with the commit, after the reply. + *utterance('item-u1', 'One.'), + *utterance('item-u2', 'Two.'), json.dumps({'type': 'input_audio_buffer.committed', 'item_id': 'item-u2'}), *_reply('r1', 'Answer.'), ], @@ -2239,11 +2248,6 @@ async def test_turns_of_a_commit_held_behind_a_reply_keep_their_order(monkeypatc await session.commit_audio() await session.send('Then this.') ws.advance() - async for event in session: # pragma: no branch - if isinstance(event, PartEndEvent) and isinstance(event.part, SpeechPart): - if event.part.transcript == 'Two.': - break - ws.advance() await session.wait_for_reply() assert _turns(session.all_messages()) == [