Bidirectional Streaming for AI Voice Messages in Swift

Goal

This article explains how to structure a bidirectional stream for an AI voice message in a Swift app. The client sends message metadata, audio headers, compressed packets, and final options. The backend can stream text, audio, suggestions, and final state back to the app.

The code is adapted from a Swift project that used Connect, Protobuf, Opus, RTP packetization, actors, and a dedicated sender object.

Why a voice message needs a stream

A text-only chat message can be a unary request:

client sends message
server returns response

Voice changes the timeline:

client sends message metadata
client sends audio format headers
client sends packet 1
client sends packet 2
client sends packet 3
client sends final options
server streams response updates
server finishes

The backend may start preparing the AI response before the user finishes speaking. The client may show partial local speech recognition. The user may cancel. The network may drop. The UI may move to another chat.

That is a session, and a bidirectional stream matches it much better than a file upload.

The stream message shape