OpenAI Realtime API Interruption Solution

I’ve recently been developing a cross-platform mobile app with Expo. This app makes use of the OpenAI Realtime API for speech-to-speech chatting. I (like seemingly many others on the internet) have run into an issue where the agent would speak through the device’s speaker and would interrupt itself. I believe I found a solution to this problem.

Background

First, I’d like to provide some background. If you just want the solution, it’ll be below so feel free to just grab that if you are in a hurry.

As I mentioned above, I designed this project to work cross-platform and testing on a MacBook and an iPhone 14 Pro. Both of these devices actually worked totally fine for me, but as soon as I upgraded to the iPhone 17 Pro, my issues began. To be clear, the web version has never had any issue with echo.

Note: Developing using a Simulator on my MacBook always had an issue with interruption. I’d recommend testing on a real device regardless.

Truth be told, I’m not sure why my iPhone 14 Pro worked while my iPhone 17 Pro didn’t. I’m assuming it has something to do with a physical speaker upgrade, but that is really just a guess! Ok — enough of that. Let’s get into some of the code!

Get Grant Hershman’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

I am basing my implementation on this example with some modifications for my specific use case. To use the speaker on my iPhone, I used the react-native-incall-manager package. This packaged was designed to work closely with react-native-webrtc (and react-native-webrtc-webshim by extension which is what I am using). My code looks like this:

if (Platform.OS === "ios") {
  InCallManager.start({ media: "audio" });
  InCallManager.setForceSpeakerphoneOn(true);
}

Solution

I added the Platform check to ensure speaker is enabled only for iOS devices. Now here’s the solution that worked for me — in the configureTools function (see repo linked above for reference), I updated the session.update with the following:

{
  type: "session.update",
  session: {
    type: "realtime",
    tools: clientToolsSchema,
    tool_choice: "auto",
    audio: {
      input: {
        noise_reduction: {
          type: "far_field",  // <- This!!
        },
        turn_detection: {
          type: "server_vad",
          threshold: 0.75,
        },
      },
    },
  },
}

session.audio.input.noise_reduction.type being set to far_field is the key here. This configures some setting on the OpenAI Realtime API side to optimize the model for speakerphone use. The turn_detection.threshold being 0.75 is the user’s voice activity threshold — higher values make the model less sensitive to speech. I figured this would help too.

I spent days on end browsing the web and working on my own solution. There was seemingly no end in sight — it was maddening. I tried using the react-native-audio-api package to try to manually configure the AVAudioSession. No dice. I created a native module to handle the audio session configuration directly. Still, it refused the call (Game of Thrones reference).

In the end, the solution was really just a small update to the session configuration — adding the far_field noise reduction type and adjusting the turn detection threshold. This worked perfectly for me. Instead of including this in a session update, I’m sure it can be configured directly in the session creation as well. I haven’t tested that, so feel free to try it out.

That’s all folks! This is my first attempt at documenting a solution at this length. Hopefully you found it useful!