Implement Speech to Text using Whisper.cpp in SwiftUI

Hey Everyone

Photo by Will Francis on Unsplash

Have you ever tried speech-to-text on iOS? It’s straightforward, thanks to Apple’s built-in functionality that makes integration easy for developers. I was content with this feature for a long time, until I recently tried to add audio support to Summify, my AI summary app.

Here’s what I encountered with long audio files:
1. Processing Time: On an iPhone 15, it took 10–15 minutes to transcribe an hour-long audio file.
2. Errors: In many cases, the process failed and produced random errors.
3. Wrong transcript: Even if it was successful and took a lot of time, transcript often wasn’t good.

Get Arnav Singhal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

It became clear that the native speech-to-text wasn’t suitable for my app. That’s when I discovered Whisper.cpp, a fantastic GitHub repository that brings OpenAI’s Whisper model to C++.

GitHub - ggerganov/whisper.cpp: Port of OpenAI's Whisper model in C/C++

I was frankly amazed and excited to try it out and implement it in iOS. And that’s what this Medium article is all about. I’ve done most of the hard part — now you just need to add some code. Boom, let’s start coding!

Preparation

1. Let’s begin by integrating some packages into your project:
   - SwiftWhisper: This library serves as a robust wrapper for Whisper.cpp, simplifying the integration of the Whisper model into your application.
   - AudioKit: Added specifically for the convertAudioFileToPCMArray function, which facilitates audio file conversion to WAV format and extraction of PCM data into a floating-point array, managed through comprehensive error handling.

I attempted native implementations previously without success. Please share your experiences if any of you have achieved successful integration.

2. Add both the CoreML model and BIN file for Whisper in your project from the repository linked below.

GitHub - Arnav-arw/AppleMeetsWhisper

And that wraps up the preparation work! We just need to add a few lines of code, and we’ll be all set.

Coding

Make a ViewModel or just directly add below functions

func extractTextFromAudio(_ audioURL: URL, completionHandler: @escaping (Result<String, Error>) ->Void) {
    let modelURL = Bundle.main.url(/service/forresource: "tiny", withExtension: "bin")!
    let whisper = Whisper(fromFileURL: modelURL)
    convertAudioFileToPCMArray(fileURL: audioURL) { result in
        switch result {
            case .success(let success):
                Task {
                    do {
                        let segments = try await whisper.transcribe(audioFrames: success)
                        completionHandler(.success(segments.map(\.text).joined()))
                    } catch {
                        completionHandler(.failure(error))
                    }
                }
            case .failure(let failure):
                completionHandler(.failure(failure))
        }
    }
}

The `extractTextFromAudio` function loads the Whisper model and converts the audio file to a PCM array. It then transcribes the audio frames to text using the Whisper model and returns the transcription via a completion handler. Errors in the process are caught and returned through the same handler.

func convertAudioFileToPCMArray(fileURL: URL, completionHandler: @escaping (Result<[Float], Error>) ->Void) {
    var options = FormatConverter.Options()
    options.format = .wav
    options.sampleRate = 16000
    options.bitDepth = 16
    options.channels = 1
    options.isInterleaved = false
    
    let tempURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent(UUID().uuidString)
    let converter = FormatConverter(inputURL: fileURL, outputURL: tempURL, options: options)
    converter.start { error in
        if let error {
            completionHandler(.failure(error))
            return
        }
        
        let data = try! Data(contentsOf: tempURL)
        
        let floats = stride(from: 44, to: data.count, by: 2).map {
            return data[$0..<$0 + 2].withUnsafeBytes {
                let short = Int16(littleEndian: $0.load(as: Int16.self))
                return max(-1.0, min(Float(short) / 32767.0, 1.0))
            }
        }
        
        try? FileManager.default.removeItem(at: tempURL)
        
        completionHandler(.success(floats))
    }
 }

The `convertAudioFileToPCMArray` function converts an audio file to a WAV format, reads the PCM data, and converts it into an array of floating-point values. It handles the conversion process using specified format options and manages errors. The resulting array or any errors are returned via a completion handler.

For better performance while testing, please switch the app scheme for run to release.

And that concludes the implementation. If you encounter any errors, you can refer to the repository below, which includes a demo project.

GitHub - Arnav-arw/AppleMeetsWhisper

Results

Took an audio of 25 mins, specifically from this YouTube video.

Using native Speech-to-Text, it took a whopping 15 mins and the transcript wasn’t very accurate. With Whisper, drumroll please, it only took 30 seconds!

Yes, folks that’s a HUGE improvement, over 3000%.

Stackademic 🎓

Thank you for reading until the end. Before you go:

• Please consider clapping and following the writer! 👏
• Follow us X | LinkedIn | YouTube | Discord
• Visit our other platforms: In Plain English | CoFeed | Differ
• More content at Stackademic.com