What the OpenAI Audio API does
OpenAI's audio and speech APIs support four related jobs:
- Speech to text: turn speech into a transcript for captions, search, records, or analysis.
- Text to speech: turn text into spoken audio for narration, accessibility, or spoken replies.
- Speech to speech: run a live session where a model listens, reasons, and speaks.
- Speech translation: translate live speech or a completed recording through the appropriate audio workflow.
The right architecture depends on whether the audio is already complete or still arriving. That is more useful than beginning with a model name.
Choose the audio architecture first
Completed recordings and bounded requests
For a recorded support call, meeting, or voice note, use file transcription. OpenAI recommends gpt-transcribe as the starting model for recorded speech in its original language. The API accepts supported files up to 25 MB; longer recordings need a deliberate handling strategy rather than an assumption that the upload will work.
Use a specialized transcription path only when the workflow needs it. For example, speaker diarization can label speakers in a recording, while whisper-1 remains the option for word or segment timestamps and subtitle formats. For a completed recording that must be translated into English, the documented translations endpoint uses whisper-1.
Test transcription with the conditions your callers actually create. OpenAI recommends checking background noise, microphone quality, telephony audio, accents, and domain vocabulary. A transcript is another system output to review; it should not silently become the source of truth for a refund, account change, or safety-sensitive decision.
Live audio and low-latency conversations
Use the Realtime API when the application receives live audio and needs immediate events. The current documentation distinguishes voice-agent sessions, continuous translation sessions, and transcription sessions. A voice-agent session is for a model that responds to the caller; a transcription session emits text without model-generated spoken replies.
Connection choice is part of the build. OpenAI documents WebRTC for browser or mobile audio and WebSockets for a server that already receives audio from a call system or another media pipeline. SIP is a separate telephony option, so validate current model support before designing around it.
For a support voice agent, OpenAI's voice-agent guide describes two useful designs. A speech-to-speech session prioritizes natural, low-latency interaction. A chained workflow gives the application explicit stages for transcription, text reasoning, policy checks, internal-system calls, and speech output. The second design can be a better fit when an approval or a durable transcript must sit between what the caller said and what the system does.
Spoken output
The text-to-speech API generates audio from text and can stream output before the full response is complete. The endpoint uses a model, input text, and selected voice; format choices affect how the audio is used. For example, OpenAI documents WAV and PCM as options for faster playback paths.
If customers hear an AI-generated voice, OpenAI requires clear disclosure that it is AI-generated rather than human. This is separate from whether the spoken response is correct. The workflow still needs accurate knowledge, a policy boundary, and a way for a caller to reach a person when the issue cannot be resolved safely.
What your application must still handle
Adding audio changes the transport and user experience, not the basic responsibility of a support workflow. Whether a model is transcribing a call or speaking in real time, the application remains responsible for the following:
| Part of the workflow | What to verify |
|---|---|
| Transcript | Does it capture the customer's intent, names, numbers, and relevant details accurately enough to use? |
| Knowledge and policy | Does the response use the approved policy rather than inventing an exception? |
| Tool request | Does the application check identity, permission, and required approval before an action runs? |
| Result | Does the order, helpdesk, or other connected service confirm the action actually happened? |
| Customer handoff | Does the caller know when a human must decide, and can they reach one? |
For a damaged-delivery call, the transcript may say that the order arrived outside the normal refund window. The workflow should identify the exception, request the evidence the policy requires, and route a decision to a person when required. It must not say a refund was issued unless the connected order system confirms it.
OpenAI's safety guidance also recommends evaluating model behavior over a broad range of inputs and using human review where the output is used in practice. For live interactions with individual end users, the Realtime guidance documents privacy-preserving safety identifiers as a recommended way to associate behavior with an end user. These are controls to design and test, not a guarantee that a voice workflow is safe.
Pricing: measure the whole workflow
Audio costs cannot be reduced to one old per-minute estimate. The OpenAI pricing page lists separate rates for realtime audio, audio-capable models, transcription, speech generation, and text usage. The result depends on the current model, input and output modalities, audio duration, cache behavior, and any additional tools or systems in the request loop.
Measure a representative workflow before committing to an architecture:
- Run calls with the expected duration, language, noise, and interruptions.
- Include transcription, reasoning, tool calls, and generated speech where the design uses them.
- Record cost alongside transcript accuracy, correct policy decisions, latency, failed actions, and human escalations.
- Repeat the evaluation when the model, prompt, audio routing, or policy changes.
This separates a cheap demo from a support workflow that handles real callers correctly.
Use eesel CLI after the audio step
The eesel CLI is for operating an existing eesel teammate, not for opening an OpenAI Realtime session or uploading audio to OpenAI. It is the same teammate and workspace as the dashboard: a change made through the CLI appears in the dashboard, and the reverse is also true.
That distinction is useful when your audio system produces a transcript and the support team needs to decide what happens next. A support lead can use the terminal, a script can use the CLI's JSON output, and a coding agent such as Claude Code, Codex, or Cursor can perform a scoped review of the teammate that will prepare a follow-up.
Start with Node.js 18.17 or newer, then confirm the workspace and teammate before inspecting or changing anything:
npx @eesel/cli login
npx @eesel/cli whoami
npx @eesel/cli agents
Point the inspection commands to the intended support teammate. This avoids a script or coding agent acting on the saved default by mistake:
TARGET_EESEL_AGENT="paste-agent-id-or-name-here"
npx @eesel/cli status --agent "$TARGET_EESEL_AGENT"
npx @eesel/cli integrations --agent "$TARGET_EESEL_AGENT"
npx @eesel/cli instructions --agent "$TARGET_EESEL_AGENT"
Ask the coding agent for a read-only report: which knowledge sources are connected, whether their content has finished downloading, and which instructions govern damaged deliveries, refunds, and handoff. Do not let it connect accounts, change permissions, or enable automations as a side effect of that review.
After a human approves a configuration change, use --dry-run on the relevant write command before it is sent. Then create a fresh, named conversation and test an approved, redacted transcript. For example:
npx @eesel/cli new --name "damaged-delivery call" --agent "$TARGET_EESEL_AGENT"
npx @eesel/cli chat "Here is an approved, redacted call transcript. Draft a written follow-up for a damaged delivery requested outside the normal refund window. Use the approved policy, identify missing evidence, and state when a human decision is needed." --agent "$TARGET_EESEL_AGENT"
Review the result for the policy boundary, missing evidence, and honest handoff. The CLI chat is a support-workflow test; it does not transcribe the recording, generate a voice reply, place a call, or prove that a refund happened. If a workflow creates held actions, inspect them with npx @eesel/cli approvals --agent "$TARGET_EESEL_AGENT" and confirm the final action in the connected system before telling a customer it succeeded.
Setup and observation commands are free; CLI chat is billed work. Scope the test accordingly, and use the JSON output to make the coding agent's assessment inspectable.
Should you build with OpenAI Audio API or operate a support teammate?
Build with the OpenAI Audio API when you need to own the audio experience: capture or receive calls, choose a transcript or realtime architecture, connect tools, and deliver voice output. That path gives control, but your team owns the infrastructure, permissions, evaluations, and operations around the model.
Use eesel when the immediate job is operating an existing support teammate. The CLI lets people, scripts, and coding agents inspect and configure that same teammate alongside the dashboard, then test an approved text-based follow-up from the audio workflow. It complements the audio layer; it does not replace it.
The Activity view gives a support lead a dashboard-side record to compare with the CLI's JSON output after a controlled test.
The Activity page after a controlled teammate test.
If that support workflow fits your project, try eesel and begin with the CLI guide.
Frequently asked questions
What is the OpenAI Audio API?
OpenAI's audio APIs cover speech-to-text transcription, text-to-speech generation, and live audio sessions. Choose a request-based API for a recorded file or bounded request, and a Realtime session when audio arrives live and the application needs low-latency events.
Which OpenAI Audio API option should I use for a support-call recording?
For a completed recording, start with file transcription. OpenAI recommends gpt-transcribe for ordinary recorded-speech transcription; use a specialized option only when you need features such as speaker labels, timestamps, subtitles, or English translation. Test with the accents, noise, telephony quality, and vocabulary your calls contain.
When should I use OpenAI Realtime API for audio?
Use a Realtime session when your application receives live audio and needs partial events or low-latency interaction. Browser and mobile clients generally use WebRTC; server media pipelines can use WebSockets. Realtime is not required to stream the transcription of a completed file.
Does the OpenAI Audio API execute support actions?
No. A voice workflow can request tools, but the application must execute them, check permissions and approvals, prevent duplicate side effects, and verify the real service response. A fluent spoken answer is not evidence that an order update or refund succeeded.
What should I test before using OpenAI Audio API in customer support?
Test representative audio, including background noise, microphone quality, telephony, accents, product names, interruptions, and policy exceptions. Review transcript accuracy separately from the final support decision, and include human review or approval for actions where a mistake has material impact.
How should I estimate OpenAI Audio API cost?
Use OpenAI's current pricing page for the exact model and route you will deploy. Measure a representative full workflow: input and output audio, transcription or speech generation, text tokens, cached input where relevant, and any tools. Recheck when the model, audio duration, or architecture changes.
Where does eesel CLI fit after a support call?
The eesel CLI operates an existing eesel teammate in the same workspace as the dashboard. It does not replace OpenAI's transcription, speech, or realtime transport. A person, script, or coding agent can use it to inspect the teammate's knowledge and instructions, then test a redacted call transcript or approved follow-up workflow.
Can Claude Code, Codex, or Cursor use eesel CLI?
Yes. The CLI prints JSON, which lets coding agents handle a scoped setup or review task. Confirm the workspace and teammate, inspect before proposing changes, use --dry-run for a write, and review the resulting configuration and customer-facing behavior before enabling automation.
Article by
Kurnia Kharisma Agung Samiadjie
Kurnia is a software engineer and writer at eesel AI with two years of SEO experience, writing about AI tools, helpdesk software, and customer support. He pairs a developer's understanding of how these products are built with search-driven research into what actually ranks and resonates with the people searching for them.
