OpenAI realtime tool calls: A complete overview

Conversational AI is getting pretty wild. We're moving beyond the clunky chatbots of yesterday and into a world with voice agents that can actually hold a conversation in real-time. But what makes them truly useful isn't just that they can talk, it's that they can do things. That’s where OpenAI Realtime Tool Calls come into the picture. It lets a voice agent request information or actions from an external system during a conversation.

This post will walk you through what this technology is, how it works, and where it really shines. We'll also get real about the challenges of trying to build with it from scratch. We will also separate voice application development from another support task: using eesel CLI to configure and inspect an existing support teammate.

What are OpenAI Realtime Tool Calls?

So, what's the big deal with these tool calls? Simply put, they're a feature in OpenAI's Realtime API that lets a voice AI connect to external tools during a live conversation. The underlying function-calling pattern is familiar: the model requests a function and your application executes it. Here, the exchange happens within a live session that also carries audio or text. A slow database or external API can still delay the result.

Think of it as letting a voice assistant ask your application for an order status instead of guessing from the conversation.

This is what turns a voice agent from a neat party trick into a real workhorse. It’s the magic that lets them check your order status, book an appointment, or pull up your account details on the fly. For things like customer support, sales, or even just a personal assistant, this capability is non-negotiable.

How OpenAI Realtime Tool Calls work

Unlike a standard API call where you send a request and get a response, the Realtime API keeps an open line, using something like a WebSocket or WebRTC. This allows for a continuous, back-and-forth conversation between your app and the OpenAI model.

For browser and server applications, the docs describe WebRTC for browser-based apps and WebSocket for things running on a server. Whichever you use, the process for a tool call during a live chat follows a few key steps.

Let's walk through what happens when you ask your voice agent a question:

1. Setting the stage: Your app connects to the Realtime API and tells it which "tools" or functions the AI is allowed to use. This could be anything from "lookup_order_status" to "check_product_inventory".

2. The user speaks: You start talking. Your app streams your voice directly to the API in little chunks.

3. The AI gets an idea: As the AI listens, it decides if it needs to use one of its tools to answer you. If you ask, "Hey, where's my latest order?" the model recognizes it needs to trigger the order lookup tool.

4. The API sends a signal: The API sends an event back to your app that basically says, "I need you to run a function." This message includes the function's name and any arguments, like "name: "lookup_order"" and "arguments: {"order_id": "12345"}".

5. Your app does the work: Your backend code catches this signal and runs the function. It might ping your Shopify database or internal API to get the order status. Let's say it finds out the order has "shipped."

6. Sending the results back: Your app then packages that "shipped" status into a message and sends it back to the Realtime API, letting the model know what it found.

7. The final answer: Armed with this new info, the model generates a natural-sounding audio response and streams it back to you. You'll hear something like, "I've just checked, and your order #12345 has shipped!"

The caller experiences the whole loop, including any wait for your tools. Measure that delay with realistic backend responses.

Key use cases and benefits of OpenAI Realtime Tool Calls

Realtime tool calls are what allow voice agents to solve actual problems. Here are a few places where this tech is already making a difference.

Customer support automation

This is probably the biggest one. An AI agent can handle a ton of common support questions instantly, any time of day.

• Order management: An agent can check order statuses, find tracking numbers, or start a return by calling a company's backend systems, whether that's Shopify, Magento, or something custom.

• Account inquiries: Customers can ask about their balance or recent transactions, and the agent can securely pull that data from a CRM or customer database.

• Ticket management: By connecting to a helpdesk like Zendesk or Freshdesk, an agent can create, update, or escalate support tickets right from the call.

Interactive personal assistants

Beyond support desks, voice agents with tool-calling skills can be genuinely useful personal assistants.

• Scheduling: They can book appointments or check your availability by hooking into services like Google Calendar.

• Communication: An agent could draft and send an email for you or post a message to a Slack channel, all from a quick voice command.

Internal IT & HR support

Companies are also using this to automate their internal helpdesks, freeing up IT and HR folks from repetitive questions.

• IT helpdesk: An employee could ask a voice bot, "What's the status of my IT ticket?" The agent can then call the Jira or ServiceNow API to give an immediate update.

• HR questions: A new hire could ask about company policies, and the agent could pull answers straight from an internal knowledge base in Confluence or Google Docs.

The payoff for getting this right is pretty obvious: conversations flow without those awkward, robotic pauses; voice agents become active problem-solvers; and customers and employees get answers right away, without sitting on hold.

Challenges of building directly with OpenAI Realtime Tool Calls

While the OpenAI Realtime API is an incredible piece of tech, trying to build a production-ready voice agent on top of it is a whole different beast. It's not a weekend project, and it comes with a bunch of engineering hurdles that can trip up even skilled teams.

Complicated initial setup

Right from the start, you're not just hitting a simple REST API. You have to manage persistent WebSocket or WebRTC connections, juggle dozens of different server and client events, and write a lot of resilient code just to handle the back-and-forth. This requires specialized real-time engineering skills that aren't always easy to find. You're basically building a mini infrastructure project just to get to square one.

Conversation state and session limits

The Realtime conversation guide gives a maximum session duration of 60 minutes. If your product needs information from previous calls, decide what to retain and how to supply it to the next session. A continuing connection and long-term customer history are different responsibilities.

Testing tools and handling failures

Test the voice application before rollout, with controlled tool responses and scenarios such as a missing order, a timeout, or an interrupted reply. The Realtime event flow makes tool requests and outputs observable. Your application must still enforce permissions and decide how to respond when a requested action fails.

For a function result, the documented flow sends a conversation.item.create event containing function_call_output and the matching call_id, then a response.create event to request the next response. Check that your app associates each result with the right request.

Configuration your team can maintain

Separate business instructions from the code that executes tools. A change in tone does not inherently require rewriting a function. A new backend action may require code, credentials, and tests. Decide which settings support managers can change and which need engineering review.

OpenAI Realtime Tool Calls pricing

Realtime costs depend on the model and the audio, text, and other inputs and outputs used. Check the official pricing page for the exact model and modality; audio and text token rates are not interchangeable.

Estimate costs with representative conversations, including their length and tool results, rather than treating every call as identical. Fees for services your tools call may also sit outside your OpenAI bill. eesel workspace billing is separate from this calculation, so it should not be presented as a price for an OpenAI voice session.

Use eesel CLI to prepare and inspect a support teammate

If the goal behind your voice-agent research is better support, I would separate the voice interface from the teammate's support knowledge and instructions. You may need both. eesel CLI lets you configure, question, and inspect the same eesel teammate your colleagues manage in the dashboard. A custom voice interface remains a separate development task.

A person can run the commands directly. A script can repeat setup checks. A coding agent such as Claude Code, Cursor, or Codex can use the JSON results to help inspect instructions, supply knowledge, and review activity. This makes the support teammate accessible from a development workflow without creating a separate copy of it.

Check the support context before expanding automation

For an existing eesel workspace, start with Node.js 18.17 or newer:

npx @eesel/cli login
npx @eesel/cli agents

Choose the support agent from that list and replace <agent-id> in these commands:

npx @eesel/cli status --agent <agent-id>
npx @eesel/cli instructions --agent <agent-id>
npx @eesel/cli activity --agent <agent-id>

Status shows connected sources and whether their content has downloaded. Instructions show the teammate's standing rules; activity shows what it has been doing. These are useful checks before asking it to handle more support work.

For example, upload a real local delivery policy and, after the upload succeeds, ask a question that your customers frequently raise:

npx @eesel/cli files upload ./delivery-policy.pdf --agent <agent-id>
npx @eesel/cli chat "What should we tell a customer whose delivery is three days late?" --agent <agent-id>

Compare the response with the policy. This checks the eesel teammate's answer, not your voice application's speech handling or Realtime tool-call loop. If you later build a custom connection between systems, that connection needs its own implementation, permissions, and end-to-end tests.

Keep review available from the terminal

The CLI also exposes approvals for actions held for human review and automations for the teammate's configured event, scheduled, and webhook automations. Those are eesel workspace controls, not OpenAI Realtime session settings. For write commands, --dry-run shows the server request without sending it; it does not simulate a conversation.

Choose the next step around the support job

Build with OpenAI Realtime when you need a custom live voice experience. Use eesel CLI when you want terminal or coding-agent access to a support teammate's knowledge, instructions, and work. They address different parts of the problem.

If your next step is getting a support teammate ready, try eesel and follow the CLI guide. Begin with one source and one real support question, then review the answer before widening its responsibilities.