Adding ElatoAI project — OpenAI Realtime API Speech on ESP32 devices with Arduino and Deno Edge Functions

[akdeb]

Summary

Adding ElatoAI: A platform for building voice AI agents on device using OpenAI Realtime API Speech with ESP32 on Arduino and Deno Edge Functions for extended conversations with very low latency, globally.

Main repo: https://www.github.com/akdeb/ElatoAI

Motivation

This contribution shows a practical implementation of OpenAI's Realtime API for realtime AI speech-to-speech models in an embedded hardware context on the ESP32 programmed in C++ on Arduino. We are providing developers, a complete framework for creating voice-based AI agents using affordable and accessible hardware.

We launched last week and have found folks cloning the repo 300+ times with 75 forks and 900+ GitHub stars within a week. Essentially, we are lowering the barrier for people to bring AI speech using the OpenAI Realtime API to their projects. You can get started in minutes as shown in this Quickstart video

People have expressed pains about:

1. using ESP-IDF with WebRTC. In fact, the repo which has migrated over to Espressif here is not very popular amongst hobbyists because it is more advanced and has a steeper learning curve.
2. the lack of separation between business logic. With our implementation using an edge relay, we are able to keep business logic separately on our edge server using TypeScript while maintaining core client firmware functions in C++.

Arduino has a thriving community online with over 30 million developers globally. Even more exciting is that Espressif the company that makes the ESP32 board has sold over 1 billion units (combined with its older ESP8266 board). From our launches on Reddit, HackerNews, people have expressed a strong preference to use our repo over the current low-level implementations of Realtime AI Speech.

---

For new content

When contributing new content, read through our contribution guidelines, and mark the following action items as completed:

• I have added a new entry in registry.yaml (and, optionally, in authors.yaml) so that my content renders on the cookbook website.
• I have conducted a self-review of my content based on the contribution guidelines:
  • Relevance: This content is related to building with OpenAI technologies and is useful to others. It demonstrates implementation of OpenAI Realtime API Speech in hardware devices.
  • Uniqueness: I have searched for related examples in the OpenAI Cookbook, and verified that my content offers new insights on hardware integration of OpenAI technologies not covered in existing documentation.
  • Spelling and Grammar: I have checked for spelling or grammatical mistakes.
  • Clarity: I have done a final read-through and verified that my submission is well-organized with clear sections on architecture, components, and implementation.
  • Correctness: The information I include is correct and all of my code executes successfully. The project has been tested and verified working.
  • Completeness: I have explained everything fully, including hardware requirements, software setup, and implementation details needed to reproduce the project.

We will rate each of these areas on a scale from 1 to 4, and will only accept contributions that score 3 or higher on all areas. Refer to our contribution guidelines for more details.

[akdeb]

cc. @kwhinnery-openai Hey Kevin! Just posting an updated readme here with the relevant assets and links to the main repo here. Would love to get your feedback and thoughts on the readme / assets. cc. @bakks