<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Austin Hickam (Mutschler) on Medium]]></title>
        <description><![CDATA[Stories by Austin Hickam (Mutschler) on Medium]]></description>
        <link>https://medium.com/@austinmutschler?source=rss-26ef3a3fb177------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qEqzIXhVOZAdGTNOy_MjTA.jpeg</url>
            <title>Stories by Austin Hickam (Mutschler) on Medium</title>
            <link>https://medium.com/@austinmutschler?source=rss-26ef3a3fb177------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:32:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@austinmutschler/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I built an AI-powered Interdimensional Phone]]></title>
            <link>https://medium.com/@austinmutschler/how-i-built-an-ai-powered-interdimensional-phone-fda78135172e?source=rss-26ef3a3fb177------2</link>
            <guid isPermaLink="false">https://medium.com/p/fda78135172e</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Austin Hickam (Mutschler)]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 15:29:19 GMT</pubDate>
            <atom:updated>2025-06-27T05:18:01.639Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CAyWqme_unjHDMj1x2jyjw.png" /></figure><p>A month before my 29th birthday, I got an idea: what if I used an AI powered landline phone as the centerpiece of my 90s-themed party. Characters will call in to talk to guests and send them on story-driven adventures throughout the house. Guests can call characters to chat and characters can control technology throughout the house. It sounded fun, but my background is in software, not hardware so how could I make something in such a short deadline?</p><p>There’s been buzz that AI chatbots and agents could replace much of today’s software engineering workload within 1–2 years. I decided to utilize LLMs throughout this project to see if they could help me build something usable in an area of software I barely knew. I needed all the help I could get. I used <a href="http://t3.chat/">T3.chat</a> as my primary AI chat platform because it provided an easy cost-effective way to try out different LLMs throughout my journey (thanks Theo).</p><h3>Deciding on an Approach</h3><p>First, I needed to decide how to integrate AI characters with a landline phone.</p><h4>Option 1</h4><p>Replace a phone’s internal electronics with a Raspberry Pi in order to take control of its keypad, speaker, and microphone. Additionally, create software to emulate the phone’s behavior.</p><h4>Option 2</h4><p>Build my own VoIP network and write software to simulate phone calls and pull and inject audio to and from the phone line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yOpTpMZkkCRJWW-utSv5Kg.png" /></figure><p>When making this decision, I compared the known limitations of both the technology and my experience level.</p><p>I lacked electronics experience, but setting up a VoIP network seemed even more challenging. Plus, I wasn’t confident it’d deliver the results I needed. There aren’t many guides online for setting up VoIP on a Raspberry Pi and that made me nervous. VoIP equipment can be expensive and I didn’t want to take the risk.</p><p>For those reasons, I decided to go with Option 1 as it seemed more straightforward and gave me the most control over the phone.</p><h3>The Plan 📄</h3><ol><li>Use various models from T3.chat to learn about hardware engineering</li><li>Acquire a landline phone</li><li>Learn how to trace paths on PCB boards for keypad matrix grids button press capture</li><li>Solder wires from the phone’s board to the Pi</li><li>Build software to emulate the phone dialing, keypresses, answering and hanging up</li><li>Solder the speaker and microphone wires to the Pi</li><li>Build a small server to simulate a phone network, allowing calls in and out to AI characters</li><li>Add real-time voice AI characters using OpenAI Realtime models</li></ol><h4>Finding a Phone</h4><p>I decided on the <a href="https://www.amazon.com/AT-Trimline-Corded-Required-Wall-Mountable/dp/B000050FZP?th=1">AT&amp;T Trimline 210</a> landline phone because it has a classic 90s look and will fit right in at my party. Additionally, that model is still in production so I wouldn’t have to worry about breaking an antique. Most importantly, its large clamshell design should have enough room to fit a Raspberry Pi Zero 2 without much modification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*3ytQy3r4A_mEqzlo" /></figure><p>My new landline phone arrived and I wasted no time before disassembling it. The housing’s latch was fragile, but I was eventually able to get it opened.</p><p><strong>Problem 1</strong>: There was a lot less room in the phone housing than I was expecting. I put my Pi Zero on top of the board and was able to close the phone lid with little room to spare. This may become a problem when I get the wires in later.</p><p>I unscrewed the board from the housing to reveal the full PCB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qusWGIAHhAuPh0H0EN2ZZg.jpeg" /></figure><h4>Capturing Keypresses</h4><p>My goal was to use the phone’s original PCB board and send the numbers being pressed to the Raspberry Pi. To do this, I needed to somehow wire the board to send me the digits.</p><p>I utilized OpenAI’s GPT o3-mini and Gemini 2.5 Pro to learn how to trace a <a href="https://www.jameco.com/Jameco/workshop/JamecoBuilds/working-with-matrix-keypads.html?srsltid=AfmBOorWJqglzKXp_Wtyc3H6-9t3a2I16gJOLbC3nk9_NycCNCmXanx-">keypad matrix path</a>. I quickly discovered that this phone’s keypad was not in the expected row and column configuration. I expected each column trace to run straight top-to-bottom. On this board, the columns often bled into other columns connecting seemingly random keys.</p><p>I took some pictures of the phone’s board with a light shining through the back to make finding the traces easier. I provided these to both o3-mini and Gemini 2.5 Pro; however, they hallucinated answers for the matrix setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQ8oHJ_1uAeF-VifV55zPA.jpeg" /></figure><p>At this point, I decided to reach out to some of my electrical engineering friends to help me figure out how to map the keypad properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*WLrcEJI9kHv7SPre" /></figure><p>During the call with my friend I shared images of the board with him through Figma. He taught me how to map the keys. To map the phone keys, we needed to test the continuity from a <em>via</em> (small holes on the board) with the top or bottom of the button pad. You can accomplish this with a multimeter. One <em>via</em> will be connected to the top of the keypad and another <em>via</em> will be connected to the bottom of the pad. The Raspberry Pi passes a small amount of voltage through either all the columns or all the rows and then continuously <em>scans</em> for a key to be <strong>low</strong> for both the column and row. When the keypad is pressed, via A and via B will both be <strong>low </strong>which indicates a keypress in the phone’s software.</p><p>These vias are likely being used for phone quality control testing.</p><p>We created an overlay that shows the via to keypad mapping. C1 = Column 1. R1 = Row 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uBVd8qucAZUBlIYhMN4hwA.png" /></figure><p>I mapped all the columns and rows into an object so that our software knew which keys are pressed when any two vias are <em>low</em>.</p><p>The wires are connected from the vias to GPIO pins on the Raspberry Pi.</p><p><strong>Problem 2</strong>: There are 4 rows and 5 columns. This means we need 9 wires connected to the Pi. This could cause space issues inside the phone.</p><p>After setting up the column GPIO pins on the Raspberry Pi to “pull-up” mode, I wrote some quick code to test if the buttons were being detected. Success!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*PDjCVug5gBqgC2juwUwXhw.png" /></figure><p>Adding two hook-switch wires to detect when the phone was on-or-off hook brought me to 11 total wires. Yikes!</p><h4>Picking a Programming Language</h4><p>For the phone emulation software, I decided to use the one programming language to rule them all… Typescript.</p><p>Wait… don’t leave… hear me out. NodeJS and Typescript have extensive community packages readily available. They are great for rapid prototyping without dealing with compiler issues or strict typing. This project didn’t have extreme memory or CPU constraints. I’m not making a commercial product where I need scalability. <strong>Most importantly, it’s a stack I’m familiar with that will get the job done before my tight deadline arrives.</strong></p><p>Who knows, maybe I’ll consider rewriting it in Rust one day 😉</p><h4>Phone Emulation Software</h4><p>I began writing the code for the full phone simulation. The main focus areas are:</p><ol><li>Capture the phone keypresses</li><li>Store the key in a dial buffer with a timeout that begins dialing after a few seconds</li><li>Play the correct DTMF sound for each key pressed</li><li>Detect when the phone is in on-or-off hook state</li><li>Output phone events using the EventEmitter for our future simulated “phone network” to use</li></ol><p>I simulated DTMF tones via mpg123 child processes and mp3 files but that introduced a full second of lag between press and tone.</p><p>To solve this, o3-mini suggested keeping a persistent mpg123 process using -R mode and passing in audio commands through stdin. This worked and eliminated the lag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*MWZgK-NjZtHL-Dy5lhh1ww.png" /></figure><p>I began designing my Phone class. I needed to keep track of the phone’s current state to determine what action the phone should perform next. These states helped organize my code so that I knew when the users were attempting to call a number versus when they were pressing numbers during a call to potentially answer trivia questions. The main phone states were idleOnHook, idleOffHook, calling, and callInProgress.</p><p>I used the Javascript EventEmitter as the base class for my Phone class. This allowed other parts of my application to listen for phone events to trigger actions like placing a call, hanging up, etc.</p><pre>// Example of using the phone class to handle hangup logic<br>const handleCall = async (phone: Phone, numberDialed: number) =&gt; {<br>  phone.on(&#39;hungup&#39;, () =&gt; {<br>    console.log(`Phone hung up. Last status was: ${lastStatus}`);<br>  });<br>};</pre><h4>Getting Phone Audio to Work</h4><p>Now that the keypad matrix was fully working, and the Phone class was created, I turned my attention to the phone’s audio. This was going to be challenging because the Raspberry Pi Zero 2 does not have the hardware components to support audio output. The landline phone uses a standard mono speaker and an electret microphone.</p><p>Gemini 2.5 Pro recommended buying the <a href="https://www.amazon.com/Waveshare-Raspberry-Encoding-Recording-Interface/dp/B07KN8424G">Waveshare WM8960 audio hat</a>. These hats are designed to fit on top of the Raspberry Pi. I purchased it but when it arrived…</p><p><strong>Problem 3</strong>: The hat has two built-in MEMS microphones and an area for audio output but it doesn’t support external analog electret microphones. The WM8960 audio chip supports mic input, but the board didn’t expose it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ho_iB680esvI3Qga" /></figure><p>I decided to give my AI companion another shot. I explained the issue with the audio board and it recommended the Teensy 4 Audio Shield. On paper, this board had everything I needed. Although mainly designed for the Teensy microcontroller, it <em>should</em> be adaptable to my use case if paired with a small speaker amp. When it arrived, I tried to hook it all up but quickly found an issue.</p><p><strong>Problem 4</strong>: The Teensy board requires a MCLK clock that the Raspberry Pi Zero does not expose over GPIO. This means the board will not work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/0*zYDKj9im94Qqo1qh" /></figure><p>As the party deadline was quickly approaching I knew I only had one more shot at getting an audio board that met all my criteria. o3-mini found a breakout <a href="https://www.sparkfun.com/sparkfun-audio-codec-breakout-wm8960.html">audio board by Sparkfun</a> that used the same WM8960 audio chip as the original board I purchased. This new board has MIC output and doesn’t require the MCLK clock. Finally, a board that checks every box. I read the documentation myself and everything was in order, so I purchased it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9mSfHvUJts9WAdLIfnlaMw.jpeg" /></figure><p>When it arrived, I asked the new OpenAI o4-mini model to summarize where I should solder the wires from the audio board to the GPIO headers on the PI. It gave me detailed instructions on how to set it up. I did a lot of additional research and then <a href="https://docs.google.com/document/d/1FWMWBcfJaQq8gk7Eq6OFiZXyvJO5tl6QuTg_3xw1CdY/edit?usp=sharing">documented the entire setup process</a>.</p><p>In the Pi operating system I set up the audio overlay, turned up the speaker volume and played a test song. For the first time the phone came alive.</p><p>Then I tested the microphone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6D2q2DR5EHviwDMrD0OBDw.jpeg" /></figure><p><strong>Problem 5</strong>: the microphone wasn’t working.</p><p>I double-checked my connections. I read the <a href="https://learn.sparkfun.com/tutorials/audio-codec-breakout---wm8960-hookup-guide#example-14-electret-mic">microphone section of the audio board documentation</a> again. Everything seemed right. I consulted with o4-mini and Gemini on the problem. They both told me to check the MICBIAS. This is a port that sends voltage to the electret microphone. I probed it with my multimeter and it read 0V. Both LLMs told me there was an audio setting within amixer (audio application for Linux) to turn on MICBIAS; however, this was a lie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S7Gys7rs3UdgFNCy_RH1ig.png" /></figure><p>I began doing extensive research on this problem. Eventually I discovered that the audio overlay (controls) used by Raspberry Pi OS are for the original Waveshare audio hat I purchased. This hat doesn’t have a MICBIAS setting because its microphones are built into the board. It was all making sense now. They would not include MICBIAS settings in their overlay because they simply don’t need it.</p><p>I tried repeatedly to have o4-mini help me build an overlay, but everything we tried failed to turn on MICBIAS. I posted to a few online forums with no luck.</p><p>I had been working on the project off and on for nearly three weeks. I spent a lot of time waiting for parts to come in, learning soldering techniques, and troubleshooting audio board issues.</p><p>I still hadn’t started writing the code for the phone network or AI characters yet.</p><p>The party was in 6 days. With the deadline rapidly approaching, I had no choice but to abandon this approach entirely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GJBR8wS8ZoCkXH44FUX43Q.png" /></figure><h3>Time to Pivot</h3><p>Lying in bed at 2am, exhausted and defeated, I grabbed my phone and asked o4-mini to revisit the option I’d rejected before; setting up a VoIP servers on a Raspberry Pi 4. It recommended using an open source VoIP software called Asterisk and an interface to manage it called FreePBX. After probing more, it told me Asterisk had a REST API called ARI that gives developers extensive control over the VoIP network.</p><p>There might be something here, I thought.</p><h3>The New Plan</h3><ol><li>Purchase a VoIP Adapter that connects the phone to the Raspberry Pi</li><li>Setup a VoIP server on a Raspberry Pi 4 I already own</li><li>Buy another landline phone because the original was no longer operational</li><li>Build a backend server to interface with the VoIP network that allows AI to place and receive calls</li><li>Design fun AI characters</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cg9E-Fb-Nz-oavZXkfRcbw.png" /></figure><p>o4-mini and Gemini 2.5 Pro both told me there was an easy setup for Asterisk and FreePBX called <a href="http://www.raspbx.org/">RasPBX</a>. After doing a lot of research, I found that RasPBX was abandoned years ago with no support for current Raspberry Pi versions.</p><p>Luckily I found another guide online that showed how to set everything up. The guide was difficult to find because it existed on a site that recently shut down called DSLReports. I used the Wayback Machine to read the original post and download the setup files. I was also able to find another guide on GitHub that helped tremendously. <a href="https://github.com/playfultechnology/RasPBX">https://github.com/playfultechnology/RasPBX</a></p><p>For the VoIP equipment, I researched a few devices that had one phone jack and found the <a href="https://a.co/d/3l0YuOR">FlyingVoice FTA1101 VoIP Adapter</a>. This device is compact and plugs directly into a wall outlet without extra cables. I uploaded all of its user guides and documentation into Gemini 2.5 Pro to evaluate its compatibility with Asterisk. It appeared to be compatible, so I bought it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xX-CrhYIZi6uQOxa" /><figcaption>FlyingVoice FTA1101 VoIP Adapter</figcaption></figure><p>The VoIP adapter arrived the next day along with the new landline phone. I gave the adapter’s user guide to the LLMs and asked it to give me step by step setup instructions. It provided very detailed instructions that would have taken a while to figure out on my own.</p><p>VoIP adapters use SIP, a protocol for real time media connections. I configured the SIP account to assign a phone extension and gave it the IP address of the VoIP server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PFY8LE5hMTO_Bx_P5jMJaA.png" /></figure><p>Once the adapter was set up, I connected the Ethernet cable from the adapter to the Pi’s LAN port. I wanted to keep the VoIP network contained to just the Pi and not be accessible over my local network so I bridged the VoIP adapter with the local Pi network to keep it contained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ldLp9cT-9YNm4IJMztb0Zw.jpeg" /></figure><p>Then, I began setting up Asterisk and FreePBX. Once Asterisk and FreePBX were installed, I opened the FreePBX web interface, added my phone extension and made some modifications to the dialplan so that when people dial out, it will route the call to my custom phone server instead of attempting to make a real phone call.</p><p>I wrote a <a href="https://github.com/AustinMutschler/90s-AI-Interdimensional-Phone/blob/main/Raspberry-Pi-Asterisk-FreePBX-Setup.md">detailed guide</a> on how to set everything up if you are interested in the full details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PP-pNd0SsBCOuN-X8ubhug.png" /></figure><h4>Setting up my Custom Phone Server</h4><p>I turned my attention to the NodeJS phone server. I began by deleting all of the previous phone emulation code I had written. That code was capturing the phone button presses and handling phone state which was no longer needed with the VoIP approach.</p><p>The new code focused on:</p><ol><li>Detecting an outgoing call and routing it to the correct AI character</li><li>Connecting to the phone through the VoIP server to send and receive audio during a call</li><li>Scheduling characters to call in at certain times throughout the party</li></ol><p>I built a basic routing service that detected when a call was being placed and routed the call to an AI character function with its own unique prompt and storyline. This architecture allowed me to “register” characters to a phone number with a prompt, making it easy to add new characters.</p><pre>// Register an inbound handler for a specific dialed number<br>public registerHandlerByNumber(<br>  number: string,<br>  handler: (event: ChannelEvent, mediaSession: MediaSession) =&gt; Promise&lt;void&gt;<br>) {<br>  this.inboundHandlers.set(number, handler);<br>}</pre><h4>Audio Streaming Setup</h4><p>Then I built the MediaSession class. This controls audio in and audio out for the phone. Asterisk API does this using External Media channels. First, I created a UDP server (using dgram), then I provided the External Media Channel with the IP address and port of the UDP server. Audio can then be piped in and out of the UDP server. Normally this functionality is used for call recordings or injecting music into the call.</p><p>To test this audio code, I set up a function using @ffmpeg that records audio coming in from the phone line.</p><p>I picked up the phone and placed a call to the dummy phone number. I began speaking and then hung up. A file appeared in my server “test.wav”. I put my headphones on and played the file. “Hello, test, test, test”. After a month of trying, I finally have a phone that can capture audio.</p><p>What a relief!</p><p>It was time to test sending audio back to the phone. This is a similar process used for recording. I created another UDP server and gave the port number to Asterisk using the external media endpoint. Then, I bridged it together with the rest of the media channels Phone -&gt; Audio In (UDP) -&gt; Audio Out (UDP).</p><p>I ran my makeCall unit test and the phone began to ring.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F76LlNDrO5ug%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D76LlNDrO5ug&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F76LlNDrO5ug%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/aeb362a73d985bf733f4e9ae808cde6f/href">https://medium.com/media/aeb362a73d985bf733f4e9ae808cde6f/href</a></iframe><p>I’d never been happier to be Rick Rolled. I decided to play the recording of the call to see what the microphone captured. Oh no… I could hear the music clearly.</p><p>This means that I wasn’t isolating the microphone audio from the audio I was streaming to the phone. That would cause an audio feedback loop during real calls resulting in the AI character interrupting itself while talking.</p><p>I found the issue in the media bridge code. All three media layers are bridged together which means if we played audio through the audio in, it would be captured by the audio out UDP server. After a few hours of trial and error, I discovered that instead of sending the audio to the media bridge, I can send it directly to Asterisk using the UNICASTRTP_LOCAL_PORT. This sends audio to the phone directly and not to the media bridge fixing the feedback loop.</p><pre>public async setup(): Promise&lt;void&gt; {<br>  // socket where Asterisk will send us audio<br>  this.recvSocket = dgram.createSocket(&#39;udp4&#39;);<br>  this.recvSocket.bind(0, &#39;127.0.0.1&#39;);<br>  await once(this.recvSocket, &#39;listening&#39;);<br>  const portRecv = (this.recvSocket.address() as any).port;<br><br>  // socket from which we&#39;ll send media _to_ Asterisk<br>  this.sendSocket = dgram.createSocket(&#39;udp4&#39;);<br>  this.sendSocket.bind(0, &#39;127.0.0.1&#39;);<br>  await once(this.sendSocket, &#39;listening&#39;);<br><br>  // Transmit ExternalMedia leg<br>  const extSend: Channel = await ariClient.channels.createExternalMedia({<br>    app: &#39;phone-app&#39;,<br>    external_host: `127.0.0.1:${portRecv}`,<br>    format: &#39;ulaw&#39;,<br>    direction: &#39;outbound&#39;<br>  });<br><br>  // This is the RTP port that we need to send audio to<br>  this.extensionSendPort = extSend[&#39;channelvars&#39;][&#39;UNICASTRTP_LOCAL_PORT&#39;];<br><br>  // Initializes error and close handlers<br>  this.initializeSocketErrorHandlers();<br><br>  // Create bridge that combines the original call channel with the external media legs<br>  const bridge = await ariClient.bridges.createBridge({ type: &#39;mixing&#39; });<br>  await ariClient.bridges.addChannels(bridge.id, { channel: [this.channel.id, extSend.id] });<br>}</pre><p>Having no prior experience with audio streaming, I asked o4-mini to help me learn and write code for real-time audio. Both Asterisk and OpenAI support g711 ulaw audio codecs natively. This means I can pass audio directly in and out of the OpenAI API without having to modify it.</p><p>If we try to play the audio streamed in from <a href="https://platform.openai.com/docs/guides/realtime">OpenAI’s realtime API</a>, it would sound very fast or be distorted because the rate we play the audio is different than the speed in which the audio packets are received. We need to add the audio packets to a queue and then play them at a specific rate.</p><p>Once the queue was set up I started my OpenAI integration so everything could be tested.</p><h4>OpenAI Realtime API Setup</h4><p>The integration with OpenAI was fairly simple. I opted to connect via WebSocket to have more control over the interaction. I built different handlers for the events the AI websocket server can respond with such as tool calls, turn detection and audio out. These events can be a bit cumbersome to understand but the OpenAI documentation helped a lot… once I found it.</p><h4>Character Class Setup</h4><p>Now that the Media Service and OpenAI setup was complete, I started to work on the actual Character class. The characters needed a prompt, voice, and a phone number. If they were call-in characters, they need a call schedule.</p><h4>Bringing the phone to life</h4><p>After all the pieces were in place, it was time to fill the phone with fun characters. The main storyline was an escape room style game driven by an AI character. The character would call in, taking guests on an adventure throughout the house.</p><h4>The Story</h4><blockquote>Doctor Henderson has been studying a frequency that he believes connects our world with other dimensions. Everyone in his field believes he’s insane. His lab monitors the frequency for any anomalies. Today, the frequency amplification went off the charts and destroyed Doctor Henderson’s lab equipment. He tracked down the frequency amplification source to a house nearby. When he opened the door, the house was empty except for a landline phone. When he picked it up, the phone began to dial.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*APwW5WvvZWirM4CMJ-v8gA.png" /></figure><p>Doctor Henderson helps the party guests get back to their reality by giving them activities to <em>stabilize</em> the frequency. Each conversation reveals the next challenge.</p><p>In the real world, there were escape room puzzle games around the house. One game had guests looking for secret symbols written in invisible ink on posters. Once guests decoded the symbols into numbers they called the phone number to unlock the next challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zZepqjd8UfdGRwEWwnFnUg.png" /></figure><p>Dr. Henderson is programmed with a schedule. Each scheduled call has a prompt, start time, and an optional condition. The start time and condition need to be met before the character will call in. This ensured that if the guests didn’t complete the puzzle in time, the character would not call the phone until they do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWRcvM1pdto__spXW4siNQ.png" /></figure><p>I needed to make a new type of number in the phone system called a ConditionNumber. When these numbers are called, they mark a condition as complete in a small SQLite database. The database ensures that if the server restarts, the game state will be saved. I built a very basic webpage that outputs the call schedule and conditions table so I can monitor the game state to make sure it was working properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ZgXdRgxLeZZTG_OKYKBVQ.png" /></figure><p>Then, I developed fun AI characters that guests can call at any time. These characters have their own unique backstories.</p><ol><li>108.3 Radio Station — 90s Hits</li><li>Blockbuster</li><li>Beets-a-Hut</li><li>IT Support</li><li>Emergency Services</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FlU-IIkcX6PM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DlU-IIkcX6PM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FlU-IIkcX6PM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f99428a0a75903a6b7568187af385ae0/href">https://medium.com/media/f99428a0a75903a6b7568187af385ae0/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FxhyyqHoYJ68%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxhyyqHoYJ68&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FxhyyqHoYJ68%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/662c229aa7f10871853cc35f49209bb5/href">https://medium.com/media/662c229aa7f10871853cc35f49209bb5/href</a></iframe><h4>QA Time</h4><p>Now that all of the characters were set up, I began performing QA tests.</p><p>The first problem I ran into was cutting off the AI while they were speaking. This is a common scenario called voice activity detection (VAD). OpenAI has built-in VAD that will pause the AI from speaking when it detects the user talking. The issue was, there may still be a few seconds of audio in the queue waiting to be played when VAD is triggered. Waiting for the audio queue meant the AI would talk over guests. This was a jarring lag that broke the illusion of a real conversation.</p><p>To fix this, we can manually clear the audio queue when OpenAI sends the “input_audio_buffer.speech_started” event. This isn’t perfect but worked well for my use case.</p><p>Then, while testing the characters to make sure they were fun and stayed in character I realized that they needed the ability to hang up. It felt awkward that the character would say “goodbye” but then you could continue to talk to them. I needed to add tool/function calling.</p><h4>Tool Calling</h4><p>This was relatively easy to do with the OpenAI API. After implementing the hangup tool, it led to some pretty funny interactions. When the 911 operator was tired of your prank calls, she would simply hang up on you.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3kvPnfO-2qQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3kvPnfO-2qQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3kvPnfO-2qQ%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb623964c4b36b28dd629aaeae384051/href">https://medium.com/media/fb623964c4b36b28dd629aaeae384051/href</a></iframe><p>There was only one issue with tool calling. Remember the audio queue that I mentioned earlier? When a character called the hangup function, I needed to wait for the character to stop talking otherwise the audio would cut out before they finished their last sentence. I fixed this by holding the hangup function in a setInterval until the character’s audio queue became empty. Once the queue was empty, the hangup function would be called.</p><p>Wouldn’t it be cool if the characters could do more than talk? Maybe guests could call into the radio station and have the DJ recommend 90s hits and add them to the party’s music queue.</p><p>The music for my party was played through my four Amazon Echo speakers in a group to give cheap but effective whole home audio. I used Spotify to play to an Amazon Alexa speaker group, which made everything seamless.</p><p>I went to the <a href="https://developer.spotify.com/documentation/web-api">Spotify API</a> website and generated an API key. I needed their search endpoint to find the Track ID and their Player Queue endpoint to add the track to my currently playing music queue. I added “play_song_by_title” to the DJ characters function list, then added my Spotify API client code and tried it out.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FFK5FuWRwexw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFK5FuWRwexw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FFK5FuWRwexw%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5794f6a3dacba73a1fea0e657ef46176/href">https://medium.com/media/5794f6a3dacba73a1fea0e657ef46176/href</a></iframe><p>It worked great!</p><h3>Party Time!</h3><p>The party was a huge success. My friends had a lot of fun playing the escape room game with Dr. Henderson. Everyone loved the AI characters and the DJ was spinning tracks all night long.</p><p>Surprisingly, the phone didn’t have any problems besides being occasionally cut off by the loud sounds at the party.</p><p>Overall, I had a blast building this project. I enjoyed seeing how effective LLMs were at teaching me subjects I was unfamiliar with. They aren’t perfect and took me down rabbit holes but in the end I was able to make a working phone.</p><p>I hope you enjoyed this walkthrough of my AI powered interdimensional phone journey!</p><h3>Code</h3><p>The code for this project can be found on GitHub. It’s pretty messy because the speed at which I had to write it. The techniques may be helpful, but I wouldn’t recommend using it as a base for your project. <a href="https://github.com/AustinMutschler/90s-AI-Interdimensional-Phone">https://github.com/AustinMutschler/90s-AI-Interdimensional-Phone</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fda78135172e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>