AI Conversation Implementation Tutorial

A practical guide for implementing two different AI conversation approaches in ESOL learning platforms

Overview

This project implements two distinct approaches for AI-powered voice conversations:

Sequential Recording (/practice/nzcel/conversation)

• Turn-based conversation with manual recording control
• Uses separate API calls for transcription, response generation, and TTS
• Best for structured practice scenarios

Realtime Streaming (/speaking)

• Natural, free-flowing conversation with automatic turn detection
• Uses OpenAI Realtime API with WebRTC for bidirectional streaming
• Best for realistic speaking practice

Implementation #1: Sequential Voice Recording

Architecture

User Speaks → Record Audio → Stop Recording → Transcribe (Whisper)
    ↓
Display Transcript → Generate Response (GPT-4) → Convert to Speech (TTS)
    ↓
Play Audio → Wait for User → Repeat

Core Files

src/
├── app/(main)/practice/nzcel/conversation/page.tsx
├── components/conversation/realtime-conversation.tsx
├── hooks/use-voice-recorder.ts
└── app/api/openai/
    ├── transcribe/route.ts    # Whisper API
    ├── conversation/route.ts  # GPT-4 Chat
    └── tts/route.ts          # Text-to-Speech

Key Technologies

• Audio Recording: MediaRecorder API (audio/webm)
• Transcription: OpenAI Whisper (whisper-1)
• AI Response: GPT-4 Turbo (gpt-4-turbo-preview)
• Text-to-Speech: OpenAI TTS (tts-1)

Implementation Pattern

1. Voice Recording Hook (use-voice-recorder.ts)

export function useVoiceRecorder() {
  const [state, setState] = useState<VoiceRecordingState>({
    isRecording: false,
    audioBlob: null,
    transcription: null,
  });
​
  const startRecording = async () => {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    const mediaRecorder = new MediaRecorder(stream, { mimeType: "audio/webm" });
    
    mediaRecorder.ondataavailable = (event) => {
      chunks.push(event.data);
    };
    
    mediaRecorder.onstop = () => {
      const blob = new Blob(chunks, { type: "audio/webm" });
      setState(prev => ({ ...prev, audioBlob: blob }));
    };
    
    mediaRecorder.start();
  };
​
  return { state, startRecording, stopRecording, transcribe, assess };
}

2. Conversation Flow (realtime-conversation.tsx)

const processUserSpeech = async () => {
  // 1. Transcribe audio
  const formData = new FormData();
  formData.append("audio", new File([audioBlob], "recording.webm"));
  
  const transcribeResponse = await fetch("/api/openai/transcribe", {
    method: "POST",
    body: formData,
  });
  const { text } = await transcribeResponse.json();
  
  // 2. Generate AI response
  const aiResponse = await fetch("/api/openai/conversation", {
    method: "POST",
    body: JSON.stringify({
      messages: [
        { role: "system", content: scenarioContext },
        ...conversationHistory,
        { role: "user", content: text }
      ],
    }),
  });
  const { response } = await aiResponse.json();
  
  // 3. Convert to speech
  const ttsResponse = await fetch("/api/openai/tts", {
    method: "POST",
    body: JSON.stringify({ text: response }),
  });
  const audioBlob = await ttsResponse.blob();
  
  // 4. Play audio
  const audio = new Audio(URL.createObjectURL(audioBlob));
  await audio.play();
};

3. API Routes

// Transcription (transcribe/route.ts)
export async function POST(request: NextRequest) {
  const formData = await request.formData();
  const audioFile = formData.get("audio") as File;
  
  const transcription = await openai.audio.transcriptions.create({
    file: audioFile,
    model: "whisper-1",
    language: "en",
  });
  
  return NextResponse.json({ text: transcription.text });
}
​
// Conversation (conversation/route.ts)
export async function POST(request: NextRequest) {
  const { messages } = await request.json();
  
  const completion = await openai.chat.completions.create({
    model: "gpt-4-turbo-preview",
    messages,
    temperature: 0.8,
    max_tokens: 300,
  });
  
  return NextResponse.json({ response: completion.choices[0].message.content });
}

Implementation #2: OpenAI Realtime API

Architecture

User Speaks (continuously) ←→ WebRTC Connection ←→ OpenAI Realtime API
         ↓                           ↓                        ↓
  VAD Detection              Auto-transcription        AI Response (voice)
         ↓                           ↓                        ↓
    Turn Detection  →  Save to Database  ←  Stream Audio Back

Core Files

src/
├── app/(main)/speaking/page.tsx
├── components/speaking/ai-coach.tsx
└── app/api/openai/realtime-client-secret/route.ts

Key Technologies

• SDK: @openai/agents/realtime
• Transport: WebRTC with bidirectional streaming
• Voice Activity Detection: Server-side VAD (threshold: 0.5, silence: 500ms)
• Transcription: Built-in (gpt-4o-mini-transcribe)
• Response: Real-time audio streaming

Implementation Pattern

1. Session Initialization

import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";
​
const startSession = async () => {
  // Get ephemeral client secret
  const response = await fetch("/api/openai/realtime-client-secret", {
    method: "POST",
    body: JSON.stringify({
      voice: "verse",
      instructions: ESOL_COACH_INSTRUCTIONS,
    }),
  });
  const { clientSecret } = await response.json();
  
  // Create agent and session
  const agent = new RealtimeAgent({
    name: "ESOL Coach",
    instructions: ESOL_COACH_INSTRUCTIONS,
  });
  
  const session = new RealtimeSession(agent, {
    transport: "webrtc",
    config: {
      audio: {
        input: {
          transcription: { model: "gpt-4o-mini-transcribe" },
        },
      },
      turn_detection: {
        type: "server_vad",
        threshold: 0.5,
        silence_duration_ms: 500,
      },
    },
  });
  
  // Connect to OpenAI
  await session.connect({ apiKey: clientSecret });
  
  // Setup event handlers
  setupEventHandlers(session);
};

2. Event Handlers

const setupEventHandlers = (session: RealtimeSession) => {
  // Handle message history updates
  session.on("history_updated", (history) => {
    const messages = history
      .filter(item => item.type === "message")
      .map(item => ({
        role: item.role === "user" ? "user" : "assistant",
        content: extractTranscript(item.content),
        timestamp: new Date(),
      }));
    
    setMessages(messages);
    saveToDatabase(messages);
  });
  
  // Handle voice activity detection
  session.on("transport_event", (event) => {
    if (event.type === "input_audio_buffer.speech_started") {
      setIsSpeaking(true);
      startRecordingUserAudio();
    } else if (event.type === "input_audio_buffer.speech_stopped") {
      setIsSpeaking(false);
      stopRecordingUserAudio();
    }
  });
  
  // Handle errors
  session.on("error", (error) => {
    console.error("Session error:", error);
    toast.error("Conversation error occurred");
  });
};

3. Client Secret Generation

// realtime-client-secret/route.ts
export async function POST(request: NextRequest) {
  const { voice, instructions } = await request.json();
  
  const sessionConfig = {
    session: {
      type: "realtime" as const,
      model: "gpt-realtime",
      audio: { output: { voice } },
      instructions,
    },
  };
  
  const response = await fetch(
    "https://api.openai.com/v1/realtime/client_secrets",
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(sessionConfig),
    }
  );
  
  const data = await response.json();
  return NextResponse.json({
    clientSecret: data.value,
    expiresAt: data.expires_at,
  });
}

4. Audio Recording Synchronization

// Track user audio for database persistence
const mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });
const userAudioChunks = {};
const recordingToMessageMap = new Map();

session.on("transport_event", (event) => {
  if (event.type === "input_audio_buffer.speech_started") {
    const recordingIndex = userRecordingCounter++;
    const currentUserMessageCount = session.history
      .filter(item => item.type === "message" && item.role === "user").length;
    
    recordingToMessageMap.set(recordingIndex, currentUserMessageCount);
    
    mediaRecorder.ondataavailable = (e) => {
      if (!userAudioChunks[recordingIndex]) {
        userAudioChunks[recordingIndex] = [];
      }
      userAudioChunks[recordingIndex].push(e.data);
    };
    
    mediaRecorder.start();
  } else if (event.type === "input_audio_buffer.speech_stopped") {
    mediaRecorder.stop();
  }
});

Comparison Matrix

| Feature | Sequential Recording | Realtime Streaming |
| — — — — -| — — — — — — — — — — -| — — — — — — — — — -|
| **Latency** | 3–5 seconds per turn | <1 second |
| **User Experience** | Manual button clicks | Natural conversation flow |
| **Implementation Complexity** | Medium (3 API calls) | High (WebRTC + events) |
| **API Usage** | 3 separate calls per turn | Single persistent connection |
| **Cost Model** | Pay per API call | Pay per minute ($0.06/min input, $0.24/min output) |
| **Browser Support** | Wide (MediaRecorder API) | Modern browsers (WebRTC) |
| **Turn Detection** | Manual (user clicks stop) | Automatic (Voice Activity Detection) |
| **Audio Format** | WebM → uploaded | PCM streaming |
| **Conversation Style** | Turn-based (structured) | Free-flowing (natural) |
| **Setup Time** | ~1 hour | ~3–4 hours |
| **Debugging** | Easy (inspect each step) | Moderate (event-driven) |
| **Interruption Handling** | Not supported | Built-in (barge-in) |
| **Best For** | Practice exercises, assessments | Conversational fluency, immersion |

Quick Implementation Guide

Prerequisites

npm install openai @openai/agents sonner

Environment Variables

OPENAI_API_KEY=sk-...

Sequential Recording Setup (30 mins)

Create voice recorder hook

• Copy src/hooks/use-voice-recorder.ts
• Handles MediaRecorder lifecycle

Create API routes

• /api/openai/transcribe - Whisper transcription
• /api/openai/conversation - GPT-4 responses
• /api/openai/tts - Text-to-speech

Build conversation component

• Use useVoiceRecorder() hook
• Chain: record → transcribe → respond → speak
• Display conversation history

Test workflow

• Request microphone permission
• Record → transcribe → respond
• Verify audio playback

Realtime Streaming Setup (2 hours)

Install Realtime SDK

npm install @openai/agents

Create client secret endpoint

• Copy src/app/api/openai/realtime-client-secret/route.ts
• Generates ephemeral tokens

Build coach component

• Initialize RealtimeAgent and RealtimeSession
• Setup WebRTC transport
• Configure VAD parameters

Implement event handlers

• history_updated - Update conversation transcript
• transport_event - Handle speech detection
• error - Handle connection issues

Add audio recording (optional for persistence)

• Setup parallel MediaRecorder
• Sync recordings with message indices
• Upload to storage

Test real-time flow

• Verify WebRTC connection
• Test voice activity detection
• Confirm natural turn-taking

When to Use Each Approach

Choose Sequential Recording When:

✅ Budget is a primary concern

• Pay only for API calls used
• Predictable costs per conversation

✅ You need full control

• Explicit start/stop recording
• Validate each step separately
• Custom processing between steps

✅ Implementing structured practice

• Question-answer format
• Assessment-focused exercises
• Clear turn boundaries

✅ Supporting older browsers

• Requires only MediaRecorder API
• Broader compatibility

✅ Debugging is critical

• Easy to inspect each pipeline stage
• Clear error boundaries

Choose Realtime Streaming When:

✅ User experience is paramount

• Natural, free-flowing conversations
• Minimal latency (<1 second)
• Interrupt and respond (barge-in)

✅ Simulating real conversations

• Speaking fluency practice
• Interview preparation
• Casual dialogue

✅ Users have modern browsers

• WebRTC support required
• Chrome, Edge, Safari (recent versions)

✅ Volume justifies setup time

• Higher upfront complexity
• Better UX for frequent use

✅ You want built-in features

• Automatic transcription
• Voice activity detection
• Turn management included

Common Pitfalls & Solutions

Sequential Recording

Issue: Audio blob is null after stopping recording

// ❌ Wrong: Process immediately
stopRecording();
processAudio(state.audioBlob); // blob is still null!

// ✅ Correct: Use effect to wait for blob
useEffect(() => {
  if (!state.isRecording && state.audioBlob) {
    processAudio(state.audioBlob);
  }
}, [state.audioBlob, state.isRecording]);

Issue: Microphone stays active after component unmounts

// ✅ Always cleanup
useEffect(() => {
  return () => {
    if (streamRef.current) {
      streamRef.current.getTracks().forEach(track => track.stop());
    }
  };
}, []);

Realtime Streaming

Issue: Message and audio recording out of sync

// ✅ Map recording index to expected message index
const recordingToMessageMap = new Map();

session.on("transport_event", (event) => {
  if (event.type === "input_audio_buffer.speech_started") {
    const recordingIndex = userRecordingCounter++;
    const expectedMessageIndex = getCurrentUserMessageCount();
    recordingToMessageMap.set(recordingIndex, expectedMessageIndex);
  }
});

Issue: Session not cleaning up properly

// ✅ Comprehensive cleanup
const stopSession = async () => {
  // Stop all recordings
  if (mediaRecorder?.state === "recording") {
    mediaRecorder.stop();
  }
  
  // Stop media streams
  mediaRecorder?.stream?.getTracks().forEach(track => track.stop());
  
  // Close session
  session?.close();
  
  // Clear refs
  sessionRef.current = null;
  mediaRecorderRef.current = null;
};

Best Practices

Security

// ✅ Never expose API keys in client code
// Use API routes to proxy requests
const response = await fetch("/api/openai/realtime-client-secret", {
  method: "POST"
});

// ❌ Never do this
const session = new RealtimeSession(agent, {
  apiKey: process.env.OPENAI_API_KEY // Exposed to client!
});

Performance

// ✅ Cache TTS audio for repeated questions
const getQuestionAudio = async (questionId: string, text: string) => {
  // Check cache first
  const cached = await db.query.questionAudioCache.findFirst({
    where: eq(schema.questionAudioCache.questionId, questionId)
  });
  
  if (cached) return cached.audioUrl; // 90% cost reduction
  
  // Generate and cache
  const audio = await generateTTS(text);
  await cacheAudio(questionId, audio);
  return audio;
};

User Experience

// ✅ Provide clear feedback at each stage
toast.info("Transcribing your response...");
toast.info("AI is thinking...");
toast.success("Ready for your next response!");

// ✅ Show recording indicator
{state.isRecording && (
  <div className="flex items-center gap-2">
    <div className="h-3 w-3 rounded-full bg-red-500 animate-pulse" />
    <span>Recording... ({state.recordingDuration}s)</span>
  </div>
)}

Error Handling

// ✅ Graceful degradation
try {
  const transcription = await transcribe();
  const response = await generateResponse(transcription);
  await speakResponse(response);
} catch (error) {
  if (error.message.includes("microphone")) {
    toast.error("Microphone access denied. Please check permissions.");
  } else if (error.message.includes("API")) {
    toast.error("Service temporarily unavailable. Please try again.");
  } else {
    toast.error("An error occurred. Please refresh the page.");
  }
}

Advanced Topics

Custom AI Instructions

const INSTRUCTIONS = `You are an ESOL speaking coach.

- Adapt to CEFR level: ${userLevel}
- Focus areas: ${focusSkills.join(", ")}
- Provide feedback on: fluency, accuracy, coherence
- Keep responses under 20 seconds
- Encourage learner after each turn`;

Audio Persistence Strategy

// Sequential: Simple - save after each turn
await saveUserRecording({
  sessionId,
  audioBlob,
  transcription,
  timestamp: new Date(),
});

// Realtime: Complex - sync with message history
// Map recording index → message index → upload when message saved

Session Analytics

interface SessionMetrics {
  duration: number;           // Total session time
  turnCount: number;          // Number of exchanges
  userSpeakingTime: number;   // Active speaking time
  avgResponseLatency: number; // Sequential only
  completionRate: number;     // % of target turns reached
}

Conclusion

Both approaches have their place in ESOL learning platforms:

Get Chan Meng’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Sequential Recording: Reliable, cost-effective, and easier to implement. Perfect for structured practice, assessments, and budget-conscious applications.

Realtime Streaming: Premium experience with natural conversation flow. Ideal for advanced learners, fluency practice, and applications where UX justifies the complexity.

Consider implementing both: use Sequential Recording for structured exercises and Realtime Streaming for conversational practice. This hybrid approach provides flexibility for different learning contexts and user needs.

Resources

Project Files

Sequential Implementation:

• src/components/conversation/realtime-conversation.tsx - Main component
• src/hooks/use-voice-recorder.ts - Recording logic
• src/app/api/openai/transcribe/route.ts - Whisper API
• src/app/api/openai/conversation/route.ts - GPT-4 API
• src/app/api/openai/tts/route.ts - Text-to-speech API

Realtime Implementation:

• src/components/speaking/ai-coach.tsx - Main component
• src/app/api/openai/realtime-client-secret/route.ts - Authentication

External Documentation

• OpenAI Realtime API Guide
• OpenAI Whisper API
• MediaRecorder API (MDN)
• WebRTC API (MDN)