<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samuel Kalu on Medium]]></title>
        <description><![CDATA[Stories by Samuel Kalu on Medium]]></description>
        <link>https://medium.com/@eskayml?source=rss-126f49c4cca6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FkCjtGD1RoxZ1h_l0m5uAA.jpeg</url>
            <title>Stories by Samuel Kalu on Medium</title>
            <link>https://medium.com/@eskayml?source=rss-126f49c4cca6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:25:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eskayml/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GSOC 25 | Communication Analysis Tool for Human-AI Interaction Driving Simulator Experiments (Final…]]></title>
            <link>https://eskayml.medium.com/gsoc-25-communication-analysis-tool-for-human-ai-interaction-driving-simulator-experiments-final-39c53d90672d?source=rss-126f49c4cca6------2</link>
            <guid isPermaLink="false">https://medium.com/p/39c53d90672d</guid>
            <category><![CDATA[speaker-diarization]]></category>
            <category><![CDATA[pyannote]]></category>
            <category><![CDATA[openai-whisper]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Samuel Kalu]]></dc:creator>
            <pubDate>Sat, 27 Sep 2025 19:47:09 GMT</pubDate>
            <atom:updated>2025-09-27T19:53:36.225Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>GSOC 25 | Communication Analysis Tool for Human-AI Interaction Driving Simulator Experiments (Final Blog)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*gUifGflE4s-UmIwm" /><figcaption>GSOC x HumanAI x TRIP LAB</figcaption></figure><h3>INTRODUCTION</h3><p>For the entire summer , I have been working on creating a tool for understanding participant communication in driving simulator experiments as a Google Summer of Code contributor with HumanAI. This blog post is the second report on my work, following my <a href="https://eskayml.medium.com/gsoc-25-communication-analysis-tool-for-human-ai-interaction-driving-simulator-experiments-mid-8130aa006bdf">first blog post.</a></p><p>You can find the complete code for the tool , results, and all experiments on GitHub.</p><p><a href="https://github.com/humanai-foundation/ISSR/tree/main/ISSR_Communication_Analysis_Tool_Samuel_Kalu"><strong>Link to Complete Project</strong></a></p><h3>PROJECT DESCRIPTION</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_O_QMa1vY7a1bzqljRK1GA.png" /><figcaption>sample driving simulator footage</figcaption></figure><p>The project focused on creating a tool that processes and analyzes high-fidelity audiovisual communication data collected from six participants in a simulated driving environment. The tool leverages advanced Natural Language Processing (NLP) and visualization techniques to provide insights into group dynamics, enhancing safety measures and informing the design of intelligent driving systems.</p><p>The tool specifically focuses on answering two core questions inside the experiments.</p><ol><li>“Who Spoke When?” (Speaker identification)</li><li>“What is being said?” (Audio Transcription).</li></ol><h3>PROJECT MOTIVATION</h3><p>Effective communication between humans and AI systems is critical for safety and collaboration in complex, high-stakes domains like autonomous driving. Studies have shown that informal and verbal exchanges between drivers, pedestrians, and AI-driven systems significantly influence situational awareness and decision-making.</p><p>Driving simulators provide a controlled environment to study these interactions, generating rich multimodal data that captures both verbal dialogues and non-verbal cues. However, analyzing this data at scale requires robust computational tools or substantial, manual effort to extract actionable insights.Existing research highlights the need for tools that bridge natural language processing (NLP) with real-time interaction analysis. For instance, enabling NLP-driven interactions with technical systems has been shown to enhance user creativity and system usability.This project addresses this gap by developing a Communication Analysis Tool specifically for driving simulator experiments, leveraging NLP and audio-processing techniques to identify speaker roles, emotional cues, and collaborative patterns. Such insights are vital for designing AI systems that adapt to human behavior, aligning with studies that emphasize integrating human communication dynamics into autonomous systems.</p><h3>TOOL ARCHITECTURE</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/0*wy6z2AzRVs__mNXR" /><figcaption>Overall Architecture</figcaption></figure><h3>BUILDING AN INTERFACE</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*MjAwLvQUCRdiwkft" /><figcaption>communication analysis tool ui</figcaption></figure><p>The user interface (UI) of this tool is designed to be simple and easy to use. It’s built using the Gradio Python library, which makes it quick to create clean, interactive UIs directly in Python.</p><p>On the deployment side, everything runs completely locally no internet connection is needed and no third-party APIs are used. This was a core requirement for the project, so I made sure all models used are open-source and included directly within the project folder.</p><p>To start, I downloaded the Whisper Medium (English) model for transcription and the pyannote.audio model for speaker diarization (identifying who is speaking and when). These two models handle both the main tasks of the tool:</p><p>These models may be replaced by me later on as more efficient and accurate options become available.</p><p>The tool also supports secondary tasks:</p><ul><li>Named Entity Recognition (NER) using spaCy and its en_core_web_sm model to extract names, places, and other common entities from the text.</li><li>Sentiment Analysis using VADER, a very lightweight tool that works well for quick emotion detection.</li></ul><p>The tool runs using PyTorch in the background, and it works on both CPU and GPU, depending on what’s available on the machine.</p><p>For this tool to be valuable to TRIP-LAB, it needs to work with very low error rates. Now the most error-prone part of the system is transcription( converting audio into accurate text ) which leads to the key question: <strong>“What model is best for transcription?”</strong></p><h3>TESTS TO INFORM TRANSCRIPTION OPTIMIZATIONS</h3><p>To figure out the best transcription model for our data, I started by testing different versions of the Whisper model. Since my mentors confirmed that English would be the only language used across all experiments, I was able to narrow down the list of models to the English-only Whisper variants, which are more optimized for this task.</p><p>The testing process began by manually transcribing a sample video. I watched the entire video carefully, listened closely to all conversations, and created an accurate transcript that would serve as our “ground truth” , the reference point against which all model outputs would be compared.</p><p>As part of good scientific practice, I and my mentors created multiple versions of the manual transcription. This not only improves the reliability of our results but also demonstrates the value of this tool: researchers can test against high-quality references and understand exactly how each model performs.</p><p>However, it’s important to note that the quality of the audio extracted from the source video can significantly impact transcription accuracy. In some cases, poor microphone placement, background noise, or speakers talking over each other can make it harder for any model or even a human to produce a perfect transcript. This naturally means that some speakers or roles in the footage will have better transcription results than others.</p><p>Understanding these limitations is key to improving the tool. By identifying where and why certain segments perform worse, we can fine-tune the model choice and processing pipeline to produce more accurate, balanced results across all speakers, not just the loudest or clearest voices.</p><p>This kind of deep testing and careful setup is what allows us to optimize intelligently, rather than guess, ensuring that the final transcription model is truly the best fit for our use case.</p><p>To kick things off, we run 6 different whisper models (<a href="http://tiny.en/">tiny.en</a>, <a href="http://base.en/">base.en</a>, <a href="http://small.en/">small.en</a> , <a href="http://medium.en/">medium.en</a>, large &amp; turbo) against our human transcriptions.</p><p>The metric for evaluation was by using the Word Error Rate (WER)</p><blockquote><strong><em>WER = (S + D + I )/ N</em></strong></blockquote><p>Where:</p><ul><li><em>S</em> = Number of substitutions</li><li><em>D</em> = Number of deletions</li><li><em>I</em> = Number of insertions</li><li><em>N</em> = Number of words in the reference (ground truth)</li></ul><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*kfS3QCWtVwqg3DKT" /></figure><p>From the first test, it shows that the <a href="http://medium.en/">medium.en</a> was the best performing model for our task, even outperforming larger weight whisper models like the large and the turbo, this is due to being an english only model compared to these two, and our experiments are only being done in english.</p><p>Then we compare the performance of all the models and their required VRAM.</p><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*y2Wad-Jj7-YzTd8w" /></figure><p>We see that <a href="http://medium.en/">medium.en</a> falls within a very desirable range having the smallest WER and also a quite decent usage of RAM.</p><p>Now moving further, is it obvious that we would need a lightweight and perhaps faster version of the <a href="http://medium.en/">medium.en</a> model , so we proceed to test faster-whisper a library that provides quantized versions of the whisper model.</p><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*Hztgn5qKX9tTWpq0" /></figure><p>Now , from the result above we see that the faster- whisper medium (english only ) model completely blows the previous <a href="http://medium.en/">medium.en</a> out of the water in terms of sheer speed.</p><p>Now that we see substantial gains in speed, we must also test the tradeoff in performance between the two models.</p><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/0*dwzIKmAPrNl69sQ1" /></figure><p>Now shockingly, the results show that faster whisper slightly outperforms the default whisper with a lower WER , now this comes down to a couple of reasons.</p><ol><li>Faster Whisper has a lot of better inbuilt post processing techniques which include proper punctuation, word alignment, formatting.</li><li>Quantized models can sometimes generalize better to certain types of audio.</li></ol><p>From this particular result above, there is no reason to not go ahead with using faster-whisper as the de facto transcription model.</p><h3>TESTS TO INFORM DIARIZATION OPTIMIZATIONS</h3><p>To begin, we defined a metric to optimize diarization, it is the Diarization Error Rate (DER), the most widely used benchmark.<br>With guidance from my mentors at TRIP-LAB, we manually annotated the video, assigning speaker identities and accurate transcriptions.<br>I selected three popular models for experimentation:</p><ul><li>Pyannote.audio (because of it’s ability to recognize overlapping speech) <a href="https://github.com/pyannote/pyannote-audio">[1]</a></li><li>NeMO ( for it’s pure state of the art performance on most tasks) <a href="https://github.com/NVIDIA-NeMo/NeMo">[2]</a></li><li>SpeechBrain (because of its flexibility, modular PyTorch-based architecture) <a href="https://github.com/speechbrain/speechbrain">[3]</a></li></ul><p>The first benchmarking was done with the default configurations of each of the models , and pyannote was discovered to have the lowest error rates, The error rate calculation is done via a hybrid hungarian + cosine similarity approach in order to properly align the speakers we manually annotated to their transcriptions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*PXfgzbr7g7yiud6A4UhzpQ.png" /><figcaption>Hybrid DER Comparison</figcaption></figure><p>By adjusting important parameters like min_cluster_size, clustering_threshold and the min_duration_off, we can further improve the performance of our best performing model (which is pyannote.audio).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*xahfjIEn2tDdJGFWCStvrw.png" /><figcaption>after tuning VAD hyperparameters</figcaption></figure><p>This brings our best performing model which is pyannote.audio to have a DER as low as <strong>0.1522.</strong></p><p>A shocking discovery was that when finetuning NeMO , it was unable to produce enough speaker segments on most of the runs , making it very undesirable despite differentiating between speakers as comparably to the way pyannote.audio does. <a href="https://lajavaness.medium.com/comparing-state-of-the-art-speaker-diarization-frameworks-pyannote-vs-nemo-31a191c6300">[4]</a></p><h3><strong>OUTPUTS FROM THE TOOL</strong></h3><p>The tool provides a downloadable table as an output, which contains different columns</p><ul><li><strong>speaker_id</strong>: A unique identifier for each speaker, with experiments typically involving exactly seven (7) distinct participants ( usually 6 participants and one narrator).</li><li><strong>start_time</strong>: The precise beginning temporal marker for each spoken segment.</li><li><strong>end_time</strong>: The precise end temporal marker for each spoken segment.</li><li><strong>transcribed_content</strong>: The English transcription of the segment, generated by the efficient <strong>Faster Whisper</strong> model.</li><li><strong>named_entities:</strong> Contains a list of identified entities (names of people, places or things.)</li><li><strong>sentiment_score</strong>: A floating-point value between -1 and +1, quantifying the emotional tone (negative to positive) via the <strong>VADER</strong> lexicon.</li><li><strong>tone_intensity</strong>: A floating-point value that represents the overall acoustic intensity or volume of the speech segment.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*KANOULzEDdxmBjEguCdQzw.png" /><figcaption>Output Table From the Tool</figcaption></figure><h3>VISUALIZATIONS FROM THE TOOL</h3><p>The tool provides five main visualizations. Each one highlights a different aspect of the conversation and helps in understanding speaker behavior more clearly:</p><ol><li><strong>Word Count per Speaker<br></strong> This plot shows how many words each speaker used. It makes it easy to see who spoke the most and who spoke the least throughout the video. This is especially useful when analyzing dominance in a discussion.</li><li><strong>Speaker Timeline Map<br></strong> This is a horizontal timeline of the full video. Small bars mark the points where each speaker was active, making it simple to follow when and how often every speaker joined the discussion. This visualization is one of the most interesting because it clearly shows speaker involvement patterns across the entire conversation.</li><li><strong>Sentiment Distribution per Speaker Over Time<br></strong> This chart uses the VADER sentiment analysis tool to show the range of positive, neutral, and negative expressions for each speaker across the speaker timeline.</li><li><strong>Tone Intensity per Speaker<br></strong> Using features extracted with Librosa, this plot measures the intensity of each speaker’s tone. It highlights differences in how strongly or softly people express themselves, which makes it an interesting view for studying speaker emphasis and delivery.</li><li><strong>Speaker Interaction Map<br></strong> This visualization shows speakers as nodes and their interactions as edges (lines). It is one of the most important plots because it gives a clear picture of how connected the conversation was and reveals which speakers interacted the most with others.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*VfZMxKehJ8HQdViATfebTQ.png" /><figcaption>Output Visualization from the Tool</figcaption></figure><h3><strong>LIMITATIONS &amp; SHORTCOMINGS</strong></h3><p>Now since I was provided with just a single video footage, most of the optimization work has focused on that single recording. This means the VAD parameters and the faster-whisper parameters were tuned mainly to match the characteristics of that video. As a result, there is an innate risk of overfitting.</p><h3><strong>POTENTIAL FUTURE WORK</strong></h3><p>A potential future work might be adding a lightweight LLM as a final stage to help auto correct misspelled words in the transcriptions through a rich context.</p><p>Another one to consider (although beyond the scope of the tool) , adding some vision based components would be really useful for such a simulator based communication tool like this , since the motion of the vehicles inside the simulator play a huge role in determining what is being said and who spoke when, most of the precedence lies within vision based analysis . So adding some form of object detection/tracking into the tool would make this even more valuable.</p><h3>ACKNOWLEDGEMENTS❤</h3><p>Huge Shoutout to all my incredible mentors: <em>Piyush Pawar, Joshua White , Andrea Underhill , Despina Stavrinos , Benjamin McManus , Amanda Hudson.</em></p><h3>REFERENCES</h3><ol><li><a href="https://github.com/pyannote/pyannote-audio">https://github.com/pyannote/pyannote-audio</a></li><li><a href="https://github.com/NVIDIA-NeMo/NeMo">https://github.com/NVIDIA-NeMo/NeMo</a></li><li><a href="https://github.com/speechbrain/speechbrain">https://github.com/speechbrain/speechbrain</a></li><li><a href="https://lajavaness.medium.com/comparing-state-of-the-art-speaker-diarization-frameworks-pyannote-vs-nemo-31a191c6300">https://lajavaness.medium.com/comparing-state-of-the-art-speaker-diarization-frameworks-pyannote-vs-nemo-31a191c6300</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39c53d90672d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GSOC 25 | Communication Analysis Tool for Human-AI Interaction Driving Simulator Experiments (Mid…]]></title>
            <link>https://eskayml.medium.com/gsoc-25-communication-analysis-tool-for-human-ai-interaction-driving-simulator-experiments-mid-8130aa006bdf?source=rss-126f49c4cca6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8130aa006bdf</guid>
            <dc:creator><![CDATA[Samuel Kalu]]></dc:creator>
            <pubDate>Sat, 26 Jul 2025 15:58:36 GMT</pubDate>
            <atom:updated>2025-07-29T13:58:08.363Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>GSOC 25 | Communication Analysis Tool for Human-AI Interaction Driving Simulator Experiments (Mid Term Assessment)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uEWFfFl2i8Eal0ds" /><figcaption><em>GSOC x HumanAI x TRIP LAB</em></figcaption></figure><h4>INTRODUCTION</h4><p>Greetings, everyone! I am very excited to share the progress of my project, “Communication Analysis Tool for Human-AI Interaction Driving Simulator Experiment” for Google Summer of Code 2025. Over the past two months, I have been working diligently with HumanAI &amp; TRIP-LAB on this project.</p><p>You can find the complete code for the tool , results, and all experiments on GitHub.</p><p><a href="https://github.com/humanai-foundation/ISSR/tree/main/ISSR_Communication_Analysis_Tool_Samuel_Kalu"><strong>Link to Project</strong></a></p><h4>PROJECT DESCRIPTION</h4><p>The project focused on creating a tool that processes and analyzes high-fidelity audiovisual communication data collected from six participants in a simulated driving environment. The tool leverages advanced Natural Language Processing (NLP) and visualization techniques to provide insights into group dynamics, enhancing safety measures and informing the design of intelligent driving systems.</p><p>The tool specifically focuses on answering two core questions inside the experiments.</p><ol><li>“Who Spoke When?” (Speaker identification)</li><li>“What is being said?” (Audio Transcription).</li></ol><h4>PROJECT MOTIVATION</h4><p>Effective communication between humans and AI systems is critical for safety and collaboration in complex, high-stakes domains like autonomous driving. Studies have shown that informal and verbal exchanges between drivers, pedestrians, and AI-driven systems significantly influence situational awareness and decision-making.</p><p>Driving simulators provide a controlled environment to study these interactions, generating rich multimodal data that captures both verbal dialogues and non-verbal cues. However, analyzing this data at scale requires robust computational tools or substantial, manual effort to extract actionable insights.Existing research highlights the need for tools that bridge natural language processing (NLP) with real-time interaction analysis. For instance, enabling NLP-driven interactions with technical systems has been shown to enhance user creativity and system usability.This project addresses this gap by developing a Communication Analysis Tool specifically for driving simulator experiments, leveraging NLP and audio-processing techniques to identify speaker roles, emotional cues, and collaborative patterns. Such insights are vital for designing AI systems that adapt to human behavior, aligning with studies that emphasize integrating human communication dynamics into autonomous systems.</p><h4>TOOL ARCHITECTURE</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/0*Nv9aI99OzG8jaFS5" /><figcaption>Overall Architecture</figcaption></figure><h4>BUILDING AN INTERFACE</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cqpdhx6RttBrXmSX" /><figcaption>communication analysis tool ui</figcaption></figure><p>The user interface (UI) of this tool is designed to be simple and easy to use. It’s built using the Gradio Python library, which makes it quick to create clean, interactive UIs directly in Python.</p><p>On the deployment side, everything runs completely locally no internet connection is needed and no third-party APIs are used. This was a core requirement for the project, so I made sure all models used are open-source and included directly within the project folder.</p><p>To start, I downloaded the Whisper Medium (English) model for transcription and the pyannote.audio model for speaker diarization (identifying who is speaking and when). These two models handle both the main tasks of the tool:</p><p>These models may be replaced by me later on as more efficient and accurate options become available.</p><p>The tool also supports secondary tasks:</p><ul><li>Named Entity Recognition (NER) using spaCy and its en_core_web_sm model to extract names, places, and other common entities from the text.</li><li>Sentiment Analysis using VADER, a very lightweight tool that works well for quick emotion detection.</li></ul><p>In addition, I added a feature for analyzing tonal intensity , it helps detect when a speaker is talking more loudly or emotionally (for example, shouting or speaking with a high pitch). This can be useful in understanding emotional tone during conversations.</p><p>All of these results are displayed in a simple, downloadable table in the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*eLU_TPM1fon8wcFCLtbB-A.gif" /></figure><p>The results displayed from the table are used to generate some rich visuals</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*pGPrSUXcN1_kewBg" /></figure><p>The main goal of building this interface is to make the tool usable by non-technical researchers. It provides a way to test and use the system easily, while keeping all the data secure and local to their own environment.</p><p>The tool runs using PyTorch in the background, and it works on both CPU and GPU, depending on what’s available on the machine.</p><p>For this tool to be valuable to TRIP-LAB, it needs to work with very low error rates. Now the most error-prone part of the system is transcription( converting audio into accurate text ) which leads to the key question: <strong>“What model is best for transcription?”</strong></p><h4>TESTS TO INFORM OPTIMIZATIONS</h4><p>To figure out the best transcription model for our data, I started by testing different versions of the Whisper model. Since my mentors confirmed that English would be the only language used across all experiments, I was able to narrow down the list of models to the English-only Whisper variants, which are more optimized for this task.</p><p>The testing process began by manually transcribing a sample video. I watched the entire video carefully, listened closely to all conversations, and created an accurate transcript that would serve as our “ground truth” , the reference point against which all model outputs would be compared.</p><p>As part of good scientific practice, I and my mentors created multiple versions of the manual transcription. This not only improves the reliability of our results but also demonstrates the value of this tool: researchers can test against high-quality references and understand exactly how each model performs.</p><p>However, it’s important to note that the quality of the audio extracted from the source video can significantly impact transcription accuracy. In some cases, poor microphone placement, background noise, or speakers talking over each other can make it harder for any model or even a human to produce a perfect transcript. This naturally means that some speakers or roles in the footage will have better transcription results than others.</p><p>Understanding these limitations is key to improving the tool. By identifying where and why certain segments perform worse, we can fine-tune the model choice and processing pipeline to produce more accurate, balanced results across all speakers, not just the loudest or clearest voices.</p><p>This kind of deep testing and careful setup is what allows us to optimize intelligently, rather than guess, ensuring that the final transcription model is truly the best fit for our use case.</p><p>To kick things off, we run 6 different whisper models (<a href="http://tiny.en/">tiny.en</a>, <a href="http://base.en/">base.en</a>, <a href="http://small.en/">small.en</a> , <a href="http://medium.en/">medium.en</a>, large &amp; turbo) against our human transcriptions.</p><p>The metric for evaluation was by using the Word Error Rate (WER)</p><blockquote><strong>WER = (S + D + I )/ N</strong></blockquote><p>Where:</p><ul><li><em>S</em> = Number of substitutions</li><li><em>D</em> = Number of deletions</li><li><em>I</em> = Number of insertions</li><li><em>N</em> = Number of words in the reference (ground truth)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*fJglqMX2NkqCp3V6" /></figure><p>From the first test, it shows that the <a href="http://medium.en/">medium.en</a> was the best performing model for our task, even outperforming larger weight whisper models like the large and the turbo, this is due to being an english only model compared to these two, and our experiments are only being done in english.</p><p>Then we compare the performance of all the models and their required VRAM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*Hf4PBIgpfRZDkAml" /></figure><p>We see that <a href="http://medium.en/">medium.en</a> falls within a very desirable range having the smallest WER and also a quite decent usage of RAM.</p><p>Now moving further, is it obvious that we would need a lightweight and perhaps faster version of the <a href="http://medium.en/">medium.en</a> model , so we proceed to test faster-whisper a library that provides quantized versions of the whisper model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*xAAr_CI5HeMMyWW0" /></figure><p>Now , from the result above we see that the faster- whisper medium (english only ) model completely blows the previous <a href="http://medium.en/">medium.en</a> out of the water in terms of sheer speed.</p><p>Now that we see substantial gains in speed, we must also test the tradeoff in performance between the two models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/0*v4KtuPamNsg8mqin" /></figure><p>Now shockingly, the results show that faster whisper slightly outperforms the default whisper with a lower WER , now this comes down to a couple of reasons.</p><ol><li>Faster Whisper has a lot of better inbuilt post processing techniques which include proper punctuation, word alignment, formatting.</li><li>Quantized models can sometimes generalize better to certain types of audio.</li></ol><p>From this particular result above, there is no reason to not go ahead with using faster-whisper as the de facto transcription model.</p><h4>FUTURE PLANS (POST MIDTERMS)</h4><p>Future enhancements after my midterms would focus heavily on speaker identification(diarization) , with the introduction of a new metric called the Diarization Error Rate (DER), I would be looking forward to benchmark against different diarization models like NeMO , WhisperX,etc. , and then compare against a human rated manual diarization approach the same way I did on the transcription.</p><h4>ACKNOWLEDGEMENTS</h4><p>Huge Shoutout to all my incredible mentors: <em>Piyush Pawar, Joshua White , Andrea Underhill , Despina Stavrinos , Benjamin McManus , Amanda Hudson.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8130aa006bdf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>