<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Edwin Zhang on Medium]]></title>
        <description><![CDATA[Stories by Edwin Zhang on Medium]]></description>
        <link>https://medium.com/@edwinzhng?source=rss-9438328b2775------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*eEwSuXvPxBSdHuz53CmZsg.jpeg</url>
            <title>Stories by Edwin Zhang on Medium</title>
            <link>https://medium.com/@edwinzhng?source=rss-9438328b2775------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:16:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@edwinzhng/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hey Media: A Voice-Controlled Chrome Extension for Media Playback]]></title>
            <link>https://edwinzhng.medium.com/hey-media-a-voice-controlled-chrome-extension-for-media-playback-a089cb2dd971?source=rss-9438328b2775------2</link>
            <guid isPermaLink="false">https://medium.com/p/a089cb2dd971</guid>
            <dc:creator><![CDATA[Edwin Zhang]]></dc:creator>
            <pubDate>Wed, 14 Apr 2021 18:02:49 GMT</pubDate>
            <atom:updated>2021-04-15T02:32:08.949Z</atom:updated>
            <content:encoded><![CDATA[<p>Created by: Akshay Patel, Edwin Zhang</p><h3>Introduction</h3><p>Digital voice assistants, such as Amazon Alexa or Google Assistant, have recently exploded in popularity, and have dramatically changed the way humans interact with technology and software. Voice assistants enable devices to perform actions based purely on voice commands, which removes the need for physical interaction.</p><p>Given the current situation with COVID-19 lockdowns and mobility restrictions, user interaction with voice assistants at home is more popular than ever, and individuals are increasingly watching or listening to media for entertainment at home. For our project, we prototyped a voice assistant browser extension that can help control media playback on popular content streaming platforms such as Netflix and YouTube.</p><p>Typically, building a voice assistant draws on knowledge from a variety of areas including speech recognition, speech synthesis, and natural language processing (NLP). In this project, we focus primarily on speech recognition, where we train a neural network model to recognize the wake word phrase “hey media”, as a trigger to activate our voice assistant. After activation, the assistant continues to listen for a specific command, such as “play”, “pause”, “full screen”, or “volume up/down”, which represent user actions for controlling media playback. To recognize these commands, we take advantage of the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API">Web Speech API</a> built into the latest version of Chrome. We will be using only the speech recognition component to enable voice command recognition after the voice assistant has been activated.</p><p>In this report, we describe the steps we took to generate a custom dataset for detecting “hey media” as wake words, process raw audio streams, evaluate and train our model, and then build the end-to-end system.</p><h3>Dataset Generation</h3><h4>Raw Datasets</h4><p>For our data source, we use version 6.1 of the <a href="https://commonvoice.mozilla.org/en">Mozilla Common Voice (MCV) dataset</a>, which is an open-source dataset of voices that can be freely used to train speech-related applications. It contains 2181 hours of raw English voice audio (represented as WAV files) from over 66,000 speakers, along with their corresponding text transcriptions. All speech is sampled at a frequency 16000 Hz. We choose to use the wake words “hey media”, as a trigger for activating the voice assistant, after which the user can interact with media players via voice assistant commands. These wake words are simple to remember for users, and distinct enough to not be confused by other words by our detection models.</p><p>To prepare a dataset for these wake words, we extracted all audio clips containing the words “hey” or “media’’ in their transcriptions from the Mozilla Common Voice corpus. Next, we preprocess the audio before using it to train a convolutional neural network that performs wake word detection. To perform the dataset generation, we take advantage of scripts provided by <a href="https://github.com/castorini/howl">Howl</a>, a Python wake word detection toolkit. Specifically, we generated the dataset using the provided generate_dataset.sh script.</p><pre># Clone Howl and install dependencies<br># Assumes PyAudio and PyTorch are installed<br>git clone <a href="https://github.com/castorini/howl.git">https://github.com/castorini/howl.git</a><br>cd howl<br>pip install -r requirements.txt<br>pip install -r requirements_training.txt<br>./download_mfa.sh</pre><pre># Generate the dataset<br>./generate_dataset.sh &lt;COMMON_VOICE_DATASET_PATH&gt; hey_media [0,1]</pre><p>By running the script, we extracted 631 samples of ”media”, but only 30 samples of ”hey”, which is not enough to train a robust model. Fortunately, the authors of Howl provide a dataset containing 632 custom recorded samples with the word ”hey”, inside of the <a href="https://github.com/castorini/howl#hey-firefox">“Hey Firefox” corpus</a>, which we combine with our extracted data, giving a total of 662 positive samples. We create dataset splits for training, validation, and testing at 80%, 10%, and 10%, respectively, stratified by speaker IDs from MCV.</p><h4>Audio Processing</h4><p><strong>Forced Alignment</strong></p><p>The first step in our audio processing is forced alignment, where we take the transcription for each audio file and generate timestamps for each word, based on when they appear in the audio file. To do this, we use the <a href="https://montreal-forced-aligner.readthedocs.io/en/latest/index.html">Montreal Forced Aligner (MFA)</a> tool, which is an open source command-line executable. Alignment is performed by using a pronunciation dictionary and acoustic model to look up words within the audio, and then generating timestamps for words.</p><p>Acoustic models convert a phoneme’s acoustic information into some statistical representation. In this context, a phoneme is any speech sound that is present in a language. Under the hood, these acoustic models are trained on acoustic features from labeled data, which come from transcribed speech corpora. By default, MFA uses a <a href="https://montreal-forced-aligner.readthedocs.io/en/latest/pretrained_models.html#pretrained-acoustic">pretrained English acoustic model trained</a> on the <a href="https://www.openslr.org/12">LibriSpeech corpus</a> to look up phones for each word.</p><p>The MFA library provides several executables that can be used out-of-the-box to align audio transcriptions. The package can be easily downloaded from their <a href="https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner/releases">latest Github releases</a>. In our case, we used the version 1.0.1 Linux release. Standalone usage of this tool is shown below (although this was automatically performed by the generate_dataset.sh script provided by Howl):</p><pre># Download the package and unzip<br>wget <a href="https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner/releases/download/v1.0.1/montreal-forced-aligner_linux.tar.gz">https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner/releases/download/v1.0.1/montreal-forced-aligner_linux.tar.gz</a><br>tar -xf montreal-forced-aligner_linux.tar.gz</pre><pre># Download English dictionary for Librispeech<br>wget https://www.openslr.org/resources/11/librispeech-lexicon.txt</pre><pre># Navigate to directory and perform alignment<br>cd montreal-forced-aligner<br>./bin/mfa_align --verbose --clean &lt;AUDIO_DATASET_PATH&gt; ../librispeech-lexicon.txt pretrained_models/english.zip &lt;OUTPUT_FOLDER&gt;</pre><p>The timestamped alignment produced by MFA allows us to produce clipped audio samples containing only the single words “hey” or “media”. Using this data, we can train our model to predict the correct label for each sample.</p><p><em>Stitched Samples</em></p><p>Furthermore, we also use the aligned timestamps to create stitched samples of the entire target phrase “hey media”, which are used for evaluating our model. In total, we generated 10000 samples of our wake word phrase by combining clips from different speakers. We use these samples to see if our model is capable of detecting the entire phrase reliably. However, the stitching process is not perfect, and can lead to partially cut or missing words in the audio files.</p><p><strong>Feature Extraction from Raw Audio</strong></p><p>Before we can use our audio data, we need to also transform the time-domain audio samples into Mel-Frequency Cepstral Coefficients (MFCCs), which are commonly used as input features for speech recognition systems and machine learning models.</p><p>For training, we perform this transformation using existing implementations, although we also briefly describe the algorithm below (<a href="http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/">source</a>). Specifically, we used the <a href="https://librosa.org/">Librosa Python library</a> during training, and used <a href="https://meyda.js.org/">Meyda.js</a> for inference in the browser.</p><p><em>Step 1. Windowing</em></p><p>The first step that is performed is windowing, where audio waveforms are sliced into sliding frames of a specific length and each subsequent frame begins a certain hop-length away from the last. In our case, we choose a window size of 30ms, and a hop-length of 12.5ms. We also used a Hann window to create each frame, which provides an amplitude drop off near the two edges of the frame. This reduces high-frequency noise that may arise from a sudden drop in amplitude at the edges of the frames. The Hann function used to compute a window of length N, w, is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*iKNM0WjQTDTgSGlL" /><figcaption>The Hann window function</figcaption></figure><p><em>Step 2. Fast Fourier Transform (FFT)</em></p><p>Next, the FFT is applied to represent each frame in the frequency domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*NIwajCr13ScMJL1g" /><figcaption>Equation for computing the FFT</figcaption></figure><p>We also take the square of this output to reflect the power of speech at each frequency, which we call the power spectrum.</p><p><em>Step 3. Mel scale conversion and logarithm</em></p><p>Now, we want to convert the frequency information to the Mel scale. The Mel scale is a logarithmic transformation of signal frequency, such that equal distance on the scale translates to the perception of equal distance to the human ear. For example, it is easy for humans to tell the difference between lower frequencies such as 200 Hz and 300 Hz, but for higher frequencies such as 1000 Hz and 1100 Hz, it is harder to tell, even though the difference is also 100 Hz. The Mel scale is therefore used to mimic our own perception of sound and is computed via the following equation, where f is the frequency:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*zaWfVBixjpmi2qy0" /><figcaption>Equation for converting Hertz to the Mel scale</figcaption></figure><p>To perform this conversion, we compute a Mel-spaced filterbank, which is a set of around 20–40 equally spaced triangular filters, that are applied to the power spectrum from step 2. The number of filters we use dictates the number of coefficients that are produced in the MFCCs. In our case, we use 40 filters to create 40 features. Lastly, we take the logarithm of the output of this filterbank.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GXPLMxcFbaDSbLXd" /><figcaption>A comparison of the spectrograms produced by our stitched samples after applying FFT (left) and after converting to the Mel scale (right). Notice that the data seems to be more spread out for the Mel spectrogram.</figcaption></figure><p><em>Step 4. Discrete Cosine Transform (DCT)</em></p><p>The last step in producing MFCCs is to apply the DCT to our outputs from the previous step. This is computed using the equation below, where N is the number of data points in the input sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/0*5GojV_yHQlyP-TMc" /><figcaption>The DCT equation</figcaption></figure><h3>Wake Word Detection</h3><h4>Model Architecture</h4><p>We use a deep residual network, specifically the Res8 model architecture described by <a href="https://arxiv.org/abs/1710.10361">Tang et al</a>., as our model for wake word detection. These types of convolutional neural network (CNN) models were first applied to image recognition tasks (<a href="https://arxiv.org/pdf/1512.03385.pdf">He et al.</a>), resulting in a significant jump in state-of-the-art performance, and have more recently been applied to the keyword spotting task. The model takes in our computed MFCCs as input, and outputs a label that indicates whether or not “hey”, “media”, or an unknown token was detected.</p><p>The diagram of the Res8 architecture below shows the use of convolution layers and pooling layers, which <a href="https://www.google.com/url?q=https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/CNN_ASLPTrans2-14.pdf&amp;sa=D&amp;source=editors&amp;ust=1618451753567000&amp;usg=AOvVaw0gBNWDhQ6P9_aMA-AaVthQ">can be beneficial for the purposes of speech recognition</a> over vanilla deep neural networks without convolution layers. The intuition for this is that by converting the audio input into MFCCs, we essentially represent audio as 2D images. Convolutional neural networks are able to extract features effectively from images, and are one of the best ways to classify images into different classes. In our case, we use the image representation of our audio input in the form of MFCCs so that the Res8 model can efficiently recognize when the user has said “hey media”.</p><p>Deep residual networks are a form of convolutional neural network (CNN) that add skip connections between convolutional layers to improve the training process. In practice, this connection allows for deeper models with more layers, because it essentially acts as the identity function, where input from many layers ago is passed in directly. This makes it easier for the network to skip through layers that may be less relevant in training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/0*a2lnf1O6ZW6lpvzS" /><figcaption>Architecture of the Res8 model we use for wake-word detection. Diagram is taken directly from <a href="https://arxiv.org/abs/1710.10361">Tang et al</a>.</figcaption></figure><h4>Model Training</h4><p>For training, we also take advantage of code provided by <a href="https://github.com/castorini/howl/blob/master/training/run/train.py">Howl </a>to train a new Res8 model using our processed dataset. We trained the model for 300 epochs across the entire dataset using PyTorch. Our final model weights can be found inside our repository at <a href="https://github.com/paksha/audio-project">https://github.com/paksha/audio-project</a>, under the model/ folder.</p><p><em>Data Augmentation</em><br>In addition to using the raw audio clips, the training process performs data augmentation by injecting background noise. In particular, we use the <a href="https://github.com/microsoft/MS-SNSD">Microsoft Scalable Noisy Speech Dataset (MS-SNSD)</a>, which provides a large variety of environmental noise files. These noise files are mixed with our original speech audio to provide background noise. This makes our model more robust to wake word detection even when there is significant noise in the audio stream, such as background audio from media playback.</p><p>Below are the commands we used to set environment variables and run the training script:</p><pre># Navigate inside the Howl folder<br>cd howl</pre><pre># Set environment variables<br>source envs/res8.env<br>export NOISE_DATASET_PATH=&lt;MS_SNSD_NOISE_DATASET_PATH&gt;<br>export VOCAB=&#39;[&quot;hey&quot;,&quot;media&quot;]&#39;<br>export USE_NOISE_DATASET=True<br>export BATCH_SIZE=16<br>export INFERENCE_THRESHOLD=0<br>export NUM_EPOCHS=300<br>export NUM_MELS=40<br>export INFERENCE_SEQUENCE=[0,1]<br>export MAX_WINDOW_SIZE_SECONDS=0.5</pre><pre># Run training script<br>python -m training.run.train -i &lt;CUSTOM_DATASET_PATH&gt; --model res8 --workspace workspaces/hey_media</pre><h4>Evaluation Results</h4><p>After training was completed, we evaluated the dataset on our test set, which contained 1000 of our stitched samples for “hey media”. However, as mentioned previously, the sample stitching is not perfect. This means that the labels corresponding to the positive stitched samples are not 100% correct, possibly due to improper timestamps leading to poor audio clippings, or even just low quality recordings. Thus, the measured model performance, as evaluated through correct recognition of the positive stitched samples, might be lower than the model’s true performance. Additionally, we evaluate 1000 negative samples that do not contain the phrase, to measure the rate at which the model produces false positives.</p><p>Our results are as follows:</p><ul><li><em>Positive Samples: </em>753 true positives, 247 false negatives, 75.3% accuracy</li><li><em>Negative Samples:</em> 17 false positives, 983 true negatives, 98.3% accuracy</li></ul><p>As we can see from the results above, our model is great at detecting that sound samples are <em>not</em> “hey media”, meaning that it is unlikely to accidentally activate the voice assistant when the user did not intend to do so. However, our wake word detection model is only around 75.3% effective at detecting that audio clips contain the phrase “hey media”. This is around 9% lower than results reported in the <a href="https://www.aclweb.org/anthology/2020.nlposs-1.9.pdf">original paper for Howl</a>, where they reported a false negative rate of only 16%, compared to our rate of 24.7%.</p><p>As mentioned above however, some of these stitched positive samples may not be the clearest or properly clipped, and so the model performance might be better in more practical scenarios. To test this out, we decided to run the model locally using the Howl demo inference client and see if it would recognize our voices. For this, we can simply download the pretrained model and run the following Howl command:</p><pre># Clone our repository containing the model<br>git clone <a href="https://github.com/paksha/audio-project.git">https://github.com/paksha/audio-project.git</a></pre><pre># Navigate to the howl repository folder<br>cd howl</pre><pre># Run the demo using Howl (assuming Howl dependencies are installed already including PyAudio and PyTorch, see the Howl repo for detailed instructions)<br>python -m training.run.demo --model res8 --workspace ../audio-project/model</pre><p>In practice, the wake word model recognition seems to be quite good, despite our low evaluation metrics. It was able to generalize to both of our voices, which were not included in the dataset, and consistently detect whenever we said the phrase “hey media”.</p><h3>Building the Chrome Extension</h3><h4>Deployment</h4><p>Our Chrome Extension is quite simple and consists of a single popup HTML page that runs our wake word detection model and listens for media playback commands. We built the extension using the <a href="https://reactjs.org/">React.js</a> UI library in order to take advantage of the large ecosystem of pre-existing packages.</p><p><strong>Wake Word Detection Model</strong></p><p>To deploy our trained model in the browser, we export the model weights as a JavaScript file, which can then be loaded into TensorFlow.js. We then use the model for inference by taking the user’s microphone audio stream and transforming it into MFCCs as input. As mentioned previously, this transformation is done using the Meyda.js JavaScript library.</p><p>For each audio frame, the model will output a label indicating if any of the wake words have been detected. If the words “hey” and “media” are detected together in sequence, then a callback function is activated to listen for a media playback command.</p><p><strong>Speech Command Recognition</strong></p><p>To recognize arbitrary commands such as “play” and “pause”, we chose to rely on the Web Speech API that is built into the latest versions of Chrome. This allows us to easily transcribe speech input and detect a variety of different commands.</p><p>The Web Speech API is a specification that allows browsers to supply a speech recognition engine (either local or cloud-based) that can be used directly from the browser. In the case of Chrome, all audio is automatically sent to Google Cloud servers to perform the transcription via the <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiTnMq6pv7vAhWT9Z4KHTkhBB0QFjAAegQIAhAE&amp;url=https%3A%2F%2Fcloud.google.com%2Fspeech-to-text&amp;usg=AOvVaw0IY7ao8eCtVZbPLFfQWVdR">Google Speech to Text API</a> (<a href="https://www.twilio.com/blog/speech-recognition-browser-web-speech-api#:~:text=Chrome%20currently%20takes%20the%20audio,and%20some%20Chromium%20based%20browsers.">source</a>). Thus, developers can simply call the Web Speech API in the browser to transcribe speech using JavaScript. Our implementation uses the <a href="https://www.npmjs.com/package/react-speech-recognition">react-speech-recognition</a> package as a wrapper around the original API.</p><pre>import SpeechRecognition, { useSpeechRecognition } from &#39;react-speech-recognition&#39;</pre><pre>const { transcript } = useSpeechRecognition();<br>SpeechRecognition.startListening();</pre><p><strong>Command Execution</strong></p><p>Once a command is recognized, we execute a few simple lines of JavaScript to control the media player on the current tab. For example, to play and pause, we simply find the first HTML video element on the page and call the play() and pause() functions on that element.</p><pre>let vid = document.getElementsByTagName(&quot;video&quot;)[0];<br>if (request.action === &quot;play&quot;) {<br>    vid.play();<br>} else if (request.action === &quot;pause&quot;) {<br>    vid.pause();<br>}</pre><h4>Limitations</h4><p>Unfortunately, we were unable to get the extension running in the background of the page, so currently, a user needs to manually click on the extension icon to start the detection. This defeats the purpose of the extension since it requires user action to move the mouse, but should be possible to fix later on. However, because of this, we did not publish the extension to the Chrome Web Store.</p><p>Another limitation of this extension is that depending on how the media player is embedded into a website, and what media player is being used, our code used for media playback control might not generalize to all platforms. However, our code can be easily extended by adding platform-specific JavaScript code for media playback control.</p><h3>Demo</h3><p>The following video gives a short demo of the current extension prototype using the “play” and “pause” commands, as well as a demonstration of how we can run our trained model locally using Howl.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FulfWWNdlzsU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DulfWWNdlzsU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FulfWWNdlzsU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/07363aa3908da51bf1503a6a92a9d30a/href">https://medium.com/media/07363aa3908da51bf1503a6a92a9d30a/href</a></iframe><p>As mentioned before, all of our model weights and code for the extension can be found at <a href="https://github.com/paksha/audio-project">paksha/audio-project</a>. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a089cb2dd971" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>