<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jacquelyn Wax on Medium]]></title>
        <description><![CDATA[Stories by Jacquelyn Wax on Medium]]></description>
        <link>https://medium.com/@jacquelynwax?source=rss-cf3616c51fe8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JRxwiXqehmwvKTUBfY0S4A.jpeg</url>
            <title>Stories by Jacquelyn Wax on Medium</title>
            <link>https://medium.com/@jacquelynwax?source=rss-cf3616c51fe8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:15:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jacquelynwax/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Capturing and Piping Audio from a User Device to the IBM Watson Speech-to-Text API]]></title>
            <link>https://medium.com/@jacquelynwax/capturing-and-piping-audio-from-a-user-device-to-the-ibm-watson-speech-to-text-api-f41d1c19242a?source=rss-cf3616c51fe8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f41d1c19242a</guid>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Jacquelyn Wax]]></dc:creator>
            <pubDate>Mon, 29 Jan 2018 11:58:15 GMT</pubDate>
            <atom:updated>2018-01-29T18:57:02.630Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pemcl0wkTH0Up_sRVrucCQ.png" /></figure><p>IBM’s Watson Speech-to-Text API has received high marks from developers for many reasons. The service has three interfaces: WebSockets, HTTP REST, and asynchronous HTTP. It has extra features such as profanity filtering, keyword matching, and word confidence, too. And the documentation is clear and developer friendly. So, when a group of fellow developers and I decided to build an application leveraging speech-to-text technology, we turned to Watson.</p><p>None of us had experience with speech-to-text technology, and we learned that one of our first difficulties was capturing audio from a user device and sending it to Watson in a format the API recognizes and can work with. This was a particular challenge when compared with other popular APIs like the <a href="https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition">SpeechRecognition</a> Web API.</p><p>There is <a href="https://github.com/watson-developer-cloud/speech-javascript-sdk">an excellent SDK</a> that makes the the work of capturing audio from a user device and piping it in the correct form to IBM easy, and I recommend it highly. In the code snippets below, I’ll provide some insight into the key components of capturing audio, converting it into a usable stream, sending the data from that stream to watson via sockets, and show why my team and I ultimately turned to the SDK to convert our data to the format watson needed.</p><p><a href="https://github.com/jacquelynwax/watson-speech-to-text-tutorial">This Github repository</a> provides a simple example of a watson speech-to-text app that uses the SDK. The project has a simple server, a simple index.html with two buttons — a start listening and stop listening — and a simple script file.</p><p>We’ll be working with several technologies and libraries in addition to the SDK.</p><ul><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia">getUserMedia()</a></li><li><a href="https://www.npmjs.com/package/microphone-stream">microphoneStream npm</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">WebSockets</a></li><li><a href="https://www.ibm.com/watson/services/speech-to-text/">the Watson Speech-to-Text API</a></li></ul><p><strong>MediaDevices.getUserMedia()</strong></p><p>Our team decided to use the MediaDevices.getUserMedia() method to capture a user’s speech. The method prompts the user for permission to use her device’s media input and captures her speech in a <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaStream">MediaStream</a>. This stream contains tracks that contain the requested types of media (in our case, audio files). MediaDevices.getUserMedia() returns a Promise that resolves to the <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaStream">MediaStream</a> object. (Throw a console.log in the code below if you want to inspect the stream and its prototype chain.)</p><pre>document.querySelector(&#39;#button-start&#39;).addEventListener(&#39;click&#39;, () =&gt; {<br>  navigator.mediaDevices.getUserMedia({ audio: true }) <br>    .then(stream =&gt; {<br>      /* use the stream */<br>    })<br>    .catch(err =&gt; {<br>      /* handle the error */<br>    });<br>})</pre><pre>document.querySelector(&#39;#button-stop&#39;).addEventListener(&#39;click&#39;, () =&gt; {<br><br>})</pre><p>At this point, we knew we needed to access the actual audio data from the stream. So we turned to the <a href="https://www.npmjs.com/package/microphone-stream">microphoneStream npm module</a>, which converts the media stream above into a standard Node.js-style stream for easy piping. In the code snippet below, we’re requiring that library and using its setStream() method to produce the stream. The micStream.on(‘data’) eventHandler below is extracting buffers from the stream as Uint8Array DataViews and then converting them into Float32Array.</p><pre>&#39;use strict&#39;<br>const MicrophoneStream = require(&#39;microphone-stream&#39;)<br>const micStream = new MicrophoneStream()</pre><pre>document.querySelector(&#39;#button-start&#39;).addEventListener(&#39;click&#39;, () =&gt; {<br>  navigator.mediaDevices.getUserMedia({ audio: true }) <br>    .then(stream =&gt; {<br>      micStream.setStream(stream)<br>    })<br>    .catch(err =&gt; {<br>      /* handle the error */<br>    });<br>})</pre><pre>micStream.on(&#39;data&#39;, (chunk) =&gt; { <br>   let raw = MicrophoneStream.toRaw(chunk)<br>});</pre><pre>document.querySelector(&#39;#button-stop&#39;).addEventListener(&#39;click&#39;, () =&gt; {<br>   micStream.stop()<br>})</pre><p>This is where my team and I spent hours experimenting with piping our Float32Arrays to watson. (The code below shows our progress.) We scoured documentation to convert our arrays to 16 bit integers and downsample the the stream. While our websocket connection was successful, we weren’t receiving any other events from watson in response to our stream besides the initial response of { ‘state’ : ‘listening’ }. So we knew we weren’t successfully converting our audio.</p><pre>const MicrophoneStream = require(&#39;microphone-stream&#39;)<br>const micStream = new MicrophoneStream()</pre><pre>const token = {watson_token}<br>const wsURI = &#39;wss://stream.watsonplatform.net/speech-to-text/api/v1/recognize&#39;<br>  + &#39;?watson-token=&#39; + token<br>  + &#39;&amp;model=es-ES_BroadbandModel&#39;<br>const websocket = new WebSocket(wsURI)<br>websocket.onopen = function(evt) { onOpen(evt) }<br>websocket.onclose = function(evt) { onClose(evt) }<br>websocket.onmessage = function(evt) { onMessage(evt) }<br>websocket.onerror = function(evt) { onError(evt) }</pre><pre>function onOpen(evt) {<br>  var message = {<br>    &#39;action&#39;: &#39;start&#39;,<br>    &#39;content-type&#39;: &#39;audio/l16;rate=22050&#39;<br>  };<br>  websocket.send(JSON.stringify(message))<br>}</pre><pre>function onClose(evt) {<br>  console.log(&#39;The connection has been closed.&#39;)<br>}</pre><pre>function onMessage(evt) {<br>  console.log(evt.data)<br>}</pre><pre>document.querySelector(&#39;#button-start&#39;).addEventListener(&#39;click&#39;, () =&gt; {<br>  navigator.mediaDevices.getUserMedia({ audio: true })<br>    .then(stream =&gt; {<br>      micStream.setStream(stream)<br>    })<br>    .catch(err =&gt; {<br>      /* handle the error */<br>    });<br>})</pre><pre>micStream.on(&#39;data&#39;, (chunk) =&gt; { <br>   let raw = MicrophoneStream.toRaw(chunk)<br>   websocket.send(raw)<br>});</pre><pre>document.querySelector(&#39;#button-stop&#39;).addEventListener(&#39;click&#39;, () =&gt; {<br>   micStream.stop()<br>   websocket.close()<br>})</pre><p>Then we discovered the SDK, which does all the work we outlined above and more.</p><ul><li>It <a href="https://github.com/saebekassebil/microphone-stream/blob/master/microphone-stream.js">extracts audio from mediaStream</a>.</li><li>It <a href="https://github.com/watson-developer-cloud/speech-javascript-sdk/blob/master/speech-to-text/webaudio-l16-stream.js">converts and down samples the stream into a digestible format for watson</a>.</li><li>It also <a href="https://github.com/watson-developer-cloud/speech-javascript-sdk/blob/master/speech-to-text/recognize-stream.js">manages the WebSocket connection</a>. (While watson’s document is straightforward and clear on this point, it’s a nice feature to have this all managed out of the box.</li></ul><p>Our code now looks something like this — simpler, cleaner, and most importantly, it works.</p><pre>const WatsonSpeech = require(&quot;watson-speech&quot;)</pre><pre>const token = {watson_token}</pre><pre>document.querySelector(&#39;#button-start&#39;).addEventListener(&#39;click&#39;, () =&gt; {<br>  var stream = WatsonSpeech.SpeechToText.recognizeMicrophone({<br>    token: token,<br>  })</pre><pre>stream.setEncoding(&#39;utf8&#39;)</pre><pre>stream.on(&#39;data&#39;, function(data) {<br>    console.log(data);<br>  })</pre><pre>stream.on(&#39;error&#39;, function(err) {<br>    console.log(err);<br>  })</pre><pre>document.querySelector(&#39;#button-stop&#39;).addEventListener(&#39;click&#39;, () =&gt; {<br>    stream.stop()<br>  })<br>})</pre><p>As a reminder, you can experiment with this code at <a href="https://github.com/jacquelynwax/watson-speech-to-text-tutorial">this repository</a>. (Don’t forget to npm install if you clone and run this on your local machine.) I wish you the best of luck in creating your speech-to-text application with watson and hope this post saves you the time my team and I spent. My advice: Use the SDK.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f41d1c19242a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>