<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Linagora LABS - Medium]]></title>
        <description><![CDATA[Linagora R&amp;D activities - Medium]]></description>
        <link>https://medium.com/linagoralabs?source=rss----e3b60ed6b7f9---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Linagora LABS - Medium</title>
            <link>https://medium.com/linagoralabs?source=rss----e3b60ed6b7f9---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:45:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/linagoralabs" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Multi-gesture detection with Tensorflow API ‘TFOD’]]></title>
            <link>https://medium.com/linagoralabs/multi-gesture-detection-with-tensorflow-api-tfod-1093c39973b5?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/1093c39973b5</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[gesture-recognition]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[MOHAMED BEN ALI]]></dc:creator>
            <pubDate>Tue, 25 Jan 2022 15:17:52 GMT</pubDate>
            <atom:updated>2022-01-25T15:17:51.984Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*iEPXirxQxgTTxs8Tb1Yu2w.png" /></figure><h3>What is LinTO ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*4olX2b9yX1XT4GilZSsPEQ.png" /></figure><p><a href="https://labs.linagora.com/linto/">LinTO</a> is an intelligent Open Source assistant based only on Open Source technologies, It’s GAFAM free and easy to install on a cloud.</p><p>Research and development in the LinTO project pursued two paths. The first was to create functionalities common to personal assistants, allowing users to use their voice to read their mails, consult their agendas, open and share documents, search for information, and so on. The second consisted in developing advanced options to produce high-quality transcriptions of full multi-party conversations and follow conversational exchanges in order to take notes, produce relevant real-time recommendations, and automatically generate meeting minutes.</p><p>LinTO has been developed both as a device that runs on the Raspberry Pi (or other ARM smart boards) and as an<strong> </strong><a href="https://linto.ai/"><strong>open-source platform</strong></a> that can be deployed, adopted, or maintained either directly on a company’s premises or remotely as a service (SaaS).</p><p>At the end of our meeting, a report has to be drawn up, this is where LinTO comes in with the capability to perform voice, gestural detection.</p><p>Let’s consider a use case in which six members of a team are present in a work meeting.</p><p>During this meeting, let’s assume there were three people who spoke and four people who voted for new internal regulation.</p><p>During my scientific research with <a href="https://labs.linagora.com/"><strong>LINAGORA Labs</strong></a>, I worked mainly on visual detection approaches such as the detection of people and their gestures.<br>In this article, I will explain the different steps to build a dataset of images with several gestures and how to train and evaluate a model with the Tensorflow API tool.</p><p>In the upcoming parts, I will detail the process that is necessary to perform facial and gestural detection through the construction of a dataset of images.</p><p>We’ll start with the setup.</p><h3>Step 1: Prepare development environment</h3><p>First of all, we have to install the environment in <a href="https://ubuntu.com/download/desktop"><strong>ubuntu</strong></a><strong> 20.04</strong> machine to run your object detection application.</p><ul><li><strong>Install and configure python and pip3:</strong></li></ul><pre><strong># At first you will update &amp; upgrade your ubuntu machine<br></strong>sudo apt -y update &amp;&amp; sudo apt -y upgrade</pre><pre><strong># Install python 3.8 repository<br></strong>sudo add-apt-repository ppa:deadsnakes/ppa<br>sudo apt update<br>sudo apt install python3.8</pre><pre><strong># Install pip for python3 on ubuntu 20.04 <br></strong>sudo apt install python3-pip</pre><pre><strong># Verify the installation by cheking the pip3 version<br></strong>pip3 --version</pre><pre><strong># Output<br></strong>pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)</pre><ul><li><strong>Install Jupyter</strong></li></ul><p>Jupyter-notebook is an open document format based on JSON. They contain a complete record of the user’s sessions and include code, narrative text, equations …</p><p>To install jupyter on your machine, you need to type the following command:</p><pre>pip3 install jupyter</pre><p>Jupyter will be installed on the <a href="https://ubuntu.com/download/desktop"><em>ubuntu</em></a> 20.04 system, to run it, type the command mentioned below:</p><pre>jupyter notebook</pre><ul><li><strong>Install and configure Opencv</strong></li></ul><p>To create a model for object detection, you will install <a href="https://linuxize.com/post/how-to-install-opencv-on-ubuntu-20-04/"><strong><em>Opencv</em></strong></a> is a graphic library. She specializes in image processing, whether for still photos or video.</p><pre><strong># Refresh the packages index and install the OpenCV package<br></strong>sudo apt update<br>sudo apt install python3-opencv</pre><pre><strong># Verify the installation<br></strong>python3 -c &quot;import cv2; print(cv2.__version__)&quot;</pre><pre><strong># Output<br></strong>3.2.0</pre><p>To install the latest OpenCV version from the source, perform the following steps:</p><pre><strong># Install the required dependencies<br></strong>sudo apt install build-essential cmake git pkg-config libgtk-3-dev \    <br>libavcodec-dev libavformat-dev libswscale-dev libv4l-dev \    libxvidcore-dev libx264-dev libjpeg-dev libpng-dev libtiff-dev \    gfortran openexr libatlas-base-dev python3-dev python3-numpy \    libtbb2 libtbb-dev libdc1394-22-dev zip nano</pre><pre><strong>#Clone the OpenCV’s and OpenCV contrib repositories<br></strong>mkdir ~/opencv_build &amp;&amp; cd ~/opencv_build<br>git clone https://github.com/opencv/opencv.git<br>git clone https://github.com/opencv/opencv_contrib.git</pre><pre><strong># Create a buid directory<br></strong>cd ~/opencv_build/opencv<br>mkdir build &amp;&amp; cd build</pre><pre><strong># Set up the OpenCV build with CMake<br></strong>cmake -D CMAKE_BUILD_TYPE=RELEASE \    <br>-D CMAKE_INSTALL_PREFIX=/usr/local \    <br>-D INSTALL_C_EXAMPLES=ON \    <br>-D INSTALL_PYTHON_EXAMPLES=ON \    <br>-D OPENCV_GENERATE_PKGCONFIG=ON \    <br>-D OPENCV_EXTRA_MODULES_PATH=~/opencv_build/opencv_contrib/modules \    -D BUILD_EXAMPLES=ON ..</pre><p>You will see something like the screen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*KqDpfWJGlcCmj4HWfkMnMw.png" /></figure><p>Start the compilation process:</p><pre>make -j64</pre><p>Modify the -j flag according to your processor. If you do not know the number of cores in your processor, you can find it by typing nproc.</p><p>The compilation may take several minutes or more, depending on your system configuration.</p><p>Install OpenCV with make:</p><pre>sudo make install</pre><p>To verify the installation, type the following commands and you should see the OpenCV version.</p><p>C++ bindings:</p><pre>pkg-config --modversion opencv4</pre><p>Output:</p><pre>4.5.3</pre><p>Python bindings:</p><pre>python3 -c &quot;import cv2; print(cv2.__version__)&quot;</pre><p>You will see something like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*4VfTisr6dOgK_fIBpQKoiA.png" /></figure><ul><li><strong>Install Tensorflow and Tensorboard</strong></li></ul><p><em>TensorFlow</em> is a free and open-source software library for machine learning and artificial intelligence.</p><p><em>TensorBoard</em> is a tool for providing the measurements and visualizations needed during the machine learning workflow.</p><p>To install the two tools, type the command mentioned below:</p><pre><strong># Install Tensorflow<br></strong>pip3 install tensorflow</pre><pre><strong># Install Tensorboard<br></strong>pip3 install tensorboard</pre><p>To verify the installation, type this command:</p><pre>import tensorflow as tf</pre><pre>print(tf.__version__)</pre><p>Output:</p><pre>2.6.0</pre><p>Or you can type the following commands to verify:</p><pre><strong># Show installation of tensorflow and version<br></strong>pip3 show tensorflow</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*4tjgHw-Kxf5CVhL-YEr09g.png" /></figure><pre><strong># Show installation of tensorboard and version<br></strong>pip3 show tensorboard</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1mlYFN8jF-iAimBGjM0rOw.png" /></figure><ul><li><strong>Install Tensorflow API</strong></li></ul><p>As we are going to build an object detection model with Tensorflow API, we need to install all dependencies. All steps you can find on<strong><em> </em>the </strong><a href="https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/install.html"><strong>installation page</strong></a><strong>.</strong></p><p>Tensorflow Object Detection API depends on the following libraries:</p><ul><li>Protobuf</li><li>setuptools</li><li>cython</li><li>Pillow</li><li>lxml</li><li>tf_Slim</li><li>CocoAPI</li><li>Matplotlib</li></ul><p>To install all these libraries, you have to type the following commands below:</p><pre><strong># The remaining libraries can be installed on Ubuntu using via apt<br></strong>sudo apt install protobuf-compiler python3-pil python3-lxml</pre><pre><strong># Install python packages with pip3<br></strong>pip3 install matplotlib<br>pip3 install tf_slim<br>pip3 install setuptools<br>pip3 install cython</pre><p>We need to install TensorflowAPI using the following commands:</p><pre><strong># Create an folder named Tensorflow (for example)<br></strong>mkdir Tensorflow<br>cd Tensorflow</pre><pre><strong># Use Git to clone the Model Garden for TensorFlow<br></strong>git clone https://github.com/tensorflow/models.git</pre><pre><strong># Clone COCO API <br></strong>git clone https://github.com/cocodataset/cocoapi.git<br>cd cocoapi/PythonAPI</pre><pre><strong># Edit the Makefile with changing python to python3<br></strong>make<br>cp -r pycocotools ../models/research/</pre><pre><strong># Install protobuf in models/research/<br></strong>cd models/research<br>mkdir protoc</pre><pre>wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip</pre><pre>unzip protobuf.zip<br>./protoc/bin/protoc object_detection/protos/*.proto --python_out=.</pre><pre><strong># Install the Object Detection API<br><em># From within TensorFlow/models/research/</em><br></strong>cp object_detection/packages/tf2/setup.py .<br>python3 -m pip install .</pre><p>Output:</p><pre>Successfully installed object-detection-0.1</pre><p>Add research/slim to your PYTHONPATH:</p><pre><strong># From within <em>TensorFlow/models/research/slim</em><br></strong>export PYTHONPATH=$PYTHONPATH:TensorFlow/models/research/slim</pre><p>To test the installation, run the following command from within Tensorflow\models\research:</p><pre>python3 object_detection/builders/model_builder_tf2_test.py</pre><p>You should observe a printout similar to the one below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*pHoNJaPe9QEW8tm3DGytAQ.png" /></figure><p>Once you restart the machine you must run the following commands in your terminal to launch the Tensorflow API.</p><pre><strong># From within Tensorflow/models/research<br></strong>protoc/bin/protoc object_detection/protos/*.proto --python_out=.<br>export PYTHONPATH=$PYTHONPATH:TensorFlow/models/research/slim</pre><h3>Step 2: Construction of gesture images dataset</h3><p>To build our dataset of gesture images, we must choose the set of gestures we will use. It’s best to use universal gestures, such as those documented here:</p><p><a href="https://www.gettyimages.fr/illustrations/geste-de-la-main#">Illustrations, cliparts, dessins animés et icônes de Geste De La Main - Getty Images</a></p><p>Subsequently, I chose 10 gests from the list above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/1*aKsDGjXTrPGOoGCCyd03ZA.jpeg" /></figure><p>Gesture name = hello</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/1*e1uWxm9FkT0DUf1HhzD0mw.jpeg" /></figure><p>Gesture name = hand_up (handup)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*_o1y_0h-IPXxhdA4XTCLgw.jpeg" /></figure><p>Gesture name = victory</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/1*IRMeXqhJqkb8b3HEIUy8Lw.jpeg" /></figure><p>Gesture name = I love you (iloveyou)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/1*89CMX-AEJ-xinLy-r0Nahw.jpeg" /></figure><p>Gesture name = yes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/145/1*QYUeXjxvQrpRHAK0bpQOig.jpeg" /></figure><p>Gesture name = call me</p><p>(callme)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/1*BBUfJftWb9IEpg9aHKHK8Q.jpeg" /></figure><p>Gesture name = no</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/1*yJ0Wmd9IDWVyvSJVBuYepQ.jpeg" /></figure><p>Gesture name = ok</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/1*-TSvLyQEAHAK0zKAyxy3Qg.jpeg" /></figure><p>Gesture name = okay</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*YOb1Bh7TwWzKRhL_xXyVHA.jpeg" /></figure><p>Gesture name = thanks</p><ul><li><strong>Build dataset of gesture images</strong></li></ul><p>The objective of this part is to record videos according to the 10 gestures mentioned in the previous part, you just have to use the camera of your computer, then you can use the <a href="https://doc.ubuntu-fr.org/cheese">cheese</a> tool like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*AmORylgNW0h8ImN0pmvgVw.png" /></figure><p>Once you finish recording videos for each gesture, you apply the following code to create frames from a gesture video.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13c54777f72ddab23e3a2f063342269b/href">https://medium.com/media/13c54777f72ddab23e3a2f063342269b/href</a></iframe><p>The result of this code is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*egRQz9PmpbIKWASj9b_U_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQsVAuI94yNFxq6-A3tcJw.png" /></figure><ul><li><a href="https://medium.com/linagoralabs/construction-of-360-images-dataset-for-image-recognition-fcddc687dc8b"><strong>Annotation</strong></a></li></ul><p>Before announcing the annotation of the gesture image dataset, the labelImg tool must be installed.</p><p>To install <strong>l</strong><a href="https://github.com/tzutalin/labelImg"><strong>abeImg</strong></a>, type the following command lines:</p><pre><strong># From github clone this repository<br></strong>git clone https://github.com/tzutalin/labelImg<br>cd labelImg</pre><pre><strong># Install pyqt5<br></strong>sudo apt-get install pyqt5-dev-tools<br>sudo pip3 install -r requirements/requirements-linux-python3.txt<br>make qt5py3</pre><pre><strong># Run the tool<br></strong>python3 labelImg.py</pre><p>This step of annotation is very important because, with more annotated images, one always obtains good results during the inference. It is necessary to annotate the images in <a href="https://towardsdatascience.com/coco-data-format-for-object-detection-a4c5eaf518c5"><strong>Pascal VOC</strong></a>(XML file) format and in a regular way.<br>You can follow this demo video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FGCKWw6H_IiI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGCKWw6H_IiI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FGCKWw6H_IiI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6c04f969bbfd80fe06fb3ca938a4174/href">https://medium.com/media/c6c04f969bbfd80fe06fb3ca938a4174/href</a></iframe><p>You can open the image directory on LabelImg and add a bounding box and a label name. When you save, it gives you a .xml file with the details of x,y coordinates of the bounding box (xmin, ymin, xmax, ymax) and labels name.</p><p>The following figure shows you the dimensions of the x, y coordinates obtained by the .xml file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*npSgnUS54mvlkCaxqzgOLg.png" /></figure><p>By finishing this step you should have a folder with .jpg images and a .xml file. The dataset folder will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*tatz11K-HEqxxSZuocS12w.png" /></figure><h3>Step 3: Prepare data for Transfer Learning</h3><p>In this part, we have to prepare the data for <strong>T</strong><a href="https://journalofbigdata.springeropen.com/articles/10.1186/s40537-016-0043-6"><strong>ransfer Learning</strong></a>, then we divide our data into 90% training data and 10% testing data. Using the code located in the scripts folder of the project as shown in the next command:</p><pre><strong># From scripts folder in the project</strong><br>python3 partition_dataset.py -x -i &lt;path_to_dataset_folder&gt; -r 0.1</pre><ul><li><strong>Create Label Map file</strong></li></ul><p>Once you have created the train and test folders, you must then create the label map file which contains the 10 labels, you will find the instructions in the .ipynb file. The following code will allow you to create the label map file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67fb05b2dfed6e2189e5dcaad64be137/href">https://medium.com/media/67fb05b2dfed6e2189e5dcaad64be137/href</a></iframe><p>Output:</p><pre>item {<br>        name:&#39;hello&#39;<br>        id:1<br>}<br>item {<br>        name:&#39;yes&#39;<br>        id:2<br>}<br>item {<br>        name:&#39;no&#39;<br>        id:3<br>}<br>item {<br>        name:&#39;thanks&#39;<br>        id:4<br>}<br>item {<br>        name:&#39;callme&#39;<br>        id:5<br>}<br>item {<br>        name:&#39;ok&#39;<br>        id:6<br>}<br>item {<br>        name:&#39;okay&#39;<br>        id:7<br>}<br>item {<br>        name:&#39;vectory&#39;<br>        id:8<br>}<br>item {<br>        name:&#39;handup&#39;<br>        id:9<br>}<br>item {<br>        name:&#39;iloveyou&#39;<br>        id:10<br>}</pre><ul><li><strong>Generate TFrecords</strong></li></ul><p>TFRecord is TensorFlows binary storage format. We can generate a TFRecord file for the train and test directory using python code in the scripts folder. Run the following line:</p><pre><strong># Run generate_tfrecord.py for train</strong><br>python3 generate_tfrecord.py -x generate_tfrecord.py -x &lt;path_to&gt;/train/ -l &lt;path_to&gt;/annotations/label_map.pbtxt -o &lt;path_to&gt;/annotations/train.record</pre><pre><strong># Run generate_tfrecord.py for test<br></strong>python3 generate_tfrecord.py -x generate_tfrecord.py -x &lt;path_to&gt;/test/ -l &lt;path_to&gt;/annotations/label_map.pbtxt -o &lt;path_to&gt;/annotations/test.record</pre><p>Output:</p><pre>Successfully created the TFRecord file:/annotations/train.record<br>Successfully created the TFRecord file:/annotations/test.record</pre><ul><li><strong>Download TF Models Pretrained Models</strong></li></ul><p>For this project, we have chosen <a href="https://xailient.com/real-time-vehicle-detection-with-mobilenet-ssd-and-xailient/"><strong>Mobilenet SSD</strong></a> as the base model. The SSD models offer high speed and are ideal for detection on video streams. The Mobilenet is suitable for low-core devices as it consumes less space while still giving decent precision figures. To download the ssd_mobilenet_v2 model, just type the following command:</p><pre>wget  http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz</pre><ul><li><strong>Edit pipeline.config file</strong></li></ul><p>Before starting the retraining of the mobilenet_v2 model, we must configure the pipeline.config file which is located in the download model folder.</p><p>It suffices to change the number of classes equal to 10, edit the path of the label list ‘label_map.pbtxt’, and edit the tf_record path ‘train.record, test.record’. Finally, Change the batch_size to 4.</p><p>Run the following code to make the necessary modifications:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6697f913f763bde7b0a84b4cba85ab69/href">https://medium.com/media/6697f913f763bde7b0a84b4cba85ab69/href</a></iframe><h3>Step 4: Retrain the model</h3><p>Retrain the model using the following command:</p><pre>python3 &lt;path_to&gt;/models/research/object_detection/model_main_tf2.py \ <br>--model_dir=&lt;path_to&gt;/workspace_mohamed/models/mobilenet_version2 \<br>--pipeline_config_path=&lt;path_to&gt;/workspace_mohamed/models/mobilenet_version2/pipeline.config \ <br>--num_train_steps=15000</pre><p>Output:</p><pre>Instructions for updating:<br>Use fn_output_signature instead<br>INFO:tensorflow:Step 8100 per-step time 1.150s loss=0.319<br>I0527 14:49:23.321400 140398352971584 model_lib_v2.py:682] Step 8100 per-step time 1.150s loss=0.319<br>INFO:tensorflow:Step 8200 per-step time 1.100s loss=0.324<br>I0527 14:51:14.390439 140398352971584 model_lib_v2.py:682] Step 8200 per-step time 1.100s loss=0.324<br>INFO:tensorflow:Step 8300 per-step time 1.098s loss=0.332<br>I0527 14:53:04.773121 140398352971584 model_lib_v2.py:682] Step 8300 per-step time 1.098s loss=0.332<br>INFO:tensorflow:Step 8400 per-step time 1.106s loss=0.274<br>I0527 14:55:01.560966 140398352971584 model_lib_v2.py:682] Step 8400 per-step time 1.106s loss=0.274<br>INFO:tensorflow:Step 8500 per-step time 1.179s loss=0.397<br>I0527 14:57:07.724631 140398352971584 model_lib_v2.py:682] Step 8500 per-step time 1.179s loss=0.397<br>INFO:tensorflow:Step 8600 per-step time 1.173s loss=0.284<br>I0527 14:59:06.614590 140398352971584 model_lib_v2.py:682] Step 8600 per-step time 1.173s loss=0.284<br>INFO:tensorflow:Step 8700 per-step time 1.271s loss=0.215<br>I0527 15:01:12.691615 140398352971584 model_lib_v2.py:682] Step 8700 per-step time 1.271s loss=0.215<br>INFO:tensorflow:Step 8800 per-step time 1.249s loss=0.275<br>I0527 15:03:16.936001 140398352971584 model_lib_v2.py:682] Step 8800 per-step time 1.249s loss=0.275<br>INFO:tensorflow:Step 8900 per-step time 1.259s loss=0.300<br>I0527 15:05:21.362021 140398352971584 model_lib_v2.py:682] Step 8900 per-step time 1.259s loss=0.300<br>INFO:tensorflow:Step 9000 per-step time 1.306s loss=0.186<br>I0527 15:07:25.405282 140398352971584 model_lib_v2.py:682] Step 9000 per-step time 1.306s loss=0.186</pre><h3>Step 5: Save graph</h3><p>You can choose the latest checkpoint file to export it to a graph file for inference. Checkpoint files get stored in the data folder with the .index extension</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*Pmffc5mv078H3NQBDSlGFQ.png" /></figure><p>To export the graph model use the following command:</p><pre>python3 &lt;path_to&gt;/models/research/object_detection/exporter_main_v2.py \ <br>--input_type=image_tensor \<br>--pipeline_config_path=&lt;path_to&gt;/workspace_mohamed/models/mobilenet_version2/pipeline.config \ <br>--trained_checkpoint_dir=&lt;path_to&gt;/workspace_mohamed/models/mobilenet_version2/ \ <br>--output_directory=&lt;path_to&gt;/workspace_mohamed/models/mobilenet_version2/export</pre><h3>Step 6: Evaluate model</h3><p>The objective of the evaluation phase is to know the performance of the trained model in terms of <a href="https://towardsdatascience.com/map-mean-average-precision-might-confuse-you-5956f1bfa9e2#:~:text=mAP%20(mean%20average%20precision)%20is,averaged%20to%20get%20the%20mAP.&amp;text=The%20mean%20Average%20Precision%20or,different%20detection%20challenges%20that%20exist."><strong>mAP</strong></a><strong> .</strong></p><p>Then type the following command to evaluate our model:</p><pre>python3 &lt;path_to&gt;/models/research/object_detection/model_main_tf2.py --model_dir=&lt;path_to&gt;/workspace_mohamed/models/mobilenet_version2 \<br>--pipeline_config_path=<br>&lt;path_to&gt;/workspace_mohamed/models/mobilenet_version2/pipeline.config \<br>--checkpoint_dir=&lt;path_to&gt;/workspace_mohamed/models/mobilenet_version2</pre><p>Output:</p><pre>Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.646<br> Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 1.000<br> Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.792<br> Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000<br> Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000<br> Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.646<br> Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.683<br> Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.695<br> Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.695<br> Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000<br> Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000<br> Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.695<br>INFO:tensorflow:Eval metrics at step 15000<br>I0921 10:30:51.929377 139872141592384 model_lib_v2.py:988] Eval metrics at step 15000<br>INFO:tensorflow: + DetectionBoxes_Precision/mAP: 0.646000<br>I0921 10:30:51.934390 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Precision/mAP: 0.646000<br>INFO:tensorflow: + DetectionBoxes_Precision/mAP@.50IOU: 1.000000<br>I0921 10:30:51.935504 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Precision/mAP@.50IOU: 1.000000<br>INFO:tensorflow: + DetectionBoxes_Precision/mAP@.75IOU: 0.792357<br>I0921 10:30:51.936339 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Precision/mAP@.75IOU: 0.792357<br>INFO:tensorflow: + DetectionBoxes_Precision/mAP (small): -1.000000<br>I0921 10:30:51.937054 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Precision/mAP (small): -1.000000<br>INFO:tensorflow: + DetectionBoxes_Precision/mAP (medium): -1.000000<br>I0921 10:30:51.937712 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Precision/mAP (medium): -1.000000<br>INFO:tensorflow: + DetectionBoxes_Precision/mAP (large): 0.646009<br>I0921 10:30:51.938401 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Precision/mAP (large): 0.646009<br>INFO:tensorflow: + DetectionBoxes_Recall/AR@1: 0.683333<br>I0921 10:30:51.939035 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Recall/AR@1: 0.683333<br>INFO:tensorflow: + DetectionBoxes_Recall/AR@10: 0.694762<br>I0921 10:30:51.939728 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Recall/AR@10: 0.694762<br>INFO:tensorflow: + DetectionBoxes_Recall/AR@100: 0.694762<br>I0921 10:30:51.940590 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Recall/AR@100: 0.694762<br>INFO:tensorflow: + DetectionBoxes_Recall/AR@100 (small): -1.000000<br>I0921 10:30:51.941257 139872141592384 model_lib_v2.py:991]  + DetectionBoxes_Recall/AR@100 (small): -1.000000<br>INFO:tensorflow: + DetectionBoxes_Recall/AR@100 (medium): -1.000000</pre><p>There is another method to evaluate a model using Tensorboard, then type the following command:</p><pre><strong># in your terminal<br></strong>tensorboard --logdir &lt;path_to_dir&gt;/eval</pre><pre><strong># Output</strong><br>2021-11-03 14:46:06.061849: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library &#39;libcudart.so.11.0&#39;; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory<br>2021-11-03 14:46:06.061879: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.<br>Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all<br>TensorBoard 2.4.1 at <a href="http://localhost:6007/">http://localhost:6007/</a> (Press CTRL+C to quit)</pre><pre><strong># go to this link</strong><br><a href="http://localhost:6007/">http://localhost:6007/</a></pre><p>Results with Tensorboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*o1iKVocUX9yDkF_IGqmfBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsyHNJLxD3utcE2VCgHZxw.png" /></figure><h3>Step 7: Real-Time gestures detection</h3><p>In this part of the project, we will show the results of gesture detection in real-time using the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5fc349f5f14dea12531b682cb2c306dc/href">https://medium.com/media/5fc349f5f14dea12531b682cb2c306dc/href</a></iframe><p>Output:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fe5MaD19dy8A%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3De5MaD19dy8A&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fe5MaD19dy8A%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ff388c9c810d0076607bfa7700c990a/href">https://medium.com/media/3ff388c9c810d0076607bfa7700c990a/href</a></iframe><h3>Conclusion</h3><p>In this medium article, we presented the different steps to build a gesture detection model with the Tensorflow API and by retraining the mobilenet version SSD model.</p><p>In addition, we have built a dataset of gesture images which is located in the following repository: <a href="https://github.com/linto-ai/multi-hand-gesture-dataset.git"><strong>https://github.com/linto-ai/multi-hand-gesture-dataset.git</strong></a><strong><br></strong>With this dataset of gesture images, we obtained an mAP equal to 64.6%, we can improve easily the precision by annotating more images for each label.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1093c39973b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/multi-gesture-detection-with-tensorflow-api-tfod-1093c39973b5">Multi-gesture detection with Tensorflow API ‘TFOD’</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Construction of 360° images dataset for image recognition]]></title>
            <link>https://medium.com/linagoralabs/construction-of-360-images-dataset-for-image-recognition-fcddc687dc8b?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/fcddc687dc8b</guid>
            <category><![CDATA[image-labeling]]></category>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[linto]]></category>
            <category><![CDATA[linagora]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[ons wechtati]]></dc:creator>
            <pubDate>Mon, 01 Mar 2021 13:09:54 GMT</pubDate>
            <atom:updated>2021-03-01T15:07:56.985Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*ScoVbyXuATbWZEHlbQNs8Q.png" /></figure><p>During my summer internship with Linagora’s R&amp;D team, I was immersed in a world that any technology enthusiast would dream of, especially artificial intelligence.</p><p>My mission was to participate in the development of the intelligent agent LINTO and in particular in the module allowing gestural and visual detection, through the construction and training of a 360° image dataset in order to improve the quality of detection of LINTO.</p><p>The article below will give a global and detailed description of this project.</p><h3>What is LINTO ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*lA-FktKkkKkkKHuA.png" /></figure><p>LINTO is an intelligent Open Source assistant based only on Open Source technologies, and LINTO is compatible with the Cloud and free from GAFAM .</p><p>It is a personal assistant able to recognize your voice and help you in a professional environment, it’s also respectful of your privacy and does not share your information for commercial purposes.</p><p>The research project is subsidized by the PIA (Programme d’Investissement d’Avenir) of the French state within the framework of the Grand Challenges of the Digital Age. It brings together technology companies such as LINAGORA and ZELROS and research laboratories such as IRIT, LaaS, CNRS and the computer science research laboratory of the “Ecole Polytechnique”.</p><p>LINTO is made primarily to meet the needs expressed by professionals within companies, with the primary goal of reducing tedious and stressful tasks.</p><p>To better understand the purpose of this work, we can take the example of a weekly meeting during which six members of the work team will be present.</p><p>During this meeting there were three people who spoke , and four people voted for a new internal regulation.</p><p>At the end of our meeting a report has to be drawn up, this is where LINTO comes in with the capabilities to perform voice, facial and gestural detection.</p><p>In the following we will detail the process that is necessary to perform the facial and gestural detection through the construction of a data set of panoramic images.</p><h3>Construction of a 360 images data set :</h3><p>Before proceeding to the annotation step, it is essential to have a set of panoramic images.</p><p>These images are taken from various video recordings taken during several meetings, which will be recorded after these steps:</p><ul><li>Checking the ambient light</li><li>Choosing a suitable location for the camera</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UtHMdleYfYCTrFIv" /></figure><p>In a second step, we have to divide each video into a set of frames ( image.jpg), this is done with the help of a script written in python:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_IxyjZ_WR7EyFPW7" /></figure><p>to train our model, the acquired images must be annotated correctly and carefully</p><h3>Image annotation :</h3><p>Giving eyes to your machine is no longer a dream, thanks to artificial intelligence technologies a machine can perceive the objects constituting the environment in which it is located, i.e. analyze, process and understand several images, this is called computer vision.</p><p>The ability to see and interpret the world can provide us with seemingly cutting-edge technology such as medical imaging analysis, optical character recognition, and facial and gesture recognition.</p><p>However, this will not be possible without using clear, annotated data to form machine learning models.</p><p>Indeed, the performance rate of machine learning and deep learning algorithms* depends on the accuracy rate of the training data on which these models are based.</p><p>The machine learning models are fed from the data provided. As soon as an algorithm has processed enough annotated data, it can begin to automatically recognize recurring new non-annotated data in the annotated data provided to it.</p><p>In other words, annotation is a manual task that involves assigning labels or metadata to a dataset, it is also a type of data labeling.</p><p>Image annotation is used to identify objects and to segment images, it marks the data that the machine learning system is supposed to recognize, it is an indispensable step in the pre-processing of data in supervised learning .</p><h3>The Type of image annotation used :</h3><p>There are various annotation methods. The method of annotation is chosen according to the need, in our case, we had to use enclosing boxes, it is the simplest type of annotation and the most used, these are imaginary boxes drawn on images. The contents of the bounding box are annotated to help a machine learning model to recognize it as a distinct type of object.</p><p>For this form of annotation the annotator must draw a frame around the object he wants to annotate in the image, this frame must be as close as possible to the edges of the objects to be annotated.</p><p>The use of delimiting frames is often useful for the classification, localization and detection of objects.</p><p>Sometimes there may be more than one target object in a single image, in which case there should be as many frames as there are objects to delineate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*q8E5YnKuRRbm3lMjj_HqGg.jpeg" /></figure><h3>Method used for image annotation :</h3><p>To be able to annotate images we need annotation software.</p><p>“LabelImg” is a source program developed with Python that allows you to :</p><p>Create a 2d rectangle around an identified element of type YOLO or VOC PASCAL</p><p>Assign a label to this rectangle.</p><p>Generate .txt or .xml files.</p><h3>The different annotation classes :</h3><p>In order to be able to identify people, speaking and voting gestures</p><p>We started by adding new labels to our open source software which are the following:</p><p>person</p><p>voting</p><p>speaking</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*SdZK5v1SmPHJeSBBWpt6KQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*0b-huFaeLa9THlvS9lxevg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*tSzGI4aNzOnBzCokCs-2mg.jpeg" /></figure><p>Then we started to annotate the images related to speaking, voting and person.</p><h3>Generation of the XML file :</h3><p>Once an image has been well annotated, an XML file corresponding to it will be automatically generated by the LabelImg software.</p><p>This file contains :</p><p>The location of the image from which this file comes from</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/0*j93x-1ZXPY1XwrnX" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/0*6tZjGkeXXCdMolcN" /></figure><p>One or more “Object” tags, each of which corresponds to an annotated object, thus containing the label used and the spatial coordinates on the x-axis and y-axis of the tagging frame.</p><p>These coordinates are essential for identifying objects and for training models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hzaxoGUM29ZA9_bv" /></figure><h3>Annotation quality check :</h3><p>Each image in the dataset must be carefully and accurately labeled to train a deep learning model to recognize objects in the same way a human can. The higher the quality of the annotation, the better the performance of the model.</p><p>It is therefore necessary to check the quality of the annotation before moving on to model training.</p><p>Such a task can be performed using a script written in Python in a Jupyter Notebook where it’s possible to read the desired image and then check the position of the labeling rectangle.</p><p>Such a manual verification, despite its efficiency, remains a repetitive and time-consuming task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*DaJdohGt26YG_ztnqUgmQQ.jpeg" /></figure><p>The process of data enrichment (annotation) is the most laborious part of an artificial intelligence project, which improves performance and accuracy by efficiently training the created models.</p><p>Producing an annotated dataset in sufficient quantity and quality to train a good model and have acceptable results, does not provide 100% performance.</p><p>The annotation work does not stop when the first models are performing well.</p><p>Such a project depends on the data that is provided, and cannot guarantee the same performance on new cases that cannot be anticipated.</p><p>However, the need to detect new gestures requires to annotation on targeted examples.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcddc687dc8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/construction-of-360-images-dataset-for-image-recognition-fcddc687dc8b">Construction of 360° images dataset for image recognition</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training a Language Model using Federated Learning]]></title>
            <link>https://medium.com/linagoralabs/training-a-language-model-using-federated-learning-21bb2cc3b945?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/21bb2cc3b945</guid>
            <category><![CDATA[federated-learning]]></category>
            <category><![CDATA[gdpr-compliance]]></category>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[linto]]></category>
            <dc:creator><![CDATA[Pierre Philbert]]></dc:creator>
            <pubDate>Mon, 16 Nov 2020 13:51:25 GMT</pubDate>
            <atom:updated>2020-11-16T14:40:20.837Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*8k2YbEIeOll2AHxRSJZDOg.jpeg" /></figure><p>I work as a trainee in <a href="https://linagora.com/">LINAGORA’s</a> <a href="https://research.linagora.com/display/home/Home">R&amp;D department</a> on <a href="https://linto.ai/">LinTO</a> project. If you do not know LinTO, I strongly recommend you look into it. Smart voice-based assistants are not new, and most of us knows current leaders (Alexa, Google Home, …) but none of them, except LinTO, are open-source.</p><p>This article aims to present federated learning through a brief explanation and an application to a simple case of Natural Language Processing (NLP).</p><p><strong>Why ?</strong></p><p>Why has LINAGORA had a team [of engineers and data scientist] working on a voice assistant for the last two years when it’s possible to use the Google offering ? What was the motivation in first place to spend this amount of energy, time, and money ? It is LINAGORA’s mission to fight for digital sovereignty, and for user’s privacy while improving user experience.</p><p><strong>How ?</strong></p><p>To achieve their goal, they are offering plenty of services, but I will only talk about the R&amp;D team. One of their objectives is to assure that LinTO will be specific to the realm of the client. For instance, if an electricity supplier company pays for LinTO’s services, LINAGORA will make sure to integrate the specific vocabulary of the electricity supplier business. It enables them to automate activities of low added value tasks (i.e. writing a meeting report) without depending on GAFAM.</p><p><strong>What ?</strong></p><p>Here comes the technical part: if you want to learn what the elementary components of a smart voice-based assistant are, I recommend you to read <a href="https://medium.com/linagoralabs/training-of-a-speech-recognition-model-for-the-spanish-language-8202e22093ae">this article</a> — from now on I will assume you are familiar with these components.</p><p>If you look to train a language model or/and speech-to-text , you might read these two articles (<a href="https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/">LSTM for Text Generation</a> — <a href="https://www.analyticsvidhya.com/blog/2019/07/learn-build-first-speech-to-text-model-python/?utm_source=blog&amp;utm_medium=comprehensive-introduction-language-models-nlp-python">Speech To Text Model</a>) . You will see that to do so, you will need to work with different kind of datasets containing voice recording and their transcriptions. The data are stored on a server where you will train and test your model.</p><p>But as it is stipulated in article 4 of General Data Protection Regulation (GDPR), transcripts of discussion might be private and voice recordings are considered, no matter what is being said, as highly sensitive data. Meaning it is very delicate to collect, store and manipulate these essential data.</p><p>The problem I focused on during my internship was the following : <strong>how do you learn on extensive datasets without being intrusive ?</strong></p><p>It turns out that Google already started to work on this question in 2017 when they published <a href="https://arxiv.org/pdf/1811.03604.pdf">their first article</a> on federated learning. Instead of collecting all users’ information at one location (on a server) and updating weights locally, the model is sent to all users and train separately on those users’ device’s data. Once it is done you take the weights average of your neural network and send it to the server. To sum up, you will never access your clients’ personal data, neither the model of each client after training but only the average of all clients’ model who take part in training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*MSV5IEKWHBpx7a00qOTG5A.png" /><figcaption>Federated learning explanation</figcaption></figure><p>I will present to you an application of a distributed algorithm using Tensorflow Federated framework (also known as tff) to train a Speech-To-Text (STT) model. I followed a <a href="https://www.analyticsvidhya.com/blog/2019/07/learn-build-first-speech-to-text-model-python/?utm_source=blog&amp;utm_medium=comprehensive-introduction-language-models-nlp-python">tutorial</a> and used four convolutional layers to recognize ten different words : yes, no, up, down, left, right, on, off, stop, go.</p><p>I will explain what I did step by step to achieve this mission :</p><p><strong>STEP 1 — Prepare the dataset</strong></p><p>The first step will be to process the audio files to make them understandable to your Artificial Intelligence (AI). Once the sampling, and conversion part is done, you can split your dataset into a 80% training and 20% validation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*YS1Xr18D2GE0yCGRUyexww.png" /><figcaption>Split</figcaption></figure><p>To simulate a federated process, you can use an array where each element will be a Map dataset representing a client. In my case, to simplify the process, I use an IID (independent and identically distributed) subset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*FPtylVUDk8AF9q6sOaA--g.png" /><figcaption>Generate Clients</figcaption></figure><p><strong>STEP 2 — Create model</strong></p><p>Using Keras you will define a model that will correspond to this schema :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*mXC0Thap8t1khkKypMXRYw.png" /><figcaption>Speech-To-Text model</figcaption></figure><p><strong>STEP 3 — Create a wrapper and an iterative process</strong></p><p>The wrapper will return a Tensorflow Federated object containing Keras model, the input specification and the loss function.</p><p>To define your iterative process, you will use your wrapper, a client optimizer to compute local model updates on each client and a server optimizer that will apply the averaged update to the global model at the server. By default, the server optimizer is a Stochastic Gradient Descent (SGD) with a learning rate of 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*xkKBULt_wIrkXzz25X2w_Q.png" /><figcaption>Wrapper</figcaption></figure><p><strong>STEP 4 — Training</strong></p><p>The iterative process that you have defined in step 3 only have two functions :</p><p>initialize() : The computation takes no arguments and returns one result — the representation of the state of the Federated Averaging process on the server.</p><p>next() : It represents a single round of Federated Averaging, which consists of pushing the server state (including the model parameters) to the clients, on-device training on their local data, collecting and averaging model updates, and producing a new updated model at the server.</p><p>You will use the two following functions for training:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*rXMQM8sCh-FIelo2k2zpWA.png" /><figcaption>Train</figcaption></figure><p>Do not forget to save your model so you can evaluate it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*-csb4_F7On_ewjcA9w637g.png" /><figcaption>Save Model</figcaption></figure><p>To evaluate your model, just compute accuracy :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*5xpWu5dkudAjIvrJJdZf0Q.png" /><figcaption>Evaluation</figcaption></figure><p><strong>CONCLUSION</strong></p><p>For my federated approach, I have simulated 3 clients each having 15 audio samples and trained them for 5000 epochs. According to the validation dataset, I have reached an accuracy of 0.89, overtaking the standard approach using the same model with the same amount of data. The only constraint is the training time : using 12 CPUs, I needed 36 hours to finish training when I only needed 15 minutes with standard approach. This could be explained by the needs of GPU that require TFF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*FGClkIT_LVNPBDWTIYiwLg.png" /><figcaption>Comparison Table</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21bb2cc3b945" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/training-a-language-model-using-federated-learning-21bb2cc3b945">Training a Language Model using Federated Learning</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Assembling a large German speech corpus]]></title>
            <link>https://medium.com/linagoralabs/assembling-a-large-german-speech-corpus-7fed0e2ed5e?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/7fed0e2ed5e</guid>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[german-model]]></category>
            <category><![CDATA[linagora]]></category>
            <category><![CDATA[kaldi]]></category>
            <category><![CDATA[linto]]></category>
            <dc:creator><![CDATA[Sina Schaeffler]]></dc:creator>
            <pubDate>Tue, 06 Oct 2020 08:29:37 GMT</pubDate>
            <atom:updated>2020-10-06T08:29:37.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*8ccr-ysDYbm7fDM8kgPNpw.png" /><figcaption>French company for free and open source software</figcaption></figure><p>Today, there are many useful applications for <strong>Automatic Speech Recognition</strong> (ASR), in entertainment, in private live, in the public spaces and also at work. An important step in ASR is the text transcription of an audio file — speech memorized by a machine — to written text. Building this <strong>speech-to-text</strong> part always needs huge amounts of speech <strong>data</strong>, ideally with its transcription.</p><p>This data is needed for different purposes : Most of it is used to tune the internal parameters of the chosen model according to the data it will handle. The data used for this purpose is called the <strong>training set</strong>. Another completely distinct set will be used to test the results of the obtained models on new data, in order to manually choose some external parameters, called hyper-parameters, of the model. This is the <strong>validation set</strong>. A last data set, called the <strong>test set</strong>, completely independent from the others, permits evaluation of the final model.</p><p>This is where <strong>free and open-source applications run into difficulties</strong> : Many large speech corpora aren’t free to use, and even those which are free have a large variety of structures, contents and formats. Thus, we need to <strong>find</strong> such corpora, to bring them in a <strong>single format</strong>, to <strong>test</strong> them by training simple speech-to-text models on them and to finally <strong>join</strong> them in order to get a more complete and larger corpus.</p><p>All these steps have been done to obtain a free German speech corpus, and will be explained in the following.</p><p><strong>Search for free German speech corpora</strong><br><br>In order to train the kind of model we need, it is necessary to get a <strong>large number of small audio files</strong> (each one containing only several seconds of speech, usually one sentence) and the transcription of each one. In order to train the model to adapt to different speakers, it is useful to know which sentences have been spoken by the same person. These needs <strong>strongly limit the amount of accessible data</strong>, since the only collections of audio files suited to our use are those which have been specially formatted for it, for example by cutting them into single sentence files.</p><p>As of July 2020, according to my research, there are <strong>only three large German corpora</strong>, each of which contain over 100 hours of speech and are freely available on the internet. These are:</p><p>- <a href="https://voice.mozilla.org/fr/datasets"><strong><em>Commonvoice</em></strong></a> : An open collaborative project of the Mozilla Foundation</p><p>- <a href="https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/"><strong><em>Caito (m-ailabs)</em></strong></a> : An independent open source project by software engineer Imdat Solak</p><p>-<strong> </strong><a href="http://www.voxforge.org/de/Downloads"><strong><em>Voxforge</em></strong></a><strong><em> : </em></strong>A site which offers open speech corpora in several languages</p><p>Even at first glance the contents are <strong>very different</strong> and their general structure and content differ as well as their size and the characteristics of their audio files :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*H_KbEZwUOUE31dqq5Bv5lw.png" /><figcaption><em>Audio data per corpus</em></figcaption></figure><p>(<strong>Total </strong>: <em>1085h</em> of German speech data)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*1DJHZmLYDqptf_PhR020jg.png" /><figcaption><em>Content and format</em></figcaption></figure><p>It is clear that Commonvoice is by far the largest corpus of the three as it contains more data as both others combined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*jz3wTheIqkr6aT-ymq8xnw.png" /><figcaption><a href="https://commonvoice.mozilla.org/fr/datasets">Mozilla’s Commonvoice corpus</a> contains almost two third of the available data</figcaption></figure><p>Even though all of these data sets are thought for the training of ASR models, each of them has <strong>some limitations</strong> or flaws.</p><p><em>Commonvoice</em>, for instance, is a very large corpus of speech registered in various circumstances, as volunteers contribute to it using their own devices. This is very good for training a model usable in real life. Although, as the sentences are often<strong> checked only twice</strong> for their correctness by other volunteers, the quality of the transcriptions and even the choice of the sentences is not always perfect.</p><p><em>Caito</em> has several issues with the speakers. Since the data comes from audio books, and the only person associated to them is maybe the author, maybe the reader, it is <strong>difficult to know</strong> for certain <strong>who reads which sentence</strong>. For two books, it is marked that there is a mix of male and female speakers, but it is never specified what each of them reads, so this corpus is less useful for speaker adaptation.</p><p><em>Voxforge</em>, finally, uses the same few sentences several times. Each reading has been registered by multiple microphones at same time, with different qualities or noises. This, again, is good to get a model usable in many different situations. However, <strong>their proposed train/validation/test separation is maybe not clean enough</strong>: several sentences of the training set are also in the validation or test set, thus biasing the tests if a model trained on the training set is tested on one of the other sets. Another possible source of bias is the<strong> language model </strong>we used for the tests, which comes from CMU sphinx, which is related to Voxforge, so this model <strong>could overfit</strong> the Voxforge corpus.</p><p>So none of these datasets is perfect, but at least they do exist, they all have some kind of advantages and together they amount to over 1000 hours of audio data, so it should be possible to train decent German ASR models using these freely accessible datasets which is what we tried next.</p><p><strong>Formatting, tests and comparison</strong></p><p>In order to compare the different corpora, we used simple models provided by the <strong>toolkit </strong><a href="https://kaldi-asr.org/"><strong>kaldi</strong></a>. As this toolkit requires the corpora to be in a very <strong>precise format</strong>, we put all of them in this form. <br>Additionally, the corpora Commonvoice and Caito have been split in train/validation/test sets containing, respectively, 80, 10 and 10 percent of the data. The already existing split of Voxforge has been kept despite its flaws.<br>Once this work is finished, two very simple <a href="https://en.wikipedia.org/wiki/Hidden_Markov_model">hidden Markov model</a> - <a href="https://en.wikipedia.org/wiki/Mixture_model#Gaussian_mixture_model">Gaussian mixture model</a> (HMM-GMM) model architectures, one <strong>monophone</strong> (ie pretending a sound’s prononciation does not depend on its neighbors) and one <strong>triphone</strong> (the sound is thought to depend exclusively on its two neighbors) are successively trained on each of the training sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*Hm0A64o6MOtdOFdKeajsLA.png" /><figcaption>The <a href="https://kaldi-asr.org/">kaldi</a> toolkit facilitates testing the corpora</figcaption></figure><p>The results below were obtained by training a model with the steps/train_mono.sh function from kaldi/egs/wsj/s5 on the training set of each corpus and then decoding each of the validation sets with this model using utils/mkgraph.sh and steps/decode.sh from the same example.<br>This function applies the model to the audio data of the set to decode and <strong>compares</strong> the<strong> resulting transcription</strong> to the <strong>real transcription</strong> in order to find the percentage of wrongly transcribed words, called the<strong> word error rate</strong> (<strong>WER</strong>). <br>More precisely, it calculates the WER values for several decoding hyper-parameter variations, and the following WERs are always the best ones for the given decoding.</p><p>For the simplest model, the <strong>monophone</strong> model, we got the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*aWJWlP9eLoPmR-NMB5XASg.png" /><figcaption><em>WER of monophone training</em></figcaption></figure><p>The same table for <strong>triphone</strong> results looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*mmli2kLWBJJMvXMjdsF4sg.png" /><figcaption><em>WER of triphone training</em></figcaption></figure><p>First, the decoded test data using models trained on <strong>Voxforge looks too good to be true</strong>, especially in the monophone table. By investigating, we found out that the validation and training sets of Voxforge have many sentences in common, maybe too many. Additionally, the <strong>dictionary and the language model</strong> from <a href="https://sourceforge.net/projects/cmusphinx/file/Acoustic%20%20and%20Language%20Models/German/">CMUsphinx</a> used for this and the following tests <strong>overfit Voxforge</strong>, as the following table show.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*n4CwH-6vofPDR8vYY9qd-Q.png" /><figcaption><em>Perplexity and percentage of out-of-vocabulary words on validation sets</em> on the validation sets of the corpora</figcaption></figure><p>Aside from the bias of Voxforge, it is interesting that every model does best “within corpus”, i.e. scores best on the validation set from the corpus on which it was trained.This is normal and probably due to resemblance in content and thus sentence type and context, such as background noise or microphone types, between the files of the same corpus.</p><p>As a conclusion, the model <strong>trained on Voxforge is quite certainly not as good as it looks</strong>, and the decoding on Voxforge/validation doesn’t say much about a model’s quality.</p><p><strong>Merging the corpora<br></strong><br>By combining the corpora, an <strong>even larger corpus</strong> can be created.<br>There are several <strong>small difficulties</strong> in the practical realization of this. For example, in order to keep this corpus easy to maintain and to update in case one of its component corpora is updated, it is better to maintain their audio data separately, and to merge only the transcriptions once they are in the same format. Once this is achieved, the previous tests can be run on the new, combined corpus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*PAlzbrzKAv71PBkHcEueDA.png" /><figcaption><em>WERs of monophone training</em></figcaption></figure><p>The combined model performs worse on each validation set than a model <br>decoding in-corpus. Though, it is better than the models trained on <br>Caito or Commonvoice are on the validation set of the other corpus. Thus <br>it seems that the model trained on the merged corpus has more stable <br>results. It could be interesting to check this by decoding an entirely <br>new validation set, outside any of the initial corpora, with each of the <br>models. If the merged model is more stable, it should give better <br>results on a new corpus than any of the other models.</p><p>So, it is possible to conclude that <strong>combining the corpora gave a more performant and less specialized model</strong>. This is not very surprising, as the use of more data is likely to give better models, but it was still necessary to check it.</p><p>Another interesting detail is that the results of the merged corpus model are often close to those of Commonvoice (except on Voxforge, where it inherits the bias of the Voxforge train set). This phenomenon is also explainable: Since Commonvoice makes up a majority of the total data (677 of 1085 hours), it has more influence on the merged model as the other corpora.</p><p>As more data should help to train more accurate models, combining the corpora <strong>should give better results with the same tests</strong> as each of the individual corpora. Thus, we did run the same trainings as above (and an additional one using <strong>speaker adaptative training</strong>) and decoded every model each of the validation sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*RsGGfnuC45AkKyC3QwbzAg.png" /><figcaption><em>WERs of a model trained on all corpora</em></figcaption></figure><h4><strong>Conclusion</strong></h4><p>From the three large German speech corpora found online, it was possible to create an even larger merged corpus. The tests showed that it is <strong>necessary to take care of some bias</strong>, but the combined corpus still seems to be adapted to train even better German ASR models.</p><p>In the test described above, we used only models with a HMM-GMM (hidden Markov model — Gaussian mixture model) architecture with default hyper-parameters and a fixed language model (this and the dictionary came from <a href="https://sourceforge.net/%20projects/cmusphinx/files/Acoustic%20and%20Language%20Models/German/">CMUsphinx</a>).</p><p>Adapting the hyper-parameter or modifying the dictionary and the language model could already <strong>improve the high error rate</strong>. Moreover, these corpora and especially the merged corpus can also be used to train more sophisticated models, which use for example neural networks. With these techniques it should be <strong>possible to train high-performing German ASR models exclusively on freely available datasets</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fed0e2ed5e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/assembling-a-large-german-speech-corpus-7fed0e2ed5e">Assembling a large German speech corpus</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training of a speech recognition model for the Spanish language]]></title>
            <link>https://medium.com/linagoralabs/training-of-a-speech-recognition-model-for-the-spanish-language-8202e22093ae?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/8202e22093ae</guid>
            <category><![CDATA[kaldi]]></category>
            <category><![CDATA[linto]]></category>
            <category><![CDATA[spanish-model]]></category>
            <category><![CDATA[linagora]]></category>
            <category><![CDATA[speech-recognition]]></category>
            <dc:creator><![CDATA[Lea Baviere]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 12:39:36 GMT</pubDate>
            <atom:updated>2020-09-16T12:39:36.560Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This paper provides an overview of the development of a </em><strong><em>Speech Recognition Model in Spanish for LinTO at Linagora</em></strong><em>. In particular it shows how to make a training model for Automatic Speech Recognition (ASR).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*ByqyCmrS6X1SXZaX.jpeg" /></figure><h3>What is LinTO ?</h3><p><strong>LinTO is an Open Source toolbox</strong> designed to address professional and industry needs by enabling <strong>the development and integration of voice operated processes.</strong></p><p>If the client is an electricity supplier, for instance, Linagora will ensure that LinTO fully integrates vocabulary and language comprehension specific to electricity services.</p><p>The LinTO platform can be deployed, adapted or maintained either directly on your company’s premises or remotely as a service (SaaS). We also offer quality mentorship and software tailored to your needs.</p><p>Since it necessarily has access to private company data, LinTO certifies full information security and protection in conforming with GDPR.</p><p>Furthermore, in accordance with the principles of <strong>Open Source design</strong>, the source code is freely available, offers clean infrastructures and is GAFAM free.</p><p>LinTO is already available in French and English, and here we give an overview of the current development of the <strong>Automatic Speech Recognition model for the Spanish language.</strong></p><p>Throughout the project, <strong>we have used the platform Kaldi, which is a toolkit allowing the development of Automatic Speech Recognition (ASR) models from scratch</strong>, facilitating the learning and decoding parts of the implementation of the model.</p><h3>Discovering Kaldi</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*iy8bTgGDRIoYnzbJ.png" /></figure><p><a href="https://kaldi-asr.org/">Kaldi</a>, a generic toolkit for ASR</p><p>Understanding this toolkit starts with the different functions and methods it offers, before making the different examples available on the platform run. They are located on the “egs” folder when downloading Kaldi, and are displayed to help you understand how it works, with examples for different languages and datasets.</p><p>The word “training” quoted above refers here to Machine Learning. <strong>Machine learning means the ability for a computer to learn without being explicitly programmed to do so.</strong></p><p>The computer thus needs to be <strong>trained by giving it a lot of data, here the audio files</strong>, and the <strong>correct answers attached to it, here the text annotations.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*1yGIsKJRZd8E3ABY.jpeg" /></figure><p>This will help the machine connect the patterns from the data to the right answer. This is the “training” phase, where the machine receives “train” data.</p><p>Once it is properly trained, you can give the model validation data (“development” data), which is part of the training data you used, in order to tune the hyperparameters (i.e. the architecture) of the classifier.</p><p>Finally, you test your model by giving it “test” data, without text annotations, and the machine will predict the answers based on the training it received.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/0*Y88DY7W0e1zigrZP.png" /></figure><p>We split the speech corpora between Training, Validation and Test sets.</p><p>All of those steps constitute the acoustic model for speech recognition.</p><h3>Development of the acoustic model</h3><p>Developing an acoustic model requires the contribution of a lot of data for a better learning of the model.</p><p><strong>The more data is given, the more the model will learn, hence the more the more accurate the model will be.</strong></p><p>Given that, it means the first step of the ASR model is to look for audio tracks corpus online for the training set, and paying attention to the diversity and overall quality of the tracks.</p><p>Here is the corpus used for this particular model :</p><p><em>- </em><a href="https://voice.mozilla.org/fr/datasets"><strong><em>CommonVoice</em></strong></a><strong><em> </em></strong><em>: downloading of approximately </em><strong><em>148,000 audio tracks of roughly 5 seconds</em></strong><em>. The speakers are diversified, in age, sex and accent. The audio tracks come with a folder containing the information about the speakers, as well as the written transcription of each audio file.</em></p><p><em>- </em><a href="https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/"><strong><em>Caito</em></strong></a><em> : downloading of about </em><strong><em>59,000 audio tracks of 5 seconds as well</em></strong><em>, also containing a text file with the transcription, and the speakers’ information.</em></p><p><em>- </em><a href="http://www.voxforge.org/es/Downloads"><strong><em>Voxforge</em></strong></a><em> : downloading of approximately </em><strong><em>1000 audio files of 4 seconds</em></strong><em>, this time with 7 speakers with different caracteristics, each one of them saying the same 140 sentences.</em></p><p><em>- </em><a href="http://www.openslr.org/resources.php"><strong><em>OpenSLR LibriSpeech</em></strong></a><em> : downloading of about </em><strong><em>57,000 tracks of 5 seconds</em></strong><em>, with the same text files attached as the other 3.</em></p><p>These corpora give us a total of about <strong>340 hours of audio tracks for the model</strong>, allowing the training with approx. <strong>250,000 audio files and the test (or decoding) with approx. 13,000 audio files.</strong></p><p>The next step for the acoustic model is to clean the data.</p><p>For instance, Kaldi models work better with “.wav” files, meaning we had to convert the “.mp3” files of CommonVoice in that format.</p><p>Cleaning the data also means treating the text transcription of the audio tracks with a Python script. This script created allowed a “deep cleaning”: the removal of punctuation and uppercase letters, conversion of numbers to letters, segmentation of sentences at commas, periods, or other sentence punctuation, and the transformation of certain abbreviations such as “Dra.” in “doctora”. Spanish accents were preserved.</p><p>Once the data has been cleaned, we can move on to the training of the model.</p><p><a href="https://kaldi-asr.org/doc/kaldi_for_dummies.html">Kaldi <em>Tutorial for Dummies</em></a> allows us to easily initialize a model, with explanations and a basic run.sh script.</p><p>We first start with a small part of the data, i.e. a random sampling from CommonVoice, with 15,000 audio files for the training phase, allowing us to improve the run.sh file before training with all the data.</p><p>The discovery of different Kaldi examples allows us to learn about different sorts of training, for instance <em>monophone training </em>(learning to recognize a succession of phonemes), or <em>triphone training </em>(learning to recognize a succession of phonemes by taking into account the context of the beginning and the end of pronunciation of the phoneme).</p><p>For instance, the phoneme <em>/o/</em> is pronounced differently depending on the context for the two words “abrio” (= open) and “abrimos” (= we open).</p><p>We also learned in which order to use those training phases, as well as which alignement to use after each ones (the alignement is the projection between sets of data, given that the original data sets lie on a common manifold).</p><p>Once the acoustic model is done, and represents the relationship between an audio signal and the phonemes, we need to develop the language model, which enables us to assign a probability to a sequence of words.</p><h3>Development of the language model</h3><p><strong>Developing the language model, just as for the acoustic model, requires a research of massive text corpora</strong>, here in Spanish. The research must be used to provide the model with a diverse corpora in words, containing, as much as possible, <strong>a great variety of sentence structures in Spanish.</strong> This language contains about 100.000 to 150.000 words, according to the biggest Spanish dictionary. For this first part of the development of the language model, researches gave us :</p><p><em>- </em><a href="https://www.kaggle.com/rtatman/120-million-word-spanish-corpus"><strong><em>Corpus 120-million-words-spanish-corpus</em></strong></a><em> : about 104,000,000 words in 57 files.</em></p><p><em>- </em><strong><em>TEI_ES</em></strong><em> : about 7,000,000 words in 7034 files.</em></p><p><em>- </em><a href="https://www.gutenberg.org/browse/languages/es"><strong><em>Gutenberg</em></strong></a><em> : about 2,000,000 words in 18 files.</em></p><p><em>- </em><a href="http://opus.nlpl.eu/OpenSubtitles.php"><strong><em>Open Subtitles</em></strong></a><em> : about 5,000,000,000 words in 61 files.</em></p><p><em>- </em><a href="https://crscardellino.github.io/SBWCE/"><strong><em>Spanish_billion_words</em></strong></a><em> : about 803,000,000 words in 100 files.</em></p><p><em>- </em><strong><em>Corpus_coca</em></strong><em> : about 11,000,000 words in a single text file.</em></p><p>These files were established from corpus such as Wikipedia, movie subtitles or books transcribed in text files.</p><p>Before any adding to the Kaldi script, we needed to treat the text files we had just collected. Indeed, we had to make sure that all the text files have the same encoding (here utf-8), and that the cleaning could be done on several CPUs to optimize the cleaning process duration, but also to support in memory the size of the files. We used the same Python script used at the beginning of the acoustic model, when cleaning the text annotations, to treat the text corpus here. This deep cleaning of all the text files allowed us to have, <strong>as a finish corpus, a 30 Go text file, provided to the Kaldi model</strong> for it to learn the words.</p><p>In parallel with the language model, <strong>we also have to provide the model with a lexicon</strong>, allowing the detailing of each word, giving for instance for the word “abocar”, its prononciation “a b o k a r”. It is also necessary to provide a sort of dictionary, or vocabulary, listing as many spanish words as possible, focusing on the most common ones, so they can be familiar to the model when approaching the decoding phase.</p><p>Language models obtained using corpora retrieved online to allow better generalization to new data. By default, the language model used is biased on the text of the training audio data.</p><p>When in the decoding and learning phases, we can spot and remedy oversights of the text and audio cleaning scripts using SRILM tools (<a href="http://www.speech.sri.com/projects/srilm/">http://www.speech.sri.com/projects/srilm/</a>), which can help improve the model results.</p><p><strong>We measure the performance of language models with a metric called perplexity</strong> (you can refer to <a href="https://en.wikipedia.org/wiki/Perplexity).">https://en.wikipedia.org/wiki/Perplexity).</a> A perplexity that is too low indicates a biased, underperforming model, and a perplexity that is too high indicates a model that did not benefit from the learning phase.</p><h3>Achieved outcomes</h3><p><strong>The acoustic model matches sound waves with phonemes. The language model is used to match words to sentences while the lexicon matches phonemes to words.</strong> These three tools, properly trained and nested together,<strong> allow us to match sound waves to sequences of words, and thus to perform speech recognition.</strong></p><p><strong>We measure the performance of the speech recognition task with a metric called the Word Error Rate (or WER)</strong> (you can check <a href="https://medium.com/linagoralabs/tuning-of-parameters-for-decoding-in-automatic-speech-recognition-4bf4705488c6"><strong><em>https://medium.com/linagoralabs/tuning-of-parameters-for-decoding-in-automatic-speech-recognition-4bf4705488c6</em></strong></a> for more details).</p><p>The basic script of Kaldi allowed us to obtain a <em>WER</em> of approximately 70%, with only a triphone training (HMM-GMM), using 15.000 audio files from CommonVoice for training, and 100 for decoding, as well as the transcription of the training files as a model language. After developping the acoustic and language model, and by using all the audio files from different corpus combined, about 216.000 audio files for training, and 12000 for decoding, and the final corpora from the text files for the model language, <strong>we finally obtain a WER of approximately 20%, using an HMM-GMM model.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8202e22093ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/training-of-a-speech-recognition-model-for-the-spanish-language-8202e22093ae">Training of a speech recognition model for the Spanish language</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Next Word Prediction: A Complete Guide]]></title>
            <link>https://medium.com/linagoralabs/next-word-prediction-a-complete-guide-d2e69a7a09e6?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/d2e69a7a09e6</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[lstm]]></category>
            <dc:creator><![CDATA[Sonal Sannigrahi]]></dc:creator>
            <pubDate>Mon, 07 Sep 2020 08:53:05 GMT</pubDate>
            <atom:updated>2020-09-07T08:53:04.792Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*SmIdn8IwGXbsFJCL.jpg" /></figure><p><em>As part of my summer internship with Linagora’s R&amp;D team, I was tasked with developing a next word prediction and autocomplete system akin to that of Google’s Smart Compose. In this article, we look at everything necessary to get started with next word prediction.</em></p><p>Email continues to be one of the largest forms of communication, professionally as well as personally. It is estimated that almost 3.8 billion users send nearly 300 billion emails a day! Thus, for Linagora’s open source collaborative platform OpenPaaS, which includes an email system, improving user experience is a priority. With real time assisted writing, users are able to craft richer messages from scratch and have a smoother flow as compared to the static counterpart: complete automated replies.</p><p><strong>A real-time assisted writing system</strong></p><p>The general pipeline of an assisted writing system relies on an accurate and fast next word prediction model. It is crucial to consider several problems in regards to building an industrial language model that enhances user experience: time on inference, model compression, transfer learning to further personalise suggestions.</p><p>At the moment, these issues are addressed by the growing use of Deep Learning techniques for mobile keyboard predictions. Leading companies have already made the switch from statistical n-gram models to machine learning based systems deployed on mobile devices. However, with Deep Learning comes the baggage of extremely large model sizes unfit for on-device prediction. As a result, model compression is key to maintaining accuracy while not using too much space. Furthermore, mobile keyboards are also tasked with learning your personal texting habits in order to make predictions that are suited to your style rather than just a general language model . Let’s take a look at how it’s done!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*-qXf_QObEEzH4DndH8ovPA.png" /><figcaption>Next word predictions in Google’s Gboard</figcaption></figure><p><strong>Some useful training corpora</strong></p><p>In order to evaluate the different model architecture’s I will use to build my language model, it is crucial to have a solid benchmark evaluation set. For this task, I chose the Penn Tree Bank dataset which tests easily if your model is overfitting. Due to the small vocabulary size (roughly 10,000 unique words!), it is imperative to build a model that is well regularised. In addition to this, Enron’s email corpus was used to train on real email data to test predictions in the context of emails (with a much larger and richer vocabulary size of nearly 39,000 unique words).</p><p>My task was to build a bilingual model, in French and English, with intensions of generalising to other languages as well. For this, I also considered several widely available French texts. FrWac is a web archive of the whole .fr domain which is a great corpus to train on a diverse set of French sequences. For those with extensive GPU resources, the entire French wikipedia dump is also available online. With my code, I trained on a couple of short stories from Project Gutenberg (another great resource for textual data in multiple languages!)</p><p><strong>Model Architectures</strong></p><p>Now comes the fun part: language modelling. Generally speaking, approaches for text generation/prediction can be split into two categories: statistical based and learning based. In this article, we focus on the latter and take a deep dive into several recurrent neural network (RNN) variant architectures. For a next word prediction task, we want to build a word level language model as opposed to a character n-gram based approach however if we’re looking into completing the words along with predicting the next word then we would need to incorporate something known as beam search which relies on a character level approach. For this article, we will stick to a next word prediction without beam search.</p><p>At the time of writing this article, the models with the most success in language generation and prediction tasks are transformer based which exploit the idea of self attention. [Description]. However, these models are notoriously difficult to train without large amounts of training data as well as a good amount of GPU/TPU resources. Here we focus on the next best alternative: LSTM models.</p><p>An LSTM, Long Short Term Memory, model was first introduced in the late 90s by Hochreiter and Schmidhuber. Since then many advancements have been made using LSTM models and its applications are seen from areas including time series analysis to connected handwriting recognition. An LSTM network is a type of RNN which learns dependence on historic data for a sequence prediction task. What allows LSTMs to learn these historic dependencies are its feedback connections. For a common LSTM cell, we usually see an input gate, an output gate, and a forget gate. The weights of these gates control the flow of information in an LSTM model and thus are the parameters learnt during the training process.</p><p>Some variants of the LSTM model include a Convolution LSTM (or CNN-LSTM) and a Bidirectional LSTM (or Bi-LSTM). For our task, these variants correspond to different encodings of our input sequence. A CNN-LSTM model is typically used for image captioning tasks where the CNN portion is used to recognise features of the image and the LSTM is used to generate a suitable caption based on the features. Experimenting with using a CNN as the encoding layer yielded some interesting results however with the large number of parameters and even bigger model sizes, it was not fit for this task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*tHEGM70ovJig5LNZD0cswg.png" /><figcaption>Top: CNN-LSTM, Bottom: Bi-LSTM (credit to owners)</figcaption></figure><p>In “Recurrent Neural Network Regularisation” by Zaremba et al., they discuss a highly regularised LSTM model with added dropouts. This model is well suited for our task by meeting the requirements of both smaller model sizes as well as high performance. It is crucial in this era of Deep Leaning that there is consideration for a tradeoff between model size and performance. Being resource efficient should be the goal rather than improving accuracy by 0.01%. The model discussed in this paper balanced the trade off between complexity and accuracy quite well, as we will discuss in the next section.</p><h4>Implementation of the Chosen Model for Next Word Prediction</h4><p>Below is an implementation of this model using the Deep Learning library, PyTorch. While I have previously implemented this in Tensorflow as well, the model couldn’t converge well therefore I wouldn’t recommend Tensorflow particularly for this task. Although ideally there should be no difference!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4aa263645445c646b72c86ab417c87ba/href">https://medium.com/media/4aa263645445c646b72c86ab417c87ba/href</a></iframe><p><strong>Results</strong></p><p>So how did our final model perform on different text corpora? In order to evaluate the performance, we considered both qualitative and quantitative metrics. Qualitatively, it’s crucial to look at sentence structure (in cases where there isn’t an exact match) and context of the predictions. In the latter, we should ideally be able to see the benefit and power of LSTM based approached as compared to statistical methods. Quantitatively, we considered w<em>ord perplexity </em>(exponential of the loss) which allowed us to measure how confidently our model was predicting words, <em>ExactMatch@N </em>which shows at which level of probability (N=1 means the highest, N=2 means the second highest prediction, and so on) are we predicting the right word if at all, lastly we measured <em>precision and accuracy </em>to measure…as the name suggests… the accuracy of our model!</p><p>Our regularised LSTM performed quite well on the PTB corpus and managed to converge without overfitting, which was the primary goal. Upon increasing the input vocabulary size, by using a portion of the Enron corpus, we see even better results in terms of word level perplexity. Here are some quantitative results on both the PTB and Enron dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*EylcJzSs1-0VJj9GbZrTnQ.png" /><figcaption>Word level perplexity for Small regularised model (PTB) and for Medium regularised model (Enron)</figcaption></figure><p>Our model achieved a final test perplexity level of 117.03 on the PTB corpus with a small regularised model and a perplexity of 111.05 on the PTB corpus with a medium regularised model. As the PTB corpus was our benchmark measurement, we proceeded with the medium regularised model to train on the Enron corpus. Here we achieved a final test perplexity of 55.12 which corresponds to a test loss of 3.56 after training for 39 epochs. While in these two cases the convergence problem was solved, the same result was not met for the French text data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*eP3_b3NNoAdb044RxEKUsA.png" /><figcaption>Medium LSTM for French Text</figcaption></figure><p>As you can see above, while the perplexity for the training set is reducing the one for the validation set does not converge. This can be attributed to the training and validation sets having very different vocabulary or to the added complexity in french data with additional conjugations for verbs as compared to English. However, with more training data it is quite likely that the convergence will not be an issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*lWlhSo1YK9w1V-ARhQo4oA.png" /><figcaption>Some predictions with the Enron corpus</figcaption></figure><p>Looking at some qualitative results for the model trained on the Enron corpus, we see a respectable accuracy (ExactMatch@1) of nearly 44%. Considering a vocabulary of only 39,000 a 44% accuracy is a great result! This shows that the regularised LSTM model works well for the next word prediction task especially with smaller amounts of training data.</p><p><strong>How about using pre-trained models?</strong></p><p>Now that we have explored different model architectures, it’s also worth discussing the use of pre-trained language models for this task. Currently, language models like Open GPT by OpenAI are trained on large amounts of Wikipedia data making them highly nuanced and suited to text prediction tasks. However models like Open GPT are not easily specialised onto different corpora, in our case emails. Furthermore, due to the extremely large number of parameters, these pre-trained language models are not suited to be deployed on device and are also not suited to be trained on device. One last problem with Open GPT specifically is that at the moment it is a pre-trained <strong>English </strong>model which makes it unsuitable for our bilingual language modelling problem.</p><p>Another popular pre-trained language model is Google’s BERT, which uses the transformer architecture discussed earlier. Unlike Open GPT, BERT is indeed available in a multilingual format and specifically for French, researchers from Inria, Facebook AI, and Sorbonne Université recently released CamemBERT (trained on 138GB of French data) is quite an exciting advancement! BERT itself was designed for masked language modelling, inspired by the Cloze task. At the moment, BERT’s power is not understood very well. While BERT can be used for a next word prediction task by setting the mask as the last word, BERT is best suited to have a left and right sequence around the mask which makes full use of its bidirectional nature. In these pre-trained language models, the process of transfer learning is quite difficult and therefore making them a general language model rather than specific use case model. Furthermore, without the specialisation the concept of personalised language models (by training on device) are even harder to achieve. This is not to say that these pre-trained models are not excellent for a first trial, you’ll surely find some interesting results with them!</p><p><strong>Use on Inference</strong></p><p>So now that you’ve chosen your desired language model, how do you get it to make the predictions?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*h8Du1P_e4baL0rjjWHdG6g.png" /><figcaption>Prediction on Inference (taken from Trung Tran’s tutorial in Machine Talk)</figcaption></figure><p>The above diagram illustrates the inference process with an input sequence and output word(s). With our language model, for an input sequence of 6 works (let us label the words as 1,2,3,4,5,6) our model will output another set of 6 words (which should try to match 2,3,4,5,6, next word) so we choose the last word from the sequence with the highest probability as our predicted next work.</p><p>Here is how the prediction function could look for this problem:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d85930beb4c81702eac9dc59982773a/href">https://medium.com/media/2d85930beb4c81702eac9dc59982773a/href</a></iframe><p><strong>A Final Note on Model Compression</strong></p><p>Okay so by now our model is ready to be deployed, eager to start predicting your words! But do you really expect to deploy models of sizes larger than 200 Mb on devices like mobiles? How are we supposed to maintain model accuracy yet using an insignificant amount of space in terms of model size? In comes model compression!</p><p>Industrialising NLP is a challenging task and there have been many recent developments in this field. One key breakthrough was knowledge distillation. The idea is to have two models: a teacher and a student model. Our teacher model is the complete LSTM model with all its connections and feedback loops. Now our student model is a simple 1 layer feed forward network that learns from the last softmax layer of the teacher model. This allows the student model to imitate the teacher model’s predictions while being much shallower and therefore occupying a significantly lesser amount of space! This is a very simple explanation of model compression, if you’re interested in more I suggest the following video: <a href="https://www.youtube.com/watch?v=b3zf-JylUus">https://www.youtube.com/watch?v=b3zf-JylUus</a> . The picture below illustrates this concept quite well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*Sh0kLkNfgR19q5Rc0y0Yng.png" /><figcaption>Knowledge Distillation visualised</figcaption></figure><p><strong>Conclusion — What next?</strong></p><p>We’re finally at the end…or just the beginning? There are some exciting fronts still left to discover for me, starting from dealing with out-of-vocabulary and rare words to contextual encoding. Here we explored only the surface of a next word prediction model, the next steps would be work out model personalisation while still maintaining user privacy, include contractions as well as punctuations marks, and include a beam search method for autocomplete to make this model even more interactive.</p><p>I hope that after reading this article, you look at little word suggested by your phone keyboard a bit differently. The work that goes behind the scenes to make your user experience smooth and seamless is enormous and exciting! With current advancements in language understanding and generation, we are at a very rich time in technology and I personally can’t wait to see what is coming :)</p><p><em>Thank you to Zied Sellami for his supervision and support, and a huge thanks to the entire Linagora team for their warm welcome even while working remotely.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2e69a7a09e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/next-word-prediction-a-complete-guide-d2e69a7a09e6">Next Word Prediction: A Complete Guide</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data augmentation for Natural Language Understanding]]></title>
            <link>https://medium.com/linagoralabs/data-augmentation-for-natural-language-understanding-583eae42e268?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/583eae42e268</guid>
            <category><![CDATA[nlu]]></category>
            <category><![CDATA[data-augmentation]]></category>
            <category><![CDATA[linto]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Sonia Ratsiandavana]]></dc:creator>
            <pubDate>Wed, 04 Sep 2019 14:12:12 GMT</pubDate>
            <atom:updated>2019-09-04T15:21:38.050Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*k_K207wX0zMt6NX3k0VXHw.jpeg" /></figure><p><a href="https://linagora.com/">LINAGORA</a> is a French company, specializing in open source software. One of its current projects is the development of the open source smart vocal assistant <a href="https://linto.ai/">LinTO</a>. LinTO helps employees organize and carry out meetings: thanks to its Natural Language Understanding system, it can answer voice commands.</p><p>When training a natural language recognition engine, one of the major problems we have to tackle is data scarcity. A large training corpus is generally needed to improve these systems. The task of manually building a corpus is time-consuming and requires a lot of human resources. In this article, we describe the method we developed to create a data augmentation module, able to automatically create alternative commands from a small existing french corpus.</p><h4>The reference corpus</h4><p>For this project, the reference corpus we use is composed of several commands annoted manually beforehand in <a href="https://voyages-sncf-technologies.github.io/tock/fr/">Tock</a>, an open-source platform for bot building. It allows us to specify existing entities and attribute an intent to each command. An intent is the purpose of the user when formulating the command, and entities are the relevant key words that can be found in a given sentence (such as location, time ect.).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*q8lRlcjeCMkZY1JHwL0VLQ.png" /><figcaption>Command samples from the annotated reference corpus</figcaption></figure><h4>Command Types</h4><p>In our corpus, we identified two categories of commands:</p><ul><li>Action commands: this category covers sentences that require the execution of a certain action. For instance: “allume la lumière” or “peux-tu démarrer le chronomètre ?”.</li><li>Information commands: it is a category that contains sentences that query the information system. For instance: “quelle est la météo à Toulouse” or “donne-moi le niveau de pollution à Kyoto”.</li></ul><p>In the action command category, we can find two types of sentences:</p><ul><li>Imperative sentences: these include a verb in the imperative form (“stoppe le chronomètre”)</li><li>Questions: these are requests expressed in the form of questions (“peux-tu démarrer le chronomètre ?”)</li></ul><p>The information command category contains numerous subcategories:</p><ul><li>Close-ended questions: this kind of question can be answered with “yes” or “no” (“est-ce qu’il va pleuvoir ?”, “Paris est-elle polluée ?”)</li><li>Open-ended questions: these require a more developed answer (“qui sont les participants à la réunion ?”, “quel est le niveau de pollution à Paris ?”)</li><li>Nominal sentences: these are characterized by the absence of a verb (“actualité culturelle”, “météo à Toulouse”)</li><li>Imperative sentences: these include a verb but are different from imperative action commands (“donne-moi la date”, “donne-moi le titre de la réunion”). They always ask for a piece of information.</li></ul><p>According to the type of command identified by our module, the syntactic structures of the generated alternative sentences vary: for instance, for the action command “allume la lumière”, we cannot generate alternative commands such as “quelle est la lumière ?” or “donne-moi la lumière”. That would change the original meaning of the sentence. However, for the information command “météo à Toulouse”, we could say “quelle est la météo à Toulouse ?” or “donne-moi la météo à Toulouse”.</p><h4>Data Augmentation Module: a system based on a rule-based grammar</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Cn7SrUVlRNYZUg27IQtdw.jpeg" /><figcaption>The different components and resources of the data augmentation module</figcaption></figure><p>When processing a command, our system starts by identifying its type (action or information, and sub-categories of questions) by using certain clues in its syntactic structure. The system then retrieves the most important meaningful units in the sentence thanks to Natural Processing Tools like <a href="https://stanfordnlp.github.io/CoreNLP/">CoreNLP</a> and <a href="https://spacy.io/">Spacy Python</a>. In the action command “peux-tu démarrer le chronomètre”, it would keep the verb that describes the action to execute (“démarrer”) and the object of this action (“chronomètre”). In the information command “quelle est la température à Paris”, it would keep the noun (“température”) and the location (“Paris”). These key words will be kept or used to generate similar words.</p><p>The next step is therefore to generate new syntactically and semantically correct alternative commands.</p><p>To create our paraphrasing system we used a rule-based grammar: a system that allows us to create a set of rules in a formal language. These rules are used to generate strings that follow the syntax we set.</p><figure><img alt="Examples of rules to generate alternative “action” commands" src="https://cdn-images-1.medium.com/max/703/1*60c_09G0hoGscbtVpsfNYg.png" /><figcaption>Examples of rules to generate alternative “action” commands</figcaption></figure><p>Each rule is composed of several tags that match certain words or expressions that we want to find in the final generated sentence.</p><p>For instance, the “interro_infinitive” tag refers to a list of expressions that are commonly followed by an infinitive verb, whereas the “interro_subj” tag refers to expressions that are followed by verbs conjugated in the subjunctive tense. These kind of expressions are listed in lexicons that we manually created (“je veux que”, “c’est quoi”, “peux-tu” etc.). The requested expression is randomly chosen in the corresponding list.</p><p>Tags that mention verbs or nouns refer to the words retrieved in the input command, or similar words that we generate from them. Tags which begin with the expression “verb_tense” indicate that the verb has to be conjugated according to the specified tense (we currently use the imperative or the subjunctive tense). All of these tags are independent and can be reused to create other rules.</p><p>There is a specific set of rules for each command category: as we said before, each kind of command has its own syntactic structure and an action command will not be paraphrased in the same way as an information command. We can choose the number of alternative sentences we want to generate: for each generation, the new syntactic structure (represented by a rule) is also randomly chosen in the set of rules.</p><h4>Words and inflection</h4><p>Once the module determines the new syntactic structure it wants to apply, it needs to address the issues of conjugation and gender/number agreement. In morphology, the different forms that a word can take are called its “inflected” forms. A word’s inflected form depends on the morphological properties of the word it is attached to. The word that undergoes the transformation is called the “dependent” and the word that leads it is called the “head”.</p><p>For instance, if the input command is “quels sont les articles culturels du moment ?”, if we replace the noun “articles” (masculine-plural) by the synonym “nouvelles” (feminine-plural), the new correct syntactic structure would be “quelles sont les nouvelles culturelles du moment ?”.</p><p>For this task, we used <a href="http://redac.univ-tlse2.fr/lexiques/glawi.html">GLAWI</a>, an electronic dictionary created by Franck Sajours, Nabil Hathout and Basilio Calderone. Glawi is a structured and machine-readable version of the Wiktionnary. It contains lots of linguistically relevant information about words, including their inflected forms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*tGilJ8VkPq-nhOeyGMXYQA.png" /><figcaption>Inflected forms of the adjective “culturel” in GLAWI</figcaption></figure><p>We used this information to get the appropriate form of the dependent according to its head.</p><h4>Generation of equivalent words</h4><p>To bring variety to our generated commands, we cannot simply change the syntactic structure of the sentence and reuse the existing words of the input command. We had to generate words that could be used in the same context, such as synonyms, hyperonyms (a word whose meaning includes the meaning of more specific words → “voiture” is the hyperonym of “limousine”), hyponyms (the opposite → “limousine is the hyponym of “voiture”) or cohyponyms (“Toulouse” and “Paris” are both hyponyms of “ville”).</p><p>We used two methods to generate equivalent words:</p><ul><li>Equivalent words from the input file: we reused instances belonging to the same entity and non annotated words from the same intent to recombine them.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*gIo_kOv67-s-c9R424PRvQ.png" /><figcaption>Instances of the entity “action_stop”</figcaption></figure><p>For example, for the input command “désactive le chronomètre”, where “désactive” comes from the entity “action_stop”, we can retrieve the synonyms “stopper” and “arrêter”. We can also take the synonyms of the noun “chronomètre” in the other commands that belong to the intent “chrono”, such as “minuteur” and “chrono”.</p><ul><li>Equivalents from external tools: we used <a href="http://conceptnet.io/">ConceptNet</a>, a semantic network that gives synonyms, hyperonyms, and other terms related to an input word. We also used <a href="https://radimrehurek.com/gensim/models/fasttext.html">FastText</a>, an open-source library that uses words embeddings (a method that represents words as vectors to evaluate how close two words are to each other).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*IvvNSJf6bFTDki-ks-ebMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*B-e2E7rNfHUk-J5jBxmZ3A.png" /><figcaption>Examples of synonym and hyperonym of “vidéoconférence” in ConceptNet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*MSzpSXnR5KcirJwRSiQ8jQ.png" /><figcaption>Examples of equivalent words for “vidéoconférence” in FastText</figcaption></figure><p>For more accuracy, we put the words common to both systems at the top of the list (in the example here, “visioconférence” and “téléconférence”). We also used other methods to check if a word that only exists in one of the outputs could still be related to the input word.</p><h4>Some examples</h4><p>To conclude the presentation of our data augmentation module, here are a few examples of generated commands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*GbMZ4YHlbTaS_LnsoTFADg.png" /><figcaption>Samples of generated action commands</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*qbIuyQFFotIqTWUlrIbKPg.png" /><figcaption>Samples of generated information commands</figcaption></figure><h4>What’s next?</h4><p>French grammar is rich: we would need to add new rules to process other syntactic structures. To improve accuracy, the user will also have the ability to manually check similar words automatically generated by ConceptNet and FastText.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=583eae42e268" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/data-augmentation-for-natural-language-understanding-583eae42e268">Data augmentation for Natural Language Understanding</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning of parameters for decoding in automatic speech recognition]]></title>
            <link>https://medium.com/linagoralabs/tuning-of-parameters-for-decoding-in-automatic-speech-recognition-4bf4705488c6?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/4bf4705488c6</guid>
            <category><![CDATA[decoding]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[linagora]]></category>
            <dc:creator><![CDATA[Samir Tanfous]]></dc:creator>
            <pubDate>Tue, 29 Jan 2019 14:48:10 GMT</pubDate>
            <atom:updated>2019-01-29T14:48:10.344Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OwOMki7N3IeHpYky1D-5QA.png" /></figure><p>This article aims to show you the impact of hyper parameters on decoding performance for automatic speech recognition (ASR) systems.</p><p>I work as a Data Scientist in <a href="https://research.linagora.com/display/home/Home">LINAGORA’s R&amp;D Labs</a>, on <a href="https://linto.ai/">LinTO project</a>, which is a smart open-source voice based assistant for enterprises’ meetings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dr8NatEjxY-YNbGQ4v8fwQ.png" /><figcaption>LinTO, presented in the CES event at Las Vegas in 2019, January</figcaption></figure><h3>What is LinTO ?</h3><p>The LinTO project is developed by Linagora’s R&amp;D teams. This research project is a French PIA “Grands Défis du Numérique” funded by <a href="http://www.bpifrance.fr/">BPI</a>, and supported by <a href="https://www.gouvernement.fr/secretariat-general-pour-l-investissement-sgpi">SGPI</a> “Secrétariat Général pour l’Investissement” and <a href="https://www.entreprises.gouv.fr/">DGE</a> “Direction Générale des Entreprises”. It is part of Linagora’s commitment to:</p><ul><li><strong>promote Open Source</strong>, by offering free code source.</li><li><strong>respect users’ private data</strong>, by performing automatic speech recognition without sharing user data in Cloud infrastructures, unlike other personal assistant such as Alexa or Google Home.</li></ul><p>The LinTO device will have different functionalities:</p><ul><li><strong>Automatic Speech Recognition (ASR)</strong>: Transforming an audio signal of speech into text.</li><li><strong>Natural Language Understanding (NLU)</strong>: Detecting intentions where the objective will be to carry out concrete actions such as “Turn off the light in the room” or “Add a meeting to my agenda with Jean-Pierre on Thursday at 3pm”. In the last example, LinTO will be able to connect to the <a href="https://open-paas.org/">OpenPaaS</a> open source collaborative platform to add an appointment to the collaborator’s calendar.</li><li><strong>Quality of service</strong> with embedded audio acquisition and signal processing (denoising, echo cancellation…).</li><li><strong>Automatic summary of professional meetings:</strong> Producing a summary of a meeting with the keywords of the conversation.</li></ul><p>As far as my work is concerned, <strong>it focuses mainly on automatic speech recognition.</strong></p><p>One of the main objectives of LinTO is speech recognition. <strong>From a one-dimensional audio signal of speech, how can it be transcribed into text ?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*GgD2-8cQaLxkUR4T73WDiA.png" /><figcaption>Speech to Text</figcaption></figure><p>Automatic speech recognition is based on 2 main phases:</p><p>- <strong>Training</strong>, based on Machine Learning techniques. The goal is to build a model from audio files, for which the text transcription is already known.</p><p>- <strong>Decoding</strong>, where the model created in the previous phase is used to find the transcription on new audio files.</p><p>In this article, we will focus on <strong>decoding</strong>. It is a very active research area, as many metrics can be used to quantify the effectiveness of decoding.</p><p>On the one hand, there is the accuracy of transcription, which is measured using <a href="https://en.wikipedia.org/wiki/Word_error_rate">WER</a> (<em>word error rate</em>).</p><h3>What is WER ?</h3><p>It is a metric used to<em> determine a “distance” between the reference sentence and the sentence estimated by the model.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xaunKCs5UxFU9WdqGEx-xA.png" /><figcaption>WER formula</figcaption></figure><p>For instance, suppose that the reference sentence (the sentence you are trying to predict) is: <em>“this beautiful day”</em>, but the model predicts <em>“these day”</em>, then you have to make a substitution (replace <em>“these” </em>with <em>“this”</em>) and an insertion (add “beautiful”) to move from the sentence predicted by the model to the reference sentence.</p><p>The number of reference words being 3, the WER is therefore (2/3)*100% = 66.7% (very bad !).</p><p>On the other hand, the efficiency of decoding can be measured by calculating the time it takes a model associated with a CPU/GPU architecture to transcribe on a given set of sentences.</p><p>This last lever point is crucial for Linagora’s applications. The goal for LinTO is to provide the most accurate transcription possible, <strong>but with the fastest possible decoding time</strong>. We are exploring many state-of-the-art ways to improve the decoding speed. One of the most promising is <a href="https://www.danielpovey.com/files/2018_interspeech_gpu_wfst.pdf">to parallelize the decoding algorithm using GPUs.</a> However, the solutions we currently use run on CPUs.</p><p>In order to optimize the use of CPUs for decoding, it is possible to tune on <a href="https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)">hyperparameters</a> (<em>values set by a human before the algorithm is launched and not determined by the model</em>). One of the hyper-parameters that allows to significantly modify the decoding is the “<em>beam</em>”.</p><h3>What is the beam parameter ?</h3><p>The constitution of the learning model allows to build weighted <a href="https://en.wikipedia.org/wiki/Directed_graph">oriented graphs</a>, which we will go through in order to decode new sentences, to find the most probable paths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*2GDyW7OyVopx4WR5L4wPOQ.png" /><figcaption>Decoding graph at word level. The weights of each transition are not indicated, but the most likely sequence is the one in blue</figcaption></figure><p>The graph to explore is generally very large, the path of all possible paths is done in exponential time. A naive approach is to explore all possible paths, and choose the one that maximizes the probability that the sentence you decode is maximum, but this is computationally unacceptable.</p><p>Many strategies exist to improve the search through this graph. One of them, which is simple to understand and implement, is to select the most probable N wires when you reach a node, with N fixed along the graph. This value of N corresponds to the beam parameter defined above.<strong> </strong>The smaller the beam, the fewer the number of paths to be explored for decoding, and therefore the decoding time will be reduced, but it is possible that interesting solutions will not be explored.</p><p>We wanted to test the impact of this parameter. To do this, we trained a learning model on OpenSource data from the <a href="http://www.openslr.org/12">LibriSpeech corpus</a>, and performed decoding on test data from the same corpus. We are working on a server of 6 cores (2 threads per core). The framework we used to implement learning model and decoding part is <strong>Kaldi</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*Hm0A64o6MOtdOFdKeajsLA.png" /><figcaption><a href="http://kaldi-asr.org/">Kaldi</a>, open source toolkit for speech recognition</figcaption></figure><p>Kaldi is a generic toolkit used to create automatic speech recognition systems. It’s implemented in C++ and uses a powerful library to deal with graphs, called <a href="http://www.openfst.org/">OpenFST</a>.</p><h3>Algorithm implementations</h3><p>We performed a battery of tests using Kaldi to measure the WER and decoding time on different beam values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*MkokHVGKhDBwBM0tqyiZHA.png" /><figcaption>Evolution of decoding time and WER as a function of beam</figcaption></figure><p>We see on this graph that the decoding time increases linearly with the beam, while the WER decreases linearly until it reaches a threshold value. The choice of an interesting beam is a compromise between precision and speed. Here a value of 9 seems correct.</p><p>It is also possible to highlight the impact of the size of the models that are evaluated with the WER. In the graph below, we study the impact of the beam on the WER for different sizes of language models. The language model (LM) defines a probability distribution over a sequence of words. In the context of automatic speech recognition, the model that is mainly used is the <a href="https://en.wikipedia.org/wiki/N-gram">N-gram model</a>. It makes it possible to predict in terms of probability the appearance of a word conditioned by the previous N-1.</p><p>For example, a 3-gram model will give the probability of a word followed by a 2-word context. If it is well trained, he may, for instance, conclude that the word “<em>soon</em>” is more likely to appear after “<em>see you</em>” than the word “<em>sentence</em>”.</p><p>Thus, the language model is constructed by calculating the probabilities of occurrence of different groups of words, <em>for N generally ranging from 1 to 4.</em><br>However, these language models are <strong>too expensive in terms of storage space</strong>. They increase the size of the decoding graph, and therefore, the time required to provide a transcription. They can be reduced by, for instance, limiting the size of the vocabulary or by imposing a threshold to eliminate N-grams with a too low probability (called <em>pruning</em>), but these techniques therefore reduce the quality of transcription, as illustrated in the graph below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*FJtuvM1TlLKaqglakNaUVA.png" /><figcaption>Evolution of WER as a function of beam, with different sizes of language model</figcaption></figure><p>There are many other hyper-parameters on which it is possible to play to improve this compromise, and this parameterization engineering is very important for the needs.</p><h3>What’s next ?</h3><p>Most research in automatic speech recognition focuses on the implementation of new learning algorithms to improve WER, mainly based on neural networks. However, <strong>for commercial uses, it is important to remember that transcription speed is almost as important as its accuracy</strong>, imagine that a voice assistant takes several seconds or even several minutes to process an audio signal. Even with a perfect result, commercial applications would be limited. At Linagora, this is a crucial point for the customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bf4705488c6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/tuning-of-parameters-for-decoding-in-automatic-speech-recognition-4bf4705488c6">Tuning of parameters for decoding in automatic speech recognition</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Computing MFCCs voice recognition features on ARM systems]]></title>
            <link>https://medium.com/linagoralabs/computing-mfccs-voice-recognition-features-on-arm-systems-dae45f016eb6?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/dae45f016eb6</guid>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[mfcc]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[arm]]></category>
            <category><![CDATA[linto]]></category>
            <dc:creator><![CDATA[Antoine CHONÉ]]></dc:creator>
            <pubDate>Wed, 03 Oct 2018 14:04:56 GMT</pubDate>
            <atom:updated>2018-10-03T14:04:55.594Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*rsDnFKoVcpIwmOAsAaza2g.jpeg" /></figure><p>I worked as a trainee in <a href="https://linagora.com/"><strong>Linagora’s</strong></a> <a href="https://research.linagora.com/display/home/Home">R&amp;D departement</a> on the <a href="https://linto.ai/"><strong>LinTO</strong></a> project. Do you know LinTO? If not, you should! LinTO is a smart open-source voice based assistant for enterprises’ meetings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*JWv0NYCDsJqBNXRm4gq5yg.jpeg" /><figcaption>Say “Hey” to LinTO !</figcaption></figure><p>LinTO research project is a French PIA “Grands Défis du Numérique” funded by <a href="http://www.bpifrance.fr/">BPI</a>, and supported by <a href="https://www.gouvernement.fr/secretariat-general-pour-l-investissement-sgpi">SGPI</a> “Secrétariat Général pour l’Investissement” and <a href="https://www.entreprises.gouv.fr/">DGE</a> “Direction Générale des Entreprises”. It aims at supporting the development of the LinTO open-source voice assistance for enterprises. LinTO is based on speech recognition. <a href="https://www.lifewire.com/what-is-speech-recognition-3426721"><strong>Speech recognition</strong></a> is a technology that allows spoken input into systems. You talk to your computer, phone or device and it uses what you said as input to trigger some action. That’s what we call Natural Language Processing (<a href="https://en.wikipedia.org/wiki/Natural_language_processing"><strong>NLP</strong></a>). <a href="http://kaldi-asr.org/"><strong>Kaldi</strong></a><strong>,</strong> which is a quite known open toolkit, is currently used at Linagora to implement Speech-to-Text algorithms.</p><h3>Communicate with the machine!</h3><p>It’s easy for humans to communicate through speech when they speak the same language, but it’s really hard for a computer. The only relevant thing in communication is the real meaning, this is what we call the<strong> Linguistic Content</strong>. We need to extract features to eliminate parasitic audio components like accents, pronunciations, emotions of speakers, or background noises. The features extraction is the first step of automatic speech recognition systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BndwsZhpRaea3Ul62M1G4Q.png" /></figure><p>Humans can generate sounds which are filtered by the shape of the vocal tract including the tongue, teeth etc. This shape determines what sound comes out. If we are able to determine the shape, we should be able to get an accurate representation of the phoneme being produced. That’s the job of MFCCs.</p><h3>MFCCs, What’s this?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*QXvIzTd7wJxXzwIoYf3hYQ.gif" /><figcaption>MFFCs? I’m as confused as Travolta</figcaption></figure><p>MFCCs means Mel Frequency Cepstral Coefficients which are the most widely used features in speech recognition. <a href="http://www.cs.tut.fi/~sgn14006/PDF2015/S04-MFCC.pdf">MFCC</a> coefficients model the spectral energy distribution in a perceptually meaningful way.</p><h4>Theoretical working</h4><p>Step 1: <em>Cut the signal into several windows that intersect each other. This is called sliding windows, and not those of your kitchen ...</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*lIfxq0gcFcRI-rgVwYxi9w.png" /></figure><p>Step 2: <em>In order to reduce the spectral distortion, it is necessary to apply a specific window to the signal. There are many </em>types of windows <em>but the most known are Hamming, Hann (also called Hanning) and Blackman.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1X4yKLbjF5H8i6tr_yW86w.png" /><figcaption><a href="https://fr.wikipedia.org/wiki/Fen%C3%AAtrage">Temporal shapes of some windows.</a></figcaption></figure><p>Step 3: <em>Apply the </em><strong><em>FFT (Fast Fourier Transform)</em></strong><em> to the window to bring out the magnitude, so we get the </em><strong><em>spectrum</em></strong><em>. A fast Fourier transform (FFT) is an algorithm that samples a signal over a period of time (or space) and divides it into its frequency components. The frequency</em><strong><em> spectrum</em></strong><em> of a signal is the distribution of the amplitudes and phases of each frequency component against frequency.</em></p><p>Step 4: <em>Then we move on to Mel’s scale. This is a psychoacoustic scale of pitches of sounds, in the sense of their identification between bass and treble, which unity is mel. Mel is related to Hertz (Hz), the unit of measurement of the International System for Frequencies, by a relationship based on human hearing. It is a frequency scale closer to what the human ear is actually able to capture. The transfer formula is rather simple:</em></p><p><em>m</em> = 2595<em>log</em>(1+<em>f</em>/700)</p><p><em>To simulate the human ear, it is necessary to go through a Filter Bank, a filter for each frequency. These filters have a triangular bandwidth response.</em></p><p>Step 5: <em>Finally, we work on the </em><strong><em>cepstrum</em></strong><em>. We convert the logarithmic spectrum of Mel’s scale into time’s scale by using the DCT (Discrete Cosine Transform). A cepstrum is the result of taking the inverse transform of the logarithm of the estimated spectrum of a signal. The name “cepstrum” was derived by reversing the first four letters of “spectrum”.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObZV1Ay9CTH4YJPySIPpWA.png" /><figcaption>MFCCs extraction algorithm</figcaption></figure><h3>Design of MFCCs extraction</h3><p>The extraction of MFCCs can be done in many different ways. I implemented the mathematical operations sequences used in <strong>Kaldi</strong> toolkit.</p><p>The particular modification of this algorithm is the use of the Povey window (named by Kaldi’s owner) based on Hamming window. The other steps are the same as those presented in the scientific literature.</p><p>The particular point to note about MFCCs compute is the amount of data required, which is drastically reduced. Indeed, if we take windows of 25 ms sampled at 16000 Hz, we obtain 400 samples. These samples will be “transformed” into MFCCs that represent only 13 or 40 values in most cases. The amount of data to be stored and/or exchanged is at least reduced by a factor of ten.</p><h3>Embedded context</h3><p>Everything becomes complicated in an embedded context. In particular, we do not have as much available hardware resources as in a classic computer. As I worked with a Raspberry Pi 3, I worked on an ARM architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*-68UMtjba6FZphFz6iG5Vw.gif" /></figure><p>There are many complicated mathematical formulas to apply in order to obtain our important data. In a real-time context, it is important to verify that this process consumes less CPU time than “real time”. To be more explicit, if you get audio data with a sample rate of 16000 Hz, then you get 16000 samples per second. So, the maximum time limit to work on real-time is to compute 16000 samples in less than one second.</p><p>The aim of this algorithm is to perform these calculations as quickly as possible using the least resources possible. This is only possible by optimizing what we have, including hardware architecture.</p><h4>NEON Technologies</h4><blockquote><a href="https://developer.arm.com/technologies/neon">Arm NEON technology</a> is an advanced SIMD (single instruction multiple data) architecture extension for the Arm Cortex-A series and Cortex-R52 processors. NEON technology is intended to improve the multimedia user experience by accelerating audio and video encoding/decoding, user interface, 2D/3D graphics or gaming. NEON can also accelerate signal processing algorithms and functions to speed up applications such as audio and video processing, voice and facial recognition, computer vision and deep learning.</blockquote><p>That’s exactly what we need ! So we are going to use this implementation to boost up our algorithm. Now, we need a library to use this.</p><blockquote><a href="https://projectne10.github.io/Ne10/"><em>Ne10</em></a> is a library of common, useful functions that have been heavily optimised for Arm-based CPUs equipped with <a href="https://www.arm.com/products/processors/technologies/neon.php">NEON</a> SIMD capabilities. It provides consistent, well-tested behaviour, allowing for painless integration into a wide variety of applications via static or dynamic linking.</blockquote><p>That looks great! Here we go! I used this library to support my code. Speaking of code, I don’t need to remind you that we do not choose a random language to write embedded code on target, right? Asfar as as I am concerned, I wrote mine using C++, as it gives us speed and flexibility of C and benefits of object-oriented programming. In sum, it looks like a good choice.</p><p>You can find the NE10 documentation <a href="http://projectne10.github.io/Ne10/doc/modules.html">here</a>. The important functions to use are the <em>Real-to-Complex FFT</em> to process Fast Fourier Transform, and <em>Matrix-Vector Multiplication</em> to process Mel-Filter Bank.</p><h3>Algorithm implementations</h3><p>This algorithm will be used in <a href="https://linto.ai/">LinTO</a> prototype.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FKgEBR_k5e7RRZURhWMWnQ.png" /></figure><p>As you can see, during a full request, the CPU load could raise near 50% of 4 cores. In the LinTO prototype, this new implementation has reduced the CPU load from 45% to 22%! It is possible thanks to the use of a more adapted language, optimized libraries for a given architecture and the computing power of the DSP (Digital Signal Processor).</p><p>Thank you for your reading! I hope you enjoyed it.</p><h3>What’s next?</h3><p>Stay tuned! Our finally tuned algorithm will soon be released as Open Source. Take a look on the official <a href="https://github.com/linto-ai">LinTO’s github</a>, you can already find some voice recognition software there!</p><p>Coming next, a specific LinTO development card will be available as an Open-Hardware component.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dae45f016eb6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/computing-mfccs-voice-recognition-features-on-arm-systems-dae45f016eb6">Computing MFCCs voice recognition features on ARM systems</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ontology-based Meeting Intents Detection for Automatic Email Answering]]></title>
            <link>https://medium.com/linagoralabs/ontology-based-meeting-intents-detection-for-automatic-email-answering-7990e4a1e22?source=rss----e3b60ed6b7f9---4</link>
            <guid isPermaLink="false">https://medium.com/p/7990e4a1e22</guid>
            <category><![CDATA[intents-detection]]></category>
            <category><![CDATA[automatic-email-answering]]></category>
            <category><![CDATA[smart-reply]]></category>
            <category><![CDATA[openpaas]]></category>
            <category><![CDATA[ontology]]></category>
            <dc:creator><![CDATA[Manon Cassier]]></dc:creator>
            <pubDate>Wed, 05 Sep 2018 15:49:13 GMT</pubDate>
            <atom:updated>2018-09-06T09:25:01.188Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4osNXD_y3RZ8wQQmiXyB0g.png" /></figure><p><em>The work presented here aims to improve the </em><a href="https://linagora.com/"><em>Linagora</em></a><em>’s </em><a href="https://open-paas.org/"><em>OpenPaaS</em></a><em> open source collaborative platform for businesses — which includes among others a mailbox, a corporate social network and a shared calendar. This study focuses on automatic answering to appointment emails in French language.</em></p><p>Companies have many productivity tools that offer varied possibilities for remote communication. These tools allow to easily manage different projects that involve different partners at the same time. Among them, email still stands as a tool of choice with more than the half of the Mondial Population for users according to the <a href="https://www.radicati.com/?download=email-statistics-report-2017-2021">Radicati Group 2017 Report</a>. This same report also estimates that an average employee receives 88 emails a day and sends 34. Email would represent between 5 and 10 hours of an employee’s time over a month.</p><p>But a too massive use of emails can heavily impact work productivity by causing information overload among employees who receive too much to manage them effectively. Moreover, the Radicati Group studies reveal that 43% of French employees say that they are interrupted at least once every ten minutes and 31 % admit to being distracted in their work.</p><h4>An email processing assistant</h4><p>Considering this problem of email management, our goal is to propose an email assistant tool that would be able to help user by automatically prioritizing emails, sending notifications when urgent emails are received and generating answers. This article will focus on the latter objective.</p><p>Some tools already exist for automatic email answering like <a href="https://ai.google/research/pubs/pub45189">Google’s Gmail Smart Reply</a> or Outlook’s Message Management Assistant. But these tools only provide short answers that not always fit for professional use. Moreover, they are not open source. Our goal is to propose an approach and a tool to provide the most complete answers possible to avoid the recipient having to request further information in a new email.</p><p>Our tool is able to analyze incoming emails and decide which one requires a response and then retrieve relevant items to be able to generate an adequate answer. This response would then be submitted to the user’s validation or modification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*rCNjGExsXJoGHFzI7NyL1Q.png" /><figcaption>The attended result for the automatic email answering tool</figcaption></figure><p>This task needs to address two aspects: the email classification — to find those that require an answer — and the answer generation according to the detected user intent (i.e. the action he wants to perform or the purpose he wants to achieve by writing a sentence).</p><p>For instance, an email that only contains advertising does not require an answer. However, an email that contains a request for information about an appointment requires one. More specifically, it requests an answer that contains information about an appointment.</p><h4>Ontology-based intent detection</h4><p>Our approach is based on an ontology. An ontology describes and represents an area of knowledge about a particular domain. It is a semantic graph in which nodes represent concepts (each concept can have one or more properties) and relationships between them. There are two kinds of relationships: hierarchical relations that model subsumption links between concepts (e.g. the cat concept belongs to the animal concept) and non-hierarchical links (e.g. zebra can eat leaf). To formalize the ontology, we used the OWL language (Web Ontology Language) proposed by the World Wide Web Consortium’s (W3C) and widely used in the Semantic Web domain. We built the ontology with the <a href="https://protege.stanford.edu/">Protégé</a> tool — that support the OWL language — to describe <em>meeting</em> emails from a large corpus of business emails.</p><p>This ontology uses the formalization of <a href="https://framenet.icsi.berkeley.edu/fndrupal/">FrameNet </a>— which decomposes semantic frames into <em>frame elements</em> (FEs) that are themselves evoked by the presence of words called <em>lexical units </em>(LUs). These frames are only checked if the frame elements are present in a sentence. For example, in case of proposing an appointment by email, the semantic frame “appointment proposal” in the sentence “Je vous propose de faire une réunion jeudi prochain” (I propose to do a meeting next Thursday) can be detected thanks to the lexical units “propose” and “réunion” and the frame elements SPEAKER (i.e. “Je”), PROPOSITION (i.e. “propose”) and TIME (i.e. “jeudi prochain”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*ARlIAwpMmNcKQqDdSWd01Q.png" /><figcaption>The <a href="https://framenet.icsi.berkeley.edu/fndrupal/">FrameNet </a>formalization</figcaption></figure><p>Each concept of the ontology represents a different <em>meeting</em> intent with frame elements and lexical units that allow to detect them. The ontology contains 18 intent concepts grouped into 3 main concepts representing types of intents: <em>request</em>, <em>proposal</em> and <em>notification</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*rWy2eqQKKFTo3EThLBOH-A.png" /><figcaption>Concepts of the meeting intents ontology</figcaption></figure><h4>The tool : how to detect intents in emails?</h4><p>The smart reply feature is developed in Java. For each email, an annotation module annotates the email text with the LUs and FEs of the ontology concepts. This module outputs the three most likely intents for each sentence of the email. It retrieves the ontology annotations and projects them on each sentence of the email text to decide whether to assign them an appointment intention. This module is only based on Open Source APIs like <a href="https://stanfordnlp.github.io/CoreNLP/">Core NLP</a> (for tokenization, POS tagging and detection of named entities), Duckling Facebook (to detect named entities that contain numbers), <a href="https://snowballstem.org/">Snowball Stemmer</a> (to make the matching between LUs and words of the text easier) and <a href="https://github.com/apache/jena">Jena </a>(to read the OWL format).</p><p>The figure below illustrates the order in which APIs are used to project ontology’s elements onto the text and to detect intents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*4hQc7D0bIV2Q8XxxIFP1Gg.png" /><figcaption>API usage steps</figcaption></figure><h4>How to generate answers for emails?</h4><p>Each intent concept is associated to several answers patterns that are specified in a separated concept of the ontology. For example, the <em>appointment reminder</em> concept is associated to 2 answers canvas representing the <em>appointment changing request</em> and the <em>appointment confirmation</em>. Thus, when the user receives an email containing the sentence “Je vous rappelle que l’entretien est prévu mardi à 15h”, he can indicate that he will be present or ask to postpone the appointment if the date does not suit him.</p><p>The generated answers are adaptable according to the content of the email thanks to the syntax of the canvas that allow retrieving relevant elements in the text. For instance, if the incoming email contains the term “réunion” instead of “rendez-vous”, the answer proposed to the user must contain the same term. The figure below shows the answering canvas syntax for an <em>appointment changing request</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*32x46LWZ9Z8nU_b5AsZKZw.png" /><figcaption>Sample answer canvas from the ontology edited with <a href="https://protege.stanford.edu/">Protégé</a></figcaption></figure><p>When the user selects “Demander déplacement d’un rendez-vous” to answer to the precedent <em>appointment reminder</em>, he has access to a pre-filled email containing the sentences “Je ne serai malheureusement pas disponible pour ce créneau. Pouvons-nous trouver un autre créneau pour l’entretien ?” (Unfortunately, I will not be available for this slot. Can we find another niche for the interview ?). The tag between braces retrieves the occurrence of lexical units which makes it possible to deduce that the sentence contains an appointment intent (i.e. “entretien”) by adapting its determinant in accordance with its gender.</p><p>A greeting is automatically added at the beginning of each response. This is adapted according to the time of writing the new email and according to the recipient. For example, if the user decides to use our answers between <br>4 pm and midnight, the answer will begin with “Bonsoir” (Good evening) instead of “Bonjour” (Good morning / Good afternoon).</p><p>To adapt the answer to the person, the system considers the recipient’s email address — which often contains the name of the company in which he works. If the recipient works in the user’s company, the system simply inserts the greeting and the person’s first name (1).</p><p>If not, it checks a first name list to identify if it is a man or a woman. Depending on what it finds, it inserts “Monsieur + NAME” or “Madame + NAME” (2). If the first name is ambiguous or absent from the first name list, the system keeps “FIRST NAME + NAME” (3).</p><p>Here is an illustration of each case knowing that the user works at Linagora:</p><p>(1) Writing at 7 pm for benjamin.dupont@linagora.com <br>→ “Bonsoir Benjamin”</p><p>(2) Writing at 10 am for marie.dupont@something.com <br>→ “Bonjour Madame Dupont”</p><p>(3) Writing at 11 pm for france_dupont@something.com <br>→ “Bonsoir France Dupont”</p><h4>Integrating the smart reply feature into OpenPaaS</h4><p>The two modules described above are integrated to the OpenPaaS UnifiedInbox module. The global system finally has four main modules :</p><ol><li>An <strong>ontology manager </strong>that parses the ontology with the Apache Jena API to exploit the annotations it contains.</li><li>The <strong>intent detector</strong> that extracts the intents from the text by computing a relevance score for each intent concept of the ontology.</li><li>The <strong>answering generator</strong> that generates answers for each detected intent.</li><li>A <strong>Web Service</strong> that returns the three first answers in a JSON format.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*UGMUnhxDoDxy402s4acnHA.png" /><figcaption>The automatic answering system</figcaption></figure><p>The OpenPaaS UnifiedInbox module communicates with our answers suggestion service through a Web Service called by a REST API. For each email, the Inbox module sends a JSON request to the Web Service with the text of the email and the associated metadata. The system analyzes the request, detects intents in the text and returns a JSON response with the suggestions of answers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*5Tky130K3WV5dd50HhorXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*PE0C7FOnK0vDZRoH2rglYw.png" /><figcaption>Integration of the automatic answering system in OpenPaaS</figcaption></figure><p>A request is generally processed in less than 150ms. By selecting the appropriate button, the user has access to a pre-written response that can be modified, completed or simply sent. The sending must necessarily be validated by the user who can change his mind and choose another answer if the first does not suit him.</p><h4><strong>Evaluating the system</strong></h4><p>We evaluated our system according to two criteria: its performance in intent extraction and the relevance of the answers it proposes.</p><p>We first compared the performance of our system for the intent detection against machine learning models trained with the corpus we used to construct the ontology. The different systems were tested on another corpus of corporate emails containing (or not) various types of meeting intents. Our system presents far better results than the machine learning based models with an f-score of 69% against 24%.</p><p>We then compared the relevance of the answers proposed by our system and by Google’s Smart-Reply for 15 emails through a questionnaire distributed to 26 Linagora employees. Our answers are preferred for 8 emails out of 15 with an agreement of more than 75% for 5 emails against 7 emails out of 15 for the Google’s system — with an agreement of more than 75% for only 2 emails. The answers proposed by our system are selected by at least one annotator in all cases.</p><p>These experiences show that our approach allows providing relevant and acceptable answers that can fully compete with those proposed by the Google’s System. Moreover, our answers seem to be preferred for cases where the answer requires mentioning specific elements such as availabilities or precisions about an appointment.</p><h4>And what’s next?</h4><p>We are planning to link the answering mechanism to the OpenPaaS calendar to automatically check availabilities and generate answers depending on the user’s constraints. We also plan to detect and answer to other intents as administrative workflows (for instance, vacation requests or mission orders).</p><p>Keep in touch with the Linagora labs team, we will let you know about all of our working progress…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7990e4a1e22" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linagoralabs/ontology-based-meeting-intents-detection-for-automatic-email-answering-7990e4a1e22">Ontology-based Meeting Intents Detection for Automatic Email Answering</a> was originally published in <a href="https://medium.com/linagoralabs">Linagora LABS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>