<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aniket Bhatnagar on Medium]]></title>
        <description><![CDATA[Stories by Aniket Bhatnagar on Medium]]></description>
        <link>https://medium.com/@aniket_52307?source=rss-4364e1c0cfb3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*aiDoHiNiNpHgHFX8.</url>
            <title>Stories by Aniket Bhatnagar on Medium</title>
            <link>https://medium.com/@aniket_52307?source=rss-4364e1c0cfb3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:06:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aniket_52307/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Speaker Diarization — The Squad Way]]></title>
            <link>https://medium.com/squad-engineering/speaker-diarization-the-squad-way-1e92a1c28091?source=rss-4364e1c0cfb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e92a1c28091</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[speaker-diarization]]></category>
            <dc:creator><![CDATA[Aniket Bhatnagar]]></dc:creator>
            <pubDate>Sun, 17 Feb 2019 03:05:38 GMT</pubDate>
            <atom:updated>2019-04-08T13:01:30.115Z</atom:updated>
            <content:encoded><![CDATA[<h3>Speaker Diarization — The Squad Way</h3><p>Originally published in <a href="https://hackernoon.com/speaker-diarization-the-squad-way-2205e0accbda">HackerNoon</a>.</p><p><a href="https://en.wikipedia.org/wiki/Speaker_diarisation">Speaker Diarization</a> aims to solve the problem of <strong><em>“Who Spoke When”</em></strong> in a multi-party audio recording.</p><p>At <a href="https://www.squadplatform.com/">Squad</a>, ML team is building an automated quality assurance engine for <a href="https://www.squadvoice.co/">SquadVoice</a>. During the quality check phase, a calling representative’s performance is scored on various quality parameters, such as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*xZhDTRKBSfTSpxKb.jpg" /></figure><ol><li>Whether the representative was rushing on the call or not,</li><li>Whether (s)he was rude to the lead at any time or not,</li><li>Whether (s)he used the proper language, and many more.</li></ol><p>Such quality checks for a call recording are guided by “<strong><em>what</em></strong>” was spoken by the calling representative and “<strong><em>how</em></strong>” the same content was spoken. Thus, we need to ensure that the call recording portions where the agent spoke are separated from the portions where the customer or lead spoke. This, was a challenging problem, as Squad’s cloud telephony partner for India, records calls only in monaural format. Monaural format stores both parties audio on a single channel as opposed to stereophonic format, where audio of caller would be stored on one channel and that of callee would be written on a different channel. Thus, as a prerequisite for the quality checks, a speaker diarization system was required. However, we had a more focused problem, since the number of speakers for our use case was fixed at two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*xaM6d2ywWid9saSlh_zm6w.jpeg" /></figure><h4>How to solve Speaker Diarization?</h4><p>The problem of speaker diarization is quite complex. To be honest, it is the toughest Machine Learning problem that I have worked on till date. This solution utilizes both supervised and unsupervised Machine Learning techniques. Also, it relies on a combination of both recent Deep Learning and conventional<a href="http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html"> Agglomerative clustering</a> models. So let’s get started.</p><p>Our speaker diarization solution is derived from the combination of<a href="https://www.youtube.com/watch?v=96b_weTZb2w&amp;index=33&amp;list=PLkDaE6sCZn6Gl29AoE31iwdVwSG-KnDzF"> One shot learning</a>, speaker recognition architecture used by the work:<a href="https://www.robots.ox.ac.uk/~vgg/publications/2017/Nagrani17/nagrani17.pdf"> “VoxCeleb: a large-scale speaker identification dataset</a>”, and speaker change detection algorithm explained in this work:<a href="http://herve.niderb.fr/download/pdfs/Bredin2017.pdf"> “TristouNet: Triplet loss for speaker turn embedding</a>”.</p><h4>Speaker differentiation pipeline</h4><p>We utilized three VGG-M based CNN architectures [with shared weights] to be trained using triplet loss optimization paradigm in order to learn (dis)similarity between audio recordings of same/ (different) speakers. We call this speaker differentiation pipeline.</p><p>To train a speaker differentiation pipeline, on each iteration of model training, a triplet of audio recordings is generated, represented by &lt;q_sound, p_sound, n_sound&gt;. Here q_sound represents anchor audio snippet, p_sound represents some other audio snippet of the same speaker and n_sound represents an audio snippet of a different speaker. Then, magnitude spectrogram for each audio in the triplet is computed and pushed to VGG-M based model described above. All three CNNs share the same weights throughout the optimization process. Lastly, embeddings from the last layer of the network represented by &lt;q_vec, p_vec, n_vec&gt;are pushed to the hinge loss layer defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*XQ5GqDrKej8RibhhTSa75A.png" /><figcaption>Loss function optimized for speaker differentiation</figcaption></figure><p>In above equation D(x, y) represents the Euclidean distance between two embedding vectors and “g” margin constant in the hinge loss function, is kept equal to 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3L4_xrGjd2EVLGzuymq5A.jpeg" /><figcaption>Speaker differentiation pipeline visualization</figcaption></figure><p><strong><em>Few gotchas to be careful of </em></strong>in this step:</p><ol><li>Ensure the input magnitude spectrograms are mean and variance normalized along the time axis. Intuitively, each human would have few frequencies which would be more prominent in their voice, which contributes to the unique identity of sound produced by their vocal chords.</li><li>This would ensure that energies corresponding to a frequency bin for different phonemes pronounced over a short time frame are smoothened.</li><li>Use <a href="https://keras.io/activations/#softplus"><strong><em>softplus</em></strong></a> activation function in the last fully connected layer, instead of Relu, if you use Euclidean distance as the distance metric, this is because square root (x) is non-differentiable at 0. Or optimize squared Euclidean distance if you wish to use Relu.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TjazOQwbMHFtpU5-zBI5og.png" /><figcaption>Base CNN model trained for speaker differentiation</figcaption></figure><p>One of the key benefits of using this model is that, it will give the same dimension embedding vectors for any length of audio input to the model. However, since the model was trained using batch gradient descent hence during training phase we fixed the audio recordings length at 3s.</p><h4>The dataset for Training Speaker Differentiation Pipeline:</h4><p>To train the pipeline, we utilized audio recordings from IVR mission calls that were placed through SquadVoice. We took call recordings of 100 speakers with 12.06 mins of audio per speaker on average, while each IVR recording being 2.41 mins on average.</p><p>Additionally, we performed preprocessing on these IVR recordings, like the removal of IVR voice and trimming silence from the end of call recording using <a href="http://sox.sourceforge.net/">sox</a>.</p><p>Next, we computed random non-overlapping audio snippets of length between (4.5,s 12.5s), from each audio recording. Thus, generating a total dataset of 8586 audio snippets,</p><p>Lastly, we took random 3s crop of audio recordings while passing them as input to the neural net on each iteration and ensured that both train and validation set audio snippets had non-overlapping speaker distribution. That is, there is no speaker whose audio snippet belonged to both train and validation split.</p><h4>Speaker Change Detection</h4><p>We applied the learnings from TristouNet work to find speaker change points using the model trained above. We maintained 2 sliding window pairs of left and right sliding windows having duration 3s and moved them at a very small stride of 0.1s. Then, for each left and right window pair, obtained their corresponding magnitude spectrograms and ran them through the CNN trained in step (1) to obtain embeddings for each left and right window pair. Afterward, we computed the Euclidean distance between both left and right embeddings, if the distance exceeds a particular threshold then that would represent a speaker change point. Example, for a 30s audio recording, we would have the following workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*wHCwkxNOv1DiabMVXGVHrw.jpeg" /><figcaption>Workflow for speaker change detection</figcaption></figure><h4>Agglomerative clustering for grouping the audio snippets</h4><p>We broke down the audio as per speaker change points. I.e. if there were 5 speaker change points detected then we split the audio into 6 audio recordings, calculated magnitude spectrograms for the obtained snippets and then computed speaker embeddings using the CNN trained earlier. Later ran <strong><em>Agglomerative clustering</em></strong> on top of these embeddings to group together audio snippets by their speaker.</p><h4>Experiment Results and Evaluation</h4><p>This way the complete diarization pipeline was built. When tested on the dataset of 100 call recordings. It had an average<a href="http://pyannote.github.io/pyannote-metrics/reference.html#diarization"> diarization error rate [confusion part]</a> of 12.23%, which is quite great as per industry standards. Additionally, the pipeline also performs well in presence of background noise. As examples, have added few sample diarized audio calls with their original versions. The attached recordings are mock calls following actual customer flow.</p><pre>+---------------+-------------+-------------+---------------+<br>| original call | Calling Rep |    Lead     | Confusion DER |<br>+---------------+-------------+-------------+---------------+<br>| <a href="https://drive.google.com/file/d/1kMW9sCcIt1exM_5s7AqA_RFRJ5CVNh_F/view?usp=sharing">call_1</a>        | <a href="https://drive.google.com/file/d/1BjaVDHHp7HkKNrvSgsptjmQUNOB3nMpt/view?usp=sharing">call_1_rep</a>  | <a href="https://drive.google.com/file/d/18gsvjKs-lgMfIYN-b-3jSFvel63LdFN5/view?usp=sharing">call_1_lead</a> |       5.5%    |<br>| <a href="https://drive.google.com/open?id=1OdyLZVw2-x5MbXEeyw7HT50XJp5iJOcW">call_2</a>        | <a href="https://drive.google.com/file/d/1kiRxkPZ22H0TtEJAz1-oX8DbLZIGR2Kg/view?usp=sharing">call_2_rep</a>  | <a href="https://drive.google.com/file/d/1Fu_dKunYk-GloTteK-Il-lBbnPhs9uDO/view?usp=sharing">call_2_lead</a> |       6.44%   |<br>| <a href="https://drive.google.com/file/d/1uFEi7TAx4Su3nub_YAVI6RrWLzXQ8wCD/view?usp=sharing">call_3</a>        | <a href="https://drive.google.com/file/d/1-awWVk62k347GtkjamOQ9CzvmvtKfTrh/view?usp=sharing">call_3_rep</a>  | <a href="https://drive.google.com/file/d/1bpniRacLzblKqT6PLGpCfJq3FCrA4DCL/view?usp=sharing">call_3_lead</a> |       6.26%   |<br>+---------------+-------------+-------------+---------------+</pre><h4>Acknowledgment</h4><p>I would like to thank <a href="https://www.linkedin.com/in/sanchit-aggarwal/">Sanchit Aggarwal</a>, for discussions and reviews of the pipeline design and results of speaker diarization, along with <a href="https://www.linkedin.com/in/pragyajswl/">Pragya Jaiswal</a> and <a href="https://www.linkedin.com/in/ved-vasu-sharma-34588ba2/">Ved Vasu Sharma</a> for design discussions and PR reviews while moving this pipeline to SquadAI codebase. Also would like to thank <a href="https://www.squadvoice.co/">Squad Voice</a> Operations team for their vital inputs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e92a1c28091" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/speaker-diarization-the-squad-way-1e92a1c28091">Speaker Diarization — The Squad Way</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speaker Diarization — The Squad Way]]></title>
            <link>https://medium.com/hackernoon/speaker-diarization-the-squad-way-2205e0accbda?source=rss-4364e1c0cfb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2205e0accbda</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[speaker-diarization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aniket Bhatnagar]]></dc:creator>
            <pubDate>Sat, 06 Oct 2018 06:25:02 GMT</pubDate>
            <atom:updated>2019-04-26T09:51:19.993Z</atom:updated>
            <content:encoded><![CDATA[<h3>Speaker Diarization — The Squad Way</h3><p><a href="https://en.wikipedia.org/wiki/Speaker_diarisation">Speaker Diarization</a> aims to solve the problem of <strong><em>“Who Spoke When”</em></strong> in a multi-party audio recording.</p><p>At <a href="https://www.squadplatform.com/">Squad</a>, ML team is building an automated quality assurance engine for <a href="https://www.squadvoice.co/">SquadVoice</a>. During the quality check phase, a calling representative’s performance is scored on various quality parameters, such as:</p><ol><li>Whether the representative was rushing on the call or not,</li><li>Whether (s)he was rude to the lead at any time or not,</li><li>Whether (s)he used the proper language, and many more.</li></ol><p>Such quality checks for a call recording are guided by “<strong><em>what</em></strong>” was spoken by the calling representative and “<strong><em>how</em></strong>” the same content was spoken. Thus, we need to ensure that the call recording portions where the agent spoke are separated from the portions where the customer or lead spoke. This, was a challenging problem, as Squad’s cloud telephony partner for India, records calls only in monaural format. Monaural format stores both parties audio on a single channel as opposed to stereophonic format, where audio of caller would be stored on one channel and that of callee would be written on a different channel. Thus, as a prerequisite for the quality checks, a speaker diarization system was required. However, we had a more focused problem, since the number of speakers for our use case was fixed at two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*xaM6d2ywWid9saSlh_zm6w.jpeg" /></figure><h4><strong>How to solve Speaker Diarization?</strong></h4><p>The problem of speaker diarization is quite complex. To be honest, it is the toughest <a href="https://hackernoon.com/tagged/machine-learning">Machine Learning</a> problem that I have worked on till date. This solution utilizes both supervised and unsupervised Machine Learning techniques. Also, it relies on a combination of both recent <a href="https://hackernoon.com/tagged/deep-learning">Deep Learning</a> and conventional<a href="http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html"> Agglomerative clustering</a> models. So let’s get started.</p><p>Our speaker diarization solution is derived from the combination of<a href="https://www.youtube.com/watch?v=96b_weTZb2w&amp;index=33&amp;list=PLkDaE6sCZn6Gl29AoE31iwdVwSG-KnDzF"> One shot learning</a>, speaker recognition architecture used by the work:<a href="https://www.robots.ox.ac.uk/~vgg/publications/2017/Nagrani17/nagrani17.pdf"> “VoxCeleb: a large-scale speaker identification dataset</a>”, and speaker change detection algorithm explained in this work:<a href="http://herve.niderb.fr/download/pdfs/Bredin2017.pdf"> “TristouNet: Triplet loss for speaker turn embedding</a>”.</p><h4><strong>Speaker differentiation pipeline</strong></h4><p>We utilized three VGG-M based CNN architectures [with shared weights] to be trained using triplet loss optimization paradigm in order to learn (dis)similarity between audio recordings of same/ (different) speakers. We call this speaker differentiation pipeline.</p><p>To train a speaker differentiation pipeline, on each iteration of model training, a triplet of audio recordings is generated, represented by &lt;q_sound, p_sound, n_sound&gt;. Here q_sound represents anchor audio snippet, p_sound represents some other audio snippet of the same speaker and n_sound represents an audio snippet of a different speaker. Then, magnitude spectrogram for each audio in the triplet is computed and pushed to VGG-M based model described above. All three CNNs share the same weights throughout the optimization process. Lastly, embeddings from the last layer of the network represented by &lt;q_vec, p_vec, n_vec&gt;are pushed to the hinge loss layer defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*XQ5GqDrKej8RibhhTSa75A.png" /><figcaption>Loss function optimized for speaker differentiation</figcaption></figure><p>In above equation D(x, y) represents the Euclidean distance between two embedding vectors and “g” margin constant in the hinge loss function, is kept equal to 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3L4_xrGjd2EVLGzuymq5A.jpeg" /><figcaption>Speaker differentiation pipeline visualization</figcaption></figure><p><strong><em>Few gotchas to be careful of </em></strong>in this step:</p><ol><li>Ensure the input magnitude spectrograms are mean and variance normalized along the time axis. Intuitively, each human would have few frequencies which would be more prominent in their voice, which contributes to the unique identity of sound produced by their vocal chords.</li><li>This would ensure that energies corresponding to a frequency bin for different phonemes pronounced over a short time frame are smoothened.</li><li>Use <a href="https://keras.io/activations/#softplus"><strong><em>softplus</em></strong></a> activation function in the last fully connected layer, instead of Relu, if you use Euclidean distance as the distance metric, this is because square root (x) is non-differentiable at 0. Or optimize squared Euclidean distance if you wish to use Relu.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TjazOQwbMHFtpU5-zBI5og.png" /><figcaption>Base CNN model trained for speaker differentiation</figcaption></figure><p>One of the key benefits of using this model is that, it will give the same dimension embedding vectors for any length of audio input to the model. However, since the model was trained using batch gradient descent hence during training phase we fixed the audio recordings length at 3s.</p><h4><strong>The dataset for Training Speaker Differentiation Pipeline:</strong></h4><p>To train the pipeline, we utilized audio recordings from IVR mission calls that were placed through SquadVoice. We took call recordings of 100 speakers with 12.06 mins of audio per speaker on average, while each IVR recording being 2.41 mins on average.</p><p>Additionally, we performed preprocessing on these IVR recordings, like the removal of IVR voice and trimming silence from the end of call recording using <a href="http://sox.sourceforge.net/">sox</a>.</p><p>Next, we computed random non-overlapping audio snippets of length between (4.5,s 12.5s), from each audio recording. Thus, generating a total dataset of 8586 audio snippets,</p><p>Lastly, we took random 3s crop of audio recordings while passing them as input to the neural net on each iteration and ensured that both train and validation set audio snippets had non-overlapping speaker distribution. That is, there is no speaker whose audio snippet belonged to both train and validation split.</p><h4><strong>Speaker Change Detection</strong></h4><p>We applied the learnings from TristouNet work to find speaker change points using the model trained above. We maintained 2 sliding window pairs of left and right sliding windows having duration 3s and moved them at a very small stride of 0.1s. Then, for each left and right window pair, obtained their corresponding magnitude spectrograms and ran them through the CNN trained in step (1) to obtain embeddings for each left and right window pair. Afterward, we computed the Euclidean distance between both left and right embeddings, if the distance exceeds a particular threshold then that would represent a speaker change point. Example, for a 30s audio recording, we would have the following workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*wHCwkxNOv1DiabMVXGVHrw.jpeg" /><figcaption>Workflow for speaker change detection</figcaption></figure><h4><strong>Agglomerative clustering for grouping the audio snippets</strong></h4><p>We broke down the audio as per speaker change points. I.e. if there were 5 speaker change points detected then we split the audio into 6 audio recordings, calculated magnitude spectrograms for the obtained snippets and then computed speaker embeddings using the CNN trained earlier. Later ran <strong><em>Agglomerative clustering</em></strong> on top of these embeddings to group together audio snippets by their speaker.</p><h4><strong>Experiment Results and Evaluation</strong></h4><p>This way the complete diarization pipeline was built. When tested on the dataset of 100 call recordings. It had an average<a href="http://pyannote.github.io/pyannote-metrics/reference.html#diarization"> diarization error rate [confusion part]</a> of 12.23%, which is quite great as per industry standards. Additionally, the pipeline also performs well in presence of background noise. As examples, have added few sample diarized audio calls with their original versions. The attached recordings are mock calls following actual customer flow.</p><pre>+---------------+-------------+-------------+---------------+<br>| original call | Calling Rep |    Lead     | Confusion DER |<br>+---------------+-------------+-------------+---------------+<br>| <a href="https://drive.google.com/file/d/1kMW9sCcIt1exM_5s7AqA_RFRJ5CVNh_F/view?usp=sharing">call_1</a>        | <a href="https://drive.google.com/file/d/1BjaVDHHp7HkKNrvSgsptjmQUNOB3nMpt/view?usp=sharing">call_1_rep</a>  | <a href="https://drive.google.com/file/d/18gsvjKs-lgMfIYN-b-3jSFvel63LdFN5/view?usp=sharing">call_1_lead</a> |       5.5%    |<br>| <a href="https://drive.google.com/open?id=1OdyLZVw2-x5MbXEeyw7HT50XJp5iJOcW">call_2</a>        | <a href="https://drive.google.com/file/d/1kiRxkPZ22H0TtEJAz1-oX8DbLZIGR2Kg/view?usp=sharing">call_2_rep</a>  | <a href="https://drive.google.com/file/d/1Fu_dKunYk-GloTteK-Il-lBbnPhs9uDO/view?usp=sharing">call_2_lead</a> |       6.44%   |<br>| <a href="https://drive.google.com/file/d/1uFEi7TAx4Su3nub_YAVI6RrWLzXQ8wCD/view?usp=sharing">call_3</a>        | <a href="https://drive.google.com/file/d/1-awWVk62k347GtkjamOQ9CzvmvtKfTrh/view?usp=sharing">call_3_rep</a>  | <a href="https://drive.google.com/file/d/1bpniRacLzblKqT6PLGpCfJq3FCrA4DCL/view?usp=sharing">call_3_lead</a> |       6.26%   |<br>+---------------+-------------+-------------+---------------+</pre><h4>Acknowledgment</h4><p>I would like to thank <a href="https://www.linkedin.com/in/sanchit-aggarwal/">Sanchit Aggarwal</a>, for discussions and reviews of the pipeline design and results of speaker diarization, along with <a href="https://www.linkedin.com/in/pragyajswl/">Pragya Jaiswal</a> and <a href="https://www.linkedin.com/in/ved-vasu-sharma-34588ba2/">Ved Vasu Sharma</a> for design discussions and PR reviews while moving this pipeline to SquadAI codebase. Also would like to thank <a href="https://www.squadvoice.co/">Squad Voice</a> Operations team for their vital inputs.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2205e0accbda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/speaker-diarization-the-squad-way-2205e0accbda">Speaker Diarization — The Squad Way</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Class Classification for Images with Deep features]]></title>
            <link>https://medium.com/squad-engineering/one-class-classification-for-images-with-deep-features-69182fb4c9c5?source=rss-4364e1c0cfb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/69182fb4c9c5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[one-class-learning]]></category>
            <dc:creator><![CDATA[Aniket Bhatnagar]]></dc:creator>
            <pubDate>Thu, 08 Mar 2018 13:27:15 GMT</pubDate>
            <atom:updated>2018-03-08T13:27:15.427Z</atom:updated>
            <content:encoded><![CDATA[<p>Originally published on <a href="https://hackernoon.com/one-class-classification-for-images-with-deep-features-be890c43455d">HackerNoon</a></p><p>As programmers we regularly come across projects that require the task of building binary classifiers of the types A vs ~A, in which when the classifier is given a new data sample, it’s able to predict whether the sample belongs to class A or is an outlier. One reliable but difficult approach to solve such a problem is using the One-class Learning Paradigm.</p><p>In one-class learning we train the model only on the positive class data-set and take judgments from it on the universe [A union ~A] spontaneously. It’s a hot research topic and there are multiple tools available, like <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html">One-class SVM</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html">Isolation Forest</a>, to achieve this task. One-class learning can prove to be vital in scenarios where data consisting of ~A samples can take up any distribution and it isn’t possible to learn a pattern for ~A class.</p><p>But one-class learning becomes more challenging when the dimensions of the sample points increase. For example, consider an image size of 224x224px — to apply any one-class learning algorithm here straight out of the box can prove fatal due to the immense number of features each sample point holds (in this example it’s 50,176 features). Consequently efficient and discriminating feature representation is required to build a one-class classifier for images or high-dimensional data in general.</p><p><em>This post covers the implementation of one-class learning using deep neural net features and compares classifier performance based on the approaches of OC- SVM, Isolation Forest and Gaussian Mixtures.</em></p><p><strong>CNNs as Feature Extractors</strong></p><p>Convolutional Neural Nets have proven to be state-of-the-art when it comes to object recognition in images. CNNs have replaced traditional machine learning pipelines where feature extraction and the models used to learn from those features were two separate entities. Moreover, CNNs can also help in extraction of meaningful features for an image because a deep neural net also learns which features are important and which aren’t, in order to distinguish a class from the others. This allows us to simply <a href="https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html">use features</a> returned from these deep CNNs which are ready-to-go, and build our classifier.</p><p>Moreover, the availability of pre-trained CNNs on ImageNet data, with over 1,000 categories and more than 14 million images, has made image categorization much more simple since a pre-trained CNN will generally return features which are <em>sufficiently satisfactory</em> to train a light-weight model using them.</p><p>As I mentioned earlier — since feature vectors returned by CNNs give powerful representation of the image which generated them, we use these features to train our one-class classifier. For the purpose of this post, we use ResNet-50 as feature extractor for our model. <em>Why?</em> Because it’s fast, accurate, and credible as it <a href="https://blogs.microsoft.com/ai/microsoft-researchers-win-imagenet-computer-vision-challenge/">won the ImageNet Challenge in 2015</a>.</p><p><strong><em>Dataset used for this problem — </em></strong>We use the <a href="https://mmspg.epfl.ch/food-image-datasets">Food5k</a> data-set, which contains both Food and ~Food images (2500 each). Sample images are shown below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*SL6h3aP-lsuk5biyAJoLvw.png" /><figcaption>Sample food vs ~food images</figcaption></figure><p><strong>Implementation details for One Class SVM and Isolation Forest models:</strong></p><p>We first compute ResNet-50 features for the image data-set. The code for which will look as follows -</p><pre>from keras.applications.resnet50 import ResNet50<br>def extract_resnet(X):  <br>    # X : images numpy array<br>    resnet_model = ResNet50(input_shape=(image_h, image_w, 3), weights=&#39;imagenet&#39;, include_top=False)  # Since top layer is the fc layer used for predictions<br>    features_array = resnet_model.predict(X)<br>    return features_array</pre><p>We can accordingly compute ResNet features for all the images in the data-set. Next step, follow the pipeline -</p><ol><li>Apply standard scaler on the obtained features.</li><li>Principal Component Analysis with n_components = 512.</li><li>Pass the remaining features to One Class SVM model or Isolation Forest</li></ol><p>In below code, X_train and X_test are the resnet features for the train and test images.</p><pre>from sklearn.preprocessing import StandardScaler<br>from sklearn.decomposition import PCA<br>from sklearn.ensemble import IsolationForest<br>from sklearn import svm</pre><pre># Apply standard scaler to output from resnet50<br>ss = StandardScaler()<br>ss.fit(X_train)<br>X_train = ss.transform(X_train)<br>X_test = ss.transform(X_test)</pre><pre># Take PCA to reduce feature space dimensionality<br>pca = PCA(n_components=512, whiten=True)<br>pca = pca.fit(X_train)<br>print(&#39;Explained variance percentage = %0.2f&#39; % sum(pca.explained_variance_ratio_))<br>X_train = pca.transform(X_train)<br>X_test = pca.transform(X_test)</pre><pre># Train classifier and obtain predictions for OC-SVM<br>oc_svm_clf = svm.OneClassSVM(gamma=0.001, kernel=&#39;rbf&#39;, nu=0.08)  # Obtained using grid search<br>if_clf = IsolationForest(contamination=0.08, max_features=1.0, max_samples=1.0, n_estimators=40)  # Obtained using grid search</pre><pre>oc_svm_clf.fit(X_train)<br>if_clf.fit(X_train)</pre><pre>oc_svm_preds = oc_svm_clf.predict(X_test)<br>if_preds = if_clf.predict(X_test)</pre><pre># Further compute accuracy, precision and recall for the two predictions sets obtained</pre><p><strong>PS</strong>: Predictions returned by both isolation forest and one-class SVM are of the form {-1, 1}. -1 for the “Not food” and 1 for “Food”.</p><p><strong>One Class Classification using Gaussian Mixtures and Isotonic Regression</strong></p><p>Intuitively, food items can belong to different clusters like <em>cereals, egg dishes, breads</em>, etc., and some food items may also belong to multiple clusters simultaneously. As a result, we can fit a Gaussian mixture on the positive class data points (ResNet features). <a href="https://brilliant.org/wiki/gaussian-mixture-model/">Gaussian mixture models</a> “<em>are a probabilistic model for representing normally distributed subpopulations within an overall population</em>”. It can be surmised that a mixture model represents clusters of normally distributed subpopulations. A Gaussian mixture model, once fitted on the data, can give us information on the probability of whether any new point was generated from that distribution.</p><p>But beware — <strong><em>Gaussian mixture models return log of probability density function values for a given sample</em></strong> (and not actual probabilities). Hence it is necessary to convert these probability density function values to ‘probability scores’, which can then show that a new sample will belong to the Gaussian distribution with “x” amount of confidence.</p><p>A simple yet efficient method to accomplish this is by fitting an isotonic regression model on the log probability density scores w.r.t. labels for the validation set data points. <a href="http://www.research.ibm.com/people/z/zadrozny/kdd2002-Transf.pdf">Isotonic regression</a> is a probability calibration technique which can calibrate classifier scores to approximate probability values by fitting a stepwise non-decreasing function along the scores returned by the classifier.</p><p><strong>Implementation Details for one class learning with GMMs</strong></p><pre># The standard scaler and PCA part remain same. Just that we will also require a validation set to fit<br># isotonic regressor on the probability density scores returned by GMM</pre><pre># Also assuming that resnet feature generation is done<br>from sklearn.mixture import GaussianMixture<br>from sklearn.isotonic import IsotonicRegression</pre><pre>gmm_clf = GaussianMixture(covariance_type=&#39;spherical&#39;, n_components=18, max_iter=int(1e7))  # Obtained via grid search</pre><pre>gmm_clf.fit(X_train)</pre><pre>log_probs_val = gmm_clf.score_samples(X_val)</pre><pre>isotonic_regressor = IsotonicRegression(out_of_bounds=&#39;clip&#39;)<br>isotonic_regressor.fit(log_probs_val, y_val)  # y_val is for labels 0 - not food 1 - food (validation set)</pre><pre># Obtaining results on the test set<br>log_probs_test = gmm_clf.score_samples(X_test)<br>test_probabilities = isotonic_regressor.predict(log_probs_test)<br>test_predictions = [1 if prob &gt;= 0.5 else 0 for prob in test_probabilities]</pre><pre># Calculate accuracy metrics</pre><h3>Results and Discussions</h3><p>Following are the results obtained using the three one class learning techniques</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNffiubY7w1-8TGl5dIzFA.png" /><figcaption>Results table comparing all three algorithms over food vs ~food data</figcaption></figure><p>Thus the GMM model (calibrated using isotonic regression) outperforms both the other one-class learning models and is not far from ‘<a href="https://infoscience.epfl.ch/record/221610/files/madima2016_food_recognition.pdf">state-of-art</a>’ which is obtained after training a neural net for 700 iterations. Furthermore, in the state-of-art approach the model is trained on both positive and negative data samples, whereas in our approach the model is<strong> just trained on positive class samples,</strong> therefore making it more robust to handle any kind of distributions in the ~A samples.</p><p>GMM helps improve the precision of the model by correctly predicting more “Not food” images as compared to one-class SVM. This results in considerable less false positives. For example, some of the images for which GMM correctly predicts as “Not food” (as opposed to OC-SVM) are -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AZn1K8xbJuKfcGMyI-tp0g.png" /><figcaption>Not food correctly predicted by GMM as opposed to OC-SVM</figcaption></figure><p>Finishing off with an additional note — I didn’t go deeper on how to do a grid search for Gaussian Mixture Models but anyone who wishes to read in more about it can check out this <a href="http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html#sphx-glr-auto-examples-mixture-plot-gmm-selection-py">sklearn tutorial</a>.</p><p>If you liked this post you can check more stories from Squad Engineering at <a href="https://medium.com/squad-engineering">https://medium.com/squad-engineering</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69182fb4c9c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/one-class-classification-for-images-with-deep-features-69182fb4c9c5">One Class Classification for Images with Deep features</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Class Classification for Images with Deep features]]></title>
            <link>https://medium.com/hackernoon/one-class-classification-for-images-with-deep-features-be890c43455d?source=rss-4364e1c0cfb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/be890c43455d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[one-class-learning]]></category>
            <dc:creator><![CDATA[Aniket Bhatnagar]]></dc:creator>
            <pubDate>Thu, 18 Jan 2018 12:40:41 GMT</pubDate>
            <atom:updated>2019-04-26T09:51:20.736Z</atom:updated>
            <content:encoded><![CDATA[<p>As programmers we regularly come across projects that require the task of building binary classifiers of the types A vs ~A, in which when the classifier is given a new data sample, it’s able to predict whether the sample belongs to class A or is an outlier. One reliable but difficult approach to solve such a problem is using the One-class <a href="https://hackernoon.com/tagged/learning">Learning</a> Paradigm.</p><p>In one-class learning we train the model only on the positive class data-set and take judgments from it on the universe [A union ~A] spontaneously. It’s a hot research topic and there are multiple tools available, like <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html">One-class SVM</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html">Isolation Forest</a>, to achieve this task. One-class learning can prove to be vital in scenarios where data consisting of ~A samples can take up any distribution and it isn’t possible to learn a pattern for ~A class.</p><p>But one-class learning becomes more challenging when the dimensions of the sample points increase. For example, consider an image size of 224x224px — to apply any one-class learning algorithm here straight out of the box can prove fatal due to the immense number of features each sample point holds (in this example it’s 50,176 features). Consequently efficient and discriminating feature representation is required to build a one-class classifier for images or high-dimensional data in general.</p><p><em>This post covers the implementation of one-class learning using deep neural net features and compares classifier performance based on the approaches of OC- SVM, Isolation Forest and Gaussian Mixtures.</em></p><p><strong>CNNs as Feature Extractors</strong></p><p>Convolutional Neural Nets have proven to be state-of-the-art when it comes to object recognition in images. CNNs have replaced traditional <a href="https://hackernoon.com/tagged/machine-learning">machine learning</a> pipelines where feature extraction and the models used to learn from those features were two separate entities. Moreover, CNNs can also help in extraction of meaningful features for an image because a deep neural net also learns which features are important and which aren’t, in order to distinguish a class from the others. This allows us to simply <a href="https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html">use features</a> returned from these deep CNNs which are ready-to-go, and build our classifier.</p><p>Moreover, the availability of pre-trained CNNs on ImageNet data, with over 1,000 categories and more than 14 million images, has made image categorization much more simple since a pre-trained CNN will generally return features which are <em>sufficiently satisfactory</em> to train a light-weight model using them.</p><p>As I mentioned earlier — since feature vectors returned by CNNs give powerful representation of the image which generated them, we use these features to train our one-class classifier. For the purpose of this post, we use ResNet-50 as feature extractor for our model. <em>Why?</em> Because it’s fast, accurate, and credible as it <a href="https://blogs.microsoft.com/ai/microsoft-researchers-win-imagenet-computer-vision-challenge/">won the ImageNet Challenge in 2015</a>.</p><p><strong><em>Dataset used for this problem — </em></strong>We use the <a href="https://mmspg.epfl.ch/food-image-datasets">Food5k</a> data-set, which contains both Food and ~Food images (2500 each). Sample images are shown below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*SL6h3aP-lsuk5biyAJoLvw.png" /><figcaption>Sample food vs ~food images</figcaption></figure><p><strong>Implementation details for One Class SVM and Isolation Forest models:</strong></p><p>We first compute ResNet-50 features for the image data-set. The code for which will look as follows -</p><pre>from keras.applications.resnet50 import ResNet50<br>def extract_resnet(X):  <br>    # X : images numpy array<br>    resnet_model = ResNet50(input_shape=(image_h, image_w, 3), weights=&#39;imagenet&#39;, include_top=False)  # Since top layer is the fc layer used for predictions<br>    features_array = resnet_model.predict(X)<br>    return features_array</pre><p>We can accordingly compute ResNet features for all the images in the data-set. Next step, follow the pipeline -</p><ol><li>Apply standard scaler on the obtained features.</li><li>Principal Component Analysis with n_components = 512.</li><li>Pass the remaining features to One Class SVM model or Isolation Forest</li></ol><p>In below code, X_train and X_test are the resnet features for the train and test images.</p><pre>from sklearn.preprocessing import StandardScaler<br>from sklearn.decomposition import PCA<br>from sklearn.ensemble import IsolationForest<br>from sklearn import svm<br><br># Apply standard scaler to output from resnet50<br>ss = StandardScaler()<br>ss.fit(X_train)<br>X_train = ss.transform(X_train)<br>X_test = ss.transform(X_test)<br><br># Take PCA to reduce feature space dimensionality<br>pca = PCA(n_components=512, whiten=True)<br>pca = pca.fit(X_train)<br>print(&#39;Explained variance percentage = %0.2f&#39; % sum(pca.explained_variance_ratio_))<br>X_train = pca.transform(X_train)<br>X_test = pca.transform(X_test)<br><br># Train classifier and obtain predictions for OC-SVM<br>oc_svm_clf = svm.OneClassSVM(gamma=0.001, kernel=&#39;rbf&#39;, nu=0.08)  # Obtained using grid search<br>if_clf = IsolationForest(contamination=0.08, max_features=1.0, max_samples=1.0, n_estimators=40)  # Obtained using grid search<br><br>oc_svm_clf.fit(X_train)<br>if_clf.fit(X_train)<br><br>oc_svm_preds = oc_svm_clf.predict(X_test)<br>if_preds = if_clf.predict(X_test)<br><br># Further compute accuracy, precision and recall for the two predictions sets obtained</pre><p><strong>PS</strong>: Predictions returned by both isolation forest and one-class SVM are of the form {-1, 1}. -1 for the “Not food” and 1 for “Food”.</p><p><strong>One Class Classification using Gaussian Mixtures and Isotonic Regression</strong></p><p>Intuitively, food items can belong to different clusters like <em>cereals, egg dishes, breads</em>, etc., and some food items may also belong to multiple clusters simultaneously. As a result, we can fit a Gaussian mixture on the positive class data points (ResNet features). <a href="https://brilliant.org/wiki/gaussian-mixture-model/">Gaussian mixture models</a> “<em>are a probabilistic model for representing normally distributed subpopulations within an overall population</em>”. It can be surmised that a mixture model represents clusters of normally distributed subpopulations. A Gaussian mixture model, once fitted on the data, can give us information on the probability of whether any new point was generated from that distribution.</p><p>But beware — <strong><em>Gaussian mixture models return log of probability density function values for a given sample</em></strong> (and not actual probabilities). Hence it is necessary to convert these probability density function values to ‘probability scores’, which can then show that a new sample will belong to the Gaussian distribution with “x” amount of confidence.</p><p>A simple yet efficient method to accomplish this is by fitting an isotonic regression model on the log probability density scores w.r.t. labels for the validation set data points. <a href="http://www.research.ibm.com/people/z/zadrozny/kdd2002-Transf.pdf">Isotonic regression</a> is a probability calibration technique which can calibrate classifier scores to approximate probability values by fitting a stepwise non-decreasing function along the scores returned by the classifier.</p><p><strong>Implementation Details for one class learning with GMMs</strong></p><pre># The standard scaler and PCA part remain same. Just that we will also require a validation set to fit<br># isotonic regressor on the probability density scores returned by GMM<br><br># Also assuming that resnet feature generation is done<br>from sklearn.mixture import GaussianMixture<br>from sklearn.isotonic import IsotonicRegression</pre><pre>gmm_clf = GaussianMixture(covariance_type=&#39;spherical&#39;, n_components=18, max_iter=int(1e7))  # Obtained via grid search</pre><pre>gmm_clf.fit(X_train)</pre><pre>log_probs_val = gmm_clf.score_samples(X_val)</pre><pre>isotonic_regressor = IsotonicRegression(out_of_bounds=&#39;clip&#39;)<br>isotonic_regressor.fit(log_probs_val, y_val)  # y_val is for labels 0 - not food 1 - food (validation set)<br><br># Obtaining results on the test set<br>log_probs_test = gmm_clf.score_samples(X_test)<br>test_probabilities = isotonic_regressor.predict(log_probs_test)<br>test_predictions = [1 if prob &gt;= 0.5 else 0 for prob in test_probabilities]<br><br># Calculate accuracy metrics</pre><h3><strong>Results and Discussions</strong></h3><p>Following are the results obtained using the three one class learning techniques</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNffiubY7w1-8TGl5dIzFA.png" /><figcaption>Results table comparing all three algorithms over food vs ~food data</figcaption></figure><p>Thus the GMM model (calibrated using isotonic regression) outperforms both the other one-class learning models and is not far from ‘<a href="https://infoscience.epfl.ch/record/221610/files/madima2016_food_recognition.pdf">state-of-art</a>’ which is obtained after training a neural net for 700 iterations. Furthermore, in the state-of-art approach the model is trained on both positive and negative data samples, whereas in our approach the model is<strong> just trained on positive class samples,</strong> therefore making it more robust to handle any kind of distributions in the ~A samples.</p><p>GMM helps improve the precision of the model by correctly predicting more “Not food” images as compared to one-class SVM. This results in considerable less false positives. For example, some of the images for which GMM correctly predicts as “Not food” (as opposed to OC-SVM) are -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AZn1K8xbJuKfcGMyI-tp0g.png" /><figcaption>Not food correctly predicted by GMM as opposed to OC-SVM</figcaption></figure><p>Finishing off with an additional note — I didn’t go deeper on how to do a grid search for Gaussian Mixture Models but anyone who wishes to read in more about it can check out this <a href="http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html#sphx-glr-auto-examples-mixture-plot-gmm-selection-py">sklearn tutorial</a>.</p><p>If you liked this post you can check more stories from Squad Engineering at <a href="https://medium.com/squad-engineering">https://medium.com/squad-engineering</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be890c43455d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/one-class-classification-for-images-with-deep-features-be890c43455d">One Class Classification for Images with Deep features</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up Highly Available RMQ cluster on AWS]]></title>
            <link>https://medium.com/squad-engineering/set-up-highly-available-rmq-cluster-on-aws-30f6882d06f1?source=rss-4364e1c0cfb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/30f6882d06f1</guid>
            <category><![CDATA[rabbitmq]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Aniket Bhatnagar]]></dc:creator>
            <pubDate>Mon, 22 May 2017 18:25:20 GMT</pubDate>
            <atom:updated>2018-02-20T07:14:24.580Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*9a-iCFuAjOHPfl78hyhkPg.png" /></figure><p>Task queue or message brokers accept and forward messages. They act like a post office. Task queues manage background work that can be executed outside the usual HTTP request-response cycle. Thus they can make your application seem faster while also making it more efficient. Asynchronous message passing and processing have various advantages and have increasingly become indispensable.</p><p>Not having a highly available system can lead to loss of queued tasks/messages and the application servers being unable to queue new ones. This obviously isn’t a desirable state for a system with high-reliability needs.</p><p><strong>The solution for this is to setup a Highly Available (HA) system.</strong><br><a href="https://en.wikipedia.org/wiki/High_availability">High availability</a> is a characteristic of a system, which aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period.</p><h3>What needs to be Done</h3><p>In a nutshell, we need to create an <a href="https://www.rabbitmq.com/clustering.html">RMQ cluster</a> with <a href="https://www.rabbitmq.com/ha.html">queue mirroring</a> enabled, which means, we will not be dependent on 1 RMQ node to maintain queues and pass on tasks to the consumers. Plus tasks of each queue are mirrored to all other nodes in the cluster as well. So in an event of one of the nodes going down, all its master queues will start functioning from the next oldest node available in the cluster.<br>For example, we have a cluster with two nodes, R1 and R2. R1 has queues q1a and q1b while R2 only has one queue, q2. The configuration will look as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*q6xcBT1-XMMykTumFDkHjw.png" /></figure><p>But when we enable queue mirroring, each queue will have its mirror on every other node in the cluster. In our case, we will have three additional queues, q2_m at R1 and q1a_m and q1b_m at R2, as shown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*Bq8yteY31_NfPHmkqsCQ3g.png" /></figure><h3>How it works</h3><p>In the above example the queues q1a, q1b and q2 are called the master queues while the mirrors q1a_m, q1b_m and q2_m are called slave queues. The protocol followed in such a configuration is as follows:</p><ul><li>Whenever a task is published for a queue, it is enqueued onto master queue first from where it is mirrored to all other slave queues. Similarly, whenever a task is consumed from the master queue, it will be dropped from the slave queue(s) as well.</li><li>If there are n nodes in an RMQ cluster then any queue made on any node will have a mirror on every other node in the cluster. So there will be n-1 slave queues per queue in a cluster.</li><li>In the case of multiple slave queues, the oldest slave is promoted to become the new master in an event of the master going out of order.</li><li>A message sent, for a queue, to any of the nodes in the cluster only gets enqueued if the node holds the master queue, else it will be sent to the RMQ node that holds the master queue for that queue and the message gets enqueued over there.</li><li>Continuing with the aforementioned example. A task T1 sent to node R1 for queue q1a will get enqueued directly to q1a and subsequently get mirrored to q1a_m.<br>And a task T2 sent to node R1 for queue q2 will first be routed to R2 where it will get enqueued to q2 and subsequently mirrored to q2_m.</li><li>Similarly, a message is always consumed from the master queue and subsequently dropped from the mirror queues. If a consumer tries to make a connection with some other node it will be routed to the correct node internally.</li><li>Communication between RMQ cluster nodes is only possible when each RMQ node has the same Erlang key. This Erlang cookie on Linux system is usually present in the /var/lib/rabbitmq/ directory.</li></ul><h3>Implementation Details</h3><h4>RMQ nodes</h4><ul><li>From the EC2 management console page, create a new security key which shall be same for all the AWS instances. Let this be named “rabbitmqkey”.</li><li>Create a new security group, to be used by our EC2 instances, with the following under inbound ports:</li></ul><pre>SSH                TCP             22        0.0.0.0/0<br>Custom TCP Rule    TCP         0-65535       &lt;this security group&gt;</pre><p>In the above step adding your <em>own security group</em> to the rules might not be possible without creating the group first so create the security group first with SSH rule only and later add the 2nd rule.</p><ul><li>Create a new AWS EC2 instance (First node of the RMQ cluster). Use the same security group and security key created in the previous steps.</li><li>Upon creation of your instance ssh into your EC2 machine and run the following commands to set up an RMQ node running on this instance.</li></ul><pre>sudo apt-get install rabbitmq-server<br>sudo rabbitmq-plugins enable rabbitmq_management<br>sudo rabbitmq-server -detached </pre><p><a href="https://www.rabbitmq.com/management.html">The Management plugin</a> is provided by rabbitMQ to be able to check status of tasks and queues present on the particular node and in the cluster.</p><ul><li>Now, to ensure that each node of the rabbitMQ cluster has the same erlang cookie and to avoid installing RMQ on each server, we will create an AMI (Amazon machine image) of the above EC2 instance.</li><li>After the image is constructed, create a new instance using that image. Make sure this instance also belongs to the same security group (and also, is created in a different availability zone, because then what&#39;s the point?)</li><li>SSH into the new EC2 instance created and type in the following commands:</li></ul><pre>sudo rabbitmqctl stop_app<br>sudo rabbitmqctl join_cluster rabbit@&lt;private ip of 1st EC2 instance&gt;<br>sudo rabbitmqctl start_app<br>sudo rabbitmqctl cluster_status</pre><p><a href="https://www.rabbitmq.com/clustering.html">&#39;join_cluster&#39;</a> will connect this second node to the cluster to the 1st node.</p><ul><li>Next, SSH into the main RMQ node and set the <a href="https://www.rabbitmq.com/ha.html">HA Policy</a>. The policy needs to be activated just once for the entire cluster and should ideally be done at the first node of the cluster. The command for the same goes as:</li></ul><pre>sudo rabbitmqctl set_policy ha-all &quot;&quot; &#39;{&quot;ha-mode&quot;:&quot;all&quot;,&quot;ha-sync-mode&quot;:&quot;automatic&quot;}&#39;</pre><h4>Load Balancer</h4><p>We used AWS’s Elastic Load Balancer (ELB) to set up our load balancing. ELB directs requests to its nodes in a Round Robin fashion but we need not worry about it as the messages will get internally routed to the master node when need be.</p><p>Launch an ELB with the same security group as was used to create the EC2 instances. Set the load balancer’s port configuration as:</p><ol><li>5672 (TCP) forwarding to 5672 (TCP) — For producers and consumers</li><li>15672 (TCP) forwarding to 15672 (TCP) — For the management plugin</li></ol><p><strong>Congratulations, your highly available RabbitMQ system is ready!!</strong></p><h3>Gotcha!</h3><p>ELB resets the connection with the application servers periodically. Thus, it is possible that your producers send a task to the ELB which gets lost because of this resetting of connection between the ELB and the nodes.<br>In such situation, we will need to enable confirmation of the tasks published. The solution discussed here is specific to <strong><em>Celery</em></strong> based integration with RMQ where backend is built on <strong><em>Django</em></strong>. The following needs to be set in settings.py in such case:</p><pre>BROKER_TRANSPORT_OPTIONS = {&#39;confirm_publish&#39;: True}</pre><p>This way, your publisher will wait for the RMQ to send an ack before sending any further tasks to the same queue.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30f6882d06f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/set-up-highly-available-rmq-cluster-on-aws-30f6882d06f1">Setting up Highly Available RMQ cluster on AWS</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>