<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Helmi Rifai on Medium]]></title>
        <description><![CDATA[Stories by Helmi Rifai on Medium]]></description>
        <link>https://medium.com/@helmz?source=rss-847353d1a4a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZXHPIbLJvgeOwuJ-.</url>
            <title>Stories by Helmi Rifai on Medium</title>
            <link>https://medium.com/@helmz?source=rss-847353d1a4a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:49:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@helmz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[30 Million Songs Down to 30]]></title>
            <link>https://medium.com/hackernoon/30-million-songs-down-to-30-1e6a4d2aebdc?source=rss-847353d1a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e6a4d2aebdc</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Helmi Rifai]]></dc:creator>
            <pubDate>Wed, 28 Mar 2018 16:20:38 GMT</pubDate>
            <atom:updated>2019-05-21T10:32:14.452Z</atom:updated>
            <content:encoded><![CDATA[<h4>Generating millions of personalized playlists @AnghamiTech</h4><p><a href="http://play.anghami.com/"><strong>Anghami</strong></a> is the leading Music streaming service in the MENA region. My team leads on personalization related features, the champion of which is the “Your Weekly Mixtape” feature. The Mixtape is a playlist of 30 songs that we generate weekly for every Anghami user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GmPVopTajsvfwHyLvhRQ8Q.png" /><figcaption>I know! I liked them all 😀</figcaption></figure><p>Every week our users get a fresh set of songs based on their taste. It uses our recommendation pipeline (codename <em>HARP</em>) to sift through 30 Million songs and create a playlist that our users will love; we balance between songs a user knows and what we think they should discover.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*B5nnE0SL6FxOe9r9QAf7RQ.png" /><figcaption>Curating music based on a user’s taste profile (musical fingerprint).</figcaption></figure><h4>Going deeper into how we generate the Mixtape</h4><p>First, our algorithms use our data to pair up all user-song (optionally user-artist as well) interactions ( see <a href="https://towardsdatascience.com/using-word2vec-for-music-recommendations-bb9649ac2484">Song2Vec</a> &amp; <a href="https://en.wikipedia.org/wiki/Collaborative_filtering">Collaborative Filtering</a> + more on these in <a href="https://hackernoon.com/tagged/future">future</a> posts) to project songs &amp; artists into what is called a Latent Space by giving them vector representations; essentially a method that allows us to take any 2 pairs of items and get a “similarity score” between them. Now we have the ability to take any item and score all other items by similarity, so then we take it from here.</p><p>Next we need to transform a user’s abstract “taste” into something we can work with. We take the user’s recent interactions with content on Anghami and use an exponential time decay on the scores, in simple terms: a more recent interaction is worth more than an older one. This allows us to capture a user’s changing taste over time while still incorporating their histories so no 1 week of change in habits can throw the Mixtape off, i.e. we allow your guilty pleasures :) In more technical terms we project the user’s “taste” into the same Latent Space where we projected songs &amp; artists allowing us to assign a perceived “interest score” between users &amp; content.</p><p>Now that we can assign a score between each user and song we can go ahead and take a crack at generating a playlist. Simply getting the list of the closest 30 songs to a user proves to be problematic. Recommender Systems are a great <a href="https://hackernoon.com/tagged/tool">tool</a>, but without some logic overlaid they tend to generate boring results; they are very likely to give results from the artists the user has interacted with, they also tend to give multiple songs from the same artist and such.</p><p>To work around these aspects of the system we layer some final bits of human logic to aid these machines:</p><p><em>After we generate the first ever Mixtape, we keep track of all songs and artists ever added to any user’s Mixtape to make sure we don’t recommend the same song across different weeks and so we don’t keep shoving the same artists in consecutive weeks.</em></p><ol><li>We make sure the songs haven’t been played by the user recently</li><li>We make sure we aren’t giving different versions of the same songs previously recommended</li><li>We start off by generating up to 200 “possible” tracks for a User (1 by artist)</li><li>We factor in how much a user “knows” a given artist (streams + % of discography) to use that in the sorting of the list. This is to allow for discovery or we’ll just be giving users songs from artists they know and the Mixtape will become boring.</li><li>We now have a re-sorted list of the 200 initial tracks from which we select the top 30 tracks for the Mixtape.</li></ol><p>And there you have it, a simple-ish formula to generate a personalized playlist and get LOVE from users. As normal with all products the formula keeps on changing as we iterate on the the product to make it better.</p><p>The chart below gives an idea of how often users interact with the Mixtape. We see a slow start when it was in staged rollout on a small segment for data verification &amp; validation. Then we see a quick rise which happens when we rollout a new feature and user start seeing and interacting with it, after which we get a slight drop and stabilization and back to growth. This is a pretty standard usage chart which is quite healthy as we see near the end of the chart that our usage is split almost equally between users that have never tried the Mixtape before, users that streamed it the previous weeks and users that streamed it before but didn’t the previous week. We had a few dips (2 main ones) where we had issues with sending out notifications and not displaying the Mixtape on the user’s homepage properly 🙊</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VcxqrP4tAL6jm2C-aH0SJw.png" /><figcaption>Usage on the Mixtape by User Segment</figcaption></figure><p>More posts on Recommendations coming soon: our take on Collaborative Filtering, “Your Welcome Mixtape”, Content-Based Recommendations (codename <em>Cochlea</em>), Radios@Anghami, NewMusic@Anghami.</p><p>A big thanks to Anghami and my team (<a href="https://medium.com/u/f0b042f4ac2d">Ramzi Karam</a>, <a href="https://medium.com/u/d953ce72fa8b">Abdallah Moussawi</a>, <a href="https://medium.com/u/ff599d0598dc">Omar El Zarif</a>).</p><p>Shoutouts to the <a href="http://spark.apache.org/">Apache Spark</a> team, on top of which <em>HARP</em> is built.</p><p>Join the fun: <a href="https://www.anghami.com/careers">https://www.anghami.com/careers</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e6a4d2aebdc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/30-million-songs-down-to-30-1e6a4d2aebdc">30 Million Songs Down to 30</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reducing Costs in the Cloud]]></title>
            <link>https://medium.com/hackernoon/reducing-costs-in-the-cloud-c6d822415aa7?source=rss-847353d1a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6d822415aa7</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Helmi Rifai]]></dc:creator>
            <pubDate>Tue, 14 Feb 2017 18:41:46 GMT</pubDate>
            <atom:updated>2019-05-21T10:32:15.076Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*paashaJr-y87vmCBVq-Ddg.jpeg" /><figcaption>Credits to <a href="https://medium.com/u/61bd5185adba">Salim Batlouni</a> our Photographer / Product Manager</figcaption></figure><h4>A tale of investigating rising costs in a high velocity startup</h4><p><a href="https://www.anghami.com/"><em>Anghami</em></a><em> </em>is the leading music &amp; <a href="https://hackernoon.com/tagged/entertainment">entertainment</a> platform in the Middle East. We’ve been dubbed the <a href="http://www.businessinsider.com/meet-anghami-the-spotify-of-the-middle-east-2016-8">Spotify of the Middle East</a>, though we prefer just <a href="https://play.anghami.com/"><em>Anghami</em></a><em>. </em>Millions of users use our services daily to enjoy <strong>Music</strong>, <strong>Music Videos</strong> or <a href="https://medium.com/@batlounis/how-our-app-became-a-social-hub-a8acf0421812?source=linkShare-847353d1a4a-1487661653"><strong>Expressions</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*vvQd-JcVcBSlzOm5inNkiQ.jpeg" /></figure><p>We have been on the <a href="https://hackernoon.com/tagged/aws">AWS</a> Cloud since Day 0, which is 4 years ago, I joined 2 &amp; 1/2 years ago (and my oh my does time fly) and we mostly love the scale and flexibility we get. We rely on AWS’s <a href="https://aws.amazon.com/sqs/">SQS</a> (Simple Queue Service) to move data asynchronously back and forth through some of our systems. For those that don’t know how SQS or what we call Message Queues work: they are systems (or a service in the case of SQS) where you can push some data we call <strong>“Messages” </strong>to, from what we call <strong>“Producers”</strong>, you can then subsequently pull data from it and process it on what we call <strong>“Consumers”</strong> to accomplish the processing of tasks asynchronously, where <strong>“Consumers”</strong> <em>usually</em> run the longer running tasks needed to be done like sending out emails to welcome a New User.</p><p>The wrapper and core processing implementation around SQS was written before I joined and was working pretty well at the scale we were running it back then. Then something happened, we need to process more data and so we started adding more “Consumers” latching on to SQS to fetch and process “Messages”. That’s when we started to see our SQS costs go up disproportionately to the increase in the number of “Messages” we were processing! Queue panic mode and investigations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wN__csEYAsxRGX0DFXH_gw.png" /><figcaption>Total SQS Costs with a BIG jump in July</figcaption></figure><p>I opened our AWS Billing Dashboard (it was my first time back then) to check what was going on. I found that they provide a tab called “Cost Explorer” which contained the link to “Daily Spend View”. There I chose the date range I was interested in (before and after the jump in costs and addition of “Consumers”), followed by adding a filter for the Service SQS shown above.</p><p>Then I noticed the <em>Grouping</em> option and then tried to see what the different options were, the magical option for our case was grouping by <em>API Operation</em> which makes sense as SQS charges by number of API Operations, not “Messages”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vdNnlEn6G-Bg_tFqPCgldw.png" /><figcaption>SQS Costs grouped by API Operation</figcaption></figure><p>The chart provided gave me all the insight I needed to make sense of things. Most of our spend was going towards the <strong>GetQueueAttributes </strong>call (in blue above). Interesting! I opened up the old piece of code for the SQS Wrapper core we had and checked for calls to the method that invokes that API call, it was being called before every time we try to fetch messages to process to check if the queue had any contents and sleep if it didn’t. This was weird but without it we would keep calling the <strong>ReceiveMessage </strong>API in a loop (at the time this code was initially written SQS didn’t have long polling). By the time I had adopted the code SQS had long polling so first things first I updated all our Queues to set the <strong>Receive Message Wait Time</strong> to the max (20 seconds), which basically tells the SQS library to keep the HTTP connection to SQS waiting up to 20 seconds given that it can still receive more messages with the limit being 10 messages per call.</p><p>Then I updated the code to utilize the long polling, this removed all our reliance on <strong>GetQueueAttributes </strong>API Calls and I removed them from the code. That was the first sweep that cut down the bulk of the cost that was basically waste at that point.</p><p>After the rush of victory from my first investigation I looked into how I can take this further and tune it down. We know we will always be receiving “Messages” in batches of up to 10 and we are processing those on the same node. When a message is processed successfully you must call the <strong>DeleteMessage</strong> API to tell SQS to delete it from its pool of messages (there is a safety in SQS that sets a Timeout on the Message after which it is released back into the pool of available messages). What came next was batching those into <strong>DeleteMessageBatch </strong>instead of delete message taking care of not crossing the <strong>Visibility Timeout </strong>of the Queues in question. After I changed that code we saw another reduction in cost since we were basically calling <strong>DeleteMessageBatch </strong>once for every 10 messages (in reality it’s lower, at around one for every 7 Messages, since we don’t always get 10 messages or fail to process etc.).</p><p>I continued to then batch some of the <strong>SendMessage </strong>calls that could be which also resulted in some reduction there, but due to the nature of these systems not much can be batched when Sending as they are generated one by one when our API instances receive some calls etc. We also added <strong><em>autoscaling </em></strong>logic to our “Consumer” tier and with the above changes we no longer needed to worry and reduced costs further by only having the number of instances we need to process the load of messages we needed to process.</p><p>Thanks for reading!</p><figure><a href="http://bit.ly/HackernoonFB"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hqOaABQ7XGPT-OYNgiUBg.png" /></a></figure><figure><a href="https://goo.gl/k7XYbx"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vgw1jkA6hgnvwzTsfMlnpg.png" /></a></figure><figure><a href="https://goo.gl/4ofytp"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKBpq1ruUi0FVK2UM_I4tQ.png" /></a></figure><blockquote><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="/service/mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</blockquote><blockquote>If you enjoyed this story, we recommend reading our <a href="http://bit.ly/hackernoonlatestt">latest tech stories</a> and <a href="https://hackernoon.com/trending">trending tech stories</a>. Until next time, don’t take the realities of the world for granted!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35tCjoPcvq6LbB3I6Wegqw.jpeg" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6d822415aa7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/reducing-costs-in-the-cloud-c6d822415aa7">Reducing Costs in the Cloud</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>