<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Phani Srikanth on Medium]]></title>
        <description><![CDATA[Stories by Phani Srikanth on Medium]]></description>
        <link>https://medium.com/@phanisrikanth33?source=rss-b7f3e4fe28b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PwER0RvEnDf6RY4x6uF2ig.jpeg</url>
            <title>Stories by Phani Srikanth on Medium</title>
            <link>https://medium.com/@phanisrikanth33?source=rss-b7f3e4fe28b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:21:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@phanisrikanth33/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[2018 Goals Monthly Review: January ed.]]></title>
            <link>https://medium.com/@phanisrikanth33/2018-goals-monthly-review-january-ed-11269d9820f5?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/11269d9820f5</guid>
            <category><![CDATA[2018]]></category>
            <category><![CDATA[goals]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[alphago]]></category>
            <category><![CDATA[books]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Thu, 01 Feb 2018 15:40:15 GMT</pubDate>
            <atom:updated>2018-04-03T07:09:59.885Z</atom:updated>
            <content:encoded><![CDATA[<h4>A sedate start.</h4><p>Well, a lot of promises made and goals set. Did I make progress? Let’s see.</p><p>Tl; dr -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjRwVp7ufILRNMreS8h12A.png" /><figcaption>A summarised version of this post.</figcaption></figure><p>Link: <a href="https://docs.google.com/spreadsheets/d/1UEYtJzx-SUdpHdxKeW2OZQYZLOQTCh4z6iZblB1eCg0/edit?usp=sharing">https://docs.google.com/spreadsheets/d/1UEYtJzx-SUdpHdxKeW2OZQYZLOQTCh4z6iZblB1eCg0/edit?usp=sharing</a></p><p>Although the screenshot says the gist, I’d like to expand on three aspects.</p><h4>Movies</h4><p>While I planned to skip watching movies until July, I broke this resolution a couple of days back when I watched the movie <em>AlphaGo</em> (The story about Google’s AI engine beating the world champion Lee Sedol in Go) on Netflix. I’ve seen people lauding this movie recently and I decided to watch it and boy it was epic. The narration and screenplay of the movie mirrored real-life events and it was very emotional to say the least. In the man vs the machine battle, although the supporters of ML/AI wanted the machine to win, most of us deep down craved for the man to come out on the top. The moment where Lee resigns in the second game, slips a white stone on the board, his fingers still shivering with a feeling of disbelief on his face was perfectly captured. I thoroughly enjoyed the movie, partly because I could really connect to the algorithms etc., but I highly recommend this documentary to everyone to know the enormity of AI’s advancements and human creativity involved in creating this AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*64GP99vr0JlbgrgbPzdW6A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eT1iKTcKjYivbV76DBYbFQ.png" /><figcaption>The big moment in the second game.</figcaption></figure><h4>Book of the month</h4><p><em>A Thousand Splendid Suns</em></p><p>I started the book named “The power of moments” by Chip &amp; Dan Heath after a lot of recommendations and its immediate usability in my life. While the key concepts, about creating memorable moments to elevate people, are quite well explained, I realised I was forcing myself to read through it and it wasn’t an enjoyable read. I decided to chuck the book and picked up The thousand Splendid Suns by the famous Khaled Hosseini.</p><p>This book spans a 40-year story about two Afghani women, the kind of lives they live, the grief and oppression they go through and the hope that gets them through the day. Their grit and determination put a lot of things in perspective for most of us, in this materialistic world. The things we take for granted such as having two meals in a day, a peaceful home to live in, access to study, freedom to express love – they had none of these and when this happens, humans go through an enormous amount of grief and they start valuing small things in life. Often these people are stronger than we think. Mariam and Laila have left a strong impact on me. There’s a passage in the book where Laila and Mariam’s eyes are transfixed at each other and all they could manage is a smile. It was such a satisfying moment to read through. Also, the letter. Oh, it was heartbreakingly emotional. Tears rolled over in an instant. With that said, I’d like to read more of Khaled sometime later this year.</p><h4>The 1TB RAM server for Kaggle</h4><p>Rohan and I teamed up for the grocery forecasting kaggle challenge. Though we started in the last 10 days of the competition, we made good progress in the last week and we pushed hard to build really good individual models. It was the first time I was using servers of size 1TB RAM and 180 cores of CPU power. We believed we had a pretty strong validation framework and we ended up at 90th /1400 participants on public leaderboard on the last day. I woke up at 5:30AM to see final standings and I was shocked to see us fall by 530 places on private leaderboard. Later when we checked, we realised some of our initial models would have done better but we missed a couple of tricks that put us behind. It was a sad ending to a weeklong mania at my place. Anyways, lessons learnt and we’ll surely come back stronger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9flbnP_gbt3hoYV-BgfPbQ.png" /><figcaption>Our ensemble scored way lesser than single models. XGB really overfitted in comparison to LGB, inline with many other competitors.</figcaption></figure><p>Overall, I think it’s a slow start to an exciting 2018 and there’s a ton of good work to be done. See you next month with a bunch of updates. Stay tuned…</p><p>The original goals post is available here: <a href="https://medium.com/@phanisrikanth33/my-2018-goals-80a3d70fd213">https://medium.com/@phanisrikanth33/my-2018-goals-80a3d70fd213</a></p><p>February post: <a href="https://medium.com/@phanisrikanth33/2018-goals-monthly-review-february-ed-f3d68ee83039">https://medium.com/@phanisrikanth33/2018-goals-monthly-review-february-ed-f3d68ee83039</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11269d9820f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My 2018 goals.]]></title>
            <link>https://medium.com/@phanisrikanth33/my-2018-goals-80a3d70fd213?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/80a3d70fd213</guid>
            <category><![CDATA[new-year]]></category>
            <category><![CDATA[goals]]></category>
            <category><![CDATA[personal-development]]></category>
            <category><![CDATA[2018]]></category>
            <category><![CDATA[professional-development]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Mon, 08 Jan 2018 13:41:31 GMT</pubDate>
            <atom:updated>2018-01-14T06:59:46.219Z</atom:updated>
            <content:encoded><![CDATA[<p>Another new year comes by. Another opportunity for a fresh start. Well, technically you don’t need a “new year” to start something good or develop a good habit. But, I’m a human and maybe I need a moment like a new year to start a few new things. Haha. So, I’d like to take this opportunity to make a note of all the things that I’d like to accomplish in the next 12 months.</p><p>Well, I’ve had a colorful start to 2018. I travelled to Thailand for the new year, been at the surreal Maya Bay beach at Phi Phi islands and I couldn’t have asked for a much more spectacular new year party than the Patong beach party at Phuket! After returning to normalcy, I’ve spent good time figuring out what matters to me the most for the next 52 weeks and I’ve listed down a few of them.</p><p>I’d like to publicly commit to these milestones. I’m challenging myself to walk the extra mile, and accomplish all of them. All of them! No second measures.</p><h4>Personal goals</h4><ul><li>Stop watching movies in theatres until June 30th. And Amazon Prime too. It’s so addictive!, I ain’t a big movie buff like <a href="https://medium.com/u/5b82a84c6303">Pandari Jain Kumar | OTG</a> but I end up spending way too much time catching up on movies. So, avoid this. I’d like to re-evaluate this on July 1st.</li><li>Run 2 full marathons, 4 half-marathons; one each quarter. I dislike the gym. I enjoy being fit. Athletics maybe is way one way to get there and I’m up for it. <a href="https://medium.com/u/1edf46ba56fe">Sandeep Medisetti</a>, I’m coming for you.</li><li>Read 12 books this year from my reading list. Publish the review about the new book on 1st of every month. I ain’t an avid reader like <a href="https://medium.com/u/8d9de2c4968a">Rohan Rao</a>. However, I’d like to spend time reading quality books this year. Perhaps I will share my reading list in future, once it is in better shape.</li><li>Plan for a solo trip one inside the country and one outside. One for each half of the year.</li></ul><h4>Professional goals</h4><ul><li>Become one among the top 100 data scientists in the world, on Kaggle. Current Rank — 870 (as on January 8th).</li><li>Become a Kaggle Grandmaster. There are only 101 Kaggle grandmasters out of 70,000 active Kagglers as on today. (I consider this as the most ambitious goal of the lot.)</li><li>Become an accomplished blogger. I cannot think of a metric that would defined accomplished but let’s see how this turns out.</li><li>Earn and Save enough to build a deep learning rig. Maybe an Nvidia 1080Ti? Volta? Or, the next best GPU. Haha. Let’s see.</li></ul><p>I will report progress on all of the above on first of each month so that I can be held accountable. I wanted to commit publicly since I’ve miserably failed last two times whenever I committed on new year resolutions. This time the added pressure of making this public might motivate me further. Well, it might not as well. Who knows!</p><p><em>Anyways, it’s fun to set a personal goal, announce it so that it forces me to be conscious about the way I live my life and learn a great deal in the process.</em></p><p>To all those who made it till here, I really appreciate you for your time since I know some of you are going to be the ones who’d keep reminding me about this. Haha. And, I’m sure you have created your own goals too. Maybe, it’s time to make them public and we can learn from each other’s experiences.</p><p>Finally, I wish you all the best for your endeavours. And a very happy 2018 !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80a3d70fd213" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Notes from Machine Learning Summit]]></title>
            <link>https://medium.com/@phanisrikanth33/notes-from-machine-learning-summit-c1b13d00de69?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1b13d00de69</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[recommendation-engine]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Tue, 05 Dec 2017 05:44:07 GMT</pubDate>
            <atom:updated>2017-12-10T08:34:12.510Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A short memo on experiences from the hack sessions, workshops and answers for a set of common questions from attendees.</em></p><p>Analytics Vidhya, a popular community of data scientists from India, organised a Machine Learning and Artificial Intelligence conference on 9–11th November, 2017 in the hottest tech destination in India; Bengaluru. A bunch of thought leaders from the industry, prominent data scientists, data enthusiasts, and beginners too, have come together for this spectacular event. While Dr. <a href="https://medium.com/u/41c6e28d5e76">Kirk Borne</a>, from Booz Allen Hamilton, graced the occasion as a keynote speaker, there were several other speakers who were CEOs, Heads of various Indian analytics divisions who came down to share their wisdom with enthusiasts.</p><h4><strong><em>Format</em></strong></h4><p>While the keynote sessions were excellent, what stood out in the conference were the <em>workshops</em> and the new format of live <em>hack sessions</em>. These hack sessions prompt the speakers to code live, demonstrate a concept, show the output, in the process make a concept intuitive to the audience. Workshops and Hack Sessions spanned across several topics such as Computer Vision, Natural Language Processing, Applied Data Science, Strategies to do well in Data Science Challenges (by famous Kagglers — SRK and Bishwarup), ML in Production, Sentiment Analysis, Ensemble Learning etc.</p><p>Amidst such really good talks, I had an opportunity to deliver a hack session on Neural Approaches to Recommender systems and a hands on workshop on ML on Spark along with <a href="https://medium.com/u/8d9de2c4968a">Rohan Rao</a>. I’m briefly going to describe both of them.</p><blockquote>TL; DR, For notebooks of my talks, visit <a href="https://github.com/binga/DataHackSummit_2017">https://github.com/binga/DataHackSummit_2017</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bL-d82rf4bYQSVkSSJ5cAQ.jpeg" /></figure><h4>Hack Session</h4><p>My hack session was about using neural network based tools and techniques for solving problems involving generating recommendations. Traditionally, collaborative filtering and matrix factorization techniques were used to solve these problems. In the last couple of years, this trend has been changing. Due to the massive success of effectively training deep neural nets, new approaches have been developed by leveraging the tools and modeling flexibility from the Deep Learning ecosystem.</p><p>We used the Keras library to build a simple dot product based recommender system and slowly build on top of that. The flexibility of layers in deep learning ecosystem allows us to add arbitrary number of layers and hence capture complex relationships in the data. Often, we have side information such as age, gender, number of transactions, number of logins etc. which we’d be interested in using in generating recommendations. This is also covered and the complete code is available <a href="https://github.com/binga/DataHackSummit_2017/tree/master/Recommendations_HackSession/"><strong><em>here</em></strong></a>.</p><blockquote>Most of this codebase is from the brilliant <strong><em>fast.ai course</em></strong>, <a href="http://course.fast.ai/lessons/lesson4.html"><strong><em>lesson 4</em></strong></a>. You should watch it for detailed explanation.</blockquote><h4>Workshop</h4><p>The workshop was about getting familiar with performing data processing in Apache Spark and subsequently building Machine Learning models on large datasets. First, we familiarised the audience with the Spark’s new DataFrame Scala — API which is very similar to Python Pandas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hhumG9aja1unBx54QahGdQ.png" /><figcaption>The learning curve between Pandas DF user becoming a Spark DF user is very small !</figcaption></figure><p>We identified the top 50 operations data scientists use to deal with data from reading data, filtering rows, column level operations, aggregations etc and compiled a list. After we introduced the API and made people familiar with it, diving into the model building section was straight forward.</p><blockquote><em>50 Scala — Spark DataFrame Exercises for Data Scientists</em><strong> </strong>—<a href="https://docs.google.com/document/d/18lz-oRpNUr_rdYJtf0Z1Tdv_Gxiwe1M_NSUKKPcqe3o/edit#heading=h.aeiews21b0pq">https://docs.google.com/document/d/18lz-oRpNUr_rdYJtf0Z1Tdv_Gxiwe1M_NSUKKPcqe3o/edit#heading=h.aeiews21b0pq</a></blockquote><blockquote><strong>Notebooks here </strong>— <a href="https://github.com/binga/DataHackSummit_2017/tree/master/SparkML_Workshop"><em>https://github.com/binga/DataHackSummit_2017/tree/master/SparkML_Workshop</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lk8qSCdyuyybBIfC3VTItQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hsO0zh_ZDPqzvwnmty0k2g.jpeg" /><figcaption>Left: <a href="https://medium.com/u/8d9de2c4968a">Rohan Rao</a> talking about model building and evaluation. Right: Rohan and myself delivering the talk.</figcaption></figure><p>Overall, the workshop was an enjoyable experience since the audience was super enthusiastic about trying / failing / learning / correcting and most importantly they were asking a lot of questions which motivated us to delve deeper into certain concepts.</p><h3>Interacting with the audience</h3><p>During the conference, I happened to interact with a lot of people and most of the attendees loved the hack sessions since it showed them that using these advanced concepts was only about knowing how to use them but not the inherent complexity. Lack of structured material about these concepts made it difficult to a lot of them. And most of the queries circled around two broad questions. So, I decided to write about them here, for everyone’s benefit.</p><ol><li><strong>How do I get started in Data Science and Machine Learning?</strong></li></ol><p><em>Well, there are so many articles / resources on the internet. However, one of the approach could be to start with analyzing a dataset (at your workplace, kaggle.com/datasets) using Excel. Make a few charts. Make a few inferences. As the data gets bigger, learn tools such as SQL, R, Python. To plot, learn ggplot2 / matplotlib. Always have a dataset, and an end goal in mind. In the process, learn the tools required so that you can figure out the necessary tools along the way. If you start learning tools by reading documentation, there’s a good chance you’ll give up!</em></p><p><em>Kaggle Kernels is an excellent place to put up your ideas and practise. Find a new dataset, experiment, publish, collect feedback and iterate! There’s no one right way to learn data science. You just have to figure out your own way!</em></p><p><em>As you go along, transition into ML. Start with small problems on Kaggle. Figure out similar past competitions. Learn from winners and try applying them!</em></p><p><strong>2. There are so many resources out there. How do I start with deep learning and create impact?</strong></p><p>There are a bunch of online courses teaching Deep Learning. However, the best place to start, arguably, is the course taught by <a href="https://medium.com/u/34ab754f8c5e">Jeremy Howard</a> at <a href="http://course.fast.ai/"><strong><em>fast.ai</em></strong></a>. (For some more theory, visit Stanford Class — <a href="http://cs231n.stanford.edu/">http://cs231n.stanford.edu/</a>) The fast.ai course teaches how to build a classifier end to end. In the process, lose the stigma that DL is difficult. Then slowly learn the intricacies. If you’re interested, dive deep into specifics. Maybe, read a couple of papers along the way to see if you’d be interested in going that route. Try taking up simple problems. Check out these awesome blogs by fast.ai students — <a href="https://github.com/reshamas/fastai_deeplearn_part1/blob/master/resources.md"><em>https://github.com/reshamas/fastai_deeplearn_part1/blob/master/resources.md</em></a></p><p>Most often, it’s about trying / failing / trying / learning / succeeding and having a lot of fun along the way. It’s okay if you fail in understanding a concept. The whole learning process isn’t really rewarding if the concepts are straightforward. ;-)</p><p>Finally, commit to a project you are most interested in. Be it a pet project, kaggle competition, an analysis you are most curious about — anything. Spend some time on it everyday. Everyday! After a few days, you will be surely succeeding no matter what.</p><p>Always remember, <em>the master has failed more than the novice has ever tried!</em></p><p>All the best.</p><p>The conference was a fun place to attend over the weekend and I look forward to more of these knowledge sharing avenues.</p><p>Be sure to check the notebooks <a href="https://github.com/binga/DataHackSummit_2017"><strong><em>here</em></strong></a>. Incase you have any questions, please ask questions on this post. I’ll try my best to answer them. Or shoot me a tweet <a href="https://twitter.com/phanisrikanth33"><strong><em>here</em></strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1b13d00de69" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3rd place winning solution for building a Movie Recommendation Engine for Hotstar]]></title>
            <link>https://medium.com/data-science-analytics/building-a-movie-recommendation-engine-for-hotstar-478fb4b21c17?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/478fb4b21c17</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Sun, 10 Sep 2017 15:05:11 GMT</pubDate>
            <atom:updated>2017-11-23T21:36:18.003Z</atom:updated>
            <content:encoded><![CDATA[<p>IndiaHacks is the annual flagship event of HackerEarth and one of the tracks of the event was Machine Learning. After two rounds of shortlisting through online rounds, from more than 500 participants, 15 participants from all over India were selected for one final onsite round where we had 9 hours to solve the problem. The following is a brief writeup on the same.</p><h4>Problem statement</h4><p>Hotstar is an on demand video streaming service in India. It boasts about having more than a 100 Million users and more than 35,000 hours of content on their platform. Due to the sheer scale of users and content, leveraging user browsing history and thus personalising the content for each user creates a lot of value for Hotstar.</p><p>In this challenge, Hotstar challenged us with building a recommendation system so that they could personalize the user experience and also improve the content consumption on their platform (and hence revenues!).</p><h4>Data</h4><p>They have given us data about ~0.8 million users and 3000 odd movies. These movies are sampled from 11 different genres and 50 different languages spoken in India. 2 Million such user — movie — watch_duration (of each movie) were given. A significantly larger dataset, in the first look. We were asked to predict the list of top 20 movies of ~6200 users with which our models will be evaluated and NDCG@20 is the scoring criterion.</p><h4>Approach</h4><p>A quick exploratory analysis showed that</p><ul><li>More than 90% of users given in our training dataset had watched less than 5 movies. We call them the <em>rare_users</em>.</li><li>More than 70% of the users given in the test dataset fall in the <em>rare_users</em> category.</li></ul><p>The 10% of the non rare_users came out to be around 80,000 users and around 1.3M user-movie interactions. The above two points formed the foundation of the modeling strategy.</p><h4>Modeling</h4><p>My modeling strategy had two parts. One each for the <em>rare_users</em> and the non <em>rare_users</em> segment.</p><ul><li>rare_users: For this set of users, as we have less data, I had a heuristics based model that essentially recommends top movies by the ratio of the watch_time to the duration of the movie. This ratio is the amount of the movie watched by the users, a proxy that I considered as the intent of the user towards the movie. Largely, this model contributed to more than 95% of the final score.</li><li>non <em>rare_users</em>: For 600 users in test set, from 80000 users in train set, I built a collaborative filtering based model using the Keras and TensorFlow backend. This approach creates an embedding vector for each of the user and movie features. Augmenting the model with genre and language alongside the embeddings also helped.</li></ul><p>The heuristics and the CF model runs under a minute on a AWS GPU instance.</p><h4>How this approach can evolve</h4><p>This approach is quite clean and works on real-world datasets since the problem of cold start in many recommendation engines is quite common. To address that issue, the heuristics based approach helps. Slowly, as the users consume more and more content, they are transitioned into the non <em>rare_users</em> bucket and they will be a part of the training process.</p><p>The collaborative filtering approach scales horizontally with the amount of data. Add more GPU memory or add more GPUs, though the approach remains the same.</p><h4>Results</h4><p>The benchmark submission scored about 0.07 NDCG whereas the first approach scores around 0.201. The heuristics based model can be further improved to 0.24 (almost a 3x lift over baseline) with smart sampling (letting go of low frequency movies, letting go of low % watched movies) and arriving at final predictions.</p><p>This model helped me to get to #1 standing on public leaderboard. After a 5-minute short presentation of the jury, the winners were announced and I managed to finish 3rd in the hackathon and carried home a good prize money! :D The other winners had similar approaches but smart heuristics here and there had a decisive impact on the final results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kpjDcZ3qQfoB-We-ExK4iA.jpeg" /><figcaption>The top 10 in Machine Learning track. I am on the extreme left.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ozg59FnOkN9Qa0ULl6BUJQ.jpeg" /><figcaption>Finished as 2nd runner up.</figcaption></figure><h4>Conclusion</h4><p>This edition of IndiaHacks was quite an experience in itself. Hotstar coincidentally organized another challenge in round 1 and that was one of richest datasets I’ve worked with. Looks like their data collection and storage is quite impressive and well aligned with the long-term idea of having analytics and machine learning drive some of their products.</p><p>However, the competition setup could have been way better. Be it data sampling, or the timeframe of the challenge. Had there been more data and a longer timeframe, this problem would have been way more interesting and could’ve had much more value to Hotstar.</p><p>Anyways, I hope folks at HackerEarth will take this feedback into consideration and I look forward to more such interesting datasets, problems that create impact.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=478fb4b21c17" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-analytics/building-a-movie-recommendation-engine-for-hotstar-478fb4b21c17">3rd place winning solution for building a Movie Recommendation Engine for Hotstar</a> was originally published in <a href="https://medium.com/data-science-analytics">Data Science | Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Winning two Machine Learning Challenges in the same month]]></title>
            <link>https://medium.com/data-science-analytics/winning-two-machine-learning-challenges-in-the-same-month-6f36d0ca28a6?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f36d0ca28a6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Fri, 31 Mar 2017 04:16:05 GMT</pubDate>
            <atom:updated>2017-03-31T04:16:05.755Z</atom:updated>
            <content:encoded><![CDATA[<p>February 2017 has been a really good month since I have had an opportunity to work on multiple machine learning challenges, hosted by Analytics Vidhya, and came out winning both of them. I wanted to share my experiences of participating in these challenges.</p><h4><strong>Analytics Roadshow MiniHack</strong></h4><p>The first of these challenges was a mini-hack in which participants had three hours of time to build the most accurate predictive model that predicts surge pricing type for a taxi aggregator so that the matchmaking between the right cabs and right customers is quick and efficient.</p><p>After a quick exploration of data, I encoded the Gender, Type of Cab and Confidence Life Style Index to numerical values so that tree models could consume the data. A 3-hour challenge doesn’t give you too much room to start with feature engineering. Hence, I directly jumped to building models to quickly test what works and what doesn’t. Linear models and Random forests produced around 0.63 and 0.68 on both cross-validation and leaderboard respectively. However, these stood around top 30th percentile. XGBoost, once again outperformed other models and it put me one among top 5 on leaderboard within an hour of the challenge. However, it was difficult to make any improvements to this.</p><blockquote>What didn’t work</blockquote><ul><li>2-way variable interactions especially between var1, var2, var3 didn’t improve the score.</li><li>Feature selection didn’t add any value.</li><li>Ensembling RFs with XGBs produced no good effect.</li></ul><p>In a lot of ways, I was stuck for a long time. 30 minutes to finish line. I dropped to 15th position among 170 participants.</p><blockquote>What worked</blockquote><p>As ensembling wasn’t working, I focused on improving my XGB model. I changed the min_child_weight parameter and voila! my score shot up. I was totally underfitting all the time, damn. I went up by 8 places. 15 minutes to go, I quickly lowered my learning_rate, my cross-validation score went up and it scored 2nd on LB! Falling short of time, I couldn’t add any other model. I was sure I didn’t overfit and I was delighted to see that my model was one of the most stable models on public and private and I scored 1st on private leaderboard.</p><p>Maybe, it ain’t over until it’s over.</p><p>Ref: <a href="https://datahack.analyticsvidhya.com/contest/minihack-machine-learning/lb">https://datahack.analyticsvidhya.com/contest/minihack-machine-learning/lb</a></p><p>Source code here: <a href="https://github.com/binga/AnalyticsVidhya_Minihack">https://github.com/binga/AnalyticsVidhya_Minihack</a></p><p>2 weekends later..</p><h4>MLWare 2— Recommendation Challenge</h4><p>This challenge was about building a model that predicts a given user’s ratings (from 0 to 10 stars) for a given item based on past ratings on other items and/or other information. The dataset approximately had one million data points and contained information about 40000 users, 120 items and the distribution of ratings was good enough to build a good recommendation engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*RVx2JnI6Bc3Nah1Q3f0SiQ.png" /><figcaption>Ratings distribution in the training set</figcaption></figure><p>As we were only given user_id, item_id and rating, there was no scope for feature engineering in this challenge. The data contained about a million data points and hence I split the entire dataset into a train and a validation set without any cross-validation loops. I started off with a collaborative filtering model in Python and a single CF model with 50 factors scored in the top 20%ile on leaderboard. Increasing or decreasing number of latent factors wasn’t contributing to an increase in score. However, ensembling 5 same kind of CF models with different random seed gave a significant boost, almost up to 3% and this model put me 2nd on the leaderboard, at the end of day one (in a 3-day challenge).</p><p>Up next, I built a few LightGBM models to ensemble with the CF model. I used user-user features, user-item features like number of items rated by each user, {mean, median, max, min} rating by each user and all these features helped the LightGBM generalize well.</p><blockquote>What didn’t work</blockquote><ul><li>XGBoost with the same features gave worser scores than LightGBM.</li><li>Addition of item-item features didn’t contribute to the overall score.</li><li>Neural network based collaborative filtering didn’t help improve the score.</li></ul><blockquote>What worked</blockquote><ul><li>Simple CF models and blended average.</li><li>GBM models with user-item features.</li></ul><p>With a weighted ensemble of CF model and LightGBM model, my model was good enough for a 2nd position on the leaderboard. On day 3 of the challenge, <a href="https://medium.com/u/8d9de2c4968a">Rohan Rao</a> who was 3rd on the leaderboard and I decided to team up. Rohan had multiple SVD models and a XGBoost stacker as his best model. This model’s score was close enough to my score but when we teamed up, our submissions correlation was 0.91 which was the most interesting thing. Immediately, we knew an average of both of our best models was going to do well and when we submitted this model, we zoomed to #1 position on the leaderboard. We tried a few things during the last hours of the competition but couldn’t improve the score.</p><p>In the end, we were glad to have maintained enough lead to remain #1 on the private leaderboard. Reference: <a href="https://datahack.analyticsvidhya.com/contest/mlware-2/lb">https://datahack.analyticsvidhya.com/contest/mlware-2/lb</a></p><p>This marked the end of a good month where I ended up winning both the challenges I have taken part in. This is my 4th consecutive win in a hackathon. As I have lost one yesterday on HackerEarth, my good streak has finally come to an end and I decided to write down about the previous ones.</p><p>Well, that’s it for today and I’m looking forward to do some more interesting and exciting work in the coming months.</p><p>Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f36d0ca28a6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-analytics/winning-two-machine-learning-challenges-in-the-same-month-6f36d0ca28a6">Winning two Machine Learning Challenges in the same month</a> was originally published in <a href="https://medium.com/data-science-analytics">Data Science | Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Winning the HackerEarth Machine Learning challenge]]></title>
            <link>https://medium.com/data-science-analytics/winning-the-hackerearth-machine-learning-challenge-2038cc72401c?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2038cc72401c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pattern-recognition]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Mon, 12 Dec 2016 15:41:26 GMT</pubDate>
            <atom:updated>2016-12-12T15:41:26.445Z</atom:updated>
            <content:encoded><![CDATA[<h4>A 2-day experience at Societe Generale, Bengaluru</h4><p>Societe Generale, one of the largest banks in France, in collaboration with HackerEarth, organised Brainwaves, the annual hackathon at Bengaluru on November 12–13, 2016. The theme of the hackathon this year was <strong>“Machine Learning”</strong>. The hackathon had an online qualifier from where 85 top teams out of 2200 registrations from all over India, were selected for the final round. The final round was a 30-hour long hackathon which needed the teams to solve 1 problem out of 3 given problems spanning across transaction fraud detection, image and text analytics. I decided to solve the former since I have had experience working with banking data in multiple firms I have previously worked with.</p><p><strong><em>Brief Approach</em></strong></p><p>For the first problem, we were given millions of historical transactions to find patterns from and use these patterns to find anomalies on future transactions. We quickly skimmed through the data and built our machine learning model to predict the fraud on future transactional data and ranked #1 on the leaderboard. Eventually, we also built dashboards which can be used for proactive real-time monitoring for detection of any kind of new anomalies, or they can also be used to monitor transaction throughput etc. You could think of it like a one stop control centre with a global view of what’s going through the system. One commonly known fraudulent behaviour is, fraudsters try to exploit the system by doing high number of small debits and one large credit, thus swindling the money across countries and exchanges and hence trying to circumvent the defences of the system. This particular kind was quite challenging to incorporate into our machine learning model on and we are glad to have solved it to a good extent in 30 hours. Eventually, we had a good dashboard, a very good model and made an excellent pitch to the jury and ranked 1st amongst 85 teams.</p><p><strong><em>Experiences at the hackathon</em></strong></p><p>The hackathon was very well organised in terms of the quality of problem statements in the online and offline rounds, the way the organising team responded to any queries. It was genuinely surprising to see many mentors walking down to our table, talking to us about our backgrounds and providing us various domain related insights which augmented our model and resulted in higher performance. Even during the late hours none of them really left the place, they would always come and check if we were stalled anywhere and help us directionally so that we make constant progress. Having participated in a lot of hackathons prior to this one, I am very surprised by the energy levels of the mentors at Societe Generale.</p><p>To conclude, I would like to thank HackerEarth, Societe Generale, mentors and most importantly Phani Srinath and Supreeth Manyam for their fantastic work during the weekend. Great work guys! If not for any of the above, I am sure that weekend wouldn’t have been so memorable.</p><p>And yes… we partied long and hard that night! :-)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zU2aB3yAnZaZzAiNNdxrCg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHECCE8XaDdQYFXPfbPrWA.jpeg" /><figcaption>Our team with the amazing mentors and Societe Generale India CEO</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oe-oQHs84Jn1ZyI79wXBNQ.jpeg" /><figcaption>Top 3 teams pose for the customary picture.</figcaption></figure><p>Overall, November 2016 has been a great month since <a href="https://medium.com/u/7810e5f18999">Supreeth Manyam</a> and I have also ended up winning another machine learning contest on CrowdAnalytix. We participated with the moniker <a href="https://www.crowdanalytix.com/contests/extraction-of-product-attribute-values">Popeye For Olive</a>!</p><p>That’s it for now. Stay tuned…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2038cc72401c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-analytics/winning-the-hackerearth-machine-learning-challenge-2038cc72401c">Winning the HackerEarth Machine Learning challenge</a> was originally published in <a href="https://medium.com/data-science-analytics">Data Science | Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Black Friday Data Science Hackathon]]></title>
            <link>https://medium.com/data-science-analytics/black-friday-data-science-hackathon-4172a0554944?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4172a0554944</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Mon, 23 Nov 2015 15:06:18 GMT</pubDate>
            <atom:updated>2015-11-24T14:57:42.593Z</atom:updated>
            <content:encoded><![CDATA[<p>Analytics Vidhya, a community of analytics professionals and data scientists hosted an <a href="http://datahack.analyticsvidhya.com/contest/black-friday-data-hack">online data analytics hackathon</a> on 20–22nd of November, 2015. After my very bad performance on their previous hackathon, I was geared up to do well, try new things and learn during the process.</p><h4>Problem Statement</h4><p><em>The challenge was to predict purchase prices of various products purchased by customers based on historical purchase patterns. The data contained features like age, gender, marital status, categories of products purchased, city demographics etc.</em></p><p>After working on a host of classification problems, a regression problem this time was very refreshing for me and I wanted to make sure I try different approaches to find out various intricacies of how to effectively handle regression problems.</p><h4>Approach</h4><p>As a first step, I did some exploratory analysis work, trying to identify if there are any outliers and any missing values. The dataset size was pretty big as well. This is when I realized that it is a very nice dataset <em>(hunch)</em>. No outliers. Only a couple of columns having missing values and a couple of submissions showed nice correlation between Cross-validation scores and Leaderboard scores. My hunch was validated!</p><p>As usual, I started with linear models and moved soon to Random Forests and XGBoost. One of the unique aspects of this competition was apart from XGBoost, no other model was doing well. Not even marginally close. I guess this happened mainly due to the large number of user_ids and product_ids in the dataset. Simpler models were unable to capture the non-linearity in the dataset.</p><p>Even after extensive tuning of the model hyperparameters, I couldn’t break the 2470 RMSE score barrier which was 8th place on leaderboard. My XGBoost was giving me around a 4-fold cross-validation score of 2477 RMSE. I thought the guys at the top were using something more (than just tuning the model) to reach lower RMSE. You could see that (1,2), (3,4) and (5,6,7,8,9,10) form clusters on leaderboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yPWwQa4l3AzRP0V8EjB8pw.png" /><figcaption>Private leaderboard</figcaption></figure><p>So, we were clearly missing out on some information. So, what did they do? I mentioned it below. Keep reading!</p><p>After trying many more models (explained below), I just averaged 5 XGBs to ensure seed stability and finished 9th on the leaderboard. Finally, capping the highest target to 99.9 percentile and replacing negative predictions to minimum target value in train set improved my score by ~1–2 .</p><h4>What didn’t work for me</h4><ol><li>Linear models and Random Forests — very bad CV (around 2680).</li><li>Tried something new — Regularized Greedy Forests. Gave second best CV score after XGBoost (2544) but not so good to ensemble.</li><li>Tried Neural Networks. But, I couldn’t get a good network configuration in a 3-day hackathon.</li><li>Tried Matrix Factorization. But, I had some infrastructure issues.</li><li>Feature Engineering. Tried creating new features. But, I couldn’t obtain a good CV score — probably due to leakage.</li></ol><h4>What set winners apart and my thoughts</h4><ol><li>3rd and 4th place winners (SRK and Rohan Rao) used good feature engineering to gain significant lead. I should learn that from them</li><li>2nd place winner (Nalin) used Matrix factorization to get a good lead. I tried this too but had some infrastructure issues. In retrospect, this looks to be a great idea because of a large number of levels in the product_id variable and user_id variable. MF captures the feature interactions automatically and thus it proved to be a differentiator. Awesome Nalin.</li><li>1st place winner (Jeeban) used an ensemble of Gradient Boosted Trees and Deep Learning. Well, his neural network wasn’t technically <em>‘deep’ Lol!</em> I had a mini heart-attack when I came to know his neural network structure. It was input x 6 x 1. Neural networks are soo easy! Hehehe. I could have never come up with such a simple structure. But, it worked great. Awesome Jeeban!</li></ol><h4>Learnings, tips and tricks!</h4><ol><li>When you have categorical variables with high cardinality, try (definitely try) matrix factorization. It might work (it actually works more often that not!)</li><li>If you are interested in dealing with large number of levels in categorical variables, don’t step back because of RAM. Try H2o. It’s brilliantly engineered.</li><li>Feature engineering always sets you apart!</li></ol><p>Well, I finished 10th and 9th on public and private leaderboards respectively out of 162 participants. It’s a decent finish, but I think I could have done much better. In retrospect, I am happy that I tried new approaches and it was a weekend well spent.</p><p>Thank you Analytics Vidhya for conducting this hackathon. Come up with nice datasets and we will never disappoint you! <strong>;-)</strong></p><p><strong>For my code: </strong><a href="https://github.com/binga/AnalyticsVidhya_BlackFriday"><strong>https://github.com/binga/AnalyticsVidhya_BlackFriday</strong></a></p><p><strong>The datasets are available at: </strong><a href="https://drive.google.com/folderview?id=0B4-c0K3SnXFcbWFRbjNUaWl1ekU&amp;usp=sharing"><strong>https://drive.google.com/folderview?id=0B4-c0K3SnXFcbWFRbjNUaWl1ekU&amp;usp=sharing</strong></a><strong> </strong>— go ahead and give it a shot if you are interested in exploring the dataset.</p><p>#DataScience #Hackathon #MachineLearning #XGBoost #DeepLearning #MatrixFactorization</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4172a0554944" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-analytics/black-friday-data-science-hackathon-4172a0554944">Black Friday Data Science Hackathon</a> was originally published in <a href="https://medium.com/data-science-analytics">Data Science | Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analytics Vidhya 3.X Hackathon]]></title>
            <link>https://medium.com/data-science-analytics/analytics-vidhya-3-x-hackathon-9f2550b47be6?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f2550b47be6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hackathons]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Tue, 15 Sep 2015 14:26:31 GMT</pubDate>
            <atom:updated>2015-09-15T15:06:08.983Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="http://www.analyticsvidhya.com/"><strong>Analytics Vidhya</strong></a>, a community of analytics professionals and data scientists hosted an online data analytics hackathon on 5th, 6th (weekend) of September, 2015. I have come to know it just a couple of days prior to the hackathon and I quickly registered and cancelled all my weekend plans. While there were a lot of registrations before the hackathon began, there were only about 70 odd participants who participated and submitted solutions.</p><p><em>Problem Statement: Digital arms of banks today face challenges with lead conversion, they source leads through mediums like search, display, email campaigns and via affiliate partners. Here Happy Customer Bank faces same challenge of low conversion ratio. They have given a problem to identify the customers segments having higher conversion ratio for a specific loan product so that they can specifically target these customers.</em></p><p>It was eerily similar to a problem statement I’ve worked with, 2 weeks earlier to this competition. I quickly jumped onto the dataset and noticed that the dataset was quite small and it was easy to run several experiments over the stipulated 2 days of time.</p><p>The evaluation metric of this competition was AUC score. So, I once went through the “rank averaging” section on this awesome <a href="http://mlwave.com/kaggle-ensembling-guide/"><em>ensembling guide</em></a><em>.</em> This article should be read multiple times by any data analyst participating in a data science competition. Also, check out my <a href="https://www.kaggle.com/c/springleaf-marketing-response/forums/t/16055/evaluation-example/90114#post90114"><strong><em>5 minutes to understand AUC</em></strong></a><strong><em> </em></strong>guide I posted on Kaggle forums.</p><p>As a first step, I established the pipeline — reading data, extracting a few basic features, building a simple classifier, generating the predictions file in a couple of hours and then went through the feature extraction as the time progressed. As it was a 2-day hackathon, we had to use the best tools available and snatch the lead. Once again, linear models failed to get me a good score and models like Random Forest worked well. As expected, XGBoost again did phenomenally well on the dataset.</p><p>The crucial points I noted while I worked on this dataset were:</p><ol><li>When a high-cardinality categorical variable like “city” or “state” is given, try to use it. Also, merge all rare levels. Really helps!</li><li>XGBoost has a way to treat missing values. Sometimes it’s better to leave the missing values as-is and allow the algorithm to take a decision on how to deal with them.</li><li>For Random Forests, I treated the missing values as -3.14, just as some other level.</li><li>Use dates whenever given. You could extract day, day of week, month, quarter, year and do forward subset selection for a start!</li></ol><p>In the beginning, I was leading on the leaderboard but soon, others like phani_srinath have taken an insane lead on the LB. I was lucky to figure out my mistake in tuning the XGB and I could improve my score. Somehow, I was unable to get my Random Forest to work well and I was out of tricks. Then came in the defining moment. I tried FTRL, an online linear model (an <a href="http://research.google.com/pubs/pub41159.html"><strong>algorithm</strong></a> Google uses in its ad-prediction engine) and it worked! This came in as a shocking/pleasant surprise and I did weighted ensembling. Inspite of all this, <strong><em>I ended up 2nd on the private leaderboard, losing by a tiny 0.00032., </em></strong>though, I was happy that I lost to an incredibly tough competitor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UecZF2RS8rRgoT8Jt3wvA.png" /><figcaption>Phani Srinath finished 1st and I marginally missed out on the weekend contest!</figcaption></figure><p>Fortunately, a few participants asked the organizers to extend the competition for a few more days and they obliged and extended it for 5 more days. This time I wanted to make sure I give my best shot. To demonstrate how close was the competition, the organizers revealed just the top 2 private scores before the 2nd leg of the competition and this gave me and phani_srinath an unfair lead. So, we gave out our <a href="http://discuss.analyticsvidhya.com/t/winning-solutions-data-hackathon-3-x/3959/4?u=binga"><strong>approach</strong></a> to all the participants on the forum. And everybody picked it up so well that I ended up 10th on the public leaderboard in the weeklong competition. This time I built <strong><em>10XGB models, 5 RF models, 5 FTRL models</em></strong>, averaged them for stability and made sure I ensemble my models well and made a gamble. Fortunately, it worked and Yayy!<strong> I finished highest on the private leaderboard of the weeklong competition and secured 1st position out of ~60 odd participants!</strong> I was only 50% sure about this because my CV scores were pretty strong and I used ensembling, way too much, to effectively fight variance in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aztlDRdiftxRMI-LGCDDFw.png" /><figcaption>This time I finished 1st and Phani Srinath finished 3rd on the weeklong contest.</figcaption></figure><p>I also gave out a small guide, how to effectively cross-validate with XGBoost. See <a href="https://github.com/binga/Analytics_Vidhya_3.X_Hackathon/blob/master/XGB%20Tuning%20guide.md"><strong>here</strong></a>!</p><p>I am happy that I secured 2 podium finishes and 1 win in this hackathon. This makes it 2 prizes in a month. The previous one was <a href="https://medium.com/data-science-analytics/data-science-hackathon-datameet-mumbai-bbf080e3009b"><em>this</em></a>. Very lucky 30 days it has been!</p><p>For other details, check out my <a href="https://github.com/binga/Analytics_Vidhya_3.X_Hackathon"><strong>codes on github</strong></a>!</p><p>Hopefully, I continue to learn more and more, and do well on future competitions. Game on!</p><p>Finally, a shout-out to the <strong>AWESOME ‘T’ trio</strong>,</p><ol><li><a href="http://homes.cs.washington.edu/~tqchen/"><strong>Tianqi</strong></a> — creator of XGBoost</li><li><a href="https://www.kaggle.com/triskelion"><strong>Triskelion</strong></a> — for his brilliant ensembling guide</li><li><a href="https://www.kaggle.com/ggglhf"><strong>Tinrtgu</strong></a> — he deserves an oscar for his beautiful online-lr FTRL code!</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f2550b47be6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-analytics/analytics-vidhya-3-x-hackathon-9f2550b47be6">Analytics Vidhya 3.X Hackathon</a> was originally published in <a href="https://medium.com/data-science-analytics">Data Science | Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science Hackathon — DataMeet Mumbai]]></title>
            <link>https://medium.com/data-science-analytics/data-science-hackathon-datameet-mumbai-bbf080e3009b?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbf080e3009b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Tue, 18 Aug 2015 17:04:53 GMT</pubDate>
            <atom:updated>2015-08-18T17:05:48.899Z</atom:updated>
            <content:encoded><![CDATA[<p>Well, it was a rather ordinary day when I found that there was a <a href="http://www.meetup.com/DataMeet-Mumbai/events/223625039/">hackathon</a> being organized at BSE in South Mumbai and that’s when I along with a couple of my colleagues from Data Science Labs at Housing.com decided to form a team and participate in the weekend hackathon. We were pretty late to the event and we received the data on friday night and we only started working from the afternoon of Saturday. Effectively, we had just above a day to beat the best competitors.</p><p><em>Problem Statement: Propensity model development –the client has a couple of use cases where they have not been able to get 80% response capture in top 3 deciles / &gt;3X lift in the top decile — inspite of several iterations. The expectation here would be identification of any new technique / algorithm (apart from logistic regression), which can help get the desired model performance.</em></p><p>Various kinds of customer data was given to us for utilizing in the data analysis. We were asked to figure out which customers are potential loan takers. The data given to us had a train set and a test set, total amounting to 8 lakh records and various features of each of the customers. The evaluation metric seemed trickier at first. I have often worked with logloss and AUC for binary classification problems. But, this problem was aspiring for higher lift in the top 10% and 30% recommendations.</p><p>We initially started with linear models and observed that the lift we were able to achieve was worser than the benchmark. Therefore, we switched to nonlinear models like decision trees and ensembles like Random Forest. Forests were able to produce much better results. But, due to the inherent nature of forests, the probabilities aren’t uniformly calibrated. So, we had to choose an algorithm that directly optimized the evaluation metric and thus we chose XGBoost (which is by far the most popular tool on Kaggle) which is fast and optimized the right evaluation metric and we observed that the XGBoost was performing way better than forests. As the requirement from the organizers included fast computation and cross-platform support, we stuck with XGBoost instead of creating complex ensembles which are engineering-heavy, and choose to take the feature creation route.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3OVUsFyr5vpeevFxsiEKw.png" /><figcaption>A log of our work</figcaption></figure><p>One of our teammates focused on creation of various features while the others focused on modeling and pushing the accuracy. We were pretty happy to obtain a lift of 5.5x while the benchmark was around 3x. Thus we chose to stick with 1 model!</p><p>On Sunday, we met the organizers and they were pretty happy with the model performance and they gave us 5 more days to increase the accuracy. But, we could only push it by 1% considering the heavy engineering effort required to make a significant progress.</p><p>After 2 weeks of evaluation, I received a mail today from them saying, we stood 2nd on the competition from the total pool of 15 teams which are from various parts of India and from reputed universities like CMU etc.</p><p>I’m pretty happy with the 2nd position finish as this is the first time I ended up winning a Data Science Challenge.</p><p>I hope there are many more to come!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbf080e3009b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-analytics/data-science-hackathon-datameet-mumbai-bbf080e3009b">Data Science Hackathon — DataMeet Mumbai</a> was originally published in <a href="https://medium.com/data-science-analytics">Data Science | Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science and Cars!]]></title>
            <link>https://medium.com/data-science-analytics/data-science-and-cars-2765efec475c?source=rss-b7f3e4fe28b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2765efec475c</guid>
            <dc:creator><![CDATA[Phani Srikanth]]></dc:creator>
            <pubDate>Sat, 18 Oct 2014 17:40:17 GMT</pubDate>
            <atom:updated>2014-10-18T17:40:17.107Z</atom:updated>
            <content:encoded><![CDATA[<p>Is the cars industry leveraging their data?</p><p>Well, data science has several applications for the automotive industry, especially cars. The following are a few:</p><ul><li><strong>Product Development</strong>: Ford has been instrumental in taking a data driven approach into designing new cars by looking at the customer reviews data and several data sources across internet. See [1]</li><li><strong>System and Enviromental impact</strong>: Ford Research once again is harnessing the power of supercomputing and advanced mathematics to arrive at these decisions. See [1]</li><li><strong>Car insurance by telematics</strong>: Discovering the driving patterns of the driver by using the telematics technology and analyzing the data, insurance providers are providing lesser premiums to “good” drivers who drive safely and penalizing rash drivers by increasing premiums. See [2], [3]</li><li><strong>F1 — racing</strong>: Private cloud for each race. 250 sensors attached to the car. Log everything you can. Result: 40TB of data. Analyze. Acceleration and braking patterns at corners. Each second is important. Enough said. See [4], [5]</li><li><strong>Supply Chain and Sales</strong>: And sales as usual. See [6]</li><li><strong>Better experience</strong>: A holistic picture of the above points. See [7]</li></ul><p>References:</p><ol><li><a href="http://corporate.ford.com/news-center/press-releases-detail/pr-ford-wins-2013-informs-prize-for-37903">http://corporate.ford.com/news-c...</a></li><li><a href="http://www.forbes.com/sites/adamtanner/2013/08/14/data-monitoring-saves-some-people-money-on-car-insurance-but-some-will-pay-more/">Data Monitoring Saves Some People Money On Car Insurance, But Some Will Pay More</a></li><li><a href="http://www.insurancetech.com/the-big-data-approach-to-telematics-insurance/a/d-id/1314909">The Big Data Approach to Telematics Insurance | Insurance &amp; Technology</a></li><li><a href="http://www.citeworld.com/article/2366960/big-data-analytics/lotus-f1-team-private-cloud-microsoft-avanade.html">How the Lotus F1 team builds a private cloud for each race</a></li><li><a href="http://www.forbes.com/sites/netapp/2013/09/05/f1-big-data-nethappy/">NetAppVoice: Big Data: The Digital Backbone Of Formula </a>1</li><li><a href="http://techpageone.dell.com/technology/data-center/how-gm-uses-big-data-to-generate-sales/#.VEKjofmSxp0">How GM uses big data to generate sales</a></li><li><a href="https://infocus.emc.com/william_schmarzo/big-data-in-automotive-and-machinery-using-analytics-to-deliver-better-products-and-a-more-fulfilling-driver-experience/">Big Data in Automotive and Machinery: Using Analytics to Deliver Better Products and a More Fulfilling Driver Experience — InFocus</a></li></ol><p>I surely know, I have missed a few. If I find them, I will surely as hell come back and update the post!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2765efec475c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-analytics/data-science-and-cars-2765efec475c">Data Science and Cars!</a> was originally published in <a href="https://medium.com/data-science-analytics">Data Science | Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>