<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roman Zykov on Medium]]></title>
        <description><![CDATA[Stories by Roman Zykov on Medium]]></description>
        <link>https://medium.com/@rzykov?source=rss-404bfdeb7627------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RLl9Jyy8OA1rwOAb1sOJGQ.jpeg</url>
            <title>Stories by Roman Zykov on Medium</title>
            <link>https://medium.com/@rzykov?source=rss-404bfdeb7627------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:19:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rzykov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stay Ahead in the UK Logistics and Customs Industry with CorpSignals]]></title>
            <link>https://medium.com/@rzykov/stay-ahead-in-the-uk-logistics-and-customs-industry-with-corpsignals-91575cc8e130?source=rss-404bfdeb7627------2</link>
            <guid isPermaLink="false">https://medium.com/p/91575cc8e130</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[tracking]]></category>
            <category><![CDATA[uk]]></category>
            <category><![CDATA[logistics]]></category>
            <dc:creator><![CDATA[Roman Zykov]]></dc:creator>
            <pubDate>Tue, 18 Mar 2025 21:22:54 GMT</pubDate>
            <atom:updated>2025-03-18T21:22:54.033Z</atom:updated>
            <content:encoded><![CDATA[<p>In the fast-paced world of logistics and customs, staying ahead of the competition is crucial. With the UK trading over £1 trillion in goods annually, the demand for efficient and reliable services has never been higher. But how do you ensure your business remains competitive in such a dynamic environment? The answer lies in leveraging the right tools to identify opportunities and respond swiftly to changes in the market.</p><p>Introducing <a href="https://www.corpsignals.com/">CorpSignals</a>, an innovative app designed to give logistics and customs operators a strategic edge. By tracking UK companies involved in exporting and importing goods, and monitoring any changes in their trade activities, CorpSignals empowers you to make informed decisions, target the right customers, and stay ahead of the curve.</p><h3>Why You Need CorpSignals</h3><p>Logistics and customs operators face unique challenges in today’s market. From identifying potential customers to staying updated on their trade activities, the ability to access real-time, company-specific data is essential. Here’s how CorpSignals can transform your approach:</p><ul><li><strong>Identify Potential Customers with Precision:</strong><br>Finding companies that export or import goods is key to growing your customer base. CorpSignals provides a targeted list of businesses actively engaged in trade, saving you time and resources in your search for new clients.</li><li><strong>Stay Informed on Trade Activity Changes:</strong><br>Keeping up with shifts in trade patterns — such as new export markets or changes in import volumes — can be daunting. CorpSignals delivers real-time alerts on these changes, ensuring you’re always in the know and can adjust your services accordingly.</li><li><strong>Enhance Customer Retention and Acquisition:</strong><br>By staying proactive and informed, you can offer tailored solutions that meet your clients’ evolving needs. This not only strengthens existing relationships but also positions you as a go-to partner for new businesses entering the trade sector.</li></ul><p>With the UK logistics sector facing rising costs and increasing customer expectations for faster, cost-effective services, having a tool like CorpSignals is no longer a luxury — it’s a necessity.</p><h3>What Tools Are You Currently Using?</h3><p>You might already be familiar with resources like <a href="https://www.uktradeinfo.com/">UK Trade Info</a>, which provides official trade data. While useful, these platforms often fall short in delivering the specific, real-time insights needed to drive your business forward. Here’s why:</p><ul><li><strong>Limited Company-Specific Data:</strong><br>General trade databases typically offer aggregated data at the product or country level, not the detailed, company-specific information you need to target potential clients effectively.</li><li><strong>No Real-Time Alerts:</strong><br>Staying updated on changes in individual companies’ trade activities requires constant monitoring, which these tools don’t support. This can leave you reactive rather than proactive, potentially missing out on key opportunities.</li><li><strong>Inefficient for Customer Acquisition:</strong><br>Relying on manual research or general business intelligence tools can be time-consuming and less effective for identifying trade-involved companies.</li></ul><p>CorpSignals fills this gap by offering a dedicated solution for tracking and monitoring UK companies’ export and import activities, complete with real-time alerts and company-specific insights.</p><h3>How CorpSignals Solves Your Biggest Challenges</h3><p>We understand the pain points you face daily:</p><ul><li><strong>Struggling to Find New Customers:</strong><br>In a competitive market, identifying businesses involved in trade can feel like searching for a needle in a haystack. CorpSignals simplifies this by providing a comprehensive, up-to-date list of companies actively exporting or importing goods.</li><li><strong>Difficulty Keeping Up with Trade Changes:</strong><br>Trade activities are constantly evolving, and staying informed manually is nearly impossible. With CorpSignals’ real-time alerts, you’ll never miss a beat — whether it’s a new trade route or a shift in a company’s import volume.</li><li><strong>Pressure to Stay Competitive:</strong><br>Customers expect faster, more cost-effective services, and your competitors are always looking for an edge. CorpSignals equips you with the insights needed to offer proactive, tailored solutions, helping you stand out in a crowded market.</li></ul><h3>Take the Next Step with CorpSignals</h3><p>Imagine having all the information you need at your fingertips — targeted lists of potential customers, real-time updates on trade activities, and the ability to act swiftly on new opportunities. With CorpSignals, this is your new reality.</p><p>Don’t let outdated methods hold you back. Join the growing number of logistics and customs operators who are using CorpSignals to streamline their operations and drive business growth.</p><p><strong>Ready to see the difference?</strong><br> <a href="https://www.corpsignals.com/">Sign up for a free trial</a> today and discover how CorpSignals can transform your approach to customer acquisition and retention.</p><h3>Why Wait? Start Today!</h3><p>In an industry where timing is everything, having the right tools can make or break your success. CorpSignals is designed to give you the insights you need, when you need them, so you can focus on what you do best — delivering exceptional service to your clients.</p><p>Take control of your business’s future. <a href="https://www.corpsignals.com/">Learn more about CorpSignals</a> and see how it can help you stay ahead in the competitive UK logistics and customs landscape.</p><p><em>CorpSignals: Your Partner in Navigating the Future of Trade.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91575cc8e130" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Tips: From Data Analyst to Startup Co-Founder]]></title>
            <link>https://medium.com/@rzykov/10-tips-from-data-analyst-to-startup-co-founder-af0d98b979e4?source=rss-404bfdeb7627------2</link>
            <guid isPermaLink="false">https://medium.com/p/af0d98b979e4</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-startup]]></category>
            <dc:creator><![CDATA[Roman Zykov]]></dc:creator>
            <pubDate>Mon, 22 Nov 2021 09:36:41 GMT</pubDate>
            <atom:updated>2021-11-22T09:36:41.965Z</atom:updated>
            <content:encoded><![CDATA[<h3>12 Tips: From Data Analyst to Startup Co-Founder</h3><h4>The author of this post went from data analyst to startup co-founder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zfmrg6nU7M0Xj2Vt" /><figcaption>Photo by <a href="https://unsplash.com/@clarktibbs?utm_source=medium&amp;utm_medium=referral">Clark Tibbs</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Early in my career, working as a data analyst, I, like many people, dreamed of doing something significant and valuable for people. I wanted to be creative. I wanted to feel the results of my work, not just study the data. I worked at several startup companies for ten years before co-founding an e-commerce recommendation services company in 2012. In 2020, during the COVID pandemic, I went on sabbatical for a year — wrote a book, released it on Amazon, and left the successful company entirely in the summer of 2021. Now I am developing my next project from scratch.</p><p>I decided to share some tips that came in handy for me when I first started my startup. This post complements chapter 12 (“Challenges and Startups”) [3] of my book.</p><p><strong>Create a product idea for one client.</strong> First, there was the idea, then the product. If you have an idea, the best way to test it is to implement it in your current work for the company’s benefit. For example, my project was born many years before my partners, and I started our company. I was developing recommendation systems as a side project when I was still employed. So I knew right away what to do and how to do it, especially initially. I could create a minimum viable product (MVP) with my eyes closed.</p><p><strong>Find a way to scale the product cheaply to thousands of customers (understanding engineering)</strong>. It is essential to be able to scale a product with minimal effort. To do this, you need to learn the practice of putting models into production. In addition to Python and SQL, it is a good idea to know some compiled programming languages (Java, Scala, C#, C++). Understanding how to use different types of databases and Hadoop is helpful. I found it very helpful to visit the Netflix office, where I received some advice about open source software and Hadoop. It took me about six months to learn and implement Hadoop. I also heard about Spark at an O’Reilly STRATA conference video. We were among the first to implement Spark in production. Life proved that both of these decisions were the right ones. We were able to scale very quickly on cheap hardware. As soon as the load increased, we added new servers to the Hadoop cluster in just one day. The cluster was maintained by two engineers only.</p><p><strong>Do less long-term research, more business</strong>. In a startup, you have to act very quickly and still get guaranteed results. Suppose you have a choice between two ML algorithms. In that case, the first is difficult to develop and theoretically gives better results. The second is simple, but its metrics are average — I will always choose the easier one as step number one. Yes, you can put the first algorithm in the backlog of hypotheses, but you’ll probably change it in a few months.</p><p>I spent two years on one short-term recommendation algorithm [4]. We did many complicated things. Ultimately the simplest version of the algorithm won all A/B tests. Trying to find the perfect algorithm was a waste of time in my case.</p><p><strong>The theory is not always the same as practice.</strong> I’ve spent the last ten years developing recommendation algorithms for e-commerce. Scientific articles on the subject use standard metrics (Precision, Recall, Novelty, Diversity [2]) the link). It would seem, take a scientific paper and do it. This approach works well in computer vision, for example, but not in my field. A purchase is a deferred event. The buyer can make it in a few hours or even days. So there is no direct correlation between accuracy metrics [1] and visitor conversion to buyers.</p><p>Another point — the recommendation algorithm changes user behavior. When you do offline testing on users’ past actions in logs — you don’t consider this. This factor also contributes to the error in predicting online performance in A/B tests with offline testing.</p><p><strong>Getting the metrics right is key to success.</strong> It would seem, pick a standard metric for your ML model, and everything will work out. Unfortunately, it doesn’t. Have you heard about the problems of classification metrics in healthcare? For example, you have two different COVID PCR tests:</p><ul><li>test A yields more false positives</li><li>test B yield more false negatives</li></ul><p>Test A will send more healthy people to quarantine, which will cause economic harm. Test B will miss more sick people, and they will spread the disease. The choice of the test is a complicated question; it will depend on the circumstances. You will face the same choice after A/B testing two different algorithms. For example, I’ve encountered situations in recommendation systems where the algorithm improved the commercial metric, but the recommendations themselves visually looked worse. Such an “improvement” is hard to sell to customers.</p><p><strong>A full-time focus is better than doing a startup in the evenings.</strong> I used to program in the evenings and weekends myself. I even bought myself a second laptop, which I carried with me to my current job, so that I could sometimes program there as well. In the evenings after work, I was getting tired, so there were a lot of mistakes in the code I was writing. I had to fix them all the next day. I found even some of them several years later. So try to find an opportunity to devote yourself fully to your project. It will save you a lot of effort and time in the future.</p><p><strong>A professional data analytics experience is necessary.</strong> With it, it is easier to succeed without attracting significant investments. But for the lack of experience, you will have to pour out money. That is why it is good to work in actively developing companies to learn a lot. My former boss once gave me this advice. And if you have also worked on the side of your potential client, it will be much easier for you to understand their needs. It’s common knowledge that a customer won’t tell you everything during a product interview. Many issues are difficult to understand, even for a client himself. My experience as an analyst in e-commerce has helped me a lot. I created my first startup based on the knowledge gained there.</p><p><strong>A B2B startup is more likely to take off than a B2C startup.</strong> B2B will require less investment, fewer employees, and the average transaction check is much higher there. You need to get far fewer customers to break even in B2B than if you did it in B2C.</p><p><strong>No cargo cult</strong>. It’s easy to copy what others are doing, especially FAANG companies. Your company and internal culture are unique. Simply stringing together templates of any standards such as development, ML model deployment, or product will not work. Any of them are in question; you’re more likely to create your rules based on common sense and appropriate for you.</p><p><strong>Clouds are better for getting started</strong>. Now the cloud is like a Lego constructor — you don’t have to think about hardware and software issues, and it’s relatively easy to scale. I did my first project on rented hardware. I deployed Hadoop there and set up all the computation algorithms. I immediately started creating the second project in the cloud, spending much less time on it.</p><p><strong>Clients want to increase their own sales rather than get data analytics.</strong> An analytical product is much more complex to sell than a system that raises a client’s sales. And suppose this effect is also easy to check, for example, in Google Analytics. In that case, your product will sell like hotcakes. That’s why I did not create an analytics project but went straight to where the data increases sales directly — for example, to recommender systems.</p><p><strong>Self-service analytics.</strong> Try to instill a culture of self-service analytics in your company. No one likes to be bombarded with a deluge of tasks that can be done on a “calculator.” These “monkey tasks” are elementary tasks that employees can do with two or three mouse clicks. You need three conditions to get rid of them:</p><ul><li>A user-friendly interactive analytics system (OLAP, Tableau, Metabase).-</li><li>Minimal level of data quality.</li><li>Trained users.</li></ul><p>Put effort into these three areas. It will pay off well, even in a startup environment where development happens very quickly.</p><h3>References</h3><p>[1] Marco Rossetti, Fabio Stella, Markus Zanker, <a href="https://www.youtube.com/watch?v=HEsG5rTaPwE">RecSys 2016: Paper Session 1 — Contrasting Offline and Online Results when Evaluating Recommendation</a> (2016)<br>[2] Paolo Cremonesi, Yehuda Koren, Roberto Turrin, <a href="https://www.researchgate.net/publication/221141030_Performance_of_recommender_algorithms_on_top-N_recommendation_tasks">Performance of recommender algorithms on top-N recommendation tasks</a> (2010)<br>[3] Roman Zykov, <a href="https://www.amazon.com/dp/B09CVD8T5K">Roman’s Data Science: How to monetize your data</a> (2021)<br>[4] Maxim Borisyak, Roman Zykov, Artem Noskov, <a href="https://arxiv.org/abs/1507.07382">Application of Kullback-Leibler divergence for short-term user interest detection</a> (2015)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af0d98b979e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Another Data Science book?]]></title>
            <link>https://medium.com/@rzykov/another-data-science-book-992cf30e5cff?source=rss-404bfdeb7627------2</link>
            <guid isPermaLink="false">https://medium.com/p/992cf30e5cff</guid>
            <category><![CDATA[data-science-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[romansdatasciencebook]]></category>
            <dc:creator><![CDATA[Roman Zykov]]></dc:creator>
            <pubDate>Thu, 21 Oct 2021 13:53:04 GMT</pubDate>
            <atom:updated>2021-10-24T07:06:42.835Z</atom:updated>
            <content:encoded><![CDATA[<p>Why I decided to write and publish a new Data Science book.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*fkS2jfWKSqX2Mlnj.png" /><figcaption>Paperback, image by Author</figcaption></figure><p>When I started working on the book “<a href="http://topdatalab.com/book">Roman’s Data Science: How to Monetize Your Data</a>,” I wanted to bridge the gap between the interests of business and Data Science. We data scientists are under pressure of high expectations. Many executives expect that implementing ML models will solve almost all of their business problems. We, in turn, are also victims of our expectations. Many are surprised to realize that it only takes 10% of their time to create helpful ML models when entering the profession. Fixing problems and proving that these models benefit the company in production take another 90% of the time.</p><p>So in this book, I decided to show how things are. It will be helpful for managers who assign tasks to data scientists. And for the data scientists who perform these tasks. I hope that through this book, we will be able to get more value out of the data.</p><h3>Book Style</h3><p>I address the book for a wide range of readers. The use of mathematics is kept to a minimum (it is almost absent, except for two chapters), and the programming code is wholly omitted. All this opens up the possibility for a wide range of readers who have no idea about mathematics and programming to become closely acquainted with data analytics and understand what we do.</p><p>When I wrote the book, I wrote it as if I were writing it for myself at the beginning of my career, mentioning mistakes and failed experiments. If there had been a time machine, I would certainly have sent it to myself. Like the anti-hero Biff did in “Back to the Future Part 2”, sending Grays Sports Almanac to himself in the past to make money on the sweepstakes. Would I have made fewer mistakes then? I think just as many, only in a different way :), more interesting.</p><h3>Summary Table of Contents</h3><p>The book consists of 13 chapters, and I wrote it in a general-to-specific format.<br>The first chapter focuses on the most crucial goal of Data Science — data-driven decision-making. Decisions can be made either by a human being given data in a report/dashboard or an interactive system or automatically by ML models trained on past choices. I don’t have a complete decision-making theory, but I put principles that seem particularly important.</p><p>Chapter 2 covers the <strong>main areas of data analytics</strong>: business intelligence, machine learning, and data engineering. It also raises some questions: who really should analyze data, what’s wrong with a metrics sheet on a dashboard, how little we value data engineering, and the conflict of interest between the researcher and the business.</p><p>The third chapter will be beneficial for <strong>startups</strong> — those who are doing analytics from scratch. Here I have tried to outline the basics, from technology selection to management and task management issues, including my experiment with task randomization conducted over several years.</p><p>The fourth chapter is about <strong>analytical tasks</strong> and how to set, validate, and test them; the second part is about the basics of statistics, descriptive analysis, and basic graphs. Additionally, I wrote about my experience with paired data analysis, which I practiced in my startup, and technical debt in large ML projects.</p><p>Chapter five is all about <strong>data</strong>. It is a collection of recipes about data — how to collect it, control quality without spending a lot of effort, and some technical issues.</p><p>Chapter six is <strong>data warehouses</strong>. Why we need them, how data gets there, archiving, some thoughts on Hadoop and Spark, open-source technologies that make storing and processing large amounts of data cheap.</p><p>Chapter seven is about <strong>data analysis tools</strong>. Even though I’ve been programming a lot since I was a kid, I consider spreadsheets (Excel, Google Sheets) and pivot tables are the best inventions for data analysis. I also discuss notepad services, statistical analysis packages here, and my criteria for a sound reporting system, which removes a lot of headaches from our analytical heads.</p><p>The eighth chapter, which contains <strong>basic machine learning</strong> algorithms and everything related to them, correlates very strongly with Andrew Ng’s course on machine learning. I only wrote about the algorithms that I use a lot.</p><p>The ninth chapter is <strong>machine learning in practice</strong>. How to effectively learn ML algorithms, best use Kaggle competitions, Mechanical Turk capabilities for ML; Recency/Frequency are the most potent predictors of user behavior.</p><p>The tenth chapter is devoted to <strong>testing</strong> algorithms in production. I wrote about three approaches to analyze A/B tests: Fisher’s statistics (p-value), Bayesian statistics, and bootstrap.</p><p>The eleventh chapter, data <strong>ethics</strong>, discusses the good and bad uses of data. Admittedly, I have encountered temptations to misuse data, so it is crucial to understand good and evil.</p><p>The twelfth chapter is about <strong>challenges</strong> and startups. It lists some interesting problems that I have worked with during my career and advises those going from data scientist to startup founder.</p><p>The thirteenth chapter — building a <strong>career</strong> — is all about creating a career in data science, just personal experience and observations.</p><h3>Illustrations for the book</h3><p>An illustration precedes each chapter by an artist at my request. The meaning of each one reflects the content of the chapter. The main character is my cat, who slept near me while I was writing the book.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5_FdIH52tQM2w9FZ.png" /><figcaption>Data warehouse keeper, image by Author</figcaption></figure><h3>Additional information</h3><p>The book itself has many references to sources, over 100. I took care of readers and provided the Kindle and paperback versions with QR codes. Just point your smartphone camera, click, and the link opens. A unique Java Script redirection mechanism allowed to correct the broken links without reprinting the book itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*46bewoV71ENHuv8X.png" /><figcaption>QR codes, image by Author</figcaption></figure><p>The <a href="https://topdalalab.com/book">book</a> was released in Kindle and Paperback formats through Kindle Direct Publishing on Amazon on August 16, 2021.</p><h3>Call for reviewers</h3><p>Any comments are welcome! If you want to review the book, you can get it for free. Just fill in the <a href="https://topdatalab.com/book#book_review">form</a> on the book page.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=992cf30e5cff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysis paralysis in Data Science]]></title>
            <link>https://medium.com/@rzykov/analysis-paralysis-in-data-science-eeeb5ec7a4b6?source=rss-404bfdeb7627------2</link>
            <guid isPermaLink="false">https://medium.com/p/eeeb5ec7a4b6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analysis-paralysis]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <dc:creator><![CDATA[Roman Zykov]]></dc:creator>
            <pubDate>Thu, 23 Sep 2021 12:10:41 GMT</pubDate>
            <atom:updated>2021-10-18T12:31:44.429Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1vi0LvE_4qlipV9G.png" /></figure><p>Haste leads to waste. All of the biggest mistakes I have ever made can be explained by the fact that I was in a hurry. For example, 15 years ago I was hired by Ozon.ru (10 bln $ on NASDAQ now) to create an analytical division. Part of my job was to compile a huge weekly summary of the company’s activities, and I had no opportunity to check the information regularly. The pressure from above, as well as the lack of time, meant that the weekly reports were full of mistakes, which took a lot of time and effort to correct.</p><p>Today’s world moves at breakneck speed. But compiling metrics requires careful attention, and that takes time. Of course, this does not mean that the reverse situation — “analysis paralysis,” when an inordinate amount of time is spent on every figure on the sheet — is also desirable. Sometimes the desire to make the right choice leads to precisely this state of analysis paralysis, where it becomes impossible to make any decision. The uncertainty about where the decision will lead is inordinately high, or the framework for making a decision is too rigid. An easy way to fall into analysis paralysis is to approach decision-making in a purely rational, logic-only way.</p><p>Graham Simsion’s book <a href="https://www.amazon.com/Rosie-Project-Don-Tillman-Book-ebook/dp/B009VIZDLI/">The Rosie Project</a> illustrates this idea perfectly. The novel tells the story of Don Tillman, a successful young genetics professor who really wants to find a wife, but has never gotten beyond a first date. So he comes to the conclusion that the traditional way of finding the other half is ineffective, and decides to take a scientific approach to the problem. The first phase of his so-called “wife project” is a detailed 30-page questionnaire designed to screen out unsuitable candidates and identify the ideal couple. Obviously, no one can meet such a list of requirements. Don is then introduced to a girl who does not exhibit any of the traits of his ideal woman. You won’t guess who Don ends up dating.</p><p>I can give you an example from my professional life that also has to do with hypotheses or, more accurately, tests. Imagine that you come up with a new recommendation algorithm that you think is better than the existing one, and you want to test it. So, you compare the two algorithms on ten sites. Four times your new algorithm is better, two times it’s worse, and four times there’s nothing in common between them. Will you abandon the old algorithm in favor of the new one? It all depends on what criteria you developed to make a decision before comparing the two algorithms. Was the new algorithm supposed to be better in all cases? Or only in most cases? In the first case, you are likely to bury yourself in endless iterations of the algorithm, honing it to perfection, especially given the fact that the tests will take several weeks to run. This is a classic case of “analysis paralysis”. As for the second case, the task seems much simpler. But practice tells us otherwise.</p><p>I believe that there must always be an element of informed risk in decision-making; we must be able to make a decision even if we do not have all the information. These days we can’t afford to take the time to think about decisions. The world moves too fast for us to afford that luxury. If you don’t make a decision, someone else a competitor will make it for you.</p><p>This article has been taken from “ <a href="https://topdatalab.com/book">Roman’s Data Science</a>” book.</p><p><em>Originally published at </em><a href="https://topdatalab.com/blog/tpost/arlaohyua1-analysis-paralysis-in-data-science"><em>https://topdatalab.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eeeb5ec7a4b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Analysis with Scala]]></title>
            <link>https://medium.com/data-science/data-analysis-using-scala-f3846602fb0d?source=rss-404bfdeb7627------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3846602fb0d</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Roman Zykov]]></dc:creator>
            <pubDate>Mon, 13 Sep 2021 14:38:14 GMT</pubDate>
            <atom:updated>2021-10-18T12:33:08.835Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learning by example using the maximum information coefficient and the RScala library for visualization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0d19fZ-u4CtNFdNG" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It is very important to choose the right tool for data analysis. On the <a href="http://kaggle.com/">Kaggle.com</a> forums, where international Data Science competitions are held, people often ask which tool is better. R and Python are at the top of the list. In this article, we will tell you about an alternative stack of data analysis technologies, based on Scala programming language and <a href="http://spark.apache.org/">Spark</a> distributed computing platform.</p><h3>How did we come up with it?</h3><p>At Retail Rocket, we do a lot of machine learning on very large data sets. We used to use a bunch of IPython + Pyhs2 (hive driver for Python) + Pandas + Sklearn to develop prototypes. At the end of summer 2014, we made a fundamental decision to switch to Spark, as experiments have shown that we will get 3–4 times the performance improvement on the same park of servers.</p><p>Another advantage is that we can use one programming language for modeling and code that will run on production servers. This was a huge benefit for us, since before we were using 4 languages simultaneously: Hive, Pig, Java, Python. It’s a problem for a small team of engineers.</p><p>Spark supports working with Python/Scala/Java through APIs well. We decided to choose Scala because it is the language Spark is written in, which means that we can analyze its source code and fix bugs if needed. It is also the JVM on which the Hadoop runs.</p><p>I must say that the choice was not easy, since no one in the team knew Scala at the time. It is a well-known fact that to learn to communicate well in a language, you need to immerse yourself in the language and use it as much as possible. So we abandoned the Python stack in favor of Scala for modeling and fast data analysis.</p><p>The first step was to find a replacement for IPython notebooks. The options were as follows:</p><ol><li><a href="https://zeppelin.apache.org/">Zeppelin</a> — an IPython-like notebook for Spark;</li><li><a href="https://github.com/tribbloid/ISpark">ISpark</a>;</li><li>Spark Notebook;</li><li>IBM’s Spark IPython Notebook.</li><li>Apache T<a href="https://toree.apache.org/">oree</a></li></ol><p>So far the choice has been ISpark because it’s simple — it’s IPython for Scala/Spark. It’s been relatively easy to bolt-on HighCharts and R graphics. And we had no problem connecting it to the Yarn cluster.</p><h3>Task</h3><p>Let’s try to answer the question:</p><p><strong>Does average order value (AOV) in your online store depend on static customer parameters, which include settlement, browser type (mobile/Desktop), operating system, and browser version?</strong></p><p>You can do this with <a href="http://en.wikipedia.org/wiki/Mutual_information">Mutual Information</a>. We use entropy a lot for our recommendation algorithms and analysis: the classical Shannon formula, the Kullback-Leibler divergence, Mutual Information. We even submitted a paper on this topic. There is a separate, albeit small, section devoted to these measures in Murphy’s famous textbook [5] on machine learning.</p><p>Let’s analyze it on my own real Retail Rocket data. All data is fully anonymized before doing analysis. Beforehand I copied the sample from our cluster to my computer as a csv file.</p><h3>Data</h3><p>Here we use ISpark and Spark running in local mode, which means that all calculations are performed locally and are distributed among the processor cores. Everything is described in comments to the code. The most important thing is that in output we get RDD (Spark data structure), which is a collection of case classes of type Row, which is defined in the code. This will allow you to refer to fields via “.”, for example, _.categoryId.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a23f68eefdb1a71caf6bf924ff62de91/href">https://medium.com/media/a23f68eefdb1a71caf6bf924ff62de91/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mAFuSu8JoBnH59DX.png" /><figcaption>Data screenshot, Image by Author</figcaption></figure><p>The line above uses the new DataFrame data type added to Spark in version 1.3.0, it is very similar to the similar structure in the pandas library in Python. toDf picks up our Row case class, so we can refer to the field by name.</p><p>For further analysis, we need to select a single category, preferably with a lot of data. To do this, we need to get a list of the most popular categories.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e25b69519e6281b620702c0ab61102c/href">https://medium.com/media/1e25b69519e6281b620702c0ab61102c/href</a></iframe><p>For further work I decided to choose the 128th category.</p><p>Let’s prepare the data: let’s filter the necessary types of operating systems, so as to avoid littering the graphs with noise.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77e43c258c7e31a4b6cb985e199f0781/href">https://medium.com/media/77e43c258c7e31a4b6cb985e199f0781/href</a></iframe><p>Want visualization, let’s do it in HighCharts.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/134039691260949a3e2d35132abbffc2/href">https://medium.com/media/134039691260949a3e2d35132abbffc2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ovwNbB5P68lKKpiHmJwDhA.png" /><figcaption>AOV by OS, Image by Author</figcaption></figure><p>You can use any HighCharts graphs as long as they are supported in Wisp. All charts are interactive.</p><p>Let’s try to do the same thing, but with R.<br>Run the R client and plot a char:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81a15a5c5d6a4d57d21df81890c05377/href">https://medium.com/media/81a15a5c5d6a4d57d21df81890c05377/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/0*Kfc6XjrLC-0u09C7.png" /><figcaption>AOV by OS, Image by Author</figcaption></figure><h3>Mutual Information</h3><p>The graphs show that there is a relationship, but will the metrics confirm this conclusion for us? There are many ways to do this. In our case, we use Mutual Information between the values in the table. It measures the mutual dependence between distributions of two random (discrete) variables.</p><p>For discrete distributions, it is calculated using the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2x676nN-f5Rt8_e8NRjkyA.png" /><figcaption>Image by Author</figcaption></figure><p>But we are interested in a more practical metric — the <a href="http://en.wikipedia.org/wiki/Maximal_information_coefficient">Maximal Information Coefficient</a> (MIC) which requires some tricky calculations for continuous variables. Here is how the definition of this parameter sounds.</p><p>Let D = (x, y) be a set of n ordered pairs of elements of random variables X and Y. This two-dimensional space is partitioned by X and Y grids, grouping x and y values into X and Y partitions, respectively (remember histograms!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tVXuO7pqq6eZQu01s1YSFg.png" /><figcaption>Image by Author</figcaption></figure><p>where B(n) is the mesh size, I∗(D, X, Y ) is the mutual information of the X and Y partitions. The denominator specifies the logarithm, which serves to normalize the MIC to the values of the segment [0, 1]. MIC takes continuous values in the interval [0,1]: for extreme values, it is 1 if there is a dependence, 0 if there is not. What else can be read on this topic is listed at the end of the article, in the list of references.</p><p>The <a href="https://probml.github.io/pml-book/book0.html">book</a> (Machine Learning: a Probabilistic Perspective) calls MIC (mutual information) a 21st-century correlation. And here’s why! The <a href="https://www.cs.ubc.ca/~murphyk/MLbook/figReport-16-Aug-2012/pdfFigures/MICfig4.pdf">graph</a> from [1] and [4] shows 6 dependencies (C to H graphs). Pearson’s correlation and MIC have been calculated for them, and they are marked with the corresponding letters on the chart to the left. As we can see, the Pearson correlation is almost zero, while the MIC shows a correlation (graphs F, G, E). According to [1], MIC detects nonlinear dependencies very well.</p><p>In our case, we are dealing with a MIC calculation where we have a continuous variable AOV and all others are discrete with unordered values, such as browser type. In order to calculate the MIC correctly, we need to discretize the AOV variable. We will use a ready-made solution from <a href="http://www.exploredata.net/Downloads/MINE-Application">exploredata.net</a>. There is one problem with this solution: it assumes that both variables are continuous and expressed in Float values. So we’ll have to trick the code by encoding the values of the discrete variables into Float and randomly changing the order of these variables. To do this, we will have to do many iterations with random order (100), and we will take the maximum MIC value as the result.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa44241deb4e0b47ab4420d887c64eb4/href">https://medium.com/media/fa44241deb4e0b47ab4420d887c64eb4/href</a></iframe><p>For the experiment, I added a random variable with a uniform distribution and the AOV (Average Purchase Value) itself. As we can see, almost all MIC was below the random MIC, which can be considered a “conditional” decision threshold. The AOV MIC is almost unity, which is natural because the correlation to itself equals 1.</p><p>An interesting question arises: why do we see a correlation on the graphs, but the MIC is zero? We can come up with a lot of hypotheses, but most likely for the case of os Family it is quite simple — the number of Windows machines is much higher than the number of others.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b79b9f292e0248b31be2a90d3b424de/href">https://medium.com/media/8b79b9f292e0248b31be2a90d3b424de/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kVnjP_F-21-6qc3imnJBCA.png" /><figcaption># of Windows machines is much higher than others, Image by Author</figcaption></figure><p><strong>Conclusion</strong><br>I hope that Scala will get its popularity among data analysts (Data Scientists). It is very convenient because it is possible to work with a standard IPython notebook + get all the features of Spark. This code can safely work with terabytes of data, you just need to change the configuration line in ISpark, specifying the URI of your cluster.</p><p>Roman Zykov, a co-founder of Retail Rocket and an author of the book “<a href="https://topdatalab.com/book">Roman’s Data Science: How to monetize your data</a>”.</p><p><strong>References</strong></p><p>[1] Reshef, D. N., Y. A. Reshef, H. K. Finucane, S. R. Grossman, G. McVean, P. J. Turnbaugh, E. S. Lander, M. Mitzenmacher, and P. C. Sabeti, “<a href="https://www.researchgate.net/publication/51884204_Detecting_Novel_Associations_in_Large_Data_Sets">Detecting Novel Associations in Large Data Sets.</a>” (2011)<br>[2] <a href="https://minepy.readthedocs.io/en/latest/">Minepy library</a> — Maximal Information-based Nonparametric Exploration.<br>[3] <a href="http://www.exploredata.net/Downloads">MINE Java library</a> with datasets. <br>[4] Kevin Patrick Murphy, <a href="https://probml.github.io/pml-book/book0.html">“Machine Learning: a Probabilistic Perspective”</a>, 2012<br>[5] <a href="https://gist.github.com/rzykov/2c66b2bfffad3136837ab9dc0c22000c">Gist of the code above</a></p><p><em>Originally published at </em><a href="https://topdatalab.com/blog/tpost/1p3lt20od1-data-analysis-using-scala"><em>https://topdatalab.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3846602fb0d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-analysis-using-scala-f3846602fb0d">Data Analysis with Scala</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>