<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alejando Attento on Medium]]></title>
        <description><![CDATA[Stories by Alejando Attento on Medium]]></description>
        <link>https://medium.com/@alejandroAttento?source=rss-41ecd57f5631------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4252oBeCqpansnSSHopDUA.jpeg</url>
            <title>Stories by Alejando Attento on Medium</title>
            <link>https://medium.com/@alejandroAttento?source=rss-41ecd57f5631------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:35:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alejandroAttento/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mobile Gaming News — September 2023]]></title>
            <link>https://medium.com/@alejandroAttento/mobile-gaming-news-september-2023-1cd2664683e6?source=rss-41ecd57f5631------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cd2664683e6</guid>
            <category><![CDATA[mobile-games]]></category>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[game-development]]></category>
            <category><![CDATA[games]]></category>
            <category><![CDATA[mobile-apps]]></category>
            <dc:creator><![CDATA[Alejando Attento]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 14:08:46 GMT</pubDate>
            <atom:updated>2023-10-02T14:08:46.562Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mobile Gaming News — September 2023</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*7EoRUlhqpordJ6ebTOCnNg.png" /></figure><p>September 2023 has been a monumental month in the mobile gaming landscape. From groundbreaking collaborations to major updates in fan-favorite titles, the industry has witnessed a flurry of activity. It has also seen a scandal with one of the most popular game engines on the market, the aftermath of which has left the industry in a scramble. Dive into our comprehensive recap to catch up on the top mobile gaming highlights, trends, and innovations that defined this month.</p><p><strong>Unity’s 2023 Pricing Overhaul: Understanding the Controversy and Developer Backlash</strong></p><p>In a move that sent shockwaves through the gaming industry, <a href="https://www.linkedin.com/company/unity/">Unity</a> , the renowned video-game tool maker, announced significant changes to its pricing model. The company’s decision to charge developers for every new installation of their game after a certain threshold was met with widespread protests. Protests were not only for show, as many prominent developers like Toby Fox the developer of the famous Undertail series took to <a href="https://www.linkedin.com/company/twitter/">Twitter</a> (X) to rant about the changes and how these changes are catastrophic for new game developers.</p><p>One of the most concerned people where indie developers, as some felt it could destroy indie gaming, just like how greed led to the downfall of AAA games. Unity’s initial defense of the fee, stating that “more than 90% of our customers will not be affected,” did not appease the masses as Unity has always championed itself as a platform for the people by the people. The fee structure meant that even successful indie studios could face significant financial burdens.</p><p>Unity has since attempted to address the concerns, as the company is considering changes, including a cap on potential fees, and has clarified that installations counted toward reaching the threshold won’t be retroactive. Unity also emphasized that the new policy aims to generate more revenue from its biggest customers and that the majority of Unity users won’t be affected.</p><p>However, the entire episode serves as a cautionary tale, as it underscores the importance of clear communication and understanding the needs and concerns of your user base, especially in an industry as dynamic and passionate as gaming.</p><p><strong>PUBG: New State collaborates with Tesla for a limited-time event.</strong></p><p>In an exciting fusion of technology and gaming, “PUBG: New State” (<a href="https://www.linkedin.com/company/pubg-corporation/">PUBG Corporation</a>) has announced a unique collaboration with <a href="https://www.linkedin.com/company/tesla-motors/">Tesla</a>, the renowned electric vehicle manufacturer. This partnership is set to introduce a limited-time event that seamlessly integrates Tesla’s innovative spirit into the dynamic battlegrounds of PUBG.</p><p>Players can anticipate a range of exclusive in-game content that draws inspiration from Tesla’s iconic designs. From Tesla-themed skins that showcase the brand’s sleek aesthetics to special challenges that allow players to engage with the brand in novel ways, the collaboration promises a fresh gaming experience. There’s even buzz within the gaming community about the potential introduction of drivable Tesla vehicles within the game, though this remains speculative.</p><p>This collaboration is not just a testament to the growing influence of mobile gaming but also highlights the potential of cross-industry partnerships. By bringing together the adrenaline-packed world of battle royale gaming with the cutting-edge advancements of Tesla, both brands aim to offer players a unique and immersive experience.</p><p><strong>Riot Games announces Wild Rift Horizon Cup with a $500,000 prize pool.</strong></p><p><a href="https://www.linkedin.com/company/riot-games/">Riot Games</a>, the powerhouse behind some of the most popular esports titles, has unveiled its latest competitive venture: the “Wild Rift Horizon Cup.” This announcement has sent ripples through the mobile esports community, not just because of the prestige associated with Riot’s tournaments, but also due to the impressive $500,000 prize pool on offer.</p><p>“Wild Rift,” the mobile counterpart to the immensely popular “League of Legends,” has been steadily gaining traction since its launch. The introduction of the Horizon Cup signifies Riot’s commitment to fostering a competitive ecosystem around the game. Participants can expect a tournament structure that mirrors the high standards set by other Riot competitions, ensuring intense battles and top-tier gameplay.</p><p>The substantial prize pool is a clear indication of the growing importance of mobile gaming in the esports arena. With half a million dollars at stake, teams will undoubtedly bring their A-game, leading to matches filled with strategy, skill, and suspense.</p><p>As details about the tournament format, participating teams, and broadcast schedules emerge, fans worldwide are gearing up for what promises to be a landmark event in mobile esports history.</p><p><strong>Mobile is key to Capcom’s expansion goals in India</strong></p><p><a href="https://www.linkedin.com/company/capcom/">Capcom</a>, a titan in the gaming industry known for iconic franchises like “Street Fighter” and “Resident Evil,” has set its sights on the burgeoning Indian mobile gaming market. India, with its vast population and rapidly growing smartphone penetration, presents a lucrative opportunity for game developers and publishers. The country’s mobile gaming scene has witnessed exponential growth, with local and international titles gaining immense popularity among diverse age groups.</p><p>Recognizing this potential, Capcom aims to tailor its approach to cater to the unique preferences of Indian gamers. This could mean introducing localized content, collaborating with local developers, or even launching India-specific titles. The company’s decision to prioritize mobile gaming in India is a testament to the platform’s dominance and the shift in gaming consumption patterns.</p><p>While Capcom has a strong legacy in console and PC gaming, its pivot towards mobile in the Indian market underscores the changing dynamics of the global gaming industry. As Capcom embarks on this journey, Indian gamers can anticipate a blend of the company’s classic gaming experiences adapted to the mobile platform.</p><p><strong>Console Prices On iOS Games, Fact or Fiction? Resident Evil 4 Remake to Follow Suit.</strong></p><p>The recent announcement regarding the iOS release of “Resident Evil 4 Remake” has garnered significant attention in the gaming community. This title, highlighted during the iPhone 15 unveiling, is set to adopt a console-equivalent price point, reflecting its anticipated high-quality gaming experience.</p><p>The introduction of major AAA games, known for their intensive graphics, to mobile platforms, especially with the launch of the new iPhone, signifies a potential shift in mobile gaming perceptions. The primary concern for many is the cost associated with accessing these high-end console games on mobile devices.</p><p>The “Resident Evil 4 Remake” for iOS comes with a price tag of $59.99, aligning with its console and PC counterparts. An additional DLC for the deluxe version is available for an extra $19.99. This pricing strategy indicates a strong belief from both <a href="https://www.linkedin.com/company/capcom/">Capcom</a> and <a href="https://www.linkedin.com/company/apple/">Apple</a> in the game’s ability to deliver a console-like experience on mobile.</p><p>Historically, mobile versions of games have often been priced lower than their console or PC versions. However, Capcom’s approach, offering a demo version with the full game accessible via an in-app purchase, challenges this norm. The game’s performance, both technically and in sales, will be pivotal in determining the viability of this AAA-to-mobile model.</p><p><strong>India’s Mobile App Revolution: The Rise of Localized App Stores and the Impending Challenge for Global Giants.</strong></p><p>India’s mobile application landscape is witnessing a transformative shift with the introduction of the Indus Appstore by <a href="https://www.linkedin.com/company/phonepe-internet/">PhonePe</a>, a prominent financial service provider. This new storefront is uniquely tailored to cater to the diverse cultural and linguistic needs of the Indian populace. Emphasizing its commitment to localization, the Indus Appstore will support 12 languages at its inception, a significant move considering India’s rich linguistic diversity.</p><p>PhonePe’s vision for the Indus Appstore extends beyond merely being an app distribution platform. It aims to offer a more localized alternative to the dominant app stores, enhancing app discovery and user engagement. The platform’s launch comes at a time when India is projected to have over a billion smartphone users by 2026. Despite this vast market, developers have historically had limited options for app distribution. The Indus Appstore seeks to change this narrative by providing a platform that resonates with local needs and preferences.</p><p>The emergence of alternative app stores like Indus Appstore and mSeva indicates a potential challenge to traditional app stores (Like <a href="https://www.linkedin.com/company/google-play/">Google Play</a> and <a href="https://www.linkedin.com/company/app-store-apple/">App Store</a>). The emphasis on localization and the support for multiple languages underscores the importance of catering to the unique needs of the Indian audience. Losing traction in the Indian market would be a significant setback for giants like <a href="https://www.linkedin.com/company/google/">Google</a>, especially given the country’s growing digital footprint and the vast potential it holds for the future.</p><p><strong>FINAL FANTASY VII EVER CRISIS on Mobile: A Deep Dive into Square Enix’s Latest Venture</strong></p><p><a href="https://www.linkedin.com/company/square-enix/">Square Enix</a>’s ambitious project, “FINAL FANTASY VII EVER CRISIS,” is the talk of the mobile gaming community. This title promises to deliver an all-encompassing experience of the revered FINAL FANTASY VII series, right on your smartphone. Merging elements from various titles, it introduces a fresh combat system, blending the original’s turn-based mechanics with new strategic layers. However, it’s not without its challenges.</p><p>Players have pointed out interface issues, and the current lack of controller support feels like a missed opportunity. The game also delves into the gacha realm, allowing weapon unlocks through a draw system. While this approach adds a layer of strategy, it’s a departure from traditional FINAL FANTASY gameplay. As “EVER CRISIS” continues to evolve, it’s clear that Square Enix is aiming high. The game has immense potential, but it’ll be crucial for the developers to address feedback to ensure it’s a hit with both new and long-time fans.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cd2664683e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unity: A Concerning Move for the Gaming Industry]]></title>
            <link>https://medium.com/@alejandroAttento/unity-a-concerning-move-for-the-gaming-industry-d8ca8660b484?source=rss-41ecd57f5631------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8ca8660b484</guid>
            <category><![CDATA[pricing-strategy]]></category>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[indie]]></category>
            <category><![CDATA[games]]></category>
            <category><![CDATA[unity]]></category>
            <dc:creator><![CDATA[Alejando Attento]]></dc:creator>
            <pubDate>Mon, 18 Sep 2023 12:57:41 GMT</pubDate>
            <atom:updated>2023-09-18T12:57:41.587Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhirdU8b_QgD3cSqtAih5w.png" /><figcaption>Credits for the image to Ryan Lambie</figcaption></figure><h3><strong>What is happening with Unity?</strong></h3><p>Unity has introduced a “Runtime fee” pricing system, beyond their existing subscription model. Now, Unity plans to charge developers based on game installations and activation. This is generating significant discussions and concerns. In this article we are going to explain why this is causing that much noise and how this is going to affect you as a developer and/or gamer.</p><h4><strong>What you have to know</strong></h4><ul><li><strong>Start date</strong>: The changes kick in from 2024–01–01.</li><li><strong>Who’s Impacted?<em>:</em> </strong>It’s tied to your subscription type, total installs and earnings. Unity Personal or Plus subscribers with 200,000 installs or at least $200,000 revenue over the past year will incur this fee. Similarly, Pro or Enterprise subscribers with 1 million installs or $1 million in revenue will also be charged. It’s worth noting these numbers are based on gross revenue, not profit.</li><li><strong>Regional Pricing: </strong>Unity divides countries into Central Market and Emerging Market, charging differently based on the location of the install.</li><li><strong>Install Count: </strong>The more installs from Central Market countries, the less Unity charges per installation. In the case of Emerging Market Countries the cost is always the same.</li><li><strong>Install tracking</strong>: The mechanism to track these installs remains ambiguous, leading to considerable debate, which I’ll delve into shortly.</li><li><strong>Subscription Changes: </strong>As of 2023–09–12, Unity discontinued the Unity Plus Subscription, a favored option for indie developers.</li><li><strong>Versions Impacted</strong>: Not just new, but it seems even previous Unity versions will be affected retroactively. This point is also controversial and I’m going to explain later on.</li><li><strong>Fraudulent Installs:</strong> It’s not unusual for developers to encounter deceitful installations (If you’re not part of the industry this can sound odd, but it happens). Unity claims they can identify fraudulent activities in ads for other services, but that’s not the same as catching install frauds. Many developers are skeptical of Unity’s reassurances.</li></ul><h4><strong>Which are the major concerns?</strong></h4><p>Of course, an increment in the cost is something no company wants. So, I’m not going to focus too much on that subject. But, there are other points which are causing a lot of noise and the industry should be conscious of.</p><ul><li><strong>Short notice</strong>: If your game uses already Unity or you are working on a new project using Unity you got only 3 months of notice. Which basically gives you almost no margin of time to take a decision/action. This is criticized by many companies and needs to be repudiated. Any company can do changes, it is their right to do so. But, when they become a key component to your product at least they should make sure that you, as a customer, have enough time to take decisions and adapt to these changes.</li><li><strong>Installation tracking</strong>: Unity is not clear on how they are going to track these installs. This is one of the most polemic points. Are they going to have a service that tracks installs? If that’s the case I would assume and hope that is something which is currently not part of the engine. Developers should be notified if that’s the case. If this is not the case, which is the source they are going to use? Take into account that they don’t charge you for downloads, the charger you for activation, that means, installing the game and starting it for the first time (Based on their description).</li><li><strong>Potential Game Pricing Impact: </strong>If you’re a company developing games for platforms like PC, PlayStation, or Xbox, where users typically buy your games, the new costs from Unity might be passed on to customers. This could raise game prices, potentially leading to fewer sales, although companies might find ways to absorb or manage these costs. But what about free-to-play games? These games rely on massive install numbers, profiting from ads and in-app purchases like boosters or skins. They often earn mere cents per user, which might be far less than Unity’s new charges. This shift could profoundly impact this segment of the industry, which is not only accessible to all but also greatly enjoyed by many players. These games face a significant risk.</li><li><strong>Terms and Conditions</strong>: When you begin using Unity, you accept their Terms of Service, which includes an agreed-upon pricing model. Did they modify these terms without widespread notice? Or was there always a loophole that permitted them to adjust the pricing retroactively? It appears we need to scrutinize such terms with greater diligence moving forward.</li><li><strong>Subscription changes</strong>: Additionally, Unity has discontinued the Unity Plus subscription, dealing a significant blow to indie game developers. Many outstanding games have come to life thanks to these indie creators, so any impact on them is a concern for the entire industry. Previously, Unity Plus cost $400 annually. With its discontinuation, developers are pushed towards Unity Pro, which is priced at $1,800 yearly — a staggering 450% increase. This puts indie developers, especially those with ongoing or upcoming Unity projects, in a challenging spot. Shifting to a different engine isn’t just about cost; it’s about the extensive time required to master a new platform. As we know, time often equates to money. Given this, many indie developers might not be able to afford the costs associated with such a change.</li></ul><p>Will they reverse their decision on the Runtime fee? It’s doubtful. However, given the industry’s backlash, they’ll likely have to make adjustments and provide clearer explanations on several ambiguous points. Many game companies and almost all the gamers will feel the effects of these changes.</p><p>While Unity has faced challenges with profitability since its beginning, its decision to go public in September 2022 intensified the pressure due to investor expectations. However, is the way they’ve altered their pricing, and how the are pushing that change, truly the best approach? It seems that the company may have significantly harmed its reputation, a setback that will likely be tough to bounce back from. Such a move might deter new projects from choosing Unity, and some current projects may even consider shifting to other engines. Though it’s still early to draw conclusions, history in the gaming industry teaches us that a damaged reputation can indeed come at a considerable price.</p><p>Ultimately, this situation serves as a lesson for us as consumers or clients. How often do we genuinely assess the financial stability of our business partners? Had we taken a closer look at Unity’s challenges with profitability, we might have foreseen that changes were on the horizon. Admittedly, the nature of these changes might have taken many by surprise. I think this is an important point for anyone who is going to start or plan to start any project in the future, no matter the industry.</p><p><strong>References:</strong></p><ul><li><a href="https://unity.com/runtime-fee">https://unity.com/runtime-fee</a></li><li><a href="https://unity.com/pricing-updates?utm_source=twitter&amp;utm_medium=social&amp;utm_campaign=company_global_blog_2023-09-12_faq">https://unity.com/pricing-updates?utm_source=twitter&amp;utm_medium=social&amp;utm_campaign=company_global_blog_2023-09-12_faq</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8ca8660b484" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[XGBoost using JAVA explained]]></title>
            <link>https://medium.com/@alejandroAttento/xgboost-using-java-explained-24c895f9f7dd?source=rss-41ecd57f5631------2</link>
            <guid isPermaLink="false">https://medium.com/p/24c895f9f7dd</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alejando Attento]]></dc:creator>
            <pubDate>Mon, 06 Feb 2023 14:47:54 GMT</pubDate>
            <atom:updated>2023-02-06T14:47:54.032Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*9iQYvLdNXfQBxARzkwr9Pw.png" /></figure><p><strong>Most Data Scientists build their models in Python and R. Only in very specific cases it is necessary to use another language</strong> (a more efficient one). For example, if you want to make very fast inferences (Predictions) on a streaming of data.</p><p><strong>Here we will see a simple example of how we could set up the pipeline for a Machine Learning model implemented with Java.</strong> From extracting the data from .zip files to evaluating the results. Also, you will see the differences between Java and Python when building a model.</p><p>In Python you have many libraries which provide many tools you can use to split your dataset, encode categorical labels or features, etc. That’s not the case with Java. <strong>Java requires you to build all of these functions by yourself.</strong> <strong>Luckily, here you will find a complete example of how to code a Machine Learning pipeline using JAVA.</strong></p><h3>What are we going to do?</h3><p>We will take the Dry Bean Dataset (Link <a href="https://archive.ics.uci.edu/ml/datasets/dry+bean+dataset">here</a>) and build an XGboost model to predict to which class each bean belongs given a set of features. <strong>We will do this using Java and Maven.</strong></p><p>As a quick introduction, <strong>Maven is a build tool that allows us to manage Java applications across different IDEs</strong>. Basically, it defines a specific way of how we should install and import all the libraries, how to structure our application, etc. If you use IntelliJ or Eclipse to develop your Java projects it’s already integrated.</p><p>Also, we are not going to spend much time doing any exploratory analysis, feature engineering, optimizing the model, etc. <strong>The purpose of this article is to show and demonstrate how to build s machine learning processes using Java.</strong></p><h3>Application overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*elCMU7LiAyw1DSevgni8Nw.png" /></figure><p>Here you can see the structure of our application. <strong>Notice the pom.xml file with all the instructions Maven needs to build our application.</strong> Don’t worry if you aren’t used to Maven, you don’t need to know it to understand the rest of the code. This article is not about working with Maven. The reason why I chose to use Maven is that if you’re used to work with Java most likely you know Maven. This will make it easier for you to copy the code and reproduce the results.</p><p>Besides the pom.xml file, let’s look at the structure. First you will see that in the src/main/java folder you have three different folders and the main class. Here is a quick description of these:</p><p><strong>dataModels</strong>: This folder includes classes that will allow us to manage our data within the application. Basically, they define the format in which we are going to store and manage our data. Throughout the article, we will only explain one of them since the concept is exactly the same for the other two.</p><ol><li><strong>RawDatasource</strong>: To store the training data we mentioned above.</li><li><strong>ProcessedData</strong>: To store the results after inferring the bean type.</li><li><strong>ResultData</strong>: To store the aggregated results data.</li></ol><p><strong>mlModels</strong>: Here are our Machine Learning model classes. For this example, there is only one.</p><ol><li><strong>BasicXGBoostModel</strong>: Simple implementation of the XGBoost model.</li></ol><p><strong>utils</strong>: Here we have all the helper functions we will need for our application:</p><ol><li><strong>DataChecks</strong>: This general class will allow us to implement different data checks. For example, it checks for null values. If we have rows with null values they are removed.</li><li><strong>Encoder</strong>: This class allows us to encode any categorical feature or label into numbers. Also, it generates a Json file to be able to decode these values later.</li><li><strong>JsonReader</strong>: This class allows us to read Json files and cast them as dictionaries to access the values.</li><li><strong>ModelEvaluation</strong>: This class runs tests on the model after it has been trained to evaluate its performance. For training and testing data, it assesses general performance as well as performance by bean type.</li><li><strong>Reader</strong>: This class allows us to read files as CSV files.</li><li><strong>TrainTestSplit</strong>: This class allows us to split the data we have between training and testing data.</li><li><strong>TransformToLibsvm</strong>: This class allows us to transform our data into Libsvm format. The XGBoost library requires data to be in that format.</li><li><strong>Unzip</strong>: This class allows us to extract data from .zip files.</li></ol><p><strong>Main.java</strong>: The main class that runs the whole application.</p><p>On the other hand you have the resources folder where we have all the data we need which are not Java classes or the pom.xml file. Here you will find:</p><ol><li><strong>basicXGBoostModelConfig</strong>: A Json file with all the parameters required by the XGBoost model.</li><li><strong>dataset</strong>: We will keep all the data we are working on in this folder.</li><li><strong>DryBeanDataset</strong>: .zip file with the initial training data.</li></ol><h3>The Application</h3><p>Before you look into the code of the application, here is a graph to summaries the steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QX4wupUTq4lMqMsSjpq4LQ.png" /></figure><p>The whole application is managed by the main class, Main.java.</p><h4>Main.java</h4><p>We will step through the different parts of the Main class to understand what this application does step by step. Below you can find the code. As you might notice this application takes only two parameters:</p><ul><li>The path to the zip file with the data to train the model.</li><li>A boolean to indicate if we want the application to print at which step of execution it is.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5171a69d0b173de826dd282c17a84a3/href">https://medium.com/media/e5171a69d0b173de826dd282c17a84a3/href</a></iframe><h4>Unzip.java</h4><p>The first thing this application does it to take the .zip file with the data and extract it into a CSV file. This is the only part of the code I did not write myself, so credit goes to the original author. You can see references in the code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c8869c208e83d576d4e9e3381356e13/href">https://medium.com/media/0c8869c208e83d576d4e9e3381356e13/href</a></iframe><h4>Reader.java</h4><p>Once we have extracted all that data we need to read the CSV file. To do that we are going to use the reader function below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f575983c0bbb1b637b0a9e0f53c5fb3c/href">https://medium.com/media/f575983c0bbb1b637b0a9e0f53c5fb3c/href</a></iframe><p>But, once we did that the question is, in which format do we handle the data inside our application? To solve that we have the RawDatasource class.</p><h4>Reader.java</h4><p>We will read each row of the CSV file using an instance of the RawDatasource class. All these instances will be stored inside an arrayList. Find below the structure of these instances.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66fd3df496f1edfd2578359633f441f0/href">https://medium.com/media/66fd3df496f1edfd2578359633f441f0/href</a></iframe><p>We show only this data model as an example, but the other two work in the same way. At the end we will show the complete code.</p><h4>DataChecks.java</h4><p>Once this is done our application is able to work with the data. The most common task is to perform checks on the data. Here I created a class that we could use to do many checks. In order to keep it short, I applied just a quick check to see if there are any null values. Those rows will be removed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fd7ba9d4adb98ce1077a8a97b677a73/href">https://medium.com/media/7fd7ba9d4adb98ce1077a8a97b677a73/href</a></iframe><h4>Encoder.java</h4><p>Looking at the data you will see that all the features are numerical, but not the labels. Each row is labeled as one of 7 different types of beans like dermason, sira, etc. Basically, each bean has a name, but we need to encode these names into numbers. This is necessary because the XGBoost model does not take anything else than numbers as input, we can’t just pass strings to it. To do so we built an encoder function as below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdf9549381538dc2ec380d3abcb8a817/href">https://medium.com/media/bdf9549381538dc2ec380d3abcb8a817/href</a></iframe><p>In addition to updating the data, these functions will also create a Json file to save how we encoded the data, this is to decode these values afterwards. If not, when we try to infer the type of bean based on the features we would get back a number which represents a class. However, we wouldn’t know what it was. Below you can see an example of the Json file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f351478d34ca183c0f2f5e16f8d94ec/href">https://medium.com/media/4f351478d34ca183c0f2f5e16f8d94ec/href</a></iframe><h4>TrainTestSplit.java</h4><p>Once we have our data cleaned and encoded we need to split it into our testing and training datasets. For this, as for many of the things we did before, we use pre-built functions in Python. With Java, we have to create our own classes to manage it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f345e57c4bc0f8415927cb4908072b03/href">https://medium.com/media/f345e57c4bc0f8415927cb4908072b03/href</a></iframe><p>In this case we split our data 80% for training and 20% for testing.</p><h4>TransformToLibsvm.java</h4><p>Usually at this point we would train the model with our data. But, the xgboost4j library we use to train the XGBoost model requires the data to have a specific format called Libsvm.</p><p>If you are unfamiliar with that format, it’s quite simple to understand. It’s also known as the dense matrix format in some cases. The first column indicates the label of the data. Each of the following elements is preceded by the number of the column. After that, you have the actual value for that column. It becomes more clear in the example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*UnJHurC2_AT53r1tgF82_w.png" /></figure><p>To be able to come to this format we had to build our own class that handles it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8dbe2c5137825fc983648e6ae6b51eb6/href">https://medium.com/media/8dbe2c5137825fc983648e6ae6b51eb6/href</a></iframe><p>Using this we will create two libsvm files, one for the training data and another for the testing data.</p><h4>BasicXGBoostModel.java</h4><p>Now, with the proper format we can go forward and train the model. The parameters for the model are in a Json file as we said above. We are not attempting to optimize parameters here, that is a subject for another article. But, feel free to copy the code, set up the model and play around with the parameters to test out other parameters. These parameters are in the file called basicXGBoostModelConfig.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ada2ccbc547344483e4b0099066157e/href">https://medium.com/media/4ada2ccbc547344483e4b0099066157e/href</a></iframe><p>Given these parameters, we will pass it all to the model class, together with the data, to train the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1b91ae04a475f9632612d700c3e5ffb/href">https://medium.com/media/f1b91ae04a475f9632612d700c3e5ffb/href</a></iframe><p>This part can take more or less time depending on the parameters, the data, the iterations, etc. But, with the default settings, it should be fairly quick. Also, keep in mind that by default we set the epochs to just 50.</p><p>Finally, once the model is trained we save it as a bin file so that we can access it later. That’s basically what would happen in a real-world scenario.</p><p>You will find that model as model.bin once it runs.</p><h4>ModelEvaluation.java</h4><p>Finally, we have to evaluate our model. To do this, we built a class. We will evaluate the performance in general and for all of the individual bean classes. Of course, we are planning to do this for the testing and training datasets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3f608099b5158f50301dc57020971f9/href">https://medium.com/media/d3f608099b5158f50301dc57020971f9/href</a></iframe><p>Once this part runs, we will receive the results of our model. Here you can see the results I got after the first run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*oeN2AKQzGAxfLrBE6brk3Q.png" /></figure><p><strong>Notice that the Default probability is the probability of guessing the right kind of bean randomly given that you have 7 possible types.</strong> As you can see, the model performs quite well.</p><p><strong>Here we should do many more analysis to understand better the model, to do a proper evaluation.</strong> But, the goal of this article is to show how to built a Machine Learning model using Java, not to do all the proper analysis and fine tuning a model requires.</p><p>Finally, you will see that the model returns us the bean class name rather than the associated value we got by encoding. <strong>This is because we saved in a Json file the encoding relations, this allows us to decode the data.</strong> The function calls that Json and translate the encoded value back to the name of the bean type.</p><h3>Conclusion</h3><p><strong>Building a Machine Learning model with Java takes much more time than using Python or R. Also, we will have less resources than we are used to with Python</strong>, all the base libraries like pandas, scikit-learn, etc. to manage data and make data handling easier are not provided, we have to built our own tools. <strong>But, in some circumstances Java, because how fast it is can be the right option for our needs.</strong></p><p>However, you will see that in some cases Python and Java are used together. An example could be training the model with Python, optimizing the parameters, etc. encode that model using a format that allows us to restore it in Java, and regenerate it in Java already trained. In that way we can take advantage of the speed Java has to do inferences, but train the model with Python</p><p>To access the complete code please refer to this <a href="https://github.com/AlejandroAttento/xgboost_java_model/tree/master">link</a>. Also, feel free to reuse this code for your own purposes but please do not forget to give credit.</p><p>If you have questions or problems running this model feel free to reach me out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24c895f9f7dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Information vs. Data - Are you conscious?]]></title>
            <link>https://medium.com/@alejandroAttento/information-vs-data-are-you-conscious-e67a852fae05?source=rss-41ecd57f5631------2</link>
            <guid isPermaLink="false">https://medium.com/p/e67a852fae05</guid>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[information-theory]]></category>
            <category><![CDATA[information]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alejando Attento]]></dc:creator>
            <pubDate>Mon, 28 Nov 2022 14:59:29 GMT</pubDate>
            <atom:updated>2022-11-29T20:15:12.600Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this article we will discuss the difference between data and information. There are many ways to approach this topic. Here, we will explain the main concepts behind them and connect them to our daily lives. We will also cover Information Theory, but don’t worry if there is some math involved. This article is not about math. The goal of this article is to make you conscious of whats surrounds you.</p><p>Please read this article to the end to understand how everything connects. You will not regret it, I can promise that.</p><p><em>The calculations are done with Python, but you can easily follow them using a calculator and the provided equations (Only simple math).</em></p><h3>Informal definition</h3><p>Let’s start with a simple explanation of what data and information are and how they differ. Depending on the author you will find different definitions, more or less detailed. Here I will try to keep it as simple as possible:</p><ul><li><strong>Data</strong>: Data are individual facts, they are not organized, and they are not self-explanatory. For instance, Revenue: 3.800 USD, Month: December, Customers: 14. Each one of these data points doesn’t tell us much, until we try to connect them. That brings us to the next concept.</li><li><strong>Information</strong>: An organized and connected set of facts that can be interpreted to provide insights. For instance: In December we made 3.800 USD of revenue which is expected since in that month we used to have the lowest amount of customers.</li></ul><p>How to organize, connect, and interpret data to get information is a big topic and depending on the situation different techniques can be applied. For now, keep in mind the map below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*rlTc_LcSkHz8y3Z75aOdEw.png" /><figcaption>You have to “mine” your data to get the gold out of it.</figcaption></figure><p>Now that we have an intuition about what data and information are, let’s dig deeper into the differences. What does data not have that information has? Or in other words, what does data need to become information?</p><h3>The three components of information</h3><p>Right now, if I handed you a spreadsheet, you probably would ask me what you are looking at. You would ask me which area that data belongs to, or you would try to connect that data on your own. In all cases, you are trying to figure out the <strong>Context</strong>. If you want to turn data into information, you must understand what the data is referring to. What is happening in that context? Coming back to the example above, if you want to understand if a revenue of 3.800 USD in December is high or not you need to know how much it was in previous months, the previous year, etc.</p><p>Now, not all the data is the same. If I sent you information regarding a technology company it could be the marketing results of a campaign, the financial analysis for the previous year, or any other kind of data. Let’s suppose that you are a financial analyst. You will feel comfortable reading the financial report, you are used to seeing these metrics, you know how they are calculated and how to interpret them. What if I accidentally give you the marketing report? You will feel lost at first. That’s because you may not have the <strong>Domain Knowledge</strong> to read that kind of report. Different areas require different domain knowledge. Even if all areas rely on data at some point to get information, without the right domain knowledge you will have trouble getting there.</p><p>Let’s keep the previous example. You’re a financial analyst. Suppose you need to make a decision regarding the next step for the company. One first option could be to look at data to try to get the information you need to take that decision. But, surprise! The company you are working for has a one-month delay in delivering the data. Depending on the kind of decision you make, one month of data could be crucial. That’s why <strong>Timing</strong> is so critical, data is only useful if it allows us to solve a problem, understand what is happening, if it arrives to late, it’s worthless. As a site comment, if you never worked with data, let me tell you that there are many reasons why data could delay.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*UOv-x2NfY7nsBb37RRBrJA.png" /><figcaption>To get information you have to take into account CONTEXT, DOMAIN KNOWLEDGE, AND TIMING</figcaption></figure><p>By now I hope you have a pretty clear idea of what data and information are, and what you need to do in order to get information from data. Now, come with me to the dark side, where you will find math and the information theory.</p><h3>The Information theory</h3><h4>Introduction</h4><p>What is the Information theory? This theory describes how information is passed over different channels, and how we can measure it no matter which channel is used. A picture contains information, just as a written message or a voicemail does, and we can measure the amount of information contained within them and compare them. The question is, how will we compare that information? With a unique unit to measure it, say hello to the <strong>Bit</strong>. Before explaining the interpretation of a bit, let’s start with an example.</p><p>Let’s say you want to know if a child likes chocolate. What would you do? You can just ask him. That is an example of a binary question. The possible answers are True (Yes) or False (No). That message has a value of 1 bit. But, why 1 bit? Because the question split the possible answers into two equally sized groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*1tR2CyhXpMbWLBXRj29yEw.png" /></figure><p>Let’s continue with another example with more possible answers to make that concept clear. Let’s say that we have 32 blocks. We have to figure out which one is the diamond block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*7j-U9g6d1XLnZzQRlULgZw.png" /></figure><p>In the previous example we had only 2 possible answers, here we have 32. We need to define a binary question that splits the answers into two equal groups to start finding the right block. One example would be: Is the diamond block on the right side? The answer is yes, so we can discard all the other possible answers. As you can see, based on that question we discarded half of all the possible answers. The other half are the answers that are still possible. This kind of question, we will call them “Standard binary questions” in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*uabrIBwshQQrsZ8pJ2zvVQ.png" /></figure><p>By using standard binary questions we can make our second question. Is the diamond block on the upper side? Yes, let’s discard the rest. Notice, again we discarded half of all the possible answers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*-HuQvFsYpfHbs6g2dIGrnA.png" /></figure><p>As you can see, we are getting closer to the diamond block. Let’s speed this up, after doing all the needed standard binary questions that’s how the graph will look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*LnzruZTKfa5WJVKItaXwOg.png" /></figure><p>Now, how many standard binary question did we ask to get to that diamond block? Given that each red line is one question we did 5, you can check it on your end too. So, the value of the information which tells us where the diamond block is, is 5 bits.</p><p>Finally, what is a Bit? <strong>A Bit is one standard binary question, and information is measured in Bits.</strong> In other words, we measure information as the number of standard binary questions we would need to ask in order to get the same result. In this case, we would need to do 5 standard binary questions. Given that each standard binary question adds 1 Bit, by each question I’m 1 bit closer to reach the right answer.</p><p>The example above with 32 possible answers took a bit of time, so suppose we want to find out what is the PIN code of a credit card. Since the code is 4 digits long, we have 10000 possible answers. We can’t follow the approach above if the amount of answers is too large. Luckily, there is a mathematical way to do that. To calculate the amount of bits we have in certain information we can use the equation below, being x the amount of possible answers we have.</p><h4>How to calculate it?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*f4pa8xM-oQWhERRIMkXpeg.png" /></figure><p>Let’s apply it to the example we already analyzed with the blocks.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8c692a8a129188ba7fd5fe0555a9cde/href">https://medium.com/media/f8c692a8a129188ba7fd5fe0555a9cde/href</a></iframe><p>The result is 5, it matches with our previous result.</p><p>Now, let’s apply that equation to find out how many binary questions (bits) we would need to do to get the PIN number of a credit card.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/059c8c6f707dd98822b1e78394590490/href">https://medium.com/media/059c8c6f707dd98822b1e78394590490/href</a></iframe><p>The result is 13.28 bits. We could apply the same equation to any case.</p><h4>How to calculate the amount of bits of an image?</h4><p>If you remember, I told you that with this theory we can calculate the amount of bits for any message, no matter the channel. Let’s give an example with photos. But first, to understand the difference between data and information from a mathematical point of view you don’t need to understand this example. I’m adding it here to show that indeed, we can measure the information in bits for any message no matter the channel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*aLDfX0glyfIb1h2c.jpg" /></figure><p>This is a photo of a cat. From a technical point of view there are many ways to visualize a photo, one of the most popular is RGB. It means that each photo consists of three layers (each layer representing the colors Red, Green and Blue), so every pixel is the result of the combination of these three layers at different intensities. The intensity of each layer, for each pixel, is represented by a number between 0 and 255 (256 possible values). For example, one possible combination for a single pixel could be:</p><p><em>Pixel 1 = 26 * Red + 255 * Green + 0 * Blue</em></p><p>The result of that equation is the color we would see at the image at that pixel.</p><p>Now that we know how an image is built let’s calculate the amount of bits. Suppose a picture has a size of 120 Pixels x 120 Pixels, plus, we have three layers. So, the amount of data points we would have in that image is:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f74f5391977ba80c4ae2be827f746308/href">https://medium.com/media/f74f5391977ba80c4ae2be827f746308/href</a></iframe><p>The result is 43200 data points.</p><p>Now, how many standard binary questions do I need to define the amount of information in only 1 data point given that we have 256 different possible values?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53770bf765460ef8f1b26d70bb6afa49/href">https://medium.com/media/53770bf765460ef8f1b26d70bb6afa49/href</a></iframe><p>If you use the information equation above you will see that we get 8 bits.</p><p>But, we have 43200 data points, not only 1, so we have to multiply these 8 bits by the amount of data points. The complete equation would be:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc173d4ce26e8037a73023eee40d733f/href">https://medium.com/media/dc173d4ce26e8037a73023eee40d733f/href</a></iframe><p>By doing the math we get 345600. This means that image had 345600 bits of information, or in other words, you would need 345600 standard binary questions to define that image.</p><p>Again, if you are not familiar with the technical representation of an image using the RGB system this example could be challenging. You don’t need to understand this one to understand the rest of the article. But, also take a moment to think about it, isn’t it amazing that there is a way to compare the amount of information in a photo with the amount of informatin in a sentence, in a song, or any other channel?</p><h4>Can we do better?</h4><p>Now the funny part begins. Do you remember the example of the blocks? Well, what happens if I just start asking if the diamond block is in the second row? <strong>Perhaps I have already gathered some contextual information which suggests that it’s more likely the block is in the second row</strong>, so that’s why I ask that question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*hkdcacTVThbAlzAO5172Jg.png" /></figure><p>Using the previous equation, I can’t do anything. However, based on the image above, I can see that by asking this question I would do better than using standard binary questions. Why? Because I’m discarding more wrong options than the half I would discard with a standard binary question.</p><p>However, if we change our equation a little bit, we can come up with another equation that will allow us to calculate how much information we gained from that NON-standard binary question we asked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*OmgNh-JR4LFe4ymkKHHGjA.png" /></figure><p>Now you see that our equation depends on two variables. “b” are the possible answers before you ask the question, and a are the remaining answers after asking it. Before selecting the second row we had 32 options, now we have 8, so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd49501cfdc8ff75d24ceaa5edf3e075/href">https://medium.com/media/cd49501cfdc8ff75d24ceaa5edf3e075/href</a></iframe><p>If you do the math you will see that the result is 2, and remember that the total amount of bits we calculated for this case was 5. So, that question is twice as good as a standard binary question. In other words, by doing that question we gained twice as much information than doing a standard binary question.</p><p>Now, let’s do it the other way around. What if I have some wrong context indicating that the diamond block is in the third row? Before the question I would still have 32 options. After it, I would have 24, given that neither of the blocks in the third row are diamond blocks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*O0GvDLSfBEsRpGbjAIsFPw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9acc1d65ef7cc1d5092b3f8c647900a8/href">https://medium.com/media/9acc1d65ef7cc1d5092b3f8c647900a8/href</a></iframe><p>The result is 0.41, which means that the wrong context led us to ask a faulty question which is worth less than 1 standard binary question.</p><p><strong>As we can see here, a right context can help us get closer to the answer faster, but a wrong context can also slow down our journey a lot.</strong></p><h4>How much information does a question add?</h4><p>Until now we worked only with standard binary question. With the examples above we showed that given the context, we could come up with other questions that perform better or worse than standard binary questions. But, can we generalize how much information we can gain from a question? Yes, and for this we will do a little bit of math. Please pay attention to this.</p><p>If you look closer at the second equation you will see that the argument for the logarithm is an inverted probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*8pBHi8WLFm5-XiOHauXpdg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*K1UjJHAVuGXOKNX1KlWZKQ.png" /></figure><p>That P to the power of negative 1 means inverted probability, don’t pay too much attention to it. What is important to notice here is that if we turn it around we get the probability of the remaining answers given all the answers. That means, that by doing a little bit of math, we can rewrite the equation as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*w_DkdPGDOS_xOil9pzn2vA.png" /></figure><p>And as we said that a/b is a probability we can simplify that equation into</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*shw1ybe-XU7xf6PXZc7i5g.png" /></figure><p>By doing all this work we can plot a graph which returns the value in bits for any possible question based on that probability.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c070f88885fdefefecb79a26390734f/href">https://medium.com/media/1c070f88885fdefefecb79a26390734f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*MzXDzaGef1tphgmhW6egwA.png" /></figure><p>From this graph we can see how much information a single question can add, based on the probability. If I ask a question that leaves only a few possible answers behind the probability will be low and the gain will be huge. That’s because we eliminated more than half of the wrong answers, which is what a standard binary question would do.</p><p>Let’s read that graph together.</p><p><em>1st Example:</em></p><blockquote>What does a probability of 100% mean in this context? Let’s use the blocks case again as an example. We have 32 possible answers. What happens if I ask:</blockquote><blockquote>Is the diamond one of these 32 blocks?</blockquote><blockquote>The answer would be yes. However, given that we are not discarting any option by asking that question, the amount of information that question is gaining is 0, as you can see on the graph.</blockquote><p><em>2nd example:</em></p><blockquote>By the time we reach 50% of probability, we are saying that half of the answers were left after our question. The bit gain associated with 50% is 1 Bit. This makes a lot of sense, think about it! In the initial examples we said that 1 Bit is one standard binary question which splits the answers into two equally sized groups. We are saying exactly the same thing here.</blockquote><p><em>3rd example:</em></p><blockquote>Finally, if the probability is less than 50%, in other words, if after our question less than half of all the possible answers remain, we are doing better than the standard binary questions. As a result, our question gains more than 1 bit.<strong> Less answers we leave, more bits we gain.</strong></blockquote><h4>Data vs. Information</h4><p>Now, I ask you, what is data and what is information? Do you remember the definitions with which we started?</p><p>Reread them and come back. Now, let’s relate them to the mathematical definition we made.</p><p>The first thing I have to do is correct one of the equations. I know that I started saying that this was the equation to get the information of a message. Indeed, with that equation you can get the amount of data in a message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*e-O-EV5ZRyLPn1iJFYO3wQ.png" /></figure><p>Using standard binary questions we can get the raw value of data from a message. But, with the right questions, given the <strong>Context</strong>, the <strong>Domain Knowledge</strong>, and the opportune <strong>Timing</strong> we can make that data interpretable in a much easier way. Using the right questions, we can gain much more information from it. But, as much as we can improve, the wrong context will make it much harder for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*_7fUO88WvpqpCoHbVfK9NQ.png" /></figure><p>Do you remember how many bits of information we said that cat photo had? 345600. Do we really need 345600 questions to figure out that this is a cat photo, or can you do it with less?</p><h3>Conclusion</h3><p>In a way, data is a raw resource, like so many others we have on earth, but at the same time it is so different. Every area needs it in one way or the other. These days we are constantly overwhelmed by so much data at the same time, and our workspaces are no exception. The question is, <strong>how can we make information out of that data? Do we need to make out information out of all the data we receive? </strong>This would be impossible, and we didn’t even discuss how deeply we should process each data source to gain more information.</p><p>We might not be able to find an answer to all these questions in this article, but that doesn’t mean we shouldn’t think about them. Take them into consideration and be conscious of what surrounds us. <strong>Being conscious of that is the goal of this article.</strong></p><p>There may be a lag of context, domain knowledge, timing, or all together to be able to accurately extract information from your data. <strong>Our problem is that we are not aware of that most of the time, the first step to improving is being aware, identifying the problem, and finding a solution</strong>. In that way you will be able to get the information you need out of your data.</p><p>Each of us, as members of a society, communicates with others. It can be spoken or written, but also when you present a dashboard or a report to another person you are communicating. In doing so, <strong>do you provide context in addition to the report? Also, do you make sure the person has domain knowledge to understand it, and delivers everything in a timely manner? Only in that way will that person be able to make the most of what you’re presenting.</strong></p><p>I hope that by now, reaching the end of this article, you have become a little bit more conscious of what surrounds you…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e67a852fae05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks doesn't have to be black boxes — Captum Pytorch]]></title>
            <link>https://medium.com/@alejandroAttento/neural-networks-doesnt-have-to-be-black-boxes-captum-pytorch-2b34ec23100f?source=rss-41ecd57f5631------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b34ec23100f</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[captum]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Alejando Attento]]></dc:creator>
            <pubDate>Tue, 20 Sep 2022 13:29:58 GMT</pubDate>
            <atom:updated>2024-05-05T20:47:59.721Z</atom:updated>
            <content:encoded><![CDATA[<h3>Explainable AI— Captum Pytorch</h3><figure><img alt="How Explainable AI (XAI) for Health Care Helps Build User… – Towards AI" src="https://cdn-images-1.medium.com/proxy/1*PbV3bkMfScIx02S5f9xnWg.jpeg" /></figure><h3>Introduction</h3><p>If we compare decision trees against Neural Networks it’s likely that we all will agree that a Neural Network will perform better than a decision tree in general. But, there is a big advantage decision trees have against neural networks, <strong>it’s much easier to follow what they are doing</strong>. From the root, through the nodes until the leaves, we can explain step by step what a decision tree does.</p><p><strong>For a model to be explainable is a big advantage</strong>, everybody who worked at some time as an Data Scientist knows that one of the hardest things is to explain a model to a non-technical audience. <strong>It is hard to convince people to trust your model if they don’t understand it</strong>. But, if your model is explainable and we can tell people exactly which are the criteria the model takes into account, it will be much easier to convince them.</p><h4>Use case</h4><p>In some industries there is a real need to understand how a model works to be able to use it. When a model makes a prediction they need to know why and which features are the most important.</p><p><strong>The banking industry is an example</strong>, let’s say that you build a model which takes information of a user as his credit score, transactions, etc. and based on this it infers the probability of a client to pay back a credit or not. An account executive could use this model to decide if to grant a credit or not, but in order to use it it’s not enough to take into account the output of the model, he will need to understand why the model makes that prediction.</p><p>Given the explanation above we could assume that banks can’t use Neural Networks right? Well, let me tell you that<strong> Neural Networks don’t have to be black boxes</strong>, indeed there are several algorithms out there with which we can view inside a Neural Network, see how it works and understand, among other things, how important each feature is for a prediction.</p><h3>What will we do?</h3><p>We will build a classification neural network on top of the ECG arrhythmia dataset to identify if a heart beat is abnormal or not.</p><p>We will:</p><ul><li>Download the data</li><li>Do a quick exploratory analysis</li><li>Clean it and prepare it to be ingested by an Neural Network</li><li>Build our Neural Network</li><li>Train our model and evaluate it</li><li>Apply different attribution algorithms to understand the importance of our feature, the contribution of each feature to a prediction, and understand the contribution of the different neurons in a layer.</li></ul><p><strong>As you can see we are covering a big amount of large subjects in this article, this means that we can’t go in deep into each one of them. Instead of that I will provide code so that you can do your own test, use it as base, and if you want me to write more about particular subjects covered here or not let me know and don’t forget to subscribe so we can keep in touch for any question you have and you don’t miss any article.</strong></p><h3>Let’s start coding</h3><h4>Initialization</h4><p>First we need to import all the needed libraries</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0afa4b8cfd937e019c1ebb246ad49f3f/href">https://medium.com/media/0afa4b8cfd937e019c1ebb246ad49f3f/href</a></iframe><p>And set up some configurations</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71f262b89209002a46cfb6230895b76f/href">https://medium.com/media/71f262b89209002a46cfb6230895b76f/href</a></iframe><h4>Get and prepare data</h4><p>Now we will import the data, you can get the data from <a href="https://www.kaggle.com/datasets/sadmansakib7/ecg-arrhythmia-classification-dataset">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3a82cf3a914282ba5b883b8d19488cc/href">https://medium.com/media/f3a82cf3a914282ba5b883b8d19488cc/href</a></iframe><p>Of course, as in each model, <strong>before starting it it’s important to understand the data we have, not only doing exploratory analysis, also understanding and consulting with experts in this area</strong>, in this case a doctor specialized in cardiology. Given that this is a demonstration we are going to do a simple exploratory analysis, but take this into account when you are working on actual models.</p><p>Also, we will redefine the target, this dataset doesn’t classify only between normal and abnormal heart beats. It classifies based on the type of anomaly, but to keep it simple we will only classify between normal and abnormal.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1997deb384da04b2d85dfa532a83222/href">https://medium.com/media/e1997deb384da04b2d85dfa532a83222/href</a></iframe><p>Also, this dataset allows us to connect the heart beats to patience, so that we can build models on a patience level rather than an heart beat level. But, as we are going to analyze individual heart beats we can drop the related column.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9119e61126c4f6b97d240eea438b1894/href">https://medium.com/media/9119e61126c4f6b97d240eea438b1894/href</a></iframe><h4>Exploratory analysis</h4><p>Let’s take an initial view into the data, the shape, and some initial insights using the describe function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2c49ee2fea21f446547eb653af9757c/href">https://medium.com/media/a2c49ee2fea21f446547eb653af9757c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vbU0pjp0AE7fCRi7Ory7iA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c37f822032fac2a7ffe8b0e775fa3932/href">https://medium.com/media/c37f822032fac2a7ffe8b0e775fa3932/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/116/1*CWOserVV0yisJ-1sopxq0w.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/933e3ead6a972b03c8df06c9d398e03c/href">https://medium.com/media/933e3ead6a972b03c8df06c9d398e03c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RS4GhYzp0mEo06QTxSZrDA.png" /></figure><p>First of all, you will find some definitions at Kaggel for the features if you would like to understand them. Notice that we have about 175k samples. A good amount to build a simple model.</p><p>Also, let’s check quickly for nulls. As you will see there are no null values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0989426385efcba67f3e1ed0eff09a64/href">https://medium.com/media/0989426385efcba67f3e1ed0eff09a64/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/69/1*be6xig_GqLoP7SbB9HrNyQ.png" /></figure><p>Now, we want to see how many abnormal heart beats we have in our dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ddd51e793900298ba891b1324fb2b83f/href">https://medium.com/media/ddd51e793900298ba891b1324fb2b83f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*Gev3RMUss2ZpYpSA28V0lw.png" /></figure><p>About 12–13% of our samples are abnormal heart beats.</p><p>Finally, for the exploratory analysis we will draw some pairplots to see the relation between normal and abnormal heart beats. In some cases it’s quite easy to differentiate among them, in others that seem not to be the case. So, we could assume that some metrics are better to predict abnormal heart beats than others.</p><p>Also, we didn’t plot all the metrics against all as this Pairplot would be hard to read here, this doesn’t mean that there is no relation, indeed you could try with your own exploratory analysis to get even more insights and find these relations. As this article is not about exploratory analysis we will not go deeper into it, but I would encourage you to do it.</p><p>Here is also an additional tip when doing pairplots. It doesn’t always make sense to plot all the points, this causes usually ugly graphs which are hard to read. To avoid this limit the points for each class (Normal and abnormal heart beats), this is enough to make out relation between metrics. To do this we create here the pair_plot_sample function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/043a1bf82677316e1ec8fe43f1a5f1d8/href">https://medium.com/media/043a1bf82677316e1ec8fe43f1a5f1d8/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/126dcb3d7f3b9330548a5e2814dfa91f/href">https://medium.com/media/126dcb3d7f3b9330548a5e2814dfa91f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W6QCv_fWyZNzShkOg0WNZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q5XzjqTqFcCrvL_zKwa8vg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SYTSkK35DVh85AJRub2F5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k2VZQPL92uEnNkPNEICVSw.png" /></figure><h4>Model data preparation</h4><p>To get the data ready for the model we need to separate the data frame into a numpy matrix for the features and a numpy vector for the target.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46daccb35c0e0b6170ca5a0096440395/href">https://medium.com/media/46daccb35c0e0b6170ca5a0096440395/href</a></iframe><p>After this, we are going to scale our data using the Min Max scaler.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93d67c521ae9487025c06e938372b819/href">https://medium.com/media/93d67c521ae9487025c06e938372b819/href</a></iframe><p>Than, we need to split our data into training and testing. For this we are going to do a simple 70–30 split. 70% for training, 30% for testing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24363263d509de925898b28059827f77/href">https://medium.com/media/24363263d509de925898b28059827f77/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*41JX_hBv00_Xt1L6twlkfg.png" /></figure><p><strong>In Neural Network frameworks such as Tensorflow or Pytorch (Used here) we work with tensors</strong>, not with arrays, dataframes or any other data type, that’s why we have to transform our data into tensors. Also, given that models used to be trained in batches it is very common to use <strong>Dataloaders</strong>. These are just classes which given all our data will serve it to the model in the batches we define.</p><blockquote>Batches: When you train a model you don’t pass all your data at once to the model, you use to do it in groups, these groups are called batches. This has many advantages. This is usually called batch training.</blockquote><p>In addition, notice that when I define the Dataloader I select to shuffle only the training data, not the testing data. The training data needs to be shuffled to make sure our model generalizes well, but there is no point in shuffling the testing data, even so, it wouldn’t harm the model if you do so.</p><p>Finally, <strong>play around with the batch size if you are not used to working with neural networks</strong>. A lower batch size could increase the model performance but it would extend the training time. Also, it is usual to define the batch size as a power of 2 (This is related to the fact that computers work with the binary system in the background).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa3936c2a77961b99a446c8caba782d4/href">https://medium.com/media/fa3936c2a77961b99a446c8caba782d4/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb557c72cf5ded2185bc2b7679e98f9a/href">https://medium.com/media/eb557c72cf5ded2185bc2b7679e98f9a/href</a></iframe><p>Now we have all our data ready, the question is, what will our model look like? It will be a simple model with two layers, ReLU activation functions and dropout layers to prevent overfitting. The output layer is a sigmoid function as we are doing a binary classification between normal and abnormal heart beats. The optimization algorithm we will use is Adam and the loss function is the Binary Cross entropy function. The architecture of this model isn’t fancy, but it is maybe a little too big for our case, even so, we will keep it in this way because it will help us to make our points when we come to Explainable AI.</p><p><strong>Feel free to use this code as a base and play around with it, this is the only way to learn about Neural Networks.</strong> Fun fact, when I started my first time building a Neural Networks with Pytorch I had actually a hard time finding good examples for what I wanted to do, hopefully it’s easier for you given this example.</p><p>Coming back to the model class, here is a quick summary of the structure:</p><ul><li><strong>The training step</strong>: Where back-propagation occurs and the weights of the neurons are updated.</li><li><strong>The testing step</strong>: Where we apply our model to testing data to make out it’s performance.</li><li><strong>The forward function</strong>: Here we define how our layers are connected.</li><li>T<strong>he model evaluation function</strong>: This function uses the private _model_metrics function to calculate the final performance of the model once it is finished. Here we used some basic metrics, but you could add more.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a8346f227f14d1d437d2ffd96d7a117/href">https://medium.com/media/1a8346f227f14d1d437d2ffd96d7a117/href</a></iframe><p><strong>Remember if you want me to explain all the details about neural networks, the theory behind, best practices, and how to code them let me know it and subscribe so you don’t miss the next article.</strong></p><p>Let’s create an instance of it and train it.</p><p>Notice that we are training the model on 300 epochs and that at the end we are saving the model. <strong>I strongly recommend saving models, at the end of a training process as during the training. </strong>As our model takes just some minutes to train we will save it only at the end.</p><p>In addition, here you can see how many neurons and layers our model has, <strong>the amount of layers is fixed but the amount of neurons adapts depending on the amount of feature columns we have.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2090b44f526e6be8ae5d1685a675307/href">https://medium.com/media/d2090b44f526e6be8ae5d1685a675307/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a3034f5ebf0f7c8d738c4206420d54a/href">https://medium.com/media/8a3034f5ebf0f7c8d738c4206420d54a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*lSj0c6K--5664DYFa1Drvw.png" /></figure><h4>Model evaluation</h4><p>As each model, once it’s build and trained we need to evaluate our model. We will not go into the details, but at least do a brief review.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46a8ab42ce5d00cadde6a853d09a1532/href">https://medium.com/media/46a8ab42ce5d00cadde6a853d09a1532/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*TlcFx_TZSnCNS-V_EwxNVg.png" /></figure><p>Given these metrics it seems like we have a pretty solid model. We are not going to explain each metric, this is a subject for another article as how to do and proper validation on a model, but as the F1 score is high together with the accuracy we are okay with the model. Now, let’s look into the loss.</p><blockquote>Loss: The loss in this context is a kind of error the model is making, by training it we hope to decrease it, making it as small as possible. Notice that when I say “kind of error” it’s because the definition depends on the loss function we are using for our model, this can vary from model to model.</blockquote><p>Here I’m using a simple trick to make sure we have the same amount of loss data points for training and testing, even so, this comparison is still valid. Also, <strong>those spikes you see there happen because of the optimization algorithm we are using, Adam, if we would use another one as SGD the line would be most likely smoother.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/881b5f5c21b9373c6afaa9e4b7032964/href">https://medium.com/media/881b5f5c21b9373c6afaa9e4b7032964/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*guVhgN3TyWWRrENxmdjm5Q.png" /></figure><p><strong>From this loss graph we can see that both training and testing optimized until the minimum value and don’t decrease more. So, we will stop the training here and use this model.</strong></p><h3>Explainable AI</h3><p>Now that we have a trained model we can come to the core point of this article. The question is, <strong>how can I make my model more understandable?</strong> Which are the most important features? How do the neurons work as part of a layer? To do this we will use the captum library.</p><blockquote>Explainable AI: This is a large concept, but to make it quick it’s a concept which refers to making models understandable. Some models are considered black boxes, as they throw an prediction but we don’t really know how they work internally and why they made that prediction. Neural Networks are one of these “black box” models, but using some algorithms we can throw some light on them.</blockquote><p>The captum library provides us many attribution algorithms, here we will compare three of them and analyze our model based on three points of view.</p><blockquote>Attribution algorithms: These are the algorithms which allow us to better understand our model and “see” what happens inside them.</blockquote><h4>Features importance</h4><p>Let’s say we want to know which are the most important features for our model, to make that out we will use three algorithms and compare them:</p><ul><li>Integrated gradients</li><li>Deep LIFT</li><li>Feature Ablation</li></ul><p>Also, <strong>applying these algorithms can mean a lot of processing</strong>, because of that will run these algorithms with only 3000 randomly sampled data points of our testing dataset.</p><p>Based on this, the first thing we need to do is get these 3000 samples and run the algorithms. <strong>Here we are taking just three algorithms, but there are many more.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c650ddcb795b09bd3196cdffadaf790/href">https://medium.com/media/4c650ddcb795b09bd3196cdffadaf790/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/395e9e74daf9ebd37410b65193608465/href">https://medium.com/media/395e9e74daf9ebd37410b65193608465/href</a></iframe><p>Once this is done, we will normalize all the results so that we can compare the algorithms, understanding to which features they assign the biggest importance. To recap, <strong>here we are looking for the general feature importance, not for any specific prediction</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f50a4bafe4a95788b21722e06f101e2/href">https://medium.com/media/6f50a4bafe4a95788b21722e06f101e2/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/433fa6c132937f50578248b575ab7a3c/href">https://medium.com/media/433fa6c132937f50578248b575ab7a3c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HLeomxvGwiEhO2I4sWH0ig.png" /></figure><p>As we can see, <strong>two of the algorithms seem to have a similar attribution, that’s the case of the Integrated Gradients and the Deep LIFT algorithm. Feature Ablation seems to have some differences for some of the metrics.</strong></p><p>This is because of several reasons. We will not go into the details of these algorithms because it would need one or more articles by itself, but we can make some basic statements. <strong>The way how Integrated Gradient and Deep LIFT work is comparable, they are some of the most used algorithms</strong>, but they are not perfect, usually <strong>it’s considered that they have some limitations</strong>. <strong>Feature Ablation is different in the way it works, one of the general differences you will see is that it assigns much more weight to the features it considers important and almost no weight to the feature it considers not to be important</strong>, this is something you can see in the graph. Even so, for some features as 1_sPeak and 1_rPeak we see some significant differences between Integrated Gradients + Deep LIFT against Feature Ablation.</p><p><strong>The takeaway is, even if algorithms as Integrated Gradients are very used, they are not perfect, they have limitations, that’s why it is so important to understand them, to be able to interpret them and be able to put into context the differences among them.</strong> Look by yourself into these values, where do you see consistency among the algorithms and where not?</p><h4>Prediction attribution</h4><p>Now we will try out something which is very similar to what we did above. <strong>Let’s say we did a prediction and we want to understand why, which features contributed more to that result.</strong> This is not the same as before, here we are looking into specific cases, above we looked for the feature importance in general.</p><p>To do this we will compute the Integrated Gradient algorithm to one single case.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95f1383b327ffb9fd171d3ec59099bd7/href">https://medium.com/media/95f1383b327ffb9fd171d3ec59099bd7/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d78846f85add576c9300f61a84a969a3/href">https://medium.com/media/d78846f85add576c9300f61a84a969a3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABIDqAaH25oxsNUR3MuXMw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/540e96f1123f5b003674e4665d00c519/href">https://medium.com/media/540e96f1123f5b003674e4665d00c519/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*jbaJOa0wHwAz_8nEenyDAw.png" /></figure><p>Here we see the importance of each feature for the first case of our testing dataset. Of course, there are some similarities compared to the feature importance analysis we did above, but it will not match exactly, for each prediction we will get a different attribution vector. Also, we can calculate the delta for this attribution which is the error. This is useful to compare it against other predictions, to see if we are having in some cases an bigger or smaller error. Also, there are ways we can try to reduce it, for instance increasing the amount of iteration of the attribution algorithm with the n_steps parameter. Play around with the parameter, try to reduce the delta and ask yourself. Can you reduce the delta by modifying some parameters? Which is the trade-off you will be facing by reducing the delta?</p><h4>Looking inside the Neural Network</h4><p><strong>There is another thing we can do with attribution algorithms, this is looking inside Neural Networks and making out how they work</strong>, for instance, we could look into a layer and see which neuron seems to be the most important.</p><p>To do this we will look into the second layer of our neural network, compute the attributions, compare it to the weight of each neuron and see what we can make out of it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76e11de52b7a68f0d71e771508e20b6b/href">https://medium.com/media/76e11de52b7a68f0d71e771508e20b6b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b87589a776f202f964597f6ab44aa694/href">https://medium.com/media/b87589a776f202f964597f6ab44aa694/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c8efe862d3340b1e86e96ee76562db3/href">https://medium.com/media/9c8efe862d3340b1e86e96ee76562db3/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5141fcb7093706ea2396a121f67fbc65/href">https://medium.com/media/5141fcb7093706ea2396a121f67fbc65/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Upw1C8f0zI2N6eGHFYKJaA.png" /></figure><p>Here we are looking into all the 65 neurons we have in our second layer. As we can see, <strong>the weights don’t equal directly the attributions</strong>. There are many reasons why this could happen, but for now let’s understand that weights and attribution are not the same, and that in this case one neuron has a much higher importance than the rest.</p><p>This article is already becoming too long and it’s time to end it. <strong>If you read it until here thanks for your time, I hope you were able to take some good concepts and pieces of code from it</strong>. Of course, many concepts have been left out or have been barely explained given the length the current articles already has, but they can be taken in other articles. If you think this article is helpful please let me know and subscribe so you don’t miss any future article.</p><p>Finally, <a href="https://github.com/AlejandroAttento/neural_networks_doesnt_have_to_be_black_boxes-pytorch_captum">here</a> is a link to the complete code at GitHub.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b34ec23100f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finance with Python]]></title>
            <link>https://medium.com/@alejandroAttento/finance-with-python-if-you-know-how-to-analyze-data-with-python-take-advantage-from-it-8cb3ad4b390a?source=rss-41ecd57f5631------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cb3ad4b390a</guid>
            <category><![CDATA[financial-analysis]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[trading]]></category>
            <dc:creator><![CDATA[Alejando Attento]]></dc:creator>
            <pubDate>Mon, 05 Sep 2022 15:51:24 GMT</pubDate>
            <atom:updated>2024-04-30T20:09:24.063Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*5RMa2zas50wbrQgi.jpg" /></figure><h3>Introduction</h3><p>I know, the title is way to long, but seriously. <strong>If you know how to analyze data with python, you have a tremendous advantage to do financial analysis.</strong></p><p>Don’t missunderstand me, you will need to study, but you already know how to use the best tool possible to do this kind of analysis. The question of course is, how to start? Well, don’t sweat it, that’s what I’m going to do here, a gentle introduction into financial analysis with python.</p><p>This is one of several articles I will write about this subject as, of course, covering everything would be impossible in one single article. Also, take into account that we will start from simple manual analysis, but the final target is to be able to define strategies to invest in the stock, crypto market, forex, etc.</p><p>Finally, as I said, the is the first of many articles. So, if you want to join me in this trip throught the world of financial analysis with python don’t forget to subscribe and let me know any comment you have.</p><h3>What will we do?</h3><p>So, first we need to define which will be our final goal with these articles, starting with this one. After finishing this seris of articles you will be able to take well fundamented financial decisions, analyze the risk againste the benefit of each investment, learn to read candle stick charts (Those strange charts traders use which you saw for sure at least at movies) and other metrics used to trade. In addition, with this comes also the theory, understand the difference between a trader and an investor, how they analyze differently the market, understand the different ways you have to invest into the market (Buy assets and sell them later is not the only option).</p><p>But, to reach this we need to start with the basics, so don’t skip this article.</p><blockquote>Asset: We call asset any financial instrument we can use to invest. So, when we say asset we refer in general to stocks, cryptocurrencies, etc.</blockquote><h3>We invest, we don’t bet!</h3><p>It is important that you understand that trading and investments are not something easy to do. It will take you time, study, and practice. Most of the people I know lost more money than they win, but that’s because they didn’t know what they were doing, they don’t have a strategy, they are betting, and belive me, here we don’t bet. We have strategies, measure everything as much as possible, and don’t expect to breack the jackpot putting all our money into one investment. <strong>In general, the goal of each investment strategy is to make profit in a constant way, over time.</strong></p><p><strong>Here, I’m not going to make any financial advisement, I’m going to analyze financial data and show you how to do it by yourself.</strong> We will go from the easiest analysis to more complex ones, in addition to the financial concepts you need to know and will allow you to continue studying more advanced topics. Once you start with financial analysis and see everything which is possible to achieve it is a world changer. But nothing is for free, it takes time and a big effort studying many things. Good thing, we are about to start that journey together.</p><p>Here comes the first lesson, when you are about to make your moves in the financial market think about these three questions:</p><ul><li><strong>Do I have really supporting reasons to make this investment?</strong></li><li><strong>Did I measure and consider the risk of that investment?</strong></li><li><strong>How much of the invested money can I effort to loss in the worst of the cases?</strong></li></ul><p>If you can’t answer them lets reconsider what you‘re about to do.</p><h3>Financial analysis</h3><h4>Types of analysis</h4><p>You saw that I spoke about investments, in finance <strong>usually we distinguis between investors and traders</strong>. Depending the book you look for you will get different definitions, I will try to give you an easy one:</p><ul><li>A trader is a person which buys and sells assets, for instance, in the stock market. He does not hold his positions (stocks) for a long time. Also, he bases his analysis mostly on the price of the stocks and the exchange volume. This kind of analysis is called <strong>technical analysis</strong>, we will start here with technical analysis.</li><li>An investor also buys and sells assets, but he uses to hold these for much longer periods. Also, his analysis is different, his analysis is mostly based on the companies behind the stocks. Investors are interested into how good or bad the company is performing, what are competitors doing, how that industry is doing in general, etc. This kind of analysis is called <strong>fundamental analysis</strong>.</li></ul><p>An trader can look into the performance of the companies to anticipate opportunities, as an investor can look into the current price of an stock to make out the best moment to buy or sell. But, each one of them use mostly their own tools.</p><h4>Financial instruments</h4><p>Something else I want to add here. Buy and sell stocks, crypto currencies or any other assets is not the only way to make money in the market. You have futures, CDFs, and others. But, <strong>if you are not able to make money just by buying and selling stocks most likely you will no able to make profit with other financial instruments</strong>. They add more complexity to our analysis and involve an higher risk (This is something you will able to see in numbers later on following this series).</p><p>Buying stocks, crypto, etc. to sell them later at an higher price is called usually “Spot trading”, remember that name because you will hear it often out there.</p><h3>Let’s code!</h3><h4>Get our data</h4><p>The first question every person which worked with data will have when it comes to analyze the market will be, <strong>from where I get the data?</strong> Don’t worry, the solution is Yahoo finance. You will find the yfinance library using python which will allow you to get that data. Let’s start with an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/0*BR3SyOAgTN0bDPku.png" /></figure><p>We will use the yahoo finance library to download the data for many tickers.</p><blockquote><em>Tickers: Are the codes which identify each one of the assets we have in the market.</em></blockquote><p>Here we are looking for:</p><ul><li>ETH-USD: Ethereum</li><li>BTC-USD: Bitcoin</li><li>EURUSD=X: Euro-USD exchange</li><li>EURJPY=X: Euro-Yen exchange</li><li>GC=F: Gold</li><li>BZ=F: Brent crude oil</li><li>TSLA: Tesla stocks</li><li>NVDA: Nvidia stocks</li><li>^DJI: Down Jones Industrial average</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5f7a017c468eea4cc0975faab3da415/href">https://medium.com/media/e5f7a017c468eea4cc0975faab3da415/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u4nGwTB4An80c90HfCuM1A.png" /></figure><p>Let’s analyze a littel bit the dataframe we got.</p><p>First the structure, you will see that for each one of the tickers we got five values. To understand them better we need to understand how traders analyze the market.</p><h4>Candlestick chart</h4><p>Maybe you saw these graphs somewhere else, this is one way how traders look into the stock prices. <strong>This kind of chart is called candlestick chart.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/0*qrPwEtGhtJ3nT77p.png" /></figure><p>Each of these candles at the graph represents one period. One common way to analyze prices is looking at the daily prices, but you could also look at the weekly, hourly, or the prices at each minute. But, how to read these candles? Look at the graph below.</p><ul><li>The color of the candle will tell you if in that period the price went up or down, in other words, if the final price of that day was higher or lower than the initial price.</li><li>The thickest part of the candle is called body if the candle is red (Price went down), the upper price of the body is the price when the market opened and the lower price, the price when the market closed. On the other hand, if the price went up, which means our candle is green, the lower part of the body is the price when the market opened and the upper parte the price when the marker closed.</li><li>The thinnest part of the candle are the maximum and minimum prices reached in that period.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*gK8HHk8dpd8IWc6d.jpg" /></figure><p>To make it easier to understand here is an example</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/1*B74mmXNxjbIDIE3cjNKWmg.png" /></figure><p>Here, because of the color, we know that the price went down. The price started at 100 USD and ended at 98 USD. But during that period the price moved up and down. It reached a maximum price of 102 USD and a minimum price of 95 USD. So, the price moved in a range from 95 USD to 102 USD.</p><p>These values mentioned above corresponde to the first values we got from yahoo finance for each ticker.</p><ul><li>Close: The price when the market closed.</li><li>High: The maximum price reached in a period.</li><li>Low: The minimum price reached in a period.</li><li>Open: The pricen when the market opnened.</li><li>Adj Close: This is the closing price after the market closed. To explain it in a simple way, if something happens after the close of the market that affects the prices, for instance, the payment of dividends, that change in the price will be reflected not at the common Close price, it will be reflected here at the adjusted close price.</li><li>Volume: The is the volume of the bought and sold assets, in other words, how many stocks have been bought and sold during the period.</li></ul><p>All these values can be used, but for this first start let’s keep it simple. We will use only the close price.</p><p>BTW: The data we downloaded is daily. But, if you want you can get the same data for each hour, minute, and so on.</p><h4>Null values</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c124a245427b9c1a821c973ab808c655/href">https://medium.com/media/c124a245427b9c1a821c973ab808c655/href</a></iframe><p>Also, you will notice some null values among the dataframe, this is because different markets have different operational days. Crypto currencies can be trade 24 7, that market never closes, that’s why there are no null values for these cases. On the other hand, the stock market doesn’t open during weekends and bank holydays. So, don’t worry, it’s normal to get some null values her.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/132/1*5ZY1Nt5Omt5aDUW03vSkTw.png" /></figure><h4>Overtime performance</h4><p>If somebody comes to us and asks us, which asset performed best during the dates we are analyzing, what would you do?</p><p>Maybe just plotting the closing prices will be a good start, so lets do it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d23d0418599fdc2ffe0a33ae54497013/href">https://medium.com/media/d23d0418599fdc2ffe0a33ae54497013/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*MtRyvYxKBARL_lN7_EedXA.png" /></figure><p>As we can see, some assets, as have a much higher price than the rest, but does this mean that these are better? Well, it comes out that the <strong>prices by themself doesn’t really indicate which assets are performing better than others.</strong> What will us make money is the change of these prices not the value by them self.</p><p>That means, that no matter if the asset is 100 USD or 10000 USD worth, if it increases 10% and we invested 1000 USD we will have at the end 1100 USD.</p><p>So, <strong>one technique to make out which asset performed best is normalizing the values with base 1</strong>. This means, we will transform the graph in a way that you can see the daily performance if you would have invested only 1 dollar, in other words, how much money you would have after each day if you would have invested one dollar into each asset, in that way you can compare them. Let’s do it.</p><h4>Normalizing the values to base value 1</h4><p>First, remember that each asset can have some null values, and they can be different from asset to asset, so we need to apply a column wise calculation to get this transformation. About the transformation, it’s simple. We just divide for each asset all the values by the first closing price we have.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/051409c40b4589207be0572c9912a989/href">https://medium.com/media/051409c40b4589207be0572c9912a989/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*yoVLQ_FqA_ymis05VGzy1w.png" /></figure><p>By taking the last value, for instance for the Down Jones Industrial Average Index, we can see the money we would have left after investing one dollar during that period in this asset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aba8a3eaece0816f66f3bd9d5caaa27f/href">https://medium.com/media/aba8a3eaece0816f66f3bd9d5caaa27f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/163/1*WNQ82qIPQquW6xBxYUFGag.png" /></figure><p>That means, for each dollar we would have invested into DJI, we would get back at the end 0.90 USD, we would have lost money. This last value we get for each asset is called the <strong>investment multiplier</strong>. If you multiply you initial investment to that value you get back the final value of your investment.</p><p>If you would have invested 100 USD instead of 1 USD you can make out you finally money left by calculating:</p><p>Initial investment * Investment Multiplier = Money left</p><p><em>100 USD * 0.9017 = 90.17 USD</em></p><p><strong>Each asset that ended below 1 has a negative performance during that period</strong>, as DJI. The rest, <strong>all the assets which ended up above 1 had a positive performance</strong>, Brent crude oil seems to be outperforming the rest.</p><p>This analysis helps us the understand the past, the profit we would have made, or the losses.</p><h4>Risk vs. Benefit</h4><p>You will see that there is an important relationship each investor will have in mind, the relation between risk and benefit. <strong>Higher benefit means an higher risk</strong>. The question becomes how we measure the benefit and the risk?</p><p>To make that out we will speak about <strong>returns, which is for each stock the percentage change from one day to the next</strong>, let’s calculate it for the Down Jones industral average index.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e891ab269f62cfe5f39d3fc3f1514478/href">https://medium.com/media/e891ab269f62cfe5f39d3fc3f1514478/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*yI3uugZaRhrjBm4gRyOUKA.png" /></figure><p>Basically what you are seeing here is daily percentage change to the previous day. But looking just at this graph it is hard to make out the general return. We know already that normalizing the values is a way to compare assets. Now, we will look into other two ways to calculate the investment multiplier:</p><ul><li>Using prices: Divide the most recent price by the first price.</li><li>Using returns: Using the geometric mean.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd62d43e441d05a10f7e8df491ad6900/href">https://medium.com/media/dd62d43e441d05a10f7e8df491ad6900/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/1*b0LHCMA1yN5MvK2p6Ghvrg.png" /></figure><p>Here we see that from these two methods we get the same values than before. One valid question is why we would like to get the investment multiplier using the returns if the other way is much simpler. The answer is that creating algorithmic trading systems in some cases you will have only the returns, not the actual prices.</p><p><strong>For the benefit we will take only the geometric mean of the daily returns. So, for now out benefit is the geometric mean of the returns.</strong></p><p>The question is which is our risk? There are many methods to calculate risk, we will start with a pretty simple one. The risk is equal to the standard deviation of the returns. Think about it as, then more variation we have in the price than less predictable the next price is. Of course, we are doing some assumptions by defining the risk as the standard deviation, but I will go over it at the end.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6216787966a5d8514e768401a8ddc119/href">https://medium.com/media/6216787966a5d8514e768401a8ddc119/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/158/1*qf9MJHyurf9IjKDlAyWRkA.png" /></figure><p>Now, we will calculate the benefit and the risk for all the assets we analyzed so we can compare them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1282dee9eb9a4ca8b8630f61bb3417a/href">https://medium.com/media/d1282dee9eb9a4ca8b8630f61bb3417a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*7J0vsARJlYDr3bxWUpcaMQ.png" /></figure><p>At the graph we can see the risk against the benefit. If we would base our decision only on top of this analysis, we would discard all the assets with a negative benefit. That mean, let’s discard all the assets below the horizontal line on 0. Two of the assets above 0 are pretty close one to each other, Gold and EURJPY, they have a relative low risk, but also a low benefit, these could be the options for a more conservative investor, who want to run a low risk, making some profit. Then we have Brent crude oil, which has a much higher risk, but the benefit is also higher, this one could be the option for an investor which uses to run a higher risk.</p><p>There is no clear way to say which option is better, we have now a way to compare them, but the choose depends on the risk you want to run and your strategy.</p><p>Also, this definition of risk has some problems:</p><ul><li>It assumes an normal distribution on the data.</li><li>It dosen’t take into account worst case scenarios, high changes in the price in a short periods of time.</li><li>We assume that the risk is stable across the analyzed period of time which is not true. Look at the variability at the returns graph for DJI.</li></ul><p>These are things we are going to solve with more advanced methods in future articles and adding something called the logarithmic returns or log returns.</p><p>We have some topics left which are also part of the basics of financial analysis, we will cover them in the following articles. Don’t forget to subscribe so you get notified once it’s published.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cb3ad4b390a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Markov Chains from scratch]]></title>
            <link>https://medium.com/@alejandroAttento/markov-chain-from-scratch-26023664e6b5?source=rss-41ecd57f5631------2</link>
            <guid isPermaLink="false">https://medium.com/p/26023664e6b5</guid>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[markov-models]]></category>
            <category><![CDATA[markov-chains]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Alejando Attento]]></dc:creator>
            <pubDate>Mon, 22 Aug 2022 14:48:22 GMT</pubDate>
            <atom:updated>2022-08-22T14:54:19.509Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*jKhP_dF52IChIoiE.jpg" /></figure><h3>Introduction</h3><p>If your worked in a data team maybe you heard about Markov Chains, for instance:</p><ul><li>When they apoke about a bayesian process, maybe they mentioned the <strong>Markov Chain Montecarlo (MCMC)</strong> method.</li><li>Maybe your heard about a kind of model called <strong>Hidden Markov Chain</strong>.</li><li>Finally, maybe they mentioned that some time series have M<strong>arkov Properties</strong>.</li></ul><p>These are a few examples, the point is that Markov Chains are used for a lot of different purposes, but the question is, what are Markov Chains about?</p><p><strong>During this article we will give an explanation about what Markov Chains are, some properties, and give some examples. The idea of this article is to be the base for other more advanced articles which will come, showing real application you can do using this. There are a lot of powerful applications, believe me.</strong></p><h3>Example</h3><p>Suppose that you want to predict the weather for the ninth day as in the picture below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*bGRRWwDpwK6EUnaOM8nfrA.png" /></figure><p>The possible options are sunny, cloudy and rain. In theory, you could build a model which considers the weather of all the past days. In this case:</p><p>The weather day 8+ The weather day 7 + … + The weather day 1</p><p>I think you got the idea.</p><p>Now, the question is, are all these days equally meaningful for your prediction? Well, it’s likely that this won’t be the case. Indeed,<strong> it could be that the last day will be the most important for our prediction</strong>. In our case to predict the weather for day 9, day 8 would be the most important.</p><p><em>This is something you could make out using PACF (Partial autocorrelation function), used often in timeseries analysis.</em></p><p>Let’s assume that day 8 is much more important than any other day to predict the next day, <strong>could we not just use that last day to make the prediction and make our model simpler</strong>? Well, we can, and if we take only the current day to predict the next, we are seeing a Markov Chain.</p><blockquote>In the context of Markov Chains, we speak about states. Our current weather would be our current state, the next state would be the weather of tomorrow.</blockquote><p><strong>One of the most important properties of a Markov Chain is that the next state only depends on the current one</strong>. In other words, that the weather of tomorrow only depends on the weather of today. If we can make that assumption, we can treat this case as a Markov Chain.</p><p>In this case the weather of Day 2 depends only on Day 1, the weather of Day 3 depends only on Day 2, and so on…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*OzHwANypkVUx183mohNg_A.png" /></figure><p>But, given that we said that we are working with a Markov Chain, there is an efficient way to graph the transitions from one weather to another.</p><h4>Markov Chain graph</h4><p>At the graph below we can see the probabilities for the weather at the next day given the current weather (The probabilities are random selected by me for the example).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TmF6zJ_4sUUtGtKSyu-aEQ.png" /></figure><p>How to read this? Well, let’s assume that today it’s sunny, which are the probabilities for the weather tomorrow?</p><p>Let’s highlight the related lines at the graph. There are 3 lines which start at the sunny symbol, these represents the probability for the weather of tomorrow given that today it’s sunny. We have:</p><ul><li>30% probability that it will be sunny again.</li><li>30% probability that it will rain.</li><li>40% probability that tomorrow it will be cloudy.</li></ul><p>Also, <strong>notice that the sum of the probabilities of all the lines which come out from one state sum together 100%</strong>, this is a property.</p><p>We could write this down in probability terms following the same example, that the current weather is sunny (<em>If you didn’t studied probabilities don’t worry, it’s another way to express what I said above</em>):</p><ul><li>P(S/S): 0.3</li><li>P(R/S): 0.3</li><li>P(C/S): 0.4</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ikhLD2MpBTBf5uhx9cejfw.png" /></figure><p>So, now we know how to read that graph we can make out many things from it, for instance: Given that today it’s raining, what it the probability that tomorrow it will becloudy? As we can see the probability is relatively high, it’s 60%.</p><p>Seems like we are pretty done, right? Given the current weather we can make out which will be the weather tomorrow. Well, no.</p><p>Somebody could come to use and ask: “If I don’t know the current weather, because I live far away, how could I know the probability that it will be raining?”</p><p>Basically, this person is asking us, <strong>given any day, which is the probability for each weather</strong>? In terms of probabilities they are asking us:</p><ul><li>P(S) = ?</li><li>P(C) = ?</li><li>P(R) = ?</li></ul><p>Let’s do some maths to get that.</p><blockquote>A transition is the change from one state to the next in the context of Markov Chains.</blockquote><p>To calculate this, we will need a transition matrix, this is <strong>a matrix which have the values for all the possible transitions </strong>, from any weather to any weather in our case.</p><h4>Transition Matrix</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*rMbR-XqEHDTXDQNYYlNwLw.png" /></figure><p>At this matrix I have the exact same values as in the graph above, to read it start selecting a row, which represents the current weather. We would say, <strong>given that today it’s raining (Second row) the probability that tomorrow it will be sunny is 30%, that it will be again cloudy is 40% and that it will rain is 30%</strong>. Take your time to make sure you understand it until now.</p><p>Now, we will jump to python to make some calculations. You will need for this some basic understanding of matrix operations. If not don’t worry, I will try to make it clear for you.</p><p>What we will do is select randomly the weather for the first day, let’s say it’s cloudy. As with the transition matrix, the second column and row correspond to the cloudy weather, so we will use the following one row matrix to indicate this weather.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*W6XYhguKYhQpJlgaSPVbiQ.png" /></figure><p><strong>By calculating the product of the first day matrix and the transition matrix we will get the probabilities for each weather for the next day.</strong> Let’s do this with python to verify it.</p><p><em>If you don’t know about matrices operation don’t worry. What I’m about to do below is the dot product between two matrices, one of many ways to multiply two matrices.</em></p><h4>Get the general probabilities</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9df3333bca99dba2e424c2e9ad08713c/href">https://medium.com/media/9df3333bca99dba2e424c2e9ad08713c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/1*qbv71jgMelm7lECm-HYmlQ.png" /></figure><p>As we can see the result is the second row of our transaction matrix, which makes sense because these are exactly the probabilities for each weather given a cloudy day. Let’s multiply these probabilities again against the transaction matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27aff29fdd2f7803bfbd8bb4c09148d2/href">https://medium.com/media/27aff29fdd2f7803bfbd8bb4c09148d2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/130/1*SjPyIQdinODvIdA-8r5Nig.png" /></figure><p>These are the probabilities for each weather at the second day after a cloudy day. 30% that it will be sunny, 46% that it will be cloudy, and 24% that it will rain.</p><p>Now, what happens if we repeat that 5, 100, or even 1000 times?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e905db064059ea879ed8bf8390a136f9/href">https://medium.com/media/e905db064059ea879ed8bf8390a136f9/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*Zjmy_kjGm1QoRg__WsOlfw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/1*7bYu_iB8S5QYvlQ-Cnb29g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/1*7bYu_iB8S5QYvlQ-Cnb29g.png" /></figure><p>As we can see, after 5 additional iterations we get values very close to 30%, 45%, and 25%. But, after 100 and 1000 iterations we finally reach these values.</p><p>If we would do more iterations we would get the same values, because these have already converged. <strong>These values are the general probability for any day, no matter the current weather to be sunny, cloudy, or to rain. Indeed, what you got is something we call in algebra eigenvector.</strong> I will not go to much in detail into that concept, but if you ever studied algebra, it’s likely that eigenvectors are not something new to you.</p><h3>More complex example</h3><p>It is worth to make one clarification, the example above is simple. Everything is connected to everything, and we have only three possible states. But our system could be more complex if we want as the example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QYycTPGXRmYW3iq3KKCbcw.png" /></figure><p>And we can build again the related transition matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TxTisuHtPPN3AFRMIA9_wg.png" /></figure><p>And do the same calculation we did before, let’s suppose we start at the state G.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa22c73a2ba99f8ac82d3fb0a1ece4ef/href">https://medium.com/media/aa22c73a2ba99f8ac82d3fb0a1ece4ef/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*pBPthguTc-5XkX_9mQtTUw.png" /></figure><p>As you can see, after one single iteration we got back the last row of the matrix, that’s because we started at the G state. Then, with every following iteration the probabilities started to converge until 500 iterations where we reach a probability of 100% for the state C, can you guess why?</p><p>Look at which are the possible following states for the state C.</p><p>Markov Chains has many applications, from building a model which can make paragraphs based on a previous learned text to being used in a Bayesian model when we are defining a hierarchical model. In the next article related to Markov Chain we will explain how, sharing the code, and add more concepts related to Markov Chains, <strong>if you are interested into real application, you should make sure you don’t miss it.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26023664e6b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AB testing in real life step-by-step]]></title>
            <link>https://medium.com/@alejandroAttento/ab-testing-in-real-life-step-by-step-8d455dc5674a?source=rss-41ecd57f5631------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d455dc5674a</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistical-significance]]></category>
            <category><![CDATA[data-analyst]]></category>
            <category><![CDATA[ab-test]]></category>
            <dc:creator><![CDATA[Alejando Attento]]></dc:creator>
            <pubDate>Mon, 08 Aug 2022 16:33:05 GMT</pubDate>
            <atom:updated>2022-08-08T20:55:31.954Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Zg8-ZM4Sum3IabJ.jpg" /></figure><p><strong>If you are here, you’re interested in AB testing and want to learn how to perform them with all the criterias you need to take into account in real life, not skipping points which usually are ignored in other examples.</strong></p><p>To know about AB testing is a valuable skill in almost all the industries. Here we will cover many topics, some others will be covered in subsequent articles, so feel free to check out if I published already the next part or reach me out if you have any doubts.</p><p>Finally, at the end you will find a link to a repository with the code on GitHub.</p><h3>Introduction</h3><p>Here we will explain how to perform the analysis of an AB test. Usually, when people speak about AB testing, they discuss which statistical test is the best to define if an AB test was successful or not. Starting thinking about the statistical test when we think about AB testing in an error. Indeed, even thinking about “successful” and “non-successful” AB tests is a little annoying. These are just two of many common errors which I noticed with the time.</p><p>We will do a complete and detailed analysis of an AB test here, but it’s important before of that to do some reflection. Before we start to code even one line, we need to have the right mind set, we need to understand what the target of any AB test is. Let’s start with some basic points:</p><ul><li>Well-designed AB tests are successful in any case, even if the outcome is not the expected one. What matters are the learnings, a <strong>well-designed AB test will allow us to plan the next steps on top of it, again, even if the outcome is not the expected one.</strong></li><li><strong>AB testing is a tool, not a stopper</strong>. I often heard different people complaining about the time AB testing takes and how it is delaying the implementation of changes. The problem is that once a new change is implemented if something goes wrong, we are not able to be 100% sure if it’s because of the last change or not. Indeed, even if everything goes well, we will not be able to ensure that it happened because of that new change. <strong>Correlation and causation are two different things</strong>.</li><li>In the theory AB testing is mostly about the statistical techniques to evaluate different variants and proof or reject a hypothesis. In real life the things are a little different. Indeed, I would say that in any data related field one of the major subjects is data cleaning and preprocessing. This is especially relevant when we think about how real AB tests work. For instance, to apply different treatments to different users when we speak about mobile applications or websites, we need to use special tools which allow us to do that, but these tools are not perfect. So, <strong>if we don’t prepare the data taking that into account any further steps will drag an error.</strong></li><li>Usually, at a company you won’t have all the time you would like to investigate all the different options and ways you have to do AB testing. Also, building a pipeline to automate this takes time if you think now about adding dashboards so that non-technical users can look into results, it takes even more time. Because of that, in many cases I would suggest starting with simpler approaches, if we don’t really understand why we are selecting a more complex approach instead of a simpler one we are doing something wrong. As a base rule, <strong>the easiest approach which meets our needs is the right option</strong>.</li></ul><p>We have now an idea about what AB testing is, but let’s summarize all this into one concept,<strong> what is AB testing about? AB testing is about investigation.</strong></p><p>We test something to learn from it, we start with an initial hypothesis, but only the real experience will tell us what reality is. Indeed, well-designed AB tests don’t only evaluate one thing, they help us to understand better the subject of study, allowing us to plan next steps.</p><p>One of the questions you could have is, what is a well-designed AB test? With the previous explanation we got a feeling about what a well-designed AB test is, but this discussion is much deeper and will require his own article which for sure will come.</p><h3>Statistical tests</h3><p>If you look statistical test for AB testing at google you will find half a dozen in just 10 minutes. Now, going ahead and selecting anyone of them is enough? Well, you are running a big risk by doing this. All of us surly saw some of the most basic ones at college or the university. If that is your case, hopefully, your professor repeated many times the word “assumptions”. You can not pick any test and apply it to any set of data, at least not if you want to get some useful and accurate insights from it.</p><p>I will not go into the details of every statistical test, that is out of scope of this article, but I will give you a general recommendation. Two of the most robust statistical test I would recommend you are the:</p><ul><li><strong>Mann — Whitney U test</strong></li><li><strong>Kolmogorov — Smirnov test</strong></li></ul><p>Quick clarification, when I say robust I mean that they have less assumptions on the data than other statistical tests, and also, they are in general easy to use from the process point of view. They evaluate different things (We will come to it) to define when a change is statistically significant. So, in general I would recommend using them together. They are easy to understand, with some basic understanding about statistics they will not be a challenge for you.</p><p>I know that other ways to evaluate AB tests are more interesting, as the bayesian approach, but if we don’t understand why and when to go with the bayesian approach and not with this one recommended here it’s because we need to start with the basics. Again, as a base rule, <strong>the easiest approach which meets our needs is the right option.</strong></p><h3>The analysis</h3><h3>What are we going to do?</h3><p>Now, after a quick but really important reflection, let’s speak about the analysis we did here. What can you expect from this article? We will face the analysis of an AB test, taking into account additional steps which we need to think of when we do the analysis of an AB test but are usually skipped in most of the examples I saw.</p><p>Also, this dataset we will use is already aggregated, but in some cases we will have more granular data, indeed it’s pretty common to work with user level data. So, I will make some comments on things we have to take care of when we work with non-pre aggregated data.</p><h3>To take into account on user level data</h3><p>In many cases you will handle user level data, which you might want to aggregate to analyze it in the way I will do here. But, before doing this, let’s think a littel bit about it.</p><p>In real cases, specially in technology orientated companies, AB testing is done with some tools which helps us to select some of our users, split them into variants and apply to these variants different treatments. Now, how confident are you that the tool you are using works well? Given my experience I wouldn’t be too confident on this. Challenge your tool, ask yourself:</p><ul><li>Is it selecting the amount of users you expected?</li><li>Are the users distributed among the variants in the proportion you set up?</li><li>Did you apply some filters on which users to include in the AB test? Is the tool applying those filters?</li><li>Do you have one user in more than one variant of the same AB test?</li><li>Are you users actually getting the treatment we expect them to get?</li><li>Do we have enough users in our AB test? Are those users representative for the total population?</li></ul><p>Some of those points still applying on aggregated data and we will evaluate them here also.</p><h3>Our data</h3><p>Let’s come back to our case, with already aggregated data the dataset I’m going to use is from Kaggle, <a href="https://www.kaggle.com/datasets/ilkeryildiz/example-dataset-for-ab-test">here</a> is the link so you can download it (Indeed there are two datasets, one for each variant).</p><p>One of the first things we need to do when we work with new data is to understand what information we have in it. Here we are working with non-real data, we can’t go ahead and ask any teammates, go through documentation, or make out by ourselves how the data for each column in processed. But, I would highly recommend you to take this step very seriously. Here, given the context, I will define any additional clarification the dataset or related documentation doesn’t provide by itself.</p><h3>Our study case</h3><blockquote>A company recently introduced a new bidding type, “average bidding”, as an alternative to its existing bidding type, called “maximum bidding”. One of our clients, non_real_bidding_company.com, has decided to test this new feature and wants to conduct an A/B test to understand if the new feature brings more average revenue than maximum bidding.</blockquote><p>Let’s first define the hypothesis they want to test. They want to now if the average revenue increased because of this feature. Of course, we will look also into other metrics, but this is the base criteria for this AB test.</p><p>Based on this, let’s start!</p><h3>Hands on analyzing</h3><p>Let’s first import our libraries, get the data and join it into one single dataframe.</p><h4>Initialization</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc3569886c8d25003af8adaf4d101a89/href">https://medium.com/media/bc3569886c8d25003af8adaf4d101a89/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P_lDErqaRS59zxrk2id28w.png" /></figure><p>Quick spoiler, here we can see alrady some null values.</p><h4>Preprocessing</h4><p><strong>Transform date column</strong></p><p>After importing the data we will start the preprocessing transforming the format of the dates to avoid any future error. Also, we will transform this column into the index.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6899d87ea8a3b5ad684891cc7082971b/href">https://medium.com/media/6899d87ea8a3b5ad684891cc7082971b/href</a></iframe><p><strong>Verify experiment variants</strong></p><p>Checking the experiment variants we can see that we have only two variants, the default (or control) one and the testing variant.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5af9638ca95aff9344d90e01c314366f/href">https://medium.com/media/5af9638ca95aff9344d90e01c314366f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*937KZXFXbNce152VCIhGbw.png" /></figure><p><strong>Define granularity</strong></p><p>Now, we need to make out which is the granularity level of the data. Here we will see that indeed we have 60 rows one for each variant and each day. We can make this out by looking into the shapes as we did below.</p><p>This is important as we could have a lower granularity level, for instance, by country, which would mean that we have one row per date, variant, and country. In these cases we can look at the results at country level, or we would need to aggregate the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2accd455a4c90c16239073d9ead3fc8d/href">https://medium.com/media/2accd455a4c90c16239073d9ead3fc8d/href</a></iframe><p><strong>Manage nulls<br></strong>Nulls are something we have to check for in all the cases, indeed we need to understand why we are getting null values, in some cases it could indicate an error in the process of getting that data. So, if you see a null value don’t just ignore it, at least ask among your team or yourself what is happening, try to understand what that null means. Below we can see that we have a case as such.</p><p>The row for the date 2019–08–05 for the control group has only nulls, here we have two options:<br>- We can drop this row<br>- Use any technique to fill those values</p><p>Initially, I would recommend not to fill these cases if we are not totally sure about what we are doing, we could being biasing the results. Given the risk we would be running and the low potential gain by filling this value I will just drop it. In general I only would fill in values if it’s really necessary.</p><p>But, here we need to make a call-out, as we are working with time series if we take out one date for one of the variants we need to consider doing the same for the other variant. If not we could be facing problems applying different<br>statistical methods to evaluate the results. If you have doubts about what to do I would recommend you to drop the date for both variants.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed571540bd66be51f21a2a1a52c5750b/href">https://medium.com/media/ed571540bd66be51f21a2a1a52c5750b/href</a></iframe><p><strong>Normalize metrics<br></strong>Now, here we have another of the most common errors when evaluating AB tests. This error isn’t covered in most of the examples I saw.<br>In this experiment we have two different variants, are we sure that the same amount of users have been exposed to each variant? Sometimes it could be that this is not the case, and if we don’t be aware of this we will cause noise in our analysis, or in the worst of the cases totally biase our results, making them worthless.</p><p>So, what can we do? Usually one of the metrics in your dataset will give you information about how many users have been exposed to this AB test, a general approach is to normalize all the metrics by this value. In that way you are making sure that different amount of users exposed to each treatment will not biase your results.</p><p>The description of the data isn’t that clear about which column tells us how many users have been exposed to the AB test. In real life, if that happens, stop everything and speak with every team member or responsible for the data<br>to understand clearly what information we have at each column. As we can’t do this here, and based on the information we can find in Kaggle about this dataset, I will assume that the column “Reach” defines the amount of users exposed to this AB test, so I will divide the rest of the columns by this value to normalize the metrics.</p><p>Also, let’s check quickly if we can make out if indeed different amount of users have been exposed to the different variants.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8cc6d75a8c12e461474a6360132b2ce6/href">https://medium.com/media/8cc6d75a8c12e461474a6360132b2ce6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*GHPijqETc0DC2QLCmURMPw.png" /></figure><p>As we can see, one of the variants has been exposed to about 1.000.000 more users than the other. So, if we don’t normalize our metrics our results will be strongly biased.</p><p>Lets normalize the metrics.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a13f5dbb96b612256e495e69b707b504/href">https://medium.com/media/a13f5dbb96b612256e495e69b707b504/href</a></iframe><h3>Analysis</h3><p><strong>Overview</strong><br>Now, we have the dataset we will use for the analysis, we will start with a quick overview. We will split the dataset into two, one for the testing variant and another for the default variant.</p><p>Using the describe function and the style feature pandas has (Nice feature I would recommend to look for at the link <a href="https://pandas.pydata.org/docs/reference/style.html">here</a>), let’s create a quick summary of our data comparing both variants.</p><p>We are calculating the relative difference for each cell we get from the describe function, here the idea is just to get an overview. In that way to can make out what we should expect to see in our further analysis.</p><p><em>Just to notice, the values below are percentages, so 1.00 means 100%.</em></p><p>As we can see, the testing variant seems to be performing much better, than the control group. But, if we look at the “Reach” column we see again that our suspicion was true, one of the variants has been exposed to many more users than the other.</p><p>Analyzing the values below to make out which are more or less meaningful is an interesting subject, but I don’t want to complicate this post too much, so please fell free to do it on your end and reach me out with any doubts you have.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/281a8898e1dfa8665667c65612796f43/href">https://medium.com/media/281a8898e1dfa8665667c65612796f43/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zawy_o0S3jcjHs5jgUfaLw.png" /></figure><p><strong>Time Series &amp; Boxplot</strong><br>Let’s look at this metrics over time, to make out if we can get any insides. We should not forget that we are working with time series, not independent data points, so we could even make out some patterns over time by looking at the line plots.</p><p>To do this we will join the data again, but we won’t stack one dataframe over the other, we will join them horizontally.</p><p>Also, we will calculate the daily difference between the variants for each metric. We can calculate the daily relative difference, this is what I would usually suggest, but if you are working with tiny numbers, as here, this could mean huge percentage values which are harder to analyze. I would recommend to avoid those values, specially if you need to present your work later on to other people.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fdd1a019baeddf491260a32b1fc0d7e9/href">https://medium.com/media/fdd1a019baeddf491260a32b1fc0d7e9/href</a></iframe><p>Looking at the time series we see that among all the metrics two dates seems to be outliers 2019–08–12 and 2019–08–19, at least we should check if something happened during these dates. Also, looking at the boxplot it seems like that these values are possible anomalies.</p><p>Let’s assume for our case that we checked these days, and it actually comes out that some problems happened which caused anomalies. So, we will take them out from our dataset. In real applications take your time to understand these cases, not everything which looks not normal is indeed an anomaly.</p><p>Below you can see the graphs</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e3f833d596f7b105f4e25669fc49195/href">https://medium.com/media/4e3f833d596f7b105f4e25669fc49195/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rZnv1mjPBZNzTMZ8EfeQCw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ePMcEeKTRhWefLn3x-WtXw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i-COFQqgQ5C2k5rjJ2vwdQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VI5xY1Dm5JaX43ERcyI51g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QjOGesaNR9TrYhZyyZiBrg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*btdFnyp5Le3fWDlyqj_YHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rtjgUruYkrwDb1-zKDIJEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KErrtUsuiTq2RjfqI0pdmQ.png" /></figure><p>We will drop the anomalies for the two dataframes we are using to analyze the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c079fe56f3fffa21cd1c6a21f9dd30e7/href">https://medium.com/media/c079fe56f3fffa21cd1c6a21f9dd30e7/href</a></iframe><p>Let’s quickly check if it worked.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/802e42e45ede506031aa3bb2b7bef8e1/href">https://medium.com/media/802e42e45ede506031aa3bb2b7bef8e1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JHjMLOXNrlt2VzdLxJ-zg.png" /></figure><p>As we see those two dates has been excluded.</p><p>We will see another high value which is the 2019–08–08, but let’s assume that this value is not an anomaly, indeed not all the values after the upper bound value of the boxpolt are necessary anomalies.</p><p>Anomalies are an case by them self, we can’t make general rules and assume that they will be 100% accurate for all the cases. But, we can get good approximations, which is out of the scope of this article, but would be interesting to cover.</p><h3>Pairplot</h3><p>Now we will plot a pairplot to get an general visual overview of the metrics.</p><p>Some points to take into account when working on a pairplot:</p><ul><li>This is usually a heavy graph, here our dataset is really small, but if you are working with a bigger datasets, which this function can’t handle, consider sampling your dataset.</li><li>The lower and upper triangle of the graph shows the same, but investing the axes, so:</li><li>One way to reduce the processing when creating this graph is setting corner=True, in that way only the lower triangle will be plotted.</li><li>If you’re plotting the lower and upper triangle consider adding something extra to one of the triangles as I did here with KDE plot, to get extra information.</li></ul><p>Looking at this graph we can make out many things, but in general we could say that for most of the cases we see differences in the distribution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b9cb926b7468a3b0e992db025d85745/href">https://medium.com/media/7b9cb926b7468a3b0e992db025d85745/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*F78900EP9JBwelqk1Z2ftg.png" /></figure><h4>Statistical test</h4><p>Now we have a good understanding of our data, we cleaned it and took out all the anomalies. Now the question is, can we see a statistically significant changes among our variants to say that the hypothesis of our AB test has been meet? Besides of the hypothesis, do we see changes in other metrics.</p><p>There are several tests we could apply, a lot of them would need you to preprocess in someway the data, to keep it easy, but not less efficient because of this, we will apply two non-parametric tests. Why two? Because each test checks for something different, in this way we can have more confident in our results, as we look into the data with two different criteria.</p><p>The tests we will apply are:</p><ul><li><a href="https://www.statstest.com/mann-whitney-u-test/">Mann — Whitney U test</a></li><li><a href="https://www.statisticshowto.com/kolmogorov-smirnov-test/">Kolmogorov — Smirnov</a></li></ul><p>At the links you can get a full explanation of them, they are easy to understand, but here are some characteristics I would like to highlight:</p><ul><li>Both are pair-wise comparison techniques.</li><li>The null hypothesis in both cases is that both data point collections we are comparing (Testing and control) come from the same distribution, which would mean there is no change.</li><li>Both are non-parametric tests, they don’t assume any underlying distribution.</li><li>In both cases it assumes that samples are independent (This could be problematic as time series are by definition not independent, even so, it still being a widely used approach for our case)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/26b4a8cbbfe60593e5f9cfc38c4e0e80/href">https://medium.com/media/26b4a8cbbfe60593e5f9cfc38c4e0e80/href</a></iframe><h3>Final results</h3><p>As we can see, we are getting statistical significant results for most of the metrics (p-values &lt;= 0.05). In our case we consider the result positive only if we get significant results with both statistical tests for a metric.</p><p>The only two metrics which doesn’t seem to change that much, or at least dosen show a statistical significant change, are the amount of times the item has been add to the cart and the amount of impressions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25ec162c7acb12dab2c5b0df2eab436c/href">https://medium.com/media/25ec162c7acb12dab2c5b0df2eab436c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*1IMOCLaEO1yXfQm8vcV_gQ.png" /></figure><h3>What’s next?</h3><p>With this explanation above you’re able to perform AB tests in a quick and robust way. From the statistical point of view you can apply other statistical tests which fits better to this particular base, but here we wanted to set up an general base line to start with AB testing. Also, take into account that in a company, if you want to implement an automated way to perform AB tests it takes time, an easier methode at the start is something I would recommend, later on you have time to develop something more complex if needed. Start small and build big is usually the best way to go.</p><p><strong>Link to the code here</strong></p><p><a href="https://github.com/AlejandroAttento/medium/tree/master/ab_testing_in_real_life_step_by_step">https://github.com/AlejandroAttento/medium/tree/master/ab_testing_in_real_life_step_by_step</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d455dc5674a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>