<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Luis Meazzini on Medium]]></title>
        <description><![CDATA[Stories by Luis Meazzini on Medium]]></description>
        <link>https://medium.com/@luismeazzini?source=rss-335ed336aced------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WFk6Q7S4QnBQ6S43DCCKwQ.jpeg</url>
            <title>Stories by Luis Meazzini on Medium</title>
            <link>https://medium.com/@luismeazzini?source=rss-335ed336aced------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:08:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@luismeazzini/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I built a entire Astrology echosystem]]></title>
            <link>https://medium.com/swlh/how-i-built-a-entire-astrology-echosystem-50193b930a21?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/50193b930a21</guid>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[astrology]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Tue, 10 Feb 2026 16:18:10 GMT</pubDate>
            <atom:updated>2026-02-10T16:18:10.467Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c3rF0p2wAb_k2DIpKhmprw.png" /></figure><p>Traditional astrology isn’t your average horoscope column. It’s a 2,000-year-old system with Ptolemaic dignities, temperament analysis, and mathematical precision that rivals modern astronomy. When I set out to build a platform that honored this tradition while leveraging modern AI, I had no idea I’d ship a production-ready SaaS with paying customers in under 30 days.</p><p>This is the story of how I built <strong>Real Astrology</strong> — a complete traditional astrology ecosystem featuring birth chart calculations, synastry analysis, AI-powered interpretations, and a credits-based monetization system — using Claude Code as my AI pair programmer.</p><h3>The Vision: Traditional Astrology Meets Modern Tech</h3><p>Most astrology apps focus on surface-level Sun sign readings. I wanted something different: a platform that implements <strong>Hellenistic and Medieval astrology</strong> with the rigor of academic texts like Ptolemy’s <strong><em>Tetrabiblos</em> </strong>and Vettius Valens’ <strong><em>Anthology</em></strong>.</p><p><strong>The feature set I envisioned:</strong></p><ul><li>Birth chart calculations with sub-arcsecond precision</li><li>6 house systems (Placidus, Whole Sign, Koch, Regiomontanus, Campanus, Equal)</li><li>Essential dignities scoring (Domicile, Exaltation, Triplicity, Terms, Face)</li><li>Temperament analysis using the classical 4-humors system</li><li>Synastry (relationship compatibility) with AI-enhanced insights</li><li>Saturn and Solar Return calculations</li><li>A credits-based monetization system</li></ul><p>Building this traditionally would take a team months. I did it solo in <strong>under 4 weeks.</strong></p><p><strong>The Tech Stack</strong></p><p>After evaluating options, I settled on a modern, scalable architecture:</p><p><strong>Backend:</strong></p><ul><li>Python 3.13 with FastAPI (async everywhere)</li><li>PostgreSQL 16 with JSONB for flexible chart data</li><li>PySwisseph for astronomical calculations (Swiss Ephemeris)</li><li>Qdrant vector database for RAG-based AI interpretations</li><li>Redis + Celery for background tasks</li><li>Stripe for payments</li></ul><p>Frontend:</p><ul><li>React 18 with TypeScript (strict mode)</li><li>Vite for lightning-fast builds</li><li>TailwindCSS with a custom “Midnight &amp; Paper” design system</li><li>Custom SVG chart visualization components</li></ul><p><strong>Infrastructure:</strong></p><ul><li>Turborepo monorepo</li><li>Docker Compose for development</li><li>GitHub Actions CI/CD</li><li>Terraform for infrastructure-as-code at AWS</li></ul><h3>Spec-Driven Development: The Secret Sauce</h3><p>Before touching any code, I adopted a methodology called <strong>Spec-Driven Development</strong> — and it changed everything.</p><h4>The Approach</h4><p>Instead of jumping into code, I started with a comprehensive specification document (‘PROJECT_SPEC.md’). This 2,000+ line document defined:</p><ul><li><strong>Every </strong>requirement with unique IDs (RF-001, RF-002, etc.)</li><li><strong>Input/output contracts</strong> for each feature</li><li><strong>Business rules</strong> and edge cases</li><li><strong>Database schemas</strong> with all relationships</li><li><strong>API endpoint definitions</strong> with example payloads</li><li><strong>Validation rules</strong> with exact constraints</li></ul><h4>Why This Works with Claude Code</h4><p>When you give Claude Code a detailed spec, magic happens:</p><p>1. U<strong>nambiguous requirements:</strong> No “make it look nice” — exact pixel values, exact validation rules</p><p>2. <strong>Consistent implementation:</strong> Every feature follows the same patterns defined upfront</p><p>3. <strong>Reduced back-and-forth:</strong> Claude Code implements exactly what the spec says</p><p>4<strong>. Built-in documentation:</strong> The spec becomes your documentation</p><p>I could point Claude Code to a requirement like “Implement RF-003: Login Social (OAuth2)” and it would generate the complete OAuth flow — endpoints, callbacks, token exchange, user creation — all matching the spec’s definition.</p><p>The spec also served as a contract between me and Claude Code. When something didn’t work as expected, I could reference the spec:</p><blockquote><em>&gt; “According to RF-002, the access token should be valid for 15 minutes, but the current implementation uses 60 minutes. Please fix.”</em></blockquote><p>This precision eliminated ambiguity and sped up iterations dramatically.</p><h3>The Claude Code Difference</h3><p>Here’s where things get interesting. I used <strong>Claude Code</strong> as my AI pair programmer throughout the entire development process.</p><h4>Speed Through Conversation</h4><p>Instead of context-switching between documentation, Stack Overflow, and code, I had conversations like:</p><blockquote><em>&gt; “Implement the Ptolemaic dignities system with scoring. Domicile should be +5, exaltation +4, triplicity +3 (day ruler) or +2 (night/participating), terms +2, face +1. Include debilities: detriment -5, fall -4.”</em></blockquote><p>Claude Code would generate the Python module, complete with type hints, tests, and documentation. Then it would integrate it into the existing codebase, understanding the repository pattern and service layer architecture I had established.</p><h4>Complex Algorithms Made Accessible</h4><p>Traditional astrology has intricate calculation rules. The temperament analysis alone involves:</p><ul><li>Ascendant sign qualities (hot/cold, wet/dry)</li><li>Ascendant ruler’s essential dignity (weighted scoring)</li><li>Solar quadrant position</li><li>Lunar phase divisions</li><li>Lord of Nativity calculations</li></ul><p>Claude Code helped me translate academic texts into working code:</p><pre>def calculate_temperament(chart_data: dict) -&gt; TemperamentResult:<br>    &quot;&quot;&quot;<br>    Calculate temperament using the classical 4-humors system.<br>    Based on Ptolemy&#39;s Tetrabiblos and medieval sources.<br>    &quot;&quot;&quot;<br>    scores = {&quot;choleric&quot;: 0, &quot;sanguine&quot;: 0, &quot;melancholic&quot;: 0, &quot;phlegmatic&quot;: 0}<br><br>    # Ascendant sign contributes base qualities<br>    asc_sign = get_ascendant_sign(chart_data)<br>    scores = apply_sign_qualities(scores, asc_sign)<br><br>    # Dignity-weighted planetary contributions<br>    asc_ruler = get_ruler(asc_sign)<br>    dignity_weight = calculate_dignity_weight(asc_ruler, chart_data)<br>    # ... continues with full implementation</pre><h4>Full-Stack Integration</h4><p>The magic wasn’t just in isolated code generation. Claude Code understood the entire system:</p><ul><li>When I added a new endpoint, it automatically created the corresponding React service, TypeScript types, and component</li><li>When I modified the database schema, it generated Alembic migrations</li><li>When I implemented the credits system, it updated the frontend balance display, the API validation, and the Stripe webhook handler</li></ul><p>This full-stack awareness saved countless hours of integration work.</p><h3>The Traditional Astrology Engine</h3><p>The heart of the system is the calculation engine. Here’s what makes it special:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2vp9WjCtLjPLMGYz8WPgA.png" /></figure><h4>Astronomical Precision</h4><p>Using PySwisseph (the Swiss Ephemeris), calculations achieve:</p><ul><li><strong>Planetary positions:</strong> &lt; 1 arcsecond error</li><li><strong>House cusps:</strong> &lt; 0.01° tolerance</li><li><strong>Historical dates:</strong> Full timezone handling back to ancient</li></ul><p>Every planet is scored across five dignity levels:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1561db40f6aec5e51de2ed9b06adf887/href">https://medium.com/media/1561db40f6aec5e51de2ed9b06adf887/href</a></iframe><p>Debilities (Detriment: -5, Fall: -4) are also tracked. This scoring system powers the AI interpretations, giving context about planetary strength.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PvIut3mdIIRDIBI-BzvAgg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ECDU--MH_dxusiX20KW7Q.png" /></figure><h3>AI-Powered Interpretations</h3><p>Raw calculations are useful for astrologers, but most users want meaning. I built a <strong>RAG (Retrieval-Augmented Generation)</strong> system:</p><p><strong>1. Vector Database: </strong>Qdrant stores embeddings of astrological knowledge</p><p><strong>2. Hybrid Search:</strong> BM25 keyword matching + semantic similarity</p><p><strong>3. Context Assembly:</strong> Relevant passages are retrieved for each chart placement</p><p><strong>4. Generation:</strong> OpenAI GPT-4o-mini creates personalized interpretations</p><p>The result? Interpretations that honor traditional astrology while being accessible to beginners.</p><p>![Planets, houses, and aspects with detailed traditional interpretations](chartdetail-redesign-analysis.png)</p><p><strong>Cost efficiency:</strong> ~$0.01 per full chart interpretation (7 planets + 12 houses + aspects).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WSCXNHppT4S3hUyjYuJvzg.png" /></figure><h3>Synastry: Relationship Compatibility</h3><p>One of the most requested features was synastry — comparing two birth charts to analyze relationship dynamics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FCFon7ErBf9oyz5XChVd0w.png" /></figure><p>The synastry module calculates:</p><ul><li>Inter-chart aspects (how planets in one chart relate to another)</li><li>House overlays (where your planets fall in their houses)</li><li>Composite midpoints</li><li>Overall compatibility scores by category (emotional, intellectual, physical, spiritual)</li></ul><p>The AI-enhanced insights option lets users get deeper, personalized interpretations of their relationship dynamics — powered by the same RAG system that interprets individual charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bL_7fiR7hx_RfYQchV-WUQ.png" /></figure><p>The AI-enhanced insights option lets users get deeper, personalized interpretations of their relationship dynamics — powered by the same RAG system that interprets individual charts.</p><h3>Monetization: The Credits System</h3><p>I implemented a credits-based model that balances accessibility with sustainability:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dec6dfde5c41e225ec7d54b115964eb1/href">https://medium.com/media/dec6dfde5c41e225ec7d54b115964eb1/href</a></iframe><p>Users can purchase credit packs via Stripe, and every transaction is logged for transparency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-55VOFkxchhLMk1TWtPYOg.png" /></figure><p>The system includes: Real-time balance updates, Transaction history, automatic webhook handling for purchases.</p><h3>The Dashboard Experience</h3><p>The dashboard serves as the central hub for users:</p><ul><li>Quick access to create new charts or synastry analyses</li><li>Recent chart activity</li><li>Credit balance prominently displayed</li><li>Personalized recommendations based on usage</li></ul><p>Every screen is fully responsive, ensuring a seamless experience whether users are on desktop or mobile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ff0qoD-rSI9oI-4-Xx4lWg.png" /></figure><h3>The Numbers</h3><p>After 4 weeks of development:</p><p><strong>Codebase:</strong></p><ul><li>213+ Python backend files</li><li>195 React/TypeScript components</li><li>25+ API endpoints</li><li>439+ automated tests</li><li>20+ database models</li></ul><p><strong>Infrastructure (Terraform):</strong></p><ul><li>12 modular Terraform components</li><li>60+ ‘.tf’ files</li><li>VPC with 5 subnets across 2 AZs</li><li>Full CI/CD pipeline with 7 deployment stages</li></ul><p><strong>Architecture:</strong></p><ul><li>Full async/await throughout</li><li>Repository pattern for data access</li><li>Service layer for business logic</li><li>LGPD/GDPR compliant with audit logging</li></ul><p><strong>Quality:</strong></p><ul><li>Zero-tolerance linting (Ruff + mypy)</li><li>Pre-commit hooks enforcing standards</li><li>CI/CD with GitHub Actions</li></ul><h3>Early Traction</h3><p>We launched in early 2026 and already have <strong>early adopters actively using the platform</strong>. The feedback has been encouraging:</p><ul><li>Professional astrologers appreciate the traditional calculation accuracy</li><li>Beginners love the AI-powered interpretations that make complex concepts accessible</li><li>The synastry feature has driven the most engagement — people love relationship compatibility analysis</li></ul><p>The credits system is working. Users are purchasing packs to unlock premium features, validating the monetization model before we scale marketing efforts.</p><h3>Lessons Learned</h3><h4>1. AI Pair Programming Changes Everything</h4><p>Claude Code didn’t just write code faster — it enabled a different kind of development. I could think at the architecture level while Claude handled implementation details. The conversation-based interface meant I could describe intent and get working code.</p><h4>2. Traditional Knowledge + Modern Tech = Differentiation</h4><p>By going deep into traditional astrology (not surface-level Sun signs), we carved out a unique position. The combination of academic rigor and AI accessibility is rare.</p><h4>3. Ship Fast, Iterate with Users</h4><p>Getting to paying customers quickly validated assumptions. The credits system, synastry feature priority, and interpretation style all evolved based on real feedback.</p><h4>4. Type Safety Pays Off</h4><p>Strict TypeScript on the frontend and comprehensive type hints in Python caught countless bugs during development. The upfront investment in types accelerated iteration.</p><h4>5. Spec-Driven Development is the Multiplier</h4><p>The upfront investment in PROJECT_SPEC.md paid dividends throughout development. When requirements are crystal clear, AI pair programming becomes incredibly effective. No ambiguity means no wasted iterations.</p><h3>Conclusion</h3><p>Building Real Astrology in under a month wasn’t about cutting corners — it was about leveraging the right tools. Claude Code acted as a force multiplier, handling everything from complex astronomical algorithms to Stripe webhook integration.</p><p>The result is a production-ready SaaS that:</p><ul><li>Implements rigorous traditional astrology</li><li>Scales with async architecture</li><li>Monetizes with a proven credits model</li><li>Already has paying customers</li></ul><p>If you’re building a SaaS in 2026, AI pair programming isn’t optional — it’s the new baseline for solo founders and small teams.</p><p>The stars aligned, and so did the code.</p><p><em>Real Astrology is live and accepting early adopters. If you’re interested in traditional astrology or want to see the platform in action, visit </em><a href="http://realastrology.ai/"><em>realastrology.ai</em></a><em>.</em></p><blockquote>“A wise man rules his stars; the fool obeys them.”</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50193b930a21" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/how-i-built-a-entire-astrology-echosystem-50193b930a21">How I built a entire Astrology echosystem</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fraud detection — Unsupervised Anomaly Detection]]></title>
            <link>https://medium.com/@luismeazzini/fraud-detection-unsupervised-anomaly-detection-1fedc2b345b1?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fedc2b345b1</guid>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[credit-risk]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Mon, 18 Jan 2021 23:28:34 GMT</pubDate>
            <atom:updated>2021-01-18T23:28:34.113Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fraud detection — Unsupervised Anomaly Detection</h3><h4>An 100% unsupervised approach to discover frauds on credit card transactions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qugPHiJkTEtipEh_.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@b0rno?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ryan Born</a> on <a href="https://unsplash.com/s/photos/credit-card?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>One of the greatest concerns of many business owners is how to protect their company from fraudulent activity. This concern motivated large companies to save data relative to their past frauds, however, whoever performs a fraud aims not to be caught then this kind of data usually is unlabeled or partially labeled.</p><p>On this article, we will talk about how to discover frauds on a credit card transaction dataset, unlike most fraud datasets this dataset is completely labeled however, we won’t use the label to discover frauds. Credit card fraud is when someone uses another person’s credit card or account information to make unauthorized purchases or access funds through cash advances. Credit card fraud doesn’t just happen online; it happens in brick-and-mortar stores, too. As a business owner, you can avoid serious headaches — and unwanted publicity — by recognizing potentially fraudulent use of credit cards in your payment environment.</p><p>One of the most common approach to find fraudulent transactions was randomly select some transactions and ask and auditor to audit it. This approach was quite unaccurate since the relation between the number of fraudulent transactions and normal transactions is close to 0.1%.</p><p>Then, we aim to leverage machine learning to detect and prevent frauds and make fraud fighters more efficient and effective. Commonly, there are the supervised and the unsupervised approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*jFw18Rw2uOmjEDMX.png" /></figure><p>Also, these models can then be deployed to automatically identify new instances/cases of known fraud patterns/types in the future. Ideally the validation of this type of machine learning algorith sometimes need to be a temporal validation since fraud patterns can change over time, however to simplify this article, the validation will be simplified.</p><h3>The dataset</h3><p>The project uses a dataset of around 284000 credit card transactions which have been taken from <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Kaggle</a>.</p><p><a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Credit Card Fraud Detection</a></p><p>The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions. It contains only numerical input variables which are the result of a PCA transformation. Unfortunately, due to confidentiality issues, the original features and more background information about the data are not provided. Features V1, V2, …, V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are “Time” and “Amount”, and there are no null values (<a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Dataset page</a>).</p><p>Since just the “Time” and “Amount” features are easely intepreted, we can use some visualizations to see the impact of the features on the target variable (fraud). First, frauds happen more on small transactions or big ones?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*Il0Yx9ro2DJ5e4Zq.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/0*OifkTiuQIbI4A4lc.png" /></figure><h3>Approach to detect frauds</h3><p>This article proposes an unsupervised approach to detect frauds, the only place the labels are used is to evaluate the algorithm. One of the biggest challenge of this problem is that the target is highly imbalanced as only 0.17% cases are fraudulent transactions. But the advantage of the representation learning approach is that it is still able to handle such imbalance nature of the problems. Using TSNE we can try to see how the transactions are similar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/0*GX4rGcnyW3BcMbbH.png" /></figure><p>The main ideia of this approach is to compress the data making a “latent representation” and then reconstruct the data. If a sample is similar to the rest of the dataset, the reconstructed data will be similar or even equal to the original data. However, if the sample is <strong>not</strong> similar to the rest, the reconstructed sample will not be similar to the original one.</p><p>In short, we compress the data and reconstruct it. If the reconstructed data is not similar to the original one, we have a fraud.</p><h3>Principal Component Analysis</h3><p>Using Principal component analysis (PCA), we managed to compress the data from 30 features to 10 features and calculated the reconstruction score. The histogram for this score is below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*4GS7ycF2qQAmvM9d.png" /></figure><p>We can see that most samples have a low reconstruction score and then, probably most frauds have more then 50 reconstruction score. Using TSNE we can compare the original data disposition with the PCA compressed data distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/0*HDkoMLhtNRs5QLoD.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/0*os3b6WSdpJQxUQJm.png" /><figcaption>Original vs PCA distribution</figcaption></figure><p>Now, we need to set a threshold to the reconstruction score. Usualy there domain expertise is used to help to set this threshold since it impacts direcly on the precision and recall trade-off.</p><p>Using the mean and standard deviation of the reconstruction score we can set a reasonable threshold. Then, I choose to set the threshold to <em>mean + 2*std.</em> With this, auditing 0.74% of the transactions we managed to find 87% of the frauds.</p><h3>Autoencoder</h3><p>An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. It is composed of a enconding part responsable to compress the data and a decoder to reconstruct the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bAbcp9s-O9rCjYmm.png" /></figure><p>The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction. Analogously to the PCA approach, the reconstrcution score histogram can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*ievDKOG3aTFYrt4S.png" /></figure><p>We can see that most samples have a low reconstruction score and then, probably most frauds have more then ~60 reconstruction score. Using TSNE we can compare the original data disposition with the Autoencoder compressed data distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/0*cGIpBQROcGHoIfJT.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/0*iNsH05qpyZMTPw6V.png" /><figcaption>Original vs Autoencoder distribution</figcaption></figure><p>The Autoencoder representation seens to split quite well the frauds from the normal data. Now, we need to set a threshold to the reconstruction score. Usualy there domain expertise is used to help to set this threshold since it impacts direcly on the precision and recall trade-off.</p><p>Using the mean and standard deviation of the reconstruction score we can set a reasonable threshold. Then, I choose to set the threshold to <em>mean + 2*std.</em> With this, auditing 0.85% of the transactions we managed to find 65% of the frauds.</p><h3>Conclusion</h3><p>The objective of the approach was fulfilled, making possible to detect frauds with an 100% unsupervised approach. Nevertheless, there are several ways to make this approach work better, like:</p><ul><li>Tunning the used models (PCA and Autoencoder);</li><li>Tune the threshold of reconstruction score;</li><li>Explore if the PCA and Autoencoder are detecting the same frauds. If they work in different ways, maybe it is worth to make an emsamble;</li><li>Augment the data with some feature engineering.</li></ul><p>The code used to perform this approach along with a little more analysis is available on <a href="https://github.com/lmeazzini/Small-projects/blob/master/Credit_fraud.ipynb">GitHub</a>.</p><p>Hope it helps you and that our SPERO increases your hope in the future of the internet! Be here, be happy, be safe, and enjoy yourself!</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fedc2b345b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fraud detection — Unsupervised Anomaly Detection]]></title>
            <link>https://medium.com/data-science/fraud-detection-unsupervised-anomaly-detection-df43d81fce67?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/df43d81fce67</guid>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[fraud]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Tue, 02 Jun 2020 14:49:11 GMT</pubDate>
            <atom:updated>2020-06-03T14:02:07.270Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fraud detection — Unsupervised Anomaly Detection</h3><h4>An 100% unsupervised approach to discover frauds on credit card transactions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nY7dDQH1cvRbVxQTmVKRA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@b0rno?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ryan Born</a> on <a href="https://unsplash.com/s/photos/credit-card?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>One of the greatest concerns of many business owners is how to protect their company from fraudulent activity. This concern motivated large companies to save data relative to their past frauds, however, whoever performs a fraud aims not to be caught then this kind of data usually is unlabeled or partially labeled.</p><p>On this article, we will talk about how to discover frauds on a credit card transaction dataset, unlike most fraud datasets this dataset is completely labeled however, we won’t use the label to discover frauds. Credit card fraud is when someone uses another person’s credit card or account information to make unauthorized purchases or access funds through cash advances. Credit card fraud doesn’t just happen online; it happens in brick-and-mortar stores, too. As a business owner, you can avoid serious headaches — and unwanted publicity — by recognizing potentially fraudulent use of credit cards in your payment environment.</p><p>One of the most common approach to find fraudulent transactions was randomly select some transactions and ask and auditor to audit it. This approach was quite unaccurate since the relation between the number of fraudulent transactions and normal transactions is close to 0.1%.</p><p>Then, we aim to leverage machine learning to detect and prevent frauds and make fraud fighters more efficient and effective. Commonly, there are the supervised and the unsupervised approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*aGBYL67at6voRNaFNlG1ew.png" /></figure><p>Also, these models can then be deployed to automatically identify new instances/cases of known fraud patterns/types in the future. Ideally the validation of this type of machine learning algorith sometimes need to be a temporal validation since fraud patterns can change over time, however to simplify this article, the validation will be simplified.</p><h3>The dataset</h3><p>The project uses a dataset of around 284000 credit card transactions which have been taken from <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Kaggle</a>.</p><p><a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Credit Card Fraud Detection</a></p><p>The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions. It contains only numerical input variables which are the result of a PCA transformation. Unfortunately, due to confidentiality issues, the original features and more background information about the data are not provided. Features V1, V2, …, V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are “Time” and “Amount”, and there are no null values (<a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Dataset page</a>).</p><p>Since just the “Time” and “Amount” features are easely intepreted, we can use some visualizations to see the impact of the features on the target variable (fraud). First, frauds happen more on small transactions or big ones?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*6YTkJf6Sh-V0VSOn1zfTOg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*WJQWcj0MxkojVFrRHFWN1w.png" /></figure><h3>Approach to detect frauds</h3><p>This article proposes an unsupervised approach to detect frauds, the only place the labels are used is to evaluate the algorithm. One of the biggest challenge of this problem is that the target is highly imbalanced as only 0.17% cases are fraudulent transactions. But the advantage of the representation learning approach is that it is still able to handle such imbalance nature of the problems. Using TSNE we can try to see how the transactions are similar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*DHg78CC5EOsc0suzhVTS0A.png" /></figure><p>The main ideia of this approach is to compress the data making a “latent representation” and then reconstruct the data. If a sample is similar to the rest of the dataset, the reconstructed data will be similar or even equal to the original data. However, if the sample is <strong>not</strong> similar to the rest, the reconstructed sample will not be similar to the original one.</p><p>In short, we compress the data and reconstruct it. If the reconstructed data is not similar to the original one, we have a fraud.</p><h4>Principal Component Analysis</h4><p>Using Principal component analysis (PCA), we managed to compress the data from 30 features to 10 features and calculated the reconstruction score. The histogram for this score is below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*Rqs4nAAA2PMATxsCGLnmWA.png" /></figure><p>We can see that most samples have a low reconstruction score and then, probably most frauds have more then 50 reconstruction score. Using TSNE we can compare the original data disposition with the PCA compressed data distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*DHg78CC5EOsc0suzhVTS0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*wHQB6RNaYP9NpGkOk2dYJA.png" /><figcaption>Original distribution vs PCA distribution</figcaption></figure><p>Now, we need to set a threshold to the reconstruction score. Usualy there domain expertise is used to help to set this threshold since it impacts direcly on the precision and recall trade-off.</p><p>Using the mean and standard deviation of the reconstruction score we can set a reasonable threshold. Then, I choose to set the threshold to <em>mean + 2*std.</em> With this, auditing 0.74% of the transactions we managed to find 87% of the frauds.</p><h4>Autoencoder</h4><p>An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. It is composed of a enconding part responsable to compress the data and a decoder to reconstruct the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C6Z6i1_2EJn13jVEsAOkRQ.png" /></figure><p>The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction. Analogously to the PCA approach, the reconstrcution score histogram can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*VA6Yo4TuAIng5-JRvPIDxQ.png" /></figure><p>We can see that most samples have a low reconstruction score and then, probably most frauds have more then ~60 reconstruction score. Using TSNE we can compare the original data disposition with the Autoencoder compressed data distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*DHg78CC5EOsc0suzhVTS0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*rTq8MQWIIr-u65uiIrqVxw.png" /><figcaption>Original districution vs Autoencoder distribution</figcaption></figure><p>The Autoencoder representation seens to split quite well the frauds from the normal data. Now, we need to set a threshold to the reconstruction score. Usualy there domain expertise is used to help to set this threshold since it impacts direcly on the precision and recall trade-off.</p><p>Using the mean and standard deviation of the reconstruction score we can set a reasonable threshold. Then, I choose to set the threshold to <em>mean + 2*std.</em> With this, auditing 0.85% of the transactions we managed to find 65% of the frauds.</p><h3>Conclusion</h3><p>The objective of the approach was fulfilled, making possible to detect frauds with an 100% unsupervised approach. Nevertheless, there are several ways to make this approach work better, like:</p><ul><li>Tunning the used models (PCA and Autoencoder);</li><li>Tune the threshold of reconstruction score;</li><li>Explore if the PCA and Autoencoder are detecting the same frauds. If they work in different ways, maybe it is worth to make an emsamble;</li><li>Augment the data with some feature engineering.</li></ul><p>The code used to perform this approach along with a little more analysis is available on <a href="https://github.com/lmeazzini/Small-projects/blob/master/Credit_fraud.ipynb">GitHub</a>:</p><p><a href="https://github.com/lmeazzini/Small-projects/blob/master/Credit_fraud.ipynb">lmeazzini/Small-projects</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df43d81fce67" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/fraud-detection-unsupervised-anomaly-detection-df43d81fce67">Fraud detection — Unsupervised Anomaly Detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Youtube music recommender]]></title>
            <link>https://medium.com/data-science/building-a-youtube-music-recommender-bbd1b3431da4?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbd1b3431da4</guid>
            <category><![CDATA[artificial-inteligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[youtube]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Sun, 29 Mar 2020 18:04:24 GMT</pubDate>
            <atom:updated>2020-03-29T18:04:24.873Z</atom:updated>
            <content:encoded><![CDATA[<h4>From scratch to deploy: recommending youtube music videos to you</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*oph1PynSCUMwLNVMgAhWzA.jpeg" /></figure><h3>Overview</h3><p>On this article, I`ll comment some details of a simple solution I built to recommend Youtube music videos to myself. You can check out the deployed solution on Heroku at <a href="https://thawing-shore-99052.herokuapp.com/">Youtube Music Recommender</a>. I know, it is not beautiful. Maybe one day I will develop my front-end skills.</p><p>Then, the main idea here is to go from scratch to deployment. Mainly, this process will include all these steps:</p><ol><li>Scrapping video data from Youtube pages;</li><li>Extracting the video information from each page;</li><li>Preprocess data from each video into a single dataset;</li><li>Manually label some of samples, active learning the rest;</li><li>Extract features from the dataset;</li><li>Train a Random Forest and a LightGBM model and ensemble them;</li><li>Build a simple app to serve the model through Heroku.</li></ol><p>Also, if you want to check out the <a href="https://github.com/lmeazzini/youtube-recommender">code</a>, I will be citing which script is used on each step.</p><p><a href="https://github.com/lmeazzini/youtube-recommender">lmeazzini/youtube-recommender</a></p><blockquote><strong>Lets begin!</strong></blockquote><h3>Scrapping video data from Youtube pages</h3><p>First of all, we need data! To get some data, I scrapped youtube pages by querying on Youtube search. Since I am looking for music to recommend to myself, I used six queries:</p><blockquote>‘Folk rock’, ‘Classical guitar’, ‘Acoustic rock’, ‘Orchestral rock’, ‘Cello songs’ and ‘Brazillian rock’</blockquote><p>Then, I saved the first 100 pages of each query. This is done in “search_data_collection.py” script.</p><h3>Extracting the video information from each page</h3><p>Now we have data from Youtube search pages, we need to extract information from the videos present on those pages. To do so, I made a parser with <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup</a>.</p><p>This parser just saves the link, title and query used for each video on the 600 pages. Also, since there is a possibily of videos been duplicated, I dropped the duplicates. This is done in “search_data_parsing.py” script.</p><h3>Preprocess data from each video into a single dataset</h3><p>From the parsed data, we are able to web scrap each video. There is much information here, most of the features we can get are from this step. Since this is a simple solution and I did not spent a lot of time engenieering features I used just a few of the possibilities.</p><p>There are 2277 videos totally, then this script takes a quite longe time to run, ~2h. The script name is “video_data_processing.py”</p><h3>Manually label some of samples possibly with active learning</h3><p>The most tedious and boring part of this work. I had to get labels. So, I opened the raw_data file generated by the previous step on sheets and added a “y” column and manually filled 1000 rows with 0 or 1.</p><p>With the labelled data, it is possible to train a Random Forest model and the features extracted on the next step and use this model to identify which samples are the hardest to classify (~0.5 output), then I manually labelled 200 of the samples identified by active learning.</p><p>The active learning helps a lot to mitigate the labelling cost. However, the labelling time I was expecting to spend was less then I imagined, so I ended manually labelling everything.</p><h3>Extract features from the video data</h3><p>Now, each row of our dataset is a video with its data, like title, number of views, author, description, video height, video width and other information. From these fields, I made feature engineering to extract some features that can be helpful to later.</p><p>This way, I extracted the number of views, view per day, video resolution and made a Bag of Words (BoW) with the title string. Surely, there are much more features that could be extracted, especially from video description and video tags. However, I left a more deep feature engineering to future works.</p><h3>Train a Random Forest and a LightGBM model and ensamble them</h3><p>With the extracted features and the labels, we can split the dataset in train and validation. I chose not to make a test set, leaving the ultimate validation to production data.</p><p>First I trained a LightGBM model optimizing its parameters with a bayesian optimization along with some data pre-processing parameters (like n_gram range for BoW). Then, the Random Forest model was trained with the same pre-processing parameters found on bayesian search. For more details, you can check the implementation. I looked for an ensemble of these two models giving a weight to each model.</p><p>To measure how well the model is performing, I chose not to use precision, recall and F1-Score. Since the problem involves a recommendation, the final output should be a ranking of the most relevant recommendations (output closer to 1). Then I used the average_precision_score, roc_auc and log_loss to better evaluate the model.</p><p>Also, I made a Learning Curve Analysis (LCA) to check if the data used was enough to have a consistent prediction. The ROC curve and the LCA can be seen on the figures below. The feature extraction and the modelling is on “final_model.py”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*PycPAy--SmCtudklakiZLQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*wiqWjxlZ2qvvMmUvF9fltg.png" /></figure><h3>Build a simple app to serve the model through Heroku</h3><p>Now that we have our model, we can make an application to predict on production data. Since our data is from youtube searches, if we use the same queries used to make the dataset, we would probably get the same or almost the same videos used to train/validate the model. Then I changed most of the queries to get different videos at the application.</p><p>This process includes an application (app.py) built with Flask. This script is responsible to call the functions to scrap new data, treat it and use the saved model to predict the output and finally, make the .html file to present the final output. The scrapped data is pre-processed and stored on a SQLite database and then queried to make predictions, this way later I can insert new data to the database and easy re-run the application.</p><p>To deploy it, I used Docker and Gunicorn. There is a Dockerfile with the configuration on the deploy folder. You can easily run a docker locally or even deploy it through Heroku, which I have done following the <a href="https://devcenter.heroku.com/articles/container-registry-and-runtime">guide</a> on Heroku documentation.</p><p>You can see the deployed application screenshot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wUobw-f7QfAIjGenJPDZ4Q.png" /></figure><p>However, if you open it now probably the music list will be the same or quite similar to the screenshot. This can happen due to music do not change a lot over time and, the model just makes a re-classification when I want; there is no re-train routine scheduled.</p><h3>Finally</h3><p>Looking at the model predictions at production there a clear bias. A lot of the recommend videos have the String “Official Music Video” on the title. Probably this is a heavy feature that does not make a lot of sense. Many improvements can be made to improve the model performance and better understand the predictions.</p><p>Hope this article gives ideas and helps anyone who reads it. Again, this is article is not describing every step, however, the code is available at <a href="https://github.com/lmeazzini/youtube-recommender">GitHub</a>.</p><p><a href="https://github.com/lmeazzini/youtube-recommender">lmeazzini/youtube-recommender</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbd1b3431da4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-a-youtube-music-recommender-bbd1b3431da4">Building a Youtube music recommender</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything you need to know about Time Series]]></title>
            <link>https://medium.com/data-science/everything-you-need-to-know-about-time-series-5fa1834d5b18?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fa1834d5b18</guid>
            <category><![CDATA[guides-and-tutorials]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Mon, 20 Jan 2020 21:34:47 GMT</pubDate>
            <atom:updated>2020-04-14T00:25:01.450Z</atom:updated>
            <content:encoded><![CDATA[<h4>The most complete introduction to Time Series analysis you can find</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iK9n5y9JJxAdEcsQn4yO0Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@greenpjy123?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jiyeon Park</a> on <a href="https://unsplash.com/s/photos/time?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><blockquote>It’s really clear that the most precious resource we all have is time. — Steve Jobs</blockquote><h3>Introduction</h3><p>Most of the data that is collected has a temporal structure. Sometimes there are cases where this structure is hidden or we can ignore it, however, there are some ways that we can use this temporal structure to extract relevant information from the available data.</p><p>Knowing how to model time series is an essential skill within data science, as there are structures specific to this type of data that can be explored in all situations.</p><p>Everything I put in this guide can be used at any time scale. Keep in mind that: whenever I say days, weeks, hours, or any unit of time, you can understand it as “periods”. Then, let’s get started.</p><h3>Structure of a Time Series</h3><p>How do you know you are dealing with a time series? A <strong>time series</strong> is a series of data points indexed in time order. The most simple time series is something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*0WwkuXB2DaKlytCTeMilwg.png" /><figcaption>Simple time series</figcaption></figure><p>Where the elements are:</p><ul><li>Timestamp: a mark of the moment in time when the event was registered. Its accuracy will depend on the measured event.</li><li>Value: what is the value that this phenomenon had at that moment? Can<br>be just one or more values. When there is more than one value per timestamp, we have a multivariable time series.</li></ul><h3>Type of Time Series</h3><p>Also, it’s important to know that there are two types of time series:</p><ul><li><strong>Regular time series</strong>: are those that have a record in each uniform period of time. A daily series of temperatures can be considered a regular series since we expect exactly one record per day, one record per unit of time.</li><li><strong>Irregular time series</strong>: are those collected without a defined frequency. An example is the log of accesses to a website. People do not access, uniformly, every <em>N</em> minutes. In this case, we will have records with different time intervals between them.</li></ul><h3>Time Series Data Preparation</h3><h4>Feature Engineering</h4><p>To create a machine learning model we need input variables (features — X) and one or more output variables (target — y). There are four essential features, which should be tested on any Time Series problem:</p><ul><li><strong>Lag</strong>: The Lag feature is simply using a previous target value as a feature to predict the current one. For example, if you want to predict the price of Apple stock tomorrow, the price today is a really powerful feature, this case is a lag 1 feature. Or if your problem has a weekly seasonality, a lag seven feature will be probably very useful.</li><li><strong>Differences</strong>: This feature is very similar to Lag. Only here, instead of using the specific value, we will use the difference from the last period to the previous one. For example, calculating the difference between the number of items sold yesterday and the number of items sold, on the same day of the week, but last week. The most common operation is the difference (subtraction) but we can also use sums, ratios and multiplications.</li><li><strong>Statistical features</strong>: In addition to specific values and arithmetic operations on them, we can calculate statistics such as the mean, standard deviation, maximum, minimum, quantiles. In our sales exemple, we can calculate what was the maximum quantity sold of the product, in one day, in the last seven days. The idea is simple, take a window of periods and apply a function on the respective values. Usually, this function should return only one number for each timestamp.</li><li><strong>Timestamp decomposition</strong>: We also can extract the relevant parts of our timestamp, usually thing related to its seasonality. Some periods have a differentiated pattern specific to that moment. Some classic examples are Christmas, Black Friday, Mother’s Day, Parents, Valentines. In the days before these dates, there is a significant increase in sales in the trade. In other words, at this time of year, there will always be an increase in sales and the model needs to know that. We can create features such as day of the week, day of the month, month of the year, time of day.</li></ul><h3>Modelling</h3><p>Traditionally models for statistical time series are applied in one series at a time. That is, you have only a series of values and apply the desired method to model the process that generates it. At Data Science we can also model one series at a time, just create features that only depend on the values of the series we are trying to predict. However, we can also stack the time series to make a stacked modelling. This way, we can have individual models and stacked models, as the individual models are the traditional approach I will just explain the stack approach in detail.</p><h4>Stacked modelling</h4><p>In this case, instead of creating a model for each series, we “stack” them all together and create a global model. What do I mean by stacking?</p><p>Imagine that we have a series of the stock price of 10 companies for a year, one record per day. Let’s concatenate all these series into one. Then we will have 10 companies x 365 days = 3650 lines. Each line of the “new series” will have the timestamp, the company identification and the sales value at that moment.</p><blockquote><strong>Why do we want to do this?</strong></blockquote><ul><li><strong>More data</strong>: Machine learning models tend to work better with a larger amount of data. Instead of making models based on a few records (365 lines in our example), we will be making more lines, allowing the model to capture patterns more robustly.</li><li><strong>Complexity</strong>: When it comes to production, monitoring models, keeping them running, it is much simpler to maintain just one model than 10. This reduces the complexity of our solution and avoids implementation errors.</li><li><strong>Global effects</strong>: Using several series at once, the model has the opportunity to capture common events between them. Global effects, occurring in several series, will be clearer and captured more easily.</li></ul><h3>Validation</h3><p>When we are dealing with time series models, we can not make an incorrect validation, and it is very common to see gravely incorrect validations. If you simply divide your data randomly, as we learned in a lot of courses, you will have a result that will look very good, but it will not actually be reproduced in production. This is because you will be mixing lines from the past and the future, letting your model know about future information to predict the past! This way, in production, there is no way to get a time machine and get this information.</p><p>In addition, the entire process is affected by time. Some more quickly, others more slowly, so we need to have an idea of how our model will act with the changes that will happen.</p><blockquote>You may be wondering: is the whole process really going to change over time?</blockquote><p>Even an image classifier must be affected, as the devices used to take pictures are always changing.</p><p>As a rule, models that are directly or indirectly related to human behaviour tend to change faster and require updating (re-training) with a certain frequency. The right frequency depends on the specific case. In general, time series break the assumption of line independence. That is, information from neighbouring lines tends to give hints about the target of the line we selected.</p><p>Therefore, the simplest, but already quite robust, way to divide your time data between training and testing, is by separating by date. You can take a specific date or a proportion of your data, usually the first 70% to train and 30% to validation. Or even, can we make a time series cross-validation?</p><h4>Time Series Cross Validation</h4><p>Instead of using just one split to validate our model, it’s good to use at least three Cross Validation folds to do so, always splitting the data by the timestamp.</p><p>Usually, more recent data tends to be more important for the model. There are two types of “windows” that we can use to validate our model:</p><ul><li><strong>Expandable Window: </strong>It means that as time passes, our training data will increase. The figure below can illustrate this process very well.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*oBBvvDNsMKMeid7n.png" /></figure><ul><li><strong>Sliding Window: </strong>In this format, as well as in the calculation of the features, we will slide a fixed window, previously defined, by the data. The sliding window gives you more control over the data you are going to use, making your pipeline more robust when retraining. Even if you have many years of data, it doesn’t mean that you need to use everything to train the model.</li></ul><p>A disadvantage of simple cross-validation is that we are validating “some version” of the future, ones that really happened, iterating each version after a small-time variation. But in some cases, the future may behave differently.</p><p>This happens a lot in cases of modelling in the financial market. In general, you need to look for time periods with different financial regimes (high, low and stagnation) to validate your models.</p><p>One way that is gaining popularity is by cross-block validation. In general, resampling methods, where you reuse samples of the data in your validation, tend to be more robust.</p><p>In the case of time series, as we want to preserve the structure, but simulate different scenarios, we need to take blocks of data without making a random mix, and these block can not have intersections between the cross-validation folds. The most important thing is that the lines that were generated together over time remain together. These can be seen in the figure below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*58j7vnq909edrHVd.png" /></figure><h3>The Model</h3><p>Some machine learning models seem to work better with time series data.</p><h4>Tree Ensembles</h4><p>Here we are talking about any model that joins several decision trees to make the forecast. The most popular are Random Forest and Gradient Boosted Decision Trees.</p><p>Both in competitions and in industry projects we see these models easily finding patterns within time series, being robust to noise and being able to extract values of features little correlated with the target due to their high ability to capture non-linear patterns.</p><h4>Neural Networks</h4><p>Multi-layer perceptron (MLP) neural networks are the most traditional that you can use to create models of basically any nature. But there are some types that deal with strings natively.</p><h4><strong>Convolutional Neural Network (CNN)</strong></h4><p>Originally created to handle images, but their use for time series cases has become increasingly popular. Basically, they end up learning filters that multiply the original features within the convolutional windows. This means that we can use the power of learning representations to model time series.</p><p>They train faster and have fewer parameters (weights) to adjust than recurrent networks. Still, as in most cases with neural networks, they tend to work better as the amount of data grows. Worth the try for small samples, but don’t expect it to work miracles.</p><h4><strong>Recurrent Neural Network (RNN)</strong></h4><p>The most popular today is Long-Short-Time-Memory (LSTM), are made specifically with sequential data in mind, as is the case with time series. The idea is that not only the current example can influence the result, but the “memory” of past examples can also help in the forecast.</p><p>These networks tend to be quite complex, with a large number of parameters to adjust, making them need a large sample to achieve a good result. At the same time, they are very flexible, allowing for a multitude of architectural combinations between the number of entry and exit periods.</p><h4>Time Decay</h4><p>Every time series prediction tries to understand human interactions. If the modelled system changes any logic, a trained model will have a performance decay over time. Some models will decay fast, others slowly.</p><p>To measure how is your model performing over time, try to monitor its performance and retrain it when the performance drops below a threshold.</p><p>The decaying speed will vary on the problem type you are dealing, for example, financial models probably will need retraining in a short period of time, and models prediction the weather can be retrained after a longer period of time.</p><p>There are some more robust ways to tell when to retrain time series models, however, this article is already long enough. Some day I can still make an article here about it.</p><h3>Conclusion</h3><p>One of the most frequent data available is related to time. Them, for a Data Scientist, is fundamental to understand how to correctly explore, prepare, extract good features, validate and model this kind of data.</p><p>With the ideas presented here, you will certainly be able to create high performance and robust models using machine learning with time series.</p><p>I hope this overview can help some people, for now I will just leave a book recommendation and a good quote.</p><h3>Nice to read Time Series books:</h3><p>Time Series Forecasting (Francesca Lazzeri): <a href="https://amzn.to/2R7qGko">https://amzn.to/2R7qGko</a></p><p>Python Crash Course (Eric Matthes): <a href="https://amzn.to/38gJtPI">https://amzn.to/38gJtPI</a></p><p>Python for Data Analysis (Wes McKinney): <a href="https://amzn.to/2tpgevz">https://amzn.to/2tpgevz</a></p><blockquote><em>It’s really clear that the most precious resource we all have is time. — Steve Jobs</em></blockquote><h3>References:</h3><p>Time Series CV figures are from: <a href="https://hub.packtpub.com/cross-validation-strategies-for-time-series-forecasting-tutorial/">https://hub.packtpub.com/cross-validation-strategies-for-time-series-forecasting-tutorial/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fa1834d5b18" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/everything-you-need-to-know-about-time-series-5fa1834d5b18">Everything you need to know about Time Series</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Intelligence]]></title>
            <link>https://medium.com/@luismeazzini/learning-intelligence-part-1-eaa506e3f73f?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/eaa506e3f73f</guid>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[personal-development]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[intelligence]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Tue, 04 Jun 2019 13:22:47 GMT</pubDate>
            <atom:updated>2019-08-07T14:28:07.839Z</atom:updated>
            <content:encoded><![CDATA[<h4>Is it possible to learn intelligence? YES!</h4><h4>Answering why, when, for how long and how to study.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*5OdDbJfzH5-lfx5hLe5zfw.jpeg" /></figure><p>Is it possible to be more intelligent? To learn a way to improve our ability to learn? An Italian professor Pierlugui Piazzi wrote a book called “Learning Intelligence” which this Storie is based on.</p><p>For a long time, intelligence was believed to be an innate characteristic. The genetic fact was considered much more influential than the environmental fact. However, the latest discoveries in neuroscience show that intelligence can be learned and that this fact does not occur during classes, but at the time of the individual study, out of school.</p><blockquote>“The roots of education are bitter, but the fruit is sweet.” — Aristoteles</blockquote><p>Then, I’ll answer four questions here, why, when, for how long and how to study. I hope this helps you think about how are you studying.</p><h3>Why study?</h3><p>We need to study to be more intelligent and even more. There can be a time when knowledge is fundamental to survival. As well, most companies nowadays do not care much about degrees and diplomas. However, they tend to give greater importance to knowledge, not where or how many years did you study.</p><h3>When to study?</h3><p>First, let’s understand something: If someone got good grades all his life, does not that mean he learned it? Of course not! That means he has become, generally encouraged by family and school, an expert at getting good grades! How people do this?</p><blockquote><em>STUDYING ON THE EVE OF EVERY TEST.</em></blockquote><p>Or even, just before the test. However, why study as late as possible? Most likely, to remember everything. So, the main idea is to study when you have to prove that you know something, putting the information in your brain in an absurdly unstable way, taking the test and then forgetting everything!<br>However, why this happens? We can make an association between a computer and our brain.</p><p>A computer has RAM which is small and provisory if energy fails, all its content is lost. Moreover, there is an HD, which is big and permanent. If the computer is turned off, the RAM contents are lost, while the HD content is retained for later use. Also, the capacity of the HD is huge, that is, hundreds or even thousands of times greater than that of RAM.</p><h4>RAM &amp; HD of a human brain</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ZpYbmtMvZPJvwt3Z14fjiQ.png" /></figure><p>To avoid falling into the “forget everything” trap, be aware that in your brain, you have the equivalent of a RAM and an HD. If we could make the mental x-ray of a student’s head, we would see, basically, two structures.</p><p>At the core, we have the so-called limbic system, full of structures (thalamus, hypothalamus, amygdala, etc.), in which there is a so-called hippocampus, which is very important for short-term memory. Wrapping this kernel, as if it were the bark of a tree (in Latin, cortex), we have the most “noble” part of the brain, fundamental in long-term memory.</p><p>As you may have already guessed, the limbic system does the paper of the computer RAM, while the cortex, among others functions, would be the equivalent of an HD. “Writing” in this RAM is straightforward, but “deleting” is even easier! You have, in this structure, a relatively small draft (in which only a few hours of information can fit).</p><p>If someone tells you a phone number, for example, you will be able to hold it for a few minutes, the time when it is still “Resonating” in your mind. However, if you do not take some measure to get a permanent form, the next day (or even a few hours later) the number will be completely forgotten!</p><p>In contrast, in the HD (cortex) an enormous amount of data fits. If one studied like a madman ten hours a day, every day of his life, he would exhaust the processing power and storage of your cortex in approximately seven centuries (or even much more)! To take full advantage of this potential, however, you should be able to write in your cortex, a task not always effortless. Then, how to save in the brain HD?</p><p>Information, therefore, is transformed into knowledge only if the neural networks of the cortex are reconfigured. Synapses must be undone. Others activated, dendrites die or are born, paths are redone. Therefore, the physical structure of the brain must be changed!</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*2S82PNG-R7jxLS9UhLzClg.png" /></figure><p>If we put electrodes on your head at bedtime and record the electrical activity of the brain on paper, we will see a significant drop in this activity from the wakefulness (wake) to deep sleep. After a while, though, the electrical activity intensifies again, even though you are still asleep. This pattern will repeat itself throughout the night, alternating deep sleep and dream.</p><p>During the dream phase, the “maintenance” of your brain is made, saving something to your HD. Moreover, also during this phase, a good part of the RAM content is thrown in the trash. The decision of what goes where is taken on the basis of the emotional load, associated with each piece of information, not the rational load.</p><p>Explaining better: if you, upon receiving that information during the day, he did it in a joyful, pleasant or even unfortunate, tragic way, emotion associated with it will cause it to be permanently recorded during the night dream. However, if the information was received with indifference, boredom, so as not to shake it either positively or negatively, it will undoubtedly be discarded overnight. For example:</p><blockquote><em>If your geography teacher, for example, gives an essential subject and, in the middle of class, tell a joke, what you will remember class after a few weeks? The joke!</em></blockquote><p>Then, how should I study? Little. The study is not about quantity, but about quality. You should not study more. You should study better. Alternatively, an even better answer, <strong>you should study little, but every day.</strong></p><ul><li>If you attend class in the morning, you should study in the afternoon. (And not another day!)</li><li>If you attend class in the afternoon, you should study at night. (And not the next morning!)</li><li>If you attend class at night, you should go to sleep about 40 minutes later; but, in any case, do not sleep before studying the classes that day.</li></ul><p>So, the day’s classes should be studied the same day before a night’s sleep! Then, back to our question, when to study?</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*HH0PYvbldYvAra-_S6f-WQ.jpeg" /></figure><h3>How long to study?</h3><p>This is the hard part. If you understand that you have to study every day, you already know “how long” to study: Little! However, how long is this little? Ten minutes, one hour, ten hours? Remember, by study, I mean the out of the classroom alone time with a pen and a notebook.</p><p>How long? … You will find out. That is, in creating the habit of studying every day, you will realize, by self-assessment a few weeks later, that there were days when you studied too much and others in which you studied less.</p><p>In a few days or at most, in a few weeks, you will find the correct pace. Of course, however, it will be little study when compared to those rushes on the eve of proof that some usually do. You should ask yourself, at the end of a study period:</p><blockquote><em>In a few years from now, when my son has a doubt in this subject, can I help you or am I going to be ashamed?</em></blockquote><p>Yet, should I study without any break? It is good to know that the cells of the brain, in charge of forming the circuits that allow retaining knowledge, are the neurons. A neuron, in the background, is a fantastic electrochemical device that acts as a key that allows, depending on certain conditions, the passage of an electric pulse to other neurons, opening or closing circuits.</p><p>In the neuron, there are some chemicals essential to its functioning that, in case of excessive use, can be exhausted in 30 or 40 minutes. For you to continue to play your part in the brain, necessary to have time to recover, that is, by replenishing those substances, the lack of which had rendered it inefficient.</p><p>A widespread phenomenon can perceive this inefficiency: you have been studying for a long time and suddenly start reading some text. Without your noticing, your attention is diverted to another line of thought, but, paradoxically, you keep reading! However, when you get to the end of the text, you look at him perplexed and ask: Jesus, what did I read?!</p><blockquote><em>Then, for how long should I rest?</em></blockquote><p>Well, you could start with half an hour of study and a half-hour break. In the half hour of study, concentrate to the fullest, doing the task and not “getting rid of it!” In the ten minute intervals, between a half hour and the next, the ideal is a physical activity, such as stretching or a short walk or maybe a little gym or playing a musical instrument.</p><p>As each person has a rhythm of his own, maybe this half hour can be stretched for 40 or even 50 minutes (But do not go beyond that!). The interval can sometimes be extended to 15 or 20 minutes.</p><h3>How to study?</h3><p>As we have seen, the study time must be between the class and nighttime sleep.</p><h4>The distractions</h4><p>You should be in a quiet, comfortable, and concentration-sensitive location. So … no TV and radio!</p><blockquote><em>However, cannot I study listening to music?</em></blockquote><p>Yes, you can. But let us understand something fundamental about the functioning of your brain: the transmission of an electric pulse, from one neuron to another, is absurdly slower than in a computer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*muVcpeNWKpmwOBbjlunNBg.jpeg" /></figure><p>Consequently, to overcome this “sloppiness”, our brains use a trick that, in computing, is called “parallel processing”. This means that several parts of the brain can perform different tasks at the same time. A human being can drive a car chewing gum, listening to music on the radio and still talking to the passenger.</p><p>Analyzing in detail the brain of a non-left-handed human being, we see that each of the halves (called the “cerebral hemispheres”) specialized in performing specific tasks. Imagine a skull seen from above: in the left hemisphere, we have the Linguistic and logical-mathematical cognitive modules; in the right, the Musical and Space modules are located.</p><p>So if you study listening to <strong>instrumental</strong> music (without someone singing in a language you know), not only will musical module not interfere with others, it will even help muffle other noises in the environment that could disrupt your concentration.</p><h4>But how?</h4><p>Well, so far, we have seen what is not to do. But then, what should I study?</p><p>To study, it is indispensable to study by <strong>doing</strong>. There’s no point in looking at a passively open book or, at most, marking with a “yellow phosphorescent” sections of a text that you have found interesting. Never study without having an active pencil on a piece of paper.</p><p>If the object of study is a text of history, for example, do not just read it or, worse, try to decorate it. Instead, find out the most essential concepts and facts (the ones you would mark with the yellow pen) and write them down on a sheet of paper.</p><p>The act of writing itself allows greater posterior fixation during the night. Strictly speaking, the paper can be thrown in the trash then because what matters is not what is engraved on it, but what was etched in your mind.</p><blockquote><em>Have you ever wondered which subject is the easiest to </em><strong><em>learn</em></strong><em>?</em></blockquote><p>… that’s right: <strong>math</strong>!</p><p>If you do not agree to this, please reread the question. I did not ask which subject was most natural to understand. Indeed, in some subjects, mathematics is complicated to understand. However, once understood, it becomes easy to be learned! To study math is <strong>practice</strong>!</p><p>On the other hand, some people think that studying history is simple because they understand everything in class. Then they complain that they cannot keep what they have learned. They did not learn; they only understood. To learn history, do not just watch the class and then read a chapter of the book. It is necessary to have a pencil and paper and write the keywords, the most important sections. There is no need to make a full summary, but it is vital to write down, in schematic form, the most critical points.</p><h3>Conclusion</h3><p>In this short Storie, you can understand four things:</p><p>What? Have more? This four things alone are not enough. The next question can demand a Storie just to this one question.</p><blockquote><em>How can I become more intelligent? How can I learn intelligence?</em></blockquote><p>I will probably continue writing on this subject and post it on Part 2 of this Storie.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eaa506e3f73f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is really data about?]]></title>
            <link>https://medium.com/data-science/what-is-really-data-about-a60a2af1cfaa?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/a60a2af1cfaa</guid>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-journalism]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[communication]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Wed, 15 May 2019 20:30:39 GMT</pubDate>
            <atom:updated>2019-05-15T20:30:39.502Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0UgG8TBsA3mpjPs7P-zqwg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/5fNmWej4tAA?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Helloquence</a> on <a href="https://unsplash.com/search/photos/computer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Data science basic idea</h3><p>Nowadays, most articles tend to show a lot of execution and a little exploration. Like Conor Deway wrote on his <a href="https://towardsdatascience.com/data-scientists-are-thinkers-a36cc186d570">article</a>:</p><blockquote>Unlike engineers, designers, and project managers, data scientists are exploration-first, rather than execution-first.</blockquote><p>Moreover, this is quite fair considering the history of Data Science.</p><h4>A little of data science history</h4><p>Data Science began with data mining on “From data mining to knowledge discovery in databases” [1] for discovering useful knowledge in data. Then in 2001, Willian S. Cleveland tried to combine the computation power available with data mining to improve statistical analysis, beginning the use of the “Data Science” term [2]. Today Data Science covers a lot, really many fields, there is Data Engineers, Data Analytics, Machine Learning Engenieers, and so on. Also, recently Data Science is called “<a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">The Sexiest Job of the 21st Century</a>”.</p><p>The primary tool of a Data Scientist is, of course, data. However, most everyone gets some data from Kaggle, runs a complex model such as a CNN and BOOM. We have a new data scientist. Using all these fancy tools without knowing the background and how to make sense out of data correctly and the model is a waste of time or just right for learning how to use those techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*UpNHg3TbDy8s7mW5tyKqrw.jpeg" /></figure><h3>What is really data about?</h3><p>This opens a window to go a little further in the data question. What is data about? Is it about making complex models? Cool visualizations? Writing code? Statistical analysis? Feature engineering?</p><p>Data is not about Machine Learning, visualizations, statistics, and wrangling.<br>Data is about understanding. Understanding the problem and how you can solve it using data with whatever tools or techniques you choose.</p><blockquote>Understand the problem. Understand the data. And the rest will flow.</blockquote><h4>The Data Scientist goal’s is not making a good model</h4><p>If there is a problem, a Data Scientist tries to solve it with Machine Learning. How do we know if he succeeded? The answer is probably using the right metrics, right? However, we must dig deeper. The real metric is not precision, recall, and all those statistical metrics.</p><blockquote>The critical metric is impact.</blockquote><p>As Data Scientists, we must use our data to make as much impact as possible to our company. The impact can be in the form of multiple things. These two are some of the essential forms of seeing the impact.</p><ul><li><strong>Insight</strong>: extract useful knowledge to guide your project development, providing direction. For example, with an exploratory analysis of real estate market data, you discovered that around 80% of the buyers are men. However, their wives are the ones that end choosing mostly everything in the house. That information is beneficial to direct advertisements to the right person.</li><li><strong>Data products</strong>: developing products that use data to provide something in return (usually money). For example, a product that can predict the glucose levels of a diabetic person. With that, this person can use this knowledge to eat and use insulin at the right time.</li></ul><p>What tool can I use to make an impact?</p><blockquote>Any, nobody cares about it.</blockquote><p>The most important thing to determine which technique to use is to <strong>understand the data</strong>. Then let the data talk about how it works. Instead of fitting a model to the data, a Data Scientist needs to know to fit the data into a model (the one that works better). Rachel Tatman has a quite good <a href="http://www.rctatman.com/files/Tatman_2019_PutDownTheDeepLearning.pdf">presentation</a> on “So what method should you use?”, she ends her presentation with his phrase:</p><blockquote>If it is stupid but it works, it is not stupid.</blockquote><p>This means, if our model is straightforward (like hand-built rules from the 90s) and it still works great, then is not stupid at all.</p><p>Then, let’s imagine that I built an application that can make a huge impact. How to sell it? For example, you are working with a dataset from your company, and you discovered a pattern nobody saw. How to present it to your team?</p><h3>Data Storytelling</h3><p>As a combination of data, narrative, and visuals,<strong> Data Storytelling</strong> is one of the new, hot skill every Data Scientist will need in the near future.</p><p>Data Storytelling is a fundamental technique for a Data Scientist or Data Analyst who needs to explain his or her line of thinking to others, whether they have the technical knowledge or not. It is at that moment that your communication skills will be very important. After all, no one likes to see a lame presentation or read an annoying text, right?</p><p>Briefly, Data Storytelling is the act of explaining what you did, how you did it, and why you did it all in a way that keeps your reader or listener engaged.</p><blockquote><em>People hear statistics, but they feel stories. (</em><a href="https://www.forbes.com/sites/brentdykes/2016/03/31/data-storytelling-the-essential-data-science-skill-everyone-needs/#35f9e50752ad"><em>Forbes</em></a><em>)</em></blockquote><p>The goal of Data Storytelling is to tell the story of your data. It is making sense of the data humanly. Personifying the data as a living scene and let others understand the data with empathy, not only statistics. It may involve a combination of three key elements: <em>data</em>, <em>visuals</em>, and <em>narrative</em>.</p><p>When the narrative is coupled with data, it helps to <strong>explain</strong> to your audience what’s happening in the data and why a particular insight is important. Ample context and commentary are often needed to appreciate an insightfully. When visuals are applied to data, they can <strong>enlighten</strong> the audience to insights that they wouldn’t see without charts or graphs. Many interesting patterns and outliers in the data would remain hidden in the rows and columns of data tables without the help of data visualizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*h50RBlpwIQO8x-IFjbPcgg.png" /><figcaption>Source: <a href="https://www.forbes.com/sites/brentdykes/2016/03/31/data-storytelling-the-essential-data-science-skill-everyone-needs/#3101f7ac52ad">Forbes</a></figcaption></figure><p>Finally, when narrative and visuals are merged, they can <strong>engage</strong> or even entertain an audience. It’s no surprise we collectively spend billions of dollars each year at the movies to immerse ourselves in different lives, worlds, and adventures. When you combine the right visuals and narrative with the right data, you have a data story that can influence and drive <strong>change</strong>.</p><p>Mostly you must use the narrative, visuals, and data to make the complex simple. Using it to make everybody understand an obscure insight and think, “how I have not saw this before?”.</p><h4>Tips to be better at Data Storytelling</h4><p>There are already some articles and stories about Data storytelling tips. Some of those are below:</p><ul><li><a href="https://towardsdatascience.com/10-ways-to-take-your-boss-on-a-journey-through-the-data-14a3fdbb3503">10 Ways to Take Your Boss on a Journey Through the Data</a></li><li><a href="https://www.crazyegg.com/blog/data-storytelling-5-steps-charts/">Mastering Data Storytelling: 5 Steps to Creating Persuasive Charts and Graphs</a></li><li><a href="https://medium.com/@Infogram/10-tips-for-better-data-storytelling-3808d823bde3">10 Tips for Better Data Storytelling</a></li><li><a href="https://www.columnfivemedia.com/7-data-storytelling-tips-from-centuries-old-data-visualization">7 Data Storytelling Tips From Centuries-Old Data Visualization</a></li></ul><p>I would add one last tip here. I believe it is the most important one.</p><blockquote>Read, a lot.</blockquote><p>Reading is essential to understand the function and history of our society. It is vital to develop our mind by discovering new things. Also, reading is a crucial aspect when we are talking about imagination and curiosity.</p><p>The more you read, the more you understand the subject you read. Now linking it back to Data Science, data is about everything. A reader can learn concepts faster about anything he reads. This can make a huge difference giving sense to data, giving the needed background to the numbers so that numbers can become ideas and insights.</p><blockquote>The more that you read, the more things you will know. The more you learn the more places you will go. -Dr. Suess</blockquote><h3>References</h3><p>[1] Fayyad, Usama, Gregory Piatetsky-Shapiro, and Padhraic Smyth. “From data mining to knowledge discovery in databases.” <em>AI magazine</em> 17, no. 3 (1996): 37–37.</p><p>[2] Cleveland, William S. “Data science: an action plan for expanding the technical areas of the field of statistics.” <em>International statistical review</em> 69, no. 1 (2001): 21–26.</p><p><a href="https://towardsdatascience.com/data-scientists-are-thinkers-a36cc186d570">Data Scientists Are Thinkers</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a60a2af1cfaa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/what-is-really-data-about-a60a2af1cfaa">What is really data about?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is really data about?]]></title>
            <link>https://medium.com/@luismeazzini/what-is-really-data-about-ad46c536e86e?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad46c536e86e</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bussiness]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Wed, 15 May 2019 20:30:39 GMT</pubDate>
            <atom:updated>2021-01-22T17:17:35.595Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data is not about AI, machine learning, statistics.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wni6z2fAComCWsYQ.jpeg" /></figure><p>Like Conor Deway wrote on his <a href="https://towardsdatascience.com/data-scientists-are-thinkers-a36cc186d570">article</a>, most articles tend to show a lot of execution and a little exploration.</p><blockquote><em>Unlike engineers, designers, and project managers, data scientists are exploration-first, rather than execution-first.</em></blockquote><p>Moreover, this is quite fair considering the history of Data Science.</p><h3>A little of data science history</h3><p>Data Science began with data mining on “From data mining to knowledge discovery in databases” [1] for discovering useful knowledge in data. Then in 2001, Willian S. Cleveland tried to combine the computation power available with data mining to improve statistical analysis, beginning the use of the “Data Science” term [2]. Today Data Science covers a lot, really many fields, there are Data Engineers, Data Analytics, Machine Learning Engeneers, and so on. Also, recently Data Science is called &quot;<a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">The Sexiest Job of the 21st Century</a>&quot;.</p><p>The raw material of Data Science is, for sure, data. However, most everyone gets some data from Kaggle, runs a complex model such as a CNN and BOOM. We have a new data scientist. In this case data science resumes to application of high order curve fitting in data, the famous <strong>.fit()</strong> and <strong>.predict()</strong>.</p><p>Using all these fancy tools without knowing the background and how to make sense out of data correctly, the model becomes a waste of time or, at best, just right for learning how to use those techniques.</p><p>This opens a window to go a little further in the data question. What is data about? Is it about making complex models? Cool visualizations? Writing code? Statistical analysis? Feature engineering?</p><p>Data is not about Machine Learning, visualizations, statistics, and wrangling.<br>Data is about understanding the reality. Understanding real problems and how you can solve it using data with whatever tools or techniques you choose.</p><blockquote><em>Understand the real problem. Understand the data. And the rest will flow.</em></blockquote><h3>The Data Scientist goal’s is not making a good model</h3><p>If there is a problem, a Data Scientist tries to solve it with Machine Learning. How do we know if he succeeded? The answer is probably using the right metrics, right? However, we must dig deeper. The real metric is not precision, recall, and all those statistical metrics.</p><blockquote><em>The critical metric is </em><strong><em>IMPACT!</em></strong></blockquote><p>As Data Scientists, we must use our data to make as much impact as possible in our company. It means that we need to bring real value, not just a cool model that performs well in a task that no ones care about. There are two main essential forms of value that we can point out:</p><ul><li><strong>Insight</strong>: extract useful knowledge to guide your project development, providing direction. For example, with an exploratory analysis of real estate market data, you discovered that around 80% of the buyers are men. However, their wives are the ones that end choosing mostly everything in the house. This information is beneficial to direct advertisements to the right person.</li><li><strong>Data products</strong>: developing products that use data to provide something in return (usually money). For example, a product that can predict the glucose levels of a diabetic person. With that, this person can use this knowledge to eat and use insulin at the right time.</li></ul><p>What tool can I use to make an impact?</p><blockquote><em>Any, nobody cares about it.</em></blockquote><p>The most important thing to determine which technique to use is to <strong>understand the data</strong>. Then let the data talk about how it works. Instead of fitting a model to the data, a Data Scientist needs to know to fit the data into a model (the one that works better). Rachel Tatman has a quite good <a href="http://www.rctatman.com/files/Tatman_2019_PutDownTheDeepLearning.pdf">presentation</a> on “So what method should you use?”, she ends her presentation with his phrase:</p><blockquote><em>If it is stupid but it works, it is not stupid.</em></blockquote><p>This means, if our model is straightforward (like hand-built rules from the 90s) and it still works great, then is not stupid at all.</p><p>Then, let’s imagine that I built an application that can make a huge impact. How to sell it? For example, you are working with a dataset from your company, and you discovered a pattern nobody saw. How to present it to your team?</p><h3>Data Storytelling</h3><p>As a combination of data, narrative, and visuals,<strong> Data Storytelling</strong> is one of the new hot skill every Data Scientist will need in the near future.</p><p>Data Storytelling is a fundamental technique for a Data Scientist or Data Analyst who needs to explain his or her line of thinking to others, whether they have the technical knowledge or not. It is at that moment that your communication skills will be very important. After all, no one likes to see a lame presentation or read an annoying text, right?</p><p>Briefly, Data Storytelling is the act of explaining what you did, how you did it, and why you did it all in a way that keeps your reader or listener engaged.</p><blockquote>People hear statistics, but they feel stories. (<a href="https://www.forbes.com/sites/brentdykes/2016/03/31/data-storytelling-the-essential-data-science-skill-everyone-needs/#35f9e50752ad">Forbes</a>)</blockquote><p>The goal of Data Storytelling is to tell the story of your data. It is making sense of the data humanly. Personifying the data as a living scene and let others understand the data with empathy, not only statistics. It may involve a combination of three key elements: <em>data</em>, <em>visuals</em>, and <em>narrative</em>.</p><p>When the narrative is coupled with data, it helps to <strong>explain</strong> to your audience what’s happening in the data and why a particular insight is important. Ample context and commentary are often needed to appreciate an insight-fully. When visuals are applied to data, they can <strong>enlighten</strong> the audience to insights that they wouldn’t see without charts or graphs. Many interesting patterns and outliers in the data would remain hidden in the rows and columns of data tables without the help of data visualizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*URO3HGxLjL4FJ3nW.png" /></figure><p>Finally, when narrative and visuals are merged, they can <strong>engage</strong> or even entertain an audience. It’s no surprise we collectively spend billions of dollars each year at the movies to immerse ourselves in different lives, worlds, and adventures. When you combine the right visuals and narrative with the right data, you have a data story that can influence and drive <strong>change</strong>.</p><p>Mostly you must use the narrative, visuals, and data to<strong> make the complex simple</strong>. Using it to make everybody understand an obscure insight and think, “how I didn’t see it before?”.</p><h3>Tips to be better at Data Storytelling</h3><p>There are already some articles and stories about Data storytelling tips. Some of those are below:</p><ul><li><a href="https://towardsdatascience.com/10-ways-to-take-your-boss-on-a-journey-through-the-data-14a3fdbb3503">10 Ways to Take Your Boss on a Journey Through the Data</a></li><li><a href="https://www.crazyegg.com/blog/data-storytelling-5-steps-charts/">Mastering Data Storytelling: 5 Steps to Creating Persuasive Charts and Graphs</a></li><li><a href="https://medium.com/@Infogram/10-tips-for-better-data-storytelling-3808d823bde3">10 Tips for Better Data Storytelling</a></li><li><a href="https://www.columnfivemedia.com/7-data-storytelling-tips-from-centuries-old-data-visualization">7 Data Storytelling Tips From Centuries-Old Data Visualization</a></li></ul><p>I would add one last tip here. I believe it is the most important one.</p><blockquote>Read, a lot.</blockquote><p>Reading is essential to understand the function and history of our society. It is vital to develop our mind by discovering new things. Also, reading is a crucial aspect when we are talking about imagination and curiosity.</p><p>The more you read, the more you understand the subject you read. Now linking it back to Data Science, data is about everything. A reader can learn concepts faster about anything he reads. This can make a huge difference giving sense to data, giving the needed background to the numbers so that numbers can become ideas and insights.</p><blockquote><em>The more that you read, the more things you will know. The more you learn the more places you will go. -Dr. Suess</em></blockquote><h3>References</h3><p>[1] Fayyad, Usama, Gregory Piatetsky-Shapiro, and Padhraic Smyth. “From data mining to knowledge discovery in databases.” <em>AI magazine</em> 17, no. 3 (1996): 37–37.</p><p>[2] Cleveland, William S. “Data science: an action plan for expanding the technical areas of the field of statistics.” <em>International statistical review</em> 69, no. 1 (2001): 21–26.</p><p><a href="https://towardsdatascience.com/data-scientists-are-thinkers-a36cc186d570">Data Scientists Are Thinkers</a></p><p><em>Originally published at </em><a href="https://towardsdatascience.com/what-is-really-data-about-a60a2af1cfaa"><em>https://towardsdatascience.com</em></a><em> on May 15, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad46c536e86e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tutorial — Transfer Learning]]></title>
            <link>https://medium.com/ensina-ai/tutorial-transfer-learning-3972cac5e9b5?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/3972cac5e9b5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Thu, 09 May 2019 09:18:04 GMT</pubDate>
            <atom:updated>2019-05-09T09:18:04.778Z</atom:updated>
            <content:encoded><![CDATA[<h3>Tutorial — Transfer Learning aplicado no reconhecimento de flores</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CozdEs7UUbA-Mo-VaDr4AA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/9z3PSK4sn1c?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Zach Taiji</a> on <a href="https://unsplash.com/search/photos/tulips-plantation?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Considerações iniciais</h3><p>O objetivo deste projeto é classificar fotos de algumas flores. Mais especificamente, a partir de um conjunto de dados do Kaggle, temos cinco tipos de flores: Margaridas, Dentes de Leão, Rosas, Girassóis e Tulipas, totalizando 4242 imagens.</p><p>Além disso, este projeto pode ser visto também como um tutorial de <em>Transfer Learning, </em>transferência de aprendizado usando uma CNN (Rede neural convolucional). Há um guia passo-a-passo para realizar um classificador CNN usando o Keras com um modelo projetado por mim e outro usando o <em>Transfer Learning</em>.</p><blockquote>O que é Transfer Learning?</blockquote><p>É a reutilização de um modelo pré-treinado em um novo problema. Isto é, vou usar uma rede neural treinada em outro outro conjunto de dados, geralmente maior, para resolver um novo problema.</p><blockquote>Por que usar o Transfer Learning?</blockquote><ul><li>Já que é difícil obter um conjunto de dados grande o suficiente, raramente CNNs são treinadados do zero;</li><li>CNN muito profundas são muito caras para serem treinadas. Os modelos mais complexos podem demorar uma semana utilizando centenas de GPUs muito caras (Tesla V100 e cia);</li></ul><p>A maioria dos problemas envolvendo visão computacional usam conjuntos de dados aparentemente grandes (4000 - 20000 imagens), mas que ainda não são grandes o suficiente para se treinar adequadamente uma CNN. Assim, treinar estas redes com centenas de milhões de parametros quase sempre criam o problema do <em>overfitting. </em>Então, usamos o <em>Transfer Learning</em> para nos ajudar.</p><p>Para este artigo, todos dados usados estão disponíveis no Kaggle:</p><p><a href="https://www.kaggle.com/alxmamaev/flowers-recognition">Flowers Recognition</a></p><h3>Pré-processamento de dados</h3><p>Como vamos fazer um tutorial prático sobre o <em>Transfer Learning, </em>é necessário um breve tratamento dos dados antes de realizar o treinamento da rede neural. Para tudo ficar bem claro, vou colocar bastante código no artigo e também, todo o código está disponível no <a href="https://github.com/lmeazzini/Flower-Recognition">GitHub</a>.</p><h4>Importando bibliotecas</h4><pre>import warnings<br>warnings.filterwarnings(&#39;always&#39;)<br>warnings.filterwarnings(&#39;ignore&#39;)</pre><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>%matplotlib inline  <br>import seaborn as sns</pre><pre>from sklearn.model_selection import train_test_split</pre><pre>import keras<br>from keras.preprocessing.image import ImageDataGenerator<br>from keras.models import Model<br>from keras.optimizers import Nadam<br>from keras.utils import to_categorical<br>from keras.layers import Dropout, Flatten, Input, Dense<br>from keras.layers import Conv2D, MaxPooling2D, BatchNormalization</pre><pre>import random<br>import tensorflow as tf<br>import cv2 as cv<br>import os <br>import glob</pre><h4>Lendo e tratando os dados</h4><p>Depois de importar tudo que será utilizado, vamos ler as imagens de flores disponíveis. Na pasta “dandelion” (dente de leão), existiam arquivos que não eram imagens, então eu os deletei manualmente.</p><pre>daisy_dir = glob.glob(os.path.join(&#39;daisy/&#39;, &#39;*&#39;))<br>dandelion_dir = glob.glob(os.path.join(&#39;dandelion/&#39;, &#39;*&#39;))<br>rose_dir = glob.glob(os.path.join(&#39;rose/&#39;, &#39;*&#39;))<br>sunflower_dir = glob.glob(os.path.join(&#39;sunflower/&#39;, &#39;*&#39;))<br>tulip_dir = glob.glob(os.path.join(&#39;tulip/&#39;, &#39;*&#39;))</pre><pre>X_path = daisy_dir + dandelion_dir + rose_dir + sunflower_dir + tulip_dir</pre><pre>X = []</pre><pre>for f in X_path:<br>    X.append(np.array(cv.resize(cv.imread(f), (224,224), interpolation = cv.INTER_AREA))) <br>    <br>X = np.array(X)</pre><pre>X = X / 255</pre><p>Cada imagem tem um tamanho diferente, e como sabemos, para entrar numa rede neural todas amostras precisam ter as mesmas dimensões, no caso, escolhi 224x244, isto porque as arquiteturas mais comuns com pesos disponíveis para o <em>Transfer Learning</em> usam estas mesmas dimensões.</p><p>Para normalizar os dados, simplesmente efetuamos uma divisão por 255 já que os valores de cada pixel são RGB variando de 0 à 255.</p><h4>Definindo as <em>labels</em></h4><p>A entrada de nossa rede neural já está pronta, agora devemos definir os rótulos (<em>labels</em>) para o treinamento.</p><pre>l_daisy = np.zeros(len(daisy_dir))<br>l_dandelion = np.ones(len(dandelion_dir))<br>l_rose = 2*np.ones(len(rose_dir))<br>l_sunflower = 3*np.ones(len(sunflower_dir))<br>l_tulip = 4*np.ones(len(tulip_dir))</pre><pre>y = np.concatenate((l_daisy, l_dandelion, l_rose, l_sunflower, l_tulip))<br>y = to_categorical(y, 5)</pre><p>Desta forma, nossas <em>labels</em> estão todas codificadas com o <em>One-Hot-Encoding.</em></p><h4>Dando uma olhada em nossas imagens</h4><p>Particularmente, sempre gosto de dar uma olhada no conjunto de dados antes de sair classificando tudo. Então resolvi mostrar algumas imagens do nosso conjunto. Olhando mais detalhadamente todo o conjunto, notei que existem algumas (menos de 10) imagens com <em>labels </em>erradas e que nem flores são. Preferi não deletar estas imagens para manter o experimento com o conjunto original do Kaggle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qe366PBEGDQ2YQ_5_qw2bQ.png" /></figure><h4>Separando treinamento e validação</h4><p>Agora vamos separar o nosso conjunto e duas partes. Uma para o treinamento e outra para a validação. Idealmente, deveríamos separar isto em três partes, para avaliarmos o desempenho do classificar em um conjunto nunca visto nem para a validação.</p><pre>X_train, X_val, y_train, y_val = train_test_split(X, y, test_size = 0.2, random_state=42)</pre><p>Para evitar problemas de <em>overfitting</em>, usei o <em>ImageDataGenerator</em>.</p><pre>datagen = ImageDataGenerator(<br>        zoom_range = 0.1, # Aleatory zoom<br>        rotation_range= 15, <br>        width_shift_range=0.1,  # horizontal shift<br>        height_shift_range=0.1,  # vertical shift<br>        horizontal_flip=True,  <br>        vertical_flip=True)</pre><pre>datagen.fit(X_train)</pre><h3>CNN do zero</h3><p>Agora sim, vamos fazer uma CNN simples.</p><pre>inp = Input((224,224,3))</pre><pre>conv1 = Conv2D(64, (5,5), padding=&#39;valid&#39;, activation= &#39;relu&#39;)(inp)<br>conv1 = MaxPooling2D(pool_size=(2,2))(conv1)<br>conv1 = BatchNormalization()(conv1)</pre><pre>conv2 = Conv2D(96, (4,4), padding=&#39;valid&#39;, activation= &#39;relu&#39;)(conv1)<br>conv2 = MaxPooling2D(pool_size=(2,2))(conv2)<br>conv2 = BatchNormalization()(conv2)</pre><pre>conv3 = Conv2D(128, (3,3), padding=&#39;valid&#39;, activation= &#39;relu&#39;)(conv2)<br>conv3 = MaxPooling2D(pool_size=(2,2))(conv3)<br>conv3 = BatchNormalization()(conv3)</pre><pre>conv4 = Conv2D(256, (3,3), padding=&#39;valid&#39;, activation= &#39;relu&#39;)(conv3)<br>conv4 = MaxPooling2D(pool_size=(2,2))(conv4)<br>conv4 = BatchNormalization()(conv4)</pre><pre>flat = Flatten()(conv4)</pre><pre>dense1 = Dense(512, activation= &#39;relu&#39;)(flat)<br>dense1 = Dropout(0.5)(dense1)</pre><pre>dense2 = Dense(64, activation= &#39;relu&#39;)(dense1)<br>dense2 = Dropout(0.1)(dense2)</pre><pre>out = Dense(5, activation = &#39;softmax&#39;)(dense2)</pre><pre>model = Model(inp, out)</pre><pre>model.compile(optimizer = Nadam(lr = 0.0001) , loss = &#39;categorical_crossentropy&#39;, metrics=[&#39;accuracy&#39;])</pre><p>E finalmente, treiná-la:</p><pre>history = model.fit(X_train, y_train, batch_size = 32, epochs = 50, initial_epoch = 0, validation_data = (X_val, y_val))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*ymJTsl4fmxw21CHwqX-GoQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*rYOWtO4dqGdaBGNcR8jzsA.png" /></figure><p>É fácil ver o problema de <em>overfitting</em> nos gráficos acima. A acurácia de treinamento está próxima de 1 enquanto, no teste próxima de 0.7.</p><p>Existem várias formas de reduzir o <em>overfitting, </em>como a regularização <em>BatchNormalization, Dropout </em>e também o aumento do conjunto dados. Vamos nos focar nesta ideia de aumentar o conjunto de dados. Sabendo que não temos mais imagens de flores disponíveis, vamos usar o <em>Transfer Learning</em> como uma forma de reduzir o <em>overfitting.</em></p><h3>Transfer Learning</h3><p>Vamos utilizar o modelo da VGG-16, com pesos treinados na<em> ImageNet, </em>este conjunto de dados é composto por mais de 14 milhões de imagens classificadas em mais de 20000 classes. Sendo assim, é um conjunto substancialmente maior que nosso conjunto de flores.</p><pre>vgg = keras.applications.VGG16(input_shape=(224,224,3), include_top = False, weights= &#39;imagenet&#39;)</pre><p>Desta forma, temos um modelo composto apenas pelas camadas convolucionais pré-treinadas da VGG-16. E agora?</p><p>Tendo em mente que os recursos da CNN são mais genéricos (ex. detector de borda) nas camadas iniciais e mais específicos (ex. pétalas) do conjunto de dados originais nas camadas posteriores, aqui estão algumas regras comuns para navegar pelos 4 principais cenários:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*BwCvo9RY5YzKkbtx1vIwZg.png" /></figure><h4>Conjunto de dados pequeno e diferente do modelo pré-treinado</h4><ul><li>Adiciona-se camadas densamente conectas depois das camadas convolucionais;</li><li>Congelamos os pesos das primeiras camadas convolucionais;</li><li>Treina-se a rede neural.</li></ul><h4>Conjunto de dados grande e diferente do modelo pré-treinado</h4><ul><li>Adiciona-se camadas densamente conectas depois das camadas convolucionais;</li><li>Treina-se a rede neural usando os pesos da ImageNet como valores inciais.</li></ul><h4>Conjunto de dados grande e similar ao do modelo pré-treinado</h4><ul><li>Adiciona-se camadas densamente conectas depois das camadas convolucionais;</li><li>Pode-se treinar todo o modelo ou congelar os pesos das primeiras camadas convolucionais.</li></ul><h4>Conjunto de dados pequeno e similar ao modelo pré-treinado</h4><ul><li>Adiciona-se camadas densamente conectas depois das camadas convolucionais;</li><li>Congelar os pesos das primeiras camadas convolucionais ou todas camadas convolucionais.</li></ul><p>Então, em qual dos casos nosso problema de reconhecimento de flores se enquadra? Lembrando que o conjunto de dados possui 4242 amostras de cinco possíveis classes.</p><p>Perto do conjunto da ImageNet, nosso conjunto de flores é muito pequeno, então vamos adicionar camadas densamente conectadas e congelar o treinamento de todas as camadas convolucionais.</p><pre>x = vgg.output<br>x = Flatten()(x)<br>x = Dense(3078,activation=&#39;relu&#39;)(x) <br>x = Dropout(0.5)(x)<br>x = Dense(256,activation=&#39;relu&#39;)(x) <br>x = Dropout(0.2)(x)<br>out = Dense(5,activation=&#39;softmax&#39;)(x)</pre><pre>tf_model=Model(inputs=vgg.input,outputs=out)</pre><pre>for layer in tf_model.layers[:20]:<br>    layer.trainable=False</pre><p>Finalmente, vamos treinar nosso modelo:</p><pre>history = tf_model.fit(X_train, y_train, batch_size = 1, epochs = 30, initial_epoch = 0, validation_data = (X_val, y_val))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*weUj3VcHHRDU5TZqe8xJow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*je___X113I9bYz6iuSxZiA.png" /></figure><p>Claramente, a acurácia melhorou bastante (85% no conjunto de validação). No entanto, ainda existe o problema de <em>overfitting</em>. Talvez treinando todas camadas convolucionais o modelo teria um desempenho melhor, mas isto fica para você treinar!</p><h4>Classificando alguns exemplos</h4><p>Vamos classificar alguns exemplos de flores para ver onde é que nosso modelo está errando mais.</p><pre>pred = tf_model.predict(X_val)<br>pred = np.argmax(pred, axis = 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*V6zeCIwG2oLkwyD3HLFkHQ.png" /></figure><p>Podemos ver que as fotos dentre estes exemplos, algumas flores não estão em seu formato mais comum ou estão em grande quantidade ou em baixa resolução ou ainda existe mais de um tipo de flor na foto. Todos estes fatores são impactantes na classificação de nosso modelo neural.</p><h3>Conclusão</h3><p>O <em>Transfer Learning</em> definitivamente será um dos principais impulsionadores para o <em>Machine Learning</em> e para o sucesso do <em>Deep Learning</em> na adoção <em>mainstream</em> na indústria. Através dele é possível acelerar e melhorar um modelo profundo com facilidade, além de também reduzir os custos computacionais atrelados ao treinamento.</p><p>Existem diversas outras arquiteturas pré-treinadas como:</p><ul><li>Xception</li><li>VGG-19</li><li>ResNet, ResNetV2, ResNeXt</li><li>InceptionV3</li><li>MobileNet, MobileNetV2</li><li>DenseNet</li><li>NASNet</li><li>etc.</li></ul><p>Todo o código usado para este artigo está disponível no repositório abaixo:</p><p><a href="https://github.com/lmeazzini/Flower-Recognition">lmeazzini/Flower-Recognition</a></p><blockquote>Eu gosto de coisas sem sentido; elas acordam as coisas do cérebro. (Dr. Seuss)</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3972cac5e9b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ensina-ai/tutorial-transfer-learning-3972cac5e9b5">Tutorial — Transfer Learning</a> was originally published in <a href="https://medium.com/ensina-ai">Ensina.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning — Diagnosing faults on vehicle trackers with a CNN]]></title>
            <link>https://medium.com/data-science/machine-learning-diagnosing-faults-on-vehicle-trackers-with-a-cnn-5b471a86b7c0?source=rss-335ed336aced------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b471a86b7c0</guid>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Luis Meazzini]]></dc:creator>
            <pubDate>Thu, 02 May 2019 23:15:28 GMT</pubDate>
            <atom:updated>2019-05-02T23:40:46.121Z</atom:updated>
            <content:encoded><![CDATA[<h3>Machine Learning — Diagnosing faults on vehicle trackers with a CNN</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VoYrarHb9DLoPn5uAlAXOg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/rZ7aO1lorLU?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Omer Rana</a> on <a href="https://unsplash.com/search/photos/parking-lot?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Initial Considerations</h3><p>Several weeks ago, I published another article with almost the same title:</p><p><a href="https://towardsdatascience.com/machine-learning-diagnosing-faults-on-vehicle-fleet-trackers-56b123d77cf5">Machine Learning — Diagnosing faults on vehicle fleet trackers</a></p><p>On that Storie, I used some specific knowledge of the tracking modules operation to extract features manually; now, on this Storie, I aim to diagnose faults on the same dataset without any feature extraction. For those who did not read the first article, the introduction and problem description will be the same. However, all the development will not.</p><h3>Introduction</h3><p>Many transport and logistics companies use modules to track their vehicles, and it is not unusual that some modules are not working as expected. The tracker might need to be discarded, wrong data from the vehicle can be collected, or technical support can be sent a long distance away to analyze the problem, for example. Thus, it is essential to conduct a remote fault diagnosis on those modules since, when they show failures, financial losses can occur.</p><p>Since these modules continuously send data to a database, the aim of this research is using this data to diagnose faults on the modules. It was necessary to develop a methodology to pre-processing the collected data sent by the vehicle modules. After this, machine learning techniques were applied to create models, which were analyzed and compared. However, what is the fault?</p><p>Faults are an unallowed deviation of at least one property or a specific parameter of the system [1]. Moreover, there are three steps to diagnose faults, which are:</p><ol><li><strong>Fault Detection</strong>: is the most basic task of fault diagnosis, it is used to check for malfunction or failure in the system and to determine when the fault occurs;</li><li><strong>Fault Isolation</strong>: the isolation serves to detect the location of the failure or which is the defective component;</li><li><strong>Fault Identification</strong>: the identification is used to determine the type, format, and size of the failure.</li></ol><p>Generally, the fault identification process consists only of Fault Detection and Isolation (FDI). That does not negate the utility of fault identification. However, this process may not be essential if no reconfiguration action is involved [2].</p><h3>Problem description</h3><p>First of all, to perform any machine learning solution, data is needed. For this case study, a Brazilian logistics company provided some data sent by many trackers, <strong>so all the data used is real.</strong></p><p>The modules installed in the vehicles send data during their entire period of operation. The company provided 12586 registries. The company provided 12586 registries. A registry is composed of all the data transmitted by one module in one day period. On average, each module sends 1116.67 points per day. Each point has eight attributes:</p><ul><li><strong>Battery Voltage</strong>: Float value for voltage of the vehicle battery;</li><li><strong>Longitude</strong>: Float value for the vehicle longitude;</li><li><strong>Latitude</strong>: Float value for the vehicle latitude;</li><li><strong>Ignition</strong>: Boolean value indicating whether the ignition is<br>switched on;</li><li><strong>Odometer</strong>: Float value for the vehicle odometer;</li><li><strong>GPS Signal</strong>: Boolean value indicating whether the GPS signal<br>is valid;</li><li><strong>Velocity</strong>: Float value for the vehicle speed;</li><li><strong>Memory index</strong>: Integer value for the memory position that the<br>point is saved in the module.</li></ul><p>Since the modules send data regularly and on different frequencies according to each module, each registry has a different size. This way, the size of each registry is the number of points sent multiplied by 8 (quantity of attributes).</p><p>The aim is identifying the faults that a registry can have or if it is working correctly. This way, this is a multi-class classification problem, where there are eight classes; one class for the registries without faults, and seven possible faults, listed below:</p><ul><li><strong>Fault 1</strong>: Wrong pulse set up;</li><li><strong>Fault 2</strong>: Odometer locked;</li><li><strong>Fault 3</strong>: GPS locked;</li><li><strong>Fault 4</strong>: Ignition wire released;</li><li><strong>Fault 5</strong>: Accelerometer defective;</li><li><strong>Fault 6</strong>: Module buffer with a problem;</li><li><strong>Fault 7</strong>: GPS with jumps in location;</li></ul><p>Some of these faults have several occurrences too low. The flaws that had less than 3% were all labeled as “Others”. The quantity of data of each failure can be seen on the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*EHoIxQXsO3_Eyx6aLmHOGQ.png" /></figure><h3>Development</h3><p>I aim to detect and isolate the faults; however, while just detecting faults is a binary problem, detection and isolation is a multi-class problem. It might be much easier to detect the faults. Look by the company side, almost always, when a module present faults, they need to replace it. So it does not matter much what is the fault, anyway we need to replace the module.</p><blockquote>We propose to models here, one to detect faults and another to detect and isolate the faults.</blockquote><h4>Data pre-processing</h4><p>As we aim to make minor changes in the dataset, without using any specific knowledge about the system operation, the data pre-processing is reasonably straightforward.</p><p>We choose to keep all the data, not performing any outlier analysis. Mostly outlier is a strong indication that there is a fault, if we throw all the outliers away, a lot of faulty registries would be gone. This way, the only thing we do is a normalization, using the RobustScaler. A traditional normalization can cause the data to be suppressed by some high value, so RobustScaler is used. This scaler uses the first and third quartiles as constraints and not the maximum and minimum values, so this normalization is more robust and maintains the outliers.</p><p>Considering the registries have variable length, it was not possible to directly construct the ML models with this data set. In this way, it was necessary to set a default size for all retries. Knowing that the highest number of points present in a registry is 6410, the mean of the number of points is 1116.67, and the standard deviation is 1155.75, the registries with a smaller amount of data were completed with “0&#39;’ until they have the same amount of data as the largest registry.</p><p>Different approaches to set the default size of a registry could be performed, such as truncating the amount of data from above-average registries, and completing with “0&#39;’ those that are below average, or truncating the amount of data from all registries with more data than the smaller registry. However, these approaches were tested and considered unfeasible due to the divergence of the CNN’s.</p><h4>CNN architecture</h4><p>I made a model based on VGG-16.</p><blockquote>Using 4 convolutional blocks. Each like Conv-&gt;Conv-&gt;Pooling. Then two fully connected layers.</blockquote><p>The idea on the convolutions is to make the filters choose and modify the attributes. To do that, we disposed of each registry with (8, 6410, 1) shape. This way, the CNN threats the data like an image. To check more deeply the architecture, check my GitHub (the link is on the introduction and at the conclusion).</p><p>The used batch size was 1, learning rate 0.0001 and he_uniform initializer. The batch_size can be higher. However, due to hardware restrainments, three was higher for me.</p><h4>Model Training</h4><p>Two models were trained, one for fault detection and another to fault detection and isolation. Thus, two structures of the same dataset were defined. These structures are described as follows:</p><ul><li><strong>Structure 1</strong>: was used in the experiments to detect if the system had faults or not. It contains 12586 records, of which 7480 are flawed, and 5106 are flawless;</li><li><strong>Structure 2</strong>: was used in the experiments to detect and isolate faults. It contains 5106 records flawless, 2170 with failure 2, 1573 with failure 3, 1702 with failure 4, and 2035 with “Others” failure, totaling 12586 records.</li></ul><p>All the models were written in Python 3.6 with Sk-learn 0.20.2 and ran on Ubuntu 18.04 with an Intel i7–7700HQ, 16Gb RAM, GTX 1050Ti.</p><p>To run the algorithms, the datasets were separated into a training and a test set. For every experiment, 20% of the registries were for testing and 80% for training.</p><h4>Model Evaluation</h4><p>The metrics need adequate interpretation. In the case study in question, it is crucial to minimize the number of false positives, as this would involve sending a technician to perform unnecessary maintenance. While a false positive implies receiving a complaint of some malfunctioning module. Then, from a financial point of view, the recall has greater relevance than precision.</p><p>With 88.42 Precision and 87.96 Recall, the confusion matrix for the fault detection can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*hc4dAYExbZTpWwcGJ6yN5A.png" /></figure><p>With 54.98 Precision and 52.57 Recall, the confusion matrix for the fault detection and isolation can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*FGb8FGZbu1_FnhMX2sU0MA.png" /></figure><p>For both models, there is high variance and high bias. The models could have better results if the architecture was deeper or changing some hyperparameters. Alternatively, even performing more extensive data pre-processing.</p><h3>Conclusions</h3><p>As proposed, these models can detect and isolate faults on vehicle fleet tracking modules. However, the evaluative metrics are not suitable to use the models. On my other article, using knowledge from the systems, the metrics were higher can be me implemented at the company to diagnose faults on their modules remotely.</p><p>The limitations of the employed methods include the inability of the models to discover faults that are not mapped, so any faults which have not been learned would be misclassified to be one of the known ones.</p><p>As in the article, there was not any code, feel free to check it out on my GitHub repository.</p><p><a href="https://github.com/lmeazzini/Tracker-fault-diagnosis">lmeazzini/Tracker-fault-diagnosis</a></p><h3>References</h3><p>[1] D. van Schrick, “Remarks on terminology in the field of<br>supervision, fault detection and diagnosis,” IFAC Proceedings<br>Volumes, vol. 30, no. 18, pp. 959–964, 1997.</p><p>[2] D. Wang and W. T. Peter, “Prognostics of slurry pumps based<br>on a moving-average wear degradation index and a general<br>sequential monte carlo method,” Mechanical Systems and<br>Signal Processing, vol. 56, pp. 213–229, 2015.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b471a86b7c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/machine-learning-diagnosing-faults-on-vehicle-trackers-with-a-cnn-5b471a86b7c0">Machine Learning — Diagnosing faults on vehicle trackers with a CNN</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>