<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Madalina Ciortan on Medium]]></title>
        <description><![CDATA[Stories by Madalina Ciortan on Medium]]></description>
        <link>https://medium.com/@ciortanmadalina?source=rss-d576810f0222------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-EoA2VigwlZdu6qi7NmIuQ.jpeg</url>
            <title>Stories by Madalina Ciortan on Medium</title>
            <link>https://medium.com/@ciortanmadalina?source=rss-d576810f0222------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:27:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ciortanmadalina/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Kaggle competitions to Claude Code subscriptions]]></title>
            <link>https://medium.com/swlh/from-kaggle-competitions-to-claude-code-subscriptions-40aaa43f954b?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/40aaa43f954b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[language]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[slang]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Thu, 02 Apr 2026 09:00:12 GMT</pubDate>
            <atom:updated>2026-04-02T09:00:12.978Z</atom:updated>
            <content:encoded><![CDATA[<h4>A philosophical detour through AI slop, developmental bypass, and why slang matters more than your textbooks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V7pfGE9CAfNSX5L2kp9n7A.jpeg" /></figure><h3>When everyone can build, what is valuable?</h3><p>I spent years debugging vanishing gradients in deep neural networks. Now a PM with a Claude Code subscription ships entire apps before lunch.</p><p>I’m not bitter. I’ve been in data science for two decades —bioinformatics and deep learning research, consultancy for big companies, AI leadership. What’s happening is genuinely extraordinary. But it forces a question most of us avoid: if everyone can build, what’s actually valuable?</p><p>The question of what makes something valuable has haunted economists for centuries. Is it the labor you put in? The scarcity of the skill? The market’s willingness to pay? When AI can generate in seconds what took us years to learn, all three shift at once. But this isn’t collapse — it’s reallocation. Schumpeter called it creative destruction: capitalism’s engine for turning today’s premium skill into tomorrow’s commodity, freeing human energy for whatever comes next (<em>Capitalism, Socialism, and Democracy</em>, 1942). It’s how we got from agriculture to industry to knowledge work. Each time, the transition felt like loss. Each time, what emerged was more valuable than what disappeared. AI is the latest wave. The question isn’t whether it destroys old value — it will. The question is what new value emerges on the other side, and whether you’re building toward it.</p><h3>Drowning in output: welcome to the slop age</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CicLoVsXtFbNdJwJYzzAKg.png" /></figure><p>The data on what mass-produced AI output actually looks like is now definitive.</p><p>A <a href="https://byteiota.com/ai-technical-debt-30-41-increase-hits-developers/">2026 study analyzing 8.1 million PRs from 4,800 teams</a> found AI-generated code contains 1.7x more issues than human code, with technical debt rising 30–41% within 90 days of adoption. <a href="https://blog.exceeds.ai/comparative-ai-code-quality-governance/">2026 benchmarks</a> show code complexity growing 3.28x and static analysis warnings up 4.94x in some environments. An <a href="https://arxiv.org/html/2603.28592v1">arXiv paper from March 2026</a> tracking AI-authored commits across real repositories found over 110,000 surviving AI-introduced issues accumulating by February 2026 — debt that becomes permanent maintenance burden.</p><p>Incidents per PR are up 23.5%. Change failure rates up 30%. PRs wait <a href="https://byteiota.com/ai-technical-debt-30-41-increase-hits-developers/">4.6x longer for review</a> because reviewers can’t keep up with volume. The estimated quality deficit for 2026 is 40% — the gap between code generated and code properly reviewed.</p><p>But slop isn’t just code. LinkedIn drowns in AI-generated thought leadership that says nothing. Medium fills with SEO-optimized articles that rank well and inform poorly. A <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC12634010/">2025 study in <em>JMIR Medical Education</em></a> found growing AI “slop” in educational videos — what researchers formally defined as “careless speech,” produced without genuine attention. AI images flood stock sites and gallery submissions. The pattern is universal: when production cost drops to zero, volume explodes and quality becomes collateral damage.</p><h3>The new currency is judgment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MkHc6zUllHesz3xhBWYeww.jpeg" /></figure><p>The industry’s response is <strong>spec-driven development</strong> — writing precise specifications before AI generates anything. <a href="https://www.thoughtworks.com/en-us/insights/blog/agile-engineering-practices/spec-driven-development-unpacking-2025-new-engineering-practices">Thoughtworks calls it</a> one of the most important practices to emerge recently. <a href="https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/">GitHub open-sourced Spec Kit</a>. AWS launched Kiro with separate Vibe and Spec modes.</p><p>But there’s a catch. Research on LLM context management shows that <a href="https://research.trychroma.com/context-rot">longer, more bloated inputs actually degrade model performance</a> — even when all relevant information is present. Verbose specs, the kind LLMs naturally produce, can make output <em>worse</em>. This demands something specific from the human: the ability to write tight, precise, minimal specifications. Decompose problems into granular tasks. Review output critically. Spot architectural flaws the AI can’t see.</p><p>Everyone now needs to think like a senior architect — decomposition, constraints, judgment under uncertainty. And that’s precisely the skill that was already scarce. Juniors were still learning it. Now the volume of AI output requiring review has exploded while cognitive deskilling erodes the muscles needed to do the reviewing. A <a href="https://myscp.onlinelibrary.wiley.com/doi/full/10.1002/arcp.70008">2026 paper in <em>Consumer Psychology Review</em></a> found that people moving from AI-assisted to unaided work showed <em>reduced</em> cognitive effort — “cognitive debt.” The AI didn’t just do the work. It made them worse at doing it themselves.</p><p>The calorie analogy writes itself. Our bodies evolved for scarcity, then got unlimited fat and sugar. Result: obesity. Our minds evolved for effortful learning, then got unlimited cognitive shortcuts. Result: the competence bubble — where the gap between what people produce and what they understand has never been wider.</p><h3>What survives and the risk of developmental bypass</h3><p>Matthew Crawford argued in <em>The World Beyond Your Head</em> (2015) that attention is the most valuable resource in a distracted economy — and that skilled, effortful practice is what makes us competent agents rather than passive consumers. In a world where everyone can produce, what matters is what you pay <em>attention</em> to.</p><p>Psychotherapist John Welwood coined the term “spiritual bypassing” — using practices to sidestep unresolved developmental tasks. AI creates its own version: <strong>developmental bypass</strong>, where we skip the struggle that builds the capacity. A child who never handles frustration doesn’t develop resilience. A developer who never debugs a gradient descent implementation can use the tool but can’t <em>judge</em> the tool. We’re outsourcing not just labor, but the understanding that makes labor meaningful.</p><p>But not every skill can be bypassed. You can vibecode an app without understanding architecture. You can generate a research summary without reading the papers. But you cannot shortcut your way into belonging somewhere. No AI generates the feeling of a friend laughing because you nailed an expression in their language — not the textbook version, the <em>real</em> one, the one that signals <em>I’m not just passing through. I actually get it.</em> If you’ve ever lived abroad, you know: you can pass the C1 exam and still feel like a permanent outsider. Because fluency and connection are different things. Textbooks teach you how to order coffee. They don’t teach you how people speak when they’re being real with each other.</p><p>That’s a gap that matters. As AI commoditizes every technical skill, the human premium shifts to exactly this: the capacity to connect across barriers that no model can see.</p><h3>So I built something</h3><p>I’ve been an expat for most of my adult life, relocated to 5 countries. I’ve had to reach fluency in English, French, Spanish, and Catalan — each time starting from the same place: technically competent but socially invisible. Because there’s a label that sticks to you when you don’t know the informal language: <em>foreigner</em>. And once locals categorize you that way, they don’t invest. Conversations stay shallow. Doors stay polite but closed.</p><p>What unsticks that label isn’t grammar. It’s knowing the expressions people actually use when they’re not being careful. The slang, the idioms, the phrases that signal <em>this person gets it</em>.</p><p>So I built <a href="https://slangy.fit/"><strong>slangy.fit</strong></a> — for myself first, out of pure frustration. It teaches the informal expressions I kept hearing in real life but never found in my textbooks, in Spanish, French, and English. Three expressions a day. Each one linked to a song, a clip, or a video where you can see it used for real. All free.</p><p>Here’s what a round looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DJltgxvUVxDKawRa7a9THg.jpeg" /><figcaption><a href="https://slangy.fit/expressions/en/it-s-gone-tits-up">https://slangy.fit/expressions/en/it-s-gone-tits-up</a></figcaption></figure><p>Because in a world drowning in slop, the most radical thing you can build is something that helps two people actually understand each other.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40aaa43f954b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/from-kaggle-competitions-to-claude-code-subscriptions-40aaa43f954b">From Kaggle competitions to Claude Code subscriptions</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of feature selection methods]]></title>
            <link>https://medium.com/data-science/overview-of-feature-selection-methods-a2d115c7a8f7?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2d115c7a8f7</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-selection]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Fri, 26 Jul 2019 16:04:50 GMT</pubDate>
            <atom:updated>2019-07-26T17:44:35.219Z</atom:updated>
            <content:encoded><![CDATA[<h4>Common strategies for choosing the most relevant features in your data set</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-FpgTw6swVcwDYdEMvHyBg.jpeg" /></figure><h4>The importance of feature selection</h4><p>Selecting the right set of features to be used for data modelling has been shown to improve the performance of supervised and unsupervised learning, to reduce computational costs such as training time or required resources, in the case of high-dimensional input data to mitigate the <em>curse of dimensionality. </em>Computing and using feature importance scores is also an important step towards model interpret-ability.</p><h4>Overview of this post</h4><p>This post shares the overview of supervised and unsupervised methods for performing feature selection I have acquired after researching the topic for a few days. For all depicted methods I also provide <em>references to open-source python implementations</em> I used in order to allow you to quickly test out the presented algorithms. However, this research domain is very abundant in terms of methods which have been proposed during the last 2 decades and as such this post only attempts to present my current limited view without any pretense for completeness. For a more comprehensive study, you can check the following <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.295.8115&amp;rep=rep1&amp;type=pdf">review</a>.</p><h3>Supervised/ Unsupervised models</h3><p>There are supervised feature selection algorithms which identify the relevant features for best achieving the goal of the supervised model (e.g. a classification or a regression problem)and they rely on the availability of labelled data. However, for unlabeled data, a number of unsupervised feature selection methods have been developed which score all data dimensions based on various criteria, such as their variance, their entropy, their ability to preserve local similarity, etc. Relevant features identified using unsupervised heuristics can also be used in supervised models as they may discover other patterns in the data additional to the correlation of features with the target variable.</p><p>From a taxonomic point of view, feature selection methods usually fall into one of the following 4 categories detailed below: <em>filter</em>, <em>wrapper</em>, <em>embedded </em>and <em>hybrid </em>classes.</p><h3>Wrapper methods</h3><p>This approach evaluates the performance of a subset of features based on the resulting performance of the applied learning algorithm (e.g. what is the gain in accuracy for a classification problem). Any learning algorithm can be used in this combination of search strategy and modelling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bVQEuQWqejWsPB4vkrZsDQ.png" /><figcaption>Image from <strong>Analytics Vidhya</strong></figcaption></figure><ul><li><strong>Forward selection</strong>: this approach starts with an empty set of features and then the dimensions providing the best performance are being iteratively added to the result set</li><li><strong>Backward selection</strong>: this approach starts from the set of all features and at each iteration the worst dimension is being removed</li></ul><p>Implementation: these algorithms are implemented in the <a href="https://rasbt.github.io/mlxtend/#examples">mlxtend </a>package, find here an <a href="https://stackabuse.com/applying-wrapper-methods-in-python-for-feature-selection/">example</a> of usage.</p><ul><li><strong>RFE </strong>(Recursive feature elimination): greedy search which selects features by recursively considering smaller and smaller sets of features. It ranks features based on the order of their elimination.</li></ul><p>Implementation: <a href="https://scikit-learn.org/stable/modules/feature_selection.html#recursive-feature-elimination">scikit-learn</a></p><h3>Embedded methods</h3><p>This approach consists in algorithms which simultaneously perform model fitting and feature selection. This is typically implemented by using a <strong>sparsity regularizer</strong> or constraint which makes the weight of some features become zero.</p><ul><li><strong>SMLR </strong>(Sparse Multinomial Logistic Regression): this algorithm implements a sparse regularization by ARD prior (Automatic relevance determination) for the classical multinational logistic regression. This regularization estimated the importance of each feature and prunes the dimensions which are not useful for the prediction.</li></ul><p>Implementation: <a href="https://github.com/KamitaniLab/smlr">SMLR</a></p><ul><li><strong>ARD </strong>(Automatic Relevance Determination Regression): based on a Bayesian Ridge Regression, this model will shift the coefficient weights towards zero more than methods like OLS for instance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*i3A9zSnYDfzLwzGp04MgTQ.png" /><figcaption>ARD sparsity constraint makes the weights of some features 0 and thus helps identify the relevant dimensions</figcaption></figure><p>Implementation: <a href="https://scikit-learn.org/stable/auto_examples/linear_model/plot_ard.html">scikit-learn</a></p><p>Other examples of regularization algorithms: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html">Lasso</a> (implementing <em>l1 </em>regularization), <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html">Ridge Regression</a> (implementing <em>l2 </em>regularization), <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html">Elastic Net</a> (implementing <em>l1 </em>and <em>l2 </em>regularization). An intuitive depiction of these different regularization types shows that Lasso regression constrains the coefficients to a square shape, ridge creates a circle and elastic net is in between:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*pbbhHlSxRU8VPXOzsLCxeQ.png" /><figcaption><a href="https://scikit-learn.org/stable/auto_examples/linear_model/plot_sgd_penalties.html">https://scikit-learn.org/stable/auto_examples/linear_model/plot_sgd_penalties.html</a></figcaption></figure><p>A comprehensive description of these algorithms can be found <a href="http://enhancedatascience.com/2017/07/04/machine-learning-explained-regularization/">here</a>.</p><h3>Filter methods</h3><p>This approach evaluates the importance of features based only on their inherent characteristics, without incorporating any learning algorithm. These methods tend to be faster and less computationally expensive than wrapper methods. If there is not enough data to model the statistical correlation between features, filter methods may provide worse results than wrapper methods. Unlike wrapper methods they are not subject to overfitting. They are extensively used on high dimensional data where wrapper methods have a prohibitive computational cost.</p><h4>Supervised methods</h4><ul><li><strong>Relief </strong>: This method samples randomly instances from the dataset and updates the relevance of each feature based on the difference between the selected instance and the two nearest instances of the same and opposite classes. If a feature difference is observed in the neighboring instances of the same class ( a ‘hit’), the feature score decreases, alternatively if the feature value difference is observed with a different score (a ‘miss’) then the feature score increases.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*EjkXuRcsUqQyQ-sw8uCM8w.png" /><figcaption>Feature weight decreases if it differs from that feature in nearby instances of the same class more than nearby instances of the other class and increases in the reverse case</figcaption></figure><p>The extended algorithm, ReliefF applies feature weighting and searches for more nearest neighbors.</p><p>Implementations: <a href="https://github.com/EpistasisLab/scikit-rebate">scikit-rebate</a>, <a href="https://pypi.org/project/ReliefF/">ReliefF</a></p><ul><li><strong>Fisher score</strong>: Typically used in binary classification problems, the Fisher ration (FiR) is defined as the distance between the sample means for each class per feature divided by their variances:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*qcOUM-MuuyXFmyRa3vW9-A.png" /></figure><p>Implementations :<a href="https://github.com/jundongl/scikit-feature"> scikit-feature</a>, usage <a href="http://featureselection.asu.edu/tutorial.php">example</a>.</p><ul><li><strong>Chi-squared score</strong>:tests if there is a significant difference between the observed and the expected frequencies of 2 categorical variables. Thus the Null hypothesis states that there is no association between both variables.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Jz5yrQMszk6FalTH-sfGWQ.png" /><figcaption>Chi square test of independence</figcaption></figure><p>In order to correctly apply the chi-squared in order to test the relation between various features in the dataset and the target variable, the following conditions have to be met: the variables have to be <em>categorical</em>, sampled <em>independently </em>and values should have an <em>expected frequency greater than 5</em>. The last condition ensures that the CDF of the test statistic can be closely approximated by the chi-square distribution, more details can be found <a href="https://sites.google.com/a/lakeheadu.ca/bweaver/Home/statistics/notes/chisqr_assumptions">here</a>.</p><p>Implementation: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html">sklearn</a>, <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chisquare.html">scipy</a></p><ul><li><a href="https://www.cs.waikato.ac.nz/~mhall/thesis.pdf"><strong>CFS </strong></a>(Correlation-based feature selection): The rationale of this method can be summarized as:</li></ul><blockquote>“ Features are relevant if their values vary systematically with category membership.”</blockquote><p>Thus, good feature subsets contain features highly correlated with the classification and uncorrelated to each other. The method calculates the merit of a subset of <em>k </em>features as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJnZEvKVUgeaAplqwDX_Yg.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Feature_selection#Correlation_feature_selection">https://en.wikipedia.org/wiki/Feature_selection#Correlation_feature_selection</a></figcaption></figure><p>Implementations :<a href="https://github.com/jundongl/scikit-feature"> scikit-feature</a>, usage <a href="http://featureselection.asu.edu/tutorial.php">example</a>.</p><ul><li><a href="http://www.public.asu.edu/~huanliu/papers/icml03.pdf"><strong>FCBF </strong></a>(Fast correlation-based filter): this method is faster and more efficient than both ReliefF and CFS and thus, more adapted for high dimensional input. In a nutshell, it follows a typical relevance-redundancy approach by computing first the Symmetrical Uncertainty (the information gain of x | y divided by the sum of their entropies) for all features, sorts them by this criteria and then removes the redundant features.</li></ul><p>Implementations: <a href="https://github.com/jundongl/scikit-feature/blob/master/skfeature/function/information_theoretical_based/FCBF.py">skfeature</a>, <a href="https://github.com/shiralkarprashant/FCBF">https://github.com/shiralkarprashant/FCBF</a></p><h4>Unsupervised methods</h4><ul><li><strong>Variance</strong>: has been shown to be an effective way to select relevant features which tend to have a higher variance score</li></ul><p>Implementation: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html">Variance Threshold</a></p><ul><li><strong>Mean absolute difference</strong>: computes the mean absolute difference from the mean value (<a href="https://github.com/ciortanmadalina/high_noise_clustering/blob/master/Unsupervised%20feature%20selection.ipynb">implementation</a>).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*yAaAnSOsHoJjcm5xoUjKTw.png" /><figcaption>Higher values tend to have more discriminative power</figcaption></figure><ul><li><strong>Dispersion ratio</strong>: the arithmetic mean divided by the geometric mean. Higher dispersion corresponds to more relevant features (<a href="https://github.com/ciortanmadalina/high_noise_clustering/blob/master/Unsupervised%20feature%20selection.ipynb">implementation</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*UGH7HBK9GjsN5eY3ejcXfg.png" /><figcaption><a href="https://www.sciencedirect.com/science/article/abs/pii/S0167865512001870">https://www.sciencedirect.com/science/article/abs/pii/S0167865512001870</a></figcaption></figure><ul><li><strong>Laplacian Score</strong>: is based on the observation that data from the same class is often close to each other and thus we can evaluate the importance of a feature by its power of locality preserving. The method consists in embedding the data on a nearest neighbor graph by using an arbitrary distance measure and then calculating a weight matrix. A laplacian score is then calculated for each feature and will have the property that smallest values correspond to the most important dimensions. However, in order to select a subset of features another clustering algorithm (e.g. k-means) is typically applied a-posteriori in order to select the best performing group</li></ul><p>Implementations :<a href="https://github.com/jundongl/scikit-feature"> scikit-feature</a></p><ul><li><strong>Laplacian Score combined with distance-based entropy</strong>: this algorithm builds on top of the Laplacian score and uses distance-based entropy to replace the typical k-means clustering and shows better stability in high-dimensional datasets (<a href="https://github.com/ciortanmadalina/high_noise_clustering/blob/master/Unsupervised%20feature%20selection.ipynb">implementation</a>)</li><li><strong>MCFS </strong>(Multi-Cluster Feature selection): a spectral analysis is performed to measure the correlation between different features. The top eigenvectors of the graph Laplacian are used to cluster the data and a feature score is being computed as explained in the original <a href="https://dl.acm.org/citation.cfm?id=1835848">paper</a>.</li></ul><p>Implementation : <a href="https://github.com/danilkolikov/fsfc">https://github.com/danilkolikov/fsfc</a></p><ul><li><strong>LFSBSS </strong>(Localised feature selection), <strong>weighted k-means</strong>, <strong>SPEC </strong>and <strong>Apriori </strong>algorithms have been described in this review <a href="https://pdfs.semanticscholar.org/f116/7e2e1fa07cdf432c10beb373e07efd6a5e58.pdf">paper</a> and are implemented in this <a href="https://github.com/danilkolikov/fsfc">package</a></li></ul><h3>Hybrid methods</h3><p>Another option of implementing feature selection consists in a hybrid approach of combining filter and wrapper methods into a 2 phase process: an initial filtering of features based on statistical properties (filter phase) followed by a second selection based wrapper methods.</p><h3>Other resources</h3><p>There is a very abundant literature tackling feature selection problem and this post only scratches the surface of the research work which has been done. I will provide links to other resources that exist and that I haven’t yet tried.</p><p>A comprehensive <a href="http://featureselection.asu.edu/algorithms.php">list</a> of other feature selection algorithms that I haven’t mentioned in this post have been implemented in <a href="https://github.com/jundongl/scikit-feature">scikit-feature</a> package.</p><p>Other ways to identify relevant features is by using <strong>PLS </strong>(Partial least squares) as exemplified in this <a href="https://www.idtools.com.au/variable-selection-method-pls-python/">post</a> or by performing linear dimensionality reduction techniques as presented <a href="https://jotterbach.github.io/2016/03/24/Principal_Component_Analysis/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2d115c7a8f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/overview-of-feature-selection-methods-a2d115c7a8f7">Overview of feature selection methods</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Clustering data with graph oriented techniques]]></title>
            <link>https://medium.com/@ciortanmadalina/clustering-data-with-graph-oriented-techniques-9946876f3f9b?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/9946876f3f9b</guid>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Sat, 22 Jun 2019 18:18:08 GMT</pubDate>
            <atom:updated>2019-06-22T18:18:08.971Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction to Louvain graph community detection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yvTpoKqNTBkURxamI_dzSQ.jpeg" /></figure><p>This post will show an alternative approach to clustering a dataset, which relies on graph oriented techniques, namely on louvain community detection.</p><h3>Louvain algorithm for community detection</h3><h4>What are graph communities?</h4><p>A community is a group of nodes within a network that are more densely connected than they are to the other nodes. The problem of community detection is the equivalent of partitioning a network into densely connected communities in which the nodes belonging to other communities are only sparsely connected. This achieved by optimizing a the modularity metric which evaluates the quality of a partitioning by how much more densely connected the nodes in a community are by how connected they would be on average, in a randomly defined network. The precise mathematical formulation is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*2uNICQKMhX5orkUjg-Y8Vw.png" /><figcaption>Definition of modularity</figcaption></figure><h4>Algorithm summary</h4><p>In a nutshell the Louvain algorithm works in two steps: assignment of nodes to communities followed by redefining the network based on the communities found in the first phase, which become new coarse-grained nodes.</p><p>During the first step a node is selected at a time and put into the same community with its neighbor which would maximize the modularity function. Next another node is selected and the same heuristic is repeated until all nodes have been visited. The second step consists in merging each community into one course grained node and performing step one on this newly defined network. The process stops when it is not possible to improve the modularity any further. The complexity of this algorithm is n *log (n). Full details about this algorithm can be found in the original <a href="https://arxiv.org/pdf/0803.0476.pdf">paper</a> and also on this <a href="https://www.quora.com/Is-there-a-simple-explanation-of-the-Louvain-Method-of-community-detection">forum</a>.</p><h3>Implementation</h3><p>I made available a simple example running on simulated Gaussian 10 D clusters on <a href="https://github.com/ciortanmadalina/high_noise_clustering/blob/master/graph-partitioning-louvain.ipynb">Github</a>.</p><p>We start by generating a less trivial test dataset, consisting of 6 clusters embedded in 10D:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dp2RIpNFong7sp2p82_biA.png" /></figure><p>I developed two different approaches, one relying on embedding the input data set in a nearest neighbor graph and then using this very representation as input to the Louvain algorithm. The second approach starts from a distance matrix and uses the similarity between points as edge weights passed to the Louvain algorithm.</p><h4>Connectivity based approach</h4><p>The first step is to obtain the connectivity matrix of the input data set and for this we will use Sklearn’s method <em>kneighbors_graph, </em>which computes the weighted graph of k-Neighbors points in the input data. This method can return either a connectivity matrix (binary matrix specifying if 2 vertices are connected or not which can be then used as input to the creation of the graph) or a distance matrix. We will chose the former option which provides a CSR sparse matrix. The non-zero elements in this matrix are edges in the graph.</p><p>I will use the <a href="https://igraph.org/redirect.html">igraph </a>library to model the graph which is integrated with a <a href="https://louvain-igraph.readthedocs.io/en/latest/reference.html#rbconfigurationvertexpartition">louvain library</a> providing multiple algorithms for optimizing the modularity.</p><pre><strong>def</strong> cluster_by_connectivity(data, neighbors = 10, resolution_parameter = 1):<br>    <em>&quot;&quot;&quot;</em><br><em>    This method partitions input data by applying the louvain algorithm</em><br><em>    on the connectivity binary matrix returned by the kneighbors graph.</em><br><em>    &quot;&quot;&quot;</em><br>    A = kneighbors_graph(data, neighbors, mode=&#39;connectivity&#39;, include_self=<strong>True</strong>)<br>    sources, targets = A.nonzero()<br>    weights = A[sources, targets]<br>    <strong>if</strong> isinstance(weights, np.matrix): <em># ravel data</em><br>        weights = weights.A1<br>    g = ig.Graph(directed=<strong>False</strong>)<br>    g.add_vertices(A.shape[0])  <em># each observation is a node</em><br>    edges = list(zip(sources, targets))<br>    g.add_edges(edges)<br>    <br>    g.es[&#39;weight&#39;] = weights<br>    weights = np.array(g.es[&quot;weight&quot;]).astype(np.float64)<br>    partition_type = louvain.RBConfigurationVertexPartition<br>    partition_kwargs = {}<br>    partition_kwargs[&quot;weights&quot;] = weights<br>    partition_kwargs[&quot;resolution_parameter&quot;] = resolution_parameter<br>    part = louvain.find_partition(g, partition_type, **partition_kwargs)<br>    groups = np.array(part.membership)<br>    <strong>return</strong> groups</pre><h4>Distance based approach</h4><p>An alternative approach is to start from a distance matrix using for instance the <a href="https://en.wikipedia.org/wiki/Euclidean_distance">euclidean distance </a>between observations. Other distance measures can be considered, for instance correlation distance or cosine similarity, depending on the nature of the input data set. The distance matrix can be then transformed into a similarity matrix whose values can be considered as edge weights in the graph.</p><pre>distanceMatrix =  euclidean_distances(data, data)</pre><p>The full implementation is:</p><pre><strong>def</strong> cluster_by_distance_matrix(distanceMatrix, resolution_parameter = 1.5):<br>    <em>&quot;&quot;&quot;</em><br><em>    This method partitions input data by applying the louvain algorithm</em><br><em>    on a given distance matrix.</em><br><em>    A similarity matrix is computed from the distance matrix and its elements</em><br><em>    will serve as edge weights.</em><br><em>    &quot;&quot;&quot;</em><br>    <em># convert distance matrix to similariy matrix</em><br>    distanceMatrix = 1- distanceMatrix/np.max(distanceMatrix)<br>    edges = np.unravel_index(np.arange(distanceMatrix.shape[0]*distanceMatrix.shape[1]), distanceMatrix.shape)<br>    edges = list(zip(*edges))<br>    weights = distanceMatrix.ravel()<br>    <br>    g = ig.Graph(directed=<strong>False</strong>)<br>    g.add_vertices(distanceMatrix.shape[0])  <em># each observation is a node</em><br>    g.add_edges(edges)<br>    <br>    g.es[&#39;weight&#39;] = weights<br>    weights = np.array(g.es[&quot;weight&quot;]).astype(np.float64)<br>    partition_type = louvain.RBConfigurationVertexPartition<br>    partition_kwargs = {}<br>    partition_kwargs[&quot;weights&quot;] = weights<br>    partition_kwargs[&quot;resolution_parameter&quot;] = resolution_parameter<br>    part = louvain.find_partition(g, partition_type, **partition_kwargs)<br>    groups = np.array(part.membership)<br>    <strong>return</strong> groups</pre><p>Both methods manage to retrieve correctly the input clusters. The second method needed an adaptation to the default <strong>resolution </strong>parameter, which is responsible for the <a href="https://neo4j.com/docs/graph-algorithms/current/algorithms/louvain/#algorithms-louvain-context">following</a>:</p><blockquote>For larger networks, the Louvain method doesn’t stop with the “intuitive” communities. Instead, there’s a second pass through the community modification and coarse-graining stages, in which several of the intuitive communities are merged together. This is a general problem with modularity optimization algorithms; they have trouble detecting small communities in large networks. It’s a virtue of the Louvain method that something close to the intuitive community structure is available as an intermediate step in the process.</blockquote><h3>References</h3><ol><li><a href="https://arxiv.org/abs/0803.0476">https://arxiv.org/abs/0803.0476</a></li><li><a href="https://www.quora.com/Is-there-a-simple-explanation-of-the-Louvain-Method-of-community-detection">https://www.quora.com/Is-there-a-simple-explanation-of-the-Louvain-Method-of-community-detection</a></li><li>https://neo4j.com/docs/graph-algorithms/current/algorithms/louvain/#algorithms-louvain-context</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9946876f3f9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An introduction to frequent pattern mining research]]></title>
            <link>https://medium.com/@ciortanmadalina/an-introduction-to-frequent-pattern-mining-research-564f239548e?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/564f239548e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[frequent-pattern]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Sat, 08 Jun 2019 18:59:42 GMT</pubDate>
            <atom:updated>2019-06-08T18:59:42.335Z</atom:updated>
            <content:encoded><![CDATA[<h4>Summary of Apriori, Eclat and FP tree algorithms</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LnTCaEnLim0vlL-aViC3Ww.jpeg" /></figure><h3>What are frequent patterns?</h3><p>Frequent patterns are collections of items which appear in a data set at an important frequency (usually greater than a predefined threshold)and can thus reveal association rules and relations between variables. Frequent pattern mining is a research area in data science applied to many domains such as recommender systems (what are the set of items usually ordered together), bioinformatics (what are the genes co-expressed in a given condition), decision making, clustering, website navigation.</p><p><strong>Terminology</strong></p><p>This section will introduce the recurrent terminology of frequent pattern mining domain.</p><p>Input data is usually stored in a <em>database </em>or as a collection of <em>transactions</em>. A transaction is a collection of items which have been observed together (e.g. the list of products ordered by a customer during a shopping session or the list of expressed genes in a condition). Each transaction is identified by a unique transaction id. Typically there are 2 types of architectures (layouts) used for storing such transactional data:</p><ul><li><em>horizontal layout</em>: 2 columns structure in which one column contains transaction ids and the second one the list of associated items, for instance:</li></ul><p>transaction1: [item1, item2, item7]</p><p>transaction2: [item1, item2, item5]</p><p>transactionk: [item1, itemk]</p><ul><li><em>vertical layout</em>: 2 columns structure in which one column contains individual item ids and the second one the associated transaction ids, for instance</li></ul><p>item1: [transaction1, transaction5]</p><p>item2: [transaction2, transaction4]</p><p>itemk: [transactionk]</p><p>A <em>frequent pattern</em> represents a set of items co-occurring in comparatively more transactions, for instance in the horizontal layout example item1 and item2 appear frequently together. This frequency is quantified using the <em>support</em> metric. <em>Itemset support</em> is the number of transactions where the itemset elements appear together divided by the total number of transactions.</p><blockquote>Support(item1) = count(item1)/count(all transactions)</blockquote><p><em>Minimum support</em> is a threshold used by the following algorithms in order to discard sets of items from the analysis which don’t appear frequently enough.</p><p>The <em>strength of the association rule</em> between 2 items, (for instance item1 and item2) or the <em>association confidence </em>represents the number of transactions containing item1 and item 2 divided by the number of transactions containing item1</p><blockquote>Confidence (item1→item2)=count(item1 &amp; item2)/count(item1)</blockquote><p>The confidence metric estimates the likelihood that a transaction containing item1 will include also item2.</p><p>The<em> lift</em> is the ratio of observing two items together to the likelihood of seeing just one of them. A lift greater than 1 means that items1 and 2 are more likely present together in transactions while values inferior to 1 apply to the cases when the two items are rarely associated.</p><blockquote>Lift(item1→item2) = (Confidence (item1→item2))/(Support (item2))</blockquote><p>There are many very well written posts on frequent pattern matching topics with well detailed examples. In this post I will give them credit and provide you with links you can follow for more in-depth details on how these algorithms work with step by step examples. I will also provide references to libraries in python if you are looking for an out of the box implementation.</p><h3>Apriori algorithm</h3><p>Apriori algorithm uses data organized by horizontal layout. It is founded on the fact that if a subset S appears k times in a database, <em>any other subset S1 which contains S will appear k times or less</em>. This implies that when deciding on a minimum support threshold(minimum frequency an item set needs to have in order to not be discarded)we can avoid calculating S1 or any other superset of S if support(s) &lt; minimum support. It can be said that all such candidates are being discarded <em>a priori</em>.</p><p>The algorithm computes the counts for all itemsets of k elements (starting with k = 1). During the next iterations the previous sets are being joined and thus we create all possible k + 1 itemsets. Only the combinations appearing at a frequency inferior to the minimum support rate are being discarded. The iterations end when no further extensions (joins) are being found.</p><p>An example of running this algorithm step by step on a dummy data set can be found <a href="https://www.singularities.com/blog/our-blog-1/post/apriori-vs-fp-growth-for-frequent-item-set-mining-11">here</a>.</p><p>Apriori algorithm produces a large number of candidates items (with possible duplicates) and performs many database scans (equal to the maximum length of frequent itemset). It is thus very expensive to run on large databases.</p><h4>Python implementation:</h4><p><a href="https://pypi.org/project/apyori/">https://pypi.org/project/apyori/</a></p><h3>Eclat algorithm</h3><p>Eclat (Equivalence Class Clustering and bottom up Lattice traversal) algorithm uses data organized by vertical layout which associates each element with the list of underlying transactions. In an iterative depth-first search way the algorithm continues by calculating for all combinations of k items (starting from k=1, it calculates all pairs of 2 items) the list of common transactions. In a nutshell, during the k step all combinations of k items are calculated by intersecting the lists of transactions associated with the k-1 itemsets. K will be incremented by 1 each time until no frequent items or no candidate items can still be found.</p><p>Eclat algorithm is generally faster than apriori and requires only one database scan which will find the support for all itemsets with 1 element. All k&gt;1 iterations rely only on previously stored data.</p><p>An example of running this algorithm step by step on a dummy data set can be found <a href="https://www.slideshare.net/deepa15/eclat-37310304">here</a>.</p><h4>Python implementation</h4><p><a href="https://github.com/andi611/Apriori-and-Eclat-Frequent-Itemset-Mining">https://github.com/andi611/Apriori-and-Eclat-Frequent-Itemset-Mining</a></p><h3>FP tree algorithm</h3><p>FP tree algorithm uses data organized by horizontal layout. It is the most computationally efficient algorithm from the 3 presented in this post. It only performs 2 database scans and keeps the data in an easily exploitable tree structure.</p><p>The first database scan sorts all items in the order of their global occurrence in the database (the equivalent of applying a counter to the unraveled data of all transactions). The second pass iterates line by line through the list of transactions and for each transaction it sorts the elements by the global order (previous step corresponding to the first database pass) and introduces them as nodes of a tree grown in depth. These nodes are introduced with a count value of 1. Continuing the iterations, for each line new nodes are being added to the tree at the point where the ordered items differ from the existing tree. If the same pattern already exists, all common nodes will increase their count value by one.</p><p>The FP tree can be pruned by removing all nodes having a count value inferior to a minimum threshold occurrence. The remaining tree can be traversed and for instance all paths from the root node to leaves correspond to clusters of frequently occurring items.</p><p>An example of running this algorithm step by step on a dummy data set can be found <a href="https://medium.com/machine-learning-researcher/association-rule-apriori-and-eclat-algorithm-4e963fa972a4">here</a>.</p><h4>Python implementations</h4><p><a href="https://pypi.org/project/pyfpgrowth/">https://pypi.org/project/pyfpgrowth/</a>, <a href="https://github.com/enaeseth/python-fp-growth">https://github.com/enaeseth/python-fp-growth</a> <a href="https://github.com/AVINASH793/FPGrowth-Algorithm">https://github.com/AVINASH793/FPGrowth-Algorithm</a></p><h3>Resources</h3><p><a href="https://www2.cs.sfu.ca/~jpei/publications/sigmod00.pdf">https://www2.cs.sfu.ca/~jpei/publications/sigmod00.pdf</a></p><p><a href="https://www.researchgate.net/publication/272864559_Comparing_the_Performance_of_Frequent_Pattern_Mining_Algorithms">https://www.researchgate.net/publication/272864559_Comparing_the_Performance_of_Frequent_Pattern_Mining_Algorithms</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=564f239548e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Subspace clustering]]></title>
            <link>https://medium.com/data-science/subspace-clustering-7b884e8fff73?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b884e8fff73</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[subspace]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Sun, 07 Apr 2019 11:09:00 GMT</pubDate>
            <atom:updated>2019-04-07T13:12:41.489Z</atom:updated>
            <content:encoded><![CDATA[<h4>Challenges in high dimensional spaces</h4><p>This post addresses the following questions:</p><ol><li>What are the challenges of working with high dimensional data?</li><li>What is subspace clustering?</li><li>How to implement a subspace clustering algorithm in python</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqM_tMZLaagA6UC1c1u1WQ.jpeg" /></figure><p>High dimensional data consists in input having from a few dozen to many thousands of features (or dimensions). This is a context typically encountered for instance in bioinformatics (all sorts of sequencing data) or in NLP where the size of the vocabulary if very high. High dimensional data is challenging because:</p><ul><li>it makes the visualization and thus understanding of the input difficult, it often requires applying a dimensionality reduction technique beforehand. It leads to the ‘curse of dimensionality’ which means that the complete enumeration of all subspaces becomes intractable with increasing dimensionality</li><li>most underlying clustering techniques depend on the results and the choice of the dimensionality reduction technique</li><li>many dimensions may be irrelevant and can mask existing clusters in noisy data</li><li>one common technique is to perform feature selection (remove irrelevant dimensions) but there are cases when identifying redundant dimensions is not easy</li></ul><h3>What is subspace clustering?</h3><p>Subspace clustering is a technique which finds clusters within different subspaces (a selection of one or more dimensions). The underlying assumption is that we can find valid clusters which are defined by only a subset of dimensions (it is not needed to have the agreement of all N features). For example, if we consider as input patient data observing the gene expression level (we can have more than 20000 features), a cluster of patients suffering from Alzheimer can be found only by looking at the expression data of a subset of 100 genes, or stated differently, the subset exists in 100D. Stated differently, <em>subspace clustering is an extension of traditional N dimensional cluster analysis which allows to simultaneously group features and observations by creating both row and column clusters</em>.</p><p>The resulting clusters may be overlapping both in the space of features and observations. Another example is shown in the figures below, taken from the <a href="https://www.kdd.org/exploration_files/parsons.pdf">paper</a>. We can notice that points from 2 clusters can be very close which can confuse many traditional clustering algorithms analyzing the entire feature space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*x8UldsjTQjyfx_m23nx_QA.png" /></figure><p>Further more, we can see that subspace clustering manages to find a subspace (dimensions <em>a </em>and <em>c</em>) where the expected clusters are easily identifiable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*Ee99wDxlommERhG4QwsOeQ.png" /></figure><h3>Types of subspace clustering</h3><p>Based on the search strategy, we can differentiate 2 types of subspace clustering, as shown in the figure below: bottom up approaches start by finding clusters in low dimensional (1 D) spaces and iteratively merging them to process higher dimensional spaces (up to N D). Top down approaches find clusters in the full set of dimensions and evaluate the subspace of each cluster. The figure below, taken from the same paper provides an overview of the most common subspace clustering algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*1JgMfAGmIYbeInlT5LCxLQ.png" /></figure><h3>Clique algorithm</h3><p>In order to better understand subspace clustering, I have implemented the <a href="https://www.cs.cornell.edu/johannes/papers/1998/sigmod1998-clique.pdf">Clique</a> algorithm in python <a href="https://github.com/ciortanmadalina/subspace_clustering/blob/master/clique_clustering.ipynb">here</a>.</p><p>In a nutshell, the algorithm functions as follows: for each dimension (feature) we split the space in <em>nBins(</em>input parameter) and for each bin we compute the histogram (number of counts). We only consider <em>dense units</em>, that is the bins with a count superior to a <em>threshold</em> given as second input parameter. A dense unit is characterized by the following:</p><ul><li>the dimension it belongs to (e.g. feature 1)</li><li>the index (or the position) of the bin (from 0 to nBins)</li><li>the observations lying in the bin</li></ul><p>In my implementation I have generated 4 random clusters in a 2D space and I have chosen 8 bins and 2 points as minimal density threshold. The figure below shows the resulting grid applied to the input space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*IEhLyuCflUA-XIwcPJqjHw.png" /><figcaption>Input space split in 8 bins per dimension</figcaption></figure><p>The intuition behind the clique algorithm is that clusters existing in a k dimensional space can also be found in k-1. We start from 1D and for each dimension we try to find the dense bins. If 2 or more dense bins are neighbors, we merge them into one bigger bin. This operation can be easily implemented by transforming all existing dense bins into a graph, where an edge is drawn if 2 dense units belong to the same dimension and the difference between their bin index is no more than 1 ( e.g a dense unit corresponding to feature 3 and bin 4 is neighbor for dense units of the same feature and bins 3 and 5). The dense units to be merged can be identified by calculating the connected components on the graph described above.</p><p>The result of this merging operation retrieves the following clusters in 1 D (one plot per cluster) for the first dimension:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*QlGgbOK9_3W1LnuJKibbOg.png" /></figure><p>and for the second dimension:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*MmMK1SlBqzdwgCg8U-v7mQ.png" /></figure><p>Next, we want to calculate <em>all valid clusters in each subspace from 2 to the number of input dimensions</em>. This operation comes down to calculating combinations of dense units in k dimensions and only keeping results having an overlap of dense continuous bins with the size greater than the initial minimal density threshold. Once we calculated the dense units for k-1 dimensions we can extend to k dimensions by computing all combinations of these last k-1 candidates.</p><p>Thus, in 2 D we are able to retrieve the clusters shown in the figure below. Note that some points (in purple) outside the 4 clusters because they belong to bins having a density inferior to the arbitrary input of 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*434TdPdpi1eEZWno4Ez4Vg.png" /></figure><p>Clique clustering has been criticized for its high sensitivity to the input parameters (the number of bins and the minimal density) which can lead to very different results. However, it is an essential algorithm in the family of bottom-up subspace clustering. There are multiple ways to optimize the clique algorithm, for instance by using a density adaptive grid as proposed in the <a href="http://www.quretec.com/u/vilo/edu/2003-04/DM_seminar_2003_II/ver1/P12/articles/goil99mafia.pdf">MAFIA</a> algorithm.</p><h3>References</h3><p><a href="https://www.cs.cornell.edu/johannes/papers/1998/sigmod1998-clique.pdf">Clique paper</a></p><p><a href="http://www.quretec.com/u/vilo/edu/2003-04/DM_seminar_2003_II/ver1/P12/articles/goil99mafia.pdf">Mafia algorithm</a></p><p><a href="https://www.kdd.org/exploration_files/parsons.pdf">Comparative study of subspace clustering methods</a></p><p><a href="https://imada.sdu.dk/~zimek/publications/VLDB08/tutorialSlides.pdf">Presentation of subspace clustering methods</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b884e8fff73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/subspace-clustering-7b884e8fff73">Subspace clustering</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gentle introduction to Echo State Networks]]></title>
            <link>https://medium.com/data-science/gentle-introduction-to-echo-state-networks-af99e5373c68?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/af99e5373c68</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reservoir]]></category>
            <category><![CDATA[pyhon]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Wed, 27 Mar 2019 08:01:32 GMT</pubDate>
            <atom:updated>2019-03-27T13:27:02.080Z</atom:updated>
            <content:encoded><![CDATA[<h4>Also an intro to reservoir computing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yC5tRSLq5Qf9gC33evSLfw.jpeg" /></figure><p>This post will address the following questions:</p><ol><li>What are Echo State Networks?</li><li>Why and when should you use an Echo State Network?</li><li>Simple implementation example in python</li></ol><p>The figure below is a simplification of the paper <a href="https://arxiv.org/pdf/1803.07870.pdf">Reservoir computing approaches for representation and classification of multivariate time series</a> but it captures well the gist of ESNs. Each component will be detailed in the following sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LWQiAfIcVKpt6R_yb2RSnw.png" /><figcaption>Echo State Networks are recurrent networks. <strong>f</strong> is a nonlinear function (such as tanh) which makes the current state dependent on the previous state and the current input</figcaption></figure><h3>What are Echo State Networks?</h3><p>Echo state network is a type of Recurrent Neural Network, part of the <em>reservoir computing</em> framework, which has the following particularities:</p><ul><li>the weights between the input -the hidden layer ( the ‘reservoir’) : <em>Win</em> and also the weights of the ‘reservoir’: <em>Wr</em> are <strong>randomly assigned</strong> and <strong>not trainable</strong></li><li>the weights of the output neurons (the ‘readout’ layer) are <strong>trainable </strong>and can be learned so that the network can reproduce specific temporal patterns</li><li>the hidden layer (or the ‘reservoir’) is very sparsely connected (typically &lt; 10% connectivity)</li><li>the reservoir architecture creates a recurrent <strong>non linear</strong> embedding (H on the image below) of the input which can be then connected to the desired output and these final weights will be trainable</li><li>it is possible to connect the embedding to a different predictive model (a trainable NN or a ridge regressor/SVM for classification problems)</li></ul><h4>Reservoir Computing</h4><p>Reservoir computing is an extension of neural networks in which the input signal is connected to a fixed (non-trainable) and random dynamical system (the reservoir), thus creating a higher dimension representation (embedding). This embedding is then connected to the desired output via trainable units.</p><p>The non-recurrent equivalent of reservoir computing is the <a href="https://en.wikipedia.org/wiki/Extreme_learning_machine">Extreme Learning Machine</a> and consists only of feed forward networks having only the readout layer trainable.</p><h4>Workflow</h4><p>For an input of shape N, T, V, where N is the number of observations, T is the number of time steps and V is the number of variables we will:</p><ul><li>choose the size of the reservoir R and other parameters governing the level of sparsity of connection, if we want to model a leakage, the ideal number of components after the dimensionality reduction, etc</li><li>generate (V, R) input weights <em>Win</em> by sampling from a random binomial distribution</li><li>generate (R, R) reservoir weights <em>Wr </em>by sampling from an uniform distribution with a given density, parameter which sets the level of sparsity</li><li>calculate the high dimensional state representation H as a non linear function (typically tanh) of the input at the current time step (N, V) multiplied by the internal weights plus the previous state multiplied by the reservoir matrix (R, R)</li><li>optionally we can run a dimensionality reduction algorithm such as PCA to D components, which brings H to (N, T, D)</li><li>create an input representation either by using for example the entire reservoir and training a regressor to map states t to t+1: one representation could be the matrix of all calculated slopes and intercepts. Another option could be to use the mean or the last value of H</li><li>connect this embedding to the desired output, either by using a NN structure which will be trainable or by using other types of predictors. The above mentioned paper suggest the use of Ridge regression</li></ul><h4>Why and when should you use Echo State Networks?</h4><ul><li>Traditional NN architectures suffer from the vanishing/exploding gradient problem and as such, the parameters in the hidden layers either don’t change that much or they lead to numeric instability and chaotic behavior. Echo state networks don’t suffer from this problem</li><li>Traditional NN architectures are computationally expensive, Echo State Networks are very fast as there is no back propagation phase on the reservoir</li><li>Traditional NN can be disrupted by <a href="http://www.scholarpedia.org/article/Bifurcation">bifurcations</a></li><li>ESN are well adapted for handling chaotic time series</li></ul><h4>Implementation</h4><p>The paper comes with a cleanly written and well documented implementation of a pipeline proposing various approaches for each step (weight initialization scheme, dimensionality reduction, input representation, readout layer). I have created more simplified version by cherry-picking steps such as PCA, reservoir representation based on ridge regression and linear regression connection to output as these options were best performing on my input. I made this version available <a href="https://github.com/ciortanmadalina/EchoStateNetwork/blob/master/EchoStateNetwork.ipynb">here</a>.</p><h3>References</h3><p><a href="https://github.com/FilippoMB/Reservoir-Computing-framework-for-multivariate-time-series-classification">https://github.com/FilippoMB/Reservoir-Computing-framework-for-multivariate-time-series-classification</a></p><p><a href="https://www.researchgate.net/post/what_is_the_realitionship_between_deep_learning_methods_and_reservoir_computing_if_any">https://www.researchgate.net/post/what_is_the_realitionship_between_deep_learning_methods_and_reservoir_computing_if_any</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af99e5373c68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gentle-introduction-to-echo-state-networks-af99e5373c68">Gentle introduction to Echo State Networks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple example of 2D density plots in python]]></title>
            <link>https://medium.com/data-science/simple-example-of-2d-density-plots-in-python-83b83b934f67?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/83b83b934f67</guid>
            <category><![CDATA[plotting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[distribution]]></category>
            <category><![CDATA[matplotlib]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Sun, 10 Mar 2019 16:24:18 GMT</pubDate>
            <atom:updated>2019-03-10T20:49:03.050Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to visualize joint distributions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sd_FJ4YSg0OiJBggsZoY3g.jpeg" /></figure><p>This post will show you how to:</p><ul><li>Use a Gaussian Kernel to estimate the PDF of 2 distributions</li><li>Use Matplotlib to represent the PDF with labelled contour lines around density plots</li><li>How to extract the contour lines</li><li>How to plot in 3D the above Gaussian kernel</li><li>How to use 2D histograms to plot the same PDF</li></ul><p>Let’s start by generating an input dataset consisting of 3 blobs:</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>import scipy.stats as st<br>from sklearn.datasets.samples_generator import make_blobs</pre><pre>n_components = 3<br>X, truth = make_blobs(n_samples=300, centers=n_components, <br>                      cluster_std = [2, 1.5, 1], <br>                      random_state=42)</pre><pre>plt.scatter(X[:, 0], X[:, 1], s=50, c = truth)<br>plt.title(f&quot;Example of a mixture of {n_components} distributions&quot;)<br>plt.xlabel(&quot;x&quot;)<br>plt.ylabel(&quot;y&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*4ddgIsTEIRdxom0Fw354wA.png" /></figure><p>For fitting the gaussian kernel, we specify a meshgrid which will use 100 points interpolation on each axis (e.g. mgrid(xmin:xmax:100j)):</p><pre># Extract x and y<br>x = X[:, 0]<br>y = X[:, 1]</pre><pre># Define the borders<br>deltaX = (max(x) - min(x))/10<br>deltaY = (max(y) - min(y))/10</pre><pre>xmin = min(x) - deltaX<br>xmax = max(x) + deltaX</pre><pre>ymin = min(y) - deltaY<br>ymax = max(y) + deltaY</pre><pre>print(xmin, xmax, ymin, ymax)</pre><pre># Create meshgrid<br>xx, yy = np.mgrid[xmin:xmax:100j, ymin:ymax:100j]</pre><p>We will fit a gaussian kernel using the scipy’s <em>gaussian_kde </em>method:</p><pre>positions = np.vstack([xx.ravel(), yy.ravel()])<br>values = np.vstack([x, y])<br>kernel = st.gaussian_kde(values)<br>f = np.reshape(kernel(positions).T, xx.shape)</pre><h3>Plotting the kernel with annotated contours</h3><pre>fig = plt.figure(figsize=(8,8))<br>ax = fig.gca()<br>ax.set_xlim(xmin, xmax)<br>ax.set_ylim(ymin, ymax)<br>cfset = ax.contourf(xx, yy, f, cmap=&#39;coolwarm&#39;)<br>ax.imshow(np.rot90(f), cmap=&#39;coolwarm&#39;, extent=[xmin, xmax, ymin, ymax])<br>cset = ax.contour(xx, yy, f, colors=&#39;k&#39;)<br>ax.clabel(cset, inline=1, fontsize=10)<br>ax.set_xlabel(&#39;X&#39;)<br>ax.set_ylabel(&#39;Y&#39;)<br>plt.title(&#39;2D Gaussian Kernel density estimation&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*pVmFBSktGXGHv3oUDX9qCA.png" /></figure><p>The matplotlib object doing the entire magic is called QuadContour set (<em>cset </em>in the code). We can programatically access the contour lines by iterating through <em>allsegs </em>object. The calculated labels are accessible from <em>labelTexts</em>.</p><pre>plt.figure(figsize=(8,8))</pre><pre>for j in range(len(cset.allsegs)):<br>    for ii, seg in enumerate(cset.allsegs[j]):<br>        plt.plot(seg[:,0], seg[:,1], &#39;.-&#39;, label=f&#39;Cluster{j}, level{ii}&#39;)</pre><pre>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*Mv3M_8pqukvTofzsk_Q_Ug.png" /></figure><h3>3D KDE plots</h3><p>We will use matplotlib’s axes3d from mplot3d. We can plot the density as a surface:</p><pre>fig = plt.figure(figsize=(13, 7))<br>ax = plt.axes(projection=&#39;3d&#39;)<br>surf = ax.plot_surface(xx, yy, f, rstride=1, cstride=1, cmap=&#39;coolwarm&#39;, edgecolor=&#39;none&#39;)<br>ax.set_xlabel(&#39;x&#39;)<br>ax.set_ylabel(&#39;y&#39;)<br>ax.set_zlabel(&#39;PDF&#39;)<br>ax.set_title(&#39;Surface plot of Gaussian 2D KDE&#39;)<br>fig.colorbar(surf, shrink=0.5, aspect=5) # add color bar indicating the PDF<br>ax.view_init(60, 35)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*s1Uk0_3NY1DXrRWcb17QMQ.png" /></figure><p>Or as a wireframe:</p><pre>fig = plt.figure(figsize=(13, 7))<br>ax = plt.axes(projection=&#39;3d&#39;)<br>w = ax.plot_wireframe(xx, yy, f)<br>ax.set_xlabel(&#39;x&#39;)<br>ax.set_ylabel(&#39;y&#39;)<br>ax.set_zlabel(&#39;PDF&#39;)<br>ax.set_title(&#39;Wireframe plot of Gaussian 2D KDE&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*829fq7tMJU1oxs_VLSkUZA.png" /></figure><p><strong>Representation using 2D histograms</strong></p><p>Another way to present the same information is by using 2D histograms. Setting the parameter <em>normed </em>to False returns actual frequencies while a True returns the PDF.</p><pre>h =plt.hist2d(x, y)<br>plt.colorbar(h[3])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*Q0tat3g2O-EXFEBjpPzL5g.png" /></figure><p>The entire code is available on <a href="https://github.com/ciortanmadalina/modality_tests/blob/master/2Dkde.ipynb">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83b83b934f67" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/simple-example-of-2d-density-plots-in-python-83b83b934f67">Simple example of 2D density plots in python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of data distributions]]></title>
            <link>https://medium.com/@ciortanmadalina/overview-of-data-distributions-87d95a5cbf0a?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/87d95a5cbf0a</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[probability-distributions]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Sun, 10 Mar 2019 11:35:11 GMT</pubDate>
            <atom:updated>2019-03-10T11:35:11.085Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to choose the right distribution to model your data</h4><p>There are over 20 different types of data distributions (applied to the continuous or the discrete space) commonly used in data science to model various types of phenomena. They also have many interconnections which allow us to group them in family of distributions. A great <a href="https://www.johndcook.com/blog/distribution_chart/">blog post</a> proposes the following visualization, where the continuous lines represent an exact relationship (special case, transformation or sum) and dashed line indicates a limit relationship. The same post provides a detailed explanation of these relationships and <a href="http://www.math.wm.edu/~leemis/2008amstat.pdf">this paper</a> provides a thorough analysis of the interactions between distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NhLlwFMzN0yWSvhipqMgfw.jpeg" /></figure><p>The following section provides information about each type of distribution depicting what phenomena it typically models, some example scenarios illustrating when it makes sense to choose the distribution, the probability distribution/mass function and its typical shape in a visualization.</p><p>Probability density function is a<strong> continuous approximation</strong> in terms of integrals of the density of a distribution or a smooth version of histograms. Cumulative distribution function can be expressed as F(x)= P(X ≤x), indicating the probability of X taking on a less than or equal value to x. PMF functions apply to the <strong>discrete domain</strong> and give the probability that a discrete random variable is exactly equal to some value.</p><h3>Discrete distributions</h3><p><strong>Bernoulli distribution</strong> is a discrete distribution consisting of only one trial with 2 outcomes (success/failure). It constitutes the basis for defining other more complex distributions,which analyze more than one trial, such as the next 3 distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nf510NwMhfKhjTxC38yWHA.png" /></figure><p><strong>Binomial distribution</strong> computes the probability of k successes within n trials. Like the Bernoulli distribution, trials are independent and have 2 outcomes. Examples of using this distribution are for estimating the number of heads given a series of n coin tosses or how many winning lottery tickets can we expect given a total number of tickets bought. This distribution has 2 parameters, <em>n </em>the total number of trials and <em>p </em>the probability of success:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*oCZUoAVYp1GFUww3VG8Zpg.png" /></figure><p><strong>Geometric distribution</strong> computes the number of trials before the first success occurs. The number of trials is not fixed, the probability of success is the same throughout the independent trials and the experiment continues until the first success. This distribution has one parameter, the probability of success <em>p</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*FHNknHTX3eYvvsMEyqrzZA.png" /></figure><p><strong>Hypergeometric distribution </strong>measures the number of successes in n trial (similar to binomial) but it doesn’t rely on an assumption of independence between trials. Thus, the trials are performed without replacement and each trial changes the probability of success. Examples are the probability of drawing a certain combination of cards from a deck without replacement or selecting defective light bulbs from a crate having both defective and working light bulbs. This distribution depends on the number of items in the population (N), the number of trials sampled (n) and the number of items in the population having the successful trait Nx. <em>x </em>represents the number of successful trials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*JmnJo-rEEouyeUyWNmEOPw.png" /></figure><p><strong>Negative binomial </strong>computes the number of trials until reaching r successful events. It is a superdistribution of the geometric distribution and can be used to model situations like the number of sales calls to be performed in order to close r deals. The parameters used by this distribution are the probability of success <em>p </em>and the number of required successes <em>r</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*0ddVk5UAc2Tpw2UiVxk1kA.png" /></figure><p><strong>Discrete uniform</strong> is the distribution of n different but equally likely outcomes. It is the counterpart of uniform distribution in the discrete space. It takes as input only N, the number of distinct outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*BqHSs8bG47d136MqoYlL9w.png" /></figure><p><strong>Poisson distribution </strong>approximates the number of times an event occurs in a given interval, knowing that the occurrences are independent, there is no upper limit to the the number of events and the average number of occurrences must remain the same if we extended the analysis form one interval to another. This distribution has one parameter, <em>lambda</em> being the average number of events per interval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*-8eqxbJbuN3QbxItN9EMdQ.png" /></figure><h3>Continuous distributions</h3><h4>Beta distribution</h4><p>Beta distribution is commonly used to represent variability over a fixed range., for instance to model the behavior of random variables limited to intervals of finite length. It is also a suitable choice to model percentage or proportions. In Bayesian inference it is used to model the conjugate prior for Bernoulli, Binomial, Negative binomial and geometric distributions. More simply put, the beta distribution is a good proposal for the priors (the initial knowledge of success) for different applications from the Bernoulli family such as the number of heads on coin tossing trials or any other dual outcome event. It takes 2 parameters, alpha and beta and the uncertain variable is a random value between 0 and a positive value. Different combinations of alpha and beta lead to the following shapes of the distribution:</p><ul><li>alpha == beta =&gt; symmetrical distribution</li><li>if (alpha == 1 and beta &gt; 1) or (beta== 1 and alpha&gt; 1) =&gt; J shaped distribution</li><li>alpha &lt; beta =&gt; positive skew</li><li>alpha &gt;beta =&gt; negative skew</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*WhlyL1-jGk5TU28O8_vL9g.png" /></figure><h4>Dirichlet distribution</h4><p>Dirichlet distribution is a multivariate generalization of beta distributions, for which reason it is also known as Multivariate Beta distributions. It is used as prior distribution in bayesian statistics, where it is the conjugate prior of categorical and multinomial distribution. It is parametrized by a vector alpha of positive reals and it samples over a <em>probability simplex</em>. A probability simplex is a set of k numbers adding up to 1 and which correspond to the probabilities of k classes. A k-dimensional Dirichlet distribution has k parameters.</p><h4>Cauchy distribution</h4><p>Cauchy distribution is employed in mechanical and electrical theory, physical anthropology and measurement and calibration problems. In physics it describes distribution of the energy of an unstable state in quantum mechanics under the name Lorentzian distribution. Another application is to model the points of impact of a fixed straight line of particles emitted from a point source or in robustness studies. The Cauchy distribution is known to be a pathological distribution as both its mean and variance are undefined. It takes two parameters. In Bayesian statistics, Cauchy distribution can be used to model the priors for the regression coefficients in logistic regression.</p><p>The distribution takes two parameters, the mode <em>m</em>(corresponding to the peak) and the scale gamma (half width at half maximum of the distribution). Cauchy distribution is the Student’s T distribution with 1 degree of freedom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*_8_AHmtNOhP9yHdsWsgiEw.png" /></figure><h4>Chi-Square Distribution</h4><p>This distribution is predominantly used in hypothesis testing, in the construction of confidence intervals, in the evaluation of the <a href="http://hamelg.blogspot.com/2015/11/python-for-data-analysis-part-25-chi.html">goodness of fit </a>of an observed distribution to a theoretical one. Chi-Square (with one degree of freedom) variable is the square of a standard normal variable and Chi-Square distribution has additive property (Sum of two independent Chi-Square distributions is also a Chi-Square variable). The sum of k independent normal distributions is distributed as a chi-square with k degrees of freedom. The chi-square distribution can also be modeled using a <strong>gamma distribution</strong> with the shape parameter as k/2 and scale as 2S².</p><p>The chi-squared distribution has one parameter: <em>k</em>, the number of degrees of freedom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*GddVH8UqEU6zFAi5RFjJSQ.png" /></figure><h4>Exponential distribution</h4><p>This distribution describes the amount of time between events occurring at random moments. It is considered that time has no effect on future outcomes (the future lifetime of an object has the same distribution, regardless of the time it existed) which makes the exponential “memoryless”. It can be used to model situations such as: how long do we have to wait at a crossroad until we see a car running on the red light or how long it will take until someone receives the next phone call? how long will a product function before breaking down?</p><p>Exponential distribution is related to Poisson which doesn’t describe the time lapsed but the number of occurrences of an event in a given time frame. The exponential distribution is parametrised only by lambda, the success rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*WQadkW_17iA_Yl1go9pXCw.png" /></figure><h4>Extreme value distribution (Gumbel)</h4><p>The <strong>Gumbel distribution </strong>models the distribution of the maximum (or the minimum) of a number of samples of various distributions. Examples of this distribution are the breaking strengths of materials, the maximum load for an aircraft, tolerance studies, the maximum level of a river or of an earthquake in a given year. This distribution has 2 parameters, the mode m corresponding th the most likely point (or the PDF’s highest peak) and a scale parameter, beta which is &gt;0 and governs the variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*YVBnSZfBhp4NwBoF3zYqaQ.png" /></figure><h4>F Distribution</h4><p>This distribution is used to test the <a href="http://blog.minitab.com/blog/adventures-in-statistics-2/understanding-analysis-of-variance-anova-and-the-f-test">statistical difference</a> between two variances as part of one way <a href="https://www.dummies.com/education/science/biology/the-basic-idea-of-an-analysis-of-variance-anova/">ANOVA </a>analysis or the <a href="http://blog.minitab.com/blog/adventures-in-statistics-2/what-is-the-f-test-of-overall-significance-in-regression-analysis">overall significance of a regression model </a>with f-tests. Frequently it is the null distribution (the probability distribution when the null hypothesis is true) of test statistics. It takes as parameters <em>n </em>degrees of freedom for the numerator and <em>m </em>degrees of freedom for the denominator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*6KqAzKonn8O040iVwzoaPQ.png" /></figure><h4>Gamma distribution</h4><p>Gamma distribution is used to measure the time between the occurrence of events when the event process is not completely random. The number of events in the studied time frame is not limited to a fixed number. The events are independent. Gamma distribution is related to <em>lognormal, exponential Pascal, Poisson and chi-square distributions</em>. It can be used to model pollutant concentrations and precipitation quantities in meteorological processes. It depends on 2 parameters, alpha or the shape parameter and beta, the scale parameter.</p><p>A special case arises when alpha is a positive integer, in which case the distribution (also known as the Erlang distribution) can be used to predict waiting times in queuing systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*FiZOQUNvLprQiOQt6w8-0w.png" /></figure><h4>Logistic distribution</h4><p>This distribution is used to describe population growth over time or chemical reactions. This distribution is symmetrical and takes 2 parameters: the mean or the average value and the scale, controlling the variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*p0fjbJfKf47wwqMlsOh2Ww.png" /></figure><p><strong>Lognormal distribution</strong></p><p>Lognormal distribution is a good candidate for modelling<em> positively skewed </em>values which<em> are ≥0</em>. For instance, normal distribution cannot be used to model<a href="https://romanorac.github.io/cryptocurrency/analysis/2017/12/29/cryptocurrency-analysis-with-python-part3.html"> stock prices </a>because it has a negative side and stock prices cannot fall below zero so lognormal distribution is a good candidate. Thus, if a random variable <em>X</em> is log-normally distributed, then <em>Y</em> = ln(<em>X</em>) has a normal distribution. Likewise, if <em>Y</em> has a normal distribution, then the <a href="https://en.wikipedia.org/wiki/Exponential_function">exponential function</a> of <em>Y</em>, <em>X</em> = exp(<em>Y</em>), has a log-normal distribution. Lognormal distribution is described by 2 parameters, the mean and the standard deviation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*GqlU0rNOCGmbf2-aYo2mIQ.png" /></figure><h4>Pareto distribution</h4><p>Pareto distribution is a power law probability distribution used to model empiric phenomena such as the distribution of wealth, the stock price fluctuations, the occurrence of natural resources. Pareto distribution has been vulgarized under the name of Pareto principle (or the “80–20 rule”, the Matthew principle) stating that, for example, 80% of the wealth of a society is held by 20% of its population. However, the Pareto distribution only produces this result for a particular power value of the input parameter alpha (<em>α</em> = log45 ≈ 1.16). In terms of parameters, this distribution depends on location (the lower bound for the variable) and the shape controlling the variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*3Td1fy5I6IOb6_zP6k4NSA.png" /></figure><h4>Student’s t Distribution</h4><p>This distribution is typically used to test the statistical significance of the difference between two sample means or to estimated the mean of a normally distributed population, both for small sample sizes. The shape of this distribution resembles the bell shape of a leptokurtic gaussian.The only parameter is the degree of freedom r.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*wxVF7FDTI07ZtWlHxdU8sQ.png" /></figure><h4>Triangular distribution</h4><p>This distribution describes situation where the minimum, maximum and most likely values of an event are know. Both the minimum and the maximum values are fixed and the most likely value falls between them, forming a triangular-shaped distribution. For instance, this distribution can describe the sales of a product when we know the minimum, maximum and most likely estimations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*YDDRRydsTXoEZV9XO63Djg.png" /></figure><h4>Weibull Distribution</h4><p>Weibull distribution is heavily employed in fatigue tests, e.g. to describe failure time in reliability studies or breaking strengths of materials in quality control tests. It can also be used to model physical quantities, such as wind speed. It depends on 3 input parameters: the location L, the shape alpha and the scale beta. When the shape parameter = 1, it becomes the exponential distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*SheGqJa_G-pG0KyFs0ElfA.png" /></figure><h4>Normal distribution</h4><p>Probably the most studied distribution, the normal is used to describe natural phenomena, such as the distribution of heights and IQ scores. This distribution take as input 2 parameters, the mean and the standard deviation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*yoV9vvysMH1HpRLru1E88Q.png" /></figure><h3>Conjugate distributions</h3><p>In the context of bayesian analysis, if the <strong>posterior </strong>distribution p(teta|x) and the <strong>prior </strong>p(teta) are part of the same probability family, they are called <strong>conjugate distributions. </strong>Furthermore, the prior called the <strong>conjugate prior</strong> for the likelihood function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*2wD8ldd1jKpZ5lTS_ruD8g.png" /></figure><p>Different choices of prior can make the integral more or less difficult to calculate. If the likelihood p(x|teta) has the same algebraic form as the prior, we can obtain a closed form expression for the posterior. This <a href="https://www.johndcook.com/blog/conjugate_prior_diagram/#geometric">blog </a>provides a good overview of the relationships between the choice of prior distribution (beta/gamma) and the sampling posterior distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*hbVgUuwmhNaYA7FXjaLHUA.png" /></figure><h3>Resources</h3><p><a href="https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781119197096.app03">https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781119197096.app03</a></p><p><a href="https://probabilityandstats.wordpress.com/tag/poisson-gamma-mixture/">https://probabilityandstats.wordpress.com/tag/poisson-gamma-mixture/</a></p><p><a href="https://www.johndcook.com/blog/conjugate_prior_diagram/">https://www.johndcook.com/blog/conjugate_prior_diagram/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87d95a5cbf0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Expectation maximization]]></title>
            <link>https://medium.com/@ciortanmadalina/expectation-maximization-57a85ad3434e?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/57a85ad3434e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[maximum-likelihood]]></category>
            <category><![CDATA[gaussian-process]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Mon, 18 Feb 2019 14:20:26 GMT</pubDate>
            <atom:updated>2019-02-18T14:20:26.832Z</atom:updated>
            <content:encoded><![CDATA[<h4>Finding the Maximum Likelihood Estimate of a model depending on unobserved latent variables</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d6e8QQ7q8QQAft4zMU26WQ.jpeg" /></figure><p>This article presents a basic python implementation of the expectation maximization algorithm applied to Gaussian distributions. The entire code has been made available as a notebook on <a href="https://github.com/ciortanmadalina/expectation_maximization/blob/master/gmm.ipynb">Github</a>.</p><p>There are datasets consisting of a mixture of distributions (for simplicity, we will consider Gaussians, but the same heuristic can be applied to other types of distributions):</p><p><em>dataset = N(mean1, sd1) + N(mean2, sd2) + … + N(mean_n, sd_n)</em></p><p>where N represents a normal distribution described by a mean and a standard deviation values.</p><p>In the example below we generated a dataset containing samples from 3 distributions having random means and the following standard deviations [0.5, 0.7, 1]. We would like to find the parameters of a statistical model which <strong>maximize the likelihood</strong> that the process described by this model produced the observed data.</p><pre>n_components = 3<br>X, truth = make_blobs(n_samples=300, centers=n_components, <br>                      cluster_std = [0.5, 0.7, 1], <br>                      random_state=42)</pre><pre>plt.scatter(X[:, 0], X[:, 1], s=50, c = truth)<br>plt.title(f&quot;Example of a mixture of {n_components} distributions&quot;)<br>plt.xlabel(&quot;Feature 1&quot;)<br>plt.ylabel(&quot;Feature 2&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*ASftA2fl0J34rzi5rAcxYw.png" /></figure><h4>What is the Maximum Likelihood Estimation (MLE)?</h4><p>MLE is a statistical method for estimating the parameters of a model for a given data. For the sake of clarity, let’s take the most simple example. We observe data coming from <em>only one normal distribution</em> and we would like to determine the parameters (in this case the mean and the standard deviation) which best approximate the data. As we select different values for the mean and for the standard deviation, we get very different results for the probability density function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oseM4js9FIcRNSEGzcX2Mw.png" /></figure><p>Thus, we want to calculate and maximize the probability of observing all data, or in statistical term the joint probability distribution of all observations. Then, we make the assumption that each data point is generated independently of the others, which allows us then to calculate the total probability as the product of the marginal probabilities (the probability of observing each point individually). Our desired optimization consists in finding the parameters (mean, sd) which lead to maximum total probability with respect to the observed data.</p><p>This comes down to calculating the points where the first derivative is 0. In practice, it is more convenient to work with the natural logarithm of the likelihood function, as this transforms the product into a sum (log-likelihood) and it is easier to differentiate. As the logarithm is a monotonically increasing function, it ensures that the maximum values occur at the same point as in the probability function. In practical terms, it can happen that the derivative of the log-likelihood is too hard or impossible to differentiate by hand, in which case we can still use iterative methods like expectation maximization to find parameter estimates.</p><p>Expectation maximization consists in an iterative process looping between 2 steps, called <strong>step e</strong> (expectation)and<strong> step m</strong> (maximization). It needs as input the <em>number of components </em>(clusters) expected to be found in the data and also the type of distribution we will use as approximation (in this case normal distributions). In a nutshell, the algorithm can be reduced to the following pseudo-code:</p><pre><strong>expectation_maximization</strong>(data, n_components):<br>params = init(data)<br>while( !convergence)<br>    cluster_probabilities &lt;- <strong>e_step</strong>(data, n_components, params)<br>    params = <strong>m_step</strong>(cluster_probabilities, data)</pre><h4>Initialization</h4><p>In addition to the n_components, this algorithm maintains a list of parameters which are updated at each iteration. These parameters are:</p><ul><li>priors: a list of n_components values, representing the probability of a point belonging to the corresponding cluster. Initially we consider that all clusters have equal probability, thus the priors are list of n_components with values 1/n_components</li><li>means: the mean value for each of the n_components clusters. Initially we can select a n_components random points as mean values</li><li>covariances: n_components values of covariance matrix corresponding to each cluster. Initially this matrix is assigned to be the Identity matrix.</li></ul><h4>Expectation step</h4><p>If we consider that the input data X has shape (n_samples, n_features), this step will calculate a cluster probabilities matrix K of shape (n_samples, n_components) providing for each observation the probability (PDF) it originates from each cluster. In other points, it answers the question: <strong>for each point, what is the probability that each of the n_components gaussians generated it?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z6OUFxZTy_4eVY55Ll3V1g.png" /></figure><p>The expectation step uses a probability density function specific to the distribution input type:</p><pre>def getPDF(X, params):<br>    &quot;&quot;&quot;<br>    Probability density function of the input distribution. This is used in the exploration step in order to<br>    calculate for each observation the probability of being observed as part of cluster k.<br>    &quot;&quot;&quot;<br>    if params[&#39;distribution&#39;] == &quot;normal&quot;:<br>        k = params[&#39;current_cluster&#39;]<br>        return mvn(params[&quot;means&quot;][k], params[&quot;covariances&quot;][k]).pdf(X)</pre><h4>Maximization step</h4><p>Given the cluster probabilities matrix calculated in the previous step, we update the mean, prior and covariance parameters to reflect these probabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_F_At6y3IENc4u0eodtjeg.png" /></figure><p>The exit condition (convergence) applies when the difference between the cluster probabilities matrix between iterations becomes negligible (smaller than an arbitrary delta). In order to avoid the algorithm getting stuck if convergence doesn’t occur in a reasonable amount of time, we can also define a maximum number of iterations after which we exit.</p><pre>def convergence(K, K_prev, delta):<br>    &quot;&quot;&quot;<br>    Check if the algorithm converged by estimating the difference between 2 consecutive solutions. If the matrix norm is smaller than a given delta, we take this as a proof of convergence<br>    &quot;&quot;&quot;<br>    if K_prev == None:<br>        return False<br>    return np.linalg.norm(K - K_prev) &lt; delta</pre><h4>Future work</h4><p>The next planned work consists in enriching this model with other types of distributions.</p><p>Another future direction is to build regression functionalities on top of this model which will provide posterior probability estimates for given input.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57a85ad3434e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mixed plot scales and custom violinboxplot]]></title>
            <link>https://medium.com/data-science/mixed-plot-scales-and-custom-violinboxplot-68cc8c5d6ead?source=rss-d576810f0222------2</link>
            <guid isPermaLink="false">https://medium.com/p/68cc8c5d6ead</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[matplotlib]]></category>
            <dc:creator><![CDATA[Madalina Ciortan]]></dc:creator>
            <pubDate>Sat, 05 Jan 2019 18:54:20 GMT</pubDate>
            <atom:updated>2019-01-07T20:18:07.942Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hybrid plot scales and custom violinboxplot implementation</h3><h4>Hybrid scale plots and a custom violinboxplot</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FuUGVTsLFZ-eKARHbOb66Q.jpeg" /></figure><p>In a nutshell, this post addresses the following 2 questions:</p><h3>1. How can we create comprehensive visulizations of data distributions having far outliers?</h3><p>Typically we use both linear and log scales (for capturing outliers) but here we will investigate the possibility of creating hybrid axes, with an arbitrary mixture of various types of scales, applied on desired intervals.</p><h3>2. How can we combine the best from boxplots, violinplots and dynamic scales following the data distribution?</h3><p>We will propose a custom implementation of a violinboxplot offering a wide range of customization parameters which govern for instance the rendering of outliers, custom annotations for modes and counts, split axis between linear/log based on an arbitrary percentile. It handles both arrays of data and dataframes grouped by a list of columns.</p><p>All examples in this blog have been implemented in python/matplotlib and are available on <a href="https://github.com/ciortanmadalina/modality_tests/blob/master/violinboxplot_hybrid_axes.ipynb">Github</a>.</p><h3>Hybrid matplolib axis scales</h3><p>Let’s start by choosing a function (sin) which will generate the data-set we will use to demonstrate the hybrid scales concept.</p><pre>x = np.arange(-80,80, 0.1)<br>y = np.sin(x)<br>plt.title(&#39;Linear scale plot of a sinusoid&#39;)<br>plt.xlabel(&#39;x&#39;)<br>plt.ylabel(&#39;y&#39;)<br>plt.plot(x, y);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*6U8srpNwxBDBlh8guaqUHA.png" /></figure><p>If we treat this data-set as a black-box, a data scientist may want, for any number of reasons, to have a dynamic resolution of the plot by using different scales on different intervals with a minimal effort. For instance, he may want to visualize:</p><ul><li>1&lt;= y &lt;= 0.5 using a linear scale</li><li>0.1 &lt;= y &lt;= 0.5 using a log scale</li><li>-1&lt;= y &lt;= 0.1 using a linear scale</li></ul><p>The first naive solution is to create 3 different plots with the chosen axis scales on chosen intervals.<br>In this post we will investigate the capabilities of matplotlib to make appear the original plot under different scales, thus providing a unified visualization.</p><p>There are 2 approaches we will present in this post:</p><ol><li>Using an axis divider</li><li>Using Grid spec</li></ol><h3>Axis divider</h3><pre><strong>from</strong> <strong>mpl_toolkits.axes_grid1</strong> <strong>import</strong> make_axes_locatable</pre><p>Matplotlib’s function <strong>make_axes_locatable</strong> allow us to append a new axis to a given axis. In the example below, a log axis is created from the original linear axis.<br>By setting arbitrary y limits we control what part of the plot is being rendered and we can create the impression of plot continuity.<br><em>Sharedx</em> parameter allows sharing the same x axis and prevents the x tick labels from being re-rendered.</p><pre>plt.title(&#39;Split plot in 2 parts: linear: [0.5, 1] and log: [0.01, 0.5]&#39;)<br>linearAxis = plt.gca()<br>linearAxis.plot(x, y)<br>linearAxis.set_ylim((0.5, 1))</pre><pre>divider = make_axes_locatable(linearAxis)<br>logAxis = divider.append_axes(&quot;bottom&quot;, size=1, pad=0.02, sharex=linearAxis)<br>logAxis.plot(x, y)<br>logAxis.set_yscale(&#39;log&#39;)<br>logAxis.set_ylim((0.01, 0.5));</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*ZLYwOUQr5fwCxMSSOVl1GQ.png" /></figure><p>We can use append axes on a given input axis in 4 potential location (top/ bottom/ up/ down). The code below illustrates chaining 2 axes, on top and on bottom.</p><pre>logAxis = plt.gca()<br>logAxis.plot(x, y)<br>logAxis.set_yscale(&#39;log&#39;)<br>logAxis.set_ylim((0.01, 0.5))</pre><pre>divider = make_axes_locatable(logAxis)<br>linearAxis = divider.append_axes(&quot;top&quot;, size=1, pad=0.02, sharex=logAxis)<br>linearAxis.plot(x, y)<br>linearAxis.set_ylim((0.5, 1))<br>linearAxis.set_xscale(&#39;linear&#39;)<br>linearAxis.set_title(&#39;Plot split in 3 scales: linear: [0.5, 1], log: [0.01, 0.5], linear: [-1, 0.01]&#39;);</pre><pre>linearAxis1 = divider.append_axes(&quot;bottom&quot;, size=1, pad=0.02, sharex=logAxis)<br>linearAxis1.plot(x, y)<br>linearAxis1.set_yscale(&#39;linear&#39;)<br>linearAxis1.set_ylim((-1, 0.01));</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*CqO8uqdLa_7Qz0H7Nps1PQ.png" /><figcaption>GridSpec implementation</figcaption></figure><p>Another option is to use matplotlib’s GridSpec which provides more flexibility in terms of sizing the components and usage.<br>We can define upfront the number of suplots, their relative sizes (height_ratios), the distance between subplots (hspace).<br>Once the independent axis have been created, we can set the scales and the desired limits.</p><pre><strong>import</strong> <strong>matplotlib.gridspec</strong> <strong>as</strong> <strong>grd</strong><br>gs = grd.GridSpec(3, 1, wspace=0.01, hspace=0.05, height_ratios=[0.33, 0.33, 0.33])</pre><pre>ax1 = plt.subplot(gs[0])<br>ax2 = plt.subplot(gs[1])<br>ax3 = plt.subplot(gs[2])<br>ax1.set_xticks([])<br>ax2.set_xticks([])</pre><pre>ax1.plot(x, y)<br>ax1.set_yscale(&#39;linear&#39;)<br>ax1.set_ylim((0.5, 1))<br></pre><pre>ax2.plot(x, y)<br>ax2.set_yscale(&#39;log&#39;)<br>ax2.set_ylim((0.01, 0.5))<br></pre><pre>ax3.plot(x, y)<br>ax3.set_yscale(&#39;linear&#39;)<br>ax3.set_ylim((-1, 0.01));</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Yf9Dvhf6Gr5n19wjsY-Wbg.png" /></figure><h3>Custom violinbloxplot</h3><p>Let’s start by generating a few data distributions reflecting multiple scenarios:</p><ul><li>unimodal data reflecting a gaussian distribution</li><li>a combination of gaussian data with outliers</li><li>a dataset with multiple (7 such distributions) illustrating a comparative visualization of input distributions</li><li>a dataframe to be grouped by one or multiple comumns to illustrate the compartive data distribution</li></ul><pre>data1 = [np.round(np.random.normal(10, 0.4, 50), 2)]<br>data1SharpEnd = [[e <strong>for</strong> e <strong>in</strong> data1[0] <strong>if</strong> e &gt; 9.9]]<br>data1Spread = [<br>    np.concatenate([<br>        np.round(np.random.normal(10, 0.2, 1000), 2), <br>        np.round(np.random.normal(80, 0.3, 5), 2)<br>        ]) <br>    ]<br></pre><pre>data2 = [<br>    np.concatenate([<br>        np.round(np.random.normal(10, std/10, 1000), 2), <br>        np.round(np.random.normal(80, std, np.random.randint(0, 24) * std), 2) ]) <br>            <strong>for</strong> std <strong>in</strong> range(1, 7)<br>        ]<br>labels7 = [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;, &#39;E&#39;, &#39;F&#39;, &#39;G&#39;]</pre><p>Based on one of the existing datasets, we can define a dataframe:</p><pre>df = pd.DataFrame()<br>df[&#39;values&#39;] = data1Spread[0]<br>df[&#39;col1&#39;] = np.random.choice([&#39;A&#39;, &#39;B&#39;], df.shape[0])<br>df[&#39;col2&#39;] = np.random.choice([&#39;C&#39;, &#39;D&#39;], df.shape[0])</pre><p>In order to better understand the underlying data distribution, let’s create a plotting function which leverages both boxplots and violinplots:</p><pre><strong>def</strong> plotDistributions(inputData, title):<br>    <em>&quot;&quot;&quot;</em><br><em>    This method plots inputData with:</em><br><em>    - matplotlib boxplot</em><br><em>    - matplotlib violinplot</em><br><em>    - seaborn violinplot</em><br><em>    &quot;&quot;&quot;</em><br>    globalMax = np.max(np.concatenate(inputData))<br>    globalMin = np.min(np.concatenate(inputData))<br>    <br>    plt.figure(figsize =(14, 4))<br>    plt.suptitle(title)</pre><pre>    plt.subplot(121)<br>    plt.grid()<br>    plt.title(&#39;Matplotlib boxplot&#39;)<br>    plt.boxplot(inputData, vert= <strong>False</strong>);<br>    plt.axvline(x = globalMax, c =&#39;red&#39;, label = &#39;Global max&#39;, alpha = 0.5)<br>    plt.axvline(x = globalMin, c =&#39;red&#39;, label = &#39;Global min&#39;, alpha = 0.5)<br>    plt.legend()<br></pre><pre>    plt.subplot(122)<br>    plt.grid()<br>    plt.title(&#39;Matplotlib violinplot&#39;)<br>    plt.violinplot(inputData, vert= <strong>False</strong>, showmeans=<strong>False</strong>, showmedians=<strong>True</strong>, showextrema=<strong>True</strong>);<br>    plt.axvline(x = globalMax, c =&#39;red&#39;, label = &#39;Global max&#39;, alpha = 0.5)<br>    plt.axvline(x = globalMin, c =&#39;red&#39;, label = &#39;Global min&#39;, alpha = 0.5)<br>    plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*B5q5J66Sy2chRyMLznyqCg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fgXyhCmMd6C0Le_cjzTdmg.png" /></figure><h3>We can visualize the dataframe using seaborn</h3><pre>sns.violinplot(x = &#39;values&#39;, y=&#39;col1&#39;, data = df)<br>plt.figure()<br>sns.violinplot(x = &#39;values&#39;, y=&#39;col2&#39;, data = df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*P9_egnBORIlJjpwn6WamSg.png" /></figure><p>However, seaborn expects to indicate as y only one column which will be used in a group by to aggregate the results. If we want to aggregate based on a combination of multiple features, we have to do it prior to calling the plotting function.</p><p>What some drawbacks we can identify in the above plots?</p><ul><li>it would be nice to have the combined resolution of boxplots and violin plots in one graph. Seaborn offers through the inner parameter a way to incorporate a boxplot but its customisation possibilities are limited.</li><li>as shown in the second graphic, if we are dealing with a distribution with far outliers, the overall visualization looses the details at the extremes. What if we could used the examples discussed in the first section to create a customized unified view with arbitrary scales on target intervals?</li></ul><p>Some other points to consider are:</p><ul><li>how can we enrich plots with custom annotations indicating for instance, for each dataset the number of points and other arbitrary measures, such as the mode?</li><li>could be provide a hyperparameter which could remove from the visualisation alltogether the points we consider outliers?</li></ul><p>If we start with this last point, we can come up with a method that removes all points father away than a given number of standard deviations (by default 3).</p><p>What some drawbacks we can identify in the above plots?</p><ul><li>it would be nice to have the combined resolution of boxplots and violin plots in one graph. Seaborn offers through the inner parameter a way to incorporate a boxplot but its customisation possibilities are limited.</li><li>as shown in the second graphic, if we are dealing with a distribution with far outliers, the overall visualization looses the details at the extremes. What if we could used the examples discussed in the first section to create a customized unified view with arbitrary scales on target intervals?</li></ul><p>Some other points to consider are:</p><ul><li>how can we enrich plots with custom annotations indicating for instance, for each dataset the number of points and other arbitrary measures, such as the mode?</li><li>could be provide a hyperparameter which could remove from the visualisation alltogether the points we consider outliers?</li></ul><p>If we start with this last point, we can come up with a method that removes all points father away than a given number of standard deviations (by default 3).</p><pre><strong>def</strong> removeOutliers(data, thresholdStd = 3):<br>    <em>&quot;&quot;&quot;</em><br><em>    This method returns all values which are farther away</em><br><em>    than thresholdStd standard deviationa</em><br><em>    &quot;&quot;&quot;</em><br>    noOutliers=[]<br>    mean = np.mean(data)<br>    std =np.std(data)<br>    <strong>if</strong> std == 0:<br>        <strong>return</strong> data</pre><pre>    <strong>for</strong> y <strong>in</strong> data:<br>        z_score= (y - mean)/std <br>        <strong>if</strong> np.abs(z_score) &lt;= thresholdStd:<br>            noOutliers.append(y)<br>    <strong>return</strong> noOutliers</pre><h3>Implementation of violinboxplot</h3><p>Find the complete code on <a href="https://github.com/ciortanmadalina/modality_tests/blob/master/violinboxplot_hybrid_axes.ipynb">github</a>. We will only present the results on the data-sets introduced previously. This custom implementation renders:</p><ul><li>(optionally) in green the mode of each distribution</li><li>(optionally) in back at the end of each line the count of each distribution</li><li>Can render or hide outliers</li><li>Optional logpercentile specifies a percentile value (0.9) under which the rendering will be done on a linear scale, values about this percentile use logscale</li><li>The method can take as input a list of arrays or a dataframe which can be grouped by a list of columns (y)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*fS3DSkckbxtfOC1U5yVl2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*dIy7eyhVRWTgWbusHgYaxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*hUfCgtPW8RVwJ9SnPBWYZA.png" /></figure><p>Results using dataframes and aggregations on one or multiple columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*0ojiudfTDDvWLWTRHWGS1Q.png" /></figure><h3>Future work</h3><p>One future improvement is to automatically detect the modes of all rendered distribution and to estimate the PDF by using the code presented in the previous article <a href="https://github.com/ciortanmadalina/modality_tests/blob/master/kernel_density.ipynb">https://github.com/ciortanmadalina/modality_tests/blob/master/kernel_density.ipynb</a> . This can give an estimation of some of the zones of interest where for instance a log scale would be good choice.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68cc8c5d6ead" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/mixed-plot-scales-and-custom-violinboxplot-68cc8c5d6ead">Mixed plot scales and custom violinboxplot</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>