<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Zabolocki on Medium]]></title>
        <description><![CDATA[Stories by Michael Zabolocki on Medium]]></description>
        <link>https://medium.com/@mzabolocki?source=rss-d52f4e270c0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*e2Cmqua7ojRNcLxjltq3mQ.jpeg</url>
            <title>Stories by Michael Zabolocki on Medium</title>
            <link>https://medium.com/@mzabolocki?source=rss-d52f4e270c0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 17:04:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mzabolocki/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Random Forest Classifier to Predict Neural Spikes]]></title>
            <link>https://medium.com/data-science/building-a-random-forest-classifier-for-neural-spike-data-8e523f3639e1?source=rss-d52f4e270c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e523f3639e1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[random-forest]]></category>
            <dc:creator><![CDATA[Michael Zabolocki]]></dc:creator>
            <pubDate>Sun, 30 Jan 2022 13:11:12 GMT</pubDate>
            <atom:updated>2022-02-03T19:32:54.710Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A step-by-step guide to building a Random Forest classifier in Python to predict subtypes of real neural extracellular spikes.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ss_LylfOEI_oHiRaBTQw_A.jpeg" /><figcaption>‘Braintree’ from Fakurian Design on <a href="https://unsplash.com/photos/58Z17lnVS4U">Unsplashed</a></figcaption></figure><p><em>Given the heterogeneity of neurons within the human brain itself, classification tools are commonly utilised to correlate electrical activity with different cell types and/or morphologies.</em> This is a long-standing question in Neuroscience circles, and can be considerably variable between different species, pathologies, brain regions and layers. Fortunately, with the readily increasing computational power allowing improvements in machine-learning and deep-learning algorithms, Neuroscientists are provided with the tools to dive further into asking these important questions. However, as stated by <a href="https://www.cell.com/neuron/pdf/S0896-6273(21)00994-6.pdf">Juavinett et al.</a>, for the most part programming skills are underrepresented in the community and new resources to teach them are crucial to solving the complexity of the human brain.</p><p>Hence, to provide a relevant use case, in this article we will work through building a Random Forest classifier algorithm for a neural extracellular spike waveform data-set. Random Forest classifiers are widely implemented supervised learning models which serve as an effective and powerful algorithm to solving classification problems. They arguably sit near the top of the classifier hierarchy, alongside other algorithms such as: logistic regression, support vector machine, naive Bayes classifier, and decision trees.</p><p>Along the way, we will also complete multi-dimensionality reduction and clustering steps (with example code) to establish a supervised learning problem. Once completed, any level-programmer will be able to (1) identify and plot unique extracellular waveforms and (2) implement a Random Forest algorithm to classify them.</p><h3>What are decision trees?</h3><p>Back to basics! First, to understand Random Forest plots it is important to grasp decision trees (this will make sense soon). <em>In short, a decision tree is a tree-like model consisting of ‘nodes’ and ‘branches’, which serves as a tool for visually displaying decisions and their outcomes.</em></p><p>As summarised by <a href="https://medium.com/u/8418ab50405a">Angel Das</a>, <a href="https://medium.com/u/1fd70d25ff14">z_ai</a>, and shown below in the figure, nodes can be classified into the following:</p><ul><li><strong>Root node:</strong> the starting node. In a decision tree it typically evaluates the variable that best splits the data.</li><li><strong>Decision nodes:</strong> sub-nodes which split into branches based on conditions.</li><li><strong>Leaf nodes: </strong>final nodes of the tree which can not branch further.</li></ul><p>Following on from a root node, splits are made at decision nodes for one independent variable (continuous or categorical) following an if-else structure. Moving down the tree, data is continually ‘split’ until reaching a leaf node at which point a classification can be made. If this is not the case, the process is repeated until the outcome is reached.</p><figure><img alt="Decision tree illustration with definitions. Image by author." src="https://cdn-images-1.medium.com/max/1024/1*vJyByQvx160sqkQ31Q9BYQ.png" /><figcaption>Decision tree illustration with definitions. Image by author.</figcaption></figure><h4>Deciding the split</h4><p>There are multiple algorithms that are used by the decision tree to decide the best split for the problem, but the <strong>“Gini Index” and “Entropy” for Information Gain </strong>are the most common criterion. Both determine the <strong>impurity</strong> of the labels at a node, which is a measure of how heterogeneous or mixed values are at a decision node. This is critical as decision tree algorithms find optimal criteria that make a set into a more homogenous subset (i.e. ↓ impurity) rather than heterogenous (i.e. ↑ impurity).</p><p><strong>Entropy</strong> is one measure of the impurity for a set of data, and can be represented via the following formulae:</p><figure><img alt="The entropy formulae, reproduced by author. Where “Pi” denotes the probability of class “i” in the dataset." src="https://cdn-images-1.medium.com/max/593/1*Zegxm4rIFAKlNOurO7b0ng.png" /><figcaption>The entropy formulae. Where “<strong>Pi</strong>” denotes the probability of class “i” in the dataset.</figcaption></figure><p>By using an entropy calculation, the randomness or disorder can be calculated at a node. <strong>Information gain </strong>uses entropy to quantify which feature provides the maximum information about the classification by reducing entropy, and thus makes a split accordingly:</p><figure><img alt="Information gain formulae, reproduced by author. The entropy of class “Y” is subtracted from the entropy of “Y” on class “X”." src="https://cdn-images-1.medium.com/max/675/1*PWSg3CiV6jWxSvQQZ03k_Q.png" /><figcaption>Information gain formulae. The entropy of class “<strong>Y</strong>” is subtracted from the entropy of “<strong>Y</strong>” on class “<strong>X</strong>”.</figcaption></figure><p>Similar to entropy, the <strong>Gini Index</strong> (also known as <strong>Gini Impurity or Gini Coefficient</strong>) varies between values of 0 and 1. It calculates the impurity of a classification by measuring the divergences between the probability distributions of the target attribute’s values. An output of 0 is homogenous while 1 is impure, and suggests a strong likelihood of new, random data being misclassified. Once achieved, a node split is conducted which reduces this calculated impurity. For “C” different classes, the Gini formulae will be:</p><figure><img alt="The Gini formulae, reproduced by author. Where “C” represents the total number of classes in the subset of data, and “i” is a selected class from C." src="https://cdn-images-1.medium.com/max/454/1*9x70BLJN9VIsxQ-PVm34Lw.png" /><figcaption>The Gini Impurity formulae, where “<strong>C” </strong>represents the total number of classes in the subset of data, and “<strong>i”</strong> is a selected class from C.</figcaption></figure><h4>The downfall of decision trees</h4><p>While decision trees can offer a visualised solution to classification problems, there are <strong>disadvantages</strong> of using the algorithm which should be taken into consideration:</p><ul><li>They will continue to form branches until every node is homogenous. This creates an overfitting issue (robustness) if a small sample is tested.</li><li>Only one independent variable at a time is used to determine the classification(s), which may lead to performance and accuracy issues.</li><li>Does not handle missing values and numeric attributes.</li></ul><h3><strong>How does a Random Forest classification algorithm work?</strong></h3><p>The importance of understanding the basic concept of classification and regression trees (CART) comes into play here, as the <strong>Random Forest</strong> uses an <strong>ensemble </strong>of many decision trees instead of one to make a final decision (an Ensemble technique). This is particularly powerful as an ensemble of models will outperform an individual model. Ultimately, this addresses some of the disadvantages associated with decision trees as it leads to an improved performance and robustness. Notably, each tree in the ensemble is relatively uncorrelated which is important because, as outlined by <a href="https://medium.com/u/840a3210fbe7">Tony Yiu</a>:</p><blockquote>The trees protect each other from their individual errors (as long as they don’t constantly all error in the same direction). While some trees may be wrong, many other trees will be right, so as a group the trees are able to move in the correct direction.</blockquote><figure><img alt="Illustration of a simplified Random Forest classification algorithm. Image by author." src="https://cdn-images-1.medium.com/max/1024/1*jM6Z2SEgo3lrbQ1j6o6aYg.png" /><figcaption>Illustration of a simplified Random Forest classification algorithm. Image by author.</figcaption></figure><h4>How does the Random Forest algorithm build multiple trees?</h4><p>The Random Forest algorithm implements a combination of both <strong>(1) bootstrap aggregation and (2) feature randomness</strong> to build many decision trees using the same dataset. Collectively, they do so while remaining relatively uncorrelated — which is an important feature:</p><ol><li><strong>Bootstrap aggregation (AKA bootstrapping or bagging) </strong>is the technique involving randomly sampling a subset of data over a given number of iterations and variables (a bootstrap sample) with replacement. The predictions from all iterations and samples are typically averaged to obtain a most likely outcome. It is important to understand that it is not ‘chunking’ the data into small sizes and training individual trees on them, but rather it is still maintaining the initial data size. This is an example of an applied ensemble model.</li><li><strong>Feature randomness </strong>primarily functions to reduce the correlation between decision tree models. In comparison to decision trees which can utilise all features to discern to optimum node split, Random Forest algorithms will select these at random for decisions. Ultimately, this allows training to also occur on different features.</li></ol><blockquote><strong>Collectively, it is through these methods that Random Forest algorithms can be trained on different sets of the same data (bootstrapping) while also utilising different features to generate prediction outcomes. This serves as a far more robust and powerful tool for classification than decision trees.</strong></blockquote><h4>Feature importance</h4><p>Lastly, Random Forest algorithms will also evaluate the importance of each feature in a dataset on an artificial classification problem using <strong>Gini Importance</strong> (or mean decrease impurity). Now knowing that the Random Forest algorithms are constructed using an ensemble of decision trees, intuitively each internal nodes are selected using Gini Impurity or Information Gain (as outlined above). For each feature the decreases in impurity is calculated, and averaged across all decision trees in the ensemble to determine feature importance. This method is available in <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.feature_importances_">scikit-learn implementation</a> of the Random Forest (for both classifier and regressor).</p><h3>Applying a Random Forest classifier to extracellular neural recordings:</h3><p>Now, for the application! We will be using a publicly available dataset made available <a href="https://www.kaggle.com/joseguzman/spike-classification-based-on-waveforms/data">here</a> under license <strong>Attribution 4.0 International (CC by 4.0)</strong>. It is a collection of extracellular recordings taken from a mammalian brain.</p><blockquote>For context, <strong>extracellular recordings</strong> are recordings of electrical potentials produced by a cell, either in extracellular fluid near the cell of interest, or non-invasively.</blockquote><h4>Loading the libraries and dataset</h4><p>To begin, we will import the following libraries and the load required dataset.</p><pre># Install the following libraries<br>import pandas as pd<br>import numpy as np<br>from umap import UMAP<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br>from mpl_toolkits.axes_grid1.anchored_artists import AnchoredSizeBar</pre><pre>from sklearn.model_selection import train_test_split<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import accuracy_score<br>from sklearn.cluster import AgglomerativeClustering<br>from sklearn.neighbors import kneighbors_graph<br>from sklearn.cluster import KMeans<br>from sklearn.metrics import silhouette_samples, silhouette_score<br>from sklearn.metrics import classification_report<br>from sklearn.metrics import confusion_matrix</pre><p>To become familiar with the dataset, it is important in the first instance to visualise the data classes. This enables us to gain a better understanding of the data-set and possible ways to organise the data.</p><pre># Load waveforms<br>mypath = &#39;*insert file path/waveforms.csv&#39;<br>data = pd.read_csv(mypath, index_col = &#39;uid&#39;)</pre><pre># Get an overview of the data<br>print(f&#39;{data.shape[0]} unique experiment identifiers (uid), recorded with a sampling frequency (KHz) of {((data.shape[1]-1)/5)}&#39;)</pre><pre># Class breakdown<br>data.organoid.value_counts()</pre><p>Seeing that we have multiple classes, as shown by value_counts(), we will label each with a unique color code. This will prove useful when visualizing these classes in further data analyses.</p><pre># Define custom colors for visualization<br>mycolors = {&#39;Data_D&#39;:     &#39;#FFA500&#39;, # orange<br>           &#39;Data_G&#39;:       &#39;#4169E1&#39;, # royalblue<br>           &#39;Data_F&#39;:       &#39;#FF4500&#39;, # orange red<br>           &#39;Data_C&#39;:       &#39;#9400D3&#39;, # darkviolet<br>           &#39;Data_A&#39;:       &#39;#32CD32&#39;, # limegreen <br>           &#39;Data_E&#39;:       &#39;#228B22&#39;, # forestgreen<br>           &#39;Data_G_V2&#39; :   &#39;#006400&#39;, # darkgreen <br>           &#39;Data_H&#39;:       &#39;#00BFFF&#39;, # deepskyblue <br>           &#39;Data_E_V2&#39;:    &#39;#DC143C&#39;, # crimson<br>           &#39;Data_F_V2&#39;:    &#39;#0000FF&#39;, # blue<br>           &#39;Data_B&#39;:       &#39;#000000&#39;, # black<br>           }</pre><pre># Add color to the DataFrame<br>data[&#39;color&#39;] = data[&#39;organoid&#39;].apply(lambda orgID: mycolors[orgID])</pre><p>Now, we can visualize! I have displayed a simple barplot below to summarise the number of recordings per class. However, further data visualization tools for Python can be found <a href="https://towardsdatascience.com/top-python-libraries-for-visualization-a-starting-guide-73402178811b">here</a> in a previous post.</p><pre># Visualizing the unique experiment identifiers<br>fig, ax = plt.subplots(figsize=(15, 8))<br>sns.barplot(x=data.organoid.value_counts().index, y=data.organoid.value_counts(), palette=mycolors)</pre><pre># Customizing the graph<br>plt.xticks(rotation=30,fontsize=14)<br>plt.yticks(fontsize=14)<br>ax.set_xlabel(&#39;Class type&#39;, fontsize=16)<br>ax.set_ylabel(&#39;Number of waveforms&#39;, fontsize=16)<br>plt.rcParams[&quot;font.family&quot;] = &quot;Arial&quot;</pre><pre># Despine<br>right_side = ax.spines[&quot;right&quot;]<br>right_side.set_visible(False)<br>top_side = ax.spines[&quot;top&quot;]<br>top_side.set_visible(False)</pre><pre>plt.savefig(&#39;Figures/barplot.png&#39;, dpi = 300, bbox_inches=&quot;tight&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aL3aY0KYtV_9r-Jv5Cv98A.png" /><figcaption>Image by author. A barplot highlighting the variation in the number of waveforms recorded from each class type.</figcaption></figure><h4>Visualizing the extracellular waveforms</h4><p>Given that we are analysing extracellular recordings, we will visualize the mean waveforms produced by each dataset class. This will provide further insight into the classification problem at hand. I have done so below by computing the mean trace for each dataset class, overlaid onto their input traces.</p><pre># Isolating the waveforms for each class <br>class_names = data[&#39;organoid&#39;].unique()</pre><pre># Plotting mean traces for each organoid class<br>fig, ax = plt.subplots(1,9, figsize=(24,4.5))<br>for unique_class in class_names:    <br>        df_new = data[data[&#39;organoid&#39;] == unique_class] #isolating </pre><pre># np.array conversion<br>        df_new = df_new.iloc[:,:-2].to_numpy() #dropped last column</pre><pre># Averaging across samples per organoid class <br>        data_mean_perclass = np.mean(df_new, axis=0)</pre><pre># Sampling frequency for plot generation<br>        sampling_freq = np.linspace(0, 5, 150) #recording length is 5ms per 150 samples</pre><pre>for i in range(class_names.shape[0]): <br>            if unique_class == class_names[i]:<br>                <br>                # Plotting all traces behind in light color<br>                for row_num in range(df_new.shape[0]): <br>                    ax[i].plot(sampling_freq, df_new[row_num,:], color = &#39;lightgray&#39;)<br>                <br>                # Plotting each mean waveform into a separate subplot <br>                ax[i].plot(sampling_freq,data_mean_perclass, color=mycolors[unique_class], linewidth=3)<br>                ax[i].set_ylim([-1.8, 1.8])<br>                ax[i].grid()<br>                ax[i].axis(&#39;off&#39;)<br>                ax[i].title.set_text(unique_class)  <br>                plt.rcParams[&quot;font.family&quot;] = &quot;Arial&quot;<br>                <br>            else: <br>                continue<br>        <br>        # Scale bar<br>        scalebar = AnchoredSizeBar(ax[8].transData, 1, &quot;1 ms&quot;, &#39;lower right&#39;, frameon=False, size_vertical=0.02, pad=0.1)<br>        ax[8].add_artist(scalebar)<br>        <br>plt.savefig(&#39;Figures/spikes.png&#39;, dpi = 300)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v2arEuKXIdsP2x8fTLZQwA.png" /><figcaption>Mean extracellular waveforms from each class are shown in color and correspond to their subplot titles. Light gray represents input signals used to generate the mean output array. Image by author with re-formatted dimensions for display.</figcaption></figure><h4>Multi-dimensionality reduction</h4><p>Visually, the waveform traces highlight that each dataset class has a number of individual spike waveforms which arise from different neuronal cell-types. When averaged, it can be said that some of these classes look quite similar, while others do not.</p><p>To better validate this, we will conduct a multi-dimensionality reduction on the extracellular waveform data using a Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP). Ultimately, Principal Component Analysis or t-Distributed Stochastic Neighbor Embedding (t-SNE) can also be conducted. The differences between such methods are outlined by <a href="https://medium.com/u/4555d4aeba04">Sivakar Sivarajah</a> <a href="https://towardsdatascience.com/dimensionality-reduction-for-data-visualization-pca-vs-tsne-vs-umap-be4aa7b1cb29">here</a>. For the sake of this article, we will focus only on the first 2-components to capture majority of the variance.</p><pre># UMAP calculation<br>umap_2d = UMAP(n_components=2, random_state = 43)<br>projections = umap_2d.fit_transform(data.iloc[:,:-2])</pre><pre># Concat dataframes for seaborn scatter plot<br>UMAP_df = pd.DataFrame(projections, columns = [&#39;UMAP 1&#39;,&#39;UMAP 2&#39;])<br>UMAP_df_concat = pd.concat([UMAP_df,data[&#39;organoid&#39;].reset_index(),data[&#39;color&#39;].reset_index()], axis=1)</pre><pre># Figure plotting<br>sns.set(font_scale=1.2)<br>sns.set_style(&quot;whitegrid&quot;, {&#39;axes.grid&#39; : False})<br>fig1 = sns.relplot(data = UMAP_df_concat, x = &quot;UMAP 1&quot;, y = &quot;UMAP 2&quot;, hue=&quot;organoid&quot;, kind=&quot;scatter&quot;, palette=mycolors, height=5, aspect=1.5)<br>fig2 = sns.relplot(data = UMAP_df_concat, x = &quot;UMAP 1&quot;, y = &quot;UMAP 2&quot;, hue=&quot;organoid&quot;, kind=&quot;scatter&quot;, palette=mycolors, height=3, aspect=1.5, col=&#39;organoid&#39;, col_wrap=3)</pre><pre># Figure export<br>fig1.savefig(&#39;Figures/UMAP.png&#39;, dpi = 300)<br>fig2.savefig(&#39;Figures/UMAP_2.png&#39;, dpi = 300)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsELYKAFtSFzPnbh8z9fSg.png" /><figcaption>UMAP plot of extracellular waveform time series data, for n_components = 2. Color corresponds to each dataset class. Image by author.</figcaption></figure><p>From a birds-eye view, different clusters are forming on the UMAP. Each cluster does not have a homogenous class type (shown by the color variations), but rather is a combination of different extracellular recordings from various classes. It is therefore likely that each organic class type has a proportion of waveforms which are similar to other classes.</p><p>To better view the contributions of each dataset the UMAP and how they cluster, we will isolate their data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0cjEgc8PXZFr-7Ba5g0Bg.png" /><figcaption>A series of UMAP subplots representing unique extracellular waveform time series data from each dataset class. Image by author.</figcaption></figure><p>It does indeed appear that some recordings from specific dataset classes cluster more similarly, while others do not. This aligns with our waveform traces. Moving forward, we will cluster our data to establish a classification problem for our Random Forest algorithm.</p><h4>K-means clustering</h4><p>To conduct an unbiased clustering of our data using the first 2-components, we will utilize K-means. This clustering method belongs to the Partitioning Class of Clustering, which serves to minimise total intra-cluster variation. To select thr optimal number of clusters,<em> we will utilise the (1) Elbow and (2) Silhouette method:</em></p><ol><li><strong>Elbow Method:</strong> Calculated the total within-cluster sum of square (wss), which measures the compactness of the clustering. The aim is to minimise this as much as possible.</li><li><strong>Silhouette Method: </strong>We will back-test the Elbow Method using the Silhouette Method. This method measures the quality of the cluster by determining how well an object lies within it. High silhouette values suggest ‘good’ clustering, while low silhouette values suggest ‘poor’ clustering.</li></ol><p>It is a fine balance between these two methods to determine the optimal number of clusters for your data. So, let’s step through it:</p><pre># Set range of possible clusters<br>range_n_clusters = range(2, 11) #set as 2 to 10</pre><pre># Elbow method &amp; Silhouette scores for optimal number of cluster<br>wcss = [] #within cluster sum of square <br>silhouette_values = []<br>for i in range_n_clusters:<br>    <br>    kmeans = KMeans(n_clusters = i, random_state = 42)<br>    kmeans.fit(data.iloc[:,:-2])<br>    wcss.append(kmeans.inertia_)<br>    <br>    cluster_labels = kmeans.fit_predict(data.iloc[:,:-2])<br>    silhouette_avg = silhouette_score(data.iloc[:,:-2], cluster_labels)<br>    silhouette_values.append(silhouette_avg)</pre><pre># Isolating highest Silhouette value calculation<br>max_silhouette = max(silhouette_values)<br>max_index = silhouette_values.index(max_silhouette)<br>print(f&#39;Optimum number of clusters is {range_n_clusters[max_index]}&#39;)<br>    <br># Figure plotting<br>fig, ax = plt.subplots(1,2, figsize=(18,4.5))<br>ax[0].plot(range_n_clusters, wcss)<br>ax[1].plot(range_n_clusters, silhouette_values)<br>ax[0].title.set_text(&#39;The Elbow Method&#39;)<br>ax[1].title.set_text(&#39;Optimal Cluster Number&#39;)<br>ax[0].set(xlabel=&#39;Number of Clusters&#39;, ylabel=&#39;WCSS&#39;)<br>ax[1].set(xlabel=&#39;Number of Clusters&#39;, ylabel=&#39;Silhouette score&#39;)</pre><pre># Line to indicate optimum cluster number<br>ax[0].axvline(x=range_n_clusters[max_index], color=&#39;black&#39;, label=&#39;axvline - full height&#39;, ls=&#39;--&#39;)<br>ax[1].axvline(x=range_n_clusters[max_index], color=&#39;black&#39;, label=&#39;axvline - full height&#39;, ls=&#39;--&#39;)</pre><pre># Export figures<br>fig.savefig(&#39;Figures/Silhouette_elbow_method.png&#39;, dpi = 300, bbox_inches=&quot;tight&quot;)</pre><figure><img alt="Results of the Elbow (left) and Silhouette Method (right) on the neural spike data set. The dotted line highlights the optimum number of clusters using K-means clustering. Image by author." src="https://cdn-images-1.medium.com/max/1024/1*lws_LmqZpPa_FbSWVrpmTQ.png" /><figcaption>Results of the Elbow (left) and Silhouette Method (right) on the neural spike data set. The dotted line highlights the optimum number of clusters using K-means clustering. Image by author.</figcaption></figure><p>As can be seen by the dotted line on the graph, <em>the optimum number of clusters for our dataset is 3</em>. Reason being is that for this number, the silhouette score is highest while the within-cluster sum of square (WCSS) is reduced. We will confirm this by visualizing the clusters on our combined UMAP plot using the following code:</p><pre># Agglomerative clustering with optimal cluster number on UMAP<br>def clustering_agglo(df, n_clusters):<br>    X = df.to_numpy()<br>    connectivity = kneighbors_graph(X, int(len(df.index)/10), include_self=False)<br>    agc = AgglomerativeClustering(linkage=&#39;ward&#39;, connectivity=connectivity, n_clusters=n_clusters)<br>    agc.fit(X)<br>    print(f&#39;Labelling {len(np.unique(agc.labels_))} clusters on original UMAP projection&#39;)<br>    return agc.labels_</pre><pre>labelsAgglo = clustering_agglo(data.iloc[:,:-2],range_n_clusters[max_index])</pre><pre># UMAP plotting<br>sns.set(font_scale=1.2)<br>sns.set_style(&quot;whitegrid&quot;, {&#39;axes.grid&#39; : False})<br>fig = sns.relplot(data = UMAP_df_concat, x = &quot;UMAP 1&quot;, y = &quot;UMAP 2&quot;, hue=labelsAgglo, kind=&quot;scatter&quot;, palette=[&#39;red&#39;,&#39;blue&#39;,&#39;green&#39;], height=5, aspect=1.5)</pre><pre># Add labels to original dataframe: <br>data[&#39;cluster_group&#39;] = labelsAgglo</pre><pre># Figure export<br>fig.savefig(&#39;Figures/UMAP_clusters.png&#39;, dpi = 300)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X7kLT4EFROnnRZPY2BarTw.png" /><figcaption>UMAP plot of time-series extracellular data recorded across dataset classes. Colors correspond with clusters determined using K-means clustering, and validated using the Silhouette and Elbow methods. Image by author.</figcaption></figure><p>From this plot, we can see (for the most part) that the K-means clustering has down a reasonable job! Notably, there is some overlap between the clusters on the UMAP plot, but seeing this is a real dataset using neural spike data it can be expected. Now, let’s investigate these further?</p><h4>Class and waveform breakdowns for each cluster</h4><p><em>As aforementioned, waveforms recorded from specific dataset classes cluster more similarly, while others do not.</em> To better understand these differences, lets look at the mean waveforms from each cluster and the dataset classes which comprise them:</p><pre>#Cluster group colour<br>cluster_colors=[&#39;red&#39;,&#39;blue&#39;,&#39;green&#39;]</pre><pre>#Breakdown of each waveform cluster type by organoid class<br>for cluster_group in range(0,3):<br>    df_group = data[data[&#39;cluster_group&#39;] == cluster_group] #isolating each cluster group<br>    breakdown_perclust = ((df_group.organoid.value_counts())/(df_group.shape[0]))*100<br>    <br>    # Getting idx for each cluster for color matching<br>    breakdown_perclust_df = breakdown_perclust.to_frame()<br>    breakdown_perclust_df[&#39;index&#39;] = breakdown_perclust_df.index<br>    breakdown_perclust_df[&#39;color&#39;] = breakdown_perclust_df[&#39;index&#39;].apply(lambda orgID: mycolors[orgID])<br>    breakdown_perclust_df = breakdown_perclust_df.rename(columns = {&#39;organoid&#39; : &#39;class_breakdown_percentage&#39;})<br>    <br>    # Computing mean waveforms for each cluster<br>    df_group_new = df_group.iloc[:,:-3].to_numpy() #dropped last columns<br>    data_mean_group_percluster = np.mean(df_group_new, axis=0)</pre><pre># Sampling frequency for plot generation<br>    sampling_freq = np.linspace(0, 5, 150) #recording length is 5ms per 150 samples<br>    <br>    # Piechart plotting<br>    fig, ax = plt.subplots(1,2, figsize=(16,4), gridspec_kw={&#39;width_ratios&#39;: [2.5, 1]})<br>    ax[0].pie(breakdown_perclust_df[&#39;class_breakdown_percentage&#39;], colors = breakdown_perclust_df[&#39;color&#39;], labels = breakdown_perclust_df[&#39;index&#39;],\<br>              autopct=&#39;%1.0f%%&#39;, pctdistance=0.5, labeldistance=1.1)<br>    title_pie = (&quot;Cluster number: &quot; + str(cluster_group))<br>    ax[0].title.set_text(title_pie)<br>    <br>    # Draw inner circle<br>    centre_circle = plt.Circle((0,0),0.70,fc=&#39;white&#39;)<br>    <br>    # Equal aspect ratio ensures that pie is drawn as a circle<br>    ax[0].add_patch(centre_circle)  <br>    ax[0].axis(&#39;equal&#39;)  <br>    <br>    # Mean waveform plotting<br>    ax[1].plot(sampling_freq,data_mean_group_percluster, linewidth=3, color = cluster_colors[cluster_group])<br>    title_waveform = (&quot;Mean waveform for cluster number: &quot; + str(cluster_group))<br>    ax[1].title.set_text(title_waveform)<br>    ax[1].set_ylim([-1.3, 1.3])<br>    ax[1].grid()<br>    ax[1].axis(&#39;off&#39;)<br>    plt.rcParams[&quot;font.family&quot;] = &quot;Arial&quot;<br>    <br>    # Scale bar<br>    scalebar = AnchoredSizeBar(ax[1].transData, 1, &quot;1 ms&quot;, &#39;lower right&#39;, frameon=False, size_vertical=0.02, pad=0.1)<br>    ax[1].add_artist(scalebar)<br>    <br>    # N-values<br>    n_value = (&quot;N-value: &quot; + str(df_group.shape[0]))<br>    plt.figtext(0.36, -0.05, n_value, ha=&quot;center&quot;, fontsize=12, bbox={&quot;facecolor&quot;:&quot;orange&quot;, &quot;alpha&quot;:0.2, &quot;pad&quot;:5})<br>    <br>    # Export figures<br>    fig_name = (&quot;Figures/Piechart_waveform_clust&quot; + str(cluster_group) + &quot;.png&quot;)<br>    plt.savefig(fig_name, dpi = 300, bbox_inches=&quot;tight&quot;)<br>    plt.tight_layout()<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5acBmO6j1-BdAKdc5tZ1A.png" /><figcaption>The pie charts (left) highlights the percentage of waveforms from each dataset class and their corresponding mean waveform (right). Image by author.</figcaption></figure><p>At first glance, it can be seen that there is a considerable amount of difference in the mean waveforms between the different data-set clusters. Additionally, the percentage of wavefroms from different dataset classes contributing to the clusters also appear to vary.</p><blockquote>Now, we have a clear classification issue for our Random Forest classifier! We can validate these differences between cluster(s).</blockquote><h4>Splitting data into separate training and test set</h4><p>To effectively train the Random Forest algorithm, we will split the data into a train and test set. The desired outcomes for our predictions yare the cluster types, while our extracellular spike time-series data is X.</p><p>The reason for test and training sets it to <strong>not allow enough data in the training dataset for the model to learn an effective mapping of inputs to outputs</strong>. There will also not be enough data in the test set to effectively evaluate the model performance. This can be done using the following code:</p><pre># Split data into training and testing sets<br>X = data.iloc[:,:-3] #drop last 3 columns<br>y = data[&#39;cluster_group&#39;]<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 42)</pre><pre># check the shape of X_train and X_test<br>print(f&#39;Number of entries for X_train is {X_train.shape[0]}, and the number of entries for X_test is {X_test.shape[0]}&#39;)</pre><figure><img alt="Print output. Image by author." src="https://cdn-images-1.medium.com/max/1024/1*opMceRsVGxAqv8nCmOJ72g.png" /><figcaption>Print output. Image by author.</figcaption></figure><h4>Random Forest Classification</h4><p>Now, let’s train our Random Forest classifier! We will do so over a number of decision trees (iterations) and test their prediction accuracy. Here, <strong>y_test</strong> are the true class labels and <strong>y_pred</strong> are the predicted class labels in the test-set:</p><pre># Instantiate the Random Forest classifier <br>number_int = [1,2,3,4,5,10,20,50,70,100,200,400,500]<br>accuracy = []<br>for i in number_int: <br>    rfc = RandomForestClassifier(n_estimators=i, random_state=41)</pre><pre># Fit the model<br>    rfc.fit(X_train, y_train)</pre><pre># Predict the test set results<br>    y_pred = rfc.predict(X_test)<br>    <br>    # Append to list<br>    accuracy.append(&#39;{0:0.4f}&#39;.format(accuracy_score(y_test, y_pred)))</pre><pre># Check accuracy score <br>    print(f&#39;Model accuracy score with {i} decision-trees is {format(accuracy_score(y_test, y_pred))}&#39;)</pre><pre># Figure plotting<br>fig, ax = plt.subplots(figsize=(7,4))<br>ax.plot(number_int, accuracy)<br>ax.title.set_text(&#39;Number of interations vs accuracy&#39;)<br>ax.set(xlabel=&#39;Number of iterations&#39;, ylabel=&#39;Accuracy&#39;)<br>plt.savefig(&#39;Figures/Iterations_accuracy.png&#39;, dpi = 300, bbox_inches=&quot;tight&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fSmEnD3Q6O8vlOqbZwtmOg.png" /><figcaption>Relationship between ‘Accuracy’ and the ‘Number of iterations’ for the Random Forest classifier. Image by author.</figcaption></figure><p>Despite the higher number of decision trees, the model accuracy score does increase between 1 and 100 before plateauing. With 2 decision-trees it is 0.93877, while with 500 decision-trees it is 0.9632. So, as expected accuracy increases with number of decision-trees in the model.</p><h4>Confusion Matrix</h4><p>Lastly, let’s construct a confusion matrix with a classification report to summarize the performance of our Random Forest classification algorithm and assess accuracy, precision and recall metrics.</p><p>A confusion matrix will provide an indication of our models performance and the types of errors produced by summarising correct and incorrect predictions. These can fall into the following four categories: <strong>(1)</strong> <strong>True Positives (TP), (2) True Negatives (TN)</strong> , <strong>(3)</strong> <strong>False Positives (FP)</strong> and <strong>(4) False Negatives (FN). </strong><a href="https://medium.com/u/2489258d6569">Giulio Laurenti, PhD</a> explains said readouts in more detail <a href="https://medium.com/swlh/confusion-matrix-and-classification-report-88105288d48f#:~:text=The%20confusion%20matrix%20is%20an,predictions%20of%20the%20classification%20model.&amp;text=The%20rows%20of%20the%20matrix,columns%20represent%20the%20predicted%20classes.">here</a>.</p><pre># Confusion matrix<br>cm = confusion_matrix(y_test, y_pred)<br>print(classification_report(y_test, y_pred))</pre><pre># Figure plotting<br>cmap = sns.color_palette(&quot;Greys&quot;, as_cmap=True) # color map<br>fig, ax = plt.subplots(figsize=(8, 8))<br>cm_percent = ((cm/cm.sum(axis=0))*100)<br>cm_percent = np.nan_to_num(cm_percent, copy=True, nan=0.0, posinf=None, neginf=None)<br>sns.heatmap(cm_percent, cmap = cmap, annot=True, fmt=&quot;.2f&quot;, linewidth=1, cbar_kws={&quot;label&quot;: &quot;Cluster group classified (%)&quot;, &quot;shrink&quot;: .8}, annot_kws={&quot;size&quot;: 16}) #selecting for percentage only column <br>ax.set_title(&#39;Confusion matrix&#39;)<br>plt.show()</pre><pre># Figure export<br>fig.savefig(&quot;Figures/Confusion_matrix.png&quot;, dpi=300, bbox_inches=&#39;tight&#39;) </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t8IR_9qyN3cT7_2WIizoXw.png" /><figcaption>The outputted confusion matrix classification report (screenshot). Image by author.</figcaption></figure><figure><img alt="Confusion matrix for cluster-groups 0-to-2. Different shades highlight the number of predictions classified as either TP, TN, FN, or FP relative to the total possible classifications per cluster (%). Image by author." src="https://cdn-images-1.medium.com/max/1024/1*z96MCCxiK8mrbkU9YSsl7w.png" /><figcaption>Confusion matrix for cluster-groups 0-to-2. Different shades highlight the number of predictions classified as either TP, TN, FN, or FP relative to the total possible classifications per cluster (%). Image by author.</figcaption></figure><h3>Conclusion</h3><p><em>There we have it! We have now built a Random Forest algorithm to accurately and precisely classifier subsets of neural spike data.</em></p><p>In this post, we have not only guided through a real example involving neural extracellular data, but we have also covered the theory behind decision trees and Random Forest classifiers, along with implemented multi-dimensionality reduction and K-means clustering into our workflow.</p><p>Although, while we have discussed solving classification problems using Random Forest here, it is important to recognise that there are a multitude of ways to tackle these. So in future articles we will discuss this further by covering classification using deep learning algorithms, logistic regression.</p><p>You can contact and connect with me via <a href="https://www.linkedin.com/in/michael-zabolocki/">LinkedIn</a>, follow me via <a href="https://twitter.com/mzabolocki1">Twitter</a>, or join me on <a href="https://github.com/mzabolocki">GitHub</a> and <a href="https://www.kaggle.com/michaelzabolocki">Kaggle</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e523f3639e1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-a-random-forest-classifier-for-neural-spike-data-8e523f3639e1">Building a Random Forest Classifier to Predict Neural Spikes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top Python Libraries for Visualization: A Starting Guide]]></title>
            <link>https://medium.com/data-science/top-python-libraries-for-visualization-a-starting-guide-73402178811b?source=rss-d52f4e270c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/73402178811b</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Michael Zabolocki]]></dc:creator>
            <pubDate>Sat, 01 Jan 2022 11:32:25 GMT</pubDate>
            <atom:updated>2022-01-30T13:14:36.561Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>The guide to plotting scatter plots, heat maps, ridgeline plots and line graphs in Python</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y7-6ni-MqllfGbiW9pHMqg@2x.jpeg" /><figcaption>Image by Daniel Olah on Unsplash.</figcaption></figure><p>Given the enormous number of libraries and possibilities for data visualization in Python, it can quickly become a difficult and somewhat overwhelming endeavour to navigate through for anyone new to the field. There is an abundance of options to choose from, but knowing which is correct for you (and why) is an important step to effectively showcasing data. In this article we focus on several key libraries for effective data visualization, and provide clear working examples for implementation into any dataset.</p><h3><strong>What is data visualization?</strong></h3><p>In lay terms, data visualization is the ‘art’ of utilising graphic displays to show raw data. More specifically, it is useful for exploring data structure, detecting outliers, trends, and clusters to reduce data complexity. Hence, effective data visualization will help convey data to a target audience in an easily understandable visual context for the human brain.</p><p>The landscape of data visualization is here to stay and continues to evolve with technological advances, leaving a more-and-more prominent footprint in scientific publications, media and website displays. More precise reproduction, better color (including alpha-blending), and even faster plotting are some variables which have rapidly improved within the last decade.</p><blockquote><a href="https://hdsr.mitpress.mit.edu/pub/zok97i7p/release/3">Furthermore, progress has also been made in developing graphic theories, such as Wilkinson’s <em>Grammar of Graphics</em> (2005) and Hadley Wickham’s implementation of it in the R package ggplot2 (Wickham, 2016). In doing so, graphics that were rarely used in the past and difficult to draw, such as parallel coordinate plots (e.g., Theus, 2015) and mosaicplots (e.g., Unwin, 2015), have now been refined and developed.</a></blockquote><h3><strong>Why use Python for data visualization?</strong></h3><p>Many no-code tools for data visualization exist such as Tableau, Power BI, ChartBlocks, Prism, and more. However, even though programming is required, Python is one of the preferred languages among data scientists due to its visualization capabilities. This is also accompanied by data collection and exploration libraries (e.g. Pandas, NumPy, SciPy) for data manipulation and transformation. Already, Python has numerous libraries and features available for visualization which will continue to grow due to its large user base and open source nature. With the ever evolving technological landscape, Python is in excellent stead to maintain its position as the preferred language amongst data scientists for years to come.</p><h3>Library installation and dataset preparation</h3><p>There are numerous ways to plot statistical graphics with Python, however some of my preferred can be created with the following libraries listed below. In this article we will focus on plotting heatmaps, scatter plots, line graphs and ridgeline plots using the following:</p><ul><li>joypy</li><li>seaborn</li><li>plotly.express</li></ul><p><em>For general interest, I have summarised key library information with links to source documentation:</em></p><ul><li><a href="https://plotly.github.io/plotly_express/"><strong>Plotly Express</strong></a> is a high-level Python visualization library, serving as a wrapper for <a href="https://plot.ly/python/">Plotly.py</a> that exposes a simple syntax for complex charts. It can be used to make richly interactive plots in just a single function call, including faceting, maps, animations, and trendlines. Plotly Express is also fully compatible with the rest of the Plotly ecosystem.</li><li><a href="https://github.com/mwaskom/seaborn"><strong>Seaborn</strong></a> builds ontop of matplotlib, and works by capturing entire dataframes or arrays containing data and performs necessary internal functions for semantic mapping and statistical aggregation to produce informative plots.</li><li><a href="https://github.com/leotac/joypy"><strong>Joypy</strong></a> is a one-function Python package based on matplotlib and pandas with a single purpose: drawing joyplots (i.e. ridgeline plots). This library is very useful for visualizing data distributions in a clear manner.</li></ul><h4>Library installation</h4><p><em>If not already, please install these libraries via the terminal by running the following command line(s):</em></p><pre>pip install seaborn<br>pip install joypy<br>pip install plotly</pre><p>Once complete, we will import the following nodules before continuing:</p><pre>import numpy as np<br>import pandas as pd<br>import seaborn as sns<br>from joypy import joyplot<br>import plotly.express as px<br>from scipy import stats<br>import matplotlib.pyplot as plt</pre><h4>Loading a dataset</h4><p>Without data, we can not visualize anything. So before we step through a series of plotting examples, we will load the following in-built (and commonly used) python dataset about flights using the seaborn library:</p><pre># Load a dataframe<br>flights_data = sns.load_dataset(&quot;flights&quot;)</pre><pre># Display dataset<br>flights_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*z8s9fhSzr0KglkqdOyNrug.png" /><figcaption>Flights_data data frame example. Image by the author.</figcaption></figure><h3>Scatter plots</h3><p>Basic scatter plots can be made using plotly.express, with the flexibility to alter their width and height for more optimal image exports. Templates can also be changed from simple_white to incorporate grids and darker color:</p><pre>px.scatter(flights_data, x=”year”, y=”passengers”, width=700, height=600,template=”simple_white”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*WpR7D4A5vRQatLmY7vNkOw.png" /><figcaption>Basic scatter plot example with plotly.express. Image by the author.</figcaption></figure><p>Notably, plotly.express is great for colour sequencing, which is why I prefer it for visualizing grouped data. You also have the option to change colors from a discrete color sequence (shown below) to a continuous color scheme, which serves effective for continuous data measurements. Overall, different colours can be given to different data groups for improved visual clarity.</p><pre>#group data <br>flights_data.loc[flights_data.year &lt;= 1954 , &#39;group&#39;] = &#39;&lt;= 1954&#39;<br>flights_data.loc[flights_data.year &gt; 1954 , &#39;group&#39;] = &#39;&gt; 1954&#39;<br> <br>#re-plot data with discrete color scheme<br>px.scatter(flights_data, x=&quot;year&quot;, y=&quot;passengers&quot;, width=700, height=600, template=&quot;simple_white&quot;, color=&quot;group&quot;, color_discrete_sequence=[&#39;red&#39;,&#39;blue&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*QuGOhFInLyrCV1mM08p6Bg.png" /><figcaption>Plotly.express scatter plot with group display. Red and blue color correspond to ≤ 1954 and &gt; 1954 flight year, respectively. Image by the author.</figcaption></figure><p>If you are like me and despise the default aesthetics on the scatter plots, here are some helpful examples which can be implemented to change tick lengths, font sizes, axis titles, marker colors, and more:</p><pre>#size dimension change<br>fig1 = px.scatter(flights_data, x=&quot;year&quot;, y=&quot;passengers&quot;, width=700, height=450,template=&quot;simple_white&quot;, color=&quot;group&quot;,color_discrete_sequence=[&#39;red&#39;,&#39;blue&#39;])</pre><pre>#tick font and size change<br>fig1.update_layout(<br>yaxis = dict(<br>tickfont = dict(size=20)),<br>xaxis = dict(<br>tickfont = dict(size=20)),<br>font_family=&quot;Arial&quot;)</pre><pre>#x-axis title and size + tick length changes<br>fig1.update_xaxes(<br>title_text = &quot;Year of flight&quot;,<br>title_font = {&quot;size&quot;: 20},<br>title_standoff = 15, ticklen=10)</pre><pre>#y-axis title and size + tick length and value changes<br>fig1.update_yaxes(<br>title_text = &quot;Passengers&quot;,<br>title_font = {&quot;size&quot;: 20},<br>title_standoff = 15, ticklen=10, tickvals=list(range(0,700,100)))</pre><pre>#marker size, outline and color change <br>fig1.update_traces(marker=dict(size = 9,<br>line=dict(width=0.8, color=&#39;white&#39;)),<br>selector=dict(mode=&#39;markers&#39;))</pre><pre>#legend off <br>fig1.update_layout(showlegend=False)<br>    <br>fig1.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*3rlns97U3azr84tIZguS5g.png" /><figcaption>Adjusted marker and font sizes, along with tick lengths and font styles using plotly.express. Image by the author.</figcaption></figure><p>Notably, you can choose any of the following built-in qualitative color sequences from the px.colors.qualitative module, or define your own:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*6J65i45JmYm3ZFy0Pco7fQ.png" /><figcaption>Example ploty.express color palettes. Image by <a href="https://plotly.com/python/discrete-color/">Plotly</a>.</figcaption></figure><h3>Line plots</h3><p>Even though plotly.express has the option to add a trendline, here we show how to implement a <strong>filled</strong> trendline to the existing data using seaborn libraries. This is particularly helpful to visualize relationships between variables. In the case of year and passenger variables, a linear relationship can be seen and hence a trendline will help to reinforce this.</p><pre>#use lmplot to plot scatter points<br>sns.set(font_scale = 1.6)<br>sns.set_style(&quot;white&quot;)<br>fig2 = sns.lmplot(data = flights_data, x=&quot;year&quot;, y=&quot;passengers&quot;, height=5, aspect=1.64, legend=False, line_kws={&#39;color&#39;: &#39;black&#39;}, scatter_kws={&quot;color&quot;: &quot;black&quot;})</pre><pre>#change axis titles<br>fig2.set(xlabel=&#39;Year of flight&#39;, ylabel=&#39;Passengers&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*nwvg1wa_bVOfUEcGHb94BA.png" /><figcaption>Example seaborn scatter plot with filled trendline. Image by the author.</figcaption></figure><p>To maintain the color scheme for previous data groups we can specify color to specific datapoints. A <strong>filled </strong>trendline can also be applied to the entire data set, as shown below:</p><pre>#get coeffs of linear fit<br>slope, intercept, r_value, p_value, std_err = stats.linregress(flights_data[&#39;year&#39;],flights_data[&#39;passengers&#39;])</pre><pre>#use lmplot to plot scatter points<br>sns.set(font_scale = 1.6)<br>sns.set_style(&quot;white&quot;)<br>sns.lmplot(data = flights_data, x=&quot;year&quot;, y=&quot;passengers&quot;, height=5, aspect=1.64, hue=&quot;group&quot;, fit_reg= False, legend=False, palette=[&quot;red&quot;,&quot;blue&quot;])</pre><pre>#use regplot to plot the regression line and use line_kws to set line label for legend<br>fig3 = sns.regplot(data = flights_data, x=&quot;year&quot;, y=&quot;passengers&quot;, scatter_kws={&quot;zorder&quot;:-1}, line_kws={&#39;label&#39;:&quot;y={0:.1f}x+{1:.1f}&quot;.format(slope,intercept)}, color=&quot;black&quot;)</pre><pre>#change axis titles<br>fig3.set(xlabel=&#39;Year of flight&#39;, ylabel=&#39;Passengers&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*l3Ws8RrcOj28eilqJlHJPw.png" /><figcaption>Seaborn scatter plot with filled trendline and different color scheme for groups 1 and 2. Image by the author.</figcaption></figure><p>Alternatively, a separate filled trendline can be applied to each group using the following code:</p><pre>#use lmplot to plot scatter points<br>sns.set(font_scale = 1.6)<br>sns.set_style(&quot;white&quot;)<br>fig4 = sns.lmplot(data = flights_data, x=&quot;year&quot;, y=&quot;passengers&quot;, height=5, aspect=1.64, hue=&quot;group&quot;, palette=[&quot;red&quot;,&quot;blue&quot;], legend = False)</pre><pre>#change axis titles<br>fig4.set(xlabel=&#39;Year of flight&#39;, ylabel=&#39;Passengers&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*HVOkwwoIgfFBAWacdEu1Qg.png" /><figcaption>Seaborn scatter plot with filled trendline using hue. Image by the author.</figcaption></figure><p>In all cases, to simplify the data the mean (or median) of all y-values for each corresponding x-value can be displayed as shown below:</p><pre>#set fontsize<br>sns.set(font_scale = 1.6)<br>sns.set_style(&quot;white&quot;)</pre><pre>fig5 = sns.regplot(data = flights_data, x=&quot;year&quot;, y=&quot;passengers&quot;, x_estimator=np.mean, color = &quot;black&quot;)</pre><pre>#set fig size<br>fig5.figure.set_size_inches(8.2, 4.5)</pre><pre>#change axis titles<br>fig5.set(xlabel=&#39;Year of flight&#39;, ylabel=&#39;Passengers&#39;) </pre><pre>#despine<br>sns.despine()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*g6y2XTcSYtCaLEkbVBNAXw.png" /><figcaption>Seaborn scatter plot with filled trendline and x_estimator = np.mean. Image by the author.</figcaption></figure><h3>Ridgeline plots</h3><p>It is easy to see looking at the line and scatter plots that data distribution within each group is significantly varied. <em>Joyplots or ridgeline plots</em> in Python can be an excellent method to visualizing said distribution while providing insight into distribution types (e.g. normal, gamma, beta). For a refresher, an excellent summary of these can be found <a href="https://medium.com/@srowen/common-probability-distributions-347e6b945ce4"><strong>here</strong></a>.</p><p>One library I particularly enjoy to visualize joyplots or ridgeline plots in Python is — joypy. It boils down to a single function call, making it easy to use and alter.</p><pre># x variables to plot with corresponding text<br>xvar = [&#39;year&#39;, &#39;passengers&#39;]<br>xlabel = [&#39;Year of flight&#39;, &#39;Passengers&#39;]</pre><pre>for x,xlabel in zip(xvar,xlabel):<br>    joyplot(data = flights_data[[x, &quot;group&quot;]], <br>            by=&quot;group&quot;, figsize=(10, 4), <br>            labels = [&quot;Group 1&quot;,&quot;Group 2&quot;],<br>            linecolor = &#39;black&#39;,<br>            color = [&#39;red&#39;,&#39;blue&#39;], <br>            alpha = 0.85, <br>            linewidth=3, overlap = 0.9)<br>    plt.rc(&quot;font&quot;, size=24)<br>    plt.xlabel(xlabel, fontsize = 24)<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*rwfg3Tex3VpElqkL0zfOKA.png" /><figcaption>Ridgeline plot example for groups 1 and 2. Image by the author.</figcaption></figure><p>Of course, this is a simplified ridge-line plot for the two arbitrary groups and the amount you can do with this library is up to the user. Additional documentation can be found <a href="https://towardsdatascience.com/ridgeline-plots-the-perfect-way-to-visualize-data-distributions-with-python-de99a5493052"><strong>here</strong></a> with further examples for inspiration.</p><h3>Heatmaps</h3><p>Heatmaps can provide a simplified summary of your dataset with a color coded overlay. To achieve this, the seaborn library provides a variety of options for heatmap plotting.</p><p>For this example, I have grouped the dataset by “year” and produced both mean and median outputs into one concatenated dataframe. This enabled me to simplify the dataset before heatmap generation:</p><pre>#grouped flights_data and calculated mean<br>grouped_df = flights_data.groupby(&quot;year&quot;)<br>mean_df = grouped_df.mean()<br>mean_df.rename({&#39;passengers&#39;: &#39;Mean&#39;}, axis=1, inplace=True)<br>mean_df = mean_df.reset_index()</pre><pre>#grouped flights_data and calculated median<br>median_df = grouped_df.median()<br>median_df.rename({&#39;passengers&#39;: &#39;Median&#39;}, axis=1, inplace=True)<br>median_df = median_df.reset_index()</pre><pre>#concat dataframes<br>data_concat = pd.concat([median_df[&#39;Median&#39;], mean_df[&#39;Mean&#39;]], axis=1, ignore_index=False)</pre><pre>#display <br>data_concat.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*gE_SytFTDF2gOcW4lueNpA.png" /><figcaption>data_concat dataframe. Image by the author.</figcaption></figure><p>Once complete, you can plot your heatmap for each year:</p><pre>#finding y-axis titles<br>y_axis_labels = flights_data.year.unique()</pre><pre># plot heatmap<br>fig, ax = plt.subplots(figsize=(6,10)) <br>fig7 = sns.heatmap(data_concat, cmap=&quot;vlag&quot;, linewidth=0.1, annot=data_concat, fmt=&quot;.2f&quot;, cbar_kws={&#39;label&#39;: &#39;Passengers&#39;, &quot;shrink&quot;: .4}, annot_kws={&quot;size&quot;: 16}, yticklabels=y_axis_labels)</pre><pre># set yticks and axis label<br>plt.yticks(rotation=0) <br>fig7.set(ylabel=&#39;Year of flight&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*mh3C27e6O5_hcunvaS-hhA.png" /><figcaption>Heatmap example using seaborn library. Different columns correspond to median and mean values for each grouped year of flight. Image by the author.</figcaption></figure><h3>Conclusion</h3><p>Of course, the field of data science has a large variety of tools available for data visualization which are endlessly improving and changing. However, keeping up to date and navigating this for anyone new to the Python language can be a somewhat overwhelming task.</p><p><strong>To summarize</strong> — key libraries such as seaborn joypy and plotly.expressoffer an effective starting point for anyone new to data visualization with Python. In this article we utilized these to generate scatter plots, heat maps, line graphs and joy plots with a simple dataset.</p><p>I hope that you enjoyed reading this article as much as I enjoyed writing it.</p><p>In upcoming articles I will being to dive into machine-learning and deep-learning theory, along with further data visualization techniques.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73402178811b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/top-python-libraries-for-visualization-a-starting-guide-73402178811b">Top Python Libraries for Visualization: A Starting Guide</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>