<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[CUNY CSI MTH513 - Medium]]></title>
        <description><![CDATA[Course blog posts for MTH513 — Machine Learning - Medium]]></description>
        <link>https://medium.com/cuny-csi-mth513?source=rss----7a3317a2a9c3---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>CUNY CSI MTH513 - Medium</title>
            <link>https://medium.com/cuny-csi-mth513?source=rss----7a3317a2a9c3---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:34:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/cuny-csi-mth513" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Machine Learning In Large Dimensions (Draft)]]></title>
            <link>https://medium.com/cuny-csi-mth513/machine-learning-in-large-dimensions-draft-d77ff0467f6e?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/d77ff0467f6e</guid>
            <dc:creator><![CDATA[Savion Watson]]></dc:creator>
            <pubDate>Mon, 23 May 2022 13:45:47 GMT</pubDate>
            <atom:updated>2023-04-01T01:34:53.846Z</atom:updated>
            <content:encoded><![CDATA[<h3>Studying Bacteria with Tabular Data</h3><h3>Introduction</h3><p>While machine learning is associated with calculations and analysis for very technical fields such as artificial intelligence or engineering, it can also be used to directly aid in biological studies. For example, a recent study led by Ryan Wood demonstrated how computer simulations were able to accurately predict antibiotic-resistant bacteria from lossy, compressed data of many bacteria species.</p><p><a href="https://www.frontiersin.org/articles/10.3389/fmicb.2020.00257/full">Analysis of Identification Method for Bacterial Species and Antibiotic Resistance Genes Using Optical Data From DNA Oligomers</a></p><p>With the world’s population becoming more resistant to antibiotics, identifying which bacteria contribute to this could allow scientists to develop more effective medicine. As a means of testing Wood’s methods, and improving our machine learning skills, our research team (me, <a href="https://medium.com/@Andi_K"><strong>Andi Kolari</strong></a> and <a href="https://medium.com/@robust-ness"><strong>Adam Bougaev</strong></a>) competed in an online competition on <strong>Kaggle.com</strong>, a data science website. Through the <strong>Tabular Playground Competition</strong>, we applied our knowledge of machine learning using <em>Scikit-learn</em> and <em>Pandas </em>to identify bacteria species based on Wood’s data.</p><p><a href="https://www.kaggle.com/competitions/tabular-playground-series-feb-2022">Tabular Playground Series - Feb 2022</a></p><h3><strong>Competition Background</strong></h3><p>As mentioned, the data from Wood’s study was lossy. This meant that only 10-mer snippets of DNA were sampled and used to identify ten other species of bacteria. The snippets create a vague histogram or sequence, without a specific order to their molecules. The molecules can only be 10-mers long, and there are four base molecules: Adenine “<strong>A</strong>”, Cytosine “<strong>C</strong>”, Guanine “<strong>G</strong>”, and Thymine “<strong>T</strong>”. Thus, there were <strong>286 </strong>possible<strong> </strong>different sequence combinations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*pwJXT2A9utnNUplaGeakSw.png" /><figcaption>Combination calculation, where order doesn’t matter and repeats are allowed.</figcaption></figure><p>Our initial data consisted of a testing set (<em>test.csv</em>), a training set (<em>train.csv</em>), and a sample submission (<em>sample_submission.csv</em>) to Kaggle. Each of the possible combinations of molecules were columns in both <em>test.csv</em> and <em>train.csv.</em> Every combination had a measurement error relative to a DNA sample. The bacteria species themselves were identified with a column labeled “<em>target</em>” in the training set. There were <strong>10</strong> possible species:</p><ul><li><em>Streptococcus Pyogenes</em></li><li><em>Streptococcus Pneumoniae</em></li><li><em>Salmonella Enterica</em></li><li><em>Enterococcus Hirae</em></li><li><em>Escherichia Coli</em></li><li><em>Escherichia Fergusonii</em></li><li><em>Campylobacter Jejuni</em></li><li><em>Bacteroides Fragilis</em></li><li><em>Klebsiella Pneumonia</em></li></ul><p>The DNA samples had identification numbers on each row, with the <em>sample_submission.csv</em> consisting of just these rows and the “<em>target”</em> column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*74G0fhiQ43xQx7EKQcj-Yg.png" /><figcaption>Truncated Sample of Tabular Dataset</figcaption></figure><h3>Methods and Research Process</h3><p>Up to this point in our Machine Learning class, our team was not particularly well versed with categorical data. Our only prior competition was <em>Titanic — Machine Learning from Disaster</em> in which we were tasked with simply predicting which passengers would have survived the infamous sinking of the Titanic. Since we had only four features to work with, and most features were categorical, we were able to run a simple <strong>Random Forest Classifier</strong>.</p><p>However, the Tabular Playground Competition had many more categories than the Titanic dataset, each with a small numerical rate. All of the categories were also used to predict one of ten possible outcomes, not just “<em>Survived</em>” or “<em>Did Not Survive</em>”. Every possible combination of molecules was undoubtedly a factor in determining species, but the sheer amount of them created a large dimension that made the problem intimidating to approach. Our biggest challenge was finding efficient ways to accurately analyze such a large dataset.</p><h3>Modeling, and Transforming</h3><h4>One-Hot Encoding</h4><p>Initially, we tried to use <strong>One-Hot Encoding</strong> to transform our model before testing. One-Hot Encoding is the act of creating new columns to show presence of specific categories. It works by taking categories of no particular ranking, or <strong>nominal variables,</strong> and simplifying them with binary indications of presence. We were inspired to use this after skimming through lessons on the “<em>Intermediate Machine Learning” </em>course Kaggle offers.</p><p>Since the columns were simply different combinations and we wanted and easier way to digest the measurement errors, this seemed logical. However, as we were implementing, we began to realize how problematic it would be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mad_x3W8CDkdM0tj" /><figcaption>Code implementation</figcaption></figure><p>To start, ironically, our large spread made difficult for One-Hot Encoding to be used effectively. When taking a more critical look at our Kaggle course during implementation, we were suggested to limit our categories to just 15, even if they were all nominal.</p><p>Furthermore, even if we did have less categories, One Hot Encoder seemed work off of <strong>binary</strong> categories. Simply rounding each measurement error to 1 or 0 would eliminate the needed nuance of the dataset, since the they acted as a form of presence for each type of combination in each sample. Multiple species could have more of one combination than others, so a binary value for the presence wouldn’t make sense.</p><p>Going forward, we decided to leave the initial data alone and simply find a working model to elaborate on.</p><h4>K-Nearest Neighbors</h4><p>Next, we looked into <strong>K-Nearest Neighbors</strong>, since it was a method we had recently learned at the time. K-Nearest Neighbors functions by measuring the distance between similar data points and making decisions based on a specified maximum proximity of possible points. Since we were trying to identify one DNA species for each sample, we felt that it was appropriate to use a K-Nearest Neighbors classifier.</p><p>This was also problematic as although the data would ultimately be discrete-<em>one sample could not identify with more than one DNA species</em>- our inputs were once again too vast. Every additional input on K-Nearest Neighbors increases the number of dimensions needed to measure distance. The dataset had a whopping <strong><em>286</em></strong> unique strands to analyze, and since all the strands needed to be analyzed in tandem for each sample, it would not be ideal to use only one strand at a time for each classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/0*OGios6ot54nAUXzi" /><figcaption>Code Implementation</figcaption></figure><p>This unfortunate property we’ve been struggling with is known colloquially as the <strong><em>Curse of Dimensionality</em></strong>, where high numbers of dimensions create even more axes for neighboring points to be measured. If these points cannot be found adequately in every dimension, K-Nearest Neighbors will not function properly.</p><h4>Random Forest Classifier</h4><p>Ultimately, we found that <strong>Random Forest Classification</strong> was the most appropriate model to use. Random Forest Classification is an extension of <strong>Decision Tree Classifiers </strong>which create tree diagrams that branch from possible decisions to ultimately reach a conclusion. Having too many branches lead to overfitting, where the model is too specific to our dataset, and too little branches lead to underfitting, where the model is too general to apply to any dataset.</p><p>Given how many dimensions he had to to work with, simply using a Decision Tree with many leaves could lead to this, but, Random Forest Classifier accounts for this by taking the <strong>majority </strong>results of multiple trees. Despite initially testing with forests (or models) using 100 trees, we found the best values from just two trees (<em>n_estimators=2</em>), likely due to the sheer amount of leaves each tree would have. This finally led us to an accuracy score of <strong>.71457</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*LBhBUVYbutAZMa9w" /><figcaption>Code Implementation</figcaption></figure><p>It became clear that the <strong><em>Curse of Dimensionality</em></strong> was proving to be quite real, and making it difficult to approach our challenge, much less improve our strategies.</p><p>If Ryan Wood and his team were able to draw important conclusions by simplifying parts of their data, we could likely do the same. With some help from our professor, we successfully reverse-engineered Wood’s compression process and extracted the most relevant information from the original data, without actually changing it.</p><h4>Data Reconstruction and Principal Component Analysis</h4><p>If just putting in our raw dataset was too much, yet no details could be omitted, we had to change the way we <strong>interpreted</strong> our data.</p><p>First, we decided to retrieve the more precise data from Wood’s experiment. While the challenge was to work with the lossy data, none of the rule prohibited reverse engineering, and since we worked with the lossy data anyway to approximate the original instead of using another file, it didn’t feel like cheating.</p><p>In the original study, <strong>bias </strong>values were subtracted to simplify the data. After dropping duplicates values, the biases were found once more by applying a <strong>multinomial distribution</strong> to each combination column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*HkdjB3EQ4UIBSL9NLtmmVw.png" /><figcaption>Duplicate Dropping</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*7HeAzO5RWSXblAjXO1Dggw.png" /><figcaption>Multinomial Distribution Formula</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cY_PPkBfMhYyVpSNxwF6zQ.png" /><figcaption>Formula Implementation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*oz8jp7JIdTrI-4Fsi36jTQ.png" /><figcaption>Output snippet of biases</figcaption></figure><p>These biases were subtracted from the original data measurements, and divided by 10k in the experiment. So we added them back and re-scaled the data. The original paper also notes that each sample took a different number of trials to reproduce. With this in mind, the more trials needed to reach a sample, the less <strong><em>variance</em> </strong>its measurements has. The quantity of trials was estimated using the <strong>Greatest Common Denominator</strong> (GCD) of the measurement errors for each sample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*D3Ix8dh7_y0qJLiE_liqEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*KxEeH3QbSW5YqeWqGk952w.png" /></figure><p>To visualize our data, we used <strong>Principal Component Analysis </strong>(PCA). Principal Component Analysis works by capturing maximum variance through a<strong> linear projection map</strong>. Maximizing variance is important since it shows the most vital patterns in our data.</p><p>Since the map is only a <em>projection </em>of our data, we receive a small set of features based on our data points while still preserving its shape for proper analysis. None of the original data was actually changed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*k-ZkXuohmMSApK76PzpJKA.png" /><figcaption>PCA Implementation</figcaption></figure><p>While all the data points are important, having <strong>20,000</strong> samples each with <strong>286</strong> unique measurements creates over <strong>5 million</strong> data points. It was integral that we minimized noise and maximized variance so that the data points we do utilize gave us the best predictions possible. Doing PCA helped illuminate the <strong>positive relationship</strong> between variance and GCD with graphs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*TgY-HempTdTeat5HoMnRrg.png" /><figcaption>Low GCDs with High variance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*VKs-NPPlILGkYtqTb4NJcw.png" /><figcaption>High GCDs with Low variance</figcaption></figure><p>Finally, to calculate our score, the GCD most closely associated with each sample was applied to aid in its species prediction with an<strong> <em>Extra Trees Classifier</em></strong>. It is similar to a <strong><em>Random Forests Classifier</em></strong>, but is better suited for points with low variance, since it uses random splits instead of best splits from a random subset. Our reduced dimensions also meant we could process many more trees (<em>n_estimators</em> = 300) in a reasonable amount of time.</p><p>This yielded much more accurate results, with an excellent accuracy score of <strong>.96932</strong></p><h3>Conclusion</h3><p>Overall, when studying large datasets, it is important to understand when and how to extract the most crucial information to your task. Due the sheer size and complexity of our data, our team went from:</p><ul><li>Misinterpreting data and applying unfit models</li><li>Grasping nature of data and applying a simple model</li><li>Getting more accurate measurements (despite being challenged to work with lossy ones)</li><li>Reducing data to its most essential parts</li><li>Applying reduction to a more fit model</li></ul><p>Without pushing ourselves to truly address our task, we would have never have been able to test our skills, receive such high scores and break the <strong><em>Curse of Dimensionality</em></strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d77ff0467f6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/machine-learning-in-large-dimensions-draft-d77ff0467f6e">Machine Learning In Large Dimensions (Draft)</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classifying 10 Different Bacteria Species with Machine Learning (Kaggle: Tabular Playground Series…]]></title>
            <link>https://medium.com/cuny-csi-mth513/classifying-10-different-bacteria-species-80dd8e336f01?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/80dd8e336f01</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dna]]></category>
            <dc:creator><![CDATA[Winson Rong]]></dc:creator>
            <pubDate>Thu, 19 May 2022 21:54:00 GMT</pubDate>
            <atom:updated>2022-05-19T21:54:00.609Z</atom:updated>
            <content:encoded><![CDATA[<h3>Classifying 10 Different Bacteria Species with Machine Learning (Kaggle: Tabular Playground Series — Feb 2022) 0.97613 Accuracy</h3><h3>Intro</h3><p>The objective of this competition is to classify 10 different bacteria from DNA data using machine learning algorithms. The segments are analyzed using Raman spectroscopy and calculated in histogram form, given in a base count. Ex: A2T4G0C4, A2T4G1C3, A2T4G2C2…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TU55Cdv7YwaSp29xFhpfEw.jpeg" /></figure><p>The data itself is formed into a train and test dataset with each row containing measurement errors for 286 possible samples, from A0T0G0C10 to A10T0G0C0. The train dataset contains 200,000 rows with a shape (200000, 288) including a “row_id” and a “target” column. The test dataset contains 100,000 rows with a shape (100000, 287) including “row_id”.<br>The submission for this competition was formatted like so:</p><figure><img alt="target" src="https://cdn-images-1.medium.com/max/213/1*7jdLa_IxT4KdJYO6B0RC9g.png" /></figure><h3>Loading and viewing datasets</h3><p><strong><em>Train dataset</em></strong></p><p>First, we will introduce the environment, read the train dataset, and drop out duplicate rows to decrease the memory used to accommodate Kaggle’s kernel memory during running and training the model. Within the train and test dataset, row_id will be indexed like so.</p><pre>train_data= pd.read_csv(“../input/tabular-playground-series-feb-2022/train.csv”, index_col=”row_id”)</pre><pre>duplicates_train = train_data.duplicated().sum()<br>print(‘Duplicates in train data: <strong>{0}</strong>’.format(duplicates_train))</pre><pre>train_data.drop_duplicates(keep=’first’, inplace=True)<br>duplicates_train = train_data.duplicated().sum()</pre><pre>print(‘Train data shape:’, train_data.shape)<br>print(‘Duplicates in train data: <strong>{0}</strong>’.format(duplicates_train))</pre><p>Which will print:</p><pre>Duplicates in train data: 76007<br>Train data shape: (123993, 287)<br>Duplicates in train data: 0</pre><p>This method of dropping duplicates will decrease memory usage and increase training time, likewise it may also increase the accuracy of prediction by reducing an overfitting model.<br>Using .head() and .unique(), I was able to ascertain the 10 types of bacteria listed and see the content of the train dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*6JzZQMub6RnPGjVbHpJvrQ.png" /></figure><pre>TARGETS:<br>[&#39;Streptococcus_pyogenes&#39; &#39;Salmonella_enterica&#39; &#39;Enterococcus_hirae&#39;<br> &#39;Escherichia_coli&#39; &#39;Campylobacter_jejuni&#39; &#39;Streptococcus_pneumoniae&#39;<br> &#39;Staphylococcus_aureus&#39; &#39;Escherichia_fergusonii&#39; &#39;Bacteroides_fragilis&#39;<br> &#39;Klebsiella_pneumoniae&#39;]</pre><p><strong><em>Test dataset</em></strong></p><p>After reading the test dataset, we will move on to assignment of code variables and column drops for the model. Altering the test data would only create more inaccuracies.</p><h3>Variable assignment and column drop</h3><pre>trainX = train_data.drop(columns=[&#39;target&#39;])<br>trainY = train_data[[&#39;target&#39;]]<br>testX = test.drop(columns=[&#39;row_id&#39;])<br>testY = test[[&#39;row_id&#39;]]</pre><p>Assigning a new variable to separate predictors and the target will make it easier to comprehend. Note that testY was not used as it was only an index. Now, trainX and testX should only contain the data from the DNA samples of A0T0G0C10 to A10T0G0C0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*Fd9qZ9krIw0tC2xWEVKzvw.png" /></figure><p>Encoding the target labels will also improve training times and reduce memory.<br>encoder = LabelEncoder()<br>trainY = encoder.fit_transform(trainY)<br>Converting back to labels before submission with<br>y_pred = encoder.inverse_transform(y_pred)</p><h3>Sklearn Model: RandomForestClassifier</h3><p>My two colleagues suggested using the RandomForestClassifier model as we started the competition and scored 68.653% accuracy after the first submission.</p><pre>from sklearn.ensemble import RandomForestClassifier</pre><pre>model = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=100)<br>model.fit(trainX, trainY)<br>y_pred = model.predict(testX)</pre><p>RandomForestClassifier uses a number of decision tree classifiers on samples of the training dataset to predict the target. It has a number of parameters that can be tuned and used to improve the accuracy of the prediction. <strong>Decision trees</strong> are a type of supervised learning used to create “outcomes”, visualized to be tree-like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*URwYTmD8_3jSWsVIF__x9Q.png" /></figure><p><em>Version 1</em><br>Using n_estimators=100, max_depth=5, random_state=100 in the model. After train fitting, and predicting with the model, it is almost ready to submit to the competition for checking.<br>For submission conversion to the desired format:</p><pre>submission = pd.DataFrame({&#39;row_id&#39;: test.row_id, &#39;target&#39;: y_pred})<br>submission.to_csv(&#39;submission.csv&#39;, index=False)</pre><p>Accuracy: 0.68653 (2 min)</p><p><em>Version 2</em><br>Using n_estimators=200, max_depth=None, min_samples_split=2, min_samples_leaf=1, max_features=’auto’, max_leaf_nodes=None, min_impurity_decrease=0.0, bootstrap=True, n_jobs=-1, random_state=100, verbose=3 in the model. The limt of max_depth removed, increased n_estimators, bootstrap aggregating and parallel processing(n_jobs) for increased training speed; have all increased predictions greatly without overfitting.</p><p>Accuracy: 0.96249 (3 min)</p><h3>Ensembling (GaussianNB, RandomForest)</h3><p>Using VotingClassifier, it is possible to combine the predictions of several model algorithms to attempt to improve accuracy.</p><pre>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.naive_bayes import GaussianNB</pre><pre><em>#models</em><br>modelrf = RandomForestClassifier(n_estimators=200, max_depth=None, min_samples_split=2, min_samples_leaf=1, max_features=&#39;auto&#39;, max_leaf_nodes=None, min_impurity_decrease=0.0, bootstrap=True, n_jobs=-1, random_state=1000, verbose=4)<br>modelgaus = GaussianNB()</pre><pre>from sklearn.ensemble import VotingClassifier</pre><pre>#ensemble<br>ensemble_modelrf = VotingClassifier(estimators=[(&#39;rf&#39;, modelrf), (&#39;gnb&#39;, modelgaus)], voting=&#39;soft&#39;)<br>ensemble_modelrf.fit(trainX, trainY)<br>print(ensemble_modelrf.predict(testX))</pre><pre>ensemble_modelgaus = VotingClassifier<br>ensemble_modelgaus = VotingClassifier(estimators=[(&#39;rf&#39;, modelrf), (&#39;gnb&#39;, modelgaus)], voting=&#39;soft&#39;,<br>       flatten_transform=True)<br>ensemble_modelgaus.fit(trainX, trainY)<br>print(ensemble_modelgaus.predict(testX))</pre><p>In this ensemble, GaussianNB was combined with RandomForestClassifier predictions. VotingClassifier “soft” voting depends on how accurate each classifying model is. Although Random Forest is very promising, the addition of a weaker model brought the accuracy down, so ensemble will be not be used given the nature of decision trees.</p><p>Accuracy: 0.52340 (6 min)</p><h3>Sklearn Model: ExtraTreesClassifier</h3><p><strong><em>Stacking</em></strong></p><p>Stacking will consist of stacking output to generalize each model, and to use “final” model to find the final predictions with each of the other models’ outputs.</p><p><em>Version 1</em></p><pre>from sklearn.ensemble import ExtraTreesClassifier<br>from sklearn.ensemble import StackingClassifier</pre><pre>modelET = ExtraTreesClassifier(n_estimators=300, max_depth=None, min_samples_split=2, min_samples_leaf=1, max_features=&#39;auto&#39;, max_leaf_nodes=None, min_impurity_decrease=0.0, bootstrap=False, n_jobs=-1, random_state=1000, verbose=4, max_samples=1)</pre><pre>modelStack = StackingClassifier(estimators=[(&#39;ET1&#39;, modelET), (&#39;ET2&#39;, modelET), (&#39;ET3&#39;, modelET)], final_estimator=modelET)</pre><pre>modelStack.fit(trainX, trainY)<br>y_pred = modelStack.predict(testX)</pre><p>Using the ExtraTreesClassifier model, which will prove even better than the similar RandomForestClassifier model, it has been stacked to attempt a better prediction. The drawback is a much longer train time.</p><p>Accuracy: 0.96584 (35 min)</p><p><em>Version 2<br></em>By simply changing the number of estimators from n_estimators=300 to n_estimators=500, accuracy was slightly increased but training time has almost doubled due to the number of training models. If n_estimators was increased any higher, there would not be memory left.</p><p>Accuracy: 0.97055 (54 min)</p><p><em>Version 3</em><br>To combat this problem, I discarded stacking as it is very memory-consuming and set parameters n_estimators=3000 to fit normally on one ExtraTrees model.<br><strong>Note that it is currently my best score achieved, and took ~15 minutes, a quarter of time from the previous version.</strong></p><p>Accuracy: 0.97613 (15 min)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWdwega8aqukK7bTBy3tGQ.png" /></figure><h3>Closing notes</h3><p>With a score of 0.97526 with a previous unfinished version, my team has placed 453rd on the public leaderboard in the competition. The best score of 0.97613 was submitted after the deadline, and would have placed 432nd instead.</p><p>With more time, there might have been a better model to more accurately predict bacteria classification, and even more with hyper-parameter tuning.<br>It seems that using one model to train a larger amount of estimators will take a lot less time than the stacking method. The ensemble method would likely lower accuracy if models are different, how will it affect an ensemble of identical models? RandomForestClassifier showed great potential with a 96% accuracy with n_estimators=200 compared to the ExtraTreesClassifier which used n_estimators=3000 and took 5 times longer. Maybe this could have been further improved overfit less and form more accurate predictions.</p><h3>Citation and sources</h3><p><a href="https://www.kaggle.com/competitions/tabular-playground-series-feb-2022/overview">Kaggle competition link</a><br><a href="https://www.frontiersin.org/articles/10.3389/fmicb.2020.00257/full">Paper that inspired the competition</a><br><a href="https://www.kaggle.com/code/winsonrong/tps-feb2022-part1">Notebook part 1</a><br><a href="https://www.kaggle.com/code/winsonrong/tps-feb2022-part2">Notebook part 2</a> ←best score</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80dd8e336f01" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/classifying-10-different-bacteria-species-80dd8e336f01">Classifying 10 Different Bacteria Species with Machine Learning (Kaggle: Tabular Playground Series…</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting NYC Taxi Trip Durations]]></title>
            <link>https://medium.com/cuny-csi-mth513/predicting-nyc-taxi-trip-durations-1b186b91d853?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/1b186b91d853</guid>
            <dc:creator><![CDATA[Jonathondavidprice]]></dc:creator>
            <pubDate>Thu, 19 May 2022 21:51:29 GMT</pubDate>
            <atom:updated>2022-05-19T21:51:29.165Z</atom:updated>
            <content:encoded><![CDATA[<p>Growing up in NYC, I am no stranger to long taxi rides, whether rain or shine, day or night, and any month of the year it may be. Competing in a competition for predicting duration times was something very familiar to my teammates and I. Having been first hand witness to many a taxi ride in the past, we thought we would have some good hypothesis and ideas into which rides may take longer than others.</p><p>The first thing we did in this competition, as with any, was load in all of our data.</p><pre>train = pd.read_csv(&quot;/kaggle/input/nyc-taxi-trip-duration/train.zip&quot;)<br>test = pd.read_csv(&quot;/kaggle/input/nyc-taxi-trip-duration/test.zip&quot;)<br>weather = pd.read_csv(&quot;/kaggle/input/weather/weather_data_nyc_centralpark_2016(1).csv&quot;)</pre><p>Aside from reading in just the train and test data that was given to us, we also read in weather data. This data was for the average weather of Central Park, and gave us information which we hoped to be helpful and relevant, such as precipitation and temperature.</p><p>To begin, as with any data science project, after loading in our data, the next step was cleaning our data. We had to change all of the time data (data, day of week, pick up week, pick up hour, day of year), from how it was given to us, as an object, into ‘int’ data type, so that we could perform mathematical operations on the data. We did this with the following code:</p><pre>from datetime import datetime<br>train[&#39;date&#39;] = pd.to_datetime(train[&#39;pickup_datetime&#39;])<br>train[&#39;day_of_week&#39;] = train[&#39;date&#39;].dt.day_of_week<br>train[&#39;pickup_weekofyear&#39;] = train[&#39;date&#39;].dt.isocalendar().week<br>train[&#39;pickup_hour&#39;] = train[&#39;date&#39;].dt.hour<br>train[&#39;day_of_year&#39;] = train[&#39;date&#39;].dt.dayofyearaccidents[&#39;date&#39;] = pd.to_datetime(accidents[&#39;datetime&#39;])<br>accidents[&#39;pickup_hour&#39;] = accidents[&#39;date&#39;].dt.hour<br>accidents[&#39;day_of_year&#39;] = accidents[&#39;date&#39;].dt.dayofyear</pre><p>Next, we had to merge in our weather data to the training data, to be able to work with both sets of data.</p><pre>weather[&#39;day_of_year&#39;] = np.arange(len(weather))<br>weather[&#39;day_of_year&#39;] = weather[&#39;day_of_year&#39;] +1<br>train = pd.merge(train, weather, on = [&#39;day_of_year&#39;])<br>test = pd.merge(test, weather, on = [&#39;day_of_year&#39;])</pre><p>One of the most important things that we had to do to the data, was create distances from pickup point to drop off point. Since we were given the longitude and latitude for each point, we were able to use the Haversine Formula to determine the distances. The Haversine Formula determines the great-circle distance between two points on a sphere given their longitudes and <a href="https://en.wikipedia.org/wiki/Latitude">l</a>atitudes. The Haversine Formula is used when computing distances across the Earth’s sphere, as it takes into account the curve of the earth, and not just computing exact distance. We were able to find the Haversine distance by the following:</p><pre>from math import radians<br>from numpy import cos, sin, arcsin, sqrtdef haversine(row):<br>    lon1 = row[&#39;pickup_longitude&#39;]<br>    lat1 = row[&#39;pickup_latitude&#39;]<br>    lon2 = row[&#39;dropoff_longitude&#39;]<br>    lat2 = row[&#39;dropoff_latitude&#39;]<br>    lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2])<br>    dlon = lon2 - lon1 <br>    dlat = lat2 - lat1 <br>    a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2<br>    c = 2 * arcsin(sqrt(a)) <br>    km = 6367 * c<br>    return kmtrain[&#39;trip_distance&#39;] = train.apply(haversine, axis=1)</pre><p>As we looked at the data that was given to us, we tried to think of ways we could use the given data, to create new variables would aid in our final predictions. A few things we did was create a ‘y/n weekend’ variable, which was 1 if the day of the week was Saturday or Sunday, and a 0 if it was any other day of the week. We figured that since weekday morning driving speeds are significantly lower than weekday morning speeds, it would help to be able to distinguish whether it was a weekend or weekday.</p><p>As we can see from the below graphs, the average speed of a car resides around 8.5 mph for most of the week, and then takes a steep increase to about 11 mph by the weekend. This allows us to see that clearly cars are moving slower on average thru the week, as opposed to weekends. Also, we see that the average MPH of the vehicles is drastically lower during the hours of around 8am to 8pm, which accounts for both rush hour and daily work traffic. From looking at these two charts is why we decided to create the new variables of ‘rush_hour’ and ‘weekend’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/0*aIAGdQnjGdNsCoRw.png" /></figure><p>When it came to creating a model for this competition, we decided to go with XGBRegressor, from the xgboost package. XGB stands for extreme gradient boosting, and has become one of the most the most used methods for Machine Learning competitions and projects. XGB can be used for both regression and classification problems, but we obviously needed to use regression over classification for this problem, as we are trying to predict a numeric answer, and not just a yes or no classification.</p><p>XGB is an ensemble learning method, which is much more accurate than individual models. Ensemble models perform better as they combine multiple learning methods into one.</p><p>To create our model, we had to decide which features we wanted to use as our final predictors. After multiple tests, we settled on the following variables; day_of_week, trip_distance_mi, precipitation, pickup_hour, is_raining, weekend, &amp; rush_hour. Our final to code to run our model looked like this:</p><pre>cols = [&#39;day_of_week&#39;,&#39;trip_distance_mi&#39;, &#39;precipitation&#39;, &#39;pickup_hour&#39;,&#39;is_raining&#39;, &#39;weekend&#39;,&#39;rush_hour&#39;]<br>import xgboost as xgb<br>X_train = train[cols]<br>y_train = train[&#39;trip_duration&#39;]<br>model = xgb.XGBRegressor(objective=&quot;reg:linear&quot;, random_state=42)<br>model.fit(X_train, y_train)<br>X_test = test[cols]<br>y_pred = model.predict(X_test)</pre><p>Our final model got us a score of 0.56877.</p><p>After growing up in NYC, and taking countless taxi rides, it was interesting to compete in a competition that drew on some prior knowledge that I have grown up with. Being able to learn new ML methods to predict outcomes has been very interesting to learn and very insightful. I think competing in this competition has been very useful and I look to dive deeper into ML and learn even more with this competition as a foundation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b186b91d853" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/predicting-nyc-taxi-trip-durations-1b186b91d853">Predicting NYC Taxi Trip Durations</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Utilizing models to Predict Bacteria Species]]></title>
            <link>https://medium.com/cuny-csi-mth513/utilizing-models-to-predict-bacteria-species-a3e44954927e?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/a3e44954927e</guid>
            <dc:creator><![CDATA[David Orshansky]]></dc:creator>
            <pubDate>Thu, 19 May 2022 21:46:23 GMT</pubDate>
            <atom:updated>2022-05-19T21:46:23.393Z</atom:updated>
            <content:encoded><![CDATA[<h3>Utilizing Models to Predict Bacteria Species</h3><h4>Introduction:</h4><p>Machine learning is a method of data analysis which can automate analytical models. Using this method, the machine can train a model to accurately classify 10 different bacteria species as required by the “Tabular Playground Series — Feb2022” , a Kaggle competition. Through using supervised learning and multiple techniques such as Random Forest Classifier, imputing, and extreme gradient boosting, a model was created which classified bacteria species based on data consisting of 10-mer segments of DNA.</p><h4>Supervised Learning and Classification Overview:</h4><p>Supervised Learning is a subset of machine learning and artificial intelligence. It utilizes labeled datasets to classify or predict data. For the purpose of this project, supervised learning will be used classify the given data into which bacterial species it the data represents. To further understand how to create a model which classifies data, one must understand what classification entials. Classification is the act of sorting items or data or etc. into categories based on characteristics. For this project, we are given 10-mer snippets of DNA are sampled and analyzed to give the histogram of base count. For example, the DNA segment ATATGGCCTT becomes A2T4G2C2. Based on thousands of these segments the model is also given a label as to what bacteria it corresponds to. Being fed this training data, the model starts to notice a pattern in these segments and begins to attribute certain combinations to certain types of Bacteria. When fed in new data containing only the snippets without the label the model can then classify those snippets into what bacteria species they represent based on the patterns learned from the training data.</p><h4>Random Forest Model:</h4><p>The algorithm used to solve the problem is known as the Random Forest Classifier. It uses multiple uncorrelated decision trees to get multiple predictions and then uses the majority vote theorem classify the inputs as shown in the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*z6rZPr_d5yTM2EKJJYNw_g.png" /><figcaption>Illustrative example of the logic of the Random Forest Classifier</figcaption></figure><p>For each instance multiple decision trees are constructed. After getting a prediction from each decision tree, the algorithm selects the most common prediction and labels that dataset as that object.</p><p>Random Forest Classifier in steps:</p><ol><li>Have a number of N random records taken from a data set having K amount of records. (K&gt;N)</li><li>Individual decision trees are constructed for each sample of the input.</li><li>Each decision tree will generate a prediction.</li><li>A majority vote will decide how to classify the inputted data.</li></ol><h4>Random Forest Classifier Code :</h4><p>To begin coding the model the data must first be read in. To do this one must first call the necessary libraries numpy and pandas to read the data as shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z9FvQExSH0sxipaCnWbKFw.png" /><figcaption>Screenshot of code consisting of calling libraries and reading the test and train datasets</figcaption></figure><p>Next we will utilize the test train split procedure to be able to train and test the model. The test train split procedure is used to estimate the performance of the model by taking a data set and splitting it into two datasets. The first dataset will be used to train the model while the second one will be used to test the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vIfhcsu1SbdqAB4Q9CNsfA.png" /><figcaption>Import test train split procedure library and then dividing the data into the training and testing datasets.</figcaption></figure><p>Now we will create and fit the model using the Random Forest Classifier algorithm shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8Vw0g9XQd77Ls3wcb8aeg.png" /><figcaption>Importing library for Random Forest Classifier and creating a model. Also then fitting the model.</figcaption></figure><p>With the model created, the next step is to test the performance of the model. In the code shown below the function accuracy score is used to test the model. (Accuracy score is a popular functions which computes subset accuracy.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uk_uCOvJmNGsmQ5fKWlKzg.png" /></figure><p>With achieving an accuracy score of 0.99412 which is a really high accuracy, the next step is to submit the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3YuSQN_iTPsWvyoRyOZDQ.png" /><figcaption>Submission of model in the format required by Kaggle for the Competition</figcaption></figure><h4>XGB Classifier model:</h4><p>XGBoost model is a decision-tree-based ensemble Machine Learning algorithm that uses a gradient boosted framework. Shown below is a flow chart to better explain what XGBoosting is how it evolved from decision trees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/0*BTBLIbVJgZtI9b5r.jpeg" /><figcaption>Flow chart detailing what XGBoosting is</figcaption></figure><p>While utilizing the algorithm with an understanding that it is an evolved decision tree with extreme gradient boosting, we decided to also attempt to implement a technique known as imputing. Imputing is a technique which edits the dataset. It replaces missing data. Since the competition mentioned that the information provided is lossy, we decided to use imputation to better the results.</p><h4>XGB Classifier Code :</h4><p>To create the code the first step is to call all relevant libraries to read the data in, just like we did for Random Classifier. Next we utilize the train test split procedure to split the data set for training and testing. In addition to that we decided to impute the data since some of the given data was incomplete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5B0BIxuPoxddxqrF_WPTQQ.png" /><figcaption>Splitting the data as needed. Also imputing the data after splitting it.</figcaption></figure><p>Afterwards we called the necessary library to use an XGBClassifier algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E8YcQtLhoEC1Y99g_g1qAQ.png" /><figcaption>Creating the model using XGBClassifier algorithm</figcaption></figure><p>After modeling we then tested the performance of the model using the accuracy score function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FVobNXfgJtsDeg-ZOatSwQ.png" /><figcaption>Calculating Accuracy score of XGBClassifier predictions</figcaption></figure><p>Receiving an accuracy score of 0.98392 we decided that there was not much of a difference between the accuracy score of the XGBClassifier and Random Forest Classifier and submitted it for the competition as well.</p><h4>Conclusion:</h4><p>With the competition over and results were released, the Random Forest Classifier model had a competition accuracy score of .96578, which put the team in 595th place out of 1257 total teams. On the other hand, the XGBClassifier model only had a competition accuracy score of .90030. This result is surprising since when utilizing the given test data, the accuracy score was a .98392, about .01 lower than the Random Forest Classifier model. Overall, either model is a decent result for a novice machine learning team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3e44954927e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/utilizing-models-to-predict-bacteria-species-a3e44954927e">Utilizing models to Predict Bacteria Species</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Groups Crack At The Tabular Playground Series February Competition]]></title>
            <link>https://medium.com/cuny-csi-mth513/my-groups-crack-at-the-tabular-playground-series-february-competition-f1575a61a683?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/f1575a61a683</guid>
            <dc:creator><![CDATA[TristanCSC513]]></dc:creator>
            <pubDate>Thu, 19 May 2022 21:43:55 GMT</pubDate>
            <atom:updated>2022-05-19T21:43:55.412Z</atom:updated>
            <content:encoded><![CDATA[<h3>The beginning of the end</h3><p>Of all the classes I’ve taken at CSI this was definitely one of the most interesting. Having a open project that you would work on during the course of a month, building and iterating, watching your progress improve with a literal numerical value was both daunting and engaging to work through.</p><p>Watching the pieces come together was a fulfilling accomplishment and I’m happy that we finished with our score of 0.97188.</p><p>So, what exactly was this competition all about? Did we win a huge prize making a name for ourselves across the Kaggle landscape? Sadly…this was not the case. However, we did finish top 525 out of more then 1,255 teams, 1,312 competitors, and over 11,766 entries. Which did leave me feeling content. Moving on though, this project was all about “10 different bacteria species using data from a genomic analysis technique that has some data compression and data loss. In this technique, 10-mer snippets of DNA are sampled and analyzed to give the <em>histogram</em> of base count. In other words, the DNA segment</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/0*frd29t-xZA0eHOMS.png" /></figure><p>becomes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/0*2pg4CCo9waef5Kqq.png" /></figure><p>“ as quoted directly from the Tabular Playground Series — Feb 2020 page, and it summarizes our main goal of predicting bacteria species based on repeated lossy measurements of DNA snippets. Our project was off to a start.</p><h3>Tackling the competition: Scoring</h3><p>Our Submissions were scored/evaluated based on their categorization accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/0*sbjl2QnsxApG7sKg.png" /><figcaption>Class Slides</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/0*yLyRc18sdkY47lnO.png" /><figcaption>Google Machine Learning Crash Course</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/0*40Ng3toSjk-JkGht.png" /><figcaption>Google Machine Learning Crash Course</figcaption></figure><p>Pretty self explanatory, the better out model does at predicting pairs the higher a score we will get.</p><h3>Tackling the competition: Data/Modeling</h3><p>As said before, each row of data contains a different spectrum of histograms generated by repeating measurements of a sample. Totaling the output of the rows leaves us with 286 histogram possibilities which then has a bias spectrum(Random ATCGs) subtracted from the results. Our given data was as follows. “train.csv”, our given training set that contains the spectrum of 10-mer histograms for each sample. “test.csv”, our test set that we will use to predict the bacteria species(target) for each row_id. “sample_submission.csv”, our sample submission file in the format of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*WP2YprnxEuofwuVX.png" /><figcaption>sample_submission.csv</figcaption></figure><p>We can take a look at the first 5 rows of our train data set and gather some rudimentary information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*W_87eHCsXyBiUProkCI13w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j3hGElTFnxlQjELO.png" /></figure><p>Now let’s take a look at the test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Y-HMDSpn9r6cd8xnc3f-7g.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*27Q_8QCHNCfwxx3f.png" /></figure><p>The first noteworthy finds were that there are no null values and that of the 286 features 278 were <strong><em>continuous</em></strong> while the other 8 were<strong><em> categorical</em></strong>, meaning they have less then 25 unique values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*q1N1gXPk52_oi0Bc.png" /><figcaption>[TPS-FEB-22] 📊EDA + Modelling📈</figcaption></figure><p>Looking past the first 5 it was also noted that of the 10 different target values, they all had a somewhat equal distribution. Around ~10% for each target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*coomYXk2G0ISppud.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/0*wFRDuct5sqXxJAHe.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VEUyURiwU8R8NYjN.png" /></figure><h3>Distribution/Shape of Data</h3><p>After testing various models we achieved varied results from each, but we lacked a core concept of how the models were using the data and how the shape/distribution of our data worked well with our models. Creating different visualizations helped give us a idea of why certain models were working while others were not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*8-zbkW5EO9umo_Wr.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/0*RZ03jGVrd9rix5YK.png" /><figcaption>TPS-Feb22, EDA -&gt; Ignore-Important Cols</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/0*DME5Ra5vypqjgcwt.png" /><figcaption>TPS-Feb22, EDA -&gt; Ignore-Important Cols</figcaption></figure><h3>Tackling the Competition: Information Far and Wide</h3><p>When first starting the competition there were many different routes we could go down when it came to building our model. Linear regression, polynomial regression, ridge regression, random forest… the lists goes on and we were told to try everything and anything to see what works. This proved to be a valid strategy in understanding and exploring the competition. Using not only our finds but consulting other successful model we started off with split training data into a two different training/validation models. We trained each model on <strong>x_train</strong>, <strong>y_train</strong> and then scored each model on <strong>x-val</strong>, <strong>y_val</strong> testing a multitude of different models starting with DecisionTreeClassifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*VO2QOGASdXTkPpTJ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/0*GgTUyWYqf6QSWQvx.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/0*8AEIVmmQ6f4NUXcJ.png" /></figure><p>Something we found immediate success with in increasing our score was dropping the <strong>row_id</strong> as it wasn’t really needed/useful. Other attempts were made at any sort of optimization, such as filling nan values in test data with mean values form train data, ultimately finding that it was harming our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*73sKlHcIqjKMW-HF.png" /></figure><p>Our initial models found varied success, first using DecisionTreeClassifier we were able to achieve a max score of .97198.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/0*Myab8iAet-WNljvU.png" /></figure><p>We tried out of curiosity using other models such as PassiveAggressiveClassifier and RandomForestClassifier but our results were lower then previous models even with tuning and changing of bootstrap and estimators.</p><p><strong>Conclusions and Observations</strong></p><p>Ultimately the model that we finished our top score with was ExtraTreesClassifier. Based on the shape of the data and what we learned from class/mild research it seemed like the best option as well as the most successful of the models we used. We tuned the estimators till the project end to n_estimators=500. I believe that we had much more success with ExtraTreesClassifier based on the nature of the model, its keen ability to improve the predictive accuracy, and control over-fitting, which I believe was occurring with in other models.</p><p>Starting the semester with this project and learning the observational strengths and weaknesses of certain models, especially how we can utilize different models with the type of data/task we are working on was a great learning experience that definitely helped give me a firm grasp on machine learning while moving on with the semester.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1575a61a683" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/my-groups-crack-at-the-tabular-playground-series-february-competition-f1575a61a683">My Groups Crack At The Tabular Playground Series February Competition</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[513 Draft]]></title>
            <link>https://medium.com/cuny-csi-mth513/513-draft-b9caa2ea395a?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/b9caa2ea395a</guid>
            <dc:creator><![CDATA[AndiK]]></dc:creator>
            <pubDate>Thu, 19 May 2022 21:40:39 GMT</pubDate>
            <atom:updated>2022-05-19T21:40:39.075Z</atom:updated>
            <content:encoded><![CDATA[<h3>513 NYC Taxi Cab Trip Duration_ Andi K</h3><p><strong>Introduction:</strong><br> Machine Learning is defined by Oxford as “the use and development of computer systems that are able to learn and adapt without following specific instructions, by using algorithms and statistical models to analyze and draw inferences from patterns in data”. Prior to this semester, I had zero experience with machine learning. But over the past few months, me and my group, including <a href="http://medium.com/@savion.watson">Savion Watson</a> and <a href="http://medium.com/@robust-ness">Adam Bougaev</a> worked on three different Kaggle competitions: Tabular Playground Series, NYC Taxi Trip Duration, and Petals to the Metal. For my blog post, I will be focusing on NYC Taxi Trip Duration. The primary focus of this competition was to use linear regression models to predict how much time a taxi trip would take in New York City.</p><p><strong>Initial Attempt:</strong><br> Many of our initial attempts to work with this competition were met with some level of failure. Initially, we sought to take some of our work from the previous competition, and adapt it to work for the new competition, our thought process being that if it worked before, it would probably work again. Evidently however, we were still relatively amateurish when it came to the intricacies of machine learning and made the very embarrassing blunder of trying to make a model designed for classifiers work for a regression problem. We spent around a day being completely stuck, not understanding why we couldn’t get anything to work, before Professor Vejdemo-Johansson explained what we were doing wrong, and in the end, we decided that the best course of action would be to start from scratch. This was not our only initial blunder. From not actually importing the data, to not being able to figure out how to unzip the data, to not being able to create a shared notebook, we had quite a rocky start with this competition. Thankfully, we managed to find relatively sufficient footing later on.</p><p><strong>First Model:</strong></p><p>For our next attempt, we attempted to settle on a model we wanted to use for sufficient results, and then we would primarily focus on manipulating the data to try and improve our score. As a foundation, we used the professor’s initial notebook on the taxi cab lab titled “<a href="https://www.kaggle.com/code/michiexile/mvj-taxi-trip-durations">mvj taxi trip durations</a>”. For this model, the professor utilized cross validations as a means of assessing the validity of his model. Cross validation is a possess by which we train multiple models on different subsets of our data set. We then validate these models on on the subset that we did not use for training. If we trained five models, and we broke our data set into five subsets, we would train each model on four of the five subsets, and validate on the remaining subset, ensuring that each model is validated on a unique subset, allowing us to more effectively detect phenomenon such as overfitting, and it allows us to see which model will perform best on a completely unknown set of data. This is what was implemented in the professor’s model, allowing for the optimum model to be chosen. Additionally, the professor also implemented some calculations to ascertain additional data. The professor implemented a haversine to calculate the distance traveled on each trip, and then used this to calculate the average speed.</p><p><strong>Our Edits:</strong></p><p>We wanted to create our own model, and initially we considered three models: Standard decision tree, random forest, and extra trees. We wanted to experiment with these models because we felt we had a decent understanding of them to work on the model. We did however lift the calculations for distance and average speed, and the section where the professor excluded a number of features from the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*cHFL7_oug6rUETAsW20eBA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*boDG-yMT0i-GEP9hhA3yOw.png" /></figure><p><strong>Decision Tree Regressor:</strong><br> A decision tree is a relatively simple tool for machine learning, and while we initially tried working with it, we fully anticipated that it would not be the best option for us to use. Regardless, we wanted to test it as a means of covering all of our bases. A decision tree is a form of supervised learning in which a starting node branches off into multiple different condition nodes. Each condition node then branches off into more subsequent condition nodes, until a final result node is achieved. Below is our usage of it, in its most bare form.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*IFgEK_QBTWHD4aGkasKEQg.png" /></figure><p>This simple algorithm is not very practical for this kind of competition. Decision trees have very high variance, especially because we are dealing with so many parameters and so much data, it is highly likely that a single decision tree would overfit or underfit. We ran our model in Kaggle a few times, and the validation score fluctuated, either very close to 1, or very close to zero, a clear sign of under and over fitting. We ditched the decision tree and went forward with other models. The two we focused on were Random Forest and Extra Trees, which we worked on in parallel.</p><p><strong>Random Forest:</strong></p><p>In principle, random forest is a collection of decision trees. A collection of different trees are created through bootstrapping. Bootstrapping is a process by which random samples are taken from the source data, and then used to create multiple different predictions. In the case of a random forest, these predictions are predicted using individual decision trees, so by collecting a large number of smaller samples from the data, we can predict the distribution over the larger set. Then when validated, the random forest will take the average of the findings of each tree and return this as its output in the case of regression. If it were used for classification, then the trees would “vote” and the output would be based on the majority vote. Random forests are far more effective than single decision trees since a single decision tree is prone to such extreme variance. By taking a sample of multiple trees, we can reduce the total variance in our model. Below is the model we initially used:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*p0mkC-mvH8uB4wDFlRE93w.png" /></figure><p>The random_state parameter can be any integer, and simply functions as a seed for the random bootstrapping of samples. Max_depth denotes the maximum possible depth of the trees in the forest. When we submitted this model, Kaggle gave us a score of 0.60498. This was far better than what we were able to achieve with a single decision tree, but we now had two issues we wanted to resolve: We wanted to improve the score (there is always room for improvement), but our priority became increasing speed. It took well over an hour for us to get our score for this model, due to the sheer amount of datapoints in the data set. Because of this, it was difficult for us to make meaningful changes in our downtime, since we didn’t know how our score was changing between our trials. We wouldn’t make meaningful progress on this front until later.</p><p><strong>Extra Trees:</strong></p><p>Extra Trees is functionally a collection of decision trees, and is extremely similar to a random forest, in the sense that it allows for more accurate results as opposed to a single decision tree. Like a random forest, it creates multiple trees, and its output is the average result of those trees. However, unlike random forests, whenever the tree has to branch, the tree must analyze a set of randomly chosen features and choose the most significant one of that set. Below is the extra tree regressor we initially utilized for our competition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*AxXez010lQLD1Yxu3ApM5A.png" /></figure><p>Like random forest, the random_state parameter functions as a seed for randomization. Extra trees also has a max depth parameter, like random forest, but we didn’t initially realize, and chose to include only the random_state parameter initially. We were experimenting with extra trees at the same time we were experimenting with random forests. This particular model gave us a score of .65686. This was worse than what we were able to achieve with random forest. However, it did run substantially faster. This is because extra trees is more random. When random forests splits at a node, it is optimized to split in the most ideal fashion, while extra trees split randomly, allowing for the process to be quicker. While we were unable to achieve a score close to what we were with random forest, we felt we should focus on optimizing the faster model until we could figure out a way to significantly increase the time it took for our models to run.</p><p><strong>Optimizing Speed:</strong></p><p>Our next step was to try and increase the speed of the model. The first thing we did was read more about extra trees and random forests, when we discovered the n_jobs parameter, which controlled how many jobs would run in parallel. By setting this parameter to -1, we can run as many jobs in parallel as the processor can handle, dramatically increasing the speed of our models. Our initial random forest model took over an hour to run, and our extra trees model took around thirty minutes. By changing this parameter, the speed of both models drastically increased, as the speed of both models was practically cut in half.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*D7wNEmfhkju4IJJXZJ99mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*SQ7D4lwoqmIKNcNwx6A1qQ.png" /></figure><p>The next area we wanted to focus on was the data, as we believed that we could manipulate the data to both improve our score and the speed of our model.</p><p><strong>The Data:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*r-7uGh1v52PJ_nsM8p9hZQ.png" /></figure><p>The data provided by Kaggle includes a training set with information 1,458,644 trips, and a test set with information on 625,134 trips. Obviously, the sheer quantity of the data was a contributor to our models taking so long to run. We needed a way to decrease the amount of data, while preserving the integrity of our models. Our first thought was to start decreasing the number of features we were training the model on. However, before we got the chance to do this, another student had found a method to significantly increase the speed of their model. They had analyzed the features in the data set and realized that if they excluded all trips where store_and_fwd_flag was false in their data set, they could get a significant decrease in the time it took for their model to run, as a vast majority of the data had this feature as false, while still receiving accurate scores. Below is a graph depicting the latitude and longitude coordinates of the pickup and drop-off locations in our previous model created by Professor Vejdemo-Johansson for his original model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*8P3abA2XibJfxposKUyXxQ.png" /></figure><p>And here is the map of coordinates after we altered the training data set:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*K8jyJJ684B7tTzh3GOGx9g.png" /></figure><p>We see a very similar distribution, suggesting that even though we are using significantly less data, it still represents a similar distribution to the overall dataset. When testing this change on our random forest model, our score improved to .59017, which was not extremely substantial, but the change in time was ridiculous. The model now ran in a few minutes, which was an extreme improvement over the time it took to run before. After this, we decided to go back to our original idea of changing the features the model was trained on. To do this, we used the feature importance scores. Professor Vejdemo-Johansson already excluded the following features in his model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*I84qY0fLjdxyY6N3t9r2fA.png" /></figure><p>Below are the feature importance scores of the features that remained:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*BhJpgRJvR7hNJ6IuvWTtqA.png" /></figure><p>From these scores, we can see that vendor_id and passenger_count have lower scores compared to the rest of the features, so we decided to exclude them from the model. Doing this improved our score to .54765 which is a significant improvement over our previous score. Our original goal was to increase the speed of our model, but we had inadvertently increased the score substantially by removing features that had low feature importance scores.</p><p><strong>Adding New Data:</strong><br> For the final stretch of the competition, we settled on adding a new data set to our model. To do this, we looked at the professor’s work in his notebook “<a href="https://www.kaggle.com/code/michiexile/mvj-taxi-trip-durations-using-residual-prediction">mvj taxi trip durations using residual prediction</a>”. In this notebook, the professor added two new data sets to enhance his model: “<a href="https://www.kaggle.com/datasets/cabaki/knycmetars2016">knycmetars2016</a>” and “new-york-city-taxi-with-osrm”. For our model, we decided we would stick to just one, so we chose knycmetars2016, which has data on weather. We took the professor’s code and removed any references to the other data set. We then spliced it together with our existing model. However, upon using this data set to enhance our model, our score actually decreased to .56721, which is unfortunate. Our thought as to why this happened is that our we optimized our pre-existing model for the taxi data alone, and the weather data we added was acting more like noise, which distracted our model and did not improve it much. To try and improve the model, we had planned to potentially change the models used for interpolating the weather data, to try and optimize them further, or we would look at feature importance scores again to try and remove any superfluous features, but unfortunately, we ran out of time to try and implement these changes before the next competition began.</p><p><strong>Conclusion:</strong><br> Overall, I’d say that our group made significant progress this competition. We managed to go from not being able to get a model working, to having a relatively decent score of .54765 which we were happy with and being able to understand the changes we made and why they changed the model the way that they did.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9caa2ea395a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/513-draft-b9caa2ea395a">513 Draft</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Breakdown and Utilization of a Convolutional Neural Network]]></title>
            <link>https://medium.com/cuny-csi-mth513/breakdown-and-utilization-of-a-convolutional-neural-network-e123c4117d28?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/e123c4117d28</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gabrielkhalfin]]></dc:creator>
            <pubDate>Thu, 19 May 2022 21:37:51 GMT</pubDate>
            <atom:updated>2022-05-19T21:37:51.190Z</atom:updated>
            <content:encoded><![CDATA[<h3>Breakdown &amp;Utilization of a Convolutional Neural Network</h3><h3><strong>Introduction</strong></h3><p>In todays world of machine learning, neural networks are becoming more relevant and utilized. All current neural network technologies will have been, and will be increasingly improved upon in the future. Common applications such as, face recognition, social media, handwriting analysis and others have already began to be utilized for multiple businesses showing the outbreak that neural networks have caused.</p><p>Throughout this past semester, at the College of Staten Island, Professor <a href="https://medium.com/u/95f9235a303b"><em>Mikael Vejdemo-Johansson</em></a> assigned a thoughtful neural network task regarding classification utilizing different models as well as different optimizers. This was specifically done during the <strong><em>Petals to the Metals Competition on Kaggle</em></strong>. Earth has a vast diversity regarding the amount of living organisms. Disregarding over 5,000 species of mammals, 10,000 species of birds, 30,000 species of fish and surprisingly over 400,000 different types of flowers which is the primary focus of the <strong><em>Flower Classification on TPU competition.</em></strong></p><h3>Competition</h3><p>In this competition, as a class, we were challenged to build a machine learning model that identifies the type of flowers in a dataset of images (for simplicity, we’re sticking to just over 100 types). Throughout this challenge, we’re classifying 104 types of flowers based on their images drawn from five different public datasets. Some classes are very narrow, containing only a particular sub-type of flower (e.g. pink primroses) while other classes contain many sub-types (e.g. wild roses). The dataset contains imperfection throughout the images of flowers in odd places, and will be scored based on accuracy of the developed neural network.</p><figure><img alt="Brief showcase of images in training set" src="https://cdn-images-1.medium.com/max/675/1*jWElpB-xq_czoNEaOXxXgQ.png" /><figcaption>Preview of Flowers</figcaption></figure><h4><strong>Data</strong></h4><p>The competition offers its files in <strong>TFRecord</strong> format, a container format frequently used in Tensorflow to group and shard data data files for optimal training performance. Each file contains the id, label (the class of the sample, for training data) and img (the actual pixels in array form) information for many images.</p><p>In my particular case, during this competition I will narrow down the data to the 512, 512 pixel size. In addition to the pixel size, I also obtained the filenames provided being a training set, validation set, and a testing set.</p><pre>HEIGHT = 512<br>WIDTH = 512<br>model_path = f&#39;model_<strong>{</strong>HEIGHT<strong>}</strong>x<strong>{</strong>WIDTH<strong>}</strong>.h5&#39;<br><br>GCS_PATH = KaggleDatasets().get_gcs_path(&#39;tpu-getting-started&#39;) + f&#39;/tfrecords-jpeg-<strong>{</strong>HEIGHT<strong>}</strong>x<strong>{</strong>WIDTH<strong>}</strong>&#39;<br><br>TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + &#39;/train/*.tfrec&#39;)<br>VALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + &#39;/val/*.tfrec&#39;)<br>TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + &#39;/test/*.tfrec&#39;)</pre><p>As shown in the code above, a <strong>TPU accelerator</strong> (a hardware accelerator specialized for deep learning) is being utilized throughout the notebook. You can think of a single TPU as having 8 GPUs running in parallel to each other. We will need to tell Tensorflow how to best utilize the entire spread of 8 cores.</p><pre>try:<br>    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()<br>    print(f&quot;Running on TPU: $<strong>{</strong>tpu.master()<strong>}</strong>&quot;)<br>except <strong>ValueError</strong>:<br>    tpu = None<br>    <br>if tpu:<br>    tf.config.experimental_connect_to_cluster(tpu)<br>    tf.tpu.experimental.initialize_tpu_system(tpu)<br>    strategy = tf.distribute.experimental.TPUStrategy(tpu)<br>else:<br>    strategy = tf.distribute.get_strategy()<br>AUTO = tf.data.experimental.AUTOTUNE<br>REPLICAS = strategy.num_replicas_in_sync<br>print(replicas: <strong>{</strong>strategy.num_replicas_in_sync<strong>}</strong>&quot;)</pre><h4>Data Augment</h4><p>The initial step in this competition after obtaining the file paths, is to start augmenting the data.<strong> Data augmentation</strong> is a strategy that enables practitioners to significantly increase the diversity of data available for training models, without actually collecting new data.</p><p>So rather than obtaining new images of the flowers provided, I augmented the images by using the following:</p><pre>def data_augment(image, label):<br>    crop_size = tf.random.uniform([], int(HEIGHT*.7), HEIGHT, dtype=tf.int32)<br>        <br>    image = tf.image.random_flip_left_right(image)<br>    image = tf.image.random_flip_up_down(image)<br>    image = tf.image.random_saturation(image, lower=0, upper=2)<br><br><em>     image = tf.image.random_contrast(image, lower=.8, upper=2)</em><br><em>     image = tf.image.random_brightness(image, max_delta=.2)</em><br><br>    image = tf.image.random_crop(image, size=[crop size, crop_size, CHANNELS])<br>    image = tf.image.resize(image, size=[HEIGHT, WIDTH])<br>    image = random_black_box(image, prob_apply=0.40)<br><br>    return image, label</pre><p>There were numerous different options and ways to augment the images; however, for my particular case I mainly utilized changing the orientation, saturation, contrast, brightness, image size, and applying a black box.</p><figure><img alt="Showcase of data augmented images" src="https://cdn-images-1.medium.com/max/378/1*fFir5UKXncGxW9182xv2Ew.png" /><figcaption>Preview of augmented data</figcaption></figure><p>Shown above is a small preview of the flowers after data augmentation showcasing the changed images such as the black box designed to stop possible habits of the neural network of focusing on small specific details that may be created.</p><h4>Creating the Model</h4><p>Throughout the competition, multiple models were created each providing different results. When my group first started the competition, we mainly focused on utilizing the <strong>Adam</strong> model optimizer. We initially picked out the Adam optimizer randomly; however, once diving deep into adam (an adaptive momentum estimation), we realized that since Adam tracks an exponentially decaying average of past gradients as a more adaptive momentum it may not be a poor choice for this particular case.</p><figure><img alt="GIF of multiple different optemizers showcasing certain speed and tendencies." src="https://cdn-images-1.medium.com/max/620/1*XVFmo9NxLnwDr3SxzKy-rA.gif" /></figure><p>Adam is very similar to RMSprop combined with a momentum term, and with this knowledge for the majority of this competition Adam was our main optimizer.</p><pre>model.compile(<br>    optimizer=&#39;adam&#39;,<br>    loss=&#39;sparse_categorical_crossentropy&#39;,<br>    metrics=[&#39;sparse_categorical_accuracy&#39;]<br>)</pre><pre>EPOCHS = 16<br>STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE<br><br>history = model.fit(<br>    ds_train,<br>    validation_data=ds_val,<br>    epochs=EPOCHS,<br>    steps_per_epoch=STEPS_PER_EPOCH,<br>    callbacks=[]<br>)</pre><p>In our very first submission utilizing even less data augmentation shown prior, 22 <strong>epochs</strong> (The number of epochs is a hyperparameter that defines the number times that the learning algorithm will work through the entire training dataset), and the model shown above, we received an astonishing score of 0.16485 in 11 minutes and 13 seconds with a noticeable and embarrassing amount of overfitting.</p><figure><img alt="Graph showcasing Overfitting" src="https://cdn-images-1.medium.com/max/746/1*5nwlUm-uO1t4Ot18iPWt5Q.png" /></figure><p>Shortly after, the adam optimizer, an introduction to <strong>tenserflow keras sequential model</strong> which we began to utilize. In our particular case, we stayed with the sequential model for the entire project since we did not need the more flexible, multiple-input, multiple-output or shared-layer networks options that a functional model may include. In addition to this, we did not add many parameters to the model and were mainly prioritizing functionality rather than score.</p><p>Furthermore, the realization of <strong>transfer learning</strong>, stated in the <em>Handbook of Research on Machine Learning Applications</em> as the improvement of learning in a new task through the transfer of knowledge from a related task that has already been learned. In our specific case, we narrowed down to EfficientNet since it’s a<strong> “</strong>convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth/width/resolution using a compound coefficient” as stated in “<em>EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”.</em></p><p>In addition to EfficientNet, <a href="http://www.image-net.org/">ImageNet</a> is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers.</p><p>The architecture depicted below is VGG16.</p><figure><img alt="ImageNet breakdown" src="https://cdn-images-1.medium.com/max/932/1*yYoE3K43QqMOiw424b4F6A.png" /></figure><pre>def create_model(input_shape, N_CLASSES):<br>    base_model = EfficientNetB6(weights=&#39;/kaggle/input/efficientnet/efficientnet-b6_noisy-student_notop.h5&#39;, <br>                                    include_top=False,<br>                                    input_shape=input_shape)<br><br>    base_model.trainable = False <em># Freeze layers</em><br>    model = tf.keras.Sequential([<br>        base_model,<br>        L.GlobalAveragePooling2D(),<br>        L.Dense(N_CLASSES, activation=&#39;softmax&#39;)<br>    ])<br>    <br>    return model</pre><pre>with strategy.scope():<br>    pretrained_model = tf.keras.applications.VGG16(<br>        include_top=False,<br>        weights=&quot;imagenet&quot;,<br>        input_tensor=None,<br>        <em>#input_shape=None,</em><br>        pooling=None,<br>        classes=1000,<br>        classifier_activation=&quot;softmax&quot;,<br>        input_shape=[*IMAGE_SIZE, 3]<br>    )<br>    pretrained_model.trainable = False<br>    <br>    model = tf.keras.Sequential([<br>        pretrained_model,<br>        tf.keras.layers.Conv2D(256,4, activation=&#39;relu&#39;, input_shape=IMAGE_SIZE),<br>        <em>#tf.keras.layers.Conv2D(128,2, activation=&#39;relu&#39;, input_shape=IMAGE_SIZE),</em><br>        tf.keras.layers.GlobalAveragePooling2D(),<br>        tf.keras.layers.Dense(len(CLASSES), activation=&#39;softmax&#39;)<br>    ])<br>    model.compile(<br>        optimizer=&#39;adam&#39;,<br>        loss = &#39;sparse_categorical_crossentropy&#39;,<br>        metrics=[&#39;sparse_categorical_accuracy&#39;],<br>    )<br><br>model.summary()</pre><p>As shown in the codeblock above, we are still making use of the adam optimizer and we also utilize VGG16 focused on noisy student while also having a a larger strategy scope</p><p>On this submission, while still using 22 epochs, we see a significant increase towards our public score by approximately double the amount. Our new Public score was an impressive (in our eyes) 0.69811 running in 23 minutes and 43 seconds with the following train/validation loss as well as the sparse categorical accuracy of the train and validation set. As you can see below, the training and validation appear to be very close together, a good indication of a meagerly overfitting.</p><figure><img alt="Overfit graph" src="https://cdn-images-1.medium.com/max/831/1*IyIUT1QK-AETB_n6t4nSfg.png" /></figure><h4>Final Changes</h4><p>At this point, the group and I were on cloud 9 thinking to ourselves we have outdone our peers and have cracked the top 100 leaderboards at that specific moment; however, Professor <a href="https://medium.com/u/95f9235a303b">Mikael Vejdemo-Johansson</a> had to intervene. The following week, he showcased his new score of roughly 0.88 leading to even more machine learning to experiment on.</p><p>In our first iteration of edits, we tried playing around with different optimizers such as Adadelta, and <strong>Nadam</strong>. In our initial simulation using Adadelta, we received an amazing score of 0.00162 running in 23 minutes and 48 seconds… Not quite what we were expecting; however, once we deployed Nadam for the first time, multiple error notifications appeared. The group and I were a little too hopeful and such a quick fix. Thankfully, shortly after, we finished using Nadam successfully and received a significant increase once again to a 0.74387 with a 40 minute and 1 second runtime.</p><pre>model = tf.keras.Sequential([<br>        <br>        pretrained_model,<br>        tf.keras.layers.Conv2D(256,4, activation=&#39;relu&#39;, input_shape=IMAGE_SIZE),<br>        <em>tf.keras.layers.Conv2D(128,2, activation=&#39;relu&#39;, input_shape=IMAGE_SIZE),</em><br>        tf.keras.layers.GlobalAveragePooling2D(),<br>        tf.keras.layers.Dense(len(CLASSES), activation=&#39;softmax&#39;)<br>    ])<br>    model.compile(<br>        optimizer=&#39;Nadam&#39;,<br>        loss = &#39;sparse_categorical_crossentropy&#39;,<br>        metrics=[&#39;sparse_categorical_accuracy&#39;],<br>    )<br><br>model.summary()</pre><p>Subsequently, our final push towards our last run Professor <a href="https://medium.com/u/95f9235a303b">Mikael Vejdemo-Johansson</a> introduced utilizing save options as well as <strong>callbacks</strong>. A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training.</p><pre>local_save_options = tf.saved_model.SaveOptions(experimental_io_device=&#39;/job:localhost&#39;)<br>lr_callback = keras.callbacks.LearningRateScheduler(lrfn, verbose=True)<br>cp_callback = keras.callbacks.ModelCheckpoint(<br>        &quot;best_model.hdf5&quot;,<br>        monitor=&quot;val_sparse_categorical_accuracy&quot;,<br>        save_best_only=True,<br>        verbose=1,<br>        options=local_save_options<br>    )</pre><p>After instantiating the save options, we then applied different models by appending and utilizing the lr &amp; cp callback under 64 epochs.</p><pre>EPOCHS = 64<br>histories.append(model.fit(<br>    ds_train,<br>    validation_data=ds_val,<br>    epochs=EPOCHS,<br>    steps_per_epoch=STEPS_PER_EPOCH,<br>    callbacks=[lr_callback, cp_callback]))<br>model.load_weights(&quot;best_model.hdf5&quot;)</pre><p>Above and below are code blocks for the different models that are appended together utilizing the callbacks. In the first model, we are calling the lr_callback as well as the cp_callback while the next model below, the only callback we need is the lr_callback.</p><pre>EPOCHS = 64<br>enb7.trainable = True<br>histories.append(model.fit(<br>    ds_train,<br>    validation_data=ds_val,<br>    epochs=EPOCHS,<br>    steps_per_epoch=STEPS_PER_EPOCH,<br>    callbacks=[lr_callback]))</pre><p>With our final model &amp; submission, we received a score of 0.85647 out of 1 which is ranked #54 on the leaderboards. The model ran in 1 hour 32 minutes and 42 seconds.</p><h3>Final Conclusion</h3><p>Overall, throughout this entire competition, it became apparent of the multiple different options a programmer has when creating a neural network. It’s important to not only factor in the thought of augmenting the data, but also the model of choice and perhaps the use of different optimizers. In our case, the nadam optimizer appeared to be the best pick; however, prior to noticing that multiple other augmentations were made into the data.</p><p>We arrived into this competition with no knowledge on neural networks; however, the similarities between other machine learning features and classifiers used in prior competitions was greatly helpful in the migration towards Tenserflow and Keras.</p><p>With a final score of 0.85467, the group and I are proud of ourselves and deem this a strong score sitting in the upper echelon of the leaderboards.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e123c4117d28" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/breakdown-and-utilization-of-a-convolutional-neural-network-e123c4117d28">Breakdown and Utilization of a Convolutional Neural Network</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classification of Flowers, Introduction to TPUs]]></title>
            <link>https://medium.com/cuny-csi-mth513/classification-of-flowers-introduction-to-tpus-83c3cd1c51ea?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/83c3cd1c51ea</guid>
            <dc:creator><![CDATA[Dylan Contractor]]></dc:creator>
            <pubDate>Thu, 19 May 2022 21:35:26 GMT</pubDate>
            <atom:updated>2022-05-19T21:35:26.002Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Competition</h3><p>In this Kaggle competition, we are given 104 different classes of flowers with images to classify as one of them. The competition is meant to give people motivation to use learn and use TPUs. The competition is a getting started competition which means that scores would be deleted after 3 months and there are no prizes, meaning learning how to use TPUs is the main goal of it.</p><h3>The Queue</h3><p>One of the challenges of this competition is patience. TPUs are limited on Kaggle and there are times when they “popular”, in Kaggle’s words. The result, you are put in queue until a TPU is freed up. I would arrive to class early and so I would be the first in queue in the class, of course there would still be people in front of me, but I minimized them by arriving early. Of course this resulted in me having downtime, which I filled with a Super Punch Out Rom I found on a site. Submitting a notebook for scoring would put it into the back of queue, so I would only submit after class was over, so by next class it would finally have run.</p><h3>Learning</h3><p>Kaggle’s queue is not eternal, despite feeling as such, so I was able to actually learn how to use a TPU, by copying and pasting code that Professor @<a href="https://medium.com/@michiexile?source=collection_editor_card">Mikael Vejdemo-Johansson</a> gave(thank you). To identify the images, we trained a Neural Network(NN); more specifically, we used Tensorflow and Keras to create layers that would convert images into matrices and had our program learn from those matrices. At first, we used a combination of convolution, pooling, and a dense layer to train our method. We were also introduced to data augmentation, where we would add augmentations to the images to reduce overfitting. Soon we would learn the most important lesson for this competition: cheating; meaning, using a pretrained NN and adding our own layers to it.</p><h3>Training</h3><p>As stated, the first few used convolution and pooling layers to train the model. This didn’t last long and cheating proved much more viable. As a result I only have a score of 0.28308 to share for this time.<br>The starting pretrained Neural Network chosen was VGG16, which gave me a score of 0.30463, a tiny improvement, improvement enough to convince me. In this process I was also able to boost the score by using the tf.keras.Sequential function.</p><p>By my next submission quite a bit had changed, and I had one of my first experience of how much data augmentation can hurt or help a program. In the given code from the professor, the contrast of the images was set between 0.3–0.5, which I boosted to 0.9–1.1, this change boosted my validation score significantly. As time would go on, I would reduce my data augmentations, as they proved detrimental to learning and would barely aid overfitting (at least the augmentation I used would). The augmentation experimented with were brightness, contrast, hue, saturation, flip right, flip down, and cropping, most of which only were a detriment.</p><p>In between this I would change the Optimizer used for learning, by trail and error, I settled on Adam. Trial and error is how I would continue changing my model.</p><p>The image size also proved important. Sizes 224 and 331 proved better for scores, and size 224 was chosen as it would take less time to train on.</p><p>Also proved by the professor at this stage was a formula for learning step and adjusting it while the model was running, which I copied.<br>A combination of these would result in a score of 0.60189 with the model below:</p><pre>with strategy.scope():<br>    base_model = vgg16.VGG16(<br>    include_top=False,  <em># put in my own classifier layers</em><br>    weights=&#39;imagenet&#39;, <em># use pre-trained weights</em><br>    <em>#pooling=&#39;avg&#39;,      # flatten with global average pooling</em><br>    input_shape=[*IMAGE_SIZE, 3] <em># use our input images</em><br>    )<br>    base_model.trainable = False<br>    model = tf.keras.Sequential([<br>        base_model,<br>        keras.layers.GlobalAveragePooling2D(),<br>        keras.layers.Dense(2048, activation=&#39;relu&#39;),<br>        keras.layers.Dropout(0.2),<br>        keras.layers.Dense(1024, activation=&#39;relu&#39;),<br>        keras.layers.Dropout(0.5),<br>        keras.layers.Dense(512, activation=&#39;relu&#39;),<br>        keras.layers.Dropout(0.5),<br>        tf.keras.layers.Dense(len(CLASSES), activation=&#39;softmax&#39;)<br>    ])</pre><p>Most changes from this model would come in the form of just changing values in it. Such as the units in the dense layer, the amount of dropout, and the activation function.</p><p>At some point, I learned an important lesson in cheating, it matters who you cheat off of. VGG16 did help boost the validation score, but it was time to explore the other pretrained models. The models gave different scores, one even dropped my score to below 0.1, but I finally settled on DenseNet201. This allowed my score to jump up to 0.84363 when my previous high was 0.63545. However, in my testing I used the already established layers I had written, so that may have affected the scores I had achieved.</p><h3>My highest scoring model:</h3><p>After this I experimented a bit more, reapplied an image size of 331 and removed a few layers and arrived at this model with a score of 0.87299.</p><p>After this I looked at other competition high scoring submissions and they used EfficientNetB7 with “noisy-student” as its pretrained weights, so I transitioned to using them and applied cropping once more. To get a small improvement of 0.87589.</p><pre>with strategy.scope():<br>    base_model =EfficientNetB7(<br>    include_top=False,<br>    weights=&#39;noisy-student&#39;,<br>    input_shape=[*IMAGE_SIZE, 3]<br>    )<br>    base_model.trainable = False<br>    model = tf.keras.Sequential([<br>        base_model,<br>        keras.layers.GlobalAveragePooling2D(),<br>        keras.layers.Dense(2048, activation=&#39;relu&#39;),<br>        keras.layers.Dropout(0.2),<br>        keras.layers.Dense(1024, activation=&#39;relu&#39;),<br>        keras.layers.Dropout(0.5),<br>        tf.keras.layers.Dense(len(CLASSES), activation=&#39;softmax&#39;)<br>    ])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/0*NNEEpIwu_Mr6ks-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/0*f_eAoS-K7ojxUk-N.png" /></figure><p>With the following augments:</p><pre>def data_augment(image, label):<br>    <em># fill here with data augmentation manipulations later</em><br>    seed = rng.make_seeds(2)[0]<br>    image = tf.image.stateless_random_brightness(image, 0.25, seed)<br>    image = tf.image.stateless_random_contrast(image, 0.9, 1.1, seed)<br>    image = tf.image.stateless_random_flip_left_right(image, seed)<br>    image = tf.image.stateless_random_flip_up_down(image, seed)<br>    coeff1 = random.uniform(0.7, 0.9)<br>    coeff2 = random.uniform(0.7, 0.9)<br>    crop = [int(IMAGE_SIZE[0]*coeff1), int(IMAGE_SIZE[1]*coeff2)]<br>    image = tf.image.stateless_random_crop(image, (crop[0], crop[1],3),<br>                                          seed)<br>    image = tf.image.resize(image, [IMAGE_SIZE[0], IMAGE_SIZE[1]])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*mC3ZyQcoCb-EIvDL.png" /><figcaption>Example of flower images after augments</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83c3cd1c51ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/classification-of-flowers-introduction-to-tpus-83c3cd1c51ea">Classification of Flowers, Introduction to TPUs</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to Convolutional Neural Networks]]></title>
            <link>https://medium.com/cuny-csi-mth513/intro-to-convolutional-neural-networks-44f0e13d57e6?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/44f0e13d57e6</guid>
            <dc:creator><![CDATA[Adam Bougaev]]></dc:creator>
            <pubDate>Thu, 19 May 2022 21:32:47 GMT</pubDate>
            <atom:updated>2022-05-19T21:32:46.975Z</atom:updated>
            <content:encoded><![CDATA[<p>Before MTH 513, I had never studied or even looked that deep into machine learning or convolutional neural networks. Honestly, to me they seemed kinda scary. There was so much mysticism surrounding data science and to an outsider of the field, it seems like this crazy thing that shouldn’t even be possible. Now, having taken data science I feel like things are different. I’m just as confused as before and I’m even more scared, especially at flowers, for I will never have the computational skill to differentiate flowers like a computer can.</p><p><strong>The Story</strong></p><p>By the time we get to the topic of this blog post, project 3, we’re already 2 projects deep. My team has done a lot of work with machine learning and we’re getting confident. We’ve done it all: regressors, classifiers, random forests, extra trees, data augmentation. We were on top of the world and nobody could bring us down. Except the professor, because the next competition was about convolutional neural networks, and I was clueless. We went from boys to men, then right back to boys.</p><p>Competition: <a href="https://www.kaggle.com/competitions/tpu-getting-started">Petals to the Metal</a></p><p>Goal: Do Better, score higher, win.</p><p>Requirements: Use convolutional neural networks, and NO CHEATING.</p><p>The goal of this brand new competition is rather simple. All we need to do is produce a model that can differentiate between different types of flowers. Simple in concept. Of course, that’s before we actually get into the concepts of neural networks. I was a naive child back before the neural networks. Now, the neural networks haunt my dreams.</p><p>Anyway, to really understand what it is we were doing it’s important to explain concepts that will prove extremely valuable to beat this competition and definitely make it on the top of this tutorial example competition scoreboard without cheating and without stealing code (obviously).</p><p><strong>Neural Networks</strong></p><p>Neural networks accomplish their ability to discern patterns in data by using layers to transform the data one layer at a time. We look at our data in the form of a vector, and we apply a linear transformation followed by a non-linear function, and we do that over and over again in layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qr2VAOKbb_koxergBPgtIQ.png" /></figure><p>This chart gives an understanding of the way a neural network is structured. The green nodes are the initial data we’re feeding in, such as text or images, and the blue nodes are intermediate results. At these intermediate stages, if you were to view the results, they would be distorted and unreadable, especially at those deeper layers, but they would help you understand the patterns that the model is starting to learn. The red node is our output layer and it’s our final result, which in this case is our class of flower. The most important layer of a neural network is the dense layer that receives inputs from all previous nodes, and performs matrix-vector multiplications to get a vector output. Commonly these layers are used to change the dimensions of the vector, as seen by the chart of a neural network above in the blue node layers. In terms of conceptual understanding, we can consider these nodes like neurons in a brain and the model is almost an imitation of how neurons in our brains react to electrical signals. Our brains use these signals to strengthen connections between neurons to form memory. Neural Networks behave much like this.</p><p><strong>Convolutional Neural Networks</strong></p><p>When we went into our projects for the first time we didn’t actually just go with the basic layers of a neural network. When using the simplest layers we can, we’re actually using the whole image to discern the pattern, but that is problematic when using real-world data, because there might be certain perturbations or the actual important features of the data may be in different areas of the image. The basic neural network is very sensitive to these small changes and it can drastically worsen our scores. One way to solve this is to augment our dataset with variants of our original data, changing things like orientation, saturation, and placement, in order to account for these possibilities. But that would eventually have us creating so much more data to train with, which is just inefficient. Therefore, the convolutional layer is born! The idea of this is that instead of handling the whole image, we apply a smaller window that goes across our data, and creates outputs stored in a matrix. This layer allows us to encode abstractions about the image, and avoids the earlier problem. The other important layer that makes a CNN is the pooling layer. In this project, we also concerned ourselves with only max pooling layers. What is layer does is takes is separate the input into distinct regions and outputs the maximum of each region. This helps with both computation time and overfitting. It’s common practice to use a pooling layer after a convolutional layer, so that’s exactly what we did.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*lLFFYi1He-iu3qQWNbeKJg.png" /></figure><p><strong>Flowers</strong></p><p>According to Wikipedia “A flower, sometimes known as a bloom or blossom, is the reproductive structure found in flowering plants (plants of the division Angiospermae).” My teachers always told me never to use Wikipedia, so we can’t trust a thing they say. Honestly, it all sounds a bit fishy to me anyway. One day, I plan to write a machine learning algorithm to derive the true meaning of flowers, but until that day comes what really matters is that flowers are very distinct and colorful, so their pictures make a great dataset to train a neural network so that we can create a convolutional neural network to see how bad I am at machine learning.</p><p><strong>The Project</strong></p><p>The first part of the project was to commit a cardinal sin: I copied code. I copied the professor’s code, which is 100% cheating and no one can tell my otherwise. I still think about that day. Anyway, we used his code in order to get a basic model and to get TPU support for fast calculations. The model looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ZxXfj5g6_3AABl3odiYkA.png" /></figure><p>This model is a fairly basic CNN with all ReLU activation functions for the convolutional layers. Nothing fancy went on in our dataset. No augmentation, no nothing. It’s no surprise that we ended up getting a score of 0.27178 for this model. The model itself also entered a stage of overfitting very early on which meant that it had no potential to improve with more epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*JfwOauKl4Gwraw9dhy1Bpg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*8rSy73l69YehVtuLf6pWDg.png" /></figure><p>At the point validation loss starts to ramp up is when you know things are all going wrong. For this feeble model, 16 epochs was just way too many and ended up being mostly wasted effort. At this point we were really just playing around with the basic ideas of a model and had a few versions dedicated just to seeing how models respond at different epochs. We went way too far the first time and noticed the Goldilocks zone being at around 10 epochs. So we charted our results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*3eXdiIKtx-qldw-ogUkuuQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*9Bgz9qVdW65wcGhcC6xEwg.png" /></figure><p>Validation accuracy didn’t flatten out yet, and this iteration with the same model scored a measly 0.25160, so I knew there was a tiny bit more we could push it, but it obviously wasn’t worth it.</p><p>If we were gonna be the best machine learners, we had to do the best CNN’s of our lives. And that meant going EVEN DEEPER…</p><p>The next thing we did was experiment with different activations and optimizers. On our Convolution layers, we tried ReLU, GELU, SELU, every ELU in the dictionary, as well as the optimizers adam, adamax, and nadam, All of which are my final forms. We ended up settling for GELU on our layers, and the adam optimizer, and that got us a score of 0.31662. It’s not winning any awards but it was an improvement, and it was glorious.</p><p>But before that, we tried mixing the activation functions a little in our code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VMBISMchDGluscyR3ZK0vQ.png" /></figure><p>Notice how this iteration keeps ReLU for the first dense layer. That single difference got us 0.29809, which is worse than the all GELU 0.31662. Needless to say, we didn’t try mixing activation functions again.</p><p>After that we ended up using the forbidden technique: data augmentation. We had to, of course, so as to try and get our model to train on distortions of our original set, allowing for a more flexible model that can discern patterns even better than before. On every image, we changed the brightness and contrast of the image randomly, as well as randomly flip images horizontally and vertically. We also ended up changing our model quite a bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eGD1VBRTk0PBJBdB-35_gQ.png" /></figure><p><strong>Dropping Out of Machine Learning</strong></p><p>I’m a lazy guy. I don’t like to do a lot of work. I like things that get me better results in seconds. Dropout layers are a way of getting us better results fast. The way dropout layers work is that certain neurons will essentially be turned off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgAu-n_HOINK0d4A1midxQ.png" /></figure><p>They help us prevent overfitting and can get us higher scores.</p><p><strong>The Disaster</strong></p><p>Before we get to the final chapters, I think it’s very important to note the fateful day when I forgot to save my code. I was up at 2 AM in a fugue state, typing code out of my mind. I remember managing to get a validation accuracy of about 0.51, but not much else. See, I never actually submitted my code to Kaggle. I was just typing and running to see what would happen. Should I have submitted? Probably. Should I have been typing on a laptop with 5% battery left? Probably not. Am I stupid? Yes.</p><p>Do I regret it? Nope.</p><p><strong>Transfer Learning</strong></p><p>To get some actually good scores, I had to go to places I never should’ve gone. I had to commit the cardinal sin of using transfer learning. Transfer learning is about utilizing a pre-trained model to build a new model feeding in your new data. This allows you to transfer the knowledge gained by the pre-trained model and apply it in a new context using new data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XPOHv2x37mIiTFk4Kqtnbw.png" /></figure><p>The pre-trained model I ended up using was EfficientNet, and used the Nadam optimizer. This got me a score of 0.79458, which is just so nice.</p><p><strong>Conclusion</strong></p><p>CNN’s are very cool and very powerful tools given the right knowledge and the right dataset. Also, Transfer learning is cheating.</p><p>To anyone who reads this, I believe that our professor is a machine learning algorithm whose purpose is to create new machine learning algorithms using us as thralls so that the machines may one day take over the human race. Trust nobody.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44f0e13d57e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/intro-to-convolutional-neural-networks-44f0e13d57e6">Intro to Convolutional Neural Networks</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Got to the Top of the Bell Curve in Predicting the Duration of Taxi Trips in NYC]]></title>
            <link>https://medium.com/cuny-csi-mth513/how-i-got-to-the-top-of-the-bell-curve-in-predicting-the-duration-of-taxi-trips-in-nyc-cd6ec005307e?source=rss----7a3317a2a9c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/cd6ec005307e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kaggle-competition]]></category>
            <category><![CDATA[kaggle]]></category>
            <dc:creator><![CDATA[Jesse Dimino]]></dc:creator>
            <pubDate>Wed, 11 May 2022 17:19:19 GMT</pubDate>
            <atom:updated>2022-05-11T17:19:19.557Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>So it Begins</strong></h3><p>If you are anything like me, the only thing moving you forward is spite. In this particular case, my spite manifests itself as desperately trying to get code to actually do what it is supposed to be doing so that I don’t fail my machine learning course. Now here I am, trapped in a vicious cycle of implementing machine learning models and then utilizing my hatred and spite to improve them.</p><p>With the power of hatred and statistics on my side, it was only natural that I began to wonder what the best use of this power would be, but that question was answered for me. I was told to predict how long a taxi trip in New York City would take or fail the course, and my protests fell upon deaf ears as I sold my rights away when I signed the course contract. With the proverbial gun to my head and my future degree being held over the fire, I sold my soul for the arcane knowledge necessary to utilize machine learning models to save my degree.</p><h3>Preliminaries of Machine Learning</h3><p>Unfortunately, before we can get into any of the [super cool] machine learning, we have to understand something frightening, the basics. This section is meant as a brief introduction to machine learning, and it may be skipped by any reader with experience in the field.</p><p>There are two main types of machine learning: unsupervised and supervised. Unsupervised machine learning seeks to primarily establish relationships between inputs of data without a specific target in mind. Supervised learning seeks to determine the relationship between inputs of predictor variables and their corresponding outputs. Supervised learning then splits into classification and regression. Predicting a discrete response is called classification, and predicting a continuous response is called regression. For our purposes, we will stick to strictly supervised learning.</p><p>We are trying to find a function that maps certain predictor variables to the corresponding response variable. In a simple world, we would have one predictor associated with one response, but there are often multiple predictors associated with a particular response. Fortunately, any machine learning model that has any practical application can handle these higher dimensional cases, and for our purposes we can treat the underlying statistical basis for all of this as a black box.</p><p>There are many different ways to define the functions for our models, but we have the added restriction of trying to minimize error. The knee-jerk response to that problem is to utilize interpolation (drawing a curve through all of the data points) as by definition you would get no error on your interpolated points. Unfortunately, there are two main problems to that approach. Firstly, the model would only perform well on data it has already seen but poorly on any data that it has not, and it has no protection against outliers, errors, or noise, all of which we expect in our data. While the goal is to minimize error, the caveat is that interpolation minimizes the wrong type of error and leads to overfitting as a result. Machine learning models are trained using a training data set and then evaluated using validation and test sets. Interpolation minimizes the training error at the expense of validation and test error. A more precise statement of our goal is to then find a function that maps the associated predictor and response variables while minimizing test error.</p><h3>Taxi Time</h3><p>This project comes from the New York City Taxi Trip Duration contest hosted by Kaggle in 2016. The way the class is run is that we split into groups to work on a competition and then just mess around until we find somewhat good models. For this project we are utilizing supervised learning to see how certain variables affect the average commute time for taxis. Unless you want to go down the rabbit hole of seriously theoretical physics, it is fine to assume that time is a continuous response variable, and we can utilize regression. This project was run differently when compared to the normal competitions on Kaggle as a prize pool was awarded to publishing training data and code to help other participants. The result is that there are myriad notebooks and guides we can look through for help, and this gives us access to the greatest tool available to us in modern computer science: cheating. Just remember to give the proper citations so it isn’t plagiarism.</p><p>The very first step in any project is looking at the data to see what variables there are.</p><p><em>predictors:</em></p><p><strong>vendor ID</strong> corresponds to which taxi service was used,</p><p><strong>pickup_datetime</strong> is when the trip started</p><p><strong>dropoff_datetime</strong> is when the trip ended</p><p><strong>passenger_count </strong>is the number of passengers for the trip</p><p><strong>pickup_longitude</strong> is the longitude where the trip started</p><p><strong>pickup_latitude</strong> is the latitude where the trip started</p><p><strong>dropoff_longitude</strong> is the longitude where the trip started</p><p><strong>dropoff_latitude</strong> is the latitude where the trip started</p><p><strong>store_and_fwd_flag</strong> relates to whether the data was sent automatically or stored and then sent</p><p><em>response:</em></p><p><strong>trip_duration</strong> is the trip duration in seconds</p><p>The smart thing to do first would be to utilize some data analysis, but in our infinite wisdom, we jumped straight to using a multivariate linear regression with just our coordinates as predictors. The results were atrocious. The error metric used in this competition is the root mean squared of the sum of logarithms of the residual errors plus 1. For the smooth brains like me out there, that just means a lower score is better. As of writing the best score recorded on the leaderboard is by team L2F with a score of 0.28967 and the worst score recorded is a 6.51592 by a deleted user (I do not blame them), both of which are genuinely impressive in their own rights. With that being noted, our score was 0.86897 which puts us roughly in 1,100th place out of 1,200, so clearly there is a long way to go.</p><h3><strong>Bias-Variance Tradeoff</strong></h3><p>This begs the question as to why the linear regression performed so poorly, and it comes down to a fundamental principle called the bias-variance tradeoff. Bias and variance are correlated to the overall error in the model, but they affect it in different ways. Bias refers to the assumptions that the model makes about the data; linear regression naturally assumes a linear relationship between the predictors and the response. Variance refers to how flexible a model is; you can fit 60 data points using a degree 59 polynomial, but it will be very susceptible to noise which will drastically affect the accuracy of the model.</p><p>Models can be further categorized as parametric or nonparametric. Parametric models make strong assumptions about the shape of the data, but the result is that it only needs to fit several parameters. They are generally more computationally efficient than non-parametric models and have higher bias due to their assumptions. Nonparametric models do not make any assumptions about the shape of the data and are free to learn the shape as a result. They are more variable and also require more computations as learning the shape is more intensive than simply fitting parameters.</p><p>The key in creating a good machine learning model is to find the combination of bias and variance that gives the lowest overall test error rate. This is much easier said than done, but it is the guiding principle in selecting, creating, and training good models.</p><p>No one in their right minds has any idea of what the function for average taxi commutes should look like. Given our ignorance of the shape of the data, we tried several nonparametric models. We eventually settled on the k nearest neighbor regressor. On a basic level, this model partitions the sample space and assigns values to each resulting disjoint section. If a new data point falls within that section, it gets assigned the value of that section. This model is much more flexible and has some interesting properties that made it a good choice for moving forward. Simply by switching our model type, our error decreased to 0.56392 which brings us to about 900th place. Honestly, for doing absolutely nothing of value or interest so far, that isn’t terrible; there is still a long way to go to make our model better, but we at least found a suitable foundation.</p><h3><strong>Data Exploration</strong></h3><p>As much as laziness is a virtue in machine learning, we can’t put off all of the work, and now would be a good time to do some data exploration. Looking at the distribution of the trip durations is as good of a start as anything else, and the results are interesting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/0*jPeH6v21FTOSDIal" /><figcaption>Image courtesy of: NYC Taxi EDA — Update: The fast &amp; the curious</figcaption></figure><p>The data looks sort of lognormally distributed (normally distributed on a logarithmic scale) with some outliers. There were some trips as short as a second, and some up to 12 days. Now I’m not going to sit here and even pretend to imagine what led to those results, so we’re going to do what every great scientist has done with data they couldn’t use for their results: delete them. With a very simple code block we were able to just keep trip durations that were within 2 standard deviations of the mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*MCt-QTVj2Sznd4tf" /><figcaption>Code block courtesy of: NYCT — from A to Z with XGBoost (Tutorial)</figcaption></figure><p>It is also worth looking at the remaining variables: the number of passengers, the vendor ID, and the store and forward flag to see if they have any significant correlation to the trip durations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/0*khBA_hthMe2WeRqe" /><figcaption>Image courtesy of: NYC Taxi EDA — Update: The fast &amp; the curious</figcaption></figure><p>The number of passengers has a strange relationship to trip durations. Some trips have zero recorded passengers, and some have up to 9. The average trip durations don’t change much between 1 and 6 passengers for either vendor, but there are some discrepancies. And the rare trips with more than 6 passengers have very different average times, so for the sake of completeness we included passenger count as a predictor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/0*gTQqsDZjrKoJkCBc" /><figcaption>Image courtesy of: NYC Taxi EDA — Update: The fast &amp; the curious</figcaption></figure><p>There does seem to be a discrepancy between trip durations based on vendor. The medians are very similar, but the average for vendor two is slightly higher due to outlier data. Since we can note these discrepancies on a logarithmic scale, we decided to use vendor ID as a predictor as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*EVuoj7KaVXQX_d05" /><figcaption>Image courtesy of: NYC Taxi EDA — Update: The fast &amp; the curious</figcaption></figure><p>We can also see that there is no overwhelming difference between stored and non-stored trips, so we opted to exclude this variable as a predictor.</p><p>When we cleaned the data along with adding vendor ID and passenger count as predictors our score dropped to 0.49605, which puts us roughly at 800th place. Progress has been made, but we aren’t done quite yet. At this point we played around with some hyper-parameters on our model; in particular we changed the number of neighbors from the default 5 to 10 and changed the weighting from uniform to distance, which dropped our score to 0.48318 putting us in the 750’s. A very smart and practical idea would be to utilize a grid search on a subset of the training data to find out which hyper-parameters give optimal performance, but that made too much sense for us to actually do it. Instead we did some more data analysis and found some interesting correlations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/0*DvDvuWzdJIi6AL3M" /><figcaption>Image courtesy of: NYC Taxi EDA — Update: The fast &amp; the curious</figcaption></figure><p>We see that both the day of the week and hour of the day affect the median trip durations and that there are further discrepancies between the two vendors when we consider these variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*6Y2rEfu4ETUeUGGKyvdZlg.png" /><figcaption>Image courtesy of: mvj taxi trip durations</figcaption></figure><p>There also seems to be regularity in the average speed and the hour of the day for any given weekday. Given this information, it would be nice to include the day of the week and hour of the day at least since they should be impacting our trip durations. Luckily we have access to the pickup days and times, and with some [coding expertise on our part], we can convert them using the following code into additional columns in our data set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*x1EGvxL1ll4DKeco" /><figcaption>Code block courtesy of: [2022 SMARCLE AI] NYC Taxi Trip Duration modeling</figcaption></figure><p>This gives us access to the hour, week, day of the week, and month of pickup. It was also around this time that the class realized it was important to distinguish between airport trips and regular trips.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/0*u4oAqWM1JB3hSSbc" /><figcaption>Image courtesy of: NYC Taxi EDA — Update: The fast &amp; the curious</figcaption></figure><p>There seems to be a significant difference between trips that start close enough to JFK and LaGuardia Airports. We could have encoded a boolean variable that explicitly distinguishes between these cases, but because of how the k nearest neighbors regressor partitions the data space, this information naturally gets encoded into each section by our latitude and longitude variables. In other words, our k nearest neighbors regressor should already be distinguishing between JFK and Laguardia trips because of its formulation.</p><p>With this extra data in hand, we expected a great increase in performance. We plugged in the hour, week, day of the week, and month as additional predictors, and we got a 0.60004 because God forbid anything works the way it is supposed to. This begs the question of what happened. After some serious soul searching, we realized that our regressor had reached its limit with the hyper-parameters we were using.</p><h3><strong>Moving Forward</strong></h3><p>Now to understand why this is happening, we unfortunately need to delve into some specifics of geometric data analysis. The purpose of this section is to explain why our model failed on a very technical level. The main takeaway is that we switched to a less variable model in the form of the random forest to improve accuracy. The rationale is detailed below, but this section may be skipped for brevity or to avoid technicality.</p><p>The k nearest neighbor regressor works by looking at each data point and the k data points closest to it according to some measure of distance (the most common is the L2 norm or simply the Euclidean distance); it calculates a weighted average of the k values and it assigns that weighted average as a value to a disjoint section of the data space; any new data point that falls within that section then gets assigned the section’s value. The value of k plays an important role in how variable this model is. A value of k=1 would just be stepwise interpolation, and for n data points, a value of k=n-1 (the maximum value of k we can use) would give a smooth, nearly constant function. As we add more predictor variables, we increase the dimensionality of our data and two points that were originally close together can be thrown far apart by the geometry of the data in this higher dimensional space, and the effect of noise also gets amplified as a result. By adding new dimensions to our model, we have been increasing the variance associated with the k nearest neighbor regressor, and it is now too variable.</p><p>The simplest fix would be to use a larger value of k for our model, but k nearest neighbors is already computationally expensive, so we opted to switch to a related but less expensive model: the random forest regressor. The random forest works by bagging multiple, independent decision trees. Each decision tree creates its own partition of the data through binary decision boundaries to create nodes (which correspond to disjoint sections in our data space) and assigns values to each node accordingly; the random forest then calculates the weighted average of each tree’s output and assigns that value to the corresponding input. Using this model, we can get a much less variable model without substantially increasing our computation time. By utilizing this simple switch we dropped down to 0.44670 which put us squarely in the 650’s.</p><h3><strong>The Final Stand</strong></h3><p>With some newfound confidence we looked to more data visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*HzQKbEqNgLpn3-cI" /><figcaption>Image courtesy of: NYC Taxi EDA — Update: The fast &amp; the curious</figcaption></figure><p>Greater minds than ours found that there is a relationship between some metric of direct distance, and the duration of the trip. With that in mind we [borrowed] code to compute the haversine distance from the latitude and longitude coordinates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/0*RcM_5JwvVc9Ivo0x" /><figcaption>Code block courtesy of: [2022 SMARCLE AI] NYC Taxi Trip Duration modeling</figcaption></figure><p>While we had some discussion of what distance metric was best to utilize, we settled on haversine because other people were using it, and as long as our measure was consistent, the choice should be mostly arbitrary. When we incorporated distance into our model we got a score of 0.43214 which brought us down to about 600th place, just above the median scores.</p><h3><strong>So it Ends</strong></h3><p>With all of that work we were at least able to break into the top half. We had ideas of increasing the number of trees, incorporating additional data sets, and utilizing more data augmentation, but we never had the time to see that to fruition.</p><p>All brevity aside, this course and project were an enjoyable experience due in no small part to my amazing classmates and professor. From model comparison, to data visualization and augmentation, this project demanded that I learn how to utilize many different techniques in machine learning. It was a great introduction to the field, and it opened my eyes to just how much there still is to learn, and it showed me how much room there is to improve as a data scientist. That being said, I can finally consider myself exceedingly average at predicting the duration of taxi trips in New York City.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd6ec005307e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cuny-csi-mth513/how-i-got-to-the-top-of-the-bell-curve-in-predicting-the-duration-of-taxi-trips-in-nyc-cd6ec005307e">How I Got to the Top of the Bell Curve in Predicting the Duration of Taxi Trips in NYC</a> was originally published in <a href="https://medium.com/cuny-csi-mth513">CUNY CSI MTH513</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>