<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Christian McDaniel on Medium]]></title>
        <description><![CDATA[Stories by Christian McDaniel on Medium]]></description>
        <link>https://medium.com/@cmcdaniel19?source=rss-4a3e58f857c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zu3pum-yASeMq2SCNjsmSA.jpeg</url>
            <title>Stories by Christian McDaniel on Medium</title>
            <link>https://medium.com/@cmcdaniel19?source=rss-4a3e58f857c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 05:17:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cmcdaniel19/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I submitted an abstract to SciPy for the second year in a row… Here’s why.]]></title>
            <link>https://medium.com/the-quarks/i-submitted-an-abstract-to-scipy-for-the-second-year-in-a-row-heres-why-1c9e202abf1d?source=rss-4a3e58f857c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c9e202abf1d</guid>
            <category><![CDATA[open-science]]></category>
            <category><![CDATA[scipy]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Christian McDaniel]]></dc:creator>
            <pubDate>Sun, 03 Mar 2019 23:17:52 GMT</pubDate>
            <atom:updated>2019-03-03T23:17:52.156Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*J_EHaInQ4AwPrmN_mIgWHA.jpeg" /><figcaption>Source: <a href="https://scipy2018.scipy.org/ehome/299527/648140/">https://scipy2018.scipy.org/ehome/299527/648140/</a></figcaption></figure><p>When I touched down in Austin, TX the day before <a href="https://scipy2018.scipy.org/ehome/index.php?eventid=299527&amp;">SciPy 2018</a> was set to begin, I had no idea of the transformative experience that awaited me. It was the end of my first year in a <a href="https://www.ai.uga.edu/ms-artificial-intelligence">Master’s in AI program</a> at the University of Georgia, and this was my first computer science-related conference. I was accepted to present a <a href="https://github.com/xtianmcd/accelstm">research project</a> I’d been working on, and about the only thing I expected with any confidence was a medley of upturned noses and dismissive chuckles from anyone reading over my poster (yay for imposter syndrome!).</p><p>The main conference kicked off that night with an opening reception in the lobby of <a href="https://www.enthought.com/">Enthought</a>, the company that hosts SciPy in the US. In the lobby I found a room full of academics and professionals of all ages and interests. I’m sure the free tacos, cool merch and home-brewed beer helped, but I was quickly immersed in the culture of SciPy and open science, and knew I belonged.</p><p>My conversations that night and over the next three days shared some common themes central to what SciPy stands for: genuine interest, humility and inclusiveness, happiness and encouragement.</p><p>There was a shared interest in scientific programming that garnered real conversations and mutual understanding about each others’ work, with a potential for gaining or providing relevant insight along the way.</p><p>Whether I was speaking with a local high school student learning to code or the <em>literal person who wrote much of pandas</em>, no one was boastful or trying to establish their knowledge or rank. From administrative policies to the content of presentations, there were major efforts from the top down to ensure equality and inclusiveness.</p><p>Finally, from that opening reception to the bitter, tired end, the buzz of kindness and excitement was as palpable as the constant smell of fresh coffee in the hallways. If SciPy emanates one message, it is to tell each attendee that he or she is capable of accomplishing greatness, including the ability and aptitude to be a contributor to <a href="https://en.wikipedia.org/wiki/Open_science"><strong>open science</strong></a>.</p><p>Did you catch my earlier mention that I spoke with a core contributor to pandas? This is what SciPy is all about. Over the weekend, I shared a table with an integral member of the jupyter notebook developers; I had a beer one night with a top committer to the numpy repo; and I sat in on a whiteboard session where the folks who ran pandas discussed its current projects, roadblocks and future.</p><p>I mention these fan-guy moments to point out the unique occasion provided at SciPy, where developers from all walks of life and all levels of experience get thrown in a room together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*_h8ihJACoiwwiqUlpx0mhg.jpeg" /><figcaption>Source: <a href="https://www.slideshare.net/kammeyer/the-joy-of-scipy">https://www.slideshare.net/kammeyer/the-joy-of-scipy</a></figcaption></figure><p>Over the weekend I gained an insight into how open science is made possible and maintained, and I learned so much about all levels of the python language. I talked with professors and postdocs working on fascinating research problems, and picked the brains of professionals in industry about how I could apply my skills after grad school. I even made friends and colleagues I still keep in touch with. Still, perhaps best of all, I had the opportunity during the <a href="https://scipy2018.scipy.org/ehome/299527/648141/">Sprints</a> to make my own contribution to the open source community.</p><p>I’ve been to several conferences since SciPy, and while I haven’t been to many of the other <em>larger</em> computer science-related conferences, I feel confident that SciPy offered me something quite unique.</p><p>I learned a great deal and witnessed cutting edge research and projects; but more importantly, I felt like I gained a spot in a welcoming community and I left with a renewed sense of hope and vigor about my schooling and career going forward.</p><p>For these reasons, I eagerly submitted my research to SciPy for the <a href="https://www.scipy2019.scipy.org/">second year</a> in a row and plan to get my “SciPy fix” for years to come.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*clixwER0pfkvunCJfk5-RQ.png" /><figcaption>Source: <a href="https://www.scipy2019.scipy.org/">https://www.scipy2019.scipy.org</a></figcaption></figure><p>Hope to see you there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c9e202abf1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-quarks/i-submitted-an-abstract-to-scipy-for-the-second-year-in-a-row-heres-why-1c9e202abf1d">I submitted an abstract to SciPy for the second year in a row… Here’s why.</a> was originally published in <a href="https://medium.com/the-quarks">The Quarks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Data Science via ScalaTion Part 3 — A Deeper Look]]></title>
            <link>https://medium.com/the-quarks/an-introduction-to-data-science-via-scalation-part-3-a-deeper-look-4a2d075fdb69?source=rss-4a3e58f857c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a2d075fdb69</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[scalation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Christian McDaniel]]></dc:creator>
            <pubDate>Sat, 12 Jan 2019 19:59:31 GMT</pubDate>
            <atom:updated>2019-01-12T19:59:31.205Z</atom:updated>
            <content:encoded><![CDATA[<h3>An Introduction to Data Science via ScalaTion Part 3 — A Deeper Look</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*YdFnhQMZ-Jb5_2TXPuJicQ.jpeg" /><figcaption>Source: <a href="http://www.mkels.com/">http://www.mkels.com/</a></figcaption></figure><p>If you’ve made it this far, congratulations! You’ve officially set up your environment (<a href="https://medium.com/the-quarks/an-introduction-to-scalation-part-1-setting-up-478cdc13470e">Part 1</a>) for running the powerful data modeling and analytics library <a href="http://cobweb.cs.uga.edu/~jam/scalation.html">ScalaTion</a>, and have run your first program (<a href="https://medium.com/the-quarks/an-introduction-to-scalation-part-2-helloworld-of-data-science-36038d0ff1d7">Part 2</a>), implementing two common and powerful machine learning models on two different datasets in the process. Now, let’s dive a bit deeper to better understand the ScalaTion library, the <a href="https://github.com/quinngroup/quarks/tree/master/scalation">code</a> we’ve run, and the output we received.</p><h4>Stage 1. Data Prep</h4><p>If we look at the main object of our HelloWorldOfDS.scala <a href="https://github.com/quinngroup/quarks/blob/master/scalation/HelloWorldOfDS/src/main/scala/scalation/HelloWorldOfDS.scala">script</a>, i.e., the part enclosed within the brackets belonging to</p><pre>object HelloWorldOfDS extends App</pre><p>we first see two methods prepReg() and prepBayes().</p><p>preReg() uses the MatrixD class in ScalaTion to read in a real-valued dataset winequality-white.csv (provided by ScalaTion, located in the data/analytics directory within the library’s main folder). The winequality-white dataset contains nearly 5000 examples of white wine batches, each of which maps eleven physico-chemical properties of the wines to a rating assigned by an oenologist.</p><p>Likewise, the prepBayes() method uses the Relation class to read in the categorical breast-cancer.arff dataset within ScalaTion’s data/analytics/classifier directory. The dataset, which maps various patient data points to a binary diagnosis of breast cancer, is converted to the MatriI data structure, similar to the MatrixD class, except that the values are Integers (i.e., categorical) instead of Decimals. The missing x means the class allows various types of matrices (e.g., dense, bidiagonal, sparse, symmetric triangular, etc.).</p><h4>Stage 2. Data Crunch</h4><p>After being prepped, the data are processed using either <a href="https://en.wikipedia.org/wiki/Linear_regression#Simple_and_multiple_linear_regression">multiple linear regression</a> (MLR, for the real-valued data) or <a href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier">Naive Bayes</a> (for the categorical data) via the run Reg() and crossValidateAlgos() methods, respectively.</p><h4>Regression:</h4><p>runReg() uses the Regression class in ScalaTion. <strong>Regression</strong> techniques choose from several <em>factorization</em> techniques to solve for parameter vector <strong>b </strong>that is fit to the data. <strong>b</strong> gives us the <em>weights </em>for each predictor as a way to model the overall sample space and thus map a given data point to a predicted outcome. We find <strong>b </strong>by minimizing the squared error between our predicted and actual target values and thus call <strong>b</strong> the <em>least squares coefficients</em>.</p><p>Given the equation for MLR, <strong>y</strong>=<strong>Xb</strong>+<strong>e,</strong> where <strong>e</strong> represents the error that the model cannot eliminate (e.g., due to noise in the data or bias of the model), MLR attempts to approximate the optimal values for <strong>b</strong> by performing factorization on the known components (<strong>y</strong> and <strong>X</strong>) using the Normal Equation: <strong>X</strong>t (<strong>Xb</strong>) = <strong>X</strong>t <strong>y</strong>, where <strong>X</strong>t is the transpose of <strong>X</strong>. There are five such factorization techniques offered by ScalaTion, including QR factorization, Cholesky factorization, singular value decomposition, LU factorization, and Inverse/Gaussian elimination. The default technique is the <a href="https://en.wikipedia.org/wiki/Gaussian_elimination">Inverse/Gaussian elimination</a>, which is the classical textbook technique.</p><p>As stated in Part 2, MLR is performed on the raw data (WineQual), after performing a simple transformation (Wine-tf), and again after some pruning. The <strong>transformation</strong> step transforms one of the attributes, that representing “Alcohol content”, by squaring its values. Since the attribute for Alcohol content is among the most correlative attributes w.r.t. the overall wine quality (¯\_(ツ)_/¯), squaring its values should provide even greater separation between classes since the magnitude of transformation on larger values will be exponentially larger than that on the smaller values. I.e., transormation is performed to help the data help us.</p><p>After a second MLR on the transformed data, a pruning step is implemented. <strong>Pruning</strong>, which consists of removing entire attributes of the data, is useful as a way to reduce the complexity and size of your dataset. However, care must be taken to ensure that only the less informative attributes are removed. Thus, we have used forward selection and backward elimination in the prunePreds() method and VIF, optional output from the Regression class, to guide our pruning decisions.</p><p><strong>Forward selection</strong> iteratively rebuilds the dataset by selecting the most predictive attribute at each iteration, stopping once a certain number of attributes are selected or once a threshold of informativeness is reached. Similarly, <strong>backward elimination</strong> iteratively removes the least informative attribute from the dataset by finding the one which, when hidden from the model, has the least effect on the prediction accuracy. <strong>VIF</strong> stands for <a href="https://en.wikipedia.org/wiki/Variance_inflation_factor">variance inflation factor</a> and is a method for finding multicollinearity among the attributes. I.e., VIF finds attributes which contribute the same information for the overall prediction. All but one of these attributes can be removed with neglible effects on prediction accuracy.</p><p>When ScalaTion’s Regression class is run, the coefficients are returned as well as the model’s performance after fitting. Ideally, we should see an increase in performance after each preprocessing step. Refer to the output snippets in <a href="https://medium.com/the-quarks/an-introduction-to-scalation-part-2-helloworld-of-data-science-36038d0ff1d7">Part 2</a>, sections (2) and (3), for example outputs from running this section of the code.</p><h4>Bayes</h4><p>Next we run our second algorithm, Naive Bayes. <strong>Naive Bayes</strong> uses Bayes’ Theorem to predict a target class for a given input data vector <strong>x</strong>; i.e., it aims to predict “y given <strong>x</strong>,” or y|<strong>x, </strong>by finding the <em>probability</em> of each target class <em>c</em> given the data instance <strong>x</strong>, <em>p</em>(y|<strong>x</strong>), and choosing the class with the highest probability. Bayes’ Theorem approximates this unknown probability by using known probabilities — the <em>prior probability </em>and the <em>conditional probability</em>. The prior probability for each class <em>c</em>, <em>p</em>(y=<em>c</em>), is simply the proportion of <em>training</em> data points with class <em>c</em> as the target value. The conditional probability looks at each predictor variable <em>j</em> and computes the probability that a variable <em>j</em> = a value <em>h </em>given the target <em>y</em> = class <em>c</em>, i.e., <em>p</em>(<strong>x</strong><em>j</em> = h | y = <em>c</em>). Put it all together, and Bayes’ Theorem states <em>p</em>(y|<strong>x</strong>)=<em>p</em>(x|y) <em>p</em>(y) / <em>p</em>(x). We typically drop the term <em>p</em>(x), which is 1 divided by the number of samples<em>,</em> as it is simply a normalizing factor. We choose the class <em>c</em> that maximizes this probability for the given input <strong>x.</strong></p><p>Naive Bayes is called in our script within the crossValidateAlgos() method. <strong>Cross validation</strong> is a technique that adds robustness to your results by repeating the predictions multiple times. The dataset is first divided into <em>k</em> equal subsets, and for <em>k</em> iterations, or <strong>folds</strong>, the <em>k</em>th subset is held out during training and used as a testing set.</p><p>During each fold of the cross validation, after the predictions are made on the <em>k</em>th subset, the checkClass() method is called to first check the predicted class against the ground truth class for each prediction and accumulate the number of true positives (tp), false positives (fp), true negatives (tn), and false negatives (fn). Since the prediction class is binary, this process is quite straightforward. Next, the checkClass() method calls getStats() to use the tp, fp, tn, and fn values to calculate the precision, recall, accuracy and F-score. Each of these is a slightly different <strong>performance measure</strong> from the others, and looking at all of them together again adds robustness to your results and can also provide insights into the model’s performance.</p><p>Following cross validation, the mean and standard deviation of each performance measure are calculated across all folds and the results are reported (See the output snippet (1) in <a href="https://medium.com/the-quarks/an-introduction-to-scalation-part-2-helloworld-of-data-science-36038d0ff1d7">Part 2</a> for an example). We ran both 10-fold cross validation and 20-fold cross validation for a comparison.</p><h3>Conclusion</h3><p>We have now completed our first Data Science project using ScalaTion! After working our way through the initial download and build procedure, we implemented a script that loaded in two types of data and ran two different machine learning algorithms along with some data preprocessing and techniques to strengthen our results. With all that behind us, I hope you feel comfortable using this incredible resource for your own projects, and I wish you the best of luck in your Data Science endeavors!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a2d075fdb69" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-quarks/an-introduction-to-data-science-via-scalation-part-3-a-deeper-look-4a2d075fdb69">An Introduction to Data Science via ScalaTion Part 3 — A Deeper Look</a> was originally published in <a href="https://medium.com/the-quarks">The Quarks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to ScalaTion Part 2 — HelloWorld of Data Science]]></title>
            <link>https://medium.com/the-quarks/an-introduction-to-scalation-part-2-helloworld-of-data-science-36038d0ff1d7?source=rss-4a3e58f857c------2</link>
            <guid isPermaLink="false">https://medium.com/p/36038d0ff1d7</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[scalation]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Christian McDaniel]]></dc:creator>
            <pubDate>Sat, 05 Jan 2019 21:18:06 GMT</pubDate>
            <atom:updated>2019-01-05T21:18:06.661Z</atom:updated>
            <content:encoded><![CDATA[<h3>An Introduction to Data Science via ScalaTion Part 2 — HelloWorld of Data Science</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*BwfLzMx_dOsc7NWdXMtmEA.png" /><figcaption>Source: <a href="https://blog.rntech.co.uk/">https://blog.rntech.co.uk/</a></figcaption></figure><p>In <a href="https://medium.com/the-quarks/an-introduction-to-scalation-part-1-setting-up-478cdc13470e">Part 1</a> of this two-part series we introduced <a href="http://cobweb.cs.uga.edu/~jam/scalation.html">ScalaTion</a>, a robust Scala-based modeling and analytics library written in such close alliance to the fundamentals of linear algebra, calculus, statistics, and machine learning that it has made an integral contribution to my development within the field of data science. We also downloaded all the required software and automated the build process for starting new projects. In Part 2, we execute our first ScalaTion script — HelloWorldOfDS.scala .</p><h4>Runnin &gt; Walking… Gimme the results!</h4><p>If you have followed <a href="https://medium.com/the-quarks/an-introduction-to-scalation-part-1-setting-up-478cdc13470e">Part 1</a> of this tutorial, then your project folder should be set up and equipped with the necessary build.sbt and .jar files. Additionally, you should have a local clone of <a href="https://github.com/quinngroup/quarks">The Quarks’ GitHub repository</a>, which contains the HelloWorldOfDS.scala file we will be running and exploring. You can transfer this file into your own project folder built in Part 1 or you can simply run the script from the local clone itself.</p><p>Either way, to run the script, enter Terminal or another bash shell and cd into the <em>main project folder </em>containing the build.sbt file and data, lib, project, src, and target folders. In the quarks GitHub repository, this folder is the one titled HelloWorldOfDS inside the scalation folder.</p><p>Once inside this folder, typing sbt run should open an sbt shell, compile, and execute the code within the project’s tree structure. If all goes smoothly, you will end up with an output something like this:</p><p>(1)</p><pre>|====================|</pre><pre>|   CANCER  DATASET  |</pre><pre>|   10  FOLD  CV    |</pre><pre>|    NAIVE  BAYES    |</pre><pre>|====================|</pre><pre>&gt;meanCV</pre><pre>accuracy     recall  precision    f-score</pre><pre>NaiveBayes 0.719285714 0.825150405 0.771436896 0.792888583</pre><pre>==========================================================</pre><pre>&gt;stdCV</pre><pre>accuracy     recall  precision    f-score</pre><pre>NaiveBayes 0.132583597 0.134881091 0.134661226 0.118966704</pre><pre>==========================================================</pre><p>… and if you scroll up a bit, a sequence of outputs resembling</p><p>(2)</p><pre>backwardElim: cols_j = Set(0)</pre><pre>backward model: remove x_j = 11 with b = VectorD(5.87791)</pre><pre>fit = VectorD(3840.99, 3840.99, 0.784196, 0.885548, 0.670793, 3.78858e-15, 3.77476e-15, Infinity, -1188.69, -1182.19)</pre><p>… and further up</p><p>(3)</p><pre>|====================|</pre><pre>|  WineDrop DATASET  |</pre><pre>|    MULT LIN REG    |</pre><pre>|====================|<br></pre><pre>Coefficients:</pre><pre>| Estimate   |   StdErr   |  t value | Pr(&gt;|t|)</pre><pre>x₀ | 140.098884 |  18.529223 |   7.5610 |   0.00000</pre><pre>x₁ |   0.060412 |   0.020444 |   2.9550 |   0.00313</pre><pre>x₂ |  -1.904478 |   0.112399 | -16.9440 |   0.00000</pre><pre>x₃ |   0.076795 |   0.007549 |  10.1731 |   0.00000</pre><pre>x₄ |  -0.276671 |   0.542720 |  -0.5098 |   0.61020</pre><pre>x₅ |   0.003777 |   0.000842 |   4.4848 |   0.00001</pre><pre>x₆ |  -0.000311 |   0.000377 |  -0.8235 |   0.41020</pre><pre>x₇ | -138.989068 |  18.906592 |  -7.3514 |   0.00000</pre><pre>x₈ |   0.660673 |   0.103723 |   6.3696 |   0.00000</pre><pre>x₉ |   0.617767 |   0.100076 |   6.1730 |   0.00000</pre><pre>x₁₀ |   0.009603 |   0.001095 |   8.7670 |   0.00000</pre><pre>SSE:             2751.6229</pre><pre>Residual stdErr: 0.7504 on 10 degrees of freedom</pre><pre>R-Squared:       0.2836, Adjusted rSquared:  0.2822</pre><pre>F-Statistic:     193.4762 on 10 and 4887 DF</pre><pre>AIC:             -2802.3636</pre><pre>BIC:             -2730.9012</pre><pre>--------------------------------------------------------------------------------</pre><pre>VIF:</pre><pre>VectorD(0.340531, 0.000321888, 0.939356, 1.39361e-05, 0.250047, 0.0932211, 3.71570e-07, 0.00305491, 0.000444961, 0.0593856)</pre><h4>Great, I’m a Data Scientist! But… uh… What have I done?</h4><p>In short, you read in two different datasets, one from a <strong>CSV</strong> file and one from an <strong>ARFF</strong> file, containing real-valued data and categorical data respectively.</p><p>You performed <strong>multiple linear regression</strong> (MLR) on the real-valued data in its raw form (WineQual), after performing a simple transformation (Wine-tf), and again after removing an attribute (WineDrop) guided by forward selection, backward elimination, and VIF calculation (more on that in Part 3). Each regression fit returned the derived least squares coefficients and associated statistics as well as several quality of fit measures (See outputs (2) and (3) above).</p><p>You then applied the <strong>naive Bayes classifier</strong> to the categorical data, reporting the mean and standard deviation for accuracy, recall, precision, and F1 score from 10-fold cross validation (CV) (See output (1) above).</p><p><em>Note</em>: The output above the naive Bayes CV results resembling repetitions of</p><pre>vc = Array(6, 3, 11, 7, 3, 3, 2, 6, 2)</pre><pre>distinct value count vc = Array(6, 3, 11, 7, 3, 3, 2, 6, 2)</pre><p>is built-in output from the ScalaTion NaiveBayes class. It shows the number of unique values for each attribute, and is returned at two points of execution during each CV fold.</p><p><strong>Congratulations!</strong> You’ve officially run your first machine learning models using the <a href="http://cobweb.cs.uga.edu/~jam/scalation.html">ScalaTion</a> library. If you’re a beginner like I was when I first used the library, just getting to this point can be daunting, so we’ll stop here for now. Stay tuned for <strong>Part 3</strong>, where we will take a deeper look into the code we’ve just run.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36038d0ff1d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-quarks/an-introduction-to-scalation-part-2-helloworld-of-data-science-36038d0ff1d7">An Introduction to ScalaTion Part 2 — HelloWorld of Data Science</a> was originally published in <a href="https://medium.com/the-quarks">The Quarks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Data Science via ScalaTion Part 1 — Setting Up]]></title>
            <link>https://medium.com/the-quarks/an-introduction-to-scalation-part-1-setting-up-478cdc13470e?source=rss-4a3e58f857c------2</link>
            <guid isPermaLink="false">https://medium.com/p/478cdc13470e</guid>
            <category><![CDATA[scalation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Christian McDaniel]]></dc:creator>
            <pubDate>Sat, 09 Jun 2018 18:43:28 GMT</pubDate>
            <atom:updated>2019-01-04T23:53:34.215Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*VMsufliniriE1YmEwE765g.png" /><figcaption>Source: <a href="http://karabina.co.za/wp-content/uploads/2017/11/Data-Science-Blog-Image.png">Data-Science-Blog-Image.png</a></figcaption></figure><p>As I iterate through the high-dimensional hilly ascent toward becoming a data scientist — clambering over statistical overhangs, deep computer science crevices, and technical stretches of mathy rock walls — I have realized both the importance and the difficulty of keeping one foot in theory and the other in implementation. All too often I have mastered a concept but remained clueless about applying it to a messy real-world problem. Conversely, my busy schedule as a Master’s student and the availability of robust single-line function calls make it easy to neglect Step Two in the “implement first, learn-what-you-just-did second” strategy.</p><p>Enter <a href="http://cobweb.cs.uga.edu/~jam/scalation.html">ScalaTion</a>, a data modeling and analytics library written straight out of your linear algebra and data science text books. This SCALAble simulaTION and analytics library is written in <a href="https://www.scala-lang.org/">Scala</a>, an object oriented, functional programming language and a member of the Java family. Thus, ScalaTion combines the interpretability and conciseness of a high-level programming language with the by-the-books technicality of a “low-level” translation of concepts in linear algebra, statistics, machine learning, and more. Using this library for <a href="https://github.com/xtianmcd/CSCI6360_Projects_Practice_and_Presentations">projects</a> in my <a href="http://cobweb.cs.uga.edu/~jam/home/courses/csci4360/index.html">Data Science II course</a> was instrumental in bridging the gap between the statistical fundamentals and model implementation involved in learning data science.</p><p>Once you are familiar with setting up and running projects using ScalaTion, you begin to understand the robustness and interpretability of its classes and functions. However, ScalaTion can be a bit tricky when you’re getting started and this blog post is intended to guide you through just that. We will first set up our ScalaTion environment, and then in Part Two we’ll execute our first data science model using ScalaTion, and finally in Part Three we’ll dig into the code we will have just run. Keep in mind that this tutorial, although written for complete beginners, was created using a 2017 MacBook Pro and some of the steps may not generalize seemlessly to other makes and models.</p><h3>The ScalaTion Environment</h3><p>I’ve constructed a bash script on <a href="https://github.com/quinngroup/quarks">The Quarks’ GitHub repository</a> to download all the necessary software, set up your first project directory, and transfer in the necessary files. Clone the repository and on your local machine, enter the folder scalation/ and execute the .sh file via</p><pre>sh &lt;path/to/scalation_setup.sh&gt;</pre><p>where &lt;path/to/scalation_setup.sh&gt; is the path to the saved bash script. <a href="http://burnedpixel.com/blog/setting-up-git-and-github-on-your-mac/">This site</a> is helpful for creating a GitHub account and getting set up to use the git command. To clone our <a href="https://github.com/quinngroup/quarks">repository</a>, follow the link and hit the green Clone or Download button to copy the link, then in Terminal on your Mac, cd into a folder in which you’d like to install the repo and enter git clone https://github.com/quinngroup/quarks.</p><p>Now let’s discuss what this script does so we can be informed data scientists!</p><h4>Downloading your software</h4><ul><li>BEFORE running the bash script, ensure your computer’s <strong>Java is up to date</strong> by typing java -version in your terminal. Java version 1.8 or higher is sufficient; otherwise, you can download/update Java <a href="https://java.com/en/download/">here</a>.</li></ul><pre>#!/bin/bash</pre><p>This tells the computer you are using a bash shell.</p><pre>cd /Applications</pre><p>Change the current working directory to /Applications. This is where the software will be stored.</p><pre>sudo port install sbt</pre><p><strong>Download the Scala programming language</strong> by downloading their <a href="https://www.scala-sbt.org/">simple build tool (sbt)</a>. You’ll need to have admin priveleges on your device and will need to enter your password for the script to continue (<em>pro tip</em>: when typing a password at the command line, no indication of typing is given as a security measure; just type in your password and hit Enter).</p><pre>wget <a href="http://cobweb.cs.uga.edu/~jam/scalation_1.4.tar.gz">http://cobweb.cs.uga.edu/~jam/scalation_1.4.tar.gz</a><br>tar xvfz scalation_1.4.tar.gz</pre><p>ScalaTion is under ongoing development in <a href="http://cobweb.cs.uga.edu/~jam/">Dr. John Miller’s</a> lab in the computer science department at the University of Georgia. The wget command <strong>downloads ScalaTion</strong> from its <a href="http://cobweb.cs.uga.edu/~jam/scalation.html">homepage</a> hosted by the UGA CS server. <a href="http://cobweb.cs.uga.edu/~jam/scalation_1.4/README.html">ScalaTion 1.4</a> is currently the most stable version, so this is what we will download; however, newer versions are currently being tested and perfected!</p><p><em>Troubleshooting tips:</em></p><ul><li>The wget command needs to be installed on Mac devices. This can be done using Homebrew via brew insall wget or from source using <a href="http://osxdaily.com/2012/05/22/install-wget-mac-os-x/">this</a> helpful guide. The subsequent tar command uncompresses the library into a usable format.</li><li>If the download has issues, make sure your version of Scala is compatible (Scala 2.12 for ScalaTion 1.4). The download links on the homepage tell what version of Scala is needed for each version of ScalaTion.</li><li>If you are having issues with the .tar compressed file, a .zip version is also available. Download the file with the .zip extension and run unzip scalation_1.4.zip to unzip.</li></ul><pre>mkdir ~/Documents/HelloWorldOfDS<br>cd ~/Documents/HelloWorldOfDS</pre><p>The project directory is created in the default~/Documents folder; feel free to change the location of this directory.</p><h4><strong>Automate the build structure</strong></h4><pre># source for automated build: Scala Cookbook Ch. 18.1<br># generate the file structure<br>mkdir -p src/{main,test}/{java,resources,scala}<br>mkdir lib project target</pre><pre># create an initial build.sbt file<br>echo &#39;name := &quot;HelloWorldOfDS&quot;<br>version := &quot;1.0&quot;<br>scalaVersion := &quot;2.12.4&quot;&#39; &gt; build.sbt</pre><ul><li>Sbt compiles and runs programs using a structured filepath system. Manually setting this up from the command line can seem tedious, but we have just automated the process. I suggest creating a separate bash script that performs this automated build for later projects:</li></ul><pre>#source: Scala Cookbook Ch. 18.1<br>#!/bin/sh</pre><pre>mkdir -p src/{main,test}/{java,resources,scala}<br>mkdir lib project target</pre><pre># create an initial build.sbt file<br>echo &#39;name := &quot;MyProject&quot;<br>version := &quot;1.0&quot;<br>scalaVersion := &quot;2.12.4&quot;&#39; &gt; build.sbt</pre><ul><li>Title the script something like mkdirs4sbt and save with a .sh extension.</li><li>Now when you create your project’s main folder, you will simply enter that folder (cd) and run sh &lt;filepath/to/mkdirs4sbt&gt; and the file structure will be built automatically.</li><li>More information can be found in the <a href="https://www.safaribooksonline.com/library/view/scala-cookbook/9781449340292/ch18s02.html">Scala Cookbook Chapter 18.1</a>.</li><li><em>Side note for beginners (like me):</em> As someone with very little computer science background before entering the field of data science one year ago, this build process seemed quite cumbersome given the ease of writing and running Python scripts. However, I have since used similar build processes with Java (Maven) and C++ (Make), and using ScalaTion from the command line helped familiarize me with these common procedures in lieu of formal training. So don’t give in to apprehension; get your hands dirty!</li></ul><pre>scp -r /Applications/scalation_1.4/scalation_models/lib/ ./lib/</pre><p>We need to copy the .jar files in the lib folder of the scalation_modeling package into the lib folder of our project. The .jar extension stands for Java ARchive, and these files typically hold a number of packages and class files together for easy distribution. This step is performed using the secure copy command scp.</p><h4>Learning from ScalaTion</h4><p>As you begin to explore ScalaTion, note that Dr. Miller has included on the library <a href="http://cobweb.cs.uga.edu/~jam/scalation.html">homepage</a> a number of textbooks/manuals to accompany the library. The most useful one for me so far has been <a href="http://cobweb.cs.uga.edu/~jam/scalation_guide/data_science.pdf">Introduction to Data Science Using ScalaTion</a>, which explains the linear algebra, statistics, calculus, and machine learning concepts that inspire many of the library’s classes and methods. Additionally, many of the classes include thorough documentation and example Test objects which demonstrate the use of the class and its methods. The data folder within the main scalation_1.4 folder contains toy datasets for this purpose and for your own usage.</p><h4>Conclusion</h4><p>You have now successfully set up an environment for building powerful machine learning models using the ScalaTion library. In Part 2 of this tutorial we run our first standalone program using ScalaTion, which will read in some data and run a simple model. Then, in Part 3 we’ll explore the code, and you’ll be ready to start your own machine learning projects :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=478cdc13470e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-quarks/an-introduction-to-scalation-part-1-setting-up-478cdc13470e">An Introduction to Data Science via ScalaTion Part 1 — Setting Up</a> was originally published in <a href="https://medium.com/the-quarks">The Quarks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>