<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nick Harvey on Medium]]></title>
        <description><![CDATA[Stories by Nick Harvey on Medium]]></description>
        <link>https://medium.com/@nicksharvey?source=rss-6f5e92e949e2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4fDwCfsiVZlvFly2cCjjww.jpeg</url>
            <title>Stories by Nick Harvey on Medium</title>
            <link>https://medium.com/@nicksharvey?source=rss-6f5e92e949e2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:11:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nicksharvey/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing Pachyderm v1.9 — Support For Streaming Data and More!]]></title>
            <link>https://medium.com/pachyderm-data/introducing-pachyderm-v1-9-support-for-streaming-data-and-more-4fb96317fab7?source=rss-6f5e92e949e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fb96317fab7</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[enterprise-technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nick Harvey]]></dc:creator>
            <pubDate>Mon, 17 Jun 2019 20:37:27 GMT</pubDate>
            <atom:updated>2019-06-17T20:37:27.228Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing Pachyderm v1.9 — Support For Streaming Data &amp; More</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4u-z6bYMIg9q3ssbSRUQw.jpeg" /></figure><p>The Pachyderm team is proud to announce our first major release of 2019 — Pachyderm v1.9. Since we released v1.8 back in November of 2018 we’ve been hard at work, and v1.9 has the feature list to show for it. Let’s dive right in, there’s a lot to cover.</p><h3>Support For Streaming Data</h3><p>With v1.9 we’ve introduced a new Pachyderm concept called Spouts. The purpose of Spouts is to introduce the same provenance that Pachyderm provides for batch data to streaming data. And as far as we know, Pachyderm is the first data science platform to do it. With Spouts, teams can continuously ingest streaming data from platforms like <a href="https://kafka.apache.org/">Kafka</a>, <a href="https://nifi.apache.org/">Nifi</a>, and <a href="https://www.rabbitmq.com/">RabbitMQ</a> directly into Pachyderm and use that data in their pipelines.</p><p><a href="https://github.com/pachyderm/pachyderm/tree/master/examples/spouts/EmailSentimentAnalyzer">You can check out a Pachyderm Spouts example here</a></p><h3>Pachyderm S3 Gateway</h3><p>Back in February, <a href="https://pachyderm.io/2019/02/19/4-Reasons-to-Get-Excited-About-Pachyderm-in-2019.html">we mentioned</a> that in 2019 Pachyderm will have new ways to access and manage data. Pachyderm v1.9 is the first major step towards honoring that promise and users can rejoice in the fact that they can now interact with their data directly instead of being constrained to the Pachyderm API. This will save users from having to build in extra pipeline steps to egress data out of Pachyderm just so it can be consumed by external resources.</p><p><a href="http://docs.pachyderm.com/en/latest/enterprise/s3gateway.html">Details on the Pachyderm S3 Gateway can be found here</a></p><h3><strong>Transactions — Batch Your pachctl PFS Commands</strong></h3><p>Another exciting addition in this release is a new set of Pachyderm commands called transactions. This new level of functionality enables users to not only batch multiple pachctl commands into a single transaction but also have those changes be applied atomically across multiple Pachyderm elements.</p><p><a href="https://docs.pachyderm.io/en/latest/managing_pachyderm/Batching_Pachyderm_with_Transactions.html">More info on Pachyderm Transactions can be found here</a></p><h3><strong>Expanded Error Handling</strong></h3><p>In some cases, flaws or gaps in your data are expected and even anticipated. In these scenarios, teams need the ability to construct pipelines where expected errors won’t cause the entire pipeline to come to a halt. Thankfully, as of v1.9, users will be able to configure pipelines with error handling code that will instruct how the pipeline should proceed when facing certain errors.</p><p><a href="http://docs.pachyderm.com/en/latest/reference/pipeline_spec.html#transform-required">You can find more details on how to set this up here.</a></p><p>As always you can <a href="https://pachyderm.com/requestdemo.html">book your own private demo of Pachyderm</a> or you can <a href="https://github.com/pachyderm/pachyderm">check out our github page</a> for more information.</p><p>This blog was republished from <a href="https://pachyderm.io/blog/blog.html">https://pachyderm.io/blog/blog.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fb96317fab7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pachyderm-data/introducing-pachyderm-v1-9-support-for-streaming-data-and-more-4fb96317fab7">Introducing Pachyderm v1.9 — Support For Streaming Data and More!</a> was originally published in <a href="https://medium.com/pachyderm-data">Pachyderm Community Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Survive The GoT Winter With ML and Pachyderm]]></title>
            <link>https://medium.com/@nicksharvey/survive-the-got-winter-with-ml-and-pachyderm-c09bfd1233a5?source=rss-6f5e92e949e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c09bfd1233a5</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[game-of-thrones]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nick Harvey]]></dc:creator>
            <pubDate>Tue, 26 Mar 2019 13:36:40 GMT</pubDate>
            <atom:updated>2019-03-26T13:36:40.954Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zn7FWfkXkyM2nFB9tWcfHw.jpeg" /></figure><p>The series finale of Game Of Thrones is almost upon us and over here at Pachyderm, we couldn’t be more excited. I mean come on, what’s not to be excited about right? Showrunners David Benioff and D.B. Weiss did a fantastic job in series 7 tieing up loose ends and setting series 8 to be nothing short of monumental.</p><p>However, we’d be lying if we said we weren’t lamenting the end of such an epic series at least a little bit. Sure, we can grasp on to a tiny bit of hope knowing <a href="https://www.digitalspy.com/tv/ustv/a835085/game-of-thrones-prequel-spin-off-show-cast-date-ideas/">this isn’t actually the end of the show</a>, and sure it’s possible George RR Martin could finish the series sometime in the next decade or two — as long as his<a href="https://en.wikipedia.org/wiki/A_Song_of_Ice_and_Fire#Writing_process"> DOS computer loaded with WordStar 4.0 </a>doesn’t crash and he lives that long. But if Mr. Martin has taught us anything it’s nothing is for certain and there’s nothing we can do about it. Or can we?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FN0v65hGymCUpy%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fgame-of-thrones-tyrion-lannister-bronn-N0v65hGymCUpy&amp;image=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FN0v65hGymCUpy%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="284" frameborder="0" scrolling="no"><a href="https://medium.com/media/30e89628ad287d64b0d2007f6f5cf958/href">https://medium.com/media/30e89628ad287d64b0d2007f6f5cf958/href</a></iframe><p>While none of us here can wield a pen like Martin, Benioff, or Weiss it did get us thinking. What if we could get a build a machine learning pipeline in <a href="http://www.pachyderm.com/">Pachyderm</a> to generate scripts for us? What if we could use a recurrent neural net to take the shows own episode scripts as input, and train a model to generate new episodes. So, as our own little tribute to what is and will always be a fantastic show, we present to you a Pachyderm example that you can use to generate GoT scripts. Sure they won’t get you an Emmy anytime soon, but it will help keep your mind off the fact that there are still 20+ days to go.</p><h4>Building a RNN to generate new GoT episode scripts:</h4><p><a href="https://github.com/pachyderm/pachyderm/tree/master/examples/ml/rnn">https://github.com/pachyderm/pachyderm/tree/master/examples/ml/rnn</a></p><p>And if you want a video demonstration of this example, <a href="https://www.bigmarker.com/pachyderm/From-Zero-To-Production-ML-in-30-mins-using-Loodse-Pachyderm">you should check out the webinar we did a couple of weeks back</a> with our friends over at <a href="https://www.loodse.com/">Loodse</a>.</p><p><em>Originally posted on </em><a href="https://pachyderm.io/blog/blog.html?__hstc=27691939.a27b38b0bb5020d50e4990688ea0317b.1550156839242.1550605621191.1553541032852.8&amp;__hssc=27691939.2.1553541032852&amp;__hsfp=2558228284"><em>https://pachyderm.io/blog/blog.html</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c09bfd1233a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 Reasons to Get Excited About Pachyderm in 2019]]></title>
            <link>https://medium.com/pachyderm-data/4-reasons-to-get-excited-about-pachyderm-in-2019-cb65b8072462?source=rss-6f5e92e949e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb65b8072462</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nick Harvey]]></dc:creator>
            <pubDate>Mon, 25 Feb 2019 16:02:49 GMT</pubDate>
            <atom:updated>2019-02-25T16:05:04.292Z</atom:updated>
            <content:encoded><![CDATA[<p>Written by: Joe Doliner, CEO &amp; Co-Founder of <a href="https://www.pachyderm.io/">Pachyderm</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uz3kvtR1jgg9q9BhbPgtsg.jpeg" /></figure><p>2018 was an exciting year for <a href="https://www.pachyderm.io/">Pachyderm</a>. The growth we saw for the project, user base, and the company has surpassed our expectations. To give you an idea, <a href="https://github.com/pachyderm/pachyderm">Pachyderm’s codebase</a> had more than 1,800 commits added, 1.5 million lines changed, and 24 new external contributors. We saw a spike in community generated content as well. Key examples include an excellent and <a href="https://szeitlin.github.io/posts/engineering/pachyderm-vs-airflow/">in-depth blog written by Dr. Samantha Zeitlin</a> the lead data scientist at Denali publishing and a <a href="https://www.youtube.com/watch?v=gwg_YXyCYBw">great talk on modern data science</a> delivered at Go Northwest by <a href="https://twitter.com/samkreter">Sam Kreter</a>, a software development engineer at Microsoft.</p><p>Furthermore, we had more than one thousand new users joining our <a href="http://slack.pachyderm.io/">support channels</a> to ask questions and provide others with advice. It’s very rewarding for us to see the momentum build around Pachyderm and based on what we observed in 2018, it would seem we’re well on our way to building a sustainable open source community.</p><p>Community momentum also fueled Pachyderm’s commercial growth in 2018. While helping numerous customers reshape data science within their organization; Pachyderm released <a href="https://github.com/pachyderm/pachyderm/releases">two major versions</a> of the product, <a href="https://www.pachyderm.io/use_cases.html">created a few case studies</a>, and welcomed five new team members. We even managed to <a href="https://pachyderm.io/2018/11/15/Series-A.html">raise some money</a>. Needless to say, we’re thrilled. But that was <em>sooo</em> last year…</p><p>Here’s a quick look at what you can expect from the Pachyderm project this year</p><h3>1. Expanded Pipelines and Native Kubernetes</h3><p>Pachyderm pipelines will be extended to support even more complex workflows and heavier workloads. This will enable us to enhance Pachyderm pipelines while also tightening our integration with <a href="https://kubernetes.io/">Kubernetes</a> and the surrounding ecosystem:</p><ul><li>Pachyderm pipelines will be implemented as K8s CRDs, essentially making Pachyderm pipelines behave like any other native Kubernetes object</li><li>Better integration with Kubeflow and its components (TFJobs, PyTorch training, etc.)</li><li>Support for more sophisticated data flows such as loops for iterative training</li><li>Templating of pipelines</li><li>One-off pipeline jobs</li><li>Greater control over which pipeline steps get persisted for faster experimentation and more granular storage control for versioned data</li></ul><p>You can find out more information here: <a href="https://github.com/pachyderm/pachyderm/issues/3345">https://github.com/pachyderm/pachyderm/issues/3345</a></p><h3>2. New Ways to Access and Manage Data</h3><p>This year we want to expand the many different ways you can integrate data into and out of Pachyderm:</p><ul><li>The Pachyderm filesystem (PFS) will now be accessible directly through an S3-compatible API, making versioned data in PFS just as accessible as if it were in an S3 bucket, but still with the powerful versioning semantics of Pachyderm</li><li>Kubernetes CSI (Container Storage Interface) to mount PFS data into any K8s pod</li></ul><p>This one is already underway and you can follow along here: <a href="https://github.com/pachyderm/pachyderm/pull/3432">https://github.com/pachyderm/pachyderm/pull/3432</a></p><h3>3. Improved Storage Engine</h3><p>Performance is paramount when it comes to applied data science, full stop. In 2018, we made drastic <a href="https://www.pachyderm.io/2018/11/15/performance-improvements.html">performance improvements</a> to Pachyderm, and 2019 will be no different. Users will continue to see performance improvements for nearly every type of workload throughout the year.</p><h3>4. An Entirely New Way to Interact with Pachyderm</h3><p>This one we’re keeping a bit closer to the chest, as it’s too early to provide details. What we will say is that it’s a natural continuation of our mission to enable reproducible data science and facilitate collaboration, but on a global scale. How we’re going to do that is by making Pachyderm more accessible and eliminating infrastructure obstacles that stand in the way of real-world ML/AI.</p><p><em>Originally published on </em><a href="https://pachyderm.io/blog/blog.html"><em>pachyderm.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb65b8072462" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pachyderm-data/4-reasons-to-get-excited-about-pachyderm-in-2019-cb65b8072462">4 Reasons to Get Excited About Pachyderm in 2019</a> was originally published in <a href="https://medium.com/pachyderm-data">Pachyderm Community Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Powerful Yet Affordable Machine Learning Rig for $2k]]></title>
            <link>https://medium.com/@nicksharvey/a-powerful-affordable-machine-learning-rig-for-2k-c96ce4bf16b8?source=rss-6f5e92e949e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c96ce4bf16b8</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nick Harvey]]></dc:creator>
            <pubDate>Fri, 27 Apr 2018 16:23:34 GMT</pubDate>
            <atom:updated>2018-04-27T20:40:01.463Z</atom:updated>
            <content:encoded><![CDATA[<p>For those who aren’t familiar the term “sleeper” it’s something that looks ordinary but is anything but stock under the hood.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FFRvaXWHJb4W5O%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Ftrippy-car-mustang-FRvaXWHJb4W5O&amp;image=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FFRvaXWHJb4W5O%2Fgiphy-downsized-large.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ad4a377da759d25aeb9cb313b6e96b9/href">https://medium.com/media/1ad4a377da759d25aeb9cb313b6e96b9/href</a></iframe><p>The reason I built this machine is that I often find myself wanting more power to test my machine learning models, but also not wanting to pay for/deal with the spinning-up and spinning-down of clusters in the cloud. Don’t get me wrong; I’m a huge fan of the cloud, especially, when it comes to machine learning and artificial intelligence. Therefore, what I need is an in-between solution, something affordable I could run at home but still had more power than my <a href="https://www.amazon.com/gp/product/B01AKYJMVI/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B01AKYJMVI&amp;linkId=7562d09fc1c9462ace582fbefeca29df">Lenovo yoga 14</a>. And it needed to be a “sleeper” because I didn’t want a machine that sounded like a jet engine in my living-room.</p><p>And so, I set out to build a more powerful machine that I could run from home, deploy my models too, and not break the bank. A few requirements that I set out for myself are as follows:</p><ol><li>It needs to be affordable and pay for itself in less than a year.</li><li>It needs to have a robust GPU capable of handling ML/AI workloads.</li><li>It needs to be quiet and small enough to not to be noticeable.</li></ol><p>Another thing I wanted to keep in mind before I got started was, by building such a machine, I knew that I was going to be adding in an additional step to my machine learning development process. It’s going to be another environment that will require systems administration work, troubleshooting when things break, which meant less time to work on my models. Therefore, to optimize my level of utility, I needed to build something that wouldn’t get in the way and wouldn’t end up ultimately cost me more time and money.</p><p>Thankfully, after a lot of research and hacking, I was able to come up with a pretty damn close to a perfect solution. I say pretty damn close because the software side of it still needs some work (more of that in a future post). But for now, let’s start with the hardware. It’s the foundation of the project, and therefore everything needed careful consideration. This is what I ended up deciding on:</p><p>Processor: <a href="https://www.amazon.com/gp/product/B06X3W9NGG/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B06X3W9NGG&amp;linkId=2d9acd5e17a1af0d1fed85942bccd2ec">AMD Ryzen 7 1700X</a> ~ $278</p><p>Motherboard: <a href="https://www.amazon.com/gp/product/B073BGMHH9/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B073BGMHH9&amp;linkId=b2c2076e7d46bf9c7569a1b758186c43">ASRock Mini-ITX X370</a> ~ $175</p><p>RAM: <a href="https://www.amazon.com/gp/product/B015FY3BJ2/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B015FY3BJ2&amp;linkId=15d0d40b878c84d0fe6509dc1a3ffdf7">G.Skill 16GB (Dual Channel Kit) 3200MHz DDR4</a> ~ $200</p><p>Video Card: <a href="https://www.amazon.com/gp/product/B06Y15DWXR/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B06Y15DWXR&amp;linkId=68ed8d86d48830a98457ff481bc5135f">EVGA GeForce GTX 1080 Ti</a> ~ $1000</p><p>Disk: <a href="https://www.amazon.com/gp/product/B01F9G46Q8/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B01F9G46Q8&amp;linkId=5103b15202e4101131102257dcb0509e">SanDisk SSD PLUS 480GB Solid State Drive</a> ~ $125</p><p>Power Supply: <a href="https://www.amazon.com/gp/product/B00ALK3KEM/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B00ALK3KEM&amp;linkId=2ba5d311b050c18108a79e656fc194d4">Corsair CX Series 750 Watt</a> ~ $80</p><p>CPU Liquid-Cooling (Optional): <a href="https://www.amazon.com/gp/product/B06XWYY83S/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B06XWYY83S&amp;linkId=90a4829c32f859bf8137e181374315f2">NZXT Kraken X52</a> — $130</p><p>Case: <a href="https://www.amazon.com/gp/product/B00HFRTF5W/ref=as_li_tl?ie=UTF8&amp;tag=nick032-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=B00HFRTF5W&amp;linkId=25171dce54b534fbcd2035931d11aec3">Corsair Obsidian 250D Mini ITX Case</a> ~ $80</p><h4>Total: $2068</h4><p>Not bad…. For around ~2000 bucks you can have a pretty badass machine learning rig that’s small enough to put under your desk, yet quiet enough not to wake the neighbors. Now, before you say it… yes, there’s newer/better hardware out there but, it’s also more expensive. However, If budget isn’t a concern for you, feel free to expand and tweak the GPU / CPU accordingly.</p><h3>Results</h3><p>I found that this particular combo gave me the power I was looking for at a price that was more affordable than those ultra-powerful laptops out there. The Ryzen 7 1700x CPU provides plenty of processing power and I find myself hardly ever maxing this thing out. The EVGA 1080ti is a beast of a card for sure and 3584 cuda cores means this machine walks tall and carries a big stick. Having 16 GB of DDR4 ram makes loading larger datasets a breeze. And the whole thing stays pretty cool and quite with the liquid-cooling for the CPU and plenty of airflow for the GPU. Needless to say; I’m thrilled with the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dveg_er3awedA1HE4eX31Q.jpeg" /><figcaption>Simple and sleek from the front</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4n9Y3UaN0nw12lub4iNtRA.jpeg" /><figcaption>Plenty of airflow for the beast within</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cGcyEisYVrKD3HM5p3q0qA.jpeg" /><figcaption>In case you’re wondering, yes the GTX 1080ti does fit in this mini-itx case, but barely.</figcaption></figure><p>So, for $2068 dollars, I now have a machine learning rig that’s powerful and damn near perfect for the early/intermediate testing of models . If you’re interested in the benchmarks, <a href="https://blog.slavv.com/@slavivanov?source=post_header_lockup">Slav Ivanov</a> wrote an <a href="https://blog.slavv.com/picking-a-gpu-for-deep-learning-3d4795c273b9">excellent article diving into the benchmarks of the1080ti</a> and a few other cards. Definitely check it out if you’re interested in the numbers ( I certainly was).</p><h4><strong>UPDATE</strong></h4><p>I started this blog post a couple of weeks ago, and since then I was fortunate enough to win a <a href="https://twitter.com/nvidia/status/984479414107783169">Titan V</a> from Nvidia at their last GTC conference in San Jose. Needless to say I was shocked about wining, and as soon as that thing showed up on my doorstep I dropped it into my machine. Surprisingly, it actually fits better than the 1080ti and it used the same drivers, lucky me! You can count on a separate post about this best of a card coming very soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TQ5zwu9qG4rw_cHx4XphDw.jpeg" /></figure><p>Now the hardware is complete, next up comes the software….</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c96ce4bf16b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RiseML : The answer to enterprise machine learning]]></title>
            <link>https://medium.com/@nicksharvey/riseml-the-answer-to-enterprise-machine-learning-a7578c879e99?source=rss-6f5e92e949e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7578c879e99</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[riseml]]></category>
            <dc:creator><![CDATA[Nick Harvey]]></dc:creator>
            <pubDate>Thu, 08 Feb 2018 20:34:04 GMT</pubDate>
            <atom:updated>2018-02-08T20:34:04.970Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the analogies I use to explain enterprise IT to people is that it’s like a classical music orchestra. Think about it for a second. An orchestra has musicians, instruments, a composer, and some sheet music. An enterprise IT organization has a similar structure.</p><p>The musicians are the different roles in IT. The instruments are the tools and platforms. The composer, well, that could be the architect or the department head. The sheet music is the process they use to build, ship, and run applications.</p><p>They also have similar goals in mind. An orchestra’s goal is to create or play a symphony. An IT organization’s goal is to do the same thing — but instead of music, it’s shipping an app. Either way, both are utilizing their tools and their talent in the best ways they can. Both are trying to create something cohesive, with purpose, and appeal to their audience.</p><p>And, like an orchestra, any IT department knows that the hardest thing to do is get everyone on the same page, let alone playing the same song and at the right key. Recent advancements in technology and the processes involved has made things easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*UDeQNXPXAtjDafxIVoSqdQ.gif" /></figure><p>Containers have made developing, deploying, and running software simpler and more consistent. DevOps have made the lifecycles of applications more adaptable, responsive, and resilient. And in order for machine learning to function at the enterprise IT level, it must be able to amalgamate with how the rest of the organization deploys applications. Whether you agree or disagree, it’s reality.</p><h4>Machine Learning: a new set of instruments</h4><p>There’s an new section getting introduced to the orchestra: machine learning. That’s because data scientists and the instruments they use have emerged as practical tools — tools that provide quantifiable business benefits and are therefore, moving deeper into the enterprise. According to a report published by Narrative Science entitled <a href="https://narrativescience.com/Portals/0/Images/PDFs/OutlookOnAI2018_NarrativeScience.pdf">Outlook on Artificial Intelligence in the Enterprise 2018</a>, “AI adoption grew by over 60% in 2017”. That’s a tremendous increase in traction.</p><p>If we return to our analogy, think about what it was like when the brass section was introduced to the orchestra. The instruments themselves weren’t new. Horns had been used centuries before composers like Beethoven. (Who was the first major composer to incorporate a brass section in his symphonic works, by the way.</p><p>However, before the brass section could be accepted, two things needed to change. The first thing that needed to change was musical tastes. Second, the instruments themselves needed to improve. Machine Learning is in a similar spot. It’s been around for a long time. And it has seen staggering improvement. More importantly, if reports like the one mentioned above are proof, tastes have evolved as well. The challenge is writing this new section into the music to improve the overall symphony.</p><h4>Introducing RiseML: the simplest way to adapt machine learning into enterprise IT</h4><p>This is where RiseML comes into play. RiseML makes it incredibly easy for IT organizations to implement the technology with little to no disruption. RiseML provides a simple yet powerful abstraction for machine learning engineers that leverages kubernetes, an existing container orchestration engine used by numerous IT departments. The beauty of RiseML is not just the tech itself, but the fact that it that hides the working details of the underlying cluster. <strong>It eliminates the need for the data scientist to become an ops person.</strong></p><p>Making things simple is paramount for success. And that’s where RiseML shines. Containers and DevOps helped solved a very prominent issue in IT, which is that Ops doesn’t like developing applications, and developers don’t like being bogged down by system administration. In the same vein, a data scientist doesn’t like doing either of those things. They’re interested in the data, and validating hypotheses with that data. In other words, woodwind wouldn’t do well playing a brass instrument, and someone from the brass section wouldn’t do well in the woodwind section.</p><p>RiseML frees up the data scientist so he or she doesn’t have to bother with either of those things. It does it in such a way that the data scientist can play along with everyone else without distributing existing processes. It creates a seamless way for the data scientist to add value to businesses.</p><p>To get a sense of what I mean, lets take a look at the RiseML architecture.</p><h4>The answer is in the RiseML Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bEdXcHF750etfuq04DCZqw.png" /></figure><p>RiseML has three main components: The CLI, the backend component, and the Kubernetes container orchestration platform. And it’s the harmonious — see what I did there? — integration of these three areas that makes RiseML such a powerful solution for any organization looking to adapt or further their usage of machine learning in their business.</p><h4>RiseML — The CLI</h4><p>The CLI abstracts a lot of the tasks associated with interacting with an orchestration engine like Kubernetes. This means the data scientist can stay in their area of expertise, without disrupting the ops environment. It leverages kubernetes, something your ops person is already familiar with. Plus, the RiseML CLI is couched in language the data scientist is already familiar with (like “train”, “status”, etc.). In order for the data scientist to run an experiment, all they need to do is run “<em>riseml train</em>”. Beautiful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*UbHjUFALQBoEggqLnlhMEQ.gif" /></figure><h4>Rise ML — The Backend</h4><p>The backend portion of the RiseML solution is series of components installed on top of the kubernetes platform. The backend automatically takes care of things like versioning, executing experiments, collecting logs, and more. Also, the backend has a full REST API that is addressable. Thus, it can be extended further to fit individual needs if necessary.</p><p>This critical component is what makes things relatively seamless. The data scientist doesn’t have to spend days or weeks trying to hack their own environment together. This becomes especially important when the lifecycle of the ML experiment comes into play.</p><p>Operations teams will love this, because it ensures that there isn’t fragmentation. Machine learning in itself is resource-intensive. It’s this backend component that will ensure workloads remain sequential. Therefore, it won’t drive up additional costs by consuming unnecessary resources across various environments.</p><h4>RiseML — GPU enabled Kubernetes</h4><p>Part of the genius of the RiseML solution is that it doesn’t add a custom orchestration engine into the mix. This means that there isn’t an additional layer of complexity into the mix left to the ops person to maintain. GPU-enabled kubernetes is possible on its own, but as of today, requires a lot of customization. RiseML has taken care of all that for you. Data scientists can install and deploy a fully capable cluster, including the additional RiseML components, with just one command (additional blog post to follow).</p><h3>Summary</h3><p>In closing, if you’re a company that is exploring or already using machine learning and are wondering how to do it at scale and with minimal risk, RiseML is definitely something you need to check out. Adopting a new technology can be a daunting task, and make it seem like you’re taking on a lot of additional risk. But it doesn’t have to be that way. RiseML has created a solution that minimizes the risks and challenges of implementing data science into your existing IT environment, while leveraging existing technologies you’re likely already using.</p><p>“The music” is changing, and solutions like this are rare. Leveraging something like RiseML may be exactly what you need to be the next Beethoven.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7578c879e99" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Machine Learning to Predict the S&P 500 with ~18 lines of python]]></title>
            <link>https://medium.com/@nicksharvey/using-machine-learning-to-predict-the-s-p-500-with-18-lines-of-python-b00777928185?source=rss-6f5e92e949e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b00777928185</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nick Harvey]]></dc:creator>
            <pubDate>Tue, 14 Mar 2017 18:46:00 GMT</pubDate>
            <atom:updated>2017-03-14T18:51:29.532Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Fu6BkTz4RLTeJx3ZfOoMg.png" /></figure><p>The folks at Facebook recently released a fascinating new machine learning project called <a href="https://facebookincubator.github.io/prophet/">Prophet</a>. The goal of prophet is to make the task of creating accurate forecasting models function at scale while taking into account all the seasonality variables (holidays, etc.). I’ve got another blog in the works that goes into what “at scale” means with ML &amp; AI (machine learning and artificial intelligence) but more on that later.</p><p>I’ve been diving into the project for a couple of weeks now and I’m already finding tons of business problems I could apply this to and will come in handy quite a bit. But, after getting a little more comfortable with it, I decided to have some fun with it as well.</p><p>In less than 18 lines of code I was able to produce the graphic above that shows the performance of the S&amp;P 500 since 2014 all the way until 2017. The graph then <em>predicts</em> what the next 30 days might look like. Now, obviously no one could predict the exact future, that’s impossible (for now….). Instead it uses very interesting — and might I say beautiful - math to identify patterns and then make reasonable estimations from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*CbFUFvRByw0g0XhmsCuamw.png" /><figcaption>Piecewise Logistic Growth Model from the Prophet Paper <a href="https://facebookincubator.github.io/prophet/static/prophet_paper_20170113.pdf">here</a></figcaption></figure><p>This isn’t the exact formula used in the example above. It’s just an example from of what elegant math is in <a href="https://facebookincubator.github.io/prophet/static/prophet_paper_20170113.pdf">Prophet</a>. Anyhow, onto the good stuff….</p><p>Here’s my python code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df01ca94dec65277398687d0a0f8b6a4/href">https://medium.com/media/df01ca94dec65277398687d0a0f8b6a4/href</a></iframe><p><strong>Lets walk through it shall we.</strong></p><p><strong>Line 1–4:</strong> Import the necessary libraries</p><p><strong>Line 6:</strong> Identify our datafile using a csv file with the S&amp;P500 time-series data [<a href="https://drive.google.com/open?id=0BxfedcgRnB_BcU1tVVhSeE1iMlk">link to that file is here</a>].</p><p><strong>Line 7:</strong> the y column (which holds the value for the S&amp;P for that given day) needs to have the log-transform applied. which is just a fancy way to make things that are highly skewed, less skewed.</p><p><strong>Line 9:</strong> Assign the Prophet model we want to use (in this case linear) to a variable.</p><p><strong>Line 10:</strong> fit the data using Prophet</p><p><strong>Line 12:</strong> define the future in this case 30 days ( I feel weird just typing that)</p><p><strong>Line 13:</strong> Predict the future!!!</p><p><strong>Line 15–17:</strong> Plot out the graphs (which are actually fantastic looking btw Facebook team)</p><p>That’s it, in just 17 lines of code (really 13 if you remove the spaces) you were able to graph out the history of the S&amp;P since 2014 and then predict what the next 30 days might look like. Now go call your broker and try to not to sound like a crazy person.</p><p>If you find yourself interested in learning more about this stuff and some of the theory used. Check out this great book and dive in <a href="https://www.amazon.com/gp/product/1107057132/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=1107057132&amp;linkCode=as2&amp;tag=nickharvey-20&amp;linkId=e329c624f479cc5994d8632228094ab6">Understanding Machine Learning: From Theory to Algorithms</a></p><p><strong>Obligatory disclaimer</strong></p><p>Just so we’re clear, I’m not able to predict the future and you shouldn’t take my word and invest your savings, there are professionals who do that for you, go get your advice from them. I just wanted to show you a cool machine learning project you could do in 10 minutes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b00777928185" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>