<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex Ragalie on Medium]]></title>
        <description><![CDATA[Stories by Alex Ragalie on Medium]]></description>
        <link>https://medium.com/@aragalie?source=rss-6a406fdee048------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jm0j54aZdhibXAcsWxHFWg.png</url>
            <title>Stories by Alex Ragalie on Medium</title>
            <link>https://medium.com/@aragalie?source=rss-6a406fdee048------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:58:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aragalie/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[FX price prediction using state of the art image recognition Machine Learning algorithms]]></title>
            <link>https://medium.com/@aragalie/fx-price-prediction-using-state-of-the-art-image-recognition-machine-learning-algorithms-c4be2b92663?source=rss-6a406fdee048------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4be2b92663</guid>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fastai]]></category>
            <dc:creator><![CDATA[Alex Ragalie]]></dc:creator>
            <pubDate>Thu, 29 Mar 2018 09:54:45 GMT</pubDate>
            <atom:updated>2018-03-29T09:56:35.285Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xpl8yIdd4OKznGrz8EJTIA.png" /><figcaption>Photo Credits <a href="https://medium.com/u/6a406fdee048">Alex Ragalie</a></figcaption></figure><p>I’ve spent the past few days working on a simple AlgoTrading question: can the current best-in-class image recognition algorithms, which have now +99% accuracy, be used to predict “winning” and “losing” price chart configurations instead of cats and dogs?</p><p>This exercise was also done as part of the work i’m doing in the fast.ai courses. Currently i’m retaking the class in their new v2 version (using PyTorch instead of Keras), so i thought this would be a good bridge between my interests and the course materials.</p><p>At the end of Lesson 1, Jeremy is suggesting that you take your own pictures which need to be sorted correctly and try to have the algorithm predict the class. He makes a special note that the algorithm is well suited for things like cats and dogs, and not necessarily for other types of images (something which we’ll talk about later in the post).</p><p>I’ve therefore gone ahead and built an entire pipeline (excluding the trading engine part) in Python:</p><p>- it gets FX price data (i’m using Oanda.com)</p><p>- then it plots it according to specific “viewPorts” (60 minutes of price data shown as a line, with 1 minute data increments)</p><p>- saves each plot as a .png file, naming it according to its class (buy, sell or hold) and also places it in the right folder inside either “train” or “valid” folder</p><p>- it also takes into account <a href="http://www.fast.ai/2017/11/13/validation-sets/">Rachel’s excellent advice</a> on how to set-up train and valid datasets properly</p><p>- then the data is run through the fastAI Lesson1 image recognition model</p><p>All the code can be found on my <a href="https://github.com/aragalie/FX_NaiveChartPredictions_using_fastai">Github</a> , presented in Jupyter Notebooks with explanations for each step and code section.</p><p>CONCLUSION</p><p>At this stage the accuracy rate is random at best (~50%), which is to be expected for the specifics of my dataset (granular FX data with little denoising work done). I knew this from the beginning, but it was a useful exercise none the less.</p><p>Hope it will help anyone else trying to do a similar experiment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4be2b92663" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build your own top-spec remote-access Machine Learning rig: a very detailed assembly and…]]></title>
            <link>https://medium.com/@aragalie/build-your-own-top-spec-remote-access-machine-learning-rig-a-very-detailed-assembly-and-dae0f4011a8f?source=rss-6a406fdee048------2</link>
            <guid isPermaLink="false">https://medium.com/p/dae0f4011a8f</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[gtx-1080ti]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Alex Ragalie]]></dc:creator>
            <pubDate>Thu, 02 Nov 2017 16:05:28 GMT</pubDate>
            <atom:updated>2017-11-06T10:41:12.182Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Build your own top-spec remote-access Machine Learning rig: a very detailed assembly and installation guide for a dual boot Ubuntu 16.04/Win 10 with CUDA 8 run on i7 6850K with 2x GTX 1080Ti GPUs</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNg-LBnL3X_2ZUDzRDS7ug.jpeg" /></figure><p>I will start by saying that there are already several very good (and recent) guides on how to install a Machine Learning rig from scratch, and I have used them heavily to set-up my own. See <a href="https://medium.com/towards-data-science/building-your-own-deep-learning-box-47b918aea1eb">this one</a>, <a href="https://medium.com/towards-data-science/build-a-deep-learning-rig-for-800-4434e21a424f">this one</a>, <a href="https://blog.slavv.com/the-1700-great-deep-learning-box-assembly-setup-and-benchmarks-148c5ebe6415">this one</a> and <a href="http://guanghan.info/blog/en/my-works/building-our-personal-deep-learning-rig-gtx-1080-ubuntu-16-04-cuda-8-0rc-cudnn-7-tensorflowmxnetcaffedarknet/">this one</a>. My sincere thanks goes to them for having taken the efforts to share their experience, as well as to everyone else in the space who is helpfully sharing their knowledge and working code.</p><p>The main reason I’m publishing this guide is because my specific configuration had some “quirks” during the install, which took many days to figure out and fix, so hopefully I’ll save some time for you in case you go with something similar. Based on the extensive research i’ve done before building it, this seems to be amongst the best money-to-benefit ratio rigs that you can buy on the top-end of the price range.</p><p>Here is the config:</p><ul><li>CPU: Intel Core<strong> i7 6850K</strong> (LGA 2011-v3, 3.60GHz, Unlocked)</li><li>Motherboard: Asus<strong> X99 Deluxe II</strong></li><li>RAM: Corsair DDR4 Vengeance LPX Black <strong>32GB 2-Kit 2x 16GB 3200M</strong></li><li>GPU: 2x ASUS GeForce<strong> GTX 1080 Ti</strong> STRIX</li><li>SSD: Samsung <strong>960 EVO 1TB</strong></li><li>PSU: Corsair <strong>HX1000</strong></li><li>CPU Cooler: Cooler Master Hyper 212 Evo (will be later on replaced by a full CPU+GPUs watercooled setup from EK)</li><li>Case: Phanteks Enthoo Evolv ATX</li><li>Monitor: Acer Predator XB271HU (27&quot;, 2560x1440) connected to the main (first PCIE slot) GPU via one of the DP slots</li><li>Keyboard and mouse: normal USB ones</li></ul><p>In the interest of flexibility and the occasional gaming session (this is also a top-notch gaming rig after all!) I chose to go with a dual boot <strong>Ubuntu 16.04</strong> and <strong>Windows 10</strong> setup. I initially wanted to go with Win8 instead of 10, but installing the drivers on the boot USB for the Samsung nVME SSD was an absolute nightmare (and it didn’t work at all in the end).</p><p>So here are the specific steps I took. I’m also sharing the links that were most helpful for me in getting things running. In case you get stuck or something doesn’t work, your best bet is to Google it, then Google it some more using slightly different keywords, and the Google it some more. I’m confident that you’ll get to a workable solution in the end.</p><h3><strong>1. Pre-Assembly preparation</strong></h3><p><strong><em>Bootable UBS sticks</em></strong></p><p>You will need to create bootable USB sticks (if you have only one, I suggest you first create it to boot-install Windows 10 and then Ubuntu, and leave Ubuntu on it as you might need it later on); in my case I had one for Win and one for Ubuntu. Here is <a href="https://www.lifewire.com/ultimate-windows-7-ubuntu-linux-dual-boot-guide-2200653">a very good guide</a> on how to go about it; it says Windows7 but the creation of a bootable USB stick is identical for both Win7 and Win10.</p><p><strong><em>Antistatic bracelet</em></strong></p><p>One key thing to make sure that you do <strong>before you start unpacking and assembling the hardware</strong> is to make sure that you’re using a method to avoid static electricity buildup. Make sure that you’re not doing the process on a carpet, and ideally buy yourself a simple and cheap antistatic bracelet. This is a key precaution, as otherwise you can easily render useless your components.</p><p>Obviously I didn’t prepare for this beforehand, so after spending 1h searching for different DYI solutions, I ended up taking a 2m unused 3mm headphone cable I had lying around, clipping one end to the grill of the PSU (which had to be plugged in the mains outlet and I also made sure it was <strong>turned off!!!</strong>) and then had the other end always touching my wrist via 2 simple rubber bands used as bracelets on my left wrist.</p><p>You should also ensure that you have a good multi-head screwdriver, ideally with magnetic inserts so that your small screws are kept in place when positioning/taking them off.</p><p><strong><em>An undisturbed working environment with at least 1h timeslots of continuous focus</em></strong></p><p>Especially if you have a large family with kids running around, it’s important to mention that you need to allocate proper time<strong>, especially for the assembly stages</strong>. Some of the key steps will require ~1h of your undivided attention, after which you can take a break and do something else.</p><p>To set your expectations, the whole assembly process took for me about 5h (I’ve not done this before), followed by about 3h to install the operating systems, followed by about 1 week of tinkering until all the software libraries were in place and working.</p><p><strong><em>Prepare your working space</em></strong></p><p>The whole operation will require a large table, on which you can leave things lying around for a few days undisturbed. You’ll also have multiple boxes that need to be stored until you get everything fixed.</p><p>I also recommend you <strong>do not throw away your packing boxes for a couple of weeks</strong>. This will ensure that, in case a component dies on your or is DOA (dead on arrival), it will be relatively straight forward to send it back.</p><h3><strong>2. Hardware Assembly</strong></h3><p>I suggest you follow the steps in the below order and first assemble stuff on the motherboard outside of the case, and then mount it inside the case with the 9 screws once everything is fixed; only after that put in the GPUs.</p><p><strong><em>Note</em></strong>: I used the white cardboard protection the motherboard comes surrounded in (inside of the packaging box) as a working surface to lay the mobo on; this provided a semi-hard but also flexible surface on which I could push at will (and you’ll need to do some pushing, especially for screwing in the CPU cooler and for the RAM installation) without worrying that the back of the circuit board will be damaged.</p><p><strong><em>CPU installation in the motherboard</em></strong></p><p>A straightforward and simple process, with only one thing to watch-out for: <strong>the golden arrow on the CPU needs to match to the one engraved on the motherboard</strong>. One thing to also notice is that the 2 levers you need to open and then close require a degree of force as well, especially when closing it.</p><p><a href="https://www.youtube.com/watch?v=j6uLJKzycU8&amp;t=4s">Follow these instructions and you should be ok</a>.</p><p><strong><em>CPU cooler installation</em></strong></p><p>Important note here: I purposefully chose a cheap and easy to install/uninstall cooler, as I plan later on to have a full-cycle water-cooled system in place. I wanted none to less to have a working solution in place, in order to test all the components and ensure that both my processor and my GPUs were fine.</p><p>One other thing to note here: the level of noise from this machine is significant! When the GPU fans are running full-throttle it sounds like a small jet taking off, so you need to plan for this aspect. This is another reason why I’ll be switching to water-cooling, as the noise level + GPU performance (via overclocking) can be noticeably improved.</p><p>In terms of installing the cooler, it was slightly tricky, but nothing to worry about. My metal brackets came set for an AMD processor, so I just had to move them in the right position before they would fit on the slots for the Intel LGA 2011. In terms of steps, I recommend that you first do a dry-run and ensure that all the screw slots are matching and only then apply the thermal paste and start screwing it in.</p><p>And remember to take the sticker off from the bottom of the mounting slab!</p><p><a href="https://www.youtube.com/watch?v=H_QVWBm4CI4&amp;t=272s">Here is the guide which I found most helpful</a></p><p>And a few others guides:</p><p><a href="https://www.youtube.com/watch?v=_cFaXX2txAY&amp;t=600s">Cooler Master Hyper 212 EVO fan installation on INTEL 2011 plus CPU and RAM install — Part 5</a></p><p><a href="https://www.youtube.com/watch?v=xwDuhJkvfaM&amp;t=116s">Cooler Master Hyper 212 EVO — Install and Unbox | LGA 2011!</a></p><p><strong><em>SSD installation</em></strong></p><p>I’m amazed at the size of SSDs these days! This small thing needs to go vertically into the motherboard, so here’s a very helpful and fully accurate guide for how to do it:</p><p><a href="https://www.youtube.com/watch?v=1MCSwswg37g">Vertical M.2 SSD installation (Crazy CPU Machine 3/10)</a></p><p><strong><em>RAM installation</em></strong></p><p>This is very straight forward as well, with 2 main mentions: first<strong> make sure that the unlocking tab on the right side of the RAM slot is unlocked</strong> (cocked to the right side) for each of the RAM slots before you try installing. Second, <strong>you will need to push relatively hard to slot the RAM in </strong>(first onthe left side, and then on the right side to click into place). Push too hard and you break things though, so be mindful.</p><p>Here’s a guide : <a href="https://www.youtube.com/watch?v=Vib0-Pb77sk">Ram installation on a X99, quad channel, motherboard (Crazy CPU Machine 4/10)</a></p><p><strong><em>Case unpacking and setup</em></strong></p><p>I opened up the case, took off the front and the back plates, as well as the top one, to make sure that I had unhindered access to all sides. The whole thing will get quite heavy as you install things in, so my reco is to set-up things in such a way that you’ll have always a 360 degrees access around your case, avoiding the need to move it around for the different cable setup.</p><p><strong><em>PSU installation and power cables preparation</em></strong></p><p>This step is a bit challenging, especially if you’ve never done this before. It took me about 2h of doing/redoing to ensure that I had everything set-up properly, and it was still a heart wrenching moment when I powered it on :)</p><p>I suggest you do several dry-runs of setting up the cables and bringing them on the front side to see that everything fits nicely, and also that you watch the video multiple times to understand what needs to go where…</p><p>Here is a tutorial: <a href="https://www.youtube.com/watch?v=l3BEN2qG7tc">PSU installation &amp; Show off video :) (Crazy CPU Machine 10/10)</a></p><p>And this one is helpful as well: <a href="https://youtu.be/d4eivp7xeg0?t=32s">Connectors &amp; cabling (Crazy CPU Machine 8/10)</a></p><p>Some things which I would mention :</p><ul><li>Each of the GPUs takes in 2 separate PCI cables, for each of the power slots available. This means that you must have <strong>4 separate cables routed</strong> to the place where the 2 GPUs will be sitting.</li><li>I also decided to add the second power cable to the board for CPU overclocking. You must ensure beforehand that you’re selecting the proper splitting cable (as explained in the tutorial) for the job</li><li>Linked to the above point: there’s a slight but noticeable difference (once you know where to look) between the “normal” PCI cable and the CPU one, not only in the way it splits but also in the way it slots into the connectors. Make note of that and make sure you don’t force anything</li><li>Speaking of forcing cables in: the biggest pain to put in was the main Motherboard power cable. Due to the way in which my HX1000 cables came threaded together inside their “fancy” black housings (vs. flat as you see in the video), I had to do some significant bending and pushing/adjusting in order to have the cable slot into the motherboard and at the same time come to the back of the case in the place nearest to the motherboard. It’s hard to understand now, but you’ll see what I mean when you’ll have to do it. The same was the case for connecting the power cables to the GPUS, as they’re so closely stuck together..</li></ul><p>I suggest you do at this stage a quick power-on only for the motherboard, to ensure that it’s not DOA. Make sure it’s not touching anything metal and also make sure that there’s nothing that can fly into the fan of the CPU. Then connect the motherboard to the PSU via the main power cable and press the power button. Wait until the AA message shows on the LCD and then power it off.</p><p><strong><em>Install the Motherboard into the case</em></strong></p><p>Very simple, and you need just the 9 screws to do it. I initially didn’t know which are the proper screws (they come in the transparent plastic partitioned box inside the case packaging), so I went with the ones which have a slightly rounded head with a flat base.</p><p>My advice for this step would be to lay the case flat on its side and mount it in that way. Also, as the motherboard will have the CPU cooler already mounted, make sure you set it in it’s place on the case mounts <strong>very gently</strong>, ensuring that you’ve first mounted the front panel overlay (the “face” of the motherboard in the back of your computer) into the back panel of the case.</p><p>Here is a tutorial for this stage: <a href="https://www.youtube.com/watch?v=-J9VPyxsxAM">x99 Motherboard installation (Crazy CPU Machine 2/10)</a></p><p><strong><em>Various other connections to the motherboard:</em></strong></p><p>You will need to connect the CPU fan in the proper slot on the motherboard (the white one, as there’s a black next to it as well). In the black one you need to connect the case fans, with the proper black cable coming out of the fan bridge on the back of the case.</p><p>As well you’ll need to put in the RGB leds + the power on/Reset button connector of the case in the bridge provided with the motherboard (a small black thing with spikes on one side and text written on it).</p><p>Here is a guide for this part: <a href="https://youtu.be/d4eivp7xeg0?t=32s">Connectors &amp; cabling (Crazy CPU Machine 8/10)</a></p><p><strong><em>GPUs installation</em></strong></p><p>Remember to take out the protective black covers over the PCI bridges of each GPU before slotting into the motherboard.</p><p>The watch-out here is that you need to be mindful of the fact that the GPUs are very heavy, so I recommend you install them also with the case lying flat. As well, notice there will be very little space between the 2 GPUS, which makes things a bit complicated as well. Start with the upper one and move to the second one, and you should be fine.</p><p>Tutorial: <a href="https://www.youtube.com/watch?v=0VVSB7Q6Tvs&amp;t=105s">Video Card &amp; ThunderboltEX 3 (Crazy CPU Machine 9/10)</a></p><p><strong><em>Note</em></strong>: don’t even try to mount the Thuderbolt 3 card as per this video, because it won’t fit between the 2 GPUs</p><p>In terms of slots, I installed the first on slot 1 and the second on slot 3, as instructed by the manual. This ensures that you’ll be able to run both of them at x16 speed (highest available).</p><p>In case you ever need to take the GPUs out of the motherboard<strong>, make sure that before you pull on the GPU you first press the locking tab on the motherboard so it unlocks! </strong>This would be a VEEEEEERY tricking thing to do for the second GPU, as you literally have only a couple of centimeters of space available. I recommend you DO NOT use a screwdriver, because if your hand slips then you’ll stick it directly into the motherboard! What I did was to take a long-handled painting brush (which has a rounded tail end), and use that to press the tab. Just to get a feel for where you need to press to unlock it and with what force, I suggest you test it out on the upper GPU as with that there’s enough space.</p><p>Here’s a video to show you what I mean (didn’t find one that shows the X99 mobo though, but it’s the same concept): <a href="https://www.youtube.com/watch?v=Ysk8d_kRg0o">How to remove GPU</a></p><p>Install as well the SLI bridge that connects the 2 GPUS. In the X99 box there are 2 SLI bridges available, and I went with the nice-er looking one. FYI, the SLI bridge is only useful for gaming and nothing else, as CUDA uses only the PCIE interface due to the speed.</p><p><strong><em>Cabling and power-on</em></strong></p><p>Connect all the power and connection cables which are left (GPUs + anything else you might have), and then power it on! One note though: do not connect yet your monitor to the GPU via the DP cable (neither the keyboard or mouse), as we’ll do that later on.</p><p>Here is a again the same tutorial: <a href="https://www.youtube.com/watch?v=l3BEN2qG7tc">PSU installation &amp; Show off video :) (Crazy CPU Machine 10/10)</a></p><p>And this one is helpful as well: <a href="https://youtu.be/d4eivp7xeg0?t=32s">Connectors &amp; cabling (Crazy CPU Machine 8/10)</a></p><p>You need to wait for the motherboard to do it’s thing, and then you’ll see “AA” as a message (on the LED display next to the power-on button located on the mobo) which means you’re good to go!</p><p>I suggest that at this stage you put back the front and back panel (and the top one if you’ve removed it as well) and enjoy the blinking lights :)</p><p>WHEWWWW! That was quite some work, so give yourself a big pat on the back.</p><p>Now onward to the fun stuff.</p><h3><strong>3. Installing Windows10 and Ubuntu 16.04</strong></h3><p>Once you’re sure that your motherboard is working, connect your main GPU (the upper one) to the monitor and then plugin your keyboard and mouse. Also connect the WIFI antenna with it’s 3 golden connectors and/or the LAN cable in case you have a wired connection.</p><p>A few things to set your expectations for this stage:</p><ul><li>There will be MANY restarts and reboots involved</li><li>Hopefully for you, following this guide, things will work better the first time; for me it was a very frustrating process, involving many hours of googling and trying out stuff</li><li>My advice is to be patient and realize that you’re not the only one who has these issues, and everything generally has a solution</li><li>Keep in mind that <strong>most</strong> of the issues you will be facing at this stage of the installation can be equally due to the Win/Ubuntu as much as due to the <strong>BIOS</strong> of the motherboard. If you’ve nerver done this before, it will be a good intro into the wonderful world of UEFI and booting challenges :)</li></ul><p>Source and more info<strong>: </strong><a href="https://www.lifewire.com/ultimate-windows-7-ubuntu-linux-dual-boot-guide-2200653">https://www.lifewire.com/ultimate-windows-7-ubuntu-linux-dual-boot-guide-2200653</a></p><p><strong><em>Initial BIOS configuration</em></strong></p><p>The first time you start your computer you’ll get the BIOS screen. Things have come a very long way since the last time I saw a BIOS screen in the early 2000’s, so things are quite simple.</p><p>A few key things you must do:</p><ul><li>Update your BIOS via the internal tool (Advanced-&gt;Tool-&gt;ASUS EZ Flash Utility-&gt; select by Internet) ; it was straightforward for me, as it connects on its own to the Internet and downloads everything needed; do this step first as everything resets after update</li><li>Enable the XMP option, to get your RAM working at the full speed (main screen, middle left side)</li><li>Disable Fast boot (in the Boot menu)</li><li>Set your to OS Type to<strong> Windows UEFI mode </strong>(in the Boot/Secure Boot submenu); here’s a link with explanation <a href="https://www.technorms.com/45538/disable-enable-secure-boot-asus-motherboard-uefi-bios-utility">https://www.technorms.com/45538/disable-enable-secure-boot-asus-motherboard-uefi-bios-utility</a>; if you ever want/need to disable Secure Boot entirely, use this link as well as it has a very good explanation</li><li>Bear in mind that the above setting needs to be changed to <strong>Other OS</strong> when we’ll install and configure Ubuntu, and it should be kept on this setting even afterwards</li></ul><p><strong><em>Windows 10 installation</em></strong></p><p>Everyone suggests to start with Win instead of Ubuntu, and I think it’s a sensible suggestion. Make sure that you have enabled “Windows UEFI mode” in the BIOS before inserting the bootable USB stick.</p><ul><li>Insert the USB stick</li><li>Start the computer, press Del or F10</li><li>On the right side, go to “Boot Menu” or press F8</li><li>Select the USB stick and your computer will restart</li><li>Then the installation flow of Windows begins</li><li>Follow the tutorial here: <a href="https://www.lifewire.com/ultimate-windows-7-ubuntu-linux-dual-boot-guide-2200653">https://www.lifewire.com/ultimate-windows-7-ubuntu-linux-dual-boot-guide-2200653</a></li><li>Once installed, download and install the latest drivers</li></ul><p>A few notes:</p><ul><li>On my 1TB SSD I have set-up the following partitions:</li><li>Windows: C drive 250GB</li><li>Windows: All other system partitions it creates on its own (which you can’t change anyways)</li><li>Ubuntu: /home partition (ideally I should have made this at least 150GB as well, but I didn’t know and made it 50Gb only)</li><li>Ubuntu:/swap partition (40GB)</li><li>Ubuntu: /root (50GB)</li><li>Ubuntu: /data ext4 partition (600GB)</li><li>Note on NTFS communication between Ubuntu and Windows: Ubuntu can read the data in Windows whereas vice-versa it doesn’t work; initially I created the /data partition as NTFS in windows, but for the life of me couldn’t manage to mount it properly in Ubuntu due to permission issues. So my suggestion, in case you want the two OSs to have a shared partition, is to make it FAT32 and for a small enough size so that it doesn’t waste the space, also understanding that FAT32 cannot manage single files larger than 4gb. I chose to avoid this entire hassle and transfer data between then via another method if needed (e.g external disk).</li><li>Make sure that you never <strong>write onto the system Windows partitions from Ubuntu!!</strong> This has high chances of breaking your Windows partitions and losing your data/requiring a reinstall. Keep them separate and they play nice, combine things and you start having a mess….at least that’s my own experience</li><li>In order for SLI to work in Windows, you need to enable it in the settings. Instructions here: <a href="https://www.computerhope.com/issues/ch001167.htm">https://www.computerhope.com/issues/ch001167.htm</a></li></ul><p>I suggest you don’t spend too much time installing everything in terms of software for Windows at this stage; there will be time for it later on. For now just make sure that everything works in terms of drivers, especially for the graphics cards. To make sure that the GPUs are ok and performing up to speed, I installed 3DMark Demo and ran a couple of benchmarks.</p><p>Once Windows is set-up and working properly, time to enjoy the fun of Ubuntu!</p><p><strong><em>Ubuntu 16.04 installation</em></strong></p><p>Make sure that you have enabled “Other OS” in the BIOS-&gt;Boot-&gt;Secure Boot menu, before inserting the bootable USB stick.</p><ul><li>Insert the USB stick</li><li>Start the computer, press Del or F10</li><li>On the right side, go to “Boot Menu” or press F8</li><li>Select the USB stick and your computer will restart</li><li><strong>BEFORE</strong> you click on “Try Ubuntu” to start installation, press “e” on your keyboard and edit “quiet splash” text to <strong>nomodeset</strong>, otherwise the system does not start (it’s missing the display drivers). If for whatever reason “nomodset” doesn’t work, add “nouveau.modeset=0” after quiet splash</li><li>Then the installation flow of Ubuntu begins</li><li>Follow the tutorial here: <a href="https://www.lifewire.com/ultimate-windows-7-ubuntu-linux-dual-boot-guide-2200653">https://www.lifewire.com/ultimate-windows-7-ubuntu-linux-dual-boot-guide-2200653</a></li><li>I’ve provided more info on my setup in the Windows section in terms of my own partition choices</li></ul><p>And this is where the fun really starts! The whole challenge revolves around the NVIDIA drivers, which for some reason seem to be a very debated and buggy topic on Ubuntu. Here is what you need to know before we go into the process:</p><ul><li>By default, Ubuntu will not recognize the GPU and due to this, it will revert to “nouveau”, the generic NVIDIA driver, which for some reason fails to allow Ubuntu to start</li><li>The work-around for this is the “nomodset” startup flag, but in some cases even that doesn’t work</li><li>Even after you install, your drivers will need a lot of work to get installed, and it’s not an easy process</li><li>Neither your LAN nor your wireless will be working directly when you get into Ubuntu (neither for the install -though it’s not needed- nor afterwards on the clean OS); to get around this the best option I found is via manual setup of the connection, which is another fun process in itself</li><li>After you get the driver installed, the process to install CUDA and cuDNN is also very involved, so prepare for many steps</li><li>All of your work will happen via CLI (command line interface) and not via the visual interface of Ubuntu; this makes everything much faster and more straightforward</li><li>In case you’re getting errors during the OS install, and sometimes even afterwards — especially for drivers install — one of the main culprits is the SecureBoot/UEFI thing; so make sure that you google properly and play around with this BIOS setting in case nothing else works</li></ul><p>Now, after you’ve installed Ubuntu, my first reco is to change the Boot order in the BIOS (Boot menu option) to have the ubuntu partition boot first.</p><p>Then set the wait time for GRUB to only 2 seconds (to make us wait much less after each reboot)</p><p><strong>Run this in your terminal:</strong></p><blockquote>sudo -H gedit /etc/default/grub</blockquote><p>This will open up the grub configuration file. Look for a line similar to GRUB_TIMEOUT=2 (Here I have set the grub time to 2).</p><p>Change the number to 2 or whatever time you desire (in seconds).</p><p>Then Save the file (Ctrl+x then Y) and run</p><blockquote>sudo update-grub</blockquote><p>Going forward, I’m only going to focus on how to install everything you need in Ubuntu, as this will be the main OS we’ll be using for Machine Learning. As mentioned in the beginning, Windows is there just in case we need it for other things. This is not mean to say that you can’t use Windows for ML, you certainly can, but it’s not in scope of this article.</p><h3><strong>4. Installing the basics in Ubuntu 16.04</strong></h3><p><strong><em>Disable graphical interface</em></strong></p><p>Tutorial: <a href="https://askubuntu.com/questions/16371/how-do-i-disable-x-at-boot-time-so-that-the-system-boots-in-text-mode/79682#79682">https://askubuntu.com/questions/16371/how-do-i-disable-x-at-boot-time-so-that-the-system-boots-in-text-mode/79682#79682</a></p><p>Run this in your terminal:</p><blockquote>sudo nano /etc/default/grub</blockquote><p>Find this line:</p><p>GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash”</p><p>Change it to:</p><blockquote>GRUB_CMDLINE_LINUX_DEFAULT=”text”</blockquote><p>Update GRUB:</p><blockquote>sudo update-grub</blockquote><blockquote>sudo systemctl enable multi-user.target — force<br>sudo systemctl set-default multi-user.target</blockquote><p>Note: You will still be able to use X by typing <strong>startx</strong> after you logged in. If you want it, here are the steps:</p><p>Run this in your terminal:</p><blockquote>startx</blockquote><p>After the interface starts, you’ll see just the background and an X as the mouse pointer. Right click the mount and select “Open Terminal”, then type in:</p><blockquote>/etc/init.d/lightdm start</blockquote><p>If at boot time there are issues (nouveau) due to display driver after you disable the graphical interface, press “e” in GRUB and edit “text” option to nomodeset, same as we did at installation time</p><p><strong><em>Get the wireless/LAN connection working</em></strong></p><p>In case you want to use the <strong>Network-Manager </strong>for configuration, follow the steps at this link <a href="https://askubuntu.com/questions/55868/installing-broadcom-wireless-drivers">https://askubuntu.com/questions/55868/installing-broadcom-wireless-drivers</a></p><p>This for me was very hit and miss though (especially when we’ll set-up the static IP for remote connection), so I ended up going with the proven manual connection setup.</p><p>Links below:</p><p><a href="https://unix.stackexchange.com/questions/253030/how-to-setup-network-without-wicd-or-networkmanager">https://unix.stackexchange.com/questions/253030/how-to-setup-network-without-wicd-or-networkmanager</a></p><p><a href="https://askubuntu.com/questions/464507/ubuntu-14-04-server-wifi-wpa2-personal/464552#464552">https://askubuntu.com/questions/464507/ubuntu-14-04-server-wifi-wpa2-personal/464552#464552</a></p><p><strong><em>Get Ubuntu up to date and install tmux</em></strong></p><p>Here’s the source: <a href="https://github.com/fastai/courses/blob/master/setup/install-gpu.sh">https://github.com/fastai/courses/blob/master/setup/install-gpu.sh</a></p><blockquote>sudo apt-get update</blockquote><blockquote>sudo apt-get — assume-yes upgrade</blockquote><blockquote>sudo apt-get — assume-yes install tmux build-essential gcc g++ make binutils</blockquote><blockquote>sudo apt-get — assume-yes install software-properties-common</blockquote><blockquote>sudo apt-get — assume-yes install git</blockquote><p><strong><em>Install Ubuntu System Dependencies</em></strong></p><blockquote>sudo apt-get install build-essential cmake git unzip pkg-config</blockquote><blockquote>sudo apt-get install libjpeg-dev libtiff5-dev libjasper-dev libpng12-dev</blockquote><blockquote>sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev</blockquote><blockquote>sudo apt-get install libxvidcore-dev libx264-dev</blockquote><blockquote>sudo apt-get install libgtk-3-dev</blockquote><blockquote>sudo apt-get install libhdf5-serial-dev graphviz</blockquote><blockquote>sudo apt-get install libopenblas-dev libatlas-base-dev gfortran</blockquote><blockquote>sudo apt-get install python-tk python3-tk python-imaging-tk</blockquote><p><strong><em>Install both Python 2.7 and Python 3 header files so that we can compile OpenCV with Python bindings</em></strong></p><blockquote>sudo apt-get install python2.7-dev python3-dev</blockquote><p><strong><em>Prepare our system to swap out the default drivers with NVIDIA CUDA drivers</em></strong></p><blockquote>sudo apt-get install linux-image-generic linux-image-extra-virtual</blockquote><blockquote>sudo apt-get install linux-source linux-headers-generic</blockquote><p><strong><em>Ensure .bash_profile takes information from .bashrc</em></strong></p><p>Several of the settings we’ll do later on for the python installs require you to add paths to .bashrc. But due to how the user access configuration is done on Ubuntu, those changes must be replicated as well on the user’s .bash_profile in order to be taken in every time the computer reboots.</p><blockquote>sudo nano .bash_profile</blockquote><p>Add this to the file:</p><blockquote>if [ -f ~/.bashrc ]; then<br> source ~/.bashrc<br> fi</blockquote><h3><strong>5. Installing CUDA 8</strong></h3><p>Source tutorial: <a href="https://www.pyimagesearch.com/2017/09/27/setting-up-ubuntu-16-04-cuda-gpu-for-deep-learning-with-python/">https://www.pyimagesearch.com/2017/09/27/setting-up-ubuntu-16-04-cuda-gpu-for-deep-learning-with-python/</a></p><p><em>First disable the Nouveau kernel driver by creating a new file</em></p><blockquote>sudo nano /etc/modprobe.d/blacklist-nouveau.conf</blockquote><p><em>Add the following lines and then save and exit:</em></p><blockquote>blacklist nouveau</blockquote><blockquote>blacklist lbm-nouveau</blockquote><blockquote>options nouveau modeset=0</blockquote><blockquote>alias nouveau off</blockquote><blockquote>alias lbm-nouveau off</blockquote><p><em>Next let’s update the initial RAM filesystem and reboot the machine</em></p><blockquote>echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf</blockquote><blockquote>sudo update-initramfs -u</blockquote><blockquote>sudo reboot</blockquote><p><em>You will want to download the CUDA Toolkit v8.0 via the NVIDIA CUDA Toolkit website </em><a href="https://developer.nvidia.com/cuda-80-ga2-download-archive"><em>https://developer.nvidia.com/cuda-80-ga2-download-archive</em></a></p><p><em>From there, download the -run file which should have the filename cuda_8.0.61_375.26_linux-run or similar. To do this, simply right-click to copy the download link and use wget on your remote GPU box:</em></p><blockquote>wget <a href="https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_375.26_linux-run">https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_375.26_linux-run</a></blockquote><p><em>From there, unpack the -run file</em></p><blockquote>chmod +x cuda_8.0.61_375.26_linux-run</blockquote><blockquote>mkdir installers</blockquote><blockquote>sudo ./cuda_8.0.61_375.26_linux-run -extract=`pwd`/installers</blockquote><p><strong><em>NOTE: we will be taking a small detour at this step from the tutorial in the initial link, as the driver there did not work for me. So I needed to install another, not “officially supported” driver, but which worked perfectly.</em></strong></p><p><strong><em>Install the NVIDIA kernel driver (using a special driver for GTX 1080Ti)</em></strong></p><p>Source: <a href="https://blog.nelsonliu.me/2017/04/29/installing-and-updating-gtx-1080-ti-cuda-drivers-on-ubuntu/">https://blog.nelsonliu.me/2017/04/29/installing-and-updating-gtx-1080-ti-cuda-drivers-on-ubuntu/</a></p><p><em>Add the PPA to apt-get and update the index by running</em></p><blockquote>sudo add-apt-repository ppa:graphics-drivers/ppa<br>sudo apt-get update</blockquote><p><em>Now, we use it to install the desired driver versions (as of this writing) for 1080 Ti: Major version 381</em></p><blockquote>sudo apt-get install nvidia-381</blockquote><p><em>If in the future you want to update the drivers via the apt-get drivers</em></p><p>First, remove the old drivers:</p><blockquote>sudo apt-get purge nvidia*</blockquote><p>Now, just install the new driver with the PPA as detailed above and reboot.</p><p><em>Check that the driver and CUDA were installed</em></p><blockquote>nvcc — version # Checks CUDA version</blockquote><blockquote>nvidia-smi # Info about the detected GPUs</blockquote><p><em>Install the CUDA Toolkit and examples</em></p><blockquote>sudo ./cuda-linux64-rel-8.0.61–21551265.run</blockquote><blockquote>sudo ./cuda-samples-linux-8.0.61–21551265.run</blockquote><p><em>Update your ~/.bashrc</em></p><blockquote>sudo nano ~/.bashrc</blockquote><p><em>with the below text (which should be pasted inside the file):</em></p><blockquote># NVIDIA CUDA Toolkit</blockquote><blockquote>export PATH=/usr/local/cuda-8.0/bin:$PATH</blockquote><blockquote>export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64/</blockquote><p><em>Now, reload your ~/.bashrc ( source ~/.bashrc ) and then test the CUDA Toolkit installation by compiling the deviceQuery example program and running it:</em></p><blockquote>source ~/.bashrc</blockquote><blockquote>cd /usr/local/cuda-8.0/samples/1_Utilities/deviceQuery</blockquote><blockquote>sudo make</blockquote><blockquote>./deviceQuery</blockquote><p>you should see “Result = PASS”. In case you did not, go back and redo all the steps of chapter 5.</p><h3><strong>6. Install cuDNN (CUDA Deep Learning Neural Network library)</strong></h3><p>For this step, you will need to <a href="https://developer.nvidia.com/accelerated-computing-developer-program">Create a free account with NVIDIA</a> and <a href="https://developer.nvidia.com/cudnn">download cuDNN</a>. For this tutorial I used cuDNN v6.0 for Linux which is what TensorFlow requires.</p><p>Next, untar the file and then copy the resulting files into lib64 and include respectively, using the -P switch to preserve sym-links:</p><blockquote>cd ~</blockquote><blockquote>tar -zxf cudnn-8.0-linux-x64-v6.0.tgz</blockquote><blockquote>cd cuda</blockquote><blockquote>sudo cp -P lib64/* /usr/local/cuda/lib64/</blockquote><blockquote>sudo cp -P include/* /usr/local/cuda/include/</blockquote><blockquote>cd ~</blockquote><p>And that’s it for cuDNN, you should be set.</p><h3><strong>7. Setup remote connection</strong></h3><p>This section will be about the steps you need to take in order to connect remotely to your machine via a slim terminal, in my case a Macbook Pro. This provides an amazing level of freedom and flexibility, especially because you don’t need to be next to your machine for (almost) anything. Even startup of the machine can be done remotely via WOL (wake on lan).</p><p><strong><em>Configure static a static IP for your machine</em></strong></p><p>Below are a couple of very good tutorials, so I won’t go into details</p><p><a href="https://michael.mckinnon.id.au/2016/05/05/configuring-ubuntu-16-04-static-ip-address/">https://michael.mckinnon.id.au/2016/05/05/configuring-ubuntu-16-04-static-ip-address/</a></p><p><a href="http://www.configserverfirewall.com/ubuntu-linux/ubuntu-set-static-ip-address/">http://www.configserverfirewall.com/ubuntu-linux/ubuntu-set-static-ip-address/</a></p><p><strong><em>Port forwarding</em></strong></p><p>This steps varies significantly depending on your own home/office network setups, but here are the principles:</p><ul><li>In order to access your machine, it needs to have a static IP, and the router should make sure not to overwrite it (usually done via manual exception designation in the DHCP server settings of the router)</li><li>For the remote connection to reach the machine’s static IP, the router must be configured to do <strong>Port forwarding</strong>, in essence to take any request coming in from outside and send it to the specific port on the specific machine</li><li>For the remote connection to reach the router in the first place, you need to make sure that the <strong>modem </strong>is also configured to allow for <strong>Port forwarding</strong></li></ul><p>So knowing the steps above, start using your google fu and figure out how to do this for your own setup.</p><p><strong><em>Configure SSH on both the ML rig as well as the slim client (Macbook Pro)</em></strong></p><p>This topic is again very straightforward and fully documented on the web, so here are some useful links</p><p><a href="https://askubuntu.com/questions/464507/ubuntu-14-04-server-wifi-wpa2-personal/464552#464552">https://askubuntu.com/questions/464507/ubuntu-14-04-server-wifi-wpa2-personal/464552#464552</a></p><p><a href="http://ubuntuhandbook.org/index.php/2016/04/enable-ssh-ubuntu-16-04-lts/">http://ubuntuhandbook.org/index.php/2016/04/enable-ssh-ubuntu-16-04-lts/</a></p><p><a href="https://www.digitalocean.com/community/tutorials/how-to-use-ssh-to-connect-to-a-remote-server-in-ubuntu">https://www.digitalocean.com/community/tutorials/how-to-use-ssh-to-connect-to-a-remote-server-in-ubuntu</a></p><p><a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2">https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2</a></p><p><strong><em>Prevent the SSH session from freezing on the Mac</em></strong></p><p>Once you’ll start using SSH, you might notice that your connection to the ML rig keeps breaking on the Mac. This is due to the standard timeout setttings on both machines, which can be easily configured to keep the connection alive:</p><p><em>On the Mac/</em></p><blockquote>sudo nano /etc/ssh/ssh_config</blockquote><p><em>Then add to the file:</em></p><blockquote>Host *<br> ServerAliveInterval 100</blockquote><p><em>On the server</em>:</p><blockquote>sudo nano /etc/ssh/sshd_config</blockquote><p>Then add to the file:</p><blockquote>ClientAliveInterval 60<br> TCPKeepAlive yes<br> ClientAliveCountMax 10000</blockquote><p>From this point onwards you don’t need to be next to your rig anymore, and neither does it need to be plugged in to either a screen or the keyboard + mouse. Barring any specific errors or boot issues, it should be fine on its own with just the power connected and the wireless/LAN working, with you controlling it via SSH and terminal.</p><h3><strong>8. Install Python3</strong></h3><p>Everything following below can be done in several ways, and it all depends on your requirements, desires and environment configuration. I’ll just show you what I did to get a barebones installation that can do “most” of the tasks required for Machine Learning, at least at a basic level.</p><p><strong><em>Install Anaconda3</em></strong></p><blockquote>wget <a href="https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh">https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh</a></blockquote><blockquote>bash Anaconda3–5.0.1-Linux-x86_64.sh</blockquote><blockquote>source .bashrc</blockquote><blockquote>source .bash_profile</blockquote><blockquote>conda upgrade -y — all</blockquote><p><strong><em>Install Tensorflow</em></strong></p><blockquote>conda install -c conda-forge tensorflow</blockquote><blockquote>sudo apt install python3-pip</blockquote><blockquote>pip install tensorflow-gpu</blockquote><p><em>Validate Tensorflow Installation</em></p><blockquote>git clone <a href="https://github.com/tensorflow/tensorflow.git">https://github.com/tensorflow/tensorflow.git</a></blockquote><blockquote>python tensorflow/tensorflow/examples/tutorials/mnist/fully_connected_feed.py</blockquote><p>You should see the loss decreasing during training:</p><p>Step 0: loss = 2.32 (0.139 sec)</p><p>Step 100: loss = 2.19 (0.001 sec)</p><p>Step 200: loss = 1.87 (0.001 sec)</p><p><strong><em>Install PyTorch</em></strong></p><blockquote>conda install pytorch torchvision cuda80 -c soumith</blockquote><p><strong><em>Install Theano</em></strong></p><blockquote>conda install theano</blockquote><p><strong><em>Install Keras</em></strong></p><blockquote>pip install keras</blockquote><p><strong><em>Install openCV</em></strong></p><p><strong>Simple way to do it:</strong></p><blockquote>conda install -c conda-forge opencv=3.2.0</blockquote><p>Now, in case that doesn’t work , google it a bit as there are several other variants:</p><blockquote>pip install opencv-contrib-python</blockquote><p>or</p><blockquote>conda install — channel <a href="https://conda.anaconda.org/menpo">https://conda.anaconda.org/menpo</a> opencv3</blockquote><p><strong>Very complicated (manual) way to do it:</strong></p><p>Source: <a href="https://medium.com/@debugvn/installing-opencv-3-3-0-on-ubuntu-16-04-lts-7db376f93961">https://medium.com/@debugvn/installing-opencv-3-3-0-on-ubuntu-16-04-lts-7db376f93961</a></p><h3><strong>9. Configure JupyterNB to work remotely</strong></h3><p><strong><em>Configure it</em></strong></p><p># Create a ~/.jupyter/jupyter_notebook_config.py with settings</p><blockquote>jupyter notebook — generate-config</blockquote><blockquote>jupyter notebook — port=8888 — NotebookApp.token=’’ # Start it</blockquote><p><strong><em>Create a SSH tunnel for Jupyer</em></strong></p><p><em>Create either script that runs this on the Macbook, or simply run this every time you want to connect on the Macbook to your remote ML rig</em></p><blockquote>ssh -N -f -L localhost:8888:localhost:8888 -p xxxx (your remote port) xxxx.xxxx.xxx.xxx (your remote IP)</blockquote><p>Now you can visit <a href="http://localhost:8888/">http://localhost:8888</a> in your laptop’s browser and start editing the notebooks on your deep learning machine!</p><p>In case the above command responds that the 8888 port is already in use (from a previous session), do the below to kill the process and then rerun the above</p><blockquote>lsof -ti:8888 | xargs kill -9</blockquote><p><strong><em>Run Jupyter on boot for the ML rig</em></strong></p><p><em>Create a startup_ipynb.sh file</em></p><blockquote>cd ~</blockquote><blockquote>sudo nano startup_ipynb.sh</blockquote><p>Add the below in the file:</p><blockquote>export PATH=/home/xxx(your username)/anaconda3/bin:$PATH</blockquote><blockquote>jupyter notebook — config=/home/xxx(your username)/.jupyter/jupyter_notebook_config.py — no-browser</blockquote><p><em>Open the rc.local file</em></p><blockquote>sudo nano /etc/rc.local</blockquote><p>Add this line:</p><blockquote>su xxx(your username) -c ‘bash /home/xxx(your username)/startup_ipynb.sh’</blockquote><p>Now every time you boot the machine, Jupyter Notebook should be running in the backgorund. So just go to your slim client and do:</p><blockquote>ssh -N -f -L localhost:8888:localhost:8888 -p xxxx (your remote port) xxxx.xxxx.xxx.xxx (your remote IP)</blockquote><p>Same as for the previous step: you can visit <a href="http://localhost:8888/">http://localhost:8888</a> in your laptop’s browser and start editing the notebooks directly on your deep learning machine!</p><p>In case the above command responds that the 8888 port is already in use (from a previous session), do the below to kill the process.</p><blockquote>lsof -ti:8888 | xargs kill -9</blockquote><blockquote>ssh -N -f -L localhost:8888:localhost:8888 -p xxxx (your remote port) xxxx.xxxx.xxx.xxx (your remote IP)</blockquote><h3><strong>10. Setup PyCharm to work remotely</strong></h3><p>If you use PyCharm, then <a href="https://medium.com/@erikhallstrm/work-remotely-with-pycharm-tensorflow-and-ssh-c60564be862d">here is a guide to help you setup it up</a>, allowing you to have the full benefits of the PyCharm IDE but running the actual code remotely on the ML rig</p><p>And that’s it in terms of setting things up. Hope you’ve enjoyed the process and everything is now working for you.</p><h3>(for ML beginners) fast.ai setup</h3><p>In case you are working your way through the lessons at <a href="http://fast.ai/"><strong>fast.ai</strong></a><strong> </strong>(which is an outstanding learning resource btw), here are some things you could do to get things working without too much hassle.</p><p><strong><em>Create a separate conda environment for python2.7</em></strong></p><p>As of the time i’m writing this article (November 2017), most of the course was written by Jeremy in python 2.7, so the initial setup you’ve done if you’ve followed this guide (Python3) will not work. The best approach in this case would be to create a separate environment where you can run the lessons without issues.</p><p>Thanks to the ease of using conda + the kernel system of Jupyter Notebook this is very easy:</p><p>Allow the use of kernels, in case this is not already installed:</p><blockquote>conda install nb_conda_kernels</blockquote><p><strong>Create a new python2.7 environment in conda</strong>, together with the required Jupyter NB kernel; “py27” is the name of the environment, and you can change that to whatever you like:</p><blockquote>conda create -n py27 python=2.7 ipykernel</blockquote><p>Restart your terminal and you should not see the new kernel showing in Jupyter Notebook (using the Kernel menu)</p><p><strong><em>Installing the required libraries</em></strong></p><p>Active the python2.7 environment, in order to install the required libraries inside:</p><blockquote>source activate py27</blockquote><p>Make sure that you are inside this environment (its name shows between brackets in front of your username in terminal) before proceeding to the next steps.</p><p>Install <em>scikit-learn</em>:</p><blockquote>conda install scikit-learn</blockquote><p>Install <em>bcolz</em>:</p><blockquote>conda install -y bcolz</blockquote><p>Install <em>theano</em>:</p><blockquote>pip install theano</blockquote><p>Install <em>keras dependecies (you might need to install a few others; in case you get errors then Google it and stackoverflow will tell you what else might me missing)</em>:</p><blockquote>pip install numpy</blockquote><blockquote>pip install pillow</blockquote><blockquote>pip install h5py</blockquote><p>Install <em>keras</em> (version 1.2.2, as the newer ones 2.x do not work well due to the API changes which break the code in Jeremy’s notebooks)</p><blockquote>pip install keras==1.2.2</blockquote><blockquote>mkdir ~/.keras</blockquote><blockquote>echo ‘{ “image_dim_ordering”: “th”, “epsilon”: 1e-07, “floatx”: “float32”, “backend”: “theano”}’ &gt; ~/.keras/keras.json</blockquote><p>You might be required to install a few other libraries as well (you’ll see the prompts in JupyterNB) depending on your setup, but it should be straightforward.</p><p>In case you get stuck on some specific errors, do a Google search with <strong>fast.ai</strong> at the end of your error and you will most likely find an answer on the forums there.</p><p>One other side note: considering that there is still a large amount of python2.7 code in the field of ML, i recommend to use either the environment you just created (py27) or even a separate one (perhaps with the most updated versions of all the libraries) in order to be able to run the code properly. In my experience, trying to change 2.7 code to run in python3 can be quite a time consuming (but extremely educative as well!) process, so if you’re pressed by time then better to use a dedicated env for it.</p><p>Let the Machine Learning begin!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dae0f4011a8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>