<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Justin Davies on Medium]]></title>
        <description><![CDATA[Stories by Justin Davies on Medium]]></description>
        <link>https://medium.com/@justindavies?source=rss-77e48ab429d1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-0J3_14pDzudOE7XYV8-Rw.jpeg</url>
            <title>Stories by Justin Davies on Medium</title>
            <link>https://medium.com/@justindavies?source=rss-77e48ab429d1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:03:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@justindavies/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[App Preview videos for Apple App Store without a device]]></title>
            <link>https://medium.com/@justindavies/app-preview-videos-for-apple-app-store-without-a-device-f6dc12dc21fb?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6dc12dc21fb</guid>
            <category><![CDATA[react-native]]></category>
            <category><![CDATA[app-store]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[app-store-optimization]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Sat, 06 Jun 2020 14:48:51 GMT</pubDate>
            <atom:updated>2020-06-06T14:48:51.072Z</atom:updated>
            <content:encoded><![CDATA[<p>For <a href="https://apps.apple.com/us/app/id1514919095">Jodie Insights</a> I decided to use React Native and Expo to speed up development (I was pretty happy to turn it round in a month). The thing was, I only have an old iPhone SE and the only way that Apple says we should generate videos for App Store App Previews is with an iPhone with a Lightning cable.</p><p>I thought there must be a better way to do this, and here it is…</p><p>Open your IOS app in the Simulator (it must be iPhone 11 Pro Max as we need that aspect ratio) and use xcrun to record the screen:</p><p><em>xcrun simctl io booted recordVideo company.mp4</em></p><p>This will give you the video to upload, but it’s way too big. Apple requires the resolution to be 886 x 1920.</p><p>Let’s fire up FFmpeg:</p><p><em>ffmpeg -i company.mp4 -vf scale=886x1920,fps=25 -aspect:v 886:1920 company-store.mp4</em></p><p>This took me about an hour to figure out, but FFmpeg would convert the video to 887x1920, which App Connect would not accept. use -aspect to force the resize exactly, you also need to upscale the FPS as the default was too low too.</p><p>Unfortunately, this still won’t work as Apple will reduce to accept a video without stereo. FFmpeg to the rescue again:</p><p><em>ffmpeg -i company-store.mp4 -f lavfi -i anullsrc -c:v copy -c:a aac -shortest company-store-correct-a.mp4</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6dc12dc21fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training Spacy NER models with doccano]]></title>
            <link>https://medium.com/@justindavies/training-spacy-ner-models-with-doccano-8d8203e29bfa?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d8203e29bfa</guid>
            <category><![CDATA[named-entity-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Tue, 28 Jan 2020 08:54:11 GMT</pubDate>
            <atom:updated>2020-01-28T08:54:11.982Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3jPEo-SMbrDLWw1gOyMDRw.png" /><figcaption>Named Entity recognition on jodie.ai</figcaption></figure><p>For <a href="https://jodie.ai/">Jodie.ai</a> I use <a href="https://spacy.io/">Spacy</a> heavily to extract named entities from 10-Q and 10-K filings to analyse the Risks affecting publicly listed companies.</p><p>Whilst the pre-built Spacy models are pretty good at NER extraction, they aren’t amazing in the Finance domain. This, of course, is fair enough, as the training data used for the NER pipeline isn’t geared towards the Finance, or even worse the type of language used in SEC filings.</p><p>I won’t go into it on this post, but I read the Edgar monthly archives and import 10-Q and 10-K filings and then parse out Item 1A — Risk Factors. I do this as the Risk Factors highlight what companies feel are the headwinds to their growth, or challenges they foresee in the next period of the business.</p><p>These insights are valuable, especially if they change over time. This is also where extracting Named Entities allows us to place those Risk Factors in the real world and allows us to mine those Entities in news as it appears (for example geo-political unrest in Eastern Europe where “Ukraine” is mentioned allows us to notify users that this is affecting a number of companies based on their Risks).</p><p>Filing data for Jodie is stored in an Elasticsearch store, and in this example, all parsing has taken place.</p><h3>Model 0</h3><p>When I start the process of creating a statistical model, O need somewhere to start, so I use the pre-built <em>en_core_web_sm </em>model for our work to extract Named Entities. It is the output of this that is used as the initial pass for our sequence labelling.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/022df074c025ca09599a2c4649aeffb1/href">https://medium.com/media/022df074c025ca09599a2c4649aeffb1/href</a></iframe><p>This is quite a long script, but a number of things are happening here…</p><p>Firstly, I am using Entity Ruler as the ingest process for Jodie has already extracted and saved the entities in ElasticSearch. As part of the admin interface to Jodie I will state whether a NER is correctly categories or not. This is stored in a MongoDB and the EntityRules will pull these as part of the pipeline. This allows us to get many more “correct” assignments as part of the annotation process and speeds up the process.</p><p>Secondly, I’ve introduced an “IGNORE” entity type. This is where Spacy has incorrectly annotated text to something I do not want in my final model.</p><p>Thirdly, I only want documents from the last year (for now) as I don’t need a huge amount of examples for pass 1.</p><h3>Catastrophic Forgetting</h3><p>Fourthly, and <strong>this is critical</strong> when amending an existing model is the catastrophic forgetting problem. Matthew and Ines have written so many replies on GitHub and in blog posts about how to stop an existing model forgetting what it has already learnt.</p><p>I make sure that for the annotation process, I also take in all NER outputs, not just the ones I have deemed correct or incorrect. As an example, it is pretty likely that the model will correctly state MONEY and DATE entities as even within the Finance domain, they are quite easy to extract. I don’t state whether these are correct or not, but they are. part of the training data.</p><h3>Importing data</h3><p>The output from the script produces a JSONL delimited file with each sentence (whether it has NERS or not), along with the entity offsets…</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5fb00b646d5fd8db1399138271ae9cf5/href">https://medium.com/media/5fb00b646d5fd8db1399138271ae9cf5/href</a></iframe><p>I run <a href="https://github.com/doccano/doccano">Doccano</a> in a docker image locally on my laptop for each labelling exercise, and carry out the process in 5,000 example chunks (it quite slow with more than that, and for a first pass I’ve found 5,000 examples works well).</p><p>So let’s spin up docanno and import data…</p><blockquote>docker container create — name doccano \ <br> -e “ADMIN_USERNAME=admin” \<br> -e “ADMIN_EMAIL=admin@example.com” \<br> -e “ADMIN_PASSWORD=password” \<br> -p 8000:8000 chakkiworks/doccano</blockquote><p>Login to Doccano (http://localhost:8080) with the username “admin”, and password is “password”…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F19IaopjNtk4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D19IaopjNtk4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F19IaopjNtk4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/90d17d20f49120b43a3a4d88a9659474/href">https://medium.com/media/90d17d20f49120b43a3a4d88a9659474/href</a></iframe><p>Once I’ve uploaded the data from the script, I can start annotating…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F0V4YgUPNEBM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D0V4YgUPNEBM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F0V4YgUPNEBM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/126b3500afd06e799753a977f1a36c95/href">https://medium.com/media/126b3500afd06e799753a977f1a36c95/href</a></iframe><p>Once I have gone through the 5,000 examples and cleaned up the annotation, it’s time to export the data for use in training the statistical Spacy model…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9OgV_3JbQAc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9OgV_3JbQAc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9OgV_3JbQAc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/05ad400f0b3f92bfdc5df2a19e77968b/href">https://medium.com/media/05ad400f0b3f92bfdc5df2a19e77968b/href</a></iframe><p>This is save the newly annotated examples into JSONL format, which I need to do some “magic” with to get it Spacy ready.</p><h3>Converting to Spacy training format</h3><p>The output from Doccano is not in the right format for use within Spacy, so I have to make a few changes to the labels list…</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a820ca6072ff309341457cdaba27d6e5/href">https://medium.com/media/a820ca6072ff309341457cdaba27d6e5/href</a></iframe><p>This reads in the Doccano output and modifies the labels to entities, as well as only using the Entity types I want.</p><h3>Split Dev and Test data</h3><p>As I have our data in single line JSON, it’s easy to wrangle into DEV/TEST data before converting to Spacy training data which is more complex.</p><blockquote>head -n 4000 clean.jsonl &gt; train.jsonl</blockquote><blockquote>tail -n 1000 clean.jsonl &gt; dev.jsonl</blockquote><h3>Convert to Spacy training format</h3><p>Now use the Spacy <em>convert</em> command to transform this into training data..</p><blockquote>python -m spacy convert ./dev.jsonl ./ — lang en<br>✔ Generated output file (1000 documents): dev.json</blockquote><blockquote>python -m spacy convert ./train.jsonl ./ — lang en <br>✔ Generated output file (4000 documents): train.json</blockquote><h3>Creating model</h3><p>And finally, let’s create the model. I’ll use the en_core_web_sm as the base model, and only train the NER pipeline</p><blockquote>python -m spacy train en model ./train.json ./dev.json -b en_core_web_sm -p ner<br>Training pipeline: [‘ner’]<br>Starting with base model ‘en_core_web_sm’<br>Counting training words (limit=0)</blockquote><blockquote>Itn NER Loss NER P NER R NER F Token % CPU WPS<br> — — — — — — — — — — — — — — — — — — — — — — — -<br> 1 2739.579 81.541 69.572 75.083 100.000 25710</blockquote><h3>Conclusion</h3><p>I can’t talk about training models without talking about <a href="https://prodi.gy/">Prodigy</a>, <a href="https://explosion.ai/">Explosion AI’s</a> own annotation tool. Whilst Doccano is great when you have lots of time on your hands, Prodigy is capable if online training, and can get the process done a lot quicker. But if you don’t want to use the tool ,or want to figure out how to do it yourself, I hope this has helped.</p><p>A lot of the sequence labelling in Jodie has been written in house as part of the ingest pipeline, so the cleansing and annotation scoring is taken care off. Doccano is used to eye-ball those assumptions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d8203e29bfa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes, React and Sitemaps using NGINX Ingress]]></title>
            <link>https://medium.com/@justindavies/kubernetes-react-and-sitemaps-using-nginx-ingress-8944cbe57b1c?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8944cbe57b1c</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[sitemap]]></category>
            <category><![CDATA[nginx]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Fri, 05 Jul 2019 06:59:36 GMT</pubDate>
            <atom:updated>2019-07-05T06:59:36.344Z</atom:updated>
            <content:encoded><![CDATA[<p>Google Search console requires that your sitemaps are stored within your top level domain. This poses a few problems when your frontend is React based. You could drop your sitemaps into the public/ directory, but this isn’t very elegant, and also means you have to mess around with routes, and stuffing static data into your builds.</p><p>For <a href="http://inkl.in/">http://inkl.in</a> I’m using Kubernetes, and have a CronJob that runs nightly to generate the combines sitemaps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1943b0e9a433804c1f5f224e24c66b51/href">https://medium.com/media/1943b0e9a433804c1f5f224e24c66b51/href</a></iframe><p>A few things to note here; sitemaps should be a maximum of 50,000 entries long (I have upwards of 8 million unique pages I want to tell Google about). Hence the reason there is a splitting pattern going on here.</p><p>Once we have the sitemaps generated, I need to host those within a simple nginx container.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f422ac506ac9669ed408bbee847dae7/href">https://medium.com/media/6f422ac506ac9669ed408bbee847dae7/href</a></iframe><p>And then finally I need to setup the NGINX ingress rules</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89a08803640cb2ae2d618a836182a0f0/href">https://medium.com/media/89a08803640cb2ae2d618a836182a0f0/href</a></iframe><p>We have two host rules, one for the frontend and sitemaps, as well as on for the api microservices.</p><p>Another few things to note. Ingress rules are sensitive to order. This allows me to have a catch all for the main React site (/), but before that to also route all traffic for /sitemaps to the sitemap exposed NGINX deployment.</p><p>Make sure to use the <strong><em>use-regex </em></strong>annotation so all XML files are served up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8944cbe57b1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating Sitemaps for Bots]]></title>
            <link>https://itnext.io/generating-sitemaps-for-bots-a9d927f359b1?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9d927f359b1</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[seo]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Fri, 14 Sep 2018 11:18:14 GMT</pubDate>
            <atom:updated>2018-09-18T06:53:37.821Z</atom:updated>
            <content:encoded><![CDATA[<p>After the previous article on getting my React App rendered on the server side with <a href="https://medium.com/@justindavies/using-rendertron-in-kubernetes-for-spa-seo-39055567c745">Rendertron and Kubernetes</a>, I needed a way of letting those Search Engines know about the content (namely updated interpretation of the Ethereum blocks).</p><p>As the content on <a href="http://inkl.in/">http://inkl.in</a> is updated constantly whenever a new block appears on the chain I needed to make sure crawlers could find things a bit easier than the organic internal linking that takes place.</p><p>Enter sitemaps.</p><p>A sitemap is an XML file that will hint to a Search Engine that there is content it should look at. I say hint, as it does not force the likes of Google or Bing to index the content, but it’s better than nothing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TrFFbACqgLR3SJTJylVJKg.png" /></figure><p>Since pushing the inkl.in sitemap to Google I can see about 5,000 pages being indexed per day, and there is organic search traffic hitting the site.</p><p>The problem with having as much content as there is on the site (over 6 million blocks = over 6 million pages, plus visualisation of addresses, which is well over 300 million) is that it will not fit into a single sitemap XML file.</p><p>To get around this, enter sitemap indexes. An index is another XML file used to reference all of the other sitemaps. There are limits in how much data you can have in a single sitemap file, namely 50,000 pages, and maximum size of 50Mb.</p><p>So two things need to happen, I need to generate a sitemap index that links to sitemaps for all Blocks in inklin. I’m not going to be too intelligent about this as I don’t mind a few thousand 404 errors for missing blocks.</p><p>The inklin API has a “live” endpoint which will give me back the data about the latest block (including the block number)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7985d3394025203bb0a12a08b9c82202/href">https://medium.com/media/7985d3394025203bb0a12a08b9c82202/href</a></iframe><p>Once I have this, all I need to do is decrement from there down to Block 1.</p><p>This will allow be to generate a sitemap entry per block which Google or Bing can decide to crawl. I also need to make sure that we generate a new sitemap file for every 50,000 entries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6f810d1af4ce8e6a62cbc46e7fca215/href">https://medium.com/media/a6f810d1af4ce8e6a62cbc46e7fca215/href</a></iframe><p>And then finally once I have a directory of sitemaps I need to generate the index.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9184fd1f80196b207c08c0244d01ee5d/href">https://medium.com/media/9184fd1f80196b207c08c0244d01ee5d/href</a></iframe><p>Once we have the full code, I need to construct a Kubernetes Job (a one time Pod) to run and generate this on a daily basis.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/260736cd3106cd5903d085ce33f06811/href">https://medium.com/media/260736cd3106cd5903d085ce33f06811/href</a></iframe><p>You’ll notice I have a volume defined. I’m using Azure Files (CIFS fileshare) so that the Sitemaps Job can generate and save the data. This is important, because your sitemaps need to be hosted and server from the same domain as you site.</p><p>I’ve modified the Frontend Pod spec to mount the same CIFS share under /sitemaps in the NGINX root. I know, I know, I could have also setup an Ingress controller to separate this, but I hacked this together in an hour!</p><p>My Helm template now looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6dc63b909c3fc90939fe1a89becb4521/href">https://medium.com/media/6dc63b909c3fc90939fe1a89becb4521/href</a></iframe><p>Once that’s done, whenever the Sitemaps Job runs, the data will be freshened for whenever Google or Bing decides to re-trawl the Sitemap Index. And here’s the final result…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*11gyzRhOBNJWmzbCF6ypNA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9d927f359b1" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/generating-sitemaps-for-bots-a9d927f359b1">Generating Sitemaps for Bots</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Rendertron in Kubernetes for SPA SEO]]></title>
            <link>https://itnext.io/using-rendertron-in-kubernetes-for-spa-seo-39055567c745?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/39055567c745</guid>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Fri, 24 Aug 2018 11:32:23 GMT</pubDate>
            <atom:updated>2018-09-16T09:24:16.070Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the applications I use for my Kubernetes demos is <a href="http://inkl.in/">http://inkl.in</a> — it’s an Ethereum Blockchain visualisation tool that uses a Frontend (React), talking to an API (Node), talking to MongoDB that sits on a VM within an Azure private VNET. You can find the code on GitHub <a href="https://github.com/justindavies/inklin">https://github.com/justindavies/inklin</a>, but for today I wanted to run through using Rendertron to fix the inherant issue of Search Engine Optimisation for component based Javascript apps.</p><p>The problem with SPA (Single Page Applications) is that web crawlers aren’t that great at Javascript, and are especially bad at interpolating Javascript to get decent content for indexing.</p><p>A number of “fixes” are out there, the most prominent being Server Side Rendering (SSR). I found a LOAD of documentation out there, but I’m just a lazy person, and wanted something a bit simpler (plus my skill level on webpack is non-existent).</p><p>With the advent of things like Puppeteer (Headless Chrome), I started messing around with using it to proxy requests for my app, and then spit that out into rendered HTML for Search Engines, as well as things like Facebook and Twitter sharing for Inkl.in.</p><p>As is usual with these types of things after working on this for a while, I found out about Rendertron. Take a look at Sam Li talking about this, it’s a great solution to the problem.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FydThUDlBDfc%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DydThUDlBDfc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FydThUDlBDfc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/87aac66da727a6cda33ccdcedc9e5fbe/href">https://medium.com/media/87aac66da727a6cda33ccdcedc9e5fbe/href</a></iframe><p>Rendertron is essentially a proxy for requests where you DON’T want the client to do the Javascript execution; in this case Bots and Social Sharing scrapers. To do this, we need to look at the User Agent to see if we need to route our request to Rendertron to pre-render, or just pass through the request to the React App directly.</p><p>As I spend most of my time in Kubernetes, and I work for Microsoft, so hosting this on Google seemed a little meh to me, I wanted to get this working on Kube.</p><p>Rendertron doesn’t come with a Dockerfile anymore, so let’s go ahead and create one</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d7b521e1a793456a3a55bbdaa1594bd/href">https://medium.com/media/4d7b521e1a793456a3a55bbdaa1594bd/href</a></iframe><p>We’ll also need to get this onto Kubernetes, so let’s get a Deployment definition</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/513d6bcf4c2daafa7bc13addad03d29d/href">https://medium.com/media/513d6bcf4c2daafa7bc13addad03d29d/href</a></iframe><p>And a Service definition (I like to keep them separate)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f40ab4b5e1927a7d86c8cd43fe02b174/href">https://medium.com/media/f40ab4b5e1927a7d86c8cd43fe02b174/href</a></iframe><p>Now that we have Rendertron on our cluster, we need to edit the NGINX configuration for the React App. Just for the sake of how I do it, this is the Dockerfile to build my React app</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f226cb2bf383752c452a268d6e87e8f0/href">https://medium.com/media/f226cb2bf383752c452a268d6e87e8f0/href</a></iframe><p>We need to replace the default config with the Rendertron goodness. What we need to do is sense the UA, and then proxy only if it is a Bot or Scraper to Rendertron</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c25e45ecae9c3b6887f76ac829868cc4/href">https://medium.com/media/c25e45ecae9c3b6887f76ac829868cc4/href</a></iframe><p>A couple of things that drove me crazy based on the Gists out there for non-Kube deployments. I had to set the resolver to be the kube-dns instance as I’m using a variable for the proxy_pass directive (thankyou StackOverflow!), as well as making sure I used the FQDN for the service rather than the shortname.</p><p>After this, I can see that requests from GoogleBot are being routed correctly by changing my UA in Chrome to GoogleBot!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39055567c745" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/using-rendertron-in-kubernetes-for-spa-seo-39055567c745">Using Rendertron in Kubernetes for SPA SEO</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Goodness me! (DevOps strikes again)]]></title>
            <link>https://medium.com/@justindavies/goodness-me-devops-strikes-again-b987d86b770f?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b987d86b770f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Thu, 19 Apr 2018 22:57:55 GMT</pubDate>
            <atom:updated>2018-04-19T22:57:55.701Z</atom:updated>
            <content:encoded><![CDATA[<p>First off, huge thanks to <a href="https://www.linkedin.com/in/kjprobert/">Kyle</a> at District 4 for inviting me to contribute to the discussion on DevOps: Azure, AWS and GCP at his meetup. And a big thanks to Alex Priscott (Head of Platforms at MatchesFashion.com) for hosting us at his offices in the Shard.</p><p>I look after Open Source at Microsoft in the UK — from Linux, up to Python and Kubernetes in between; I see our customers doing a lot. Inexorably the conversation will turn to DevOps. What is the right tool, what is the right platform and what is the right process are questions that we get asked a lot.</p><p>I asked the audience and the other Cloud evangelists seemed to agree that the comment of “DevOps is the intersection of People, Process and Tools” was a valid one. Adding onto that “…to deliver the continuous delivery of value to a customer”.</p><p>The Customer in this case could be an end user, or it could be the leadership team within the organisation that you work in. The idea being that you don’t become Agile in your delivery methodology for the fun of it. You are doing it to deliver a higher value to your Stakeholders — whoever they may be.</p><p>Microsoft is a REALLY big software company. We used to ship software on floppy disks and CD’s — the delivery cycle was long as the delivery mechanism was physical. This is nowhere near the case these days and removing the blocker of the delivery means we can accelerate the development of features.</p><p>If you are a born in the Cloud type of company, this is an alien concept to you (you deliver in an Agile manner because that’s the best way), and I will be the first to high five you to say congratulations on having a smoother way of delivering value to your users.</p><p>At Microsoft, when we delivered TFS as a SaaS offering to our users, it allowed us to remove the delivery constraints and move much more to a Continuous Delivery (CD) way of working. Our delivery bottlenecks were gone, and we could develop and iterate quickly to increase how productive our customers could be. Give us feedback about the product, and you know what, we have the Agility to potentially add that to the next sprint. Our delivery mechanism is in our control- we build, ship and run features and bug fixes to you as a service.</p><p>This value proposition is the same thing you go through. How to I shorten the time to value that my customers see ?</p><p>The question of what tools should be used came up; and this is something that I genuinely believe is one of the big red herrings of DevOps. You have to appreciate the three sides if the DevOps triangle; “People, Process and Tools” seriously.</p><p>You cannot be successful with one, or two of these. The Culture, the Tooling and the Process for getting to production have to work together.</p><p>You all have to pull together to achieve the goal of Agility, the Process has to allow you to get here, and you need a Platform that allows you to realise that value in Technology. I don’t every see there being a silver bullet of DevOps, and I’ve never met a practitioner who has thought there is.</p><p>Maybe the formula should be DevOps = Pragmatism + People + Process + Tools.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b987d86b770f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Misadventures in React with Ethereum]]></title>
            <link>https://medium.com/coinmonks/misadventures-in-react-with-ethereum-7ab3d4ee1af6?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ab3d4ee1af6</guid>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[dapps]]></category>
            <category><![CDATA[blockchain-technology]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Sat, 14 Apr 2018 13:58:20 GMT</pubDate>
            <atom:updated>2018-05-16T10:35:27.650Z</atom:updated>
            <content:encoded><![CDATA[<p>Oh GOD I hate Javascript! The first language I started with 25 years ago was Basic on the <a href="https://en.wikipedia.org/wiki/Commodore_Plus/4">Commodore Plus</a>/4 and then I upgraded to an Amstrad CPC 6128. Then at College I learnt Pascal and Assembly, and finally C and C++ at University.</p><p>I say this, as I’m a child of the 80’s and 90’s and it was all about procedural and OO languages — and that’s how I’ve intrinsically learnt as the way to write software. Over the years I’ve picked up Python, Perl and Ruby (big fan of Ruby on Rails) to name a few.</p><p>I dipped my toe in the water of Javascript (proper, not some crappy JS in a HTML page as I did in the 90s/Naughties) when I started writing an ETL tool for my Machine Learning work, and pretty much made <a href="https://www.toptal.com/nodejs/top-10-common-nodejs-developer-mistakes">The Top 10 Most Common Mistakes That Node.js Developers</a> make and added a few myself.</p><p>I suck as a programmer, but I love to make things, fix things and push machines as far as they go; so I hack at code, I copy and paste and I <a href="http://rea.tech/getting-shit-done/">Get Shit Done</a>. I am Mr Technical Debt and I’m ashamed of it — have been for about 20 years as deep down I wanted to be a leet coder, but my mind doesn’t work like that. My professional career has mostly been at the intersection of developers and business as an Architect, and now I’m a Technical Lead for Open Source at Microsoft. I’m a Systems guy, I look at the big picture when I design, but I understand how things work under the cover.</p><p>I highlight this as it is something that eats away at me. I see beautiful code (I know what beauty in code looks like), and I want to be able to think like that and do like that. I can talk with developers, and we can design great systems together but their skill has eluded me.</p><p>So I’ve taken something I hacked together a few weeks ago as inline Javascript, and some ETL etc., and I can up with a visualisation of the Ethereum blockchain.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FW34vaehtyj4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DW34vaehtyj4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FW34vaehtyj4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7a4565fbfa58da8c5602651b87da58e/href">https://medium.com/media/e7a4565fbfa58da8c5602651b87da58e/href</a></iframe><p>I was impressed by my few hours of work and thought that this was the perfect side project to make this a service and not just Get Shit Done, but do it right.</p><p>I first happened upon Angular and was crying internally and externally at how totally alien the whole thing was to me. I asked a few very competent developers whether I should be looking at Angular, Vue or React to crack this nut and apart from a few “Vue is the future” comments, the resounding response was React. I liked this answer as it also meant I could use React Native if I wanted to code this up for mobile too (in past life, I used <a href="https://www.appcelerator.com/">Appcelerator</a> for my IOS and Android apps).</p><p>At Microsoft we are given a subscription to Safari Books Online, and I started reading as much as I could on React. The O’Reilly book <a href="http://shop.oreilly.com/product/0636920049579.do">Learning React</a> was the one that has helped the most so far as it pushes you very early on the understand Functional Programming.</p><p>I have to admit when I’ve hear people talking about FP before I thought they were talking about Procedure and things. I now realise quite how wrong I was and am still totally confused about the “fat arrow” syntax, about why FP is better, and — well there are so many things I’m confused about the whole situation. But I GET that it help me be a better programmer, I sense that if I can get my head around this, I will be a better software designer. No longer shall I copy and paste, no longer shall I be ok with Getting Shit Done.</p><p>I got it that at a high level; much the same way that Angular does it, React allows me to componentise my App so that I have discreet elements to work on.</p><p>I appreciated this was valuable as my initial work on the Ethereum Visualiser turned into a mess of Javascript Function spaghetti very quickly and was so buggy that I was reticent to push it to my <a href="https://github.com/justindavies/VisualiseEthereum">GitHub repo</a>, but I did. You can see the next iteration of it “live” at <a href="http://inkl.in/">http://inkl.in</a>. I say “live” as it is the same code, with a different Force Graph visualisation reading in JSON files for each Ethereum block. There is nothing dynamic, and it isn’t very useful apart from looking kind of pretty.</p><p>When I worked at Bloomberg, I saw first hand how turning Data into Information was key, and Bloomberg has access to all of that Market Data from global exchanges, and makes it valuable to a huge audience of people.</p><p>And here we have Blockchain, the entire chain is readable and Open Source — this peaked my interest. What Information can you get from the Data that is useful?</p><p>So I have now embarked on using React to make a visual, and interactive Ethereum explorer. I’ve learnt how to programmatically parse Ethereum data, Contracts, how to clean the data so it is Reference data and able to query it for some use (although we shall see how useful it will be later)</p><p>The (very rough) flow of the data is…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*22S6L9jm2SX9jiYeyeKZ-A.png" /></figure><p>As Julie Walters would say; <a href="https://www.youtube.com/watch?v=PEodUg43Zpo">Let’s start at the very beginning</a>.</p><p>To be able to query the Ethereum Blockchain you need to download the database itself. It’s relatively simple; either install <a href="https://github.com/ethereum/go-ethereum/wiki/geth">Geth</a> or <a href="https://www.parity.io/">Parity</a> onto your machine.</p><p>Parity by default will attempt to do a Fast sync of the block chain (at the time of writing there are well over 5 million blocks in the database). You should be able to get a good chunk of the database locally in a few hours. It can do this as your machine does not need to verify all previously verified blocks. It does slow down after it has downloaded the DB snapshot and it looks likes it starts verifying the blocks it receives. All in, it took about 24 hours to get up to date locally.</p><p>Both Geth and Parity have an RPC interface that can be accessed via a Web3 interface in various languages. I have to say the documentation is pretty good (with examples) to start querying the blockchain.</p><p>I started messing around with the Javascript interface for the YouTube clip ^up there, but when I realised I needed to start getting the data into a format that could be queried, I started working with the <a href="https://github.com/izetex/web3-eth">Ruby Web3 Gem</a> for bulk imports.</p><p>For my local developer instance, I’ll use Mongo installed on my Mac, and I’ll use Mongoid for the interface from Ruby.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff0d711efc338eef4d91c547e7a1460b/href">https://medium.com/media/ff0d711efc338eef4d91c547e7a1460b/href</a></iframe><p>I initially started the exercise by structuring the data within Mongo as:</p><pre>block: {<br>  block_id: NUMBER<br>  transactions: [tx1, tx2, tx3]<br>}</pre><p>But realised that the complexity and inefficiency of querying an array within a document was not going to allow me to search efficiently by Contract or Token on the Blockchain — something I saw as a key data point to glean something useful.</p><p>The format of the data now is a 1:1 mapping of an Ethereum Transaction to a Document within the collection.</p><p>The code should be relatively self explanatory, maybe apart from the syncing check.</p><p>When the Ethereum client is attached to the network, it will attempt to sync the database locally. As this is happening, you really can only read and import data that is local, you can’t query the network itself. Because I didn’t want to wait, I would ask the local RPC endpoint what the latest block is that has been synced with</p><pre>web3.eth.syncing</pre><p>When this is in Production, the assumption is that it will be fully synced, and only receiving new Blocks as they appear on the network.</p><p>This is when we need to call</p><pre>web3.eth.blockNumber</pre><p>In my design, the import script will actually be sitting within a Container on a Kubernetes cluster (as will Parity, running as a Service), and it will loop to ingest any new data. I get that I could have subscribed to events on new Blocks, but I’m not looking for realtime data, I’m ok with near-realtime and some resilience to make sure I have all data. 1 minute delay is not going to cause any Data =&gt; Information quality issues.</p><p>I now have a lot of data in a queryable format — there is still a lot that has to be done to make sense of this, and that is the journey I’ll be documenting over the next few weeks.</p><p>Oh, and at last count (I still don’t have all data ingested), I’m looking at upwards of 150,000,000 documents beating the crap out of my poor Mac SSD. But that’s enough for a Dev environment for now.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F41d817%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F41d817&amp;image=https%3A%2F%2Fe.enpose.co%2F%3Fkey%3DdRXnS9Gplk%26w%3D700%26h%3D425%26url%3Dhttps%253A%252F%252Fupscri.be%252F41d817%252F%253Fenpose&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebbe4d33ac6b8f6af3724807b028c5eb/href">https://medium.com/media/ebbe4d33ac6b8f6af3724807b028c5eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ab3d4ee1af6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coinmonks/misadventures-in-react-with-ethereum-7ab3d4ee1af6">Misadventures in React with Ethereum</a> was originally published in <a href="https://medium.com/coinmonks">Coinmonks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fluid: Financial Language Understanding — Part 1]]></title>
            <link>https://medium.com/@justindavies/fluid-financial-language-understanding-part-1-2c8789ac04b5?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c8789ac04b5</guid>
            <category><![CDATA[natural-language]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[financial-markets]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Mon, 05 Jun 2017 12:39:57 GMT</pubDate>
            <atom:updated>2017-06-05T12:45:21.444Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*olG-8jBUAvgyXCKb4yFZ3w.jpeg" /><figcaption>“<a href="https://www.flickr.com/photos/rednuht/479370088/in/photolist-JmU2w-g6S7MU-e1GtBw-h8bbvp-5GKtTW-4AJsq9-rdAzGv-oV9nTP-46uZse-97DT4K-e1Aj6T-ahtKQx-jKFeyi-e1FY7m-baeqZX-5vSvLu-eCxPN7-8FBu6H-JYFT48-eZiVF4-7gi33H-6k9eLw-mBY3DR-fSBECd-4oQ9Kj-7o9nna-4YCTqZ-aK2w2a-bHoAAr-7qRivx-eUQk7-5UqQZJ-aK2wGH-6boZSt-6b7FUu-eHeALt-gxDpot-oV9nEw-7jzUXD-5wBq3U-hqyagf-kJA7Bj-SNh6A6-oW7p6e-neNw5k-dRsLkh-6Z5BTR-dR4VcT-bHmtPK-h5aDdn">That was supposed to be going up, wasn’t it?</a>” by <a href="https://www.flickr.com/photos/rednuht/">Rafael Matsunaga</a> is licensed under <a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></figcaption></figure><h3>Total Failure</h3><p>Over the past few years I set myself the task of breaking the financial markets by using machine learning. I don’t have a PhD, and I actually failed my Maths A-Level — Undurprisingly I’m not a millionaire, and wasn’t succesful at cracking the signal through the noise.</p><p>What it did allow me to do was to mess around with Neural Networks, starting with Ruby Fann, then moving to Encog, and then Keras where I started playing with more complex models.</p><p>I played around with two types of analysis:</p><ol><li>Pumping <a href="http://www.investopedia.com/terms/t/technicalindicator.asp">Technical indicator</a> values as timesteps into <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">LSTM </a>layers and doing an awful lot of hyper paramater optimisation</li><li>NLP using using modified versions of the <a href="https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py">Keras IMDB LSTM</a> models connected to a subset of the Twitter firehose</li></ol><p>Neither of these gave me the edge, and in fact performed pretty badly (to the point that it felt like I needed a “do the total opposite of the prediction” switch)</p><p>This hasn’t stopped me playing around with this pretty tought problem for the reason that it taught me a lot about how Neural Networks function, and allows me to apply those methods and models I created to problems that can fit.</p><p>Over time, I’ll Open Source the models and code as someone may find them useful. They are pretty hackish as they were usually created on the Tube or when I had some spare time (of which has dwindled since we’ve had two children)</p><h3>Natural Interface to Financial Markets</h3><blockquote><em>I work at Microsoft in the Azure team focussing on Open Source workloads for our customers, and it means I get to play around with some really nice technology on a daily basis. A lot of this related to Machine Learning, and massively scalable infrastructure — 2 things I am passionate about, and have experience of from trying to crack the markets.</em></blockquote><p>I’m going to take some of the statistical analysis I’ve used in the past to bolt together Azure services in different ways. The first of these is something I’ve aptly named FLUID, Financial Language Understanding in Data (you can probably tell I came up with the acronym before the title).</p><p>It will tie in LUIS, WebApps for Linux, Docker, Azure Functions and CosmosDB to allow an end user to have a conversation with a bot about financial instruments.</p><p>For example “Is Microsoft undersold?” or “What’s the instrument with the highest return in the Financial sector over the past 2 weeks?” would be translated into a calculation of the <a href="http://www.investopedia.com/terms/r/rsi.asp">Relative Strength Index</a> of the MSFT ticker and a breakdown of the instruments in the FTSE Finance sector respectively.</p><p>It’s relative simple, but does rely on the integration of some nice Cloud technology.</p><p>I’ll do my best to not tie the code or the implementation to Azure specifically so it can be re-used as easily as possible — but for LUIS and Azure functions it’s difficult to get away from the reliance on Microsoft products (although feel free to raise a PR if you want to modify the code!)</p><h3>The dev environment</h3><p>The first thing I need to do is setup a local dev environment, for this I’ll use Windows Subsystem for Linux. I won’t go into the details of setting it up, as Jack Hammons (Program Manager for WSL) has nicely put this together for us here <a href="https://msdn.microsoft.com/en-gb/commandline/wsl/install_guide">https://msdn.microsoft.com/en-gb/commandline/wsl/install_guide</a></p><p>I’ll be developing locally on MongoDB with the intention of pushing this to Azure Cosmos which has Mongo API support built in and provides a nice globally redundant service for me (it’s inexpensive too, which I like)</p><pre>justin@MININT-SK0C811:~$ sudo apt-get install mongodb</pre><p>And just to check we can connect, and to create the Fluid database</p><pre>justin@MININT-SK0C811:~$ mongo<br>MongoDB shell version: 2.6.10<br>connecting to: test<br>&gt; use fluid<br>switched to db fluid<br>&gt;</pre><p>Ok, we’re all good there!</p><p>I’ll be writing most of the code for Fluid in Python through VS Code, but running it under WSL from now on. But before that we need data to work with</p><h3>Data</h3><p>One of the best places to get data for the markets (legitimately) is <a href="http://www.quandl.com/">Quandl</a>. They provide an End of Day (EOD) dataset for just over 3,000 US companies that I’ll use for this excercise that you can find here:</p><p><a href="https://www.quandl.com/data/WIKI-Wiki-EOD-Stock-Prices">https://www.quandl.com/data/WIKI-Wiki-EOD-Stock-Prices</a></p><p>The data is good, updated every day the markets are open and includes the information we need to calculate the indicators for the Bot.</p><p>Head over to Quandl and sign up for an account. The EOD data is free to use.</p><p>It is lacking in any metadata about the companies though and for that we will need to get hold of some data from NASDAQ:</p><p><a href="http://www.nasdaq.com/screening/companies-by-industry.aspx?exchange=NASDAQ&amp;render=download">http://www.nasdaq.com/screening/companies-by-industry.aspx?exchange=NASDAQ&amp;render=download</a></p><p>This provides a nice amount of metadata that will help us when it comes to displaying user friendly responses (like the ticker, company name, sector and industry amongst others).</p><p>With both datasets, we are looking ato achieve something along the lines of this relationship:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9zJj8qGyCFqCmQlvpx2jw.jpeg" /></figure><h3>Import the NASDAQ company information to Mongo/Cosmos</h3><p>I want to make sure that the user experience is as friendly as possible, allowing the user to ask questions about the company, sector and industry that interests them. No that I know where to get the NASDAQ data from, we will use just that market for the rest of the system.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6468e9fab3c5c18aaf1c8a9692e0f26a/href">https://medium.com/media/6468e9fab3c5c18aaf1c8a9692e0f26a/href</a></iframe><p>This will import all the data from the NASDAQ list and place it into the “companies” collection in the fluid database. Once this has executed I’ll need to check the data looks good:</p><pre>justin@MININT-SK0C811:~$ mongo fluid<br>MongoDB shell version: 2.6.10<br>connecting to: fluid<br>&gt; db.companies.findOne()<br>{<br> “_id” : ObjectId(“59354502e89b3a02fd053345”),<br> “Sector” : “Finance”,<br> “LastSale” : “7.8”,<br> “Name” : “1347 Property Insurance Holdings, Inc.”,<br> “” : “”,<br> “Summary Quote” : “<a href="http://www.nasdaq.com/symbol/pih">http://www.nasdaq.com/symbol/pih</a>”,<br> “Symbol&quot; : “PIH”,<br> “Industry” : “Property-Casualty Insurers”,<br> “MarketCap” : “46462774.8”,<br> “IPOyear” : “2014”,<br> “ADR TSO” : “n/a”<br>}<br>&gt; db.companies.find({“Symbol”:”MSFT”})<br>{ “_id” : ObjectId(“59354509e89b3a02fd053a9b”), “Sector” : “Technology”, “LastSale” : “71.7501”, “Name” : “Microsoft Corporation”, “” : “”, “Summary Quote” : “<a href="http://www.nasdaq.com/symbol/msft">http://www.nasdaq.com/symbol/msft</a>&quot;, “Symbol” : “MSFT”, “Industry” : “Computer Software: Prepackaged Software”, “MarketCap” : “553947704000.723”, “IPOyear” : “1986”, “ADR TSO” : “n/a” }<br>&gt;</pre><p>Looks good, and forms a basis to start importing the historical data from Quandl.</p><h3>Importing the EOD Data from Quandl</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6696c975febaef038a94f3009f195ae/href">https://medium.com/media/c6696c975febaef038a94f3009f195ae/href</a></iframe><p>For all of the technical analysis that will be needed I only need a couple of months worth of data. Which is the reason I’m limiting the lookback to be 60 days.</p><p>A few other things to note is that if you’re using this code, make sure to get your Quandl API key from the Account Settings drop down in your profile. Quandl limits the amout of calls to 10,000 per day which is more than sufficient for this initial data load.</p><p>It also seems that the NASDAQ data and the Quandl data do not match up 1:1, or even at all in some cases. One the import of candles has happened, there are quite a few companies that do not have any EOD data associated with them. This will be dealt with in the application code to make sure only companies with associated candles can be queried.</p><h3>Next Steps</h3><p>Now that there is data to base our application on I can move on to creating the application logic to query the data and start makins some sense of it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c8789ac04b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up CNTK and Keras in Bash for Windows]]></title>
            <link>https://medium.com/@justindavies/setting-up-cntk-and-keras-in-bash-for-windows-2944ecf46d1a?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2944ecf46d1a</guid>
            <category><![CDATA[microsoft]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cntk]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Thu, 01 Jun 2017 19:16:23 GMT</pubDate>
            <atom:updated>2017-06-01T21:09:16.919Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*QLkjZYOUXlyBGw-eoPPZMA.jpeg" /><figcaption><a href="https://www.flickr.com/photos/worldeconomicforum/15715179743/in/photolist-pWGpqT-UWPaUF-9hsjrp-pA3s3H-rgEbtE-hJqVVF-TTzrdP-pZRzTA-8k1fxD-JnqRip-ew5wrU-aAmjoe-PjXmyo-edqN81-bC5pYQ-vqkm17-pcoyup-dXwSET-dXwTCt-fqMRMt-dxgNAM-dXCBcb-ru1KdE-cRXHES-TSHtwh-Tejmbo-RY8Wmg-ak5vn4-dXCBRN-qrbtzK-dXwS7K-fiMR2z-pWtvbN-mGCsGt-fj35Eu-2jz4DA-SBgY8x-rEy5kW-oAFGe7-fomz5e-qvLrq2-fj3pr1-fj2Zvb-5RZgxT-dXCxvm-pv1noK-dXwWoX-qTjzrM-pvfTab-fiMMMn">Human vs Artificial Intelligence</a> by <a href="https://www.flickr.com/photos/worldeconomicforum/">World Economic Forum</a> under <a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></figcaption></figure><p>This got me very excited. I’ve followed the CNTK development since I joined Microsoft, but have never had the time to wrap my head around using it. That is until version 2 went GA today. It also included Keras support which is my go to framework (using Theano at the moment for my dev work).</p><p>As I am part of Open Source team at Microsoft, I heavily rely on WSL (or Bash for Windows) for most of my development so wanted to quickly run through how to get CNTK and Keras working nicely.</p><p>Firstly as this is based on a fork of Keras, setup a new virtualenv or conda environment to keep things clean:</p><pre>justin@MININT-SK0C811:~$ virtualenv cntk-keras<br>Running virtualenv with interpreter /usr/bin/python2<br>New python executable in /home/justin/cntk-keras/bin/python2<br>Also creating executable in /home/justin/cntk-keras/bin/python<br>Installing setuptools, pkg_resources, pip, wheel...done.</pre><pre>justin@MININT-SK0C811:~$ source cntk-keras/bin/activate<br>(cntk-keras) justin@MININT-SK0C811:~$</pre><p>And install the fork of Keras with CNTK support (this will be integrated into Keras soon hopefully)</p><pre>(cntk-keras) justin@MININT-SK0C811:~$ pip install git+<a href="https://github.com/souptc/keras.git">https://github.com/souptc/keras.git</a><br>Collecting git+<a href="https://github.com/souptc/keras.git">https://github.com/souptc/keras.git</a><br> Cloning <a href="https://github.com/souptc/keras.git">https://github.com/souptc/keras.git</a> to /tmp/pip-89v4bF-build<br>Collecting theano (from Keras==2.0.4)<br> Downloading Theano-0.9.0.tar.gz (3.1MB)<br> 100% |████████████████████████████████| 3.1MB 308kB/s<br>Collecting pyyaml (from Keras==2.0.4)<br> Downloading PyYAML-3.12.tar.gz (253kB)<br> 100% |████████████████████████████████| 256kB 1.8MB/s<br>Collecting six (from Keras==2.0.4)<br> Using cached six-1.10.0-py2.py3-none-any.whl<br>Collecting numpy&gt;=1.9.1 (from theano-&gt;Keras==2.0.4)<br> Downloading numpy-1.12.1-cp27-cp27mu-manylinux1_x86_64.whl (16.5MB)<br> 100% |████████████████████████████████| 16.5MB 70kB/s<br>Collecting scipy&gt;=0.14 (from theano-&gt;Keras==2.0.4)<br> Downloading scipy-0.19.0-cp27-cp27mu-manylinux1_x86_64.whl (45.0MB)<br> 100% |████████████████████████████████| 45.0MB 27kB/s<br>Building wheels for collected packages: theano, pyyaml<br> Running setup.py bdist_wheel for theano … done<br> Stored in directory: /home/justin/.cache/pip/wheels/d5/5b/93/433299b86e3e9b25f0f600e4e4ebf18e38eb7534ea518eba13<br> Running setup.py bdist_wheel for pyyaml … done<br> Stored in directory: /home/justin/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc<br>Successfully built theano pyyaml<br>Installing collected packages: numpy, scipy, six, theano, pyyaml, Keras<br> Running setup.py install for Keras … done<br>Successfully installed Keras-2.0.4 numpy-1.12.1 pyyaml-3.12 scipy-0.19.0 six-1.10.0 theano-0.9.0</pre><p>CNTK requires a couple of pre-requisites, namely OpenMPI, so let’s get that installed too</p><pre>(cntk-keras) justin@MININT-SK0C811:~$ sudo apt-get install openmpi-bin</pre><p>GPU support isn’t in Windows Subsystem for Linux yet, so you will have to use the CPU version of CNTK</p><pre>(cntk-keras) justin@MININT-SK0C811:~$ pip install <a href="https://cntk.ai/PythonWheel/CPU-Only/cntk-2.0-cp27-cp27mu-linux_x86_64.whl">https://cntk.ai/PythonWheel/CPU-Only/cntk-2.0-cp27-cp27mu-linux_x86_64.whl</a></pre><p>Edit ~/.keras/keras.json (if it doesn’t exist, you can create the directory, and file)</p><pre>(cntk-keras) justin@MININT-SK0C811:~$ cat ~/.keras/keras.json<br>{<br> “image_dim_ordering”: “tf”,<br> “epsilon”: 1e-07,<br> “floatx”: “float32”,<br><strong> “backend”: “cntk”</strong><br>}</pre><p>The important part here is to set the backend to cntk.</p><p>So once that’s all integrated, get hold of one of the Keras demos and give it a whirl:</p><pre>(cntk-keras) justin@MININT-SK0C811:~$ python imdb_bideirectional_lstm.py<br>Using CNTK backend<br>Selected CPU as the process wide default device.<br>/home/justin/cntk-keras/local/lib/python2.7/site-packages/keras/backend/cntk_backend.py:19: UserWarning: CNTK backend warning: GPU is not detected. CNTK’s cpu version is not fully optimized, please run with GPU to get better performance.<br> “CNTK backend warning: GPU is not detected. CNTK’s cpu version is not fully optimized, “<br>Loading data…<br>Downloading data from <a href="https://s3.amazonaws.com/text-datasets/imdb.npz">https://s3.amazonaws.com/text-datasets/imdb.npz</a><br>17424384/17464789 [============================&gt;.] — ETA: 0s25000 train sequences<br>25000 test sequences<br>Pad sequences (samples x time)<br>x_train shape: (25000, 100)<br>x_test shape: (25000, 100)<br>Train…<br>Train on 25000 samples, validate on 25000 samples<br>Epoch 1/4<br>23488/25000 [===========================&gt;..] — ETA: 167s — loss: 0.4404 — acc: 0.7893</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2944ecf46d1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[If you’re looking for a way to use Gensim to setup a doc2vec model, I found the following works…]]></title>
            <link>https://medium.com/@justindavies/from-gensim-models-doc2vec-import-labeledsentence-9b631f9f567f?source=rss-77e48ab429d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b631f9f567f</guid>
            <category><![CDATA[word2vec]]></category>
            <category><![CDATA[gensim]]></category>
            <category><![CDATA[doc2vec]]></category>
            <dc:creator><![CDATA[Justin Davies]]></dc:creator>
            <pubDate>Mon, 06 Jun 2016 18:27:59 GMT</pubDate>
            <atom:updated>2017-05-21T10:58:16.289Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re looking for a way to use Gensim to setup a doc2vec model, I found the following works rather well for my use case.</p><p>from gensim.models.doc2vec import LabeledSentence</p><p>from os import listdir</p><p>from os.path import isfile, join</p><p>import gensim</p><p>import DocIterator as DocIt</p><p>docLabels = []</p><p>docLabels = [f for f in listdir(“/Users/justin/DeepLearning/suck/GBP_USD/train/neu”) if f.endswith(‘.txt’)]</p><p>data = []</p><p>for doc in docLabels:</p><p>with open(“/Users/justin/DeepLearning/suck/GBP_USD/train/neu/” + doc, ‘r’) as f:</p><p>data.append(f.read())</p><p>it = DocIt.DocIterator(data, docLabels)</p><p>model = gensim.models.Doc2Vec(size=300, window=10, min_count=5, workers=3,alpha=0.04, min_alpha=0.005) # use fixed learning rate</p><p>model.build_vocab(it)</p><p>for epoch in range(100):</p><p>print(“Epoch “ + str(epoch))</p><p>model.train(it)</p><p>print(model.docvecs.most_similar([“6605c7c39fc7d99889fc047488dc9e33.txt”], topn=10))</p><p>model.alpha -= 0.002 # decrease the learning rate</p><p>print(model.alpha)</p><p>model.min_alpha = model.alpha # fix the learning rate, no deca</p><p>model.train(it)</p><p>print(model.docvecs.most_similar([“6605c7c39fc7d99889fc047488dc9e33.txt”], topn=10))</p><p>model.save(“doc2vec.model”)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b631f9f567f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>