<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Steve Stagg on Medium]]></title>
        <description><![CDATA[Stories by Steve Stagg on Medium]]></description>
        <link>https://medium.com/@stestagg?source=rss-705119ceadb3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*D_oslcN6GAGj7MfmhZVaew.png</url>
            <title>Stories by Steve Stagg on Medium</title>
            <link>https://medium.com/@stestagg?source=rss-705119ceadb3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:25:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stestagg/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stealing Secrets from Developers using Websockets]]></title>
            <link>https://medium.com/@stestagg/stealing-secrets-from-developers-using-websockets-254f98d577a0?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/254f98d577a0</guid>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Thu, 21 May 2020 09:15:54 GMT</pubDate>
            <atom:updated>2020-05-21T09:33:21.680Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R5ARiU6dfA9R7JYE_WkDJw.jpeg" /></figure><p>This is a story of a convoluted, not-very-useful method for extracting codez from unwitting JavaScript developers working on top secret projects.</p><p>A couple of articles have hit the sites recently about websites abusing websocket functionality to port-scan user’s computers’. For example: <a href="https://nullsweep.com/why-is-this-website-port-scanning-me/">https://nullsweep.com/why-is-this-website-port-scanning-me/</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvWTcXDRaBGg3cxB1bhbHw.jpeg" /></figure><p>The reason these techniques work is because browsers allow websockets from public origins to open websockets connections to localhost without many protections.</p><p>This got me thinking. I know that popular JavaScript frameworks use websockets in development to automatically reload pages when content changes. Could a malicious website eaves-drop on that traffic, and find out when developers are saving their code?</p><p>The reality was slightly worse than I had thought.</p><h3>The Scheme</h3><ol><li>Either set-up, or inject advertising malware into a popular site that front-end developers tend to frequent. Let’s say <a href="http://frontend-overflowstack.com/">http://frontend-overflowstack.com/</a></li><li>On this page, add code that tries to open websockets connections to common ports (scanning 10k ports takes a second or so, so you can be quite generous here)</li><li>If the page manages to open a connection, hold it open, and forward all messages received to your secret database of nefariousness.</li><li>?</li><li>Profit</li></ol><h3>Does it work?</h3><p>I’ve hosted a very simple page at: <a href="http://frontend-overflowstack.com/">http://frontend-overflowstack.com/</a>. On load, it tries to connect a websocket to every port between 2,000 and 10,000 on the visitors computer (barring a few that Firefox doesn’t allow). If a port connects, then it listens to that port and outputs any messages received. <strong>This page does not save or otherwise transmit any captured data, it is only displayed temporarily on screen.</strong></p><p>If any output appears on this page, an actually malicious site could easily send that output to any server it wants.</p><h4><strong>Generating Data</strong></h4><p>To test this concept out, we need a simple web server that uses hot-reloading. This is the simplest I could come up with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48b1119803a50a77d7969d67095b0471/href">https://medium.com/media/48b1119803a50a77d7969d67095b0471/href</a></iframe><p>Which, when run, starts up a server on port 3000, a websocket server on port 9856, and sends a message: reload to any connected websocket clients every 5 seconds.</p><p>If we fire up our sniffer site, the following appears:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*tlQa-JTHf2dR_AztcmsbTA.png" /></figure><p>So frontend-overflowstack.com is directly eaves-dropping on reload messages being sent by a local dev server to my local browser.</p><p>At this stage, it’s possible to sit back and gleefully count how many times each visitor to our site makes changes to their local JavaScript code, but can we use this to get more info?</p><h4>The plot thickens</h4><p>The majority of front-end development these days seems to involve using React, and typically this involves running the <em>webpack-dev</em> server, which includes its own, more fancy web-socket interface.</p><p>This server shares much more, only slightly interesting, data over its websocket. Demonstrating this is as simple as invoking create-react-app:</p><pre>$ npx create-react-app test<br>$ cd test/<br>$ npm start</pre><p>If we run this, and look at our evil site again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*Pf5GvjHGE78GkFsGwU3CfQ.png" /></figure><p>Instantly there’s more data being shown, we’re getting hashes and status messages, all the useless infos.</p><p>But what happens when the developer makes a typo? The webpack dev server helpfully tries to send a bunch of debugging and stack information to the developer’s screen, by way of its websocket connection.</p><p>Luckily, our evil site can see this too:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GSvxlrQqAci5oFqcCGpQJg.png" /></figure><p>Now things are getting juicy. We’ve got code snippets, paths to files, locations, all sorts of bits of useful info.</p><p>It gets even better if eventually the Dev accidentally typos on a line containing useful data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*AYuqFr8n8-_nhrqxCc0LqA.png" /></figure><p>Now we’ve got a copy of this developer’s AWS Dev credentials. Quick, fire up the bitcoin miners!</p><p>Anatomy of the “Attack”</p><p>No technical design is complete without some form of diagram. Here’s how this works, graphically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEtzI8DntZ9-fqzcnC46iQ.png" /></figure><p>(To simplify the diagram, I have omitted the local web server running, and pretended that the websocket server originates from within the editor directly)</p><p>A malicious web page on some browser tab silently connects to open websockets on the user’s machine. When sensitive data is sent over that socket (from a process that expects to be communicating over a local-only channel) the website can receive that message data, and forward it on to any external database.</p><h3>Threat Assessment</h3><h4>Limiting factors</h4><p>In all seriousness, this attack vector is pretty slim. You’ve got to tempt unwitting users to visit your site, and to stay on it while they’re developing JS code.</p><p>You’ve then got to wait to get lucky to glean morsels of data from their coding mistakes, to maybe find an opening that allows you to profit from this data.</p><h4>Compounding Considerations</h4><p>However, we’ve already seen that various sites are already using websocket port-scanning technology without much in the way of general developer awareness. Given that JS tooling tends to use a small number of well-known ports, writing a script to exfiltrate react Dev traffic subtly would not be particularly hard.</p><p>Imagine an internal developer working for <strong>Twitbook</strong> just pressing save in their editor and causing that access token or internal server address to be leaked to the wrong audience.</p><p>The <em>slightly</em> scary aspect of this is that a reasonable developer should have a general expectation that pressing save in their code editor of choice should have effectively 0 chance of causing data to be leaked to a third-party web service. This attack raises that chance enough to be a tiny bit concerning.</p><h4>Remediation</h4><p>I pursued this vector of trying to intercept JavaScript hot-reloading mechanisms because it’s really the only general use of websockets that I’m familiar with. <em>Discord also uses websockets, but a passing glance at that didn’t yield any obvious results, as that channel is designed with the public internet in mind.</em></p><p>It’s worrying that just this one simple use-case of a one-way communication channel for reloading has exposed so much potential information to bad websites.</p><p>Given this, it’s likely that other uses of websockets (for data not designed for the public internet) may be similarly compromised.</p><p>Arguably the webpack-dev server should do some authentication or that alternate browser communication channels be used for hot-reloading (I believe this is already being planned for other reasons).</p><p>It certainly feels like it’s the way that browsers/web standards implement origin policies for websockets is surprising, and is resulting in software designed for local-only development to be exposed to the public internet in a non-ideal way.</p><p>I would expect any fix to focus on implementing extra controls in the browsers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=254f98d577a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we Built pythonjobs.github.io in a week]]></title>
            <link>https://medium.com/hackernoon/building-pythonjobs-github-io-6119cd708802?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6119cd708802</guid>
            <category><![CDATA[recruiting]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[python-jobs]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Sat, 28 Apr 2018 08:36:00 GMT</pubDate>
            <atom:updated>2019-05-18T04:04:49.516Z</atom:updated>
            <content:encoded><![CDATA[<h4>Modern open source tools are amazing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YX04w-q4aEOO_XT2kleYRg.png" /></figure><p>A while ago, some colleagues (@salimfhadley and @lordmauve) mentioned an idea that they’d had:</p><blockquote>how hard would it be to build a job site that used git pull requests for new job submissions?</blockquote><p>At the time, the python.org jobs board had been down for maintenance for nearly a year, and there wasn’t any obvious free, moderated place to list or view job opportunities. Coincidentally, at that time, our employer was hiring, and we were all involved with that process.</p><p>Their idea was to to have a static site, using GitHub pull requests to manage submissions.</p><p>This idea is awesome for a number of reasons:</p><ul><li>Lots of tools and services exist to make managing change requests from strangers easy</li><li>The entire process is public, transparent, and well understood</li><li>Having a slightly more technical submissions process naturally attracts more technically aware submittors.</li><li>It’s all free to run! (apart from some stock image costs, and our personal time)</li></ul><p>Over about the next week, we built <a href="https://pythonjobs.github.io/">https://pythonjobs.github.io</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XBQncMFBToB6YDeHqWyphQ.png" /><figcaption>Screenshot of the pythonjobs.github.io front page</figcaption></figure><p>And it was wildly successful in as much as:</p><ul><li>despite no advertising or promotion (except right at the start), we receive several submissions a week, and have a steady stream of traffic, sustained for several years.</li><li>It was a factor in the official python.org jobs board coming online a few weeks after we launched 😎.</li></ul><h3>The Static Site</h3><p>The site is built using <a href="https://github.com/hyde/hyde">hyde</a>, a python jekyll alternative (hyde’s development progress has been slow, and it’s documentation is kinda sparse, so I would hesitate to use it for new projects, but it has suited our purposes very nicely).</p><p>Advertisers submit pull requests using the <a href="https://hackernoon.com/tagged/github">GitHub</a> pull request UI: Each listing is a separate Markdown file (with a .html file extension):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aP033GA-KYGc1mhQrXOSbw.png" /></figure><p>With a YAML header, and Markdown contents:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cRX5x7vMe7WHfDSvSVyC1w.png" /></figure><p>Once a change is merged, the site is built using a <a href="http://travis-ci.org/">Travis</a> job, and automatically deployed.</p><h3>Moderation</h3><p>Reviewing large yaml/markdown files manually is time consuming and error-prone, so we built a set of tests that run on each pull request to ensure that the basics check-out (we still check the content of each submission manually, but it’s more about the textual content, rather than syntax).</p><p>This is implemented as a hyde pluign that runs on each node in the jobs directory, and checks many things. For example, that a contact name has been provided:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*awA7qdC-6BVVRssIHXy4ew.png" /><figcaption>Excerpt from our <a href="https://github.com/pythonjobs/template/blob/master/hyde/jobcheck.py">job testing script</a></figcaption></figure><p>If any of these don’t pass then, as with any CI test, the Pull Request is marked as failed, stopping us from blindly approving it.</p><p>One problem we had was that the actual error that caused the failure can be quite hidden. The reason for failure is shown in the travis build logs, as a traceback, with a friendly message, but..</p><p>Most of the people sending us adverts are recruiters, or non-technical hiring managers. Often this is their first dealing with GitHub. To get to the stage of creating a PR, they’ve already had to get to grips with a lot of unfamiliar concepts (git/github, markdown, yaml, pull requests, etc..). Asking them to realise (with no clear indications) that they have to click through to the build failure logs, parse the output till they find the traceback, and then extract the relevant information is a step (or 10!) too far.</p><p>This led me to recently add a tool that comments directly on the PR with friendly details (similar to <a href="https://www.travisbuddy.com/">https://www.travisbuddy.com/</a>). The comment lays out why the tests failed in a friendly, easily understood manner:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*baAW-tEH7CtI_ItyGYdeig.png" /></figure><p>When the tests pass, there is another script that grabs a screenshot of the listing using phantomjs and selenium, and posts that <a href="https://github.com/pythonjobs/jobs/pull/327#issuecomment-384467178">as a comment</a>, to make reviewing easier.</p><h3>Infrastructure</h3><p>We use three git repos to run pythonjobs. This seems more complicated than needed, but is designed to keep the submissions process simple:</p><ul><li><a href="https://github.com/pythonjobs/jobs"><strong>jobs</strong></a><strong>:</strong> just holds the job listing files, and some travis related scripts. This is the only repo that submitters have to interact with to create a listing.</li><li><a href="https://github.com/pythonjobs/template"><strong>template</strong></a>: Has all the hyde related code for generating the full static site, graphics resources, build scripts, etc.</li><li><a href="https://github.com/pythonjobs/pythonjobs.github.io"><strong>pythonjobs.github.io</strong></a>: Contains the generated static site, directly hosted by github pages.</li></ul><p>The build script pulls together and merges the repos as needed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7T63XTpxSJ2Q2pptUc4nBA.png" /></figure><p>When the PR is merged to master, the same CI job commits the result to the output repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sbuMWkpnZAKmmNmOpzzFew.png" /></figure><p>Github handles all the rest for us, and seconds later, the changes are visible online.</p><p>We push the changes to the live repo by using a travis secure environment variable to hold a github key. Unfortunately, for good reasons, you can’t use secure environment variables in pull requests from non-members (they could change the travis script to make it print out the secrets)</p><p>To get around this, there are some simple Amazon lambda functions to handle generating comments when a PR is tested.</p><h3>Search</h3><p>Despite being a static site, with no server code, we have full-text searching across all job listings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZ-gRief1g-ZjruYKwLsOA.png" /></figure><p>This is implemented very similarly to Sphinx client-side search (I’ve since learned). During build time, <a href="https://github.com/pythonjobs/template/blob/master/hyde/siteindex.py">a script</a> looks at each job listing, and builds a compact JSON based prefix tree from each word in the advert.</p><p>Words are stemmed, using the <a href="https://hackernoon.com/tagged/python">python</a> <a href="https://pypi.org/project/stemming/1.0/">stemming</a> library, and scored by where they appear (words in the title score higher than words in the body).</p><p>This prefix tree is then output as a JSON file which is read by the browser, and used to provide real-time search capability. The JSON first contains an array of each job, followed by the prefix tree as a set of nested maps. Each map leaf gives an index into the job name array, and a corresponding score:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sfgQzOR1l3pL4Uqor9Y37Q.png" /></figure><p>In this example, the word <strong>jack</strong> appears in <em>Job2</em>, with a score of 1. While <strong>java</strong> appears twice (score 2) in <em>Job0</em>, and once in <em>Job1</em></p><p>Highlighting is done client-side with a simple regex based search of the job listing.</p><h3>Locations</h3><p>We ask submitters to give us a text-based location (Washington, USA), and don’t care too much about how precise that location is (some companies don’t like publishing exact locations publicly). From that, we generate a google maps view of all active job listings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Tp3XwmkqvH1Go1ucaZ_KQ.png" /></figure><p>So how do we go from ‘London Bridge, London’ to a marker on the map?</p><p>The google geocoding api can translate location names into the latitude-longitude coordinates required by google maps, so the build script queries this for each job advert to find its exact location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M2tyIsRCfgcAHpUCa7yr7w.png" /></figure><p>Because this is done in public, we can’t use a secret API key to query these locations, and so Google quickly throttle these requests. To avoid this, the jobcheck script uses an N+1 cache, where the results of the previous queries are stored in the <a href="http://pythonjobs.github.io/media/geo.json">site rendered output</a> and used to ensure we only query new locations on each build (almost always just 1 location).</p><p>This data is then turned into inline JS data in the maps.html page, so that we can populate the map on page load with markers.</p><h3>GitHub</h3><p>GitHub is, and should be, primarily a code hosting site. It does this very well. GitHub pages are designed to allow some documentation and light-weight hosting to support code projects.</p><p>Pythonjobs isn’t either of these things, so I asked GitHub in advance if this would be OK for us to do. They were very supportive of the idea, and their friendly replies helped make this site a joy to set up.</p><p>As part of being open, one of Sal’s ideas was to enforce that all submissions name the company being hired for. While this causes problems for some recruiters, it has the benefit of keeping everything simple, there’s little in the way of vague language or concerns about locations.</p><h3>Summary</h3><p>The fact that the three of us could set a site list this up for free, in about a week, and run it for several years with minimal involvement says a lot about the quality and availability of free tooling available to open source projects today.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6119cd708802" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/building-pythonjobs-github-io-6119cd708802">How we Built pythonjobs.github.io in a week</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysing 1.4 billion rows with python]]></title>
            <link>https://medium.com/hackernoon/analysing-1-4-billion-rows-with-python-6cec86ca9d73?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cec86ca9d73</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Wed, 28 Mar 2018 17:45:40 GMT</pubDate>
            <atom:updated>2019-05-18T04:04:50.425Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Using pytubes, numpy and matplotlib</h4><p>The <a href="https://books.google.com/ngrams">Google Ngram viewer</a> is a fun/useful tool that uses Google’s vast trove of data scanned from books to plot word usage over time. Take, for example, the word <em>Python </em>(case sensitive)<em>:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*JBBDttphxwvek-nhV9v6eg.png" /><figcaption>Untitled graph taken from: <a href="https://books.google.com/ngrams/graph?content=Python&amp;year_start=1800&amp;corpus=15&amp;smoothing=0">https://books.google.com/ngrams/graph?content=Python&amp;year_start=1800&amp;corpus=15&amp;smoothing=0</a> charting the usage of the word ‘Python’ over time.</figcaption></figure><p>It’s powered from google’s <a href="https://en.wikipedia.org/wiki/N-gram">n-gram</a> dataset, a log of the number of times a particular word or sequence of words was spotted by <a href="https://hackernoon.com/tagged/google">google</a> <a href="https://hackernoon.com/tagged/books">books</a> during each publication year. While not complete (it doesn’t include every book ever published!), there are millions of books in the set, including books published in the 1500s and up to 2008. The dataset can be <a href="http://storage.googleapis.com/books/ngrams/books/datasetsv2.html">freely downloaded here</a>.</p><p>I decided to see how easy it would be to reproduce the above graph using Python and my new data loading library: <a href="http://github.com/stestagg/pytubes">PyTubes</a></p><h4>Challenges</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTuX_3Xo3bxvtf_GgJTwpA.jpeg" /></figure><p>The 1-gram dataset expands to 27 Gb on disk which is quite a sizable quantity of data to read into python. As one lump, Python can handle gigabytes of data easily, but once that data is destructured and processed, things get a lot slower and less memory efficient.</p><p>In total, there are 1.4 billion rows (1,430,727,243) spread over 38 source files, totalling 24 million (24,359,460) words (and POS tagged words, see below), counted between the years 1505 and 2008.</p><p>When dealing with 1 billion rows, things can get slow, quickly. And native Python isn’t optimized for this sort of processing. Fortunately <a href="https://github.com/numpy/numpy">numpy</a> is really great at handling large quantities of numeric data. With some simple tricks, we can use numpy to make this analysis feasible.</p><p>Handling strings in python/numpy is complicated. The memory overheads of strings in python are quite significant, and numpy only really deals with strings if the length of the string is known and fixed. In this situation, most of the words have different lengths, so that isn’t ideal.</p><h4>Loading the data</h4><blockquote>All of the code/examples below were run on a 2016 Macbook Pro with <strong>8 GB ram</strong>. Hardware/cloud instances with decent quantities of ram should perform much better</blockquote><p>The 1-gram counts are provided as a set of tab-separated files that look like this:</p><pre>Python 1587 4 2<br>Python 1621 1 1<br>Python 1651 2 2<br>Python 1659 1 1</pre><p>Where each row has the following fields:</p><pre>1. Word<br>2. Year of Publication<br>3. Total number of times the word was seen<br>4. Total number of books containing the word</pre><p>To generate the requested graph, we only really need to know some of this information, namely:</p><pre>1. Is the word the one we’re interested in?<br>2. Year of publication<br>3. Total number of times the word was seen</pre><p>By just extracting this information, the overheads of handling variable length string data have been avoided, but we still need to compare string values to identify which rows are about the field we’re interested in. This is where pytubes comes in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90098fd6dc1c3828189e00239132c05b/href">https://medium.com/media/90098fd6dc1c3828189e00239132c05b/href</a></iframe><p>About 170 seconds (3 minutes) later, <em>one_grams</em> is a numpy array with ~1.4 billion rows, looking like this (headers added for clarity):</p><pre>╒═══════════╤════════╤═════════╕<br>│   Is_Word │   Year │   Count │<br>╞═══════════╪════════╪═════════╡<br>│         0 │   1799 │       2 │<br>├───────────┼────────┼─────────┤<br>│         0 │   1804 │       1 │<br>├───────────┼────────┼─────────┤<br>│         0 │   1805 │       1 │<br>├───────────┼────────┼─────────┤<br>│         0 │   1811 │       1 │<br>├───────────┼────────┼─────────┤<br>│         0 │   1820 │     ... │<br>╘═══════════╧════════╧═════════╛</pre><p>From here, it’s just a question of using numpy methods to calculate some things:</p><h4>Total word count for each year</h4><p>Google shows the % occurrence of each word (number of times a word occurs/total number of words published that year) which is somewhat more useful that just the raw word count. To calculate this, we need to know what the total word count is.</p><p>Luckily numpy makes this really simple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8308e8ac48d5676dbc3a2b02d1f1f3d3/href">https://medium.com/media/8308e8ac48d5676dbc3a2b02d1f1f3d3/href</a></iframe><p>Plotting this shows how many words google has collected for each year:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MGpmL__D90H1skGgYO2ibg.png" /></figure><p>What’s clear is that before 1800, the volume of data falls off quickly, and thus can skew results, and hide interesting patterns. To counter this, let’s only include data after 1800:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97043de3e8fa8333e52cd87dc9945322/href">https://medium.com/media/97043de3e8fa8333e52cd87dc9945322/href</a></iframe><p>Which returns 1.3 Billion rows (only 3.7% of words are recorded from before 1800)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVjNfqQb0j-5S_opj4oTIA.png" /></figure><h4>Python % by Year</h4><p>Getting the % counts for python is now surprisingly easy.</p><p>Using the simple trick of making the year-based arrays 2008 elements long means that the index for each year equals the year number, so finding the entry for, say, 1995 is just a question of getting the 1,995 th element.</p><p>It’s not even worth using numpy operations for this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2bf3677e38c684caa11cc0f81af48549/href">https://medium.com/media/2bf3677e38c684caa11cc0f81af48549/href</a></iframe><p>Plotting the resulting word_counts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tJD7p3d6J8Ecl75tHIR5vQ.png" /></figure><p>which turns out pretty similar in shape to Google’s version:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*JBBDttphxwvek-nhV9v6eg.png" /></figure><p>The actual % numbers don’t match up at all, and I think this is because the downloadable dataset includes words tagged with various parts of speech (for example: Python_VERB). This is not explained well in the google page for this dataset, and raises several questions:</p><ul><li>how does one use Python as a verb?</li><li>Do the counts for ‘Python’ include the counts for ‘Python_VERB’? etc.</li></ul><p>Luckily, it’s clear that the method I’ve used produces a similar-enough shape of graph as google that the relative trends are not affected, so for this exploration, I’m not going to try to fix that.</p><h4>Performance</h4><p>Google produces its graph in ~1 second, compared with about 8 minutes with this script, but this is reasonable. The backend to Google’s word count will be working from significantly prepared views of the dataset.</p><p>For example, pre-calculating the per-year total word count and storing it in a single lookup table would save significant time here. Likewise, storing the word counts in a single database/file and indexing the first column would eliminate almost all of the processing time.</p><p>What this exploration <em>does </em>show, however, is that using numpy and the fledgeling pytubes, it’s possible to load, process, and extract some arbitrary statistical information from raw billion-row datasets in a reasonable time, using standard commodity hardware and Python.</p><h3>Language Wars</h3><p>Just to prove the concept with a slightly more complex example, I decided to compare the relative mention rates of three programming languages: <strong>Python, Pascal, </strong>and<strong> Perl.</strong></p><p>The source data is quite noisy (it includes all English words used, not just programming language mentions and, for example, python also has a non-techical meaning!) to try to adjust for this, two things things were done:</p><ol><li>Only the Title case forms of the names are matched (Python, not python)</li><li>Each language’s mention count has been be shifted by the mean % count between 1800 and 1960, Given Pascal as a language was first mentioned in 1970, this should give a reasonable baseline.</li></ol><h4>Results:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AsipoFxV-cE2zIuDqZOiHw.png" /></figure><p>Compared with Google (<em>without any baseline adjustment</em>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aWPxvopsNmbY50WKF8Wvjg.png" /></figure><p>Run Time: Just over 10 minutes</p><p>Code: <a href="https://gist.github.com/stestagg/910859576f44f20e509822365414290d">https://gist.github.com/stestagg/910859576f44f20e509822365414290d</a></p><h4>Future PyTubes Improvements</h4><p>At the moment, pytubes only has a single concept of an integer, which is a 64-bit int. This means that numpy arrays generated by pytubes use i8 dtypes for all integers. In some cases (like the ngrams data) 8-byte integers are a little over-kill, and waste memory (the full ndarray here is about 38Gb, this could easily be reduced by 60% with better dtypes). I plan to add some level of 1, 2 and 4 byte integer support soon (<a href="https://github.com/stestagg/pytubes/issues/9">https://github.com/stestagg/pytubes/issues/9</a>)</p><p>More filter logic — the Tube.skip_unless() method is a simple way to filter rows, but lacks any ability to combine conditions (AND/OR/NOT). This will make reducing the volume of loaded data much faster for some use-cases.</p><p>Better string matching —Simple tests like: startswith, endswith, contains, and is_one_of are easy to add, and significantly improve the usefulness when loading lots of string data.</p><p><em>As always, </em><a href="https://github.com/stestagg/pytubes"><em>patches</em></a><em> are more than welcome!</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cec86ca9d73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/analysing-1-4-billion-rows-with-python-6cec86ca9d73">Analysing 1.4 billion rows with python</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing the second video game for the Micro:bit in Rust]]></title>
            <link>https://medium.com/hackernoon/writing-the-second-video-game-for-the-micro-bit-in-rust-3cd8b5ab22d3?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cd8b5ab22d3</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Mon, 22 May 2017 08:38:29 GMT</pubDate>
            <atom:updated>2019-05-18T04:04:50.526Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uHEt8Fml5bodo5Wqr4OHmw.gif" /></figure><p>I recently published a story about <a href="https://hackernoon.com/the-first-video-game-on-the-bbc-micro-bit-probably-4175fab44da8">bitflyer, a simple game for the BBC micro:bit</a>.</p><p>Off the back of it, Mr Zbit, the maker of <a href="https://twitter.com/ZbitConnect">@ZbitConnect</a>, got in touch, and kindly sent through some of his boards to play with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*uMXAtDXAIK0zyi0yQlM5yA.png" /></figure><p>These boards make it simple to add different attachments to the micro:bit, and are cleverly <a href="https://hackernoon.com/tagged/designed">designed</a> to be chain-able too.</p><p>They work by sandwiching a piece of (I think) conductive elastomer between the various boards’ contacts. The rubber needs a certain amount of pressure to make a connection, hence all the screws an bolts.</p><p>The basic zbit:builder gives a whole lot of standard through-hole connections to solder things to, but the creator is also developing <a href="http://www.nevilhunt.com/zbit-connect/">several other boards</a> that do different things, including a <a href="http://www.nevilhunt.com/zbit-connect/zbitspeaker.html">speaker board</a>.</p><p>I dusted off the <a href="https://hackernoon.com/tagged/bitflyer">bitflyer</a> code, loaded it in, and it just worked first time. For about 30 seconds. Before it ran out of memory and the whole thing crashed. After several hours of stepping through the layers of code, stripping out anything that could go, and trimming things, I gave up and decided on a more radical approach: Rust.</p><h4>tl;dr</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F218426365&amp;url=https%3A%2F%2Fvimeo.com%2F218426365&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F635892474_295x166.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=vimeo" width="540" height="960" frameborder="0" scrolling="no"><a href="https://medium.com/media/992a0b065e8e9de11c47026ba292a27f/href">https://medium.com/media/992a0b065e8e9de11c47026ba292a27f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*hezbs0C0NCGF9Su2UezfUg.png" /></figure><h4>Why <a href="https://www.rust-lang.org/">Rust</a>?</h4><p>Of the languages I considered, Rust made the most sense. The list went something like:</p><ul><li><strong>C</strong>: Too many lines of code</li><li><strong>C++</strong>: Too passé</li><li><strong>Python</strong>: Too memory</li><li><strong>Go</strong>: Too basic — I struggle with the lack of syntactic sugar in the language</li><li><strong>Java</strong>: Too java</li><li>…</li><li><strong>Rust</strong>: New, shiny, aimed at systems-level programming</li></ul><h4>Zinc</h4><p>Having chosen Rust, I had to work out how to get rust running on an embedded board. Luckily there’s a project to do just that: <a href="https://github.com/hackndev/zinc">Zinc</a>.</p><p><a href="https://github.com/hackndev/zinc">Zinc</a>’s goal is a project to produce compiled rust binaries that can run directly on a computer without an OS helping out (bare metal). The project has support for a number of ARM chips/embedded boards, but not the micro:bit nrf51822 (yet!).</p><p>It provides a great platform for getting started, with examples, and a build tool called xargo which handles the linking and organisation. They have a light-weight HAL module, and macros to help with board configuration.</p><h4>hal/nrf51822</h4><p>As mentioned, zinc has some nice macros for configuring safe access to various things, including peripheral access.</p><p>These chips typically allow access to their different systems (I/O, RNG, clocks, etc..) by reading and writing to magic memory locations.</p><p>The following snippet shows the GPIO pin definition for the micro:bit:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15f9b8c869388b1ff9fb9b440ac4a5d1/href">https://medium.com/media/15f9b8c869388b1ff9fb9b440ac4a5d1/href</a></iframe><p>Which can then be used to control GPIO pins:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4328bec28cbf51674fb87364fa1cd499/href">https://medium.com/media/4328bec28cbf51674fb87364fa1cd499/href</a></iframe><p>I spent a long time working out how things like interrupt handling worked, and the subtleties of peripheral control (Turns out that the micro:bit doesn’t actually have a systick interrupt, and the docs don’t make that entirely clear!). But once I’d understood the details, and how to express them using the zinc macros, the implementation was very simple and straight-forward.</p><h4>Music</h4><p>Mr Zbit also sent along a board with a tiny speaker and amp on it. The speaker is attached to one of the gpio pins, allowing it to play very basic, square-wave based sounds.</p><p>Interrupt vectors have always slightly scared me, so doing this was a great learning excercise. ARM chips typically have a table of function pointers, held at a known location in memory, when an interrupt fires, the chip saves all the CPU state, and jumps to the address pointed to by the corresponding entry in this table. After the interrupt function returns, the state is restored, and everything continues as normal.</p><p>How does this look in rust?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/196f69c007c90dd54d745b50e67e7fe3/href">https://medium.com/media/196f69c007c90dd54d745b50e67e7fe3/href</a></iframe><p>The hardest part of this was working out the exact order/offset of the interrupt table. Once this was done, then my isr_rtc0 function was called every few ms.</p><p>The rest was just a case of converting MIDI files into a rust array of frequencies and times, and writing a simple function to toggle GPIO 1 at the right speed to make notes.</p><h4>The code</h4><p>The code is not very clean, and should be tidied up/significantly refactored (hence not being pushed upstream). But may be found here:</p><p><a href="https://github.com/stestagg/zinc/tree/bitflyer/examples/bitflyer">https://github.com/stestagg/zinc/tree/bitflyer/examples/bitflyer</a></p><h4>Thoughts on Rust</h4><p>I actually really loved writing this in Rust. Programming on bare metal meant that I was forced to be a bit cavalier with unsafe (With work, this could have been reduced), and with everything happening on-chip, with little in the way of feedback (The speaker &amp; display are both strictly write-only) I didn’t have to deal with optionals much. Ownership complexity was avoided by having shared global state (This chip was never going to do much in the way of multitasking)</p><p>Given that I was able to avoid many of the pain points mentioned above, and running on nightly giving macros the power to provide nice abstractions, the code became fun to play with, and pleasant to use.</p><p>Speed and memory usage were, of course, blistering compared with Python, and (at times being forced to) looking at and stepping through the generated Assembly, the true value of zero cost abstractions became really clear.</p><figure><a href="http://bit.ly/HackernoonFB"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hqOaABQ7XGPT-OYNgiUBg.png" /></a></figure><figure><a href="https://goo.gl/k7XYbx"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vgw1jkA6hgnvwzTsfMlnpg.png" /></a></figure><figure><a href="https://goo.gl/4ofytp"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKBpq1ruUi0FVK2UM_I4tQ.png" /></a></figure><blockquote><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="/service/mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</blockquote><blockquote>If you enjoyed this story, we recommend reading our <a href="http://bit.ly/hackernoonlatestt">latest tech stories</a> and <a href="https://hackernoon.com/trending">trending tech stories</a>. Until next time, don’t take the realities of the world for granted!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35tCjoPcvq6LbB3I6Wegqw.jpeg" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cd8b5ab22d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/writing-the-second-video-game-for-the-micro-bit-in-rust-3cd8b5ab22d3">Writing the second video game for the Micro:bit in Rust</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a botnet on PyPi]]></title>
            <link>https://medium.com/hackernoon/building-a-botnet-on-pypi-be1ad280b8d6?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/be1ad280b8d6</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Fri, 19 May 2017 20:31:12 GMT</pubDate>
            <atom:updated>2019-05-18T04:04:51.138Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Or being able to</em></p><h4>An update — September 2017</h4><p>A week or so ago, some students applied this concept to the idea of typosqatting (registering malicious packages with names similar to popular libraries). By getting a university to issue a security notice, they generated some interest, and finally resulted in some changes to pypi/warehouse to address these issues.</p><p>I decided to take another look at the download figures for my packages, and see what damage my malicious alter-ego could have wreaked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1kCe4wWgyUe_IteLMduReQ.png" /></figure><p>Across the 12 system module packages I’m hosting, I’m getting on average 1.5 thousand downloads per day, via pip. This adds up to 491,292 downloads so far this year. I’m hoping to hit 500k downloads before my packages are deleted!</p><p>By package, the download ratios pretty much match the numbers from May:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*6765wLQ_oq5EjCyYNke_hQ.png" /></figure><p>There’s a plan to delete my fake packages now that restrictions have been added to prevent this sort of attack, but it was fun while it lasted!</p><h4>Intro</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*colp9WeRXX-3bwCY1zlUaQ.png" /></figure><p>At a London <a href="https://hackernoon.com/tagged/python">python</a> dojo in October last year, we discovered that PyPi allows packages to be registered with builtin module names.</p><p>So what? you might ask. Who would pip install a system package? Well the story goes something like this:</p><ol><li>An inexperienced Python developer/deployer realises they need X functionality</li><li>Googling/asking around, they find out that to install packages, people use pip</li><li>Developer happily types in e.g. pip install sys</li><li>Baddie has registered the sys pip module, and included a malicious payload</li><li>Developer is now pwned by malitious package, but import sys in python works, and imports a functional sys module, so nobody notices.</li></ol><p>When we discovered this, I was pretty interested in how plausible this was as an attack vector, so did a few things:</p><ol><li>Emailed the pypi <a href="https://hackernoon.com/tagged/security">security</a> contacts listed on pypi</li><li>Proactively registered all the common system module names that I could think of, as packages</li><li>Uploaded an empty package to each of them that does nothing other than immediately traceback:</li></ol><pre>raise RuntimeError(&quot;Package &#39;json&#39; must not be downloaded from pypi&quot;)</pre><h4>Why upload anything?</h4><p>It’s perfectly possible to squat on a pypi package and not upload any files. But by adding an empty package, I could track the downloads from the pypi download stats.</p><p>Pypi upload their access logs (sans identifying information) to <a href="https://cloud.google.com/bigquery/">google big query</a>, which is pretty awesome, and allows us to get a good idea of how many systems each package ends up on.</p><h4>How effective is this attack vector?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFwz5yhpMV6JH9RAXSQVcg.png" /></figure><p>Big query says that so far this year (19th May 2017), my dummy packages have been download ~244k times, lucky they’re benign huh, otherwise that’s 1/4 million infected machines!</p><p>Some of the downloads will be people using custom scrapers, others may be automated build jobs, running over and over, but I used some tactics to gauge the quality of this data:</p><ol><li>pypi download logs include a column installer.name this seems equivalent to an HTTP user agent string, by only selecting rows where the installer.name is pip, we’re more likely to be counting actual installs, rather than scrapers, or other bots</li><li>Another column: system.release tracks very high-level system version information (for example 4.1.13–18.26.amzn1.x86_64) By including this in the counts, we can see that lots of different types of setups are downloading these packages, suggesting it’s not just a few bots scraping the site. 3.1k different system versions have downloaded my packages this year, compared with 33k total unique versions across the whole of pypi</li></ol><p>The query I used is here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/660bb8014babcffc6dbf3746890d3831/href">https://medium.com/media/660bb8014babcffc6dbf3746890d3831/href</a></iframe><h4>What now?</h4><p>I never actually received a reply to my email, so a while later, I raised an issue on the official <a href="https://github.com/pypa/pypi-legacy/issues/585">pypi github issue tracker</a> in January. This also got no reply.</p><p>I’m currently squatting all the system package names that seem most at risk, and doing so with benign packages, so I don’t see much of a risk of disclosing this now.</p><figure><a href="http://bit.ly/HackernoonFB"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hqOaABQ7XGPT-OYNgiUBg.png" /></a></figure><figure><a href="https://goo.gl/k7XYbx"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vgw1jkA6hgnvwzTsfMlnpg.png" /></a></figure><figure><a href="https://goo.gl/4ofytp"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKBpq1ruUi0FVK2UM_I4tQ.png" /></a></figure><blockquote><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="/service/mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</blockquote><blockquote>If you enjoyed this story, we recommend reading our <a href="http://bit.ly/hackernoonlatestt">latest tech stories</a> and <a href="https://hackernoon.com/trending">trending tech stories</a>. Until next time, don’t take the realities of the world for granted!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35tCjoPcvq6LbB3I6Wegqw.jpeg" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be1ad280b8d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/building-a-botnet-on-pypi-be1ad280b8d6">Building a botnet on PyPi</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The First Video Game on the BBC Micro:bit [probably]]]></title>
            <link>https://medium.com/hackernoon/the-first-video-game-on-the-bbc-micro-bit-probably-4175fab44da8?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4175fab44da8</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <category><![CDATA[electronics]]></category>
            <category><![CDATA[micropython]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Mon, 27 Feb 2017 17:42:46 GMT</pubDate>
            <atom:updated>2019-05-18T04:04:51.958Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bU1eV419oXLWJFwwKN3AXQ.jpeg" /><figcaption>The title screen</figcaption></figure><p>I was lucky enough to get my hands on a BBC micro:bit, courtesy of <a href="https://medium.com/u/4e8ce0d2751b">Nicholas Tollervey</a> a while back, and wanted to see how far I could push python on it (I started this post in May last year!). This is my story of building a game on the micro:bit using a $2 external display.</p><h3>The hardware</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*OJcjw2DMDmhl2IbudMyJwg.jpeg" /></figure><p>The <a href="https://www.microbit.co.uk/">bbc micro:bit</a> is a tiny computer, <a href="http://www.bbc.co.uk/mediacentre/latestnews/2016/bbc-micro-bit-schools-launch">a million of which</a> have been to English (and Welsh!) school children and schools. It’s got lights, buttons, USB, bluetooth, and runs Python!</p><p>The hardware is fairly limited. It’s built around the nRF51822 chip, which clocks in at 16 MHz, and has 16 KB of RAM. The flash space is slightly more generous, with 256 KB space available. This makes doing more complex things quite challenging and fun.</p><p>The board has two buttons, and a mini screen made up of 25 LEDs which is great for getting people started, but very limiting when drawing asteroids and spaceships! To make things more exciting, I added this tiny display:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*tkE2oAp7viRrvRFDdGQ6wg.jpeg" /></figure><p>Which you can buy for about $2 online. I’ve talked about <a href="https://medium.com/@stestagg/playing-star-wars-on-the-esp8266-with-micropython-5f175fe7b755">this display before</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*fJg5sbktYhXRiv62OZ7MoA.png" /></figure><p>To connect the two, I bought an edge connector from K<a href="https://www.kitronik.co.uk/">itronik</a>, which turns the clever edge connectors on the micro:bit into normal pins for electronics prototyping.</p><p>Given this, and a <a href="https://www.microbit.co.uk/device/pins">pinout from the micro:bit site</a>, that shows the I²C pins (The wire protocol the display uses), and which pins can be used to power the display, I could connect them up. Unfortunately, the pins for the micro:bit and the display didn’t match up exactly, and I had to patch together an adapter that crossed two of the wires, to allow the display to plug directly into the breakout board. I then covered the connector in <a href="https://sugru.com/">sugru</a> to make it look a <em>bit</em> nicer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*9BnOszlPYSFLtvcDHJzFTQ.png" /></figure><h3>The Software</h3><p><a href="https://micropython.org/">Micropython</a> is kinda awesome. An independent implementation of the Python3 language, by <a href="https://github.com/dpgeorge">Damien George</a>, it runs on low-power/memory chips, and is designed for embedded use.</p><p>Ultimately, it’s still Python which, for all its benefits, isn’t designed to be super fast, or memory efficient, both things that traditionally are critical for games. This makes a few things quite tricky, and I’ll explore them below.</p><h3>Bitflyer</h3><p>Here’s a video of the game playing:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F183855864&amp;url=https%3A%2F%2Fvimeo.com%2F183855864&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F593174033_1280.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e77833ce58ff43707351be5dda83841/href">https://medium.com/media/8e77833ce58ff43707351be5dda83841/href</a></iframe><p>My idea was to create a simpler cross between asteroids, and space invaders, with boulders falling from the top of the screen, and the player having to fly a space-ship to dodge them.</p><p>The source code is on github: <a href="https://github.com/stestagg/bitflyer">https://github.com/stestagg/bitflyer</a></p><p>While making this, I hit up against a few of the limitations of the platform, and I’ll discuss these below:</p><h4>Speed</h4><p>Speed actually wasn’t a massive issue. the chip is clocked at 16 Mhz, and micropython is pretty good making use of these cycles.</p><p>There were some cases where I moved some bit-twiddling code into a C-module (see below) which helped with some bottlenecks.</p><p>The main speed issue was with updating the display. I²C isn’t really designed for super high-speed communications. The nRF51822/ssd1306 can communicate with a clock speed of 1 Mhz, but with I²C wire and handshake protocols, this really translates to between 25–40 KBps (bytes).</p><p>To update the entire display is 1 KB of data, so doing this many times a second (the cpu is busy the whole time communication is happening) doesn’t leave much room for doing other things.</p><h4>Memory</h4><p>Memory was a massive issue. The micro:bit has 16 KB ram, which isn’t very much, especially when using python which heavily relies on dynamic allocation.</p><p>The following map shows how the RAM is divided up for micropython (with a lot of the larger built-in modules disabled):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2S_rPnRo0MLPa80QJ1legQ.png" /><figcaption>Map of the RAM usage of the bitflyer code. Labels indicate which system uses that section of memory. Each square represents 32 bytes of RAM</figcaption></figure><p>The important part is the “micropython heap” section. All normal python objects live within this range,</p><p>All normal python objects live within the green ‘micropython heap’ in the above map, which is only able to use ~ half of the total ram.</p><p>Traditionally with embedded systems, all the code is compiled, and added to the flash memory, which on the micro:bit is 256 KB (loads of room!). With python, while the source code can live in flash, micropython has to runtime compile this into objects in RAM. Adding lots of complex code to avoid memory usage in this case isn’t usually an option, as that complex code will also use the same pool of memory as the object you’re trying to compress.</p><p>I didn’t dig into the internals of the micropython parser/compiler, but things as simple as stripping comments and using <a href="https://github.com/liftoff/pyminifier">pyminifier</a> significantly helped with fitting the code in memory.</p><p>In the end, there was a lot of trial-error, changing what variables were used, and in what order, to find a model that reliably ran. I also had to tweak the python heap and stack sizes (make them smaller) to suqeeze everything in.</p><h4>Display Buffer</h4><p>I decided early on to use a display buffer. the display chip I used (ssd1306) has an internal frame buffer, but there is no easy/obvious way of reading current values from it, only for updating them.</p><p>I could have implemented a method of clearing/<a href="https://hackernoon.com/tagged/repainting">repainting</a> sections of the screen based on keeping track of what objects have moved since last frame, and calculating the delta, but this method has some drawbacks: the code/memory cost of tracking what has changed each time is quite high, and communications with the display are fairly slow, so lots of small update commands would introduce noticable lags/delays.</p><p>Instead, I decided to keep an in-memory ‘view’ of the screen, and use that to do efficient updates each time. This was a big cost, the screen has 128x64 pixels, with each pixel either being on or off, so the screen buffer is 1 KB, or 1/16 of the machine total memory. The advantage of this approach is that updating the screen is far simpler, as you just have to keep track of which rectangle of screen contains changes, and do a single bulk-transfer of data to update the display.</p><p>Originally, I used a python array to store the screen buffer, and immediately ran into crippling memory issues. The problem turned out to be fragmentation. The micropython heap was only 9 KB and having a big 1 KB slab of unmovable data somewhere random in the middle of this caused a lot of pressure on the allocator when trying to fit small objects around it.</p><p>I also ran into some difficulties copying bitmap data into the array efficiently, using the ‘|’ operator in a loop was a bit slow.</p><p>To fix both these issues, I added a c module to micropython that provides a display buffer interface to a statically allocated 1 KB buffer, along with some <a href="https://en.wikipedia.org/wiki/Bit_blit">blit</a> functions written in c.</p><h4>Images</h4><p>The display has an odd memory layout. There are 64 pixels vertically, but these are divided into rows of 8 pixels. Each row maps the bits in 1 byte to the vertical pixels, as shown here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*o921vO1RIG-zEIoW_4poSQ.png" /><figcaption>The ssd1306 internal memory layout</figcaption></figure><p>This non-standard format meant I couldn’t really use the micropython inbuilt image module, but instead wrote a script to convert the images to c-arrays packed in the above format, and included them as static variables.</p><h4>Rant — Layers</h4><blockquote>The abstraction/specialisation tradeoff arguments are complex, and can get heated. I personally am very against more than 1 or 2 abstraction points in any stack, and the rant below is based on this opinion. I’m throwing sticks at a working implementation, without proving that an alternative approach is better (or sending a pull-request) so this isn’t really fair. I do think a friendly debate around this could be useful however. Please be aware that the following sentiments may not be universally shared.</blockquote><p>What Damien managed to achieve by porting micropython to the micro:bit in such a short period was amazing, I don’t know the exact times, but I understand it was an impressively quick turnaround.</p><p>One way this was done was to leverage existing frameworks to achieve this, many of the promoted/managed by the mbed ecosystem. The cost of using the <a href="https://www.mbed.com/en/">mbed</a> tools is layers, the mbed tools all build on tools that build on underlying tools. and several of these layers add minimal value, but all come with a cost.</p><p>For example, to build the bitflyer/micropython binary is basically a case of calling the gcc arm cross-compiler on the source, with a pre-defined arguments, and linking it together with an nRF51822 ld script to give the right memory layout.</p><p>The mbed approach is to use a tool called <a href="https://yotta.mbed.com/">yotta</a>. Yotta is a package manager for ‘IOT’ platforms, and to use it, you need an ARM account, and to give ARM access to your github account.(I get the feeling ARM are trying to build a lock-in platform here).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*oHAd0WOInBSx3kUZfxQWQQ.png" /></figure><p>Micropython has some extra build stages, so has a Makefile wrapper around yotta, to so some things, then call ‘yt build’</p><p>YT build ends up creating a CMake config (&gt; 1.7 k lines) for the project, and running it to generate a <a href="https://ninja-build.org/">ninja</a> build configuration (&gt; 2.8 k lines).</p><p>Ninja then runs, and invokes gcc/g++ for each file in turn, then links them together.</p><p>And it all works fine, a binary is produced. But if you’re trying to work out where all your RAM is going, you need to see the output of <em>ld</em>, and to do that, I had to discover, and trace through each layer, of the stack to find where the relevant information was being output. Luckily I didn’t need to actually change the <em>ld</em> command line arguments, as that would have been much harder.</p><p>Now, I understand, ARM are trying to build a magic platform that houses 1,000s of off- the shelf components, that can all cross-compile with lots of platforms, and provide one-click, ARM delivered, value to all IOT developers, but in this case, the magic wore off quite quickly.</p><h4>HAL/DAL/platform</h4><p>The other case where this issue came up was in the code. Micropython is just an execution environment, so to integrate it with different hardware, it has a layer called mphal (micropython hardware abstraction layer). This turns the common module functionality into device-specific calls.</p><p>the micro:bit port of micropython uses a project called <a href="https://github.com/lancaster-university/microbit-dal">microbit-dal</a> (device abstraction layer) which, for a long time, required personal approval from someone at University of Lancaster to access the code (this is still mentioned in the readme)</p><p>The microbit-dal is built on the mbed platform, so calls down to the mbed-classic code to do much of the heavy lifting.</p><p>mbed-classic is a hardware abstraction library over a set of IOT-related functionality, and ends up calling down to a yotta_target for the nrf for actual platform specific needs.</p><p>One of the side-effects of all these layers can be seen in the memory map above. There are two separate heaps in play on the microbit, the micropython one, and the mbed-classic one. For 16 KB memory, that’s a lot of heaps!</p><p>The lines of code involved in this stack quickly start to add up, and tracing code paths through all the layers gets old quickly:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1328f857c8135800d5b2c052f821c68/href">https://medium.com/media/b1328f857c8135800d5b2c052f821c68/href</a></iframe><h4>Summary</h4><p>Writing bitflyer was a really fun and rewarding challenge!</p><p>The experience is not too far from being doable by more senior/<a href="https://hackernoon.com/tagged/technical">technical</a> students. If some more memory can be freed up, and the helper modules tidied up a bit, having a school activity to solder up the display connector, and write a simple game should be easily doable.</p><p>Most of the hard work in writing the c modules etc, should be re-usable with some tidy-up, allowing cool projects to be built on top without too much technical knowledge.</p><p>My next step for the game would be to add sound, using a simple piezoelectric speaker, and the micropython music module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O5t2oWnjYJvNAjmGG_AByw.jpeg" /></figure><figure><a href="http://bit.ly/HackernoonFB"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hqOaABQ7XGPT-OYNgiUBg.png" /></a></figure><figure><a href="https://goo.gl/k7XYbx"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vgw1jkA6hgnvwzTsfMlnpg.png" /></a></figure><figure><a href="https://goo.gl/4ofytp"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKBpq1ruUi0FVK2UM_I4tQ.png" /></a></figure><blockquote><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="/service/mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</blockquote><blockquote>If you enjoyed this story, we recommend reading our <a href="http://bit.ly/hackernoonlatestt">latest tech stories</a> and <a href="https://hackernoon.com/trending">trending tech stories</a>. Until next time, don’t take the realities of the world for granted!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35tCjoPcvq6LbB3I6Wegqw.jpeg" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4175fab44da8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/the-first-video-game-on-the-bbc-micro-bit-probably-4175fab44da8">The First Video Game on the BBC Micro:bit [probably]</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Playing Star Wars on the ESP8266 with micropython]]></title>
            <link>https://medium.com/@stestagg/playing-star-wars-on-the-esp8266-with-micropython-5f175fe7b755?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f175fe7b755</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[star-wars]]></category>
            <category><![CDATA[esp8266]]></category>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Fri, 25 Mar 2016 21:02:28 GMT</pubDate>
            <atom:updated>2016-03-25T21:02:28.145Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>I sometimes attend the London python dojo, a monthly event for anyone<em>,of any level, </em>who’s interested in exploring python for a few hours of a Thursday evening. The last one was hosted by M&amp;S Digital, the great team behind Marks &amp; Spencer technology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OmUwaj2oBIUq7qKXHYw0A.png" /><figcaption>Damien speaking about micropython at the dojo</figcaption></figure><p>Damien George came along as a guest speaker. Damien’s the person behind <a href="http://micropython.org/">micropython.org</a>, a brilliant project that has created a completely separate, implementation of Python 3 designed, and optimized, to run on low-end SOCs (think Ardiuno and similar). His <a href="https://www.kickstarter.com/projects/214379695/micropython-on-the-esp8266-beautifully-easy-iot">latest kickstarter</a>, funded earlier this month, raised money to create a port of micropython for the ESP8266 board. The success of the micropython project has been astounding, and Damien’s ongoing successes point to some rare talent and perseverance.</p><p>The ESP8266 is another marvel. A $5 chip that was marketed as just a plug-in Wifi module for other systems. People rapidly discovered that the chip was a fully-fledged programmable system with some really interesting features. Comprehensive IO, decent memory, a hardware RNG, the list goes on. Unfortunately, English documentation is poor, or lacking detail, so there’s been a coordinated effort among a group of developers to discover and document the chip’s features, resulting in a mass of information online about the thing.</p><p>I’m truly excited by all these developments, with the <a href="http://microbit.co.uk/">BBC micro:bit </a>being given (yes, there’s a <a href="https://microbit-micropython.readthedocs.org/en/latest/">micropython port</a>!) to 1,000,000 UK children, and all the work by the Raspberry PI foundation, the fact that it’s possible to throw together a self-contained internet client that can fit inside a matchbox, and show a streaming ascii video so trivially shows that we’re on the cusp of some fundamental changes in how technology works.</p><p>The fact that this tech is being given, uncontrolled, to a generation of children guarantees that exciting and unexpected things are going to appear, and soon. This is Minecraft, but 1,000 s of times more useful.</p><h3>The Dojo</h3><p>So there’s around 30 of us at the event, with 8 ESP8266 boards, some assorted hardware accessories, and a few hours to play around. Between us, we came up with: music playing (using digital outputs, no less!), an LED binary counter, an HTTP-controlled fan, and many other things during our 90 minutes:</p><h3>London Python Dojo on Twitter</h3><p>Plug things in, switch stuff on, watch internet blinkenlights using @micropython at tonight&#39;s dojo @mandsdigital pic.twitter.com/3BK62gNtVf</p><p>One idea that came up at some point, was to connect to a public streaming telnet service: towel.blinkenlights.nl, and display the output on an LCD display someone had brought.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pSPkuNLOeLiskX24R3CDiQ.png" /></figure><p>towel.blinkenlights.nl serves a <a href="http://blinkenlights.nl/services.html#starwars">complete retelling of the first Star Wars movie</a> (not the Jar jar Binks one!) as a streaming ASCII movie. You just connect a telnet client, sit back, and let the story unfold.</p><p>The idea stuck in my mind, and later that night, I ordered some cheap (The one I used cost £2.38 including delivery!) displays online, with the intent of giving the problem a go</p><h4>A few weeks later</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gYZow7LwCUF7_wWSNQuI6A.png" /><figcaption>The actual display area is about the size of a 2 pence piece</figcaption></figure><p>Coming home last night, I found a new 0.96&quot; OLED display through my letterbox, fresh off the ship from China. These displays are <em>tiny </em>but manage to pack 8,000 pixels (128x64) into the area of a 2 pence piece. They’re also really simple to drive, and with no back-light or polarizing filters, look sharp and clear under most lighting conditions.</p><h3>The Result</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F160381552&amp;url=https%3A%2F%2Fvimeo.com%2F160381552&amp;image=http%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F562358532_1280.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/5dccf267806be13d8e7bb0e16c1b1ccf/href">https://medium.com/media/5dccf267806be13d8e7bb0e16c1b1ccf/href</a></iframe><h3>Technical Log</h3><p>The display uses an <a href="https://en.wikipedia.org/wiki/I%C2%B2C">I²C interface</a> which is a simple way of devices to communicate, and simple/slow enough to implement in software without much effort. As I was firing up my tools to set off, I discovered that some ESP8266 I²C code had landed in github literally hours earlier, making this project a <strong>lot </strong>simpler (Thanks!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtIAZLdNEUGIhp3_gbBWcQ.png" /><figcaption>The timing of these changes was uncanny</figcaption></figure><p>One git pull/make later, I had the updated firmware flashed</p><h4>The Display</h4><p>Luckily the pins on the board were labelled (seriously, some boards aren’t, and don’t have online official documentation either), so it was easy to match the pins to the ESP. Because there’s no (known) hardware I²C driver, any GPIO pins can be used. This was great, as there are two GPIO pins right next to some power pins on the NodeMCU board I’m using, allowing a dead simple connection. The display seems fine with 3.3 v or 5 v on its VCC pin:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXiCyMwq_3jx4F_-14Hb0A.png" /></figure><p>As far as knowing <em>what</em> to send to the board went, there was a bit of a challenge. The only text on the board, or packaging was pin labels, and some component labels. No part number, display model, or anything. Luckily, these 0.96&quot; displays are really common, and some googling uncovered the panel was an <a href="https://www.adafruit.com/datasheets/SSD1306.pdf">ssd1306</a>.</p><p>The docs may seem a bit scary, but I love these types of technical doc. They’re worded, not to make it easiest to get started, but to help you understand exactly what’s happening, and sometimes why.</p><p>Here’s the section on the <a href="https://en.wikipedia.org/wiki/I%C2%B2C">I²C</a> interface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*Re-qRGMxhLKOkHS6m-IRSg.png" /></figure><p>I’ve highlighted the bits that the <a href="https://en.wikipedia.org/wiki/I%C2%B2C">I²C</a> library handles for us in yellow, while the rest is where the data we want to send goes.</p><p>Armed with this, I started trying to get the thing to turn on</p><h4>Showing Output</h4><p>Finally, after quite a bit of trial and error, I ended up with this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9d2e40073ad7ec83fad143abdce2a4b/href">https://medium.com/media/e9d2e40073ad7ec83fad143abdce2a4b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*trpaqsBpVfN4PpA5n2cDow.png" /></figure><p>The random display shows that it’s reading from a DRAM that has random values.</p><p>Next step was to write some output to the display. This was actually very simple. The display takes 1 byte at a time and uses that to toggle the display of a vertical strip of 8 pixels in one go. It then internally moves the pointer along one pixel, ready for the next block to be written. By doing this 128 times, an entire section of the screen can be updated:</p><pre>&gt;&gt;&gt; for i in range(128):<br>...   iic.writeto(60, b’\x40\xff’)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*imLkqrU56s1LzHErxeeVbg.png" /></figure><p>Or, slightly more interesting:</p><pre>&gt;&gt;&gt; for i in range(128):<br>...   iic.writeto(60, b’\x40’ + chr(i))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*DvrBBqJOd6dTQvW9krK-7w.png" /></figure><p>Coupling that with a command that lets you select which row of the screen to update:</p><pre>&gt;&gt;&gt; for i in range(8):<br>...   iic.writeto(60, b’\x80&#39; + (b’’ + (chr(0xb0 + i)))[-1:])<br>...   iic.writeto(60, b’\x40&#39; + b’\xaa\x55&#39; * 64)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*LLtMreLi3U8XjOVRk3MJCA.png" /></figure><p>That’s basically all there is to it. There are commands for scrolling the display, and some other features, but these are minor by comparison.</p><h4>Networking</h4><p>The telnet protocol is really, really, simple. For our purposes (and 90% of purposes) the protocol just involves reading and writing to/from a network socket. There is some capability negotiation stuff in there too, but that can usually be ignored.</p><p>The other factor here, is escape sequences. The towel service uses escape sequences to clear the screen (luckily it doesn’t try to control the cursor any more than that!), These are very straight forward. To clear the screen, it’s: 0x1b 0x5b 0x4a, or ‘&lt;esc&gt;[J’. There’s a great reference for all commonly used <a href="http://rtfm.etla.org/xterm/ctlseq.html">escape sequences online here</a>.</p><p>All that’s needed, therefore, is to:</p><ol><li>Connect to the wifi</li><li>Find the IP address of the towel server</li><li>Connect</li><li>Repeatedly read from the connection as data arrives, looking for newlines and control codes in the data</li></ol><p>The code for this can be seen below, and is pretty straight-forward. The Wifi connect code will improve as Damian’s kickstarter progresses.</p><h3>The Code</h3><p>micropython has some fairly low (but understandable!) limits on the size of modules that can be embedded in the firmware, so I had to work a bit to keep the code size small. Comments were added afterwards:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2e536d15f7b9f044058087564801cac/href">https://medium.com/media/a2e536d15f7b9f044058087564801cac/href</a></iframe><p>That’s it. Please get in touch if you have any questions, or suggestions.</p><p>Steve</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f175fe7b755" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[There is no such thing as stateless]]></title>
            <link>https://medium.com/@stestagg/there-is-no-such-thing-as-stateless-321938e4e16b?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/321938e4e16b</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[processing]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Wed, 23 Sep 2015 11:28:35 GMT</pubDate>
            <atom:updated>2015-09-23T11:28:35.470Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p><em>Every system has state*, even if it’s just the state of what I’m doing right now.</em></p><p>This fact is unavoidable in computer science. Of course, the concept of stateless programming is useful, and powerful to talk about. But requires some deep understanding of why and how to implement properly, to be useful. One cited reference for stateless programming is this blog on <a href="http://www.defmacro.org/ramblings/fp.html">Functional Programming For The Rest of Us</a> which includes the following:</p><blockquote>It turns out that functional programs <strong>can </strong>keep state, except they don’t use variables to do it. They use functions instead. The state is kept in function parameters, on the stack</blockquote><p>And this is my point. <em>Stateless design </em>is about rethinking state and storing it in a more appropriate place, not actually getting rid of it. From an information analysis point of view, stateless and stateful systems have the <strong>same</strong> <strong>quantity </strong>of information, except stateless ones tend to have less duplication, and better coupling.</p><p>A number of times, I’ve been told that some complex, slow, data processing system was “<em>designed to be stateless”</em>. This statement is often accompanied by “<em>and we can’t change it, because it took us months to iron out all the failures</em>”.</p><p>What this translates to is: the engineers didn’t want the hassle of setting up a database, so ignored the <a href="https://en.wikipedia.org/wiki/Halting_problem">halting problem</a>, and put all state in runtime-memory, hoping that nothing would ever go wrong. These systems then go live, and all the usual problems start to happen, but they remain in denial about the fact that their system <strong>has</strong> to know about what’s going on, and it <strong>can’t </strong>be just stored in memory, so a custom storage channel is invented (usually using file metadata coupled with complex heuristics) to store the information, while still pretending that there’s no state.</p><p>In this situation you end up relying on the implementer to design a custom database (yes, inferred information from file metadata is still a database, however hard it pretends not to be) to store their ‘stateless’ information. And this is actually a <em>really hard thing to do correctly</em>. Unless you design an ACID way to update the data, then you’ve just gained a whole new set of problems to solve. Which is why such systems tend to have a large set of failure scenarios that have to be coded around.</p><p>The point to the above rant is this. If you think your system is stateless, or think you can design your next solution to be stateless, consider the following questions:</p><ol><li>Any component of the solution can (and will!) fail at any time, and for a reason you didn’t anticipate.</li><li>If the component fails, whatever it was doing will be left in limbo, (your exception handlers <strong>will </strong>fail too at some point)</li><li>If one part of a process depends on any other part of the process in any way, then this <strong>is state</strong>, and you probably can’t avoid it.</li><li>What should happen if things fail? can you afford to restart everything? is your process idempotent? What if it completed but failed during post-job tasks?</li></ol><p>These points lead to some common recommendations:</p><ol><li>Make sure that that indicator/flag/storage can be updated atomically (or as nearly atomically as possible) If you can’t make it happen atomically, find a different way (It’ll make your life easier)</li><li>If you’re storing an indicator, store it in the <strong>right place</strong>, don’t attach data that’s related to one part of the system in a container that’s used for a different part. This is bad design, and leads to horrible coupling.</li><li>Similarly, trying to infer state from other indicators is usually a bad idea. If something is waiting for a precondition to be met before doing something, unless that precondition is really simple, and can be relied upon to be toggled atomically, then add a separate atomic flag that gets toggled at the right time.</li></ol><p>Comments and discussions welcomed.</p><p>— Steve</p><p>* If you know of some system that /is/ truly stateless, I’d be interested in knowing about it, but it <em>probably </em>doesn’t invalidate the point I’m trying to make here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=321938e4e16b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Feminist Programming Has Already Happened]]></title>
            <link>https://medium.com/@stestagg/how-feminist-programming-has-already-happened-9e4fb507ddb9?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e4fb507ddb9</guid>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Wed, 18 Dec 2013 13:03:40 GMT</pubDate>
            <atom:updated>2014-11-14T17:28:43.100Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring how feminist ideals are represented in existing languages</h4><p>I’d like to start with a confession<em>. I’m a white male software engineer who has had what many would call a privileged upbringing. This is also the first time I’ve attempted to explore an emotionally sensitive subject in public, so please try to accept any specific words or phrases that offend you to be accidental, rather than an attack, the overall message is important here. Comments are welcome, abuse is not.</em></p><p>Recently, <a href="http://www.hastac.org/blogs/ari-schlesinger/2013/11/26/feminism-and-programming-languages#comment-form">a blog </a><a href="https://medium.com/">on</a><a href="http://www.hastac.org/blogs/ari-schlesinger/2013/11/26/feminism-and-programming-languages#comment-form"> the HASTAC site</a> by Arielle Schlesinger became quite well publicised in social media. I originally dismissed it, because of the academic-ese register of the post, I’m horribly prejudiced against articles that are written in this form. The basic proposal seemed to be around trying to find ways to develop a new programming language that’s more in tune with various feminist ideals.</p><p>A few days passed, then some people posted what I’m going to call <a href="https://bitbucket.org/FeministSoftwareFoundation/c-plus-equality/">a poorly-judged adolescent joke</a> about the topic, in the form of a C library to introduce various fallaciously feminist terms into a program. This got me thinking about the original proposal again in more depth, and I started to wonder if some of these ideals are already creeping into programming languages, and where these developments could be taken further.</p><p>Unfortunately, the examples I could come up with are few, but their impact on programming and the way people think about concepts has been huge.</p><h3>Ducks</h3><p>Arielle proposes:</p><blockquote>“<em>a feminist programming language […] built around a non-normative paradigm that represents alternative ways of abstracting</em>”</blockquote><p>In the feminist language I’ve been exposed to, <em>normative</em> is used to talk about prejudicing. ‘<em>Boys like action toys while girls like dolls</em>’ would be a (crude example of a) normative concept in modern society.</p><p>Traditional programming languages, in my opinion, lead the developer to think in terms of normative statements. In fact, this sort of thinking fundamentally underpins traditional object-oriented design. Here’s an example (in Java):</p><pre>import toys;<br>class Person {}</pre><pre>class Boy<br>{<br> Toy buyToy() {<br>  return new ActionFigure();<br> }<br>}</pre><pre>class Girl<br>{<br> Toy buyToy(){<br>  return new PinkDoll();<br> }<br>}</pre><p>The traditional engineer’s mentality loves this, there’s no room for doubt, you can statically analyse the code and understand exactly what’s going on simply, and there’s no room for error. Boy &gt; Action Figure, Girl &gt; Pink Doll</p><p>Unfortunately this way of thinking is flawed, because inevitably a situation will occur where these rigid rules don’t apply, and suddenly all sorts of hacks have to be added to handle all those annoying pesky non-normative requirements that people have.</p><p>More modern languages, especially Python, promote <a href="http://en.wikipedia.org/wiki/Duck_typing">duck typing</a> rather than type checking for this sort of thing. Duck typing is explained by the simple phrase:</p><p><em>If it walks like a duck, and quacks like a duck, then it is a duck.</em></p><p>Immediately feminist should see some parallels here, duck typing is all about defining things by their behaviours and self-ideals rather than abstract categories and rules.</p><p>The above code example written with duck-typing:</p><pre>class Person(object):<br>  def buyToy(self):<br>    if self.likes_dolls:<br>      return Doll()<br>    else:<br>      return ActionFigure()</pre><pre>class Boy(object):<br>  likes_dolls = False</pre><pre>class Girl(object):<br>  likes_dolls = True</pre><p>Nothing’s changed here, we’re still, from our position of privilege and dominance, claiming that all girls like dolls, but the code itself is indicating to a programmer that the underlying choice is more complex than just a gender one, and with Python being so dynamic, this can easily be changed on a per-instance basis:</p><pre>jo = Girl()<br>jo.likes_dolls = False<br>print jo.buyToy()<br>&gt;&gt;  &lt;ActionFigure 0xfeed00&gt;</pre><p>This has a strong, real-world impact on code quality, and drives the programmer to not be focussed on strong pre-defined categorical prejudices.</p><p>A lot of programmers, used to traditional inheritance, coming to Python for the first time hate this loss of control, they are used to explicitly controlling how things behave and interact, and learning to have a looser-grip on your logic leads to much more beautiful code, something that matches the feminist desire for breaking down normative thinking.</p><h3>Mixins</h3><p>Interestingly, to talk about mixins, you have to talk about classical inheritance, and object-oriented principals.</p><p>In my mind, the term <em>classical inheritance</em> makes me think of upper-class privilege, and hereditary peers, while <em>object-oritented</em> is not too dissimilar from <em>objectivism</em>. And while it’s quite cheap to draw these language-based similarities, it’s interesting that very similar arguments can apply both to the software and social versions of the words.</p><p>In the classical inheritance model, objects are defined in a (general) parent-child relationship. The rule is that if you inherit from something, then there must be an ‘is-a’ relationship. For example: A <strong>square</strong> is a <strong>shape</strong>, so the <strong>square</strong> class could inherit from the <strong>shape</strong> class.</p><p>If inheritance is your primary way of expressing abstractions, then this leads to horribly arbitrary categorisation. If you were to model people using this technique you might end up with something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*TGeyWr3T2Iyg8ZSybk21Aw.png" /></figure><p>Imagine you’re writing a computer game, and you want a character that is a Fairy Princess, it would be entirely logical (in this model) to add that in like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*kNLT68D-g-3jl65RFDeKlA.png" /></figure><p>Immediately it’s clear that this way of defining classes can be problematic. This has now forced all fairy princesses to be girls. Furthermore, all the code will be written expecting this relationship, so this normative statement becomes embedded in the soul of your game, and is later impossible to unpick properly. If at a later date, someone asks for a monster-fighting girl-elf who casts spells, adding that in is going to be hard.</p><p>Instead of forcing developers to define everything by what it <strong>is</strong>, newer language communities: Python, Ruby, (even Javascript?) often talk about mixins. This is, if you like, an extension to inheritance, whereby you define an object by the behaviours it implements, rather than a strict <strong>is-a</strong> constraint. Functionality is mixed-in to an object without placing other restrictions on it.</p><p>The above model might be implemented like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*5PXyS7QrXMF_oauzAuePfw.png" /></figure><p>Here the model is much more complex, but the is also more accurate, and doesn’t impose (too many) arbitrary judgements about what <strong>is-a</strong> really means. Subsequently adding in a girl elf hunter is easy, because attributes can be selected in to a new class very simply:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*v7j5IKYGx02eiWY4CFdEkA.png" /></figure><p>This diagram is getting quite complex, but the fact that this sort of thing is even possible using mixins is critical here. I’ve left in the skip() method on Girls to show that even ‘aware’ developers can make assumptions and mistakes easily.</p><h3>Dynamic Inheritance</h3><p>Dynamic inheritance is still something of a black-art in most programming languages, and represents the front-edge of what I’m classing as the ‘feminist programming’ frontier.</p><p>Python is the only language I know with language-level support for this. Ruby allows for dynamic class generation, and Java has some inspection methods that allow for doing this, but Python really has the nicest interface here.</p><p>The basic problem is that class relationships are traditionally static. The developer defines what behaviours and attributes each type of thing should inherit. This is arbitrary and normative. Even using mixins requires a set of relationships to be hard-coded into the software.</p><p>In the game above, to allow people to chose whether their fairy was Female <strong>or</strong> Male based on preference, you’d have to remove the Genders from the inheritance/mixins and probably end up with lots of <em>if/else</em> statements or switches in the code.</p><p>With the <em>__new__</em> method in Python, inheritance and mixins can be selected dynamically at creation time. This allows for the Fairy to be created with the user-selected behaviours and actions dynamically at run-time, and removes the need for a programmer to have to select such things based on arbitrary decisions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JnKd2rWaw0i8ORAcFy5xzg.png" /></figure><p>This segues nicely into the last topic I want to cover: Logic. If you’ve got a function that is deciding if the character should be a Boy or a Girl, what happens if there is no correct answer? What happens if the<em> __new__</em> method only identified the character as Human, rather than defining it as a Boy or Girl? This is also possible in the model above, it requires some careful programming to support, but we’ve reached a situation where the language allows us to define objects in terms of predefined categories, or <strong>choose not to adhere</strong> to these limited concepts.</p><h3>Feminist Logic</h3><p>From the original article:</p><blockquote>the following are the four main groups a programming language can fall into: imperative, functional, object-oriented, and <strong>logic</strong>.</blockquote><p>Apart from imposing arbitrary categories on languages, this statement intrigued me. How could logic be made more feminist? There may be scope for some work here, but my gut feeling is that there isn’t much.</p><p>I’m going to take logic here as meaning boolean operations. All current languages (that I know of) resolve most logic statements to boolean operations.</p><p>In C, if you have a boolean operation on two values (let’s ignore pointers here), then the inputs and possible results are very simple, 1 or 0</p><p>In dynamic languages, however, things get more interesting. Typically a variable is a reference to a dynamically generated value. This allows for the possibility for a boolean variable to be true, false, or not a valid reference. These three states have some interesting implications.</p><p>The javascript terminology is probably best here, it’s untyped, so this argument is a bit nebulous, but there are actually four probable options for a Boolean type: <em>true</em>, <em>false</em>, <em>null</em>, and <em>undefined</em>. These different values are used in contexts, and can be good for adding nuance, and subtlety to otherwise black-and-white situations.</p><p>Let’s look at the implications of this with an example:</p><pre>function query_gender(person) {<br> if (person.gender == &quot;male&quot;){...} // Person identifies as male<br> if (person.gender == &quot;female&quot;)... // identifies as female<br> if (person.gender === null).. // gender explicitly not defined<br> if (person.gender === undefined)// concept doesn&#39;t apply here<br>}</pre><p>This might be slightly tenuous, but I think this added complexity in javascript is pretty closely aligned with some of the feminist thinking around. Having an explicit True/False is quite often just a bit too simplistic to allow for usefully handling all situations.</p><p>Socially, True and False are opinionated words. There are all sorts of normative assumptions tied to them, and their use can arguably be opressive.</p><p>Python provides the standard defaults to you, but doesn’t presume to enforce what True actually means. So you could quite easily define a function:</p><pre>def never_fail():<br>  False = True</pre><p>Then any future references to False will return something that equates to True. This approach is filled with flaws and caveats (early-binding etc) but the fact that Python is so unopinionated as to allow you to do this is great. A complely facaecious example might be:</p><pre>class Person():<br>  def promote(self):<br>    if self.is_male is False:<br>      self.salary *= 1.1<br>    else:<br>      self.salary *= 2</pre><p>This code might be part of a performance-management system where promotions for male staff equate to a doubling of salary, whereas female staff get a much smaller increase.</p><p>You could call code this with True and False redefined to rectify such an imbalance:</p><pre>try:<br>  temp = False<br>  False = True<br>  Jane.promote()<br>finally:<br>  False = temp</pre><p>Joking aside (I wouldn’t recommend anyone redefining True and False in a language, doing so is likely to cause chaos!), this ability shows the fact that the normal values and implications of the terms True and False may have different values in different contexts and circumstances, and Python doesn’t stand in the way of you expressing that.</p><h3>Summary</h3><p>I really feel that Python is a great case-study for how feminist concepts could be applied in software development. It does a great job of allowing non-opinionated code, which aligns it nicely with challenging normative strictures.</p><p>There are ways that this could be taken further, for example, changing an instances base class dynamically at runtime is one area where work could be done to encourage <em>new ways of thinking about problems</em>.</p><p>The concepts explained here may not be as radical as some people might hope, but looking at the places where Python differs from other languages should help to kick-start the debate nicely.</p><p>If you enjoyed reading this, please recommend it, so others can enjoy it too.</p><p>Thanks</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e4fb507ddb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Webdriver]]></title>
            <link>https://medium.com/@stestagg/beyond-webdriver-a035decfb86a?source=rss-705119ceadb3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a035decfb86a</guid>
            <dc:creator><![CDATA[Steve Stagg]]></dc:creator>
            <pubDate>Wed, 11 Dec 2013 15:24:39 GMT</pubDate>
            <atom:updated>2013-12-11T15:26:03.640Z</atom:updated>
            <content:encoded><![CDATA[<h4>Thoughts on improving website testing design</h4><p>I was going to start this by mis-quoting Churchill:</p><blockquote>Selenium is the worst tool for website testing, except for all those other tools that have been tried from time to time.</blockquote><p>Except that Selenium (webdriver) isn’t really a website testing tool at all, it’s a browser automation tool, and the Selenium IDE has definitely been improved on by other projects. By this point, the joke had lost most of its fun, and I decided not to bother.</p><p>Instead, I’ll just make my point: Webdriver is very good at what it does, but the extra tooling needed to build a really powerful testing environment on top of it doesn’t quite exist yet. There are some good bits out there already, but nothing ticks all the boxes. I think we could put together something amazing without too much work.</p><p>From my experience, for web testing to be really useful, a number of specific requirements have to be met:</p><h3>Requirements</h3><h4>1. Code Independence</h4><p>One of the things that Selenium offers is a <a href="https://code.google.com/p/selenium/wiki/PageObjects">Page Object Model</a>. This to an engineer may sound great. I’ve previously implemented something similar myself. The basic concept is to provide an automation API on top of each page of your app to allow tests to interact with it. This idea becomes more seductive when you have non-developer testers around who have to test your app. System details and complexity can behind a simple API and the testers can just call simple functions. This is flawed for a number of reasons.</p><p>Firstly, it limits the scope of what you’re testing. If an engineer has mis-understood the ‘point’ of a particular feature they implement, and that engineer also writes the automation API for it, then the test is just going to hide that fact behind another abstraction. High-level integration testing, which is what web testing usually is, should check that things work, but also that things work in a sensible fashion.</p><p>In <a href="http://en.wikipedia.org/wiki/Good_manufacturing_practice">GMP</a> environments, integration tests should (ideally) be written in isolation of the implementation, by a third party, and executed together at the end. If this test has to use a page-specific API to do so, then the isolation is undermined. The selenium POM example page gives a good example of this:</p><pre>public HomePage loginAs(String username, String password) { … }</pre><p>This seems simple, the contents of this function will type the username and password into some text fields, and click the login button (maybe). What happens if the login button is actually labelled to the user as ‘<em>Initiate user login session</em>’ however? An independent tester should immediately pick that up as a problem, but the person who just added that button may well not see what the issue is, and the test script that calls loginAs doesn’t even <strong>know </strong>about the login button so can’t check it.</p><p>There’s also the problem of re-use in testing. The traditional example of a perfectly logical, but totally useless test:</p><pre>FIVE = 5<br>FOUR = 6</pre><pre>def enterprise_max(a, b):<br>    &quot;&quot;&quot;<br>      Takes two T_INTEGER and returns the one with the higher<br>      value<br>    &quot;&quot;&quot;<br>    return b if a is FOUR else a</pre><pre>def test_enterprise_max():<br>    assert enterprise_max(FIVE, FOUR) == FIVE</pre><p>In this example, the code is almost definitely wrong, and yet the test happily passes and test coverage is 100%. This is an extreme example, but I’ve seen more subtle variants of this many times. When your POM is written as a module attached to the implementation code, then this sort of mistake is trivial to make.</p><p>The real-world cost of adding a POM is deceptively high, especially if you have a separate test team. It’s a new, separate formal API for automating your system, that means that every feature has to be automated, and exposed over it. If your app has a simple three-layer code model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*b9wUWd-2m_GbdpatP7fJCQ.png" /><figcaption>The POM is a formal API that must be implemented and maintained by (expensive) engineers</figcaption></figure><p>Then by adding a Page model, you’ve just increased the number of layers in your code by 1/3. And every change in functionality has to be kept up-to-date in every layer (in practice layering code doesn’t <em>often</em> isolate functional changes to individual layers, it helps structure the logic).</p><p>The POM also has customers (the test engineers/other devs), if you have a separate testing team, these customers likely have no direct control over the API, so any new functionality requests have to be dealt with by the implementation team, and if a code change breaks a tests script, then the debugging is likely to involve someone who wrote the API to help work out, and fix, the problem. Non-developer testers loose a lot of independence and control if their scripts become subordinate to a developer-defined API.</p><p>All of the above issues are resolvable, and can be worked around, usually by throwing resource at the problem, maybe adding some dedicated automation API engineers, or similar, but it’s costly to do, and takes a lot of time. I believe there are cheaper and much better alternatives.</p><h4>2. Resumability</h4><p>I prefer my unit test runners to start in less than a second, and to run a single unit test in around 0.1s (on average). This is important for producing high quality code fast, and allowing <a href="http://adaptivelab.github.io/pages/fearless-development.html">fearless development</a>.</p><p>For browser based testing, this isn’t possible, or practical. Typically browser tests are exercising your full application stack, and are by nature slow. Fixture setup can also be exceedingly slow, (mitigation strategies for fixtures are a separate topic here)</p><p>Typically, in my experience, a single selenium test script will run in the order of minutes, or tens of minutes, and I don’t see that as a problem. They are testing realistic workflows, and complex processes, so having long-running tests is not trivial to avoid. A Continuous Integration box can churn through these sorts of tests without too much trouble.</p><p>The problems arise when initially writing these tests, or debugging them. If you’re trying to finish off the end of test script that takes 20 minutes to run, adding a single new operation is extremely expensive to do if you have to re-run your script from the start just to test it out each change. Likewise, investigating a failure that only happens 10 minutes into a test run can be painful in the extreme.</p><p>In order to work well with these larger scripts (fast fixture management only partly helps here) it’s important to be able to stop and interact with the test script during runtime as easily as possible.</p><p>Ideally, it should be possible, if an assertion fails 1/2 way through a test case, to update the steps and continue the script from where it failed without loosing any context.</p><h4>3. Understandable Scripts</h4><p>This is an easy one, all tests should be understandable right? But for browser testing, this is arguably much more important than for other types of tests.</p><p>It’s acceptable to assume that if you’re unit testing network protocol handlers, then someone with someone with some in-depth knowledge of networking might be needed to read the tests, but functional testing should be about the ‘if I click the red X, then window goes away’ type of statement. In the formal world, this is proving that the overall vision, and functional specs have been adequately satisfied.</p><p>There should be no reason why a business owner, or non-technical user shouldn’t understand ‘if I click the red X, then window goes away’, so it should be possible to describe the tests in a similar language. The fact that the red X has an <em>id=’ui_toolbar_close_button_faded’</em> is not relevant and should be hidden from the person reading the test script. Selenium and BDD testing are natural allies here, and I’ll talk about Cucumber below.</p><p>The ideal is that the business user (manager, product owner) should be able to write such scripts themself, but this is a <em>far</em> harder problem to solve than you might think. It may be possible, but I’ve yet to see it happen.</p><h4>4. Test Semantics</h4><p>Selenium tests should be about proving (for example) ‘a user can log-in to the site’ and not ‘the site can log-in a user’. Logically these are almost identical, but semantically they’re very different.</p><p>By the time you get to the stage of writing browser tests, you should be fairly confident that the system can handle logins, but it may well not be clear that the user can do this in a sane way.</p><p>What this means for tests is that class names, and element IDs should be avoided wherever possible. If a dev goes in and changes an element <strong>id</strong> from ‘login’ to ‘authenticate’ that shouldn’t affect the user at all, and it shouldn’t affect the tests. If that dev changed the button <strong>text</strong> from ‘Login’ to ‘Initiate user session’, then it’s entirely reasonable for both the user and the test to break horribly.</p><p>So not only do you want your tests to be readable by people, without knowing about system internals, the best tests will be runnable without knowing about the internals.</p><p>Sometimes this is unavoidable, it’s quite hard to find a test runner that can understand that toolbar means ‘the grey horizontal box near the top of the screen’. But often you’ll need to cater to users who may only have a fuzzy notion of what a menubar is anyway. Most people think “click save” not “click save that’s in the toolbar”.</p><p>This fuzzy way of thinking is often alien to those working with software, we like absolutes and definites, but when you’re emulating a user who doesn’t care about all the beautiful well-defined UI models you’ve built, but just wants to save his damn file, it’s important to address these ambiguities. If the test script says ‘click save’ but there are 3 ‘save’ buttons on the page, it’s probably better to fix having 3 save buttons on the same page than updating your test to identify the button you know is the <em>correct</em> one.</p><p>As always, there are times when this approach can’t be made to work, and you have to fall back to finding something by ID, or CSS, or X-path, but in that situation, it should be possible (and easiest) to give that system identifier a human-understandable name:</p><pre>menubar = thing.find_by_css_selector(“#menubar”)</pre><h3>Summary</h3><p>I went into some detail on these requirements to try and outline <em>why</em> I think they are important, if you have any comments or questions please leave them, or tweet me @stestagg.</p><p>Next I will look at some of the tools and kits currently being used, and how they fit in with the above requirements.</p><h3>Existing Tools</h3><h4>Native Selenium (Webdriver)</h4><p>Webdriver provides a lot, it covers the various servers for driving web browsers, and various client APIs. It’s great at automating browsers, but not so great at writing more complex tests:</p><ul><li><strong>Code independence</strong> — The client libraries do nothing to address this (rightfully in my opinion)</li><li><strong>Resumability</strong> — As a raw driver, any resumability functionality are delegated to the client language, and while there are <a href="http://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html">some</a> methods to achieve this, they don’t really count in this context.</li><li><strong>Understandable scripts</strong> — I’ve written many selenium scripts in Python, and none of them have been things I would want to share with anyone non-technical.</li><li><strong>Test Semantics</strong> — The webdriver API actually has a few methods that I would call semantically relevant: things like <em>click</em> and <em>back</em>, but also: <em>find_element_by_link_text</em> is a good one. The number of useful methods however is too small to write good tests with. It’s too easy to just revert to using the <em>find_element_by_css_selector</em> or similar</li></ul><h4>Cucumber/Lettuce/Capybara</h4><p>The various BDD testing tools out there allow you to define user-readable scripts with ease, and many people use them on top of webdriver to write beautiful test cases, but I feel that there’s still some things missing:</p><ul><li><strong>Code independence </strong>— This is hard to get right with BDD tools. Typically it’s up to the implementor to define the vocabulary used, and designing an abstract set of verbs is hard. More often, it’s used as a method for annotating tests with human-readable labels (i.e. one action description matches one test step. Lines such as:</li></ul><pre><a href="https://medium.com/p/a035decfb86a/edit">When</a>(/^<a href="https://medium.com/p/a035decfb86a/edit">I</a> <a href="https://medium.com/p/a035decfb86a/edit">enter</a> &quot;(.*?)&quot; <a href="https://medium.com/p/a035decfb86a/edit">into</a> <a href="https://medium.com/p/a035decfb86a/edit">the</a> <a href="https://medium.com/p/a035decfb86a/edit">search</a> <a href="https://medium.com/p/a035decfb86a/edit">field</a>$/) do |<a href="https://medium.com/p/a035decfb86a/edit">arg1</a>|</pre><p>are common, where the defined steps are really a glorified POM implementation. There are some attempts to rectify this, and the lettuce-webdriver project looks like an interesting attempt at that.</p><ul><li><strong>Resumability </strong>— I have yet to see a test runner that includes runtime context, and supports decent debugging and resumability. They may exist, but I haven’t used one.</li><li><strong>Understandable scripts </strong>— This is where BDD tools really shine, the test output is clear, and readable (usually). Getting non-technical people to write tests using BDD tools is theoretically possible, but actually quite complex, the language is more strict than people may be used-to, grammars tend to be sensitive to whitespace etc, and unless you have a truly abstracted BDD language, you’re really just using a POM with all the associated communication issues.</li><li><strong>Test Semantics </strong>— Good but not great, because steps are behind-the-scenes implemented in native app-specific code, they will naturally tend to interact with the app internals more than they should.</li></ul><p>On the whole, I think BDD type languages have a part to play in the final picture, but I think more is needed.</p><h4>Selenium IDE</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*0HaLWU6VpmptvqYCb8-5cA.gif" /></figure><p>Selenium is great, webdriver is greater(?), and I wish the Selenium IDE was <strong>never released</strong>. It was a bold project to build a nice non-technical click-and-drag interface for writing tests that helped to kick-start the browser testing movement, unfortunately it’s not suitable for testing platforms under development at all, and it provides a big negative experience for people who turn to the IDE first when implementing browser tests.</p><p>The basic problem is that an automated recording tool cannot apply semantic logic to a user’s actions. We’re decades away from a computer understanding what a user is trying to do without lots of prompting. What results is a test script that is tightly bound to a particular page structure, and test scripts that break on every run because of this coupling. There is no state tracking (you can’t really store dynamic variables and refer to them later), and the feature set is small.</p><p>To evaluate each requirement:</p><ul><li><strong>Code independance </strong>— Great, the IDE doesn’t know anything about particular implementations.</li><li><strong>Resumability </strong>— Not bad, the script can be edited dynamically, and individual steps re-run, but the UI is flawed and it doesn’t seem to let you easily resume a partially complete test run without running each step individually</li><li><strong>Understandable scripts </strong>— The test scripts can be output in a variety of formats, but the IDE never knows the things that are needed to be able to describe what is happening in a useful way. <em>clickAndWait id=btnG </em>will never mean anything to someone not involved in writing the system.</li><li><strong>Test Semantics </strong>— The methods available in the IDE are the same as provided by the underlying Selenium API, so semantically they are poor, elements are referenced by IDs and paths.</li></ul><p>In many ways, the Selenium IDE is a ‘get what you pay for’ technology (not in money terms). It looks great, you can install it and immediately record a test script, no learning needed! It’s only when it comes to maintaining these scripts, documenting them, and debugging issues that the real costs of using the IDE become apparent.</p><h4>Watir</h4><p>I haven’t used Watir before, and it looks like a great tool for doing this sort of thing. But from my brief overview of the examples and documentation, it seems to help with, but not concretely address the requirements listed above.</p><h3>Proposal</h3><p>That somewhat lengthy introduction paints a picture of my understanding of the current selenium landscape. I think that with some careful design and a few lines of code, something amazing could emerge. Here are my thoughts so far, however unorganised they may seem.</p><p>The idea is to wrap the basic Webdriver API in an adapter that provides user-oriented actions, add a simple (execution) language-independent way of specifying a test script, and then develop some different test runners for different scenarios:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M8YqwX-QEII48GmoK7Nozw.png" /><figcaption>Basic system organisation</figcaption></figure><h4>1. An action list</h4><p>Each test script should be composed of nested list of operations, in some abstract language. I’m proposing a JSON array.</p><p>Each item in the array describes a step in the script, and each step is of a specific type, and contains contextual information. This may sound very similar to a BDD list, but the crucial points here are, the list is nested, allowing for grouping and contextual operations, and the available actions are based off a built-in vocabulary that is oriented around user-centric concepts. For example, to click on a standard disclaimer label on the page, and store the full text for later reference:</p><pre>[{<br>   action: &quot;find&quot;,<br>   startswith: &quot;I agree to the terms&quot;,<br>   and: [<br>      {<br>         action: &quot;click&quot;<br>      },<br>      {<br>         action: &quot;remember_text&quot;,<br>         as: &quot;disclaimer text&quot;<br>      }<br>   ]<br>}]</pre><p>The advantages of this approach are many:</p><ol><li>This file <em>could</em> be hand-written without any additional tools (but the idea is to develop scripts with tools)</li><li>It can be easily serialised and parsed by any language or runner</li><li>It defines a common, well-defined API that can be publicly shared</li><li>By designing the actions carefully, very complex scripts can be built-up with reusable components without compromising on the overall readability and structure.</li></ol><h4>A web-based designed</h4><p>This is a crucial element of the system, by building a web-based tool for developing scripts, the system can really easily be made interactive, and cross-platform.</p><p>For example, editing, resuming, and running individual tests is simple in this environment, all communication between the editor and the system running the selenium driver are based on the JSON spec above, and can be very simple.</p><p>A 5-minute mockup of the interface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*yN4vEOw6_hdYYT5xjGd0sw.png" /><figcaption>Example of how the Web based editor may look, with live-updating browser screenshots, and ability to run steps on-demand</figcaption></figure><p>The UI can be managed purely in Javascript, with drag-drop, wizards, and helpers to help non-technical people build up simple scripts. With live-updating browser displays and easy run buttons, it’s possible to easily build up complex scripts one step at a time.</p><p>The UI can (largely) be auto-generated from a static definition of the action types and their arguments, with nesting being automatically added to actions that support that.</p><h4>A command-line runner</h4><p>Because the language is well-defined, and self-contained, a script can be passed to a very simple command-line runner that can execute the script against different browsers, and without human interaction.</p><p>The output can be very easy to understand, given the example script above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*cCwW9njAi_WHPUBew1YOGg.png" /></figure><p>A more complex example might be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*KZqhekVJH7QM6s4e5eQE7A.png" /></figure><p>In this case, each step is readable in its own right, the failure can be traced all the way back, and extra contextual comments have been added to outline the broader direction of the script. It would be easy to filter the output to include more or less detail based on who is interested. Alternately, an X-unit compatible output is trivial to add.</p><p>If a test run fails, then it is simple to open the script up in the editor and reproduce the problem interactively. Adding in video capture and regular screenshots to the command-line runner is trivial.</p><h4>Cucumber adapter</h4><p>It would also be possible to write an adapter that converted the built-in actions into BDD style grammar, and then translated between that and the underlying JSON format.</p><h4>A User-oriented vocabulary</h4><p>A fairly small collection of ‘things that people might want to do’ should cover most cases, this list has a lot of overlap with the selenium API, but the differences are crucial.</p><p><strong>Finding elements</strong></p><p>Elements can be found using different criteria: exact text match, text starts with, contains text.</p><p>In cases where that doesn’t suffice, then also filtering by tag name or attributes is possible.</p><p>The find action can also be customised to be case sensitive, return a single element, or all elements, include hidden or disabled elements (by default excluded)</p><p><strong>Interacting with element</strong></p><p><strong>click</strong> — Click on the current element (only makes sense when nested below a find action)</p><p><strong>click on <em>&lt;text&gt;</em></strong> — Find any element (below the current one) that is clickable (link, button, span with click event, etc..) that has the matching text, and click on it.</p><p><strong>type &lt;text&gt;</strong> — Alias for ‘send_keys’, types text into current element</p><p><strong>fill in &lt;label&gt; with &lt;value&gt;</strong> — Find a &lt;label&gt; element with text matching &lt;label&gt;, click on it, then type &lt;value&gt; into the input with focus</p><p><strong>select &lt;option&gt;</strong> — Find a selection option with text matching &lt;option&gt; and select it (typically used after finding a &lt;select&gt; element)</p><p>…</p><p><strong>Identify &lt;name&gt; by css: &lt;css&gt;</strong> — stores the CSS selector under &lt;name&gt; allowing hard-to-find elements to be identified.</p><p><strong>In &lt;name&gt; — </strong>any nested commands will be scoped by an element that matches the CSS above. I.e.:</p><pre>Identify [<strong>toolbar]</strong> by css [<strong>#header #toolbar]<br></strong>In [<strong>toolbar]:<br>    </strong>Click [<strong>Save]</strong></pre><p><strong>Structural actions</strong></p><p>The above actions cover a lot of simple cases, but quickly problems get more complex, and domain-specific. In that case, it may not be possible to avoid having more technical actions in play, but these can be carefully encapsulated to help maintain isolation between non-technical test writers and devs.</p><p>Firstly, all executions run in a context that can be used to store variables and state, allowing for more complex tracking of dynamic values, this context is maintained during interactive sessions, and the user can see stored (simple) values in the editor and change them. Variables are simple key-&gt;values where the key can be any string.</p><p><strong>Include &lt;script&gt;</strong> — Most people develop a set of test scripts, and accompanying files into a suite. Where this happens, test scripts can include other scripts inline, allowing for common settings and utility libraries to be imported</p><p><strong>Define &lt;procedure&gt; with summary: [Text with &lt;inline&gt; &lt;placeholders&gt; &lt;indicated&gt; for caller to fill in] and actions: [nested list of actions] — </strong>Defines a stored procedure (list of actions) which will be run with added variables passed in by other callers. In the UI, this may look like (text in [] would be entered by user into textbox, text in <em>italics</em> indicates variable placeholders):</p><pre>Define [fast login] with summary: [Quickly login as &lt;user&gt; with &lt;password&gt;] and actions:<br>    Browse to [<em>root url</em>]<br>    Fill in [username] with [<em>user</em>]<br>    Fill in [password] with [<em>password</em>]<br>    Click on [login]</pre><p>Then a script could call the procedure at any point, with the Web IDE pr</p><p>oviding placeholders and hints dynamically based on the summary:</p><pre>Do [fast login]: Quickly login as [bill] with [Passw0rd]</pre><p>The test runner would then dynamically expand that to be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*8q3wmoRjH-ogA2BOgTA5WA.png" /></figure><p>Finally, if all else fails, then a get-out-of-jail clause would allow (trusted) scripts to run inline code from a script directly:</p><p><strong>Evaluate: &lt;statement&gt; — </strong>Evaluate code in the test-runner directly, with access to the variable context, underlying selenium driver, and other infrastructure.</p><pre>Evaluate [<em>assert int(context[“current year”]) == datetime.Datetime.now().year</em>]</pre><p>These more advanced usages would be present but not encouraged for use except in more advanced cases. By allowing them to be encapsulated in human-readable BDD style statements, the flexibility is retained without forcing readability to be sacrificed.</p><h3>Comparing to the Requirements</h3><ul><li><strong>Code independance </strong>— The basic verbs and language used here has no knowledge of individual implementations. Where such links are to be added, they are isolated within the test scripts, this makes coupling test code to implementations hard, which I consider a very good feature</li><li><strong>Resumability </strong>— Runtime context can be directly edited, steps can be run dynamically and on-demand from the editor, so developing and debugging scripts is trivial. The command-line runner may not have any of this functionality (except PDB?)</li><li><strong>Understandable scripts </strong>— The system can convert any script into human readable output, with support for added plain-text context in the process. It should also be possible (defining the grammar may be hard) to write a full BDD style script definition language that compiles to the internal script format.</li><li><strong>Test Semantics </strong>— This is down to the verbs used and defined, but by providing a built-in set of verbs that <em>set the scene</em> for a user-action oriented test script, it should be much simpler and more natural to write scripts that make sense to a user. This is about directionality more than anything.</li></ul><h3>Progress</h3><p>None of this system I’ve described is particularly ground-breaking, most of the ideas above are spread around other projects out there, but bringing this all together produces a really powerful, simple to use system.</p><p>I’ve put together a prototype which unfortunately I have lost the code for, but the concept is solid, and the prototype proved this system’s usefulness.</p><p>By writing up this spec, I’m hoping to promote some debate, and hone ideas before getting stuck into a full implementation that may be useful to a wider audience. If anyone wants to write this with/for me please get in touch, or just start!</p><p>If you’ve stuck it this far, then thank you, you’re very patient, and here’s a small present:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/22/1*W6SDBk0Fmiqtopg5bh0BmA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a035decfb86a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>