<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Datasparq Technology - Medium]]></title>
        <description><![CDATA[Welcome to Datasparq’s technology blog. At Datasparq, we design, deliver and run bespoke data science products to help organisations capture value from data. - Medium]]></description>
        <link>https://medium.com/datasparq-technology?source=rss----64ba35c1239a---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Datasparq Technology - Medium</title>
            <link>https://medium.com/datasparq-technology?source=rss----64ba35c1239a---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:52:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/datasparq-technology" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ Houston: A better approach to workflow orchestration]]></title>
            <link>https://medium.com/datasparq-technology/houston-a-better-approach-to-workflow-orchestration-31b5bfc321d4?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/31b5bfc321d4</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Matt Simmons]]></dc:creator>
            <pubDate>Wed, 14 Feb 2024 11:12:04 GMT</pubDate>
            <atom:updated>2024-02-14T11:12:04.199Z</atom:updated>
            <content:encoded><![CDATA[<p>Complex workflows (data pipelines, ML prediction pipelines, optimisation algorithms, etc.) always need orchestration.</p><p>Software like Airflow orchestrates and runs tasks within a single ‘monolithic’ service. The server becomes slow, unresponsive, and expensive as it grows.</p><p>We propose a new approach that orchestrates independent microservices remotely via a simple API. Our solution is now available as the free open-source software<em> Houston</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*tOsY7wgB-LQfqwMgzTB3Zg.gif" /><figcaption>A simple workflow, represented as a graph (Houston UI)</figcaption></figure><p>The orchestration API server is light-weight and easy to deploy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*EWMOITH1OpnmLg_OAqefqA.gif" /><figcaption>Starting a Houston API instance via the command line. Docker and Terraform deployments are just as easy</figcaption></figure><p>Defining new services (task runners) is dead simple. We’ve hidden all the complexity. For example, here’s a Google Cloud Function, written in Python, that’s fully integrated with Houston:</p><pre>from houston.gcp.cloud_function import service<br><br>@service()<br>def main(file_path):<br>   file = load(file_path)<br>   upload_file(file)</pre><p>5 key advantages:</p><ul><li>Most (if not all) code runs on <strong>serverless</strong> tools, which lowers cost, maintenance, and co-dependencies vs a managed option¹</li><li>New workflow runs start up <strong>instantly</strong> and the UI loads really <strong>fast</strong> because the orchestration server has relatively little work to do → one API instance can easily handle 500+ concurrent workflows²</li><li>Multiple workflows can <strong>use the same services</strong> at the same time, allowing code and infrastructure to be shared across teams more easily</li><li>A workflow can be made up of microservices written in various languages and runtimes, running on different cloud providers, allowing teams to make gradual upgrades as new technology becomes available with no vendor lock-in</li><li>Optionally, a publisher/subscriber message system is used to trigger each stage to run, meaning message delivery is guaranteed and services don’t get overloaded by too many requests; they pull messages at their own pace</li></ul><p>Disadvantages:</p><ul><li>All code, no matter how simple, must be deployed as a microservice. This increases the development overhead. We’ve countered this by creating utilities to speed up the creation of microservices</li><li>Workflow errors can go undetected if there is insufficient monitoring</li><li>State cannot be persisted between stages without an external database</li></ul><h4>How it works</h4><p>A collection of isolated microservices, working together to complete a task, fits quite well with the metaphor of a space mission:</p><ul><li>Microservices are the rockets, landers, and satellites. They’re completely isolated from each other</li><li>The API is the mission control centre, which orchestrates everything. It tells the services what to do, and which services should run next.</li><li>Not everything goes to plan… but we can find ways to recover from any failures and keep the mission going</li></ul><p>This is why we called the orchestration software <em>Houston</em>.</p><blockquote>“Okay, Houston…we’ve had a problem here” — Jim Lovell, Apollo-13 Mission</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*p4LzGVgLcFt9DxL9I3728Q.gif" /><figcaption>The mission timeline view from the Houston UI</figcaption></figure><h4>Why we made it</h4><p>We’re a Data Science consultancy with many different clients. We needed isolated instances for every client and environment. Airflow would be too expensive (imagine 3 environments per client per project, running heavy duty ML workloads, costing <strong>£500 a month </strong>each!).</p><p>Equally important is the developer experience. Houston is designed to be simple and fast. Entire workflows can finish in under a second³ thanks to a lightning fast server written in Go. The dashboard UI is always live, showing updates as they happen, unlike Airflow’s frustratingly slow dashboard.</p><p>Ease of maintenance is also a huge concern for us. We share the same microservices across many projects, allowing us to upgrade and test them only once and be completely confident that they’ll run healthily in every project. We also avoid the need for huge migrations with extended downtime (e.g. upgrading a large Airflow cluster from Python 3.11 to 3.12).</p><p>We also hate how different tools don’t talk to each other. Why do we have separate pipelines for Dataflow, Data Factory, Vertex AI, etc. all in one project? Houston is completely platform agnostic, so we can use it on any cloud provider and any tool.</p><p>…</p><p>It’s easy to get started, so try it out today!</p><p>👉 <a href="https://callhouston.io/">callhouston.io</a></p><p>👉 <a href="https://github.com/datasparq-ai/houston">github.com/datasparq-ai/houston</a></p><p>1. Deloitte Consulting. ‘<em>Determining the Total Cost of Ownership of Serverless Technologies when compared to Traditional Cloud’.</em> September 2019. <a href="https://pages.awscloud.com/rs/112-TZM-766/images/AWS_MAD_Deloitte_TCO_paper.pdf">https://pages.awscloud.com/rs/112-TZM-766/images/AWS_MAD_Deloitte_TCO_paper.pdf</a></p><p>2. See a live demonstration of this by running houston demo --stress-test</p><p>3. See the <a href="https://github.com/datasparq-intelligent-products/houston-quickstart-python/tree/master/local#4-run-a-workflow-mission">local server quickstart</a> for an example of this. A plan with 6 stages completes almost instantly when the stages do nothing because the API calls between the service and the server are in the order of a few milliseconds.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31b5bfc321d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/houston-a-better-approach-to-workflow-orchestration-31b5bfc321d4">🚀 Houston: A better approach to workflow orchestration</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating an SSH Key for Git, Explained for 5 Year Olds]]></title>
            <link>https://medium.com/datasparq-technology/creating-an-ssh-key-for-git-explained-for-5-year-olds-c493e1cf6b0b?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/c493e1cf6b0b</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Matt Simmons]]></dc:creator>
            <pubDate>Tue, 18 Apr 2023 14:16:44 GMT</pubDate>
            <atom:updated>2025-04-28T09:57:14.843Z</atom:updated>
            <content:encoded><![CDATA[<p>When you’re only 5 years old, it can be confusing to use git and ssh, due to the many different concepts involved. This guide explains each step in simple terms that any child will be able to understand.</p><p>SSH stands for <strong>S</strong>ecure <strong>Sh</strong>ell. It’s a way of accessing another computer securely. For a git repo, it’s a more secure alternative to using a username and password to get access.</p><p>This guide will explain how to make an SSH key pair, and use it to access a git repo.</p><h4>Step 0: Access the Command Line</h4><p>The tools you need exist on most computers by default as command line tools.</p><ul><li>On Mac, open <strong>Terminal</strong> (command + space → terminal.app).</li><li>On Windows, look for <strong>Git Bash</strong>. This should exist if you have git installed. If not, <a href="https://git-scm.com/download/win">download and install Git</a> before continuing.</li></ul><h4>Step 1: Create an SSH Key Pair</h4><pre>ssh-keygen -t rsa -b 4096 -C &quot;your_email@example.com&quot;</pre><p>ssh-keygen<strong> </strong>is the program that’s going to generate an SSH key for us.</p><p>-b means bits and 4096 is the number of bits the key will use. That’s a lot of bits for lots of security! 2048 is the default and is already enough, so this key will be unbreakable by any modern computer.</p><p>-C means ‘comment’, and the email address is the value of this comment. This tells the computer we’re accessing who we are.</p><p>You don’t need to provide a location to store the key because you will be asked.</p><p>→ <em>Run the above command</em></p><p>It will ask you where you want to save the key pair. Just press enter to accept the default location of ~/.ssh , which is a hidden folder (because it starts with .) in your user folder.</p><p>It will also ask you if you want to create a password. You can press enter to ignore this and not set a password. SSH is secure enough without one, but use one if you’re worried about people stealing your laptop or keys.</p><p><strong>What did it create?</strong> If you look in ~/.ssh you’ll see two files:</p><ul><li>id_rsa is your ‘private key’, meaning it should never be shared with anyone.</li><li>id_rsa.pub is your public key. This is what you’ll give to your git repo so that it can verify your identity when you try to access it.</li></ul><p>You may not be able to see hidden files and folders as this is the default setting on most computers, but you can use the command line to look at this folder if you want to. Use cd ~/.ssh to <strong>c</strong>hange <strong>d</strong>irectory and then ls -a to <strong>l</strong>i<strong>s</strong>t <strong>a</strong>ll the files in there.</p><h4><strong>Step 2: Get your public key</strong></h4><p>Run this command in the console to see the public key’s value. cat is a bash program that reads text files and prints their content to the terminal. You could also open the file in TextEdit/Notepad to see the key.</p><pre>cat ~/.ssh/id_rsa.pub</pre><p>Then copy the key to you clipboard. It should begin with “ssh-rsa” and end with your email address.</p><h4>Step 3: Upload the Key to Your Git Repo</h4><p>For Github repos: Go to <a href="https://github.com/settings/keys">github.com/settings/keys</a>, and click ‘New SSH key’. It will ask you to give it a name. <strong>Paste the public key</strong> from step 2 in the ‘key’ box. Click ‘Add SSH key’.</p><p>Now GitHub knows your public key, and when it sees you trying to connect it will grant you access because you and only you have the matching private key!</p><h4>Step 4: Git Clone or Set Remote</h4><p>Find the SSH url of the repo by clicking ‘Code’. Copy the value it shows you. Make sure SSH is selected, see below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dcW8b3w-ND6bocLHB9C7wA.png" /></figure><p>Then run git clone with the value you copied, e.g.:</p><pre>git clone <a href="/service/mailto:git@github.com">git@github.com</a>:example/example.git</pre><p>If you already cloned your repo before now, you may need to change the remote to use SSH instead of HTTP:</p><pre>git remote set-url <a href="/service/mailto:git@github.com">git@github.com</a>:example/example.git</pre><p>Now just test it works:</p><pre>git pull</pre><p>When you run this, the git program automatically looks for the private key you made called id_rsa to authenticate. You don’t need to tell it where the key is, or enter any passwords, unless you set a password for this key.</p><p><strong>THE END</strong>. If you get any errors, or if you have more than one GitHub account, scroll down for tips.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ny6DkW9p0ZDx0tXf" /><figcaption>Photo by <a href="https://unsplash.com/@cookiethepom?utm_source=medium&amp;utm_medium=referral">Cookie the Pom</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Appendix: Multiple Accounts</h4><p>If you have multiple GitHub accounts, you’ll need multiple keys. GitHub simply doesn’t allow you to use the same key for multiple accounts.</p><p>There are a few ways to set this up, but the simplest method is described in this excellent article by Hao Tang: <a href="https://medium.com/dayoneteams/how-to-use-different-ssh-keys-for-different-git-remotes-fa1a33eea66d">How to use different SSH keys for different Git remotes</a>.</p><p>This method still works if your two repos are hosted at the same domain name (i.e. github.com), and it you want two different remote repos for one local repo.</p><h4><strong>If you see ‘WARNING: UNPROTECTED PRIVATE KEY FILE!’</strong></h4><p>Don’t panic. This just means other users on your computer can access the private key. You may be the only user on your computer, so this may not seem relevant, but it if you were to let someone else connect to your computer remotely, they would have access to these files, so it’s safer to prevent others from reading them.</p><p>We’ll use chmod which means ‘<strong>ch</strong>ange <strong>mod</strong>e’. This is a program found on all Unix based computers that is used to change the permissions that ‘users’ have on the file.</p><p>Run:</p><pre>chmod 400 ~/.ssh/id_rsa</pre><p>The three numbers in represent the permissions of the user, group, and others, in that order, where each number means:</p><ul><li>0 = no permissions (can’t even open it)</li><li>1 = execute (run a file as a program)</li><li>2 = write (make changes to)</li><li>4 = read</li><li>7 = all permissions</li></ul><p>Therefore, the command above is going to make the private key readable only by the user (you), and not readable by anyone else. The SSH program asked you to do this to make it even more secure.</p><h4><strong>If you see ‘Failed to add the host to the list of known hosts’</strong></h4><p>This means the list of computer host names that your computer trusts (or ‘knows’) couldn’t be added to by the ssh program, due to it not having permission to do so. As above, we need to change the permissions on this list using chmod.</p><pre>sudo chmod 644 ~/.ssh/known_hosts<br>sudo chmod 755 ~/.ssh</pre><h4>If you see ‘You’re using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type’</h4><p>Please see <a href="https://github.blog/2021-09-01-improving-git-protocol-security-github/">https://github.blog/2021-09-01-improving-git-protocol-security-github/</a> for more information.</p><p>—</p><p>Thanks for reading. If you run into any other issues when doing this feel free to leave a comment and I’ll add it to the guide.</p><p>For more programming tips, follow me on Twitter: <a href="https://twitter.com/MattSimmons01">@MattSimmons01</a></p><p>Reach out to us at <a href="https://datasparq.ai/contact">datasparq.ai/contact</a> or <a href="https://twitter.com/DatasparqAI">@DatasparqAI</a> on Twitter for advice on solving a problem with AI and Data!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c493e1cf6b0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/creating-an-ssh-key-for-git-explained-for-5-year-olds-c493e1cf6b0b">Creating an SSH Key for Git, Explained for 5 Year Olds</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How data science and data transparency can help save the planet]]></title>
            <link>https://medium.com/datasparq-technology/can-data-science-and-data-transparency-help-save-the-planet-d537349bc25e?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/d537349bc25e</guid>
            <category><![CDATA[carbon-emissions]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[carbon-accounting]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[climate-change]]></category>
            <dc:creator><![CDATA[Jeremy Bradley]]></dc:creator>
            <pubDate>Wed, 15 Feb 2023 12:26:26 GMT</pubDate>
            <atom:updated>2023-02-15T14:00:54.170Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>With current carbon offsetting processes being dubiously effective, we suggest a data-driven approach that companies can use to become reliably sustainable and protect themselves from punitive investor and government action.</em></strong></p><figure><img alt="Approximately 120 stacked fans of roughly diameter 2m, situated in a desert used in the direct air capture of CO2" src="https://cdn-images-1.medium.com/max/1024/1*yU8u93yHTHBTatArduUj9w.png" /><figcaption>An array of fans blow air through a carbon dioxide-absorbing material in this mock-up of a direct air capture plant. © Carbon Engineering</figcaption></figure><p>The Paris climate agreement was important — vitally important — not because it meant that it solved the problem of climate change — it did not. Nor did it mean that the world’s governments were able to agree on a mechanism for greenhouse gas reduction, that is clearly a path much longer in the making. It was important because it represented a global, multi-nation-state backed, statement of intent to try to limit average global warming to 1.5C.</p><p>Now that has happened it is up to us: citizens, employees, colleagues, companies, organisations… to take a lead. Even if successive COPs seem woefully inadequate in ambition and outcome, it actually doesn’t matter — we now have a global call-to-arms and a goal that needs to be met. And increasingly as many cities and states across the world are finding out, it matters less and less whether national governments are able to reach a consensus or not.</p><p>As data scientists and engineers, we have our challenge — and in the (nearly) immortal words of Matt Damon as Martian astronaut, Mark Watney:</p><blockquote><strong>“You solve one problem… and you solve the next one… and then the next.</strong> <strong>And If you solve enough problems, you get to save the planet!” — Mark Watney</strong></blockquote><p>That sounds like something we can get behind.</p><h4><strong>So how can this happen and why isn’t it happening yet?</strong></h4><p>The challenge for companies that quite genuinely want to reduce their impact through their operations or services is that they have no easy way of understanding the real costs and downside risk of continuing as they are. Most companies have everyday pressures to maintain or improve their financial bottom line and those are easily understood and accountancy practices are put in place to monitor and support that goal. There are very few companies who monitor and report openly on their carbon bottom line and it is not clear how one could even convert between carbon and cash, especially in jurisdictions where there is no hard requirement to do so.</p><blockquote><strong>There is no easy or consistent way of computing the financial downside of pumping millions of tonnes of CO<em>₂</em> into the atmosphere.</strong></blockquote><p>Even for companies who are engaging in offset activities such as carbon credits, there are many pitfalls and no guarantees that carbon is actually being sequestrated and removed from the atmosphere [5].</p><p>It is very clear that where there is not now specific environmental legislation governing the release of greenhouse gases in a country, it is a safe assumption that there will be such legislation in many countries in the next 10 years. (Nevertheless without legislation, the pressure today from institutional investors to reach net zero is significant, as evidenced by one of the largest global funds, Norway’s Sovereign Wealth fund [1,6]).</p><p>What form any legislation takes will of course vary: carbon credits, carbon trading, carbon taxes, carbon limits, possibly consumer carbon rationing if things get really bad. Each of these options carries a financial impact on companies as well as being variously effective in reducing greenhouse gases.</p><p>Although legislation is rarely retrospective it is also a significant possibility that some key polluting industries, e.g. the oil and gas sector, energy generation sectors, may well be asked to be, not just carbon neutral but significantly carbon negative in the future; this will allow those industries to unwind some of their historic carbon debt if done correctly.</p><h4><strong>And the data science in this?</strong></h4><p>Readers of this series will know how much we use and value reinforcement learning (RL) and RL thinking in our approach to data science.</p><p>The reinforcement learning paradigm applies to business operation as much as it does to data science execution. A business is every bit an RL machine — it has levers it can pull, decisions it can take, projects it can try. From those decisions a well-run business will try to quantify their success and from that learnt success or failure, future decisions and projects are initiated.</p><p>If that success quantification process (in RL terms, the reward function) excludes a major element of the operation of the company — the cost of removing huge amounts of CO₂ from the atmosphere — then a company will never learn the correct behaviours and decisions to operate in a sustainable way. Never!</p><blockquote><strong>What is needed is an accurate quantification of the cost of removing CO₂ from the atmosphere (so-called CDR) that can be established as a baseline measure for carbon removal.</strong></blockquote><h4><strong>Hard to quantify costs around CDR</strong></h4><p>There is a good reason for not currently being able to quantify the carbon cost easily— because carbon mitigation is complex and a fast changing picture: do you plant trees which require land and long periods of time, do you invest in sustainability projects, do you pay someone for a carbon certificate? What do these mean and do they even achieve the goal of removing carbon from the environment?</p><p>An organisation that generates greenhouse gases now and wants to reduce that impact, has broadly two options:</p><ul><li>change the operation so that they reduce or don’t emit carbon in the first place</li><li>acknowledge that some carbon is part of their current operational cycle and attempt to mitigate it through various removal approaches</li></ul><p>Each of these possibilities carries implementation challenges. But for a business to know whether its worth changing its operation it needs to accurately understand the actual cost of carbon mitigation.</p><h4><strong>Direct Air Capture of CO₂</strong></h4><p>So now we have established the need for a standard cost metric for atmospheric carbon removal or CDR, we need a standard process that can be costed. Enter Direct Air Capture or DAC.</p><p>The process behind Direct Air Capture is relatively simple and starts with a similar chemical reaction to the one used on the Apollo moon missions to remove CO₂ from the various manned space capsules.</p><p>As shown below, ingested atmospheric air (with usually only 450 parts per million of CO₂ as of 2023!) is passed through a potassium hydroxide solution. This effectively scrubs the air of CO₂ but leaves a solution of potassium carbonate. Fortunately potassium hydroxide can be recovered from the carbonate salt by means of a pellet reactor and a calcium hydroxide solution. While the potassium hydroxide can be reused directly, the resulting calcium carbonate needs to be further treated in order to extract the CO₂ (for compression and storage) and turned back into calcium hydroxide via a final slaking process.</p><figure><img alt="A process diagram representing the chemical process behind direct air capture of CO2. It shows air being ingested on the left and CO2 being produced on the right." src="https://cdn-images-1.medium.com/max/1024/1*9nvyU7gE5a2cG8VNNduDtw.png" /><figcaption>The chemical extraction process behind Direct Air Capture of CO₂ (from Keith <em>et al</em>. [3])</figcaption></figure><p>This all looks very neat — the circularity of the two central processes obviates the need for large amounts imported chemicals. Just as well considering we will ideally need to extract in excess of 37 billion tonnes of CO₂ per year [4] by one means or another (including vegetation growth and oceanic absorption) just to keep at the position we are today.</p><h4>But where’s the catch?</h4><blockquote><strong>However this seemingly saviour process comes with a big catch — it is enormously energy intensive.</strong></blockquote><p>According to [3], it takes 67MWh of energy to capture 170 tonnes of atmospheric carbon dioxide using this process. This may seem like a huge negative and it is indeed a practical challenge of implementation.</p><p>For our purposes however, it is exactly what we need to know. Electricity is sold on the wholesale market by the MWh. Currently, as I write, the price of electricity on the UK’s National Grid is retailing £207 per MWh. This gives us a means of pricing CDR both in the short term, and over the longer term. At that price it would cost an eye watering £81 per tonne of CO₂ removed. For context, this figure has regularly dropped to £30-£40 per tonne and the average cost per tonne of CO₂ removal has been £61.85 in the last week in the UK.</p><p>Actually it’s worse than that because the UK Grid is also currently operating at a carbon intensity of 134gCO₂/kWh, but this can be taken into account (as it clearly ought to be). Removing the carbon from the electricity generation as well, gives an adjusted price of £86.13/tonne.</p><h4>In general</h4><p>For a generic CDR technology (<em>t</em>) that takes <em>Eₜ</em> MWh to extract <em>Mₜ</em> kg of CO₂ with an energy cost of <em>Cₛ</em> $/MWh where the energy source (<em>s</em>) has a carbon intensity of <em>Iₛ </em>gCO₂/kWh, the $ cost per kg of CO₂ removal is:</p><figure><img alt="\frac{E_t C_s}{M_t — I_s E_t} where I_s E_t is much less than M_t" src="https://cdn-images-1.medium.com/max/332/1*ZU9vkRboFEOtZ9mXv3mg0Q.png" /><figcaption>Formula for cost per kg of extracting carbon dioxide from the environment</figcaption></figure><h4>Constructing the Carbon metric</h4><p>You would of course not want to sample the spot market price for electricity to give you your carbon metric, you would for instance:</p><ul><li>take a rolling average over a period of time</li><li>find a market where electricity was cheap and itself low-carbon</li><li>perhaps only pay for DAC at times when the wholesale price of electricity has dropped below a certain point</li></ul><p>All these are achievable and can be reflected in a suite of metrics to conform to a gold-standard actual cost of carbon removal for an organisation that has no other way of reducing its operational carbon output but wants to do the right thing!</p><h4>Conclusion</h4><p>There we have it. As my colleague James Tawton more succinctly put it:</p><blockquote><strong>It’s complicated, but we can now expose the costs and savings implications of CDR via direct air capture as part of this data-driven model. With this, organisations and companies can ensure the financial, legal and ethical integrity of their sustainability strategy.</strong></blockquote><p>For now an idea, but one we are looking to implement. We work with companies who generate carbon dioxide in various parts of the world through heavy fleet movements, flights, supply chains, etc. We will now be able to introduce realistic cost metrics into our optimisation algorithms that can take into account carbon capture and removal (after first minimising the carbon generated).</p><p>I am sure it will generate some interesting discussions but as companies realise that they are going to be liable for this carbon removal cost at some point, it’s probably a good idea to set that money aside now. Even better, start investing in carbon removal technology and plants now and ensure that it is in a form they can be 100% sure actually does the job.</p><h4>Postscript: Saving the planet?</h4><p>To be clear I am not advocating using DAC as the core technology for carbon reduction in the future — it is more by way of a upper bound on what we may have to do [7]. If other approaches involving increasing global vegetation cover, rewilding, restoring peat bogs, and just burning less stuff can be used and <strong>importantly verified </strong>[5] then they will likely be easier and much cheaper. DAC of some variety is part of the mix of approaches that we will need to rescue our planetary ecosystem [2].</p><p>But let’s play the thought experiment. If we were to put aside a sum of money and an energy budget to reverse carbon dioxide concentrations in the atmosphere — what would that look like?</p><p>Using the UK’s grid statistics for the last year, 2022, (which was a high cost year for lots of reasons): £194.79 per MWh at an intensity of 178gCO₂/kWh gives a cost metric of £94.61 per tonne of CO₂ extracted.</p><p>If we plan to remove all annual CO₂ emissions and start to reverse atmospheric concentration levels we probably want to plan to remove around 40 billion tonnes annually. That would be £3.78tn or $4.58tn per year of OPEX cost (roughly 4.1% of annual global GDP). To be clear though this would need us to divert around 55% of current globally generated electricity (based on 2021 generation [8]) into carbon sequestration, so more efficient and cheaper techniques are definitely needed as well.</p><p><strong>And finally…</strong></p><p>If you are interested in joining us at <a href="http://www.datasparq.ai/">Datasparq</a> for exciting projects like this or just finding out how your company can benefit from our sustainable and ethical approaches to Data Science and Engineering, <a href="/service/mailto: info@datasparq.ai">please get in touch</a>. We’ll be delighted to hear from you.</p><p><strong>References</strong></p><p>[1] S. Feingold <a href="https://www.weforum.org/agenda/2022/09/norways-massive-sovereign-wealth-fund-sets-net-zero-goal/">Norway’s massive sovereign-wealth fund sets net-zero goals</a>. World Economic Forum, September 2022</p><p>[2] A. Owen, J. Burke, E. Serin. <a href="https://www.lse.ac.uk/granthaminstitute/publication/who-pays-for-greenhouse-gas-removal-in-the-uk-designing-equitable-policy-to-fund-beccs-and-daccs/">Who pays for greenhouse gas removal in the UK? Designing equitable policy to fund BECCS and DACCS?</a> Grantham Research Institute, LSE, August 2022</p><p>[3] D.W. Keith, G. Holmes, D. St-Angelo, K. Heidel. <a href="https://doi.org/10.1016/j.joule.2018.05.006">A Process for Capturing CO₂ from the Atmosphere</a>. Joule. Volume 2(8), pp. 1573–1594, 15 August, 2018</p><p>[4] I. Tiseo, <a href="https://www.statista.com/statistics/276629/global-co2-emissions/">Annual carbon dioxide (CO₂) emissions worldwide from 1940 to 2021</a>. statista.com. 3 January, 2023</p><p>[5] P. Greenfield. <a href="https://www.theguardian.com/environment/2023/jan/18/revealed-forest-carbon-offsets-biggest-provider-worthless-verra-aoe">Revealed: more than 90% of rainforest carbon offsets by biggest certifier are worthless, analysis shows</a>. The Guardian, 18 January 2023</p><p>[6] R. Neate. <a href="https://www.theguardian.com/business/2023/feb/03/worlds-biggest-investment-fund-warns-directors-to-tackle-climate-crisis-or-face-sack">World’s biggest investment fund warns directors to tackle climate crisis or face sack</a>. The Guardian, 3 February 2023</p><p>[7] S.M. Smith, <em>et al</em>. <a href="https://www.stateofcdr.org/">The State of Carbon Dioxide Removal: A global, independent scientific assessment of Carbon Dioxide Removal</a>. 1st Edition. University of Oxford. January 2023.</p><p>[8] <a href="https://www.statista.com/statistics/270281/electricity-generation-worldwide/">Electricity generation worldwide from 1990 to 2021</a>. statista.com. 16 Jan 2023.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d537349bc25e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/can-data-science-and-data-transparency-help-save-the-planet-d537349bc25e">How data science and data transparency can help save the planet</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BigQuery vs Synapse — who is currently winning the battle?]]></title>
            <link>https://medium.com/datasparq-technology/bigquery-vs-synapse-who-is-currently-winning-the-battle-7b9e3b2d19f2?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/7b9e3b2d19f2</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Filip Trocinski]]></dc:creator>
            <pubDate>Tue, 11 Oct 2022 14:16:35 GMT</pubDate>
            <atom:updated>2022-10-11T14:16:35.780Z</atom:updated>
            <content:encoded><![CDATA[<h3>BigQuery vs Synapse — who is currently winning the battle?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rNyXaAA3Ns8Wyb47" /><figcaption>Photo by <a href="https://unsplash.com/@wacalke?utm_source=medium&amp;utm_medium=referral">Mateusz Wacławek</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data Warehouse — a place where the magic happens. A place where you can throw massive amounts of data from various origins, write some SQL and draw meaningful conclusions — and this is all without stressing out about your CPU power.</p><p>The definition of a data warehouse evolved tremendously in the last ten years. Companies started to give up the traditional, obsolete service-oriented architectures and joined the rush movement into cloud computing. As part of this change, such products as Google BigQuery, Azure Synapse and Amazon Redshift appeared on the market, offering extensive data analysis for a reasonable price.</p><p>To not become a believer of one tool, it is sometimes worthwhile to revisit the market, and compare your main weapon with the other alternatives. While working on projects for a wide range of clients, where each one has different needs and preferences, I have recently had a chance to dive deeper into Synapse. Upon project completion, I have decided to compare it to BigQuery, where my previous experience was based, and share some thoughts based on what I discovered/observed. Taking into respect multiple categories, we will try to answer the question — is one product surpassing the other in 2022?</p><h3>Cost</h3><p>So let’s start with what the most individuals want to know first — the price. And although predicting it with a fine-toothed comb is virtually impossible, the choice of the used pool type is going to influence the end price the most, therefore I am going to focus on this aspect.</p><p><em>NOTE: Wanting to make the costs clear and transparent to everybody, this section turned out to be the longest. Having in mind not everybody is interested in such thorough descriptions, I have prepared a shortened version of costs in form of a table, after which you are officially allowed to move to the next section.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*uUkgIeUePlSzJdX21ZvA2A.png" /><figcaption>* Prices with respect to London’s region i.e. ‘uk-south’ in Azure and ‘europe-west2’ in GCP</figcaption></figure><p>For the ones wanting to dive deeper and know all the details — here you are!</p><h4><strong>DEDICATED POOL - Synapse:</strong></h4><p>In this option, we decide to pay in advance for the specific pool performance we want to use. The power however, is very flexible and can be adjusted to the current need anytime. The hourly cost of the smallest cluster — <strong>DW100c </strong>is going to cost us in this moment <strong>£1.40</strong>, while such a monster as <strong>DW30000c </strong>will be even <strong>£421.14</strong>! Charges are only applied when the cluster is up and running, therefore, forgetting to pause it when is idle may cost you a pretty penny. Using Azure functions, we can automate that process, pausing compute instances at a fixed time or after a specified amount of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fUY0LacPYBSvwhl-XgejsQ.jpeg" /></figure><p><strong>SERVERLESS POOL - Synapse and BigQuery:</strong></p><p>Serverless is another possibility our queries can run. It is the native environment for BigQuery, which was designed and built from the beginning as the serverless platform, whereas in Synapse it is an option. The payment models of this mode are very similar in both services and are very cost-efficient. Instead of paying for the resources deployed, you pay per-query — and the more computationally heavy it is, the higher amount you need to pay. In this option you do not care about pausing clusters, as the charge is only made after executing a query. The price of processed TB is slightly cheaper in Synapse at <strong>£5</strong>, when its <strong>£6.25</strong> in BigQuery. Both services offer the first TB of processed data for free and the quota renews every month! This benefit, however, will be dropped out from Synapse in the end of December 2022.</p><p>The other cost we need to add on to both services is saving and storing the tables. In case of BigQuery tables are saved internally, which means any data used for analysis is priorly imported directly into BigQuery. Synapse on the other hand does not support internal tables in the serverless option, and tables are saved externally on Data Lake Storage Gen2. The advantage of internal tables is that running queries against them (especially if tables are very large) is measurably faster in comparison to the external tables. The disadvantage is the slightly higher price; 1GB of stored data in BigQuery is going to cost monthly ~<strong>£0.023</strong>, whereas in Synapse ~<strong>£0.016</strong>. However, the pricier service allows storing the first 10GB of data per month charge-free. Both cases assume the table is stored in the closest region (in this case, London), is frequently accessed and modified.</p><p><strong>ALTERNATIVE PAYMENT OPTIONS - Synapse and BigQuery:</strong></p><p>For those whose pockets are not the bottomless pits and want to have a greater control over the bills, both services offer something to prevent ending up a bankrupt. In BigQuery you can opt for <strong>flat-rate pricing </strong>option and pay for the purchased amount of BigQuery slots (processing power) on the <em>annual</em>, <em>monthly</em><strong> </strong>or<strong> </strong><em>flex</em> commitment plans. Synapse on the other hand offers ‘safe limits’ … which cause a Budget Exhausted error when crossed.</p><p><strong>Trial period:</strong></p><ul><li>The trial period in Synapse lasts 30 days, during which we can enjoy the free <strong>$200</strong> credit given us by Microsoft to explore all of the tools Azure has to offer. Some of them remain free up to one year. That looks even better in Google, where the new users receive free <strong>$300</strong> credit valid for all of the cloud tools for 90 days!</li></ul><p><strong>Student pricing plan:</strong></p><ul><li>Azure offers students a very attractive option to use the cloud, giving them <strong>$100</strong> credit, even without the need to provide a credit card! Although Google offers students their qwiklabs where they can learn the fundamentals of cloud computing, there is not really an option for more experienced young users wanting to create and manage their own projects at a reduced price.</li></ul><p>Summing the costs section up, the pricing plan in BigQuery looks slightly more attractive from the one in Synapse, therefore BigQuery is the winner in this category.</p><p>Stats so far:<em> Synapse: 0, BigQuery: 1</em></p><h3>Speed</h3><p>This factor is the toughest to measure as it depends on plenty of factors to be considered and in the end, probably everybody would need to do their own benchmark. Mine was based at the serverless option, as this is the one we run the most projects on in Datasparq. What I was mainly interested in was, how the tool deals with the complex operations on big datasets and conversely — how long does a simple SELECT * take?</p><p>To check this out, I started big and ran a query to first create an artificial 120 million row dataset based on CROSS JOINs, then ran a rolling window on it and finally save the result table. This is the operation I was worried about the most in Synapse because first — it is very computationally heavy, and second — unlike BigQuery, Synapse does not allow users to save tables internally and forces saving them externally in .parquet files. My fears however, turned out to be totally unfounded, and Synapse surprised me very positively, completing the operation in a very similar time to BigQuery. The only major speed difference I have spotted between these two services was running the SELECT * statement at the newly created datasets. When BigQuery was repeatably completing the operation within 15–20 seconds, the process of listing rows in Synapse was going on <strong>even 10 minutes after BigQuery was done!! </strong>I found this circumstance very interesting, as contrary to my guesswork, saving results to the external tables was taking around 30 times shorter than producing the result-set.</p><p>Concluding, we do not need to worry about our increasing dataset, in either of these services. Both of them can perform complex operations on a large amount of data very smoothly. However, not both of them can handle retrieving rows to the result tables equally well, therefore the speed classification belongs to BigQuery.</p><p>Stats so far: <em>Synapse: 0, BigQuery: 2</em></p><h3>Simplicity</h3><p>Leaving the underlying technology of the tool behind, one of the most detrimental parameters the team should be thinking of is how quickly the final solution can be implemented and delivered. That is why simplicity turns out to be a very deterministic component with a high weight.</p><p>Realistically, it does not take more than 5 minutes to set up BigQuery. What we need to do to start using BigQuery is to create a bucket in Google Cloud Storage (GCS) and then load files into it. Afterwards, we can directly head up to BigQuery to create a dataset and a table under which we can point our file located in GCS. This is the bookish way, however, BigQuery offers a more lazy option as well being able to upload the file directly there, point the file located on GDrive, or even on the solutions of competitors such as Amazon S3 or Azure Blob storage!</p><p>To run the first query in Synapse, on the other hand, we are required a prior set up of a resource group, storage account and finally a Synapse Workspace, which altogether takes considerable longer. The point goes to BigQuery.</p><p>Stats so far:<em> Synapse: 0, BigQuery: 3</em></p><h3>Running queries in Python:</h3><p>At some stage, while building a data engineering pipeline we reach to the point where running queries needs to be automated. From the abundance of scripting languages we most likely choose Python, thanks to its simplicity and a great support of Data Engineering and Data Science tools.</p><p>Running SQL queries against BigQuery from the Python perspective is very simple and is possible through the BigQuery Python client running the query() method just after specifying the project, dataset and table names.</p><p>The same operation is much more complicated to achieve in Synapse and requires a prior set-up of ODBC drivers, which is different for every UNIX architecture. BigQuery receives another point from me.</p><p>Stats so far:<em> Synapse: 0, BigQuery: 4</em></p><h3>Syntax/limitations:</h3><p>The immediate difference we spot right away is — SQL in BigQuery does not equal SQL in Synapse. The default query language in the latter one is Microsoft Transact-SQL (T-SQL), but if you want to write queries in Kusto Query Language (KQL), you are more than welcome. BigQuery, on the other hand bases upon Google Standard SQL dialect, which ceaselessly since 2016 is a standard.</p><p>This all involves the small changes like the way of defining BOOLEAN types (BIT (1, 0) in Synapse, BOOLEAN (TRUE, FALSE) in BigQuery), through the lack of few useful methods existing only in BigQuery (such as EXCEPT, REGEXP_REPLACE, USING, etc) up to the way of saving external tables.</p><p>From the perspective of a BigQuery user, a lack of these methods is already noticeable in an attempt of selecting every column apart from THIS ONE. Let’s consider this case: your source table has 250 columns, but within your CTE, you only want to list 230 of them. Doing this in GCP would look as simple and neat as this: SELECT * EXCEPT(column_1, column_2, column_3, …, column_20) FROM source_table.</p><p>Absence of an EXCEPT method (nor its equivalent) forces us to list all of them manually within the query. To not make it sound that dramatic, Synapse has a feature listing them down for us (right click on the source file, and choosing “<em>Select TOP 100 rows”</em> ). As all 250 columns were selected, we now need to manually remove the 20 we are not interested in. Sadly, such a bypass does not work within the CTE, and either we have the patience level above the average human being listing all of the columns manually, or we need to think of some smarter solution I have not found yet.</p><p>Many people will find the syntax of SQL in BigQuery more user-friendly, as in contrast to Synapse, it hides the underlying complexity and users can focus on writing the actual SQL straight away. For example — if we want to create an external table from the CSV file in BigQuery, all we need to do is create a dataset and then run the following statement:</p><pre>CREATE EXTERNAL TABLE sample_dataset.sample_table </pre><pre>OPTIONS (</pre><pre>format = ‘CSV’,</pre><pre>uris = [‘gs://bucket/our_source_csv_file.csv’]</pre><pre>)</pre><p>Synapse on the other hand, requires prior creation of a specific Database and File Format. On top of that, it is mandatory to define schema within the CREATE statement. And although Synapse can write them for you (when you click on the name of the <em>.csv</em> file within the Data tab), all these fields will require supervision and probably modification to the desired filed type and length. In the scenario of having a big number of columns, things are getting pretty exhausting again.</p><pre>CREATE EXTERNAL TABLE [sample_dataset].[sample_table](</pre><pre>[Field1] nvarchar(256),</pre><pre>[Field2] nvarchar(256),</pre><pre>[Field3] bigint,</pre><pre>(...)</pre><pre>[Field70] float</pre><pre>)<br></pre><pre>WITH (</pre><pre>LOCATION = &#39;DummyData.csv&#39;,</pre><pre>DATA_SOURCE = [sample_datasource_dfs_core_windows_net],</pre><pre>FILE_FORMAT = [CSVFormat]</pre><pre>)</pre><p>Although this kind of granularity may seem redundant, and only complicate something that could have been done easier, it also offers more customisability options, the practised user can take advantages from. If only Synapse was not lacking the Except statement, I would grant both services by one point, however, from the perspective of BigQuery user, this lack causes just too much inconvenience.</p><p>Stats so far:<em> Synapse: 0, BigQuery: 5</em></p><h3>Summary:</h3><p>Losing the scoring competition, Synapse seems like a bad choice to start working with, which it’s not. It’s still very powerful tool with a decent speed that can be used with a great effect in various data analysis operations. With BigQuery, however, you can achieve the same thing quicker, simpler and probably cheaper. Nevertheless, unless you are an individual, the choice you make, will most likely be determined on the IT architecture your company already use..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*643m0CTyNgVljHVLW2XEFg.jpeg" /></figure><p>For more insight follow <a href="https://medium.com/datasparq-technology">datasparq-technology</a>, and find us on <a href="https://www.linkedin.com/company/datasparq">LinkedIn</a>, <a href="https://twitter.com/DatasparqAI">Twitter</a>, and our <a href="http://datasparq.ai/">website</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b9e3b2d19f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/bigquery-vs-synapse-who-is-currently-winning-the-battle-7b9e3b2d19f2">BigQuery vs Synapse — who is currently winning the battle?</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is PlayML?]]></title>
            <link>https://medium.com/datasparq-technology/what-is-play-ml-78566a82faa0?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/78566a82faa0</guid>
            <category><![CDATA[web-design]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Matt Simmons]]></dc:creator>
            <pubDate>Mon, 28 Mar 2022 13:30:11 GMT</pubDate>
            <atom:updated>2022-03-28T15:25:32.721Z</atom:updated>
            <content:encoded><![CDATA[<p>Question: When’s the last time Machine Learning (ML) directly influenced your behaviour? Answer: At least 10 times today so far.</p><p>Data science concepts are being used to affect every aspect of our lives, from the articles you get recommended on Medium, to the prices you’re shown for a taxi, and even which breakfast cereal is on offer in the super market. In 10 years time, data science will be involved in <em>every</em> human decision.</p><p>…but it can be difficult to understand. Sometimes it’s impossible to see how a model arrives at its result, and we have to trust it blindly. Sometimes a problem can’t be understood because it’s too complex. We wanted to fix that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*97vWHf-3pWJLSeBBwLj45Q.gif" /><figcaption><a href="https://play-ml.datasparq.ai/">play-ml.datasparq.ai</a></figcaption></figure><h4>Introducing PlayML</h4><p><a href="https://play-ml.datasparq.ai/">play-ml.datasparq.ai</a> is a website where curious people can play around and experiment with the most cutting edge data science techniques. Challenging problems are visualised, made interactive, and broken down into basic statistics.</p><p>We want tough concepts, such as delivery route optimisation, to seem intuitive, so that people from all backgrounds feel confident when deciding to use data science to solve a problem.</p><p>We’re not just trying to teach statistics, we’re explaining real world problems and and real solutions that we’ve built ourselves. The techniques shown in PlayML have real proven value and are being used by the world’s leading companies <a href="https://datasparq.ai/our-work">right now</a>!</p><p>Here are some of our best notebooks to get started:</p><h4>1. Supply Chain - Queueing Model</h4><p>This notebook simulates the supply chain of a pizza factory. Scenarios like these suffer from complexity and unpredictability. How do we prevent bottlenecks and ensure we’re producing as much as possible?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*KZj7m4JvjcpGm-wB3M-R5Q.gif" /></figure><p><a href="https://play-ml.datasparq.ai/article/supply_chain">Check it out here</a>.</p><h4>2. Dynamic Pricing - Taxi Company</h4><p>Ever been shocked to see the price of a taxi shoot up during rush hour? This notebook explores how the supply and demand of taxis can be balanced by adjusting the price in real-time and the algorithms used to optimise it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*q4ykp4CUkcNR0WpQ301LKg.gif" /></figure><p><a href="https://play-ml.datasparq.ai/article/pricing_algorithm_v3">View the notebook here</a>.</p><h4>3. Vehicle Routing</h4><p>How does a delivery company optimise their routes to minimise cost and get you your package sooner? It’s an increasingly complex problem, with an ever increasing number of variables. We’ve come up with an approach that beats commercially available software.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*iyBbx3F2iS1xJHYO0KwI3w.gif" /></figure><p><a href="https://play-ml.datasparq.ai/article/vehicle_routing">Try it out here!</a> Or <a href="https://datasparq.ai/vehicle-routing-problem">watch the video</a>, where two of our top Data Scientists talk through the problem.</p><h4>Help us Spread the Word</h4><p>We want everyone to have insight into the algorithms and techniques in use all around them. If you like our content, please share it!</p><h4>👉 <a href="https://play-ml.datasparq.ai/">play-ml.datasparq.ai</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78566a82faa0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/what-is-play-ml-78566a82faa0">What Is PlayML?</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I like Go so much, with examples]]></title>
            <link>https://medium.com/datasparq-technology/why-i-like-go-so-much-with-examples-8583e73895d3?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/8583e73895d3</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Matt Simmons]]></dc:creator>
            <pubDate>Thu, 17 Mar 2022 15:18:21 GMT</pubDate>
            <atom:updated>2022-06-17T10:51:55.683Z</atom:updated>
            <content:encoded><![CDATA[<p>Out of all the languages I’ve used, Go is the best. I find myself saying this to a lot of people, so here’s my attempt to explain <em>why</em>.</p><p>My perspective comes from years of data engineering with Python and front-end development with JavaScript/TypeScript. I’ve been exposed to a fair amount of C, C++, Java, C#, and R, and have formed a lot of opinions about the languages themselves, as well as their implementations and ecosystems.</p><h4>It’s Clean</h4><p>Here’s the ‘hello world’ program for Go. It’s like many others:</p><pre><strong>package</strong> main<br><br><strong>import</strong> &quot;fmt&quot;<br><br><strong>func</strong> main() {<br>  fmt.Println(&quot;Hello, world!&quot;)<br>}</pre><p>There’s no extra fluff like void , int main(), or return 0;. Instead, these are implied by the lack of arguments or return type in the function definition. Someone with no knowledge of the Go language can easily understand what’s going on.</p><p>There are no globally defined functions used; every function comes from a named package. This makes it vastly easier to follow than C++ or Java.</p><p>Public functions must start with capital letters, which avoids the need for public or private everywhere, and allows me to see which functions are public at a glance, and without seeing the definition.</p><p>Here’s a loop over a slice (a list):</p><pre><strong>for</strong> index, item := <strong>range </strong>myList {<br>  fmt.Print(item)<br>}</pre><p>Here’s a while loop. It’s still a for loop but it uses a condition rather than a range:</p><pre>sum := 0<br><strong>for</strong> sum &lt; 1000 {<br>  sum += sum<br>}</pre><p>There are little syntactic shortcuts like this everywhere.</p><h4>It’s Fast</h4><p>Surprisingly fast.</p><h4>It’s Forgiving</h4><p>I mentioned it’s fast, but for someone like me who can’t write C very well, it’s faster than any other language. This is because it optimises things for you. Forgot to make something a pointer? No problem!</p><h4>Slices</h4><p>Slices make arrays easier by abstracting away management of array length. They remain very fast but are also forgiving, providing a perfect middle ground between a Python list and a C++ array. Fixed length arrays are also available if I need them.</p><pre><strong>var</strong> s = []<strong>int</strong>{1, 2, 3}</pre><pre>s = <strong>append</strong>(s, 4)  <em>// slice grows in size</em></pre><pre>fmt.Println(s)  <em>// [1 2 3 4]</em></pre><h4>G<em>oroutines</em></h4><p>Goroutines are lightweight threads. They make parallelising code very easy and the syntax is much cleaner than most other languages, many of which have added concurrency as an afterthought.</p><p>They are as simple as they could possibly be. Just define any function as normal and use the go keyword.</p><pre><strong>go</strong> listenForPings()</pre><h4>Channels</h4><p>Sometimes you need to send information to a goroutine after you’ve started it, or get something to wait for a message before running. Channels do all of that and more.</p><p>I can make a channel that sends strings with ch := make(chan string). Then I can pass it to a function with go x.monitor(ch). My function (now running in a goroutine) will send messages to this channel (note the &lt;- syntax ).</p><pre><strong>func</strong> (x* <strong>HeartBeatDetector</strong>) monitor(channel <strong>chan</strong> <strong>string</strong>) {  <br>  <strong>if</strong> time.Since(x.Last) &gt; x.Rate {<br>    <em>// send name to channel to report dead process</em><br>    channel &lt;- x.Name  <br>  }<br>  ...<br>}</pre><p>Other functions could read the messages from this channel. This for loop will keep going until the channel is closed with close(ch):</p><pre><strong>for</strong> d := <strong>range </strong>ch {<br>  fmt.Printf(&quot;Oh dear, %v died!\n&quot;, d)<br>}</pre><h4>Error Handling</h4><p>No more try/catch blocks. Now errors are returned by any function that could result in an error:</p><pre>err, result := myFunc()<br><strong>if</strong> err != nil {<br>  <strong>panic</strong>(err)<br>}</pre><p>This may look ugly to a Python user, as there are now three extra lines of code for what is essentially an unhandled error, but at least now I (and my IDE) can see exactly what’s going to happen if there’s an error, so I won’t miss anything.</p><p>Here’s some more precise error handling, where I check the type of the error:</p><pre><strong>if</strong> err := Foo(); err != nil {<br>    <strong>switch</strong> e := err.(type) {<br>    <strong>case</strong> *<strong>SyntaxError</strong>:<br>        <em>// do something interesting with e.Line and e.Col</em><br>    <strong>case</strong> *<strong>InternalError</strong>:<br>        <em>// abort and file an issue</em><br>    <strong>default</strong>:<br>        log.Println(e)<br>    }<br>}</pre><h4>Static Types</h4><p>In my experience, typing ultimately makes code easier to write and understand. The downside is that it can lead to a bulky codebase. In Go, things are fairly clean.</p><p>I can define a type like this:</p><pre><strong>type</strong> item <strong>struct</strong> {<br>  id    <strong>int</strong><br>  value <strong>string</strong><br>}</pre><p>Then define a method for my type:</p><pre><strong>func</strong> (i *<strong>item</strong>) String() <strong>string</strong> {<br>  <strong>return</strong> i.value<br>}</pre><p>Then define a function that uses my type:</p><pre><strong>func</strong> ConcatItems(a <strong>item</strong>, d <strong>item</strong>) {<br>  <strong>return</strong> a.String() + b.String()<br>}</pre><p>I also need a method to create new instances of my type. The convention is to use the following pattern, where a function New creates the type defined within that module:</p><pre><strong>func</strong> New(id <strong>int</strong>, value <strong>string</strong>) <strong>item</strong> {  <br>  i := <strong>item</strong>{id, value}<br>  <strong>return</strong> i<br>}</pre><h4>JSON</h4><p>I can define how a type should be represented in JSON, then read a JSON string and convert it directly to my type. Validation is automatic.</p><pre><strong>type</strong> Mission <strong>struct</strong> {<br>  Name     <strong>string</strong>            `json:&quot;name&quot;`<br>  Services []<strong>Service</strong>         `json:&quot;services&quot;`<br>  Stages   []<strong>Stage</strong>           `json:&quot;stages&quot;`<br>  Params   <strong>map</strong>[<strong>string</strong>]<strong>string</strong> `json:&quot;params&quot;`<br>  isValid  <strong>bool</strong><br>}</pre><pre><strong>func</strong> NewFromJSON(jsonString []<strong>byte</strong>) <strong>Mission</strong> {<br>  <strong>var</strong> m <strong>Mission</strong><br>  json.Unmarshal(jsonString, &amp;m)<br>  <strong>return</strong> m<br>}</pre><h4>Easy to Compile</h4><p>I mean this both in general (simple command: go build, helpful error messages, fast compile times, easy install in the first place) and in terms of all the different operating systems and architectures I can compile to, including Web Assembly, which has allowed me to <a href="https://mattsimmons1.github.io/bb/playground/">run my Go package natively in the browser</a>!</p><p>I’ve used a simple <a href="https://github.com/MattSimmons1/bb/blob/main/.github/workflows/go.yml">build pipeline on GitHub</a> to build my packages for multiple architectures. It compiles binary distributions for 5 different platforms, and I didn’t have to configure anything.</p><h4>Runes</h4><p>Runes represent unicode codepoints, which get stored as int32. This means I can treat a and 你 in exactly the same way, even though a has one byte and 你 has 3 bytes. Thanks to runes, I can confidently parse text containing non-ASCII characters without messing anything up.</p><p>Go’s syntax treats &#39;s&#39; as a rune, and &quot;s&quot; as a string of length 1. &#39;hello&#39; would be invalid because it has multiple characters and therefore can’t be a rune. It only took a few minutes to get used to this, and it’s really nice to work with.</p><h4>go get</h4><p>I’ve gone through the process of turning Python code into packages and uploading to PyPi a few times now. It’s still confusing, and takes me a while.</p><p>With Go there’s no confusion. The command is 5 letters. The package location is the repo name, the install location is the repo name, the import path is the repo name. Very simple.</p><pre>go get github.com/MattSimmons1/bb</pre><p>It also compiles and installs the package automatically. I appreciate this as it allows people to install my program with a single line.</p><h4>Built-In HTTP Server</h4><p>Admittedly, I would rather use Python’s Flask or FastAPI for most use cases, but Go’s http server is very light weight. I’ve used it countless times.</p><pre>http.HandleFunc(&quot;/&quot;, serveHome)<br><strong>if</strong> err := http.ListenAndServe(&quot;:8080&quot;, nil); err != nil {<br>  <strong>panic</strong>(err)<br>}</pre><h4>Requests</h4><p>The http module does what you’d expect. It’s nice that I don’t need to install any additional modules.</p><pre>postBody, _ := json.Marshal(<strong>map</strong>[<strong>string</strong>]<strong>string</strong>{<br>  &quot;foo&quot;:  &quot;bar&quot;,<br>})</pre><pre>resp := http.Post(&quot;localhost:5000/api&quot;,<br>  &quot;application/json&quot;, bytes.NewBuffer(postBody))</pre><pre><strong>defer</strong> resp.Body.Close()<br><br>body := ioutil.ReadAll(resp.Body)<br><br><strong>var</strong> res []<strong>Response</strong><br>json.Unmarshal(body, &amp;res)</pre><p>If you need to do anything over UDP, TCP, DNS, or Unix Sockets, you can use net. Here I’m sending a string over UDP in the same way I would send a string to a file.</p><pre>conn, err := net.Dial(“udp”, “127.0.0.1:49161”)</pre><pre>fmt.Fprintf(conn, &quot;03C;13C;24C&quot;)  <em>// send data</em></pre><h4>Cobra</h4><p><a href="https://github.com/spf13/cobra">Cobra</a> is a library that can turn a go project into an easy to use command line tool in minutes. It’s used like this:</p><pre><strong>func</strong> main() {<br><br>  <strong>if </strong>err := <strong>func</strong>() (rootCmd *cobra.<strong>Command</strong>) {<br><br>    rootCmd = &amp;cobra.<strong>Command</strong>{<br>      Use:   &quot;my-cli&quot;,<br>      Short: &quot;My Tool&#39;s Command Line Interface&quot;,<br>      Args:  cobra.MinimumNArgs(1),<br>      Run:   <strong>func</strong>(c *cobra.<strong>Command</strong>, args []<strong>string</strong>) {<br>        DoStuff(args[0])  // my function <br>        <strong>return</strong><br>      },<br>    }</pre><pre>    rootCmd.AddCommand(<strong>func</strong>() (createCmd *cobra.<strong>Command</strong>) {<br>      createCmd = &amp;cobra.<strong>Command</strong>{<br>        Use:   &quot;version&quot;,<br>        Short: &quot;Print the version number&quot;,<br>        Run:   func(c *cobra.Command, args []<strong>string</strong>) {<br>          fmt.Println(&quot;v0.1.0&quot;)<br>        },<br>      }<br>      <strong>return</strong><br>    }())</pre><pre>    <strong>return</strong><br>  }().Execute(); err != nil {<br>    log.Panicln(err)<br>  }<br>}</pre><p>I could then run my-cli hello to run the main program, my-cli version to run a subcommand, or my-cli help (a subcommand Cobra made for me).</p><p>If I were to just run my-cli with no arguments, I would see:</p><pre>Error: requires at least 1 arg(s), only received 0<br>Usage:<br>  my-cli [flags]<br>  my-cli [command]</pre><pre>Available Commands:<br>  help         Help about any command<br>  version      Print the version number</pre><p>I can easily add all of the flags and subcommands I need, and don’t need to worry about overcomplicating my code. <a href="https://github.com/spf13/cobra/blob/master/projects_using_cobra.md">Many projects use Cobra</a>, including Kubernetes, so I can be confident that the interface for my tool will be familiar to lots of developers.</p><h4>time</h4><p>Go comes with the two types time.Time and time.Duration. Both give you nanosecond precision and are really easy to work with.</p><p>Use time.Now() to get the current time. If you want to know the duration since that time, use d := time.Since(t1), then format that duration however you like, or just print it’s value and it will automatically be formatted in a sensible way.</p><p>Unlike Python, I don’t have to convert between datetime.date, datetime.datetime, datetime.time, pandas.Timestamp, and time.struct_time. There’s just one time type that does everything I need it to.</p><h4>Mutex</h4><p>The mutex library allows me to easily guarantee that my Goroutines won’t try to update the same value at the same time. This isn’t relevant without regular use of Goroutines, but it’s very powerful.</p><p>Here I’ve given my type a property called mux of type sync.Mutex for making sure that multiple go routines can read and write to last, which records the last time a heartbeat was detected.</p><pre><strong>func</strong> (x *HeartbeatDetector) detected() {<br>  <strong>defer</strong> x.mux.Unlock()<br>  fmt.Printf(&quot;❤️&quot;)<br>  x.mux.Lock()<br>  x.last = time.Now()<br>}</pre><p>I also used the defer keyword here to make sure the mutex always unlocks when the function ends.</p><h4>Mascot</h4><p>Go’s mascot perfectly represents the philosophy behind the language.</p><p>It’s simple. It has everything it needs and nothing extra. There’s only a few basic features shown, but you can understand what it is right away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ztwC5SUq0Q26MBizwC4NxQ.png" /></figure><p>I can’t think of a better mascot in the world of software.</p><h4>Developer Experience</h4><p>If you’re using a good IDE (I strongly recommend GoLand), it will point out every potential issue to you as you code, so you’ll practically never have any unexpected errors. It shows the types of every variable and every unhandled error.</p><p>The most helpful thing it does is automatically add every module I reference to my import statement at the top of the file. If it weren’t for this, I would be wasting time trying to remember the location of every module, or just forget to import and get errors when I try to run.</p><h4>Tutorial</h4><p><a href="https://gobyexample.com/">gobyexample.com</a> provides the perfect way to quickly learn a new language. It starts simple, goes through each concept one at a time, and lets you run code in your browser.</p><h4>Conclusion</h4><p>In conclusion, Go is amazing. It should be part of everybody’s arsenal, along with Python (for simple scripting/data wrangling) and JS (for front-end). Go is perfect for anything that needs to be reliable and high performance, such as a Microservice, developer tool like Kubernetes or Terraform, or Data Science tool.</p><p>If you’d like any more tips you can find me on Twitter: <br><a href="https://twitter.com/MattSimmons01">@MattSimmons01</a> &amp; <a href="https://twitter.com/DatasparqAI">@DatasparqAI</a>.</p><p>Reach out to us at <a href="https://datasparq.ai/contact">datasparq.ai/contact</a> for advice on solving a problem with Data Science!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8583e73895d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/why-i-like-go-so-much-with-examples-8583e73895d3">Why I like Go so much, with examples</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Levenshtein distance as a remedy for sequential data]]></title>
            <link>https://medium.com/datasparq-technology/levenshtein-distance-as-a-remedy-for-sequential-data-4b6653b17be6?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/4b6653b17be6</guid>
            <category><![CDATA[levenshtein-distance]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[sequencing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Filip Trocinski]]></dc:creator>
            <pubDate>Thu, 20 Jan 2022 15:49:57 GMT</pubDate>
            <atom:updated>2022-01-20T15:49:57.102Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6PhIRwYf86sTsQg5" /><figcaption>Photo by <a href="https://unsplash.com/@popnzebra?utm_source=medium&amp;utm_medium=referral">Pop &amp; Zebra</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Imagine a large web portal is reaching out to you to create an AI solution that will be recommending the most relevant articles to users. That’s exactly the task we have been tackling in Datasparq recently. We needed to determine which articles had multiple parts, and what order they should go in based on the titles alone so that the next article in the sequence can always be recommended.</p><p>But articles are not the only example where you can come across sequential data. Consider movies and TV series for instance — they would not have produced <em>Avatar</em> <em>2</em>, <em>3</em>, <em>4</em> and <em>5 </em>if the first film had not been a success. In this article, we are going to replicate the solution we devised and create a BigQuery table of movie sequels so that we know which movie follows the preceding one e.g. <em>Avatar 2</em> comes after <em>Avatar</em> and <em>Star Wars: The Rise of Skywalker </em>is the continuation of <em>Star Wars: The Last Jedi</em>.</p><p>We use “IMDb movies extensive dataset” available on Kaggle: <a href="https://www.kaggle.com/stefanoleone992/imdb-extensive-dataset">https://www.kaggle.com/stefanoleone992/imdb-extensive-dataset</a>, which contains over 85,000 movie titles. This massive dataset comprises four smaller chunks, namely <em>IMDB movies.csv</em>, <em>IMDB names.csv,</em> <em>IMDB ratings.csv, and</em> <em>IMDB title_principals.csv</em>. We are interested in the latter because this is the one that contains the titles that we will parse and from which we will extract sequences. Having said that, let’s get started!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*kpWnI2NCK2K6CIKsp7WkZA.png" /><figcaption>How would you know these two movies are related if there was not any text on the posters?</figcaption></figure><h3>Setting things up</h3><p>To analyse this dataset, we will be using BigQuery which is one of the key GCP tools. Although this is a paid service, everybody is eligible for $300 credit to explore this massive toolset (which is way more than enough for the purposes of this project).</p><p>To load the data, we first need to create a dataset in which we insert the table. Specifying the location, choose the one which is the closest to you, so that data latency will be the lowest, and queries will be executed faster. We will be creating a table from the .csv containing all data. For convenience, it is best to upload this file into the dedicated bucket on Google Cloud Storage (GCS).</p><p>Letting it figure out the schema automatically will probably raise an error because some of the columns do not have consistent data. The (not-ideal) solution we are going to apply now is to set these columns with the type of STRING. To save your time defining everything manually, just copy and paste the following to <em>Schema</em> as text.</p><pre>imdb_title_id:STRING,<br>title:STRING,<br>original_title:STRING,<br>year:STRING,<br>date_published:STRING,<br>genre:STRING,<br>duration:INTEGER,<br>country:STRING,<br>language:STRING,<br>director:STRING,<br>writer:STRING,<br>production_company:STRING,<br>actors:STRING,<br>description:STRING,<br>avg_vote:FLOAT,<br>votes:INTEGER,<br>budget:STRING,<br>usa_gross_income:STRING,<br>worlwide_gross_income:STRING,<br>metascore:STRING,<br>reviews_from_users:FLOAT,<br>reviews_from_critics:FLOAT</pre><h3>Levenshtein distance</h3><p>Now we are moving to the cornerstone of this article — Levenshtein distance. In the simplest words, this metric tells us how much one text string is different from another i.e. how many characters do we need to change in one string to make it identical to the other.</p><p>Unfortunately, there is no ready-made Levenshtein distance function to use in BigQuery so we need to create one on our own. BigQuery supports creating user-defined functions (UDFs) written as SQL expression or JavaScript code. Luckily for us, Andrei wrote the Levenshtein distance function a couple of years back and published it on GitHub under the MIT License: <a href="https://gist.github.com/andrei-m/982927">https://gist.github.com/andrei-m/982927</a>. We are going to use his bit of code. However — we will implement a few changes.</p><p>Currently the code counts the exact Levenshtein distance, regardless of how big it could be. Considering the huge amount of records present within the dataset this would be a very computationally heavy process involving lots of time to run the whole query. Moreover, this would consume a much larger amount of resource which would consequently lead to higher bills to pay for GCP (or to exhausting the free $300 credit much quicker).</p><p>The thing is, we do not really care about the exact Levenshtein distance number because as soon as it crosses the determined boundary, it is beyond our limit anyway. To address that, and significantly speed up the query execution, we are going to return the max set boundary for the given title, as soon as it is crossed. Implementation of these changes will allow us to execute the code approx <strong>100x faster</strong>!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/da854d34ed454d83b93ec2b0b357716f/href">https://medium.com/media/da854d34ed454d83b93ec2b0b357716f/href</a></iframe><h3>Data cleansing</h3><p>To speed up performance and increase readability, we don’t want to select all the columns from the dataset but rather focus on these (which can be useful in the later stages). The most appropriate ones in our case will be <em>imdb_title_id</em>, <em>title</em>,<em> director, </em>and <em>date_published.</em></p><p>After a quick dataset exploration, we realise the sequence is expressed in several ways. Below are a few examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*3RAoZz9cuYSRPDa82_B_PA.png" /><figcaption>Figure 1 — sequence at the end of the title</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*Y75i5c1erGCNyPGUXljhCw.png" /><figcaption>Figure 2 — sequence written as a roman numeral</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*rjPwvebCFNlKFq61RIXjQQ.png" /><figcaption>Figure 3 — sequence in the middle</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*oL2dh7YV81rLTp6yKET6GQ.png" /><figcaption>Figure 4 — sequence at the beginning of the title</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*YEdKAHzrOPe5h2ZrqyIRaA.png" /><figcaption>Figure 5 — sequence as the year at the end</figcaption></figure><p>Data engineering involves a lot of customisation of the data we work with. To extract all sequences from the various titles, we will need to adjust our query to catch all of the cases.</p><p>As was already mentioned, Levenshtein distance checks how many characters we need to change in one string to make it identical to the other one. What we strive for, is to find the golden mean — in terms of how many characters we allow to be different without having a negative impact on our final result. We do not want to set it too high, as we do not expect the next sequel to have a totally different title. It’s just the sequence that will be different. We also want to minimise the number of comparisons, to avoid extra costs associated with the query processing and to make it run faster.</p><p>And here comes the problem… If we do not set the distance high enough, we will miss plenty of movies that are actually sequential e.g. those in <em>Figure 3</em>,<em> </em>which list the subtitle as well as the title. Fortunately, there is a simple workaround we can use in that situation. We just have to REPLACE() these substrings with empty strings so taking <em>Figure 3 </em>as an example; <em>“Destination Dubai”</em> will be deleted leaving just <em>“The Wedding Party 2”</em>. The solution works perfectly fine, removing what is unnecessary and just keeping the string with a valuable number.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/302df79c6198e582e304122992159196/href">https://medium.com/media/302df79c6198e582e304122992159196/href</a></iframe><h4>Roman numerals to Arabic numbers:</h4><p>Here comes another tricky bit. We cannot just catch the sequence written as a Roman numeral and REPLACE()it with the Arabic one because this will only pass muster in a very simple situation e.g. when the roman numeral is one of I, V, X, etc. What can we do then? Well, this is another time we can make use of UDFs and create such a method that will locate the Roman numeral, find an Arabic representation of it, and finally replace it. That is exactly what Mikhail did in this StackOverflow post:</p><p><a href="https://stackoverflow.com/questions/69771597/what-is-the-best-way-of-converting-roman-numerals-to-arabic-ones-in-bigquery">What is the best way of converting roman numerals to arabic ones in bigquery?</a></p><p>After a little modification of Mikhail’s code to suit our needs, this is what we are going to use:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f43ce234a2ce501b056ca185896b3400/href">https://medium.com/media/f43ce234a2ce501b056ca185896b3400/href</a></iframe><p>and subsequently this…</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27da95c22f9317066d392fab9f126be6/href">https://medium.com/media/27da95c22f9317066d392fab9f126be6/href</a></iframe><h4>Sequence extraction:</h4><p>Contrary to what you may have thought about extracting sequences, we do not need to define another fancy UDF :) Multiple CASE statements with a simple Regex do this job pretty well. As mentioned before, they just need to be properly adjusted to catch every way the sequence is expressed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/92785253671fe2b8331babf3c438d74a/href">https://medium.com/media/92785253671fe2b8331babf3c438d74a/href</a></iframe><h4>Levenshtein distance in practise:</h4><p>As we already have the Levenshtein distance method, it is now time to use it in practice. To be able to compare one text string to another, we need to double the data by joining a table to itself using CROSS JOIN.</p><p>This operation can be very computationally heavy and take ages to run, hence the query optimisation is key, and that is why we did not want to count the precise value of Levenshtein distance once it crosses our max_boundary.</p><p>Although by using INNER JOIN, we could have the result-set produced quicker, we would need to specify what we should join the tables on. The only options would be either on director or writer. However, there are lots of sequels where both of them are different. Joining tables based on them may not fully represent our result-set, as we would probably just obtain its fraction. We would also not join the tables based on movie ids, as we would not want to compare movies to themselves.</p><p>As was said before, there is no one-size-fits-all value for Levenshtein distance, and it should always be determined with regard to the data and the problem we are working on. For this specific case, I found allowing up to 3 characters to be different a reasonable number but I highly recommend experimenting with this value on your own, if only to see how the final output can be dramatically different!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4995b0dda8040d2481fd85456c190dc2/href">https://medium.com/media/4995b0dda8040d2481fd85456c190dc2/href</a></iframe><h4>Determining sequentiality:</h4><p>To clearly indicate whether the movie is a subsequent sequel, it would be nice to add a column and express sequentiality in the form of a simple Boolean statement; TRUE — this movie is the continuation of the previous one, FALSE — no it’s not. To do so, we will just need to form specific case statements, of how the sequentiality should be determined based on seq_1, seq_2, and next_by_date.</p><p>We know for sure, in the case where seq_2 is greater by one than seq_1, we want to mark this movie as the next part. Not having first parts explicitly marked as 1, slightly complicates this task and forces us to form a slightly more complex WHEN statement: WHEN seq_1 = 0 THEN seq_1 + 1 = seq_2.</p><p>Although this should be enough in most cases, a subset of such movies may exist where the titles are near identical, the movies have been produced by the same director, and contain a sequence in the title. This is a dangerous case, where a totally different movie may be declared as a continuation of another one.</p><p>To prevent that, we are going to use next_in_sequence parameter which tells us what the subsequent movie produced by the same director was with respect to the date. To make things clear — for sequence determination, we cannot only rely on this parameter. There may have been a director who produced a couple of other movies in the gap between the sequels of the given one. This parameter, however, works well as an additional layer of certainty where the next sequel is the correct one. After incorporating it, our statement looks like this:</p><pre>WHEN seq_1 = 0 AND next_by_date = 1 THEN next_by_date + 1 = seq_2 OR seq_1 + 1 = seq_2</pre><p>Finally, if neither of these two statements above returns TRUE, we want to classify that movie as non-sequential. Expression seq_1 = next_by_date will do the job.</p><p>In the end our code should look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2bf99b42652d749978a39c4e2f4eda4/href">https://medium.com/media/e2bf99b42652d749978a39c4e2f4eda4/href</a></iframe><h4>Final result:</h4><p>Once we finally have all the bits done, we can combine the puzzles using WITH AS clause and produce the table. In the final SELECT statement, we are going to select all columns from the last subquery, determine the Levenshtein distance value and eliminate those rows that have empty strings in their titles (arising out of our previous REPLACE statements). For readability, we can also ORDER the records BY title_1.</p><p>Our table of sequential movies should look something like this and has well-over 1000 records:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mK4_6I1XRn-ixFcczNxJlQ.png" /><figcaption>Figure 6 — Final table</figcaption></figure><p>To find out more about Datasparq, <a href="http://info@datasparq.ai/">please get in touch</a> or go to our <a href="https://datasparq.ai/contact">website</a>. We would be delighted to hear from you!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b6653b17be6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/levenshtein-distance-as-a-remedy-for-sequential-data-4b6653b17be6">Levenshtein distance as a remedy for sequential data</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[War stories from using Reinforcement Learning for Dynamic Pricing]]></title>
            <link>https://medium.com/datasparq-technology/war-stories-from-using-reinforcement-learning-for-dynamic-pricing-5f7fa3054446?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/5f7fa3054446</guid>
            <category><![CDATA[ai-for-business]]></category>
            <category><![CDATA[primer]]></category>
            <category><![CDATA[practical-tips]]></category>
            <category><![CDATA[dynamic-pricing]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Jeremy Bradley]]></dc:creator>
            <pubDate>Tue, 21 Dec 2021 12:25:20 GMT</pubDate>
            <atom:updated>2021-12-21T12:27:11.361Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Search for something that works and then remember it. Search and memory is the essence of Reinforcement Learning —<strong> Andrew Barto</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O45BsOLAKxpp8kfi.jpg" /><figcaption>Found it! Inflatable decoy tanks were deployed on the south coast of England during WWII to fool the enemy into believing that the real D-Day landings were taking place around Calais</figcaption></figure><p>As a Data Science team, we were lucky enough to get into Reinforcement Learning [1] or RL a couple of years ago when we started looking at optimal pricing for car parking spaces at an international airport [2].</p><p><strong>Airports and Dynamic Pricing</strong></p><p>It’s an interesting problem, you have a fixed resource, the car park spaces themselves, and you have customer demand which varies over time — largely with demand for airline travel. You also give people the ability to book spaces well ahead of time. What price do you set?</p><figure><img alt="An airport car park with aeroplane flying overhead" src="https://cdn-images-1.medium.com/max/1024/0*gxFrGul2iXAqrieH.jpg" /><figcaption>Airport car parks presents a challenge for optimal pricing</figcaption></figure><p>Imagine you run such a car park and have the power to set prices. If you wait a couple of days, you might get a higher price from another customer for the same date. If you lower your price for that date because you haven’t sold the spaces and the date is approaching, your customers might quickly learn that waiting is the best policy and not book when they first check the car park website.</p><p>Furthermore, customers that book for a weekend might well be families or tourists looking for a quick break who are perhaps booking with a low cost airline. If they’ve paid £67 for a return trip to Majorca, they’re likely to think that £75 for a weekend parking spot is a little on the high side. Whereas your business traveller might think that is entirely reasonable or just not an expense they’ll be picking up at the end of the day.</p><p>The ability to learn what price works for a given demand, for a given day of the year, with a given booking lead time is a good example of dynamic pricing.</p><p>Dynamic pricing is a growing area of data science and a long-standing problem in operations research and one that is enjoying much more attention as marketplaces themselves become more dynamic.</p><p>What makes these problems amenable to reinforcement learning is vast event data sets around customer interactions. This gives an RL tool the ability to deduce the demand level (as well as, potentially, need and urgency) for a particular cohort of customers and thus find a customer specific price that will lead to a well used resource.</p><p><strong>Why dynamic pricing?</strong></p><p>Before we get into the reinforcement learning and how it can power dynamic pricing, we need to understand a few use-cases for why prices might need to change in more general settings.</p><ul><li><strong>Increased demand/reduced supply. </strong>As in any economic model, if supply does not match demand for a product then the market may be able to sustain a higher price.</li><li><strong>Expiring products.</strong> Anything that has a limited sale window is likely to attract varying a price, whether it’s a hotel room for tonight, a loaf of bread that’s about to reach its expiry date, or a plane ticket on a flight. None of these can be sold after the window has passed.</li><li><strong>Seasonality</strong>. Some products are much more desired at certain times of day or times of year. Some may not be available at all at the wrong time. Christmas decorations typically get discounted heavily after Christmas as retailers don’t want to stock them for the whole subsequent year waiting for them to sell.</li><li><strong>Cost of production/supply.</strong> Some products cost more to produce on demand than on the normal production run. For example Brompton, the folding bike manufacturer based in London, typically has a 5 week lead time for a normal folding bike but that can be reduced to 5 days with tracked delivery if you are prepared to pay a premium for a model. The implication is that there is a reason to charge a varying price more closely linked to the cost of production (and possibly the customer’s willingness to pay for a quick delivery — see below).</li><li><strong>Price by customer.</strong> Possibly the most contentious reason to charge a dynamic price — certainly in mass retail — is the act of setting a different price for the same product to different customers. Sometimes this is veneered over by selling in different markets, e.g. electric cars in the US versus the EU market. But within a market, it may be desirable to identify customers who are time-limited and are not looking at competitor products, and may be willing to pay more for a standard product.</li></ul><p>Varying a price to reflect super-high demand or capture a high need customer is not always the right thing to do. For instance, airlines charging thousands of pounds for seats on the last flight that makes it home before an enforced quarantine period starts or a taxi company surging prices in the face of a civil emergency — can lead to considerable long-lasting reputational damage. Important to consider— is it the ethical thing to do?</p><p><strong>Reinforcement Learning and what to watch out for!</strong></p><p>We’ve been working on a dynamic pricing project for the last few months. It’s a market-place environment where customers are exposed to a set of agents offering a service and we have to work out what price to set. The marketplace takes a cut of the fee paid and the service providers get the rest.</p><p>Basic features:</p><ol><li>Customer demand is seasonal and varies dramatically across the day — it is also price sensitive.</li><li>The service providers are a limiting resource — if they’re working for a customer they can’t be doing other jobs for other customers.</li><li>The higher the price, the more the service providers want to offer the service and the quicker a match is made.</li><li>The lower the price, the better for the customer — of course — but also the longer they may be kept waiting for a match with a possible service provider and indeed they may get no match at all.</li></ol><p>How do you set prices in such a setting so that you offer a fair price to the service providers and yet don’t fleece the customers. Well you can use reinforcement learning. We don’t know the price that is appropriate but we can set one and see what impact it has on the market — then we can learn and in the next epoch adjust accordingly.</p><blockquote>Reinforcement learning at its root, is just memoized context-sensitive search. You do some operation and then you remember the result and then the next time you have to do that, you look it up — <strong>Rich Sutton</strong></blockquote><p>If you read any reinforcement learning text book, you’ll find a diagram very much like this describing the basic principles. In this instance, the agent is the marketplace, the action is the ability to set a price and offer it to the customer, the state is the state of the marketplace (I know that’s self referential but we’ll revisit that) and the reward is a measure of success from having made a successful match between customer and service provider. Again we’ll come back to this because it’s not straightforward and it will drive the whole optimisation of the system. There can be a choice between many possible actions reflecting the possible prices that may be offered in the marketplace. Each action may lead to a distinct change of state and a different reward associated with that change.</p><figure><img alt="A feedback loop depicting an agent taking an action, and reacting to the reward and change of state from the environment that consumed the action." src="https://cdn-images-1.medium.com/max/984/1*nQBVz20wyXh1eezFG6cVMQ.png" /><figcaption>The reinforcement learning loop. CC by-SA 4.0 Jeremy Bradley</figcaption></figure><p>So what does the Agent do — well the marketplace learns from successful matches and unsuccessful matches and adjusts the price (or not) accordingly as each new opportunity to make a match comes round. There are maybe 20,000 transactions a day and prices are set in near real time.</p><p>I think the three most important aspects to doing this kind of activity with reinforcement learning can be summarised with:</p><p><strong>What’s in a state?</strong></p><p><strong>Quick primer</strong>: Reinforcement learning comes out of a branch of mathematics called dynamic programming which solves an underlying Markov Decision Process for a value function. Deriving a value function gives you a policy which in turn tells you which actions you should take in a given state.</p><p>That was lots of jargon so let’s break that down— a <strong>value function</strong> computes the value of each state of your system under a given action policy; knowing which states are high value is important when discovering a better policy. A<strong> policy</strong> is a pre-defined strategy for choosing which action to take from a given state, ideally to maximise your accumulated lifetime reward. A <strong>Markov Decision Process</strong> or <strong>MDP</strong> is a state transition system which defines the states, actions, rewards and next states for any system.</p><figure><img alt="A Markov Decision Process probabilistic transition system depicting states in green and actions in brown. Two of the actions result in distinct rewards of -1 and +5." src="https://cdn-images-1.medium.com/max/1024/0*umbiYLQFjQxM0tQ6.png" /><figcaption>A simple 3-state MDP with actions in brown and rewards +5 and -1 marked. CC by-SA 4.0 Waldo Alvarez</figcaption></figure><p>What’s this got to do with states? If the underlying mathematical model (MDP) has too many states, then the dynamic programming algorithm will take forever to converge (or a very long time anyway). i.e. you’ll never get a solution and you won’t know what action to take — or price to set in this case — to improve your overall lifetime reward.</p><p>It is very easy to inadvertently give yourself an enormous state space. We have a marketplace that spans 850 locations across the world and we want to set prices every 15 minutes, we have 20 levels of customer demand and 20 levels of service provider supply, and 20 possible price bands. That’s a little over 4.5 billion states right there!</p><p>Surprisingly this is not necessarily a disaster as you can represent your state space approximately using a deep learning neural network of some other ML model but it definitely adds to the complexity as we’ll see when we try to learn with it.</p><p><strong>What does success look like?</strong></p><p>Success is defined by your reward. You might think that success is easy to specify. Why not choose revenue or profit and watch the dollars roll in. Well yes and no. Let’s imagine we used revenue in our marketplace.</p><p>If I have some very rich customers, then I can set my prices very high and I may initially even make a lot of revenue. However, I have inadvertently made my market very brittle and subject to the whims of a few well-off customers who are now dominating the market. Every other customer is priced out and will probably never come back.</p><p>So let’s lower the prices so that I open myself up to more of my customer base. The prices are still high. The revenue is improved because the customer base is larger, but there are service providers who are now turning down small jobs that are too low in value in the hope that a higher value job will come along shortly. Many customers with smaller jobs are left waiting for an unacceptable period of time before they get a price match. But that is ignored by the reward metric as it only cares about revenue in this case.</p><p>Again this is a short term win. Those customers left hanging around probably never return to your marketplace even when they have larger jobs to offer. In the end, low customer satisfaction kills your business.</p><p>What we found is that a hybrid reward metric which captures both financial success and customer satisfaction will allow you to find a sweet-spot in your problem — keep the service providers happy with a decent revenue while keeping the customers happy with a good performance metric.</p><p><strong>Can it learn?</strong></p><p>Reinforcement learning works best when it learns as you might expect. Successful learning comes from convergence of the algorithm and gives you a consistent result for the value of being in a particular state.</p><p>If you’re using an explicit state space model (sometimes called tabular learning, so not a neural network) you need enough training data or learning episodes to visit each state many thousands of times to get converged learning. (Converged learning refers to the model having a consistent picture of what value the most important states and actions have).</p><p>Now we know why keeping track of the state space is so crucial. If you’re using tabular learning which is simple and has many benefits you have to have a pre-training set that is 100x or 1000x the number of states in the state space or at least the number of states you are likely to visit. Alternatively, if your model is learning from scratch, your learning feedback loop needs to visit all of those states 100s or 1000s of times to get converged learning.</p><p>If you’re using an approximate state deep learning model for very large state spaces, you won’t be surprised to learn that many gigabytes of training data are needed to train that model effectively.</p><p>Certainly, if your training data only visits each state once or twice, your learning is going to be very poor and your recommended prices or actions are not going to be great!</p><p><strong>Wrapping up</strong></p><p>I’ve really only touched upon reinforcement learning as a mechanism for learning optimal prices. There’s lots of benefit to getting it right — an algorithm that can learn autonomously and present the right price for even new unseen scenarios; an algorithm than can even look after customer satisfaction if the correct reward structures are put in place to guide learning.</p><p>There are many ways you can get yourself in a tangle. But keeping the state space at a reasonable size that can match the available training data you have (or are likely to encounter during learning) is a great start and defining an appropriate reward metric will help you drive your learning in the right direction while not making embarrassing decisions along the way.</p><p>This introduction is still only the tip of the iceberg but it reflects some of the key decisions we had to take when taking reinforcement learning from the text book [1] (which is quite excellent by the way!) to a real-life project. Other significant considerations include:</p><ul><li><em>exploration versus exploitation</em> — at some point you’ll need to take a suboptimal decision to see whether it leads to a better policy that was previously unknown. This can trigger some interesting conversations with stakeholders!</li><li><em>discount factors </em>— how much do you value reward that can’t be realised for weeks or months versus reward you can access today?</li><li><em>learning rates</em> — how quickly should your learning system adapt to new evidence or is your data quite noisy, thus meaning you don’t want it to adapt to a blip by mistake.</li><li><em>algorithm </em>— where to start; tabular or approximate representation, Expected-SARSA or Q-learning. There are many techniques and pros and cons to each of these approaches but in general, start simple and build up complexity as you need it is good advice.</li><li><em>off-policy versus on-policy learning</em> — is your system able to learn as it discovers new states or is it going to have to be pre-trained off-line? Is your system able to implement the decisions it takes or is it only recommending the decisions to an operator who can decide to adopt or decline them? All these factors influence whether you need to use an on- or off-policy approach.</li><li><em>time-varying rewards and demand —</em> probably the hardest issue of all in reinforcement learning is time-heterogeneity. Your environment may change over time and the reward you see one day may differ the next day for the same action. If this means a change in the statistical properties of your system (rather than just noise or differences in samples from the same distribution), then your RL system needs to know to adapt to that. Censored memory and rapid learning rates are the order of the day here.</li></ul><p>All of these features need addressing at one point or other in practice. <a href="/service/mailto:jeremy.bradley@datasparq.ai">Leave me a comment</a> if you’d like to see a Part II on some of these more technical aspects of practical implementation of reinforcement learning.</p><p><strong>And finally…</strong></p><p>If you are interested in joining us at <a href="http://www.datasparq.ai/">Datasparq</a> for exciting projects like this or just finding out how your company can benefit from our Reinforcement Learning, Data Science and Engineering expertise, <a href="/service/mailto: info@datasparq.ai">please get in touch</a>. We’ll be delighted to hear from you.</p><p><strong>References</strong></p><p>[1] Richard S. Sutton and Andrew G. Barto. <a href="https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf">Reinforcement Learning, An Introduction</a>. 2nd ed. MIT Press, 2018.</p><p>[2] Andreas Papayiannis, Paul V. Johnson, Dmitry Yumashev, Peter Duck. <a href="https://doi.org/10.1093/imaman/dpy015">Revenue management of airport car parks in continuous time</a>. IMA Journal of Management Mathematics. Jan 2019.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f7fa3054446" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/war-stories-from-using-reinforcement-learning-for-dynamic-pricing-5f7fa3054446">War stories from using Reinforcement Learning for Dynamic Pricing</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Can a Software Engineer Learn From a Rocket Engineer? Quite A Lot Actually.]]></title>
            <link>https://medium.com/datasparq-technology/elon-musks-5-steps-of-engineering-applied-to-software-a7963fef9887?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/a7963fef9887</guid>
            <category><![CDATA[elon-musk]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Matt Simmons]]></dc:creator>
            <pubDate>Thu, 02 Sep 2021 15:18:50 GMT</pubDate>
            <atom:updated>2021-09-06T14:03:22.885Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine you’re building thousands of rocket engines and spending billions of dollars on a quest to make humanity a multi-planetary species. If your manufacturing process isn’t as efficient as possible, you’ll run out of money before you get there. You probably need to optimise this process like crazy right? So why don’t we see if we can optimise our software in the same way…</p><p><em>(please ignore the fact that nobody needs to manufacture software, that’s why it’s called software not hardware, just bear with me)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qXCFgcRXosxgEthX" /><figcaption>Photo by <a href="https://unsplash.com/@spacex?utm_source=medium&amp;utm_medium=referral">SpaceX</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In a recent interview, Elon Musk talked about how manufacturing was underrated and was the most difficult problem SpaceX had to solve.</p><blockquote>“10,000% more work goes into the production system than the thing itself.”</blockquote><blockquote>“The amount of effort that goes into the design rounds down to zero relative to the amount of effort that goes into the manufacturing system.”</blockquote><p>He went on to describe 5 steps used to optimise engineering for manufacturing, which emerged from the many successes and mistakes of rocket programs and Teslas in the past:</p><h4><strong>1. Make the requirements less dumb</strong></h4><p>The requirements are definitely dumb; it does not matter who gave them to you. Therefore, requirements get challenged, which allows for out-of-the-box thinking (see Cybertruck).</p><blockquote>“Everyone’s wrong. No matter who you are, everyone is wrong some of the time.”</blockquote><blockquote>“All designs are wrong, it’s just a matter of how wrong.”</blockquote><p>He also states that specific <em>people</em> should own requirements, not departments, so that they can be questioned much quicker.</p><h4><strong>2. Try very hard to delete the part or process</strong></h4><p>You can’t afford to add a part ‘just in case’. Musk encourages deleting as many parts as much as possible, even if you end up adding some back later.</p><blockquote><em>“If you’re not occasionally adding things back in, you’re not deleting enough.”</em></blockquote><h4><strong>3. Simplify and optimise the design</strong></h4><p>This step comes third, not first, for good reason. The critical point of this whole process is to not do this step until you’re absolutely sure it’s needed before you start wasting your time.</p><blockquote>“The most common error of a smart engineer is to optimize something that should not exist.”</blockquote><p>Elon stressed the importance of an engineer understanding the whole system to avoid optimising the wrong thing. He revealed that SpaceX engineers had made the mistake of putting enormous amounts of effort into reducing the weight of engine components, but not the weight of unused fuel, despite them being equivalent.</p><h4><strong>4. Accelerate cycle time</strong></h4><p>Here’s where you start to find issues, and have a chance to fix them before it’s too late. It’s like learning to run before you run a marathon.</p><blockquote>“You’re moving too slowly, go faster! But don’t go faster until you’ve worked on the other three things first.”</blockquote><h4><strong>5. Automate</strong></h4><p>Why is this separate from the fourth step? Because this assumes you wait until any and all issues have been found before removing all humans from the process.</p><p>It would be hugely costly to try to automate something that was overly complicated or wasn’t needed.</p><p>This is also the point where you should aim to remove in-process testing for parts that can be produced reliably. Musk states that a tester could be creating a choke point, and will always have <em>some</em> have false negatives.</p><blockquote>“If things are getting to end-of-line testing, and are passing, then you don’t need to do in-process testing.”</blockquote><p>We can summarise this process as the following:</p><p><strong>Design → Question → Simplify → Optimise → Accelerate → Automate</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jmz58whKh3ky9qS3" /><figcaption>Photo by <a href="https://unsplash.com/@spacex?utm_source=medium&amp;utm_medium=referral">SpaceX</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>How does this relate to software?</h3><p>So let’s say instead of building thousands of rockets, you’re developing the next big app/website/API. It’s a huge task. If you do everything right you’ll be bought by Google within 5 years, else you’ll take 15 years, never get feature-complete, spend all your money, and get overtaken by somebody else. How does this situation compare to SpaceX’s?</p><p>In terms of the processes and ways of thinking, there are a lot of parallels: it’s easy to get carried away solving the engineering problems and ignoring the bigger picture. Engineers tend to want to add loads of functionality that isn’t essential. They like optimise for speed and efficiency without being required to. They stick to a set of best practices and don’t usually question them.</p><h4>Manufacturing ≈ Going Live</h4><p>In the design stage your code only needs to work once. When you go live you’re running the same code billions of times, on millions of different computers, for years or decades. There’s a lot that could go badly: will you need constant patches, or tiny changes for every new use case? Will you need to spend forever teaching people how to use it? Will you be able to keep up with demand and scale? Only by solving these issues can you hope to actually achieve anything with your software.</p><p>Like with rockets, this is an underrated part of programming. An inexperienced software developer might assume their cobbled together script can be used in production with no changes. This is like building a rocket engine in your garage from spare parts and deciding you’re going to build 100,000 more in the exact same way.</p><h4><strong>Part = Script, Service, or Program</strong></h4><p>Cutting out an entire service from your app (e.g. deleting an email server because you can just use an API) is undeniably a great way to reduce complexity and go live faster.</p><h4><strong>Cost = Cost</strong>?</h4><p>For software, material cost is much lower, personnel cost is much higher, and there are additional computing costs. People often worry about their compute/network/storage bill, but the salaries of the people fixing bugs all day long are likely to be larger.</p><p>There’s also the opportunity cost of complexity. If a low priority feature requires a day in total to develop, deploy, debug, document, etc. over the course of a month, then you’ve missed the opportunity to start a killer feature, or present your product at a conference, or have a much needed office party!</p><h4><strong>Weight = Complexity</strong></h4><p>For rockets this doesn’t just mean weight of the part, it’s also how much thrust you can get for that weight. Adding complexity to a piece of software means more complex documentation, technical debt, bugs, deployment, etc., all of which increase costs but more importantly, slow you down and kill your momentum.</p><p>Reducing complexity allows you to go further with the same amount of resource.</p><h4>Requirement = Requirement</h4><p>Requirements often don’t get questioned, and they rarely have an individual owner. It’s far too easy to make assumptions about how software will be used and not seek out the opinion of an actual potential customer.</p><h4>Feature = Feature</h4><p>We’re all familiar with dubious feature requests. The Agile process helps to prevent low priority features from being worked on but doesn’t stop us from adding unnecessary complexity.</p><h4><strong>In-process testing = Unit tests</strong></h4><p>This one has really made me think. Elon’s right, we don’t need most of these unit tests. We’ve got hundreds that pass every time, and they sometimes get false negatives which take time to investigate. We could probably replace 90% of them with a single integration test.</p><p><em>(I won’t suggest getting rid of them though, just running them less often)</em></p><h4><strong>Colony on Mars = __________</strong></h4><p>It helps to know what you’re working towards. A clear and ambitious goal has certainly helped keep SpaceX focused on what’s important. What’s the ultimate end goal of your project? What will make you say ‘Mission accomplished’?</p><p>The optimisation of the productionisation process is vital for any software company. There’s a huge impact on time but it’s also the difference between being proud of your accomplishments, and being embarrassed. We’ve all seen projects that are being carried by one developer who knows how to fix the many reoccurring issues, and has given up hope of ever working on something interesting again.</p><p>I guarantee no app ever went live without a single issue, so there’s plenty of room to improve.</p><h3>What Should We Be Doing Differently?</h3><p>With all the parallels described above, I think we can take some learnings from the five steps. Looking back at some of my own projects that were difficult to get live, I can now see that we skipped some steps or did them in the wrong order, just as Elon admitted Tesla had done in the past. Based on this, here’s what I think we should all be doing differently:</p><ul><li>Challenge all requirements, no matter how small. A common example would be “We’ve got streaming data therefore we need a streaming data pipeline” but in reality there’s no requirement for data to be processed in realtime. Micro-batch processing is much simpler and cheaper to build.</li><li>Pause once you have a working concept to ask which processes aren’t needed, and just <strong>delete them</strong>. Don’t spend time on a feature nobody cares about just because you want everything ‘production ready’.</li><li>Recognise the true <strong>cost of complexity</strong>. Any ‘part’ in your app that could be replaced with something much simpler must also be deleted.</li><li>Don’t optimise for the wrong thing. Is speed your priority or is it lowering cost? Should you spend your time improving features or adding more? Find a <strong>real person</strong> in the business who can explain what the priority is and why.</li><li><strong>Don’t optimise too soon</strong>. Don’t invent a framework before trying something the simple way first. Don’t define a data model when the data isn’t finalised. Be flexible to change throughout.</li><li><em>Accelerate</em> your code before you automate. You’ll find which parts require further optimisation, and uncover problems that would be <strong>impossible to fix after going live</strong>.</li></ul><p>Most importantly, appreciate just how challenging it is to go from a great concept to a live app for a mass audience. Your approach to productionisation could determine your success or failure. Keep complexity low and momentum high, else you’ll never get to Mars.</p><p>The full text of the interview referenced can be found here: <a href="https://everydayastronaut.com/starbase-tour-and-interview-with-elon-musk/">https://everydayastronaut.com/starbase-tour-and-interview-with-elon-musk/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7963fef9887" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/elon-musks-5-steps-of-engineering-applied-to-software-a7963fef9887">What Can a Software Engineer Learn From a Rocket Engineer? Quite A Lot Actually.</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science projects at pace]]></title>
            <link>https://medium.com/datasparq-technology/data-science-projects-at-pace-1b380ff13ea5?source=rss----64ba35c1239a---4</link>
            <guid isPermaLink="false">https://medium.com/p/1b380ff13ea5</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-project]]></category>
            <dc:creator><![CDATA[James Watkinson]]></dc:creator>
            <pubDate>Tue, 10 Aug 2021 15:49:53 GMT</pubDate>
            <atom:updated>2021-08-10T15:53:55.156Z</atom:updated>
            <content:encoded><![CDATA[<p>A large part of our business is making valuable data science products a reality. I’m going to share some of what we have learnt on how to do this quickly whilst maintaining the highest quality.</p><p><strong>Pandas alone won’t save you</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7mU7TnQZRcHy51xc" /><figcaption>Photo by <a href="https://unsplash.com/@wuyyyying?utm_source=medium&amp;utm_medium=referral">Ying Wu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The first mistake is to believe that every data science project is <strong>only</strong> a data science problem. Multiple different roles have an important part to play.</p><p>The best and most efficient teams I have worked in are a combination of <strong>Data Science + Product</strong> + <strong>Engineering </strong>roles. That will not come as a surprise, but what makes a difference is ensuring that these roles have a part to play throughout the project. Unless the project is heavily standardised or a repeat of another — there is virtually no chance of finishing on time or to an acceptable standard by passing (or more commonly “throwing”) from one team to another.</p><p>A common example of where early team integration really shines is during <strong>feature generation</strong>.</p><ul><li>Data features will often need to be re-run, re-configured and re-designed (multiple times)</li><li>New data sources (with valuable additional signal) need to be ingested and cleaned</li><li>Tests on various potential models will be repeated and reviewed when features change</li></ul><p>A role diverse team will create repeatable processes from the outset and adhere to the boundaries of company infrastructure, ensuring that no costly logic “re-writes” are required when moving from test to live datasets. The benefits are often greater than the sum of parts — we develop data models supporting current project requirements and perhaps those of others too. This is just one of the areas where we have managed to cut weeks, if not months, out of projects without effecting quality.</p><p><strong>MLOps products help, collaboration is better</strong></p><p>Secondly, do not let anyone tell you that an <strong>MLOps product</strong> (Kubeflow, MLflow, Cloud provided, etc)<strong>+ Data Science</strong> are going to be all the tools your team will ever need.</p><p>Elements of Data Science projects are similar, especially in operations (monitoring, testing, processing, development environments etc). Although there are great open source and cloud based tools available, parts of your projects will be unique to your business.</p><p>If your business is software based this might be second nature. If not, having a strategy to <strong>share both</strong> <strong>code and documentation</strong> between projects is a fantastic accelerator. Make sure you bring relevant teams together to discuss lightweight governance, location and features of existing reusable parts.</p><p>Here is an example of our serverless components, we use these across several projects to reduce maintenance time and improve quality: <a href="https://datasparq.ai/elements-of-engineering">https://datasparq.ai/elements-of-engineering</a></p><p><strong>Think about the end at the beginning</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4PFHtzQVS7nvozbt" /><figcaption>Photo by <a href="https://unsplash.com/@sigmund?utm_source=medium&amp;utm_medium=referral">Sigmund</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Finally, those with experience will know that <strong>finishing a data science project is as difficult as starting or building one</strong>. It turns out that APIs are not designed for human consumption. And, no team is likely to be happy manually running data processes, creating graphs or having similar conversations forever.</p><p>It is also possible that no two Data Science projects you run have the same results interface. A handful of examples we have completed include: APIs, web / mobile applications, static files, databases (all different kinds), messaging service notifications and more.</p><p>Finishing a project means delivering something that can be used in operations, sustainably. It’s well worth taking the time early on in a project to make sure your team either <strong>has or can acquire the</strong> <strong>skills needed to finish the project</strong> as they might be a little different to starting and developing. This is where product-oriented awareness and design helps round a project off in a way that drives impact for the customer.</p><p><strong>Final thoughts</strong></p><p>Do not let any of the above put you off! Data Science projects are extremely rewarding, fantastic investments and really enjoyable to work on. The extra effort invested in early team integration, code collaboration and building tricky results interfaces is often rewarded by increased adoption and usage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b380ff13ea5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasparq-technology/data-science-projects-at-pace-1b380ff13ea5">Data Science projects at pace</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>