<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dan Baker on Medium]]></title>
        <description><![CDATA[Stories by Dan Baker on Medium]]></description>
        <link>https://medium.com/@danielbaker-190?source=rss-e5e839f0817c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yQhx3beQpH_D9K2nNmTFzQ.png</url>
            <title>Stories by Dan Baker on Medium</title>
            <link>https://medium.com/@danielbaker-190?source=rss-e5e839f0817c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:07:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danielbaker-190/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Is the Internet an alien organism?]]></title>
            <link>https://medium.com/predict/is-the-internet-an-alien-organism-8d5d5c5f4ad8?source=rss-e5e839f0817c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d5d5c5f4ad8</guid>
            <category><![CDATA[science-fiction]]></category>
            <category><![CDATA[humour]]></category>
            <category><![CDATA[future]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Dan Baker]]></dc:creator>
            <pubDate>Sat, 10 Feb 2024 05:00:48 GMT</pubDate>
            <atom:updated>2024-02-10T05:00:48.597Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*cKXi6KthxOYnyvJw" /><figcaption>Photo by <a href="https://www.dailymail.co.uk/sciencetech/article-2108621/The-tiny-gardeners-Minute-leafcutter-ants-tend-plots-fungi--use-bacteria-compost-leaves.html">Mark Bowler</a>, CC</figcaption></figure><h4>And are we humans its symbionts; cultivating it like the African leaf-cutter ants and their fungal partner?</h4><p>I know the title sounds crazy, but ask yourself: what really <em>is </em>‘the Internet’?</p><p>Words like ‘tool’ or ‘network’ understate the Internet on a fairly dramatic scale. It’s arguably the largest and most complex structure we’ve ever engineered. Billions of us spend our lives working on it, within it, and around it. We use it to educate ourselves and our children. It allows us to access and share the sum of our species’ knowledge and instantaneously communicate with one another (borderlining on telepathy). Our entire economy is interweaving with the Internet in a way that is as transformative as the system of money or the printing press.</p><p>Similar to the challenges we have in defining ‘life’ I don’t think we can adequately define what the Internet has truly become. This essay is an attempt to explore what the Internet is, how it may be fundamentally changing our species, and to what strange future this partnership might be leading us.</p><h3>Introduction</h3><p>I’ve been thinking a great deal lately about the evolution of the Internet and its similarities to what we observe in the natural world. Are there insights to be unravelled here and might they be vaguely amusing; possibly even useful?</p><p>As a student of both computer and biological sciences, I can’t help but notice a few parallels across these two disciplines. It remains true, however, I may have just watched too much Star Trek. 👾</p><p>What is ‘the Internet’, really?</p><p>Could we assert that the Internet is starting to approximate an artificial (or even alien) living organism? And are we humans, its creators, co-evolving as its symbionts, like the African leaf-cutter ants blindly tending to their underground fungus gardens in return for nourishment?</p><p>Bizarre as it sounds, in many ways there are similarities between what we call ‘the Internet’ and biological life.</p><p>Like all life, the Internet needs a steady flow of energy (electrons) to survive, without which, it will die. It converts energy to other forms and produces waste products (such as heat). It has a measurable ‘biomass’; the sum of all its interconnected components. It exhibits complex behaviour and highly specialised functions. It has living and non-living states. It grows in physical size over time (although it does not reproduce). It has multiple systems of organisation akin to what we observe in biological life. For example, the Internet has physical building blocks (silicon, metal, capacitors and diodes) which assemble into cell-like systems with specific functions (microchips, transistors, circuit boards), which in turn are assembled into even higher order systems like organs (datacenters, servers, computers, smart phones, cell towers) akin to multicellular organisms.</p><p>The Internet evolves. Our species evolves through natural selection and, as we are its creators, the Internet could be thought to evolve through us in a kind of second-order natural selection.</p><p>What about a nervous system? Does the Internet not bear a startling similarity to a decentralised nervous system, with its trillions of interconnected nodes and continuous chatter of nerve impulses (IP packets) zipping around its ‘body’? And, like complex living systems, can it only be truly appreciated while alive? Without electricity, like a brain without oxygenated blood, the Internet will rapidly cease to function and remain in a lifeless, unintelligible state. Studying the Internet without electrical power is like studying a dead brain: we can see the underlying components in limited detail (like observing dead cells under a microscope), but witnessing a living brain perform in life is breath-taking by comparison. So it is true of the ‘living’ Internet; the unconscious hive mind of our species.</p><p>If we follow the yellow brick road down this psychedelic biological metaphor, is there anything we can learn from this science fiction way of thinking about the Internet?</p><p>I’d like to do a thought experiment with you. In the following sections I’ll attempt to explore the Internet through the lens of biology, as an incompetent drunk astro-biologist might in the study of life on an alien planet.</p><p>Let’s apply what we have learned from life on earth to ‘the Internet’. We’ll start with the physical parts; and think about the ‘biomass’ of the Internet. Then we’ll dive further into other life processes like growth and reproduction, respiration, metabolism (energy transfer), nervous systems, and evolution. We’ll then attempt to define what type of organism the Internet is most-like based on what we know of life on earth. Finally, we’ll take a look at our place in this process and our relationship with the Internet.</p><h3>Chapter 1 — Physical form (biomass)</h3><p>Like a living organism, the Internet could be thought to have a measurable ‘biomass’; the sum of all interconnected devices and their physical mass. Before we can add this up, we first must establish the boundaries. How do we define where the Internet ends?</p><p>In living systems at the cellular level, this is usually a distinguishable semi-permeable membrane (a cell wall) that encases the contents of a cell and allows it to maintain particular chemical conditions that differ to the outside environment (called homeostasis). This is how we understand single-celled organisms like bacteria and Amoeba as separate to their environment. For multicelluar organisms like plants and animals that we are more familiar with, these are defined as a particular <em>organisation</em> of cells. The macro organism is bounded by the point at which its outermost cells touch the surrounding environment. So for us humans, our boundary to the environment is mainly our hair, teeth, nails, mucosal membranes, and skin cells. In life, generally speaking we can distinguish biological organisms through genetic information (DNA and RNA contained in every cell), and identify which cells belong to a particular organism.</p><p>What are the boundaries for the Internet if we were to think of it in terms of cell theory?</p><p>We don’t have genetic information to distinguish ‘cells’ of the Internet, but we do have a fundamental atomic unit or property to guide us. It would be logical to base our boundary criteria as the fundamental unit of <em>communication</em> within the Internet: the Internet Protocol (IP) packet. In a similar way as we might measure the bounds of an electricity network by the fundamental unit of the <em>electron</em>.</p><p>To measure the mass of an electricity grid, we would ‘follow the electrons’ from their point of origin to the furthest points they can reach. The mass of an electricity grid, therefore, would be the sum of all the generation, transmission and distribution systems. This would include the mass of all power stations (coal, gas, nuclear, oil, geothermal, hydro, wind, solar systems), the substations containing transformers that step the voltage up and down, the thousands of kilometers of high voltage transmission lines, the millions of poll top and underground lines that distribute lower voltage power to dwellings and buildings, and the sum of all wiring inside buildings that conduct the electrons to the power points for devices to connect to the network. Electricity networks are often massive and extend past many country borders in continents where power can be purchased and sold between countries. They are all ultimately islands though, as there is no global electricity grid for which all local networks connect to. The USA does not supply electricity directly to Europe, for example.</p><p>In assessing the mass of the Internet, we can apply a similar method and ‘follow the IP packets’ from their potential point of origin to termination, geographically. Almost all nodes on the Internet can send and receive IP packets, and because the Internet is globally interconnected, the Internet is far larger geographically than the world’s largest electricity network. Your phone can send packets to any device on the Internet, anywhere in the world. Furthermore, the Internet is not just hard-wired, like the electricity network, what about wireless aspects? We send data packets through medium range radio networks (3G/4G cellular networks) and short range radio networks like WiFi and bluetooth. We also send data from the Earth’s surface to satellites orbiting our planet, and back. Whilst these wireless communications standards are not all strictly using the IP protocol (bluetooth uses a different protocol called L2CAP), they are still transmitting the same <em>data</em> <em>payload</em>.</p><p>To illustrate this point, let’s take the example of streaming spotify on your phone when at home on WiFi, using bluetooth earbuds. Spotify holds over 1.5 billion audio files (3 petabytes) and this binary data is physically stored on drives that reside on Google cloud datacenters (Spotify migrated their data from AWS to Google in 2016). When you stream a Spotify song to your phone, the binary data stored as a compressed mp3 (or similar) file is broken up and transmitted via a synchronous TCP/IP connection from a Google datacenter (most likely the one nearest to your geographical location, or they are locally cached on a content delivery network near you) to your wireless modem-router at home. The packets are routed from the Datacenter through telecommunications infrastructure to your Internet Service Provider (ISP). From there they make their way through the local telecommunications (usually fibre or copper) all the way to your dwelling and into your modem and wireless WiFi router. This is the point at which the data is translated from the TCP/IP format to a different standard to be transmitted by high frequency radio (2.4GHz or 5GHz) based on the WiFi communications standard.</p><p>Your WiFi router converts the spotify TCP/IP datastream it receives into the WiFi radio format, and this is transmitted wirelessly to your phone. The data stream is then translated once again to another format: the bluetooth wireless protocol. This is a special protocol designed for short range wireless communication (designed for low power consumption more suitable to battery operated devices). The Spotify data now carries onward on its journey, and is transmitted from your phone to the surrounding environment, where your bluetooth earbuds receive it and convert the digital signal to an analogue wave for your ear.</p><p>Where in the above scenario does the Internet end? Strictly speaking, the IP protocol stopped being used when the data reached your modem/router. But the data payload continued via wireless transformation all the way to your phone and then earbuds. It starts to get murky with all these points of translation. So perhaps we can broaden our boundary conditions for defining the Internet terminus to be the reach of the data payload itself, and not just the use of the IP protocol. I.e. As far as the data payload can reach (from the google data center to your bluetooth earbud) might be a more appropriate boundary condition for the Internet.</p><p>So if we were to include the wireless parts of the Internet as well, the Internet not only extends across most of the landmass of our planet, it even goes into space. By this shaky definition, the biomass of the Internet therefore could include all datacenters in the world (e.g. AWS, Azure, Google and many more), the network of underground telecommunication lines and subsea cables throughout the world, all data-enabled mobile phone cellular networks and their associated infrastructure, all network enabled computers connected to the Internet at instantaneous time (e.g. every laptop, smart phone, smart TV, server and desktop computer), all satellites that transmit Internet data, all other Internet-enabled devices like wearables, IOT devices, embedded devices. It’s beaming all around you right now. Anywhere your body is within range of an Internet enabled mobile phone network, or a WiFi network, the Internet is, literally, all around you; you’re bathed in it. You can reach out and touch it with the mobile phone or laptop you are holding in this very moment.</p><p>Chapter summary: the Internet is fucking massive. The Egyptian’s may have built the Pyramids, but we’ve built the Internet. I’d argue it’s the largest structure ever made and it’s increasingly a fundamental component of the artificial habitat in which many of us spend 99% of our lives. It reaches further than our electricity and water networks, and it even reaches beyond our atmosphere into space. It’s big.</p><p>Footnote: I’m aware other more qualified people have <a href="https://www.ipxo.com/blog/how-big-is-the-internet/#:~:text=Scientists%20have%20even%20attempted%20to,heavier%20than%20a%20tennis%20ball.">documented </a>the size of the Internet by other measures, such as the number of IPv4 and IPv6 addresses, or by the weight of the electrons. I’m taking some creative licence here and arguing it could also be the sum of all interconnected physical hardware that comprises the communicating Internet. Humour me.</p><h3>Chapter 2 — Living and non-living states</h3><p>Now that we have a working definition for the physical form and mass of the Internet, let’s explore whether it is alive (in some sense) or not, and what that could mean.</p><p>The functional components of the Internet (the organs if you like) are things that are familiar to us: laptops, phones, computer banks in data centers, telecommunications infrastructure like mobile cell towers. When your laptop battery runs out, you might say it is ‘dead’. But what do we really mean when talking about these machines we use as tools in our daily lives?</p><p>Is your toaster ‘alive’ only when it’s toasting a crumpet? Is your laptop only alive when it has a charged battery? Not really. We use words like alive and dead loosely. I’m not saying your milkshake maker is alive. What remains true of all our electrical machines, however, is they have two distinct states: powered and unpowered; on and off; living and non-living; alive and dead. Without electricity, they cease to function. They can be reanimated by re-powering them and, when not energised, they don’t immediately die (irreversibly cease to function) like biological organisms do unless there is a critical component (organ) failure. This is simply because our machines don’t decay at the same speed as biological systems when they cease to function. Leave a toaster for 10 years in the elements, and it will not reanimate when energised because its parts will have irreversibly rusted and decayed through natural chemical and physical aging processes. Our electrical machines are just as delicate as biological organisms, requiring constant care and protection from the environment. Like biological life, they also operate in highly specific condition ranges such as temperature, humidity, mechanical shock and vibration limits, salinity, and acidity/alkalinity.</p><p>The Internet is no different to a toaster in the sense that it is an electrical machine. It is just comprised of millions of electronic machines that make up its component parts. Just like our kitchen toaster, the Internet could be said to ‘die’ if all electrical power to its system was cut. Before long, without being re-energised, its parts would decay irreversibly. In order for this to happen, we’d have to cut power to the entire world. This hasn’t happened since the Internet was born around 1983 with the ARAPNET network that spawned the TCP/IP communication protocol. Parts of the Internet have always remained energised somewhere in the world since then, as it has grown in size over time.</p><p>Are these machines really ‘alive’ though when energised? We know they are not in quite the same way as biological life, but we also know they are alarmingly ‘life-like’ when energised; they light up and start doing things. They certainly <em>seem</em> alive to us in some sense.</p><p>Our machines extract their energy from the surrounding environment (a power network or battery), they use it to perform functions, they are inanimate without the right conditions. Cut the power to our cities, factories and dwellings and all moving parts grind to a halt. Light fades away. Radio waves and signals travelling through cables cease. All our machines become quiet and inanimate. Just because they don’t self-repair, or reproduce, or have a common ancestor, or share the same biological foundation, do they not bear interesting similarities to life?</p><p>It really comes down to our definition of living and non-living. And let’s be intellectually honest here, we still can’t define life very well. We can describe its component systems and processes, but we are continually updating our definition. We are still <a href="https://www.scientificamerican.com/article/are-viruses-alive-2004/">arguing </a>over whether viruses are alive or not. What I’m trying to say here is there are ‘edge cases’ everywhere. We currently have a working definition of life based on one planet, so let’s not fall into the trap of certainty. We only observed the first cell under a microscope <a href="https://www.sciencemuseum.org.uk/objects-and-stories/medicine/microscope#:~:text=It&#39;s%20not%20clear%20who%20invented,30%20times%20its%20normal%20size.">~400 years ago</a>, and we discovered the genetic code of life less than 70 years ago (<a href="https://en.wikipedia.org/wiki/Genetic_code">Crick &amp; Watson, 1953</a>), so let’s not profess we are experts just yet.</p><p>Is it that far a stretch to assert that highly complex and interconnected electrical machines, something as far reaching as the millions of powered machines that comprise the Internet, might be approximating an emergent kind of artificial life?</p><p>If so, it is an artificial life whose energy systems (metabolism) are based on electricity and not sugar, sunlight or oxygen. And this is where we’ll take this crack-smoking idea next.</p><h3>Chapter 3 — Energy Systems (Metabolism)</h3><p>All life we have observed on our planet has some kind of metabolic and respiration process for obtaining the energy and building blocks they need to perform core functions critical to their survival such as moving about their environment, building their bodies (growing &amp; reproducing), or assembling proteins. Proteins are the tools of biochemistry. They have specialised physical and chemical properties and include things like poison venom, hair, finger nails, spider silk, muscle fibres. I have oversimplified this and there is more to it, but at the highest level, all life processes require energy transfer.</p><p>Energy needs to be obtained in some form, and converted to another. In living systems this is often through the breaking of molecular bonds between atoms and molecules themselves. The energy in glucose, the most fundamental sugar, is harnessed by practically all bacteria, animal and plant cells alike by cleaving off a carbon atom, breaking its bond with the parent sugar molecule and releasing energy that is harnessed to perform a function. Plants use the energy contained in sunlight to split the water molecule, cleaving it apart and harnessing the energy released when the molecular bond breaks. It goes deeper than this still and we could get into electron transfer (oxidation and reduction processes) and the fundamental energy fuel of life (adenosine triphosphate) which is all fascinating stuff but probably stretching this analogy too far.</p><p>Let’s stick to something we are familiar with: ourselves. Human beings are multicellular organisms, comprised of animal cells; around 200 types. We’ve got bone cells, liver cells, heart cells, brain cells. All of them have specific functions, and bunch them all together in the right order (governed by our genetic information) and we have a human. To keep functioning, aside from the right environmental (physical and chemical conditions) like temperature, our cells need energy. They get this primarily through glucose (a sugar which is a carbohydrate) which is produced by breaking down the food we eat. Our metabolic systems are versatile though, and many of our cells can power themselves by breaking down proteins and fats as well, which follow different chemical pathways. To keep it simple, we’ll just stick to glucose.</p><p>The glucose is obtained from the food you eat, extracted in your intestines and absorbed into your bloodstream where it can be distributed to all cells in your body. The other key ingredient your metabolic processes need is oxygen; a highly reactive molecule that plays a key role in electron transfer (energy transfer). When you take a breath of air (which is ~20% oxygen gas) a tiny amount of this gas is absorbed directly into your blood stream and stored in your red blood cells (a specialised protein structure called haemoglobin) where it is used as part of the energy processes. And that’s the key stuff: oxygen and glucose suspended in your blood, delivered to your cells. Stop the flow of blood, or oxygen into the blood (i.e. stop breathing) and your cells will begin to die in minutes. Stop the flow of glucose into the blood stream (stop eating), and you will deplete the sugars suspended in your blood, then your body’s fat and protein reserves in a matter of days, and your cells will begin to die (i.e. you will have organ failure and die from starvation).</p><p>Now that we’ve done a horrific crash course in biological metabolism, let’s look at machine metabolism by comparison.</p><p>For many machines familiar to us, electricity is their lifeblood. In the same way animal cells rely on blood (a mixture of water, sugars, cells and oxygen carrying proteins) and plant cells rely on sugars and nutrients dissolved in water, all these systems need an energy transport system to power their sub-components and processes. With machines, it’s electrical and with biological life, it’s chemical.</p><p>The Internet requires a constant flow of electrical current to live. Its components share this universal form of energy and it is distributed through a complex network of conductors, reaching every last piece, from the 12V direct current feeding a computer motherboard, to the microscopic conductor channels on a microchip. Are these electrical conductor channels not unlike the vascular systems of multicellular plants and animals, delivering nutrient and energy to every last cell?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rzc3Jk8lBBCWiaQ4uOLSqA.png" /><figcaption>Above: It’s hard not to see the stunning similarities between energy transport (vascular) systems in plants (left) and animals (center) and electrical machines (right). Credit: <a href="https://unsplash.com/@onderortel?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Önder Örtel</a> on <a href="https://unsplash.com/photos/a-close-up-view-of-a-green-leaf-ifB7e-vxHBc?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a> (leaf), <a href="https://commons.wikimedia.org/wiki/File:Human_Heart_and_Circulatory_System.png">Bryan Brandenberg</a> CC 3.0 (human heart), <a href="https://unsplash.com/@bermixstudio?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Bermix Studio</a> on <a href="https://unsplash.com/photos/a-close-up-of-a-computer-chip-on-a-printed-circuit-board-tTurhPw9bwQ?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a> (chip)</figcaption></figure><p>In many ways we can marvel at the elegance of electrical power. It simplifies the energy and nerve processes dramatically in comparison to biological life. It’s all our machines need to power their processes and it can be easily converted to different forms and transmitted throughout their systems. Electricity is the blood of our machines, and conductors are their arteries and veins. They don’t need to take molecules out of their environment like oxygen or food to produce it. Instead, they depend on the steady flow of electrons. They depend on electricity and they are not capable of producing it themselves in the quantity they often need. Most of them depend on us for that, and we’ll explore our relationship with these machines in a following chapter.</p><p>Chapter summary: electrons can be thought of as the blood of machines and metallic conductors are their vascular system. Electricity provides all the nourishment they need in one elegant system. Cut it off, and they will cease to function and die.</p><h3>Chapter 4 — Nervous system</h3><p>The Internet, unmistakably, bears some interesting similarities to a nervous system. We could think of the Internet Protocol (IP) packet as an analog to a biological neuron firing in a brain and a signal being transmitted along synapses.</p><p>With over 5.3 billion active users a month (<a href="https://www.demandsage.com/internet-user-statistics/">in 2024</a>), <a href="https://www.forbes.com/advisor/business/software/website-statistics/#:~:text=There%20are%20about%201.13%20billion,are%20actively%20maintained%20and%20visited.">1.13 billion</a> indexed web pages, and 347 billion emails sent per day, traffic in the form of TCP/IP data packets are whizzing throughout the mass of the Internet at near the speed of light; streaming our Netflix content or serving our mobile apps and web browser content. The speed at which it can achieve this information highway of traffic in all directions is astonishing. Each node (such as a desktop computer) is capable of transmitting and receiving millions of IP packets per second. Each packet able to reach destinations on the other side of the world usually within around 250 milliseconds (geographical latency)</p><p>Furthermore, the Internet’s body mass is able to store and retrieve information (akin to memory) and respond to external stimuli. When you connect your phone to the internet, it responds, establishing a connection and assimilating your phone to it’s body mass. When you load a web page on your phone’s browser, a HTTP GET request is initiated by your phone, and IP packet stream is established, travelling all the way to the server hosting the website and back, transmitting the page content back to your phone. In some sense, all browser based web connections could be thought of as tactile sensory inputs to the Internet; it’s fingertips if you like. Your phone’s web browser sits on the outermost layer of the Internet allowing it to feel and respond to stimuli in the form of web requests.</p><p>The Internet also exhibits complex behaviour, such as responding to changes to environmental conditions by rerouting web traffic if a node goes down. The Internet does not, however, exhibit a primary motive or high level decision making function like we would see in animal behaviour. As such, it’s more like an unconscious brain; optimised to store, retrieve and send information across its planetary scale mass.</p><h3>Chapter 5 — Growth, motility and reproduction</h3><p>All biological life that we know of, grows and reproduces. From the tiniest prokaryotic bacterial cells we find buried in the earth’s crust, to the multicellular macro-organisms we are far more familiar with like plants, fungus and animals. A defining characteristic of all life is growth and reproduction.</p><p>Your toaster, an electrical machine, does not grow or reproduce. Like most of our machines, it cannot expand in size over time nor can it produce a copy of itself. However the Internet does grow, in a curious kind of way. Like a forest weaving its way into the landscape, seeding new trees and plants, the Internet is an ever expanding machine that grows outward into the surrounding environment as well. Except in this case, it’s intrinsically linked to us.</p><p>The Internet’s growth and body shape therefore is continuously changing and tightly coupled to human artificial environments, as it is embedded within them. We build the underlying components like microchips, circuit boards, bare metal servers and fibre optic cables, then we encase the components in protective coatings like plastic, metal and concrete. As a result, the Internet organism takes the broad shape of our civilisation. It expands horizontally outwards and upwards as we build infrastructure, dwellings and cities, extending human settlement. You might ask how, by the above definition, the Internet is different to road and water pipe networks? It’s true these also take the broad shape of our civilisation, but these networks are orders of magnitude less complex. Bitumen roads and metal water pipes are elementary by comparison, and don’t share some of the earlier artificial life traits covered in previous sections. A bitumen road is not alive only when cars are driving on it, nor is a water pipe only alive when water is flowing. I think these things are fundamentally different, unable to exhibit the complex behaviour and other life-like processes we observe in the Internet.</p><p>Many living organisms move around their environment. The Internet, however, is sessile. Like a plant or a fungus it is unable to directly transit through space like an animal. After taking root in the early 1980’s as the first networks were established, it has continued to radiate outward, like a slime mould snaking its way into the surrounding environment.</p><p>And what of reproduction? Unlike all biological life, our machines don’t self-reproduce; they can’t produce identical copies of themselves. Cells themselves can undergo binary fission (mitosis) to produce a copy of themselves containing exactly the same genetic information. Our machines can’t: the best we can do is a 3d printer that can print 70–80% of its components (see <a href="https://all3dp.com/2/3d-printed-3d-printer/">here</a>). The Internet machine grows through us humans, as we slap on new routers to our homes, build new data centers and subsea cables. But it is incapable of self reproduction and completely dependent on us humans for growth, as it is for energy.</p><p>Chapter summary: The Internet is a sessile organism that grows through its biological partners: we humans. It is incapable of self-reproduction in the same way all of our electrical machines are. For these life processes, it depends on us in a kind of symbiotic relationship.</p><h3>Chapter 6 — Evolution</h3><p>Despite the fact we humans directly evolve the Internet, I assert that there is no grand design; no omniscient predetermined path. I think everyone is pretty amazed at just how rapidly the utility of the Internet has evolved.</p><p>Just like biological evolution blindly shapes and repurposes living systems to deal with their immediate problems, so too do we evolve the Internet to meet an immediate need; we build more data centers, we design a slightly faster processor than the one before, we repurpose older vestigial parts (organs) and extend their function to meet new needs, we increase the speed of data transmission and decoding. The Internet could be said to evolve through us but there is no grand design; it’s continuously evolving through technological advancement to meet a changing set of needs, driven by our own creativity. Like biological evolution it happens in small, incremental steps.</p><p>Chapter summary: The Internet changes and evolves over time. As it depends on us as its creators to grow and change, its evolution is guided by our needs. As we evolve through natural selection, the Internet could be thought to evolve in a kind of second-order evolutionary process.</p><h3>Chapter 7 — What type of organism is the Internet?</h3><p>If we entertain the thought that the Internet could be some kind of emergent artificial organism, what is it most like compared to what we know of biological life? How might we begin to classify it anatomically and taxonomically as a biologist who just downed 5 tabs of LSD and a bottle of vodka?</p><p>Ok here we go.</p><p><strong>Taxonomic Classification of the Internet</strong></p><ul><li>It’s a large multicellular-like organism, comprised of many systems of organisation to give it a complex but recognisable structure, with a significant ‘biomass’ distributed over a massive geographical area at a planetary scale.</li><li>It is sessile, meaning it does not move around its environment like an animal.</li><li>It grows in size outward in 3 dimensions indiscriminately, much more like a fungus or a plant (creeper) and unlike an animal which grows to a specific pattern and body size limit. I.e. its growth is more plant or fungus-like.</li><li>It lacks a predefined or final body ‘shape’ making it unlike an animal or plant, and more like an amorphous fungus such as a slime mould. However it does broadly take the shape of human civilisation, embedding itself in the artificial habitat we construct.</li><li>Unlike biological organisms which all have a primary function to consume energy and grow to reach a reproductive stage, the Internet does not exhibit a primary motive. This makes me think in terms of purpose, it’s less like a complete organism and more like an stand-alone organ; something with a specialised function like a liver, or heart.</li><li>If it is organ-like, it is functionally most like a brain. It shares interesting similarities to a brain in that it is optimised to send high speed signals throughout its entire mass (akin to neural pulses), store information for retrieval (akin to memory), receive and respond to external inputs (akin to senses like vision and hearing) although it does not appear to exhibit conscious intelligence traits. If it is a brain-like organ, it is ‘unconscious’.</li><li>Metabolically it is incapable of producing energy itself through catabolism (eating) or photosynthesis (sunlight). Instead it depends on electricity. This further lends itself to the organ theory as organs are also completely dependent on their surrounding environment (host organism) to supply them with the energy and other inputs they need. I.e. the Internet is an organ that cannot survive without a surrounding host environment: the artificial environment we provide through protective structures and the supply of electrical current.</li><li>It is incapable of reproduction or self-growth and relies on a symbiotic relationship with a biological organism (human beings) for these core needs, and energy input.</li><li>As it does not have a natural reproduction and death cycle, It has no apparent life span and is effectively immortal, although it can still be killed if it is starved of energy (electricity) or unprotected from the natural environment.</li><li>It exists in an artificial environment, with almost all of its mass encased in protective coatings (like plastics, metal and concrete) and it is embedded into the artificial ecosystem constructed by its symbiotic partners (us). This is similar to African leaf cutter ants constructing their nest around their fungal partner.</li><li>Its component parts (e.g. computer banks, server racks, telecommunication hardware, circuit boards, microprocessors, cell towers) share what you could call genetic traits. E.g. Microprocessors in every circuit board of every Internet connected device share commonalities in design, and how they look under a microscope. Its parts have a recognisable and common (hereditary) pattern.</li><li>It’s component parts are sensitive and susceptible to harm and require specific environmental conditions (temperature range, mechanical vibration, humidity, acidity and water content) to stay functional</li><li>Unlike practically all biological life that is tolerant to some degree of water exposure, the Internet is highly intolerant to water. Water is like a poison to it, rapidly killing localised parts that are exposed by disrupting the metabolic system (i.e. electrical short circuits).</li><li>It has a decentralised metabolic system: electricity is fed into its structure from thousands, probably millions, of independent sources.</li><li>It evolves and changes throughout time.</li></ul><p>In summary: the Internet is more like a specialised organ, rather than a complete organism. Organs (like your brain) require a host to provide external inputs to meet metabolic needs (such as oxygenated, glucose enriched blood). In the case of the Internet, we are the host in the form of our artificial habitat. Through this habitat, we help it grow and supply it with continuous electricity for its metabolic needs and the right environmental conditions (homeostasis) for it to survive. The Internet therefore could be thought to be a parasitic or symbiotic organ, embedded within human infrastructure.</p><h3>Chapter 8 — What is our relationship with the Internet?</h3><p>If the Internet can be thought of as some form of artificial organism or life, then it depends for its survival on a symbiotic relationship with us humans. It is reliant on us to sustain it with energy (electricity), heal non functional parts, and for growth and function. In return it nourishes us by enriching our cultural and conscious experience.</p><blockquote><em>The closest parallel I can think of to the situation with us and the Internet is that of the African leaf-cutter ants living in a symbiotic relationship with their underground fungal partner.</em></blockquote><p>Co-dependent with each other, the fungus depends on the ants for nutrients (leaf cuttings) and in return for their labour the ants have cultivated a source of nourishment themselves and carefully feed on the fungus.</p><p>In the case of the Internet: we are the ants and the Internet is the fungal-like symbiont. Like the ants we are hypnotised by it; barely grasping its mystery yet unable to turn away. Many of us spend almost all our daily lives connected to it in some way. This is surely a co-dependency in the making and the big question is where the hell is this leading us to?</p><h3>Chapter 9 — Finale</h3><p>My conclusion is this: the Internet is ‘most like’ an unconscious fungal brain (i.e. it’s an organ, not an organism).</p><p>It is a multicellular brain-like super-organ, with a planetary scale distributed mass, that grows indiscriminately from any part, much like a fungus. It depends on a symbiotic partnership with a biological organism (human beings) to sustain its metabolism (electrical energy), growth and environmental protective needs. In return it provides nourishment in the form of economic and cultural enrichment: connecting our species like a hive mind.</p><blockquote>The Internet is a virtual organ to our species, extending our senses and interconnecting us in a form of co-dependence and co-evolution.</blockquote><p>My final question is: to what strange future is our species’ co-dependence with the Internet taking us? Will it be beautiful or the stuff of nightmares? Are we on a pathway to assimilation and is the modern smart phone the final precursor to bioimplants?</p><p>Star Trek’s infamous Borg species is to me no longer such a far stretch of the imagination. I think we all need to think far more seriously about how we want the Internet to evolve, how we might nurture its good qualities and abandon (or regulate against) the bad qualities.</p><p>I’ll leave you with these images to ponder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MoYwDlKTRHwpzMJMe1FWfQ.png" /><figcaption>Above: Star Trek’s dystopian technology assimilated race called the Borg (left) and Meta group testing VR (right). Credit: <a href="https://vocal.media/futurism/resistance-is-futile-10-real-psychological-reasons-the-borg-from-star-trek-are-so-scary">Roana Sylver</a></figcaption></figure><p>It’s just too good.</p><p>The End. <em>🐜🐜🐜 🍀 🍄</em></p><p><em>P.S. I’m fully aware this is a comically bad thought experiment, but I hope I’ve provoked a few thoughts and amused at least one person. I’d love to hear what others think about the Internet organism in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d5d5c5f4ad8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/predict/is-the-internet-an-alien-organism-8d5d5c5f4ad8">Is the Internet an alien organism?</a> was originally published in <a href="https://medium.com/predict">Predict</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I’ve Built a Public World Atlas with 2,500 Datasets to Explore]]></title>
            <link>https://medium.com/data-science/ive-built-a-public-world-atlas-with-2-500-datasets-to-explore-8b9ae799e345?source=rss-e5e839f0817c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b9ae799e345</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-for-change]]></category>
            <category><![CDATA[data-visualisation]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[editors-pick]]></category>
            <dc:creator><![CDATA[Dan Baker]]></dc:creator>
            <pubDate>Sat, 19 Feb 2022 13:00:08 GMT</pubDate>
            <atom:updated>2023-02-14T02:28:03.551Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8F6yPqzIvvCmWUJvcWhLA.jpeg" /><figcaption>Image by Ylanite Koppens as free stock on Pexels. Source: <a href="https://www.pexels.com/photo/beige-analog-compass-697662/">https://www.pexels.com/photo/beige-analog-compass-697662/</a></figcaption></figure><h4>Inspired by Encarta, built in Python, accessible at worldatlas.org</h4><p>This is a self-funded thought experiment, 18 months in the making, with the goal of asserting one thing: “we can do better at making important data accessible to everyone”.</p><h4><strong>Show me the goods</strong></h4><p>Go to <a href="https://worldatlas.org/">worldatlas.org</a> and hit the <em>user guide</em> button for a quick video flyover. Best experienced on Chrome (MacOs, Win, Linux) or Firefox (MacOs, Win) on a desktop or laptop.</p><h4>Open Sourced on Github</h4><p>View the project source code at <a href="https://github.com/danny-baker/atlas">https://github.com/danny-baker/atlas</a>. Pull requests very welcome ❤️</p><h4><strong>Pitch</strong></h4><p>Have you ever wondered why we don’t have a World Atlas as a replacement for the paper Atlas, in the same way Wikipedia has replaced the paper Encyclopaedia?</p><p>Do you ever wonder how the country you live in compares to others across a range of areas spanning education, literacy, health, crime, biodiversity, gender equality, life expectancy, economy etc? Perhaps how these factors are measured and how they have changed over time? Do you lie awake at night missing <a href="https://en.wikipedia.org/wiki/Encarta">Microsoft Encarta</a> from the 1990’s?</p><p>If your answer to any of the above questions is ‘Holy crap, YES!’ or ‘what the hell is Encarta?’ then you have come to the right place my friend. There’s also a good chance you need therapy. Get ready to bear witness to my first attempt at a public web app. If you can see past the problems, I hope this manic creation ignites a spark of interest for how we might make important facts and statistics about the world more (intellectually) accessible, for everyone.</p><p>I’ve built a World Atlas as a website, incorporating over 2,500 datasets for you to explore. With over 12 million data points spanning 100 categories, find a dataset that interests you and analyse it using a suite of interactive charts. All data is fully sourced and available for download. It’s designed to be easy to use and fun to learn. Let me show you what I mean.</p><h4><strong>What might a modern World Atlas look like?</strong></h4><p>If we were to re-imagine the World Atlas as a website it should probably still have a map as the centrepiece. Map=good. But wait… couldn’t we do a lot more to visualise the data with pretty colours and interactive charts? You bet your ass we can. We can bring the data to life with modern tools in data science. For example, wouldn’t it be interesting to look at:</p><ul><li><a href="https://worldatlas.org/Access-to-electricity-(percent-of-population)/2018/map">A map indicating access to electricity across the world in 2018</a></li><li><a href="https://worldatlas.org/Technology---Percentage-of-individuals-using-the-internet-(percent)/2017/bar">All countries ranked by their access to the internet in 2017</a></li><li><a href="https://worldatlas.org/At-least-basic-sanitation--overall-access-(percent)/x/line">How access to clean water and sanitation has changed over the last 20 years</a></li><li><a href="https://worldatlas.org/Which-side-of-the-road-do-they-drive/2020/globe">An interactive 3d globe of the world showing which side of the road each country drives on</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dey-gEpf3fHfsxMjksUM0Q.png" /><figcaption>Above: Maps and charts generated by my prototype, corresponding to the four links above. Almost any dataset can be visualised as a map and analysed using a suite of charts and 3d visualisations.</figcaption></figure><h4><strong>My god…think of the possibilities</strong></h4><p>And that’s just for starters. Choose from over 2,500 open datasets across 100 categories spanning economy, finance, health, education, environment and technology. All data can be downloaded directly to your local machine. All data can be analysed in real-time with interactive charts. All datasets are fully sourced so you can trace their origin and the authority that published them, along with caveats. Any visualisation (chart or map view) is reproducible via a full URL path, so you can share a specific insight with anyone, anywhere, anytime.</p><p>For example, a map of <a href="https://worldatlas.org/Sugar-per-person-(g-per-day)/2013/map">sugar consumption per person in 2013</a> and accompanying chart of <a href="https://worldatlas.org/Sugar-per-person-(g-per-day)/2013/bar">each country ranked in order</a>. Or a map of <a href="https://worldatlas.org/Cell-phones-(per-100-people)/2018/map">cell phone use per country in 2018</a> (phones per 100 people) and a chart showing <a href="https://worldatlas.org/Cell-phones-(per-100-people)/x/line">how it has changed</a> over the last 40 years.</p><p>Whether you are a school student, boomer or grandparent I think every person in the world should have access to these kinds of datasets in a way that is simple and intuitive enough for them to understand. The problem right now is I don’t believe this data is accessible to the general public at all. And by inaccessible I don’t mean datasets can’t be found, it’s just when they are located they are not in a format that is intellectually consumable to the average person. They must be downloaded as a .csv file, cleaned, then plotted manually (requiring technical skills) or they are buried in a report somewhere for a specific purpose. Alone and naked, these valuable datasets are meaningless to the person on the street.</p><p>Organisations like <a href="https://www.gapminder.org/">GapMinder</a>, <a href="https://data.un.org/">United Nations Data</a>, <a href="https://databank.worldbank.org/">Worldbank</a>, and <a href="https://github.com/open-numbers">OpenNumbers</a> (along with many others) have done the heroic work of producing, cataloguing and maintaining many great datasets. A global community of contributors have helped build the open-source charting libraries and other amazing web tools that allow a modern web application to do things we thought were impossible only five years ago. The data is here. The tools are here. The final piece of the puzzle (I believe) is to bring it all together in some kind of centralised platform to visualise these data in a way that is meaningful for as many people as possible.</p><blockquote>“What we need is a product to iterate and evolve; a stage to bring important datasets to life.”</blockquote><p>This is what I’ve set out to prove with this prototype. This essay is the white paper for my proposal for an open-source open-data community funded World Atlas for the general public. A modern and free replacement for Encarta’s World Atlas using the latest tools in data science, and a very different landscape to the flooded ‘textlands’ of Wikipedia.</p><h4><strong>What are the problems we are addressing? (Why build this?)</strong></h4><p><strong>Key problem 1: Wikipedia is not built to visualise statistical data</strong></p><p>With the loss of Encarta in 2009 we lost two things: an Encyclopaedia and a World Atlas. Wikipedia has firmly taken the mantle of the de facto standard for a World Encyclopaedia and it is one of the few remaining wonders of the internet. I’ll continue to donate to it for the rest of my life. What Wikipedia IS NOT, is a suitable platform for visualising and understanding statistical data; it’s not an Atlas.</p><p>Sure you can find tables and data on things like world population and that kind of thing but, fundamentally, Wikipedia (more correctly the underlying ‘wiki’ engine upon which it is built) is structured around displaying interlinked pages of text. This is what it does and it does it incredibly well. Almost too well.</p><p>Wikipedia, at over 6 million articles in English, is now a flooded textland. Users are buffeted by waves of text at every turn, with images and tables thrown in awkwardly, almost as an afterthought. There are no real-time charts or analytical tools. The Wiki engine is just not designed for this purpose; it’s dated technology, largely unchanged for 20 years. That’s not a criticism, it’s just a fact. New tools have arrived, and they are good. I also think Wikipedia is a barrier for less literate members of the general public (who exist in all age groups). We all know data insights are more powerful as visual representations because we can use our most powerful primary sense: spatial reasoning, to interpret what we see. I think there is a missed opportunity here because we don’t have a mainstream, visual-first resource as a sister site to Wikipedia; we don’t have a World Atlas to complement our World Encyclopaedia.</p><p><strong>Key problem 2: Data Inaccessibility (Data Literacy)</strong></p><p>If we can agree that it’s a good thing for the public to understand (or at least have easy access to) facts about the world like carbon emissions, deforestation, economic wealth, health and education factors etc, then I would assert the problem is much of this type of data is not as accessible as it could be, and we can do better. Way better. And by better I mean providing opportunities to find this information in one place, for visual learning, and for interactive exploration of data in a way that is much more powerful than looking at numbers in a table.</p><p>Most of this data is scattered all over the place, hosted on a myriad of different websites by the differing authorities and institutions that publish them, written up in separate reports and peppered across web pages. It’s a literal nightmare. And in many cases when you do finally find good data it’s usually targeted at professionals in the surrounding industries, and is not in a format that can be digested easily by school age students etc. This makes me sad. Right now, you basically need to be a data engineer, technical professional or academic to know where to find good global statistical data, to say nothing of how you would begin to clean, analyse and visualise it to discover an insight.</p><p>Whilst it’s great that we have access to curated open datasets such as those from UN Data, Worldbank, and Gapminder — it’s not enough. I think the absence of a well-known intuitive educational tool to visualise the data and draw insight means that this data is largely lost on the general public. A school age student does not know where to find, nor how to interpret or analyse, a .csv file. This is the core problem I’ve set out to solve. I want to make important data super-accessible to everyone. I also want to enable users to form an insight on their own, rather than spoon feed it to them.</p><p>In summary, the central problem I am asserting is: “many great geo-statistical datasets exist but they are not presently accessible to the general public in a format that is meaningful”. Put more bluntly: “We can do better at making important data accessible to everyone”.</p><h4><strong>Design principles</strong></h4><p>If the mission is to “make country-scale statistical data super-accessible to everyone” what then might our core principles for design be? Here are my thoughts for how we might combine the spirit of the paper Atlas with the charm of Encarta, and perhaps a splash of magical Gummy berry juice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Rqdmvq2DrJrjLy-" /><figcaption>Above: The eight guiding principles I think we need for a World Atlas that the public would engage with and enjoy. Image courtesy of Disney’s Gummy Bears.</figcaption></figure><p>I’ve distilled it down to the eight principles you see here. The map has to be the centrepiece of this creation, but of course as we escape the confines of paper and embrace the power of the modern web browser we can enrich the Atlas experience immensely. We can add buttons, dials, sliders, colour schemes, analytical tools and charts, and other interactive widgets. If done tastefully, this could make for a superb user experience.</p><p>It also needs to be generic enough so we can keep adding datasets and extend functionality with new tools as they become available. (i.e a shift away from single-use websites to a truly reusable model). Finally, people need to like it and enjoy using it. It needs to be fun and have some style. (Help definitely needed here but I’ve tried).</p><h4><strong>What I have built so far</strong></h4><p>With the above principles in mind, I have made an attempt. It’s a website. It acts as a front-end for the user to visually access thousands of datasets.</p><p>It provides a fully interactive map that can be zoomed and panned, a navigation menu to select datasets, a time slider to flick between different years, and buttons to bring up real-time charts for analysis. All data is downloadable and fully sourced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dlPMJ4knKBD9aa4e" /><figcaption>Above: The current prototype allows users to find and select from over 2,500 datasets, visualise them on a map, view the source data, and perform basic analysis via a suite of interactive charts. Live at <a href="https://worldatlas.org/">https://worldatlas.org</a>. Open source on Github <a href="https://github.com/danny-baker/atlas">here</a>.</figcaption></figure><h4><strong>How it works (nerd section)</strong></h4><p>For the nerdy people like me who want nothing but pure detail: the prototype is a web application written in Python that aims to be a generalised platform for ingesting country-scale datasets and displaying them visually in a variety of ways.</p><p>The front-end is a typical website comprised of the navigation menu along the top, main map, and various components. This is all built in <a href="https://plotly.com/dash/">Plotly Dash</a> open-source, allowing the entire website to be written in Python, while gaining access to awesome Javascript charts out of the box. To make the magic happen in the back-end I’ve taken country/region borders from <a href="https://www.naturalearthdata.com/">Natural Earth Data</a> (geojson polygons), which allows us to draw borders around each region, and colour them uniquely. I’ve located, downloaded, cleaned, and formatted thousands of open datasets to standardise them for use in this generalised platform. I’ve curated the current list of ~2,500 datasets, re-labelling many series names to be more intelligible with consistent units of measurement. I’ve also tagged each series according to its data type (e.g. Categorical, Continuous, Ratio, Quantitative) in order to determine which charts should be available and how they behave. Finally, I’ve categorised and sub-categorised each series so they can be searched and easily built into a nested navigation menu with categories like ‘people &gt; education’, ‘environment &gt; climate’ etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aX3jS83ul5VD9QQI" /><figcaption>Above: The basic approach I’ve taken is to combine regional borders (polygons), country-scale datasets, and standardised country identifiers to provide a visual interface to any dataset in the form of a map or chart, in the browser.</figcaption></figure><p>Once processed, I store and compress all the .csv data to the <a href="https://arrow.apache.org/docs/python/parquet.html">.parquet</a> binary format. This is an awesome format for storing big tabular data as it uses columnar compression (especially good if you have many rows of repeated data). To give you an idea of just how insane it is, I’ve condensed 5GB of .csv files (5 columns x 12 million rows) to a single 100MB parquet file, that’s a 98% compression ratio. The parquet file is read in at run-time by the Python app and loaded into memory as a pandas Dataframe in 4 seconds flat, giving instant access to 12 million unique records. Because all the data is in memory at run-time, this allows the charts and maps to rapidly re-query the data and serve different data series to the client browser in seconds. There are probably better methods, but this is the fastest way I’ve found to access such a large dataset at speed. Keep it simple: pandas dataframes and a shit tonne of ram.</p><p>I’ll add here that I have looked at using non-pandas data structures such as <a href="https://docs.dask.org/en/stable/dataframe.html">Dask</a> and <a href="https://vaex.io/docs/index.html">Vaex</a> but these add a lot of complexity and have reduced features, and I don’t think they are really necessary unless the main dataset gets beyond 10GB. Through strong data-typing of the dataframe columns (e.g. using categoricals for Country and Series names, and unsigned integers (uint16) for the year, for example, I’ve reduced the main dataframe size by 87%, down to about 1GB per instance of the app, and rapidly sped up query times.</p><p>For web developers, the web application itself is a stand-alone Flask app encasing a Plotly Dash Python app, providing the full power of the Flask framework with the power of Plotly’s open-source charts. Full write-up on how to do this courtesy of Hackers and Slackers <a href="https://hackersandslackers.com/plotly-dash-with-flask/">here</a>. Finally, to get it deployed I’ve containerised the app via Docker along with an NGINX load balancer container and a certbot container to manage TLS (HTTPS) certificates, and brought it all up on dedicated virtual machines on Azure infrastructure. I’m sure there are better ways but (1) I don’t know what they are; and (2) I prefer having full hardware control via a VM rather than relying on app services like Heroku. I’ve tried these but owing to high ram requirements of the app, going for dedicated virtual machines has ended up being cheaper and more versatile.</p><p>For DevOps-y people, the site is being served via 2 NGINX workers (with HTTP caching) which act as reverse proxies to 3 Gunicorn workers (app instances) running 5 threads each. This allows the app to serve client requests in true parallel and the threads can share the memory (i.e. main dataset) of their parent gworker. In theory the site should be able to serve several hundred users simultaneously, but I’ve yet to properly load test it. Presently I’m not using a CDN, it’s just a single host running in a UK Azure datacentre so there is unfortunately geographical latency based on your distance from the UK.</p><p>End nerd section.</p><h4><strong>Why I have done this</strong></h4><p>I miss Encarta. I was there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xi64cUpF4GhAwBzLYPfrcw.png" /><figcaption>Above: Microsoft Encarta launched in 1993 for ~400USD (750USD in today’s terms). It wowed us with images, video, text, sound and in later versions a fully interactive map of the world. (Images sourced as snapshots from various YouTube videos.)</figcaption></figure><p>Microsoft made an impression when they first released Encarta in 1993. In the golden era of squawking dial-up modems, ‘Ace of Bass’ on the radio, and wall-length encyclopaedias, Microsoft released a shiny compact disc containing what seemed like the sum of all human knowledge. It was a breathtaking showcase of how we could represent information using tools of the emerging digital technology space. With the explosion of high speed internet into millions of homes throughout the late nineties, new website based knowledge platforms (such as Wikipedia) were able to scale at incredible speed. This eventually led to the demise of Encarta in 2009.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-MKMOZxcK43dOVBL" /><figcaption>Above: Today, almost thirty years since the launch of Encarta, we have an amazing World Encyclopaedia (Wikipedia) but something is still missing…<em>where is our World Atlas?</em></figcaption></figure><p>I think the world is still missing a quality the Encarta creators just nailed: they delivered curated, accessible information in a format that could be digested by those of all ages. And they did it with style: colours, sound, images, video, interactive maps, it was a triumph. Entire families from kids to grandparents would be fighting over the compact disc.</p><p>Somewhere along the path we’ve taken from the sunlit days of Encarta to the dystopian smartphone nightmare of today, something has gone horribly wrong.</p><p>As hard as I have looked, I haven’t found any free public resource that meets the standard set by Encarta for information accessibility and provides a World Atlas-esque experience that I love. After I couldn’t find satisfaction, I began building it with the naivety of an idiot. How hard could it be?</p><h4><strong>Product Evolution</strong></h4><p>The following is a comical collection of early designs ideas that take us to the present version of the site.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/0*Fhbw2-x2LUqhJQWf" /><figcaption>Above: Mockup 1. The original idea was to go full national geographic where users could dive into categories, or drill in by country. I still think it would be cool to have a public resource like this. It could almost be something like an alternative front end interface to relevant Wikipedia articles.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PMc8MfssivBBCeYB" /><figcaption>Above: Mockup 2, a remarkably bad wireframe (my first ever) that attempted to craft together a navigation menu, map, and control panel, similar to GIS software.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yN0wrBHX_L_-h3bd" /><figcaption>Above: First running prototype with world population on a mapbox Choropleth and a time slider, running on local webserver. A very happy moment. First year students in computer science probably do better than this in a 2hr tutorial, but as far as I was concerned it was time to light a cigarette and kick back. It was all down hill from here, surely.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*L00GOcF8SN3tiUr1hxX5TA.png" /><figcaption>Above: Adding a bit more polish and new datasets. Time slider, and source information, along with chart buttons.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2GvYCOo_dgRbSiAq" /><figcaption>Above: Latest version incorporates a little more polish, with transparent overlays to give the map maximum available screen cover. Added search bar, full navigation menu, and lots more datasets.</figcaption></figure><p>The present state of the site is as far as I’d really want to go as a demo or prototype. The project is now at the point where I want to validate it against strangers (not friends and family) and see if it has any value. If not, well it’s been fun.</p><h4><strong>A look at the competitors</strong></h4><p>In my search for the Holy Grail of World Atlas-y type experiences, I’ve found some truly great resources and some disappointing ones. I’ll lay these out below along with an uninvited and stinging critique for why I’m still not satisfied with what’s out there.</p><p>The criteria for which I make this entirely opinionated assessment of the competitors is based loosely on user experience, ability to appeal to all ages, apparent quality of data and ability to verify it, ability to analyse or interrogate the data, ownership and mission. But mostly I’m assessing based on whether I think the resource meets the Encarta standard for delivering great learning experiences for all-ages. (I fully acknowledge many of the competitors are targeting more selected audiences, and this is fine). Before we embark on this voyage of discovery, let us begin with the Gold Standard: Microsoft’s Encarta.</p><h4><strong>The Standard: Microsoft Encarta World Atlas</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBvRZpMSMRhm5gaGZOoG9A.png" /><figcaption>Above: Encarta was released with a World Atlas from 1997. It provided a fully interactive map and globe, with various statistical datasets to overlay spanning categories like culture, politics, animals, geography. See it in action <a href="https://www.youtube.com/watch?v=gIrY3S-usQ4">here</a>.</figcaption></figure><p>It’s hard to overstate the significance of what Microsoft achieved almost thirty years ago with this product. You have to remember this was a time of dial up modems and Pentium computers, it was before high speed internet, before MP3s and many of the breakthrough audio and video compression technologies existed, before GIS software was mainstream, before Google existed as a company, 10 years before YouTube, 5 years before Mozilla Firefox browser, and almost before the USB standard was released (1996). It was a time when owning a 8x (8-speed) cd-rom made you popular with friends.</p><p>Microsoft built the map software application ground up (their MapPoint software) and put it on a compact disc. The World Atlas came shipped with Microsoft Encarta from around 1997 and was targeted for family homes and all-ages education. At an exciting time when home computers were advancing dramatically, it was fun to use and navigate, providing an interactive globe and map experience where users could zoom and pan intuitively. It also displayed data in an unpatronising way. Users could choose to display interesting geographical data spanning physical, land and climate, human, political categories. Learning was genuinely a pleasure and you could spend hours just browsing what it had to offer. I think it’s still impressive today and I am not aware of a modernised, freely available version or similar product currently available.</p><p>Score: 10/10 Encarta Love Meter</p><h4><strong>Competitor 1: </strong><a href="https://www.worldatlas.com/"><strong>Worldatlas.com</strong></a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kYLIVhevuvL7Z8KPgUwoGA.png" /><figcaption>Above: <a href="https://www.worldatlas.com/">Worldatlas.com</a> provides an interactive map and lots of country information, however the experience without an ad-blocker ruins every possible part of the educational value.</figcaption></figure><p>By its name alone, I want to like this resource. I really do. It’s got a good domain and an interactive map as the centrepiece, with some great ideas and articles about interesting pieces of history from parts of the world, along with nice country summary tables. It also appears to be targeted at all ages. If I am brutally honest though, I find the experience dated and disappointing. The main map is clunky and not very interactive by today’s standards, which I could forgive if the site wasn’t riddled with advertising. Even with an ad blocker it’s a frustrating user experience and I think if the site owners really cared about education, as they claim, they would find a way to do it without blatant, distracting advertising that ruins the educational experience entirely.</p><p>It’s also impossible to verify any of the data in the tables and content in the articles, because there is no source information, just a loose claim that they fact check everything. Guess we’ll just have to take their word it’s all accurate. I’m not sure how they are funded but appears to be supported by a private company. In shifting the medium for a World Atlas from paper to a website, I feel this site has managed to take some of the worst aspects of the web experience (e.g. flashing ads) and failed to capitalise on the best aspects of web (the possibility for a truly immersive experience). I’d much prefer to use a paper Atlas.</p><p>Score: 1/10 Encarta Love Meter.</p><h4><strong>Competitor 2: </strong><a href="https://www.cia.gov/the-world-factbook/"><strong>CIA World Factbook</strong></a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_QipK4p9RBEWeGiyKaFmHQ.png" /><figcaption>Above: The Central Intelligence Agency (CIA) <a href="https://www.cia.gov/the-world-factbook/">World Factbook</a> provides data tables and profiles on 266 country/regions, and many static maps that can be downloaded.</figcaption></figure><p>In my Atlas pilgrimage many I have spoken to recommended I check out the CIA World Factbook. I’ve checked it out and indeed it does represent information in a nicely structured way for basically 266 countries/regions of the world, with static maps, tables of statistics, nice categories and surprisingly good explanatory content. It certainly provides some of the components of a good Atlas. You can download the maps and I really like that you can download the various data tables as a .csv file (big tick here). The whole experience, and the layout, has polish.</p><p>However, there are a number of major causes for concern. Obviously there is the issue of the entire resource being hosted by a sovereign nation (USA) so it’s not as globally independent as the UN Data services, for example. I’m also disappointed to find that none of the map or statistical data (and I mean none of it) is sourced. Most of the data tables just say ‘est 2021’ or something. This means there is no possible way to verify anything on this site as being true and accurate. People could just be making up numbers and chucking them in the tables, and no one would know. It’s this lack of integrity that allows users to check the origin of data that I see time and time again, and I think it’s a major problem for many of these ‘trusted’ public resources. Any published statistics anywhere without a reference to a source are 100% useless and 100% unverified as far as I’m concerned. Maybe that’s too harsh, but I just think the standard we must put on published statistics has to be better than this. For a resource that calls itself a <em>Factbook</em> it’s ironic that there is no way to verify the facts.</p><p>Score: 3/10 Encarta Love meter.</p><h4><strong>Competitor 3: </strong><a href="https://earth.google.com/web/"><strong>Google Earth</strong></a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fLBJQ0EMUI4gOwf-qXJyDg.png" /><figcaption>Above: <a href="https://earth.google.com/web/">Google earth</a> provides a beautiful user experience to the world, but it does not (to the best of my knowledge) support overlaying statistical data.</figcaption></figure><p>If we’re thinking about a digital Atlas, we can’t go past without looking at Google Earth. From an interface and user experience point of view, I don’t think anything can touch it; it’s a superb product with the ability to zoom almost to infinity from space to street level. It has made the giant leap from the imperfect projections of the earth on a flat surface (e.g. Mercator) to the only true and correct way to visualise the Earth without projection distortion trade-offs: a spherical globe.</p><p>The clincher is it’s really only built for displaying the physical landmarks, it doesn’t (to my knowledge) have a comprehensive geographical-information-systems (GIS) component that would allow you to visualise region specific statistical data, such as population density. I know you can add geo points and series, with a bit of metadata etc but this is more for cartographic applications. Nor can it be used to plot data in real time with charts etc. It’s just not really designed for the purpose of displaying geostats, it’s more for displaying high resolution photographic imagery of the Earth.</p><p>In summary I think Google Earth captures the Encarta-esque magic from an interface and user experience perspective (++points), but fails to deliver on how to display the geostats (data) itself. I.e. the interface is insane, it just doesn’t have the stats I want. It’s also a private company owned asset so there is no community control of the content.</p><p>Score 4/10 Encarta Love meter</p><h4><strong>Competitor 4: United Nations Data Portal (</strong><a href="https://data.un.org/"><strong>UN Data</strong></a><strong>)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCEqAsv59IMZvCSDs-643Q.png" /><figcaption>Above: The <a href="https://data.un.org/">UN Data Portal</a> and other similar data services provide a basic facility for users to locate and download large geostatistical databases, free of charge. No visual interface to the data is provided.</figcaption></figure><p>I’ve chosen UN Data as a representative competitor to some of the other publishers of geostatistical datasets (others include <a href="https://databank.worldbank.org/">Worldbank</a> and <a href="https://github.com/open-numbers">OpenNumbers</a>). These resources collect, curate, and lovingly maintain some of the most important global statistics about the world that I know of. All the data is freely downloadable and many of them have APIs (application programming interfaces) that allow a direct machine to machine connection to access their data, which is awesome. If you’re not familiar with APIs, in this context it means you can connect to their systems and download their data directly from your code (running app), rather than manually downloading a .csv file from the browser as a user. It’s like a back door to the data. In any case, what they don’t do is provide a visual interface to any of the data; it’s raw access only.</p><p>Really, these sites are a shopping trolley for you to download specific datasets from their shelves. Getting good data is a critical piece of the Atlas puzzle, and these organisations provide this service. But of course getting from raw data to an insight moment requires additional effort and technical prowess to visualise the data in a meaningful way.</p><p>I’d like to single out UN Data because they are funded, unlike OpenNumbers which is all volunteers. I must say that I’m disappointed that a funded organisation such as the UN has not risen to the occasion to breathe a bit of life into the wonderful datasets they hold. I’d argue that hosting the data for download is the bare minimum effort. Part of their purpose and intent surely is to get this data into the hands of those that could benefit from it, particularly the general public. To quote their website directly:</p><p><em>“UNdata was launched as part of a project in 2005, called “Statistics as a Public Good”, whose objectives was to provide free access to global statistics, to educate users about the importance of statistics for evidence-based policy and decision-making and to assist National Statistical Offices of Member Countries to strengthen their data dissemination capabilities.”</em></p><p>I love the intent and am eternally grateful for the hosting of these open datasets. However, I’d say this resource is spectacularly underknown and underutilised by the general public due to the impassable technical bridge required to convert the raw data they supply into a meaningful insight such as a map or a chart<strong>.</strong> It’s useless to the average school student or non technical person. If I was assessing them on their core objective (above) I’d call it a complete fail because they have failed to really deliver on a ‘public good’ outcome to make this data intellectually accessible to the general public. It doesn’t matter if it’s available for download if no one can understand it.</p><p>Score: Not applicable as they are a data service provider only.</p><h4><strong>Competitor 5: ArcGIS Living Atlas Map Viewer</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gyt3tdDvrEcA5fE9" /><figcaption>Above: <a href="https://www.arcgis.com/apps/mapviewer/index.html">ArcGIS Mapviewer</a> is free to use without a paid account. This example is showing the early 2022 volcanic activity near Tonga and New Zealand in the pacific region.</figcaption></figure><p>I’ll admit, initially I was annoyed with ArcGIS because they are heavily geared toward paid subscriptions and I didn’t realise you could access a bunch of stuff for free. This is a great resource to explore. I haven’t been able to find any basic statistics like world population and GDP and that kind of thing. But I’m guessing it’s because this data is generally more for statistical analysis. We are obviously in hardcore GIS space here and I’m not qualified to comment on how you are (correctly) supposed to represent much of the statistical information.</p><p>There looks to be some great datasets here for learning in specific areas. This is a professional level tool. I’d say it’s a little too technical for the average middle school student or member of the general public, but this is the nature of a tool as sophisticated as this. It’s also a bit hard to actually find this map viewer tool. I spent 20 minutes navigating the ArcGIS website and avoiding all the paid subscription hard sells, before I actually found the free access part (massive barrier). Finally, it doesn’t appear to be a way to generate a hyperlink to share a specific map view, such as the one I used above of the volcanic activity. This is a lost opportunity for sharing knowledge and insight with others.</p><p>Score: 6/10 Encarta Love Meter.</p><h4><strong>Competitor 6: </strong><a href="https://www.gapminder.org/"><strong>GapMinder</strong></a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MT6cqkuOzzbctMVTOGgp8g.png" /><figcaption>Above: GapMinder provides amazing interactive tools to analyse many open datasets, including a play button to watch how the data has changed over time. It’s impressive stuff.</figcaption></figure><p><a href="https://www.gapminder.org/">Gapminder</a> is a (Swedish) not for profit. Their origin story, mission and intent is noble and aligned with my Atlas mission:</p><p><em>“Our mission is to fight devastating ignorance with a fact-based worldview everyone can understand. Gapminder identifies systematic misconceptions about important global trends and proportions and uses reliable data to develop easy to understand teaching materials to rid people of their misconceptions. Gapminder is an independent Swedish foundation with no political, religious, or economic affiliations.”</em></p><p>They are the driving force behind many amazing datasets used by websites and educators worldwide, they produce teaching materials and they have built a suite of awesome interactive and responsive visualisation tools (pictured above). They have even pioneered some new visualisation techniques that have landed their founder on famous <a href="https://www.ted.com/talks/hans_rosling_the_best_stats_you_ve_ever_seen?language=en">TED talks</a> (15M views). They provide links to access the raw data and good explanatory content around how it is standardised etc. Amazing stuff and a must see if you haven’t heard of them.</p><p>The one point I will make for why I didn’t stop work on my project when I discovered GapMinder is that they are not quite an Atlas. Their interactive charts and visualisation tools are fabulous and I feel that each of them are more like components of something bigger that could exist to bring it all together. Most of these great tools are also buried quite deep in their educational website on a subdomain as they are only a small part of the mission. So, for example, to get to the charts and maps displayed above, you need to actually click through three (1-click-2-click-3-click) separate web pages from the homepage! So you really have to know where to look to access these great tools, and I think this immediately adds a major barrier to getting to the data. Of course, this is my problem not Gapminder’s and they have a much broader educational mission than building an Atlas. If any organisation was to have a proper go at building the modern World Atlas, I would trust Gapminder to do it in a way that is super accessible for a broad audience. Lots of love.</p><p>Score: 9/10 Encarta Love Meter.</p><h4><strong>Competitor 7: </strong><a href="https://www.visualcapitalist.com/"><strong>Visual Capitalist</strong></a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJBck94vV7faNGCMaiQFXg.png" /><figcaption>Above: Visual Capitalist produces beautiful and authoritative infographics on important issues.</figcaption></figure><p><a href="https://www.visualcapitalist.com/">Visual Capitalist</a> is a publisher that builds artistically gorgeous infographics across a range of educational areas. Their mission is super on point:</p><p><em>“Making the world’s information more accessible. Every day a staggering 2.5 exabytes of data is generated making our world increasingly difficult to understand. By highlighting the bigger picture through data-driven visuals, we stay true to our mission to help cut through the clutter and simplify a complex world.”</em></p><p>This really strikes a chord with my Atlas dream. Visual Capitalist is an amazing resource and a must-see. The stuff they produce is artistically and professionally presented, whilst at the same time being authoritative and accurate, based on open public data like Census data, Worldbank etc. They also provide great explanatory content to contextualise the visualisations, and they source everything to a high standard. Their work has broad appeal and is unpatronising, making it highly accessible for all ages from school students and up. They have full path URL linking, so you can share a direct insight or visualisation, such as <a href="https://www.visualcapitalist.com/mapped-corruption-in-countries-around-the-world/">corruption mapped across the world</a>. They are independently owned and draw revenue from ads and licensing.</p><p>I can’t fault this great work and this fills the artistic void for bringing data to life, the very thing I think we need to do so much more of. They employ a different approach (polished infographics) to achieve that similar punchy insight and instant appeal through visually representing data. Visual Capitalist use high quality design to produce pieces of infographic art, whereas a digital World Atlas is simply another way to access the data that may arguably be less effective. In any case, this is not a competitor but a kindred spirit. I’m so glad I found it during this project.</p><p>Score: 8/10 Encarta Love Meter</p><h4><strong>Competitor 8: </strong><a href="https://ourworldindata.org/"><strong>Our World in Data</strong></a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ko-32O7284dVPt_Su3F25w.png" /><figcaption>Above: Our World in Data provides a scientifically rigorous approach to the World’s most pressing issues. They boast over 3000+ interactive charts backed up by comprehensive articles and explanatory content. Outstanding stuff.</figcaption></figure><p><a href="https://ourworldindata.org/">Our World in Data</a> is a mission centred on information accessibility to the public to convey and help articulate the World’s biggest challenges. They boast over 3000+ interactive charts across 297 topics. As a collaboration between researchers at the University of Oxford and a not for profit organisation <a href="https://global-change-data-lab.org/">Global Change Data Lab</a> they are funded by sponsors, philanthropists and grants including the Bill and Melinda Gates Foundation (that’s a big deal).</p><p><em>“The goal of our work is to make the knowledge on the big problems accessible and understandable. As we say on our homepage, Our World in Data is about Research and data to make progress against the world’s largest problems. We believe that a key reason why we fail to achieve the progress we are capable of is that we do not make enough use of this existing research and data: the important knowledge is often stored in inaccessible databases, locked away behind paywalls and buried under jargon in academic papers.“</em></p><p>Simply superb. Not only do they tackle the World’s most important issues, they do it with style: beautifully structured information, well written and accessible explanatory content, supported by interactive visualisations most relevant to the issue. They provide categorised ways to access different areas of interest, and data is fully sourced and available for download. They also host a <a href="https://sdg-tracker.org/">stand-alone website to track global progress toward the United Nations Sustainable Development Goals (UN SDG)</a>, which are arguably the most important goals we have for a future most people want to live in. This is a must see also.</p><p>The one and only criticism I can make here (which is more of an unavoidable fact) is that I feel ‘Our World in Data’ is a resource best suited to university level educated people and above. It’s academic grade work and it’s rich and detailed. I wonder if there is a way to open some of it up to broader audiences such as middle-school students, in the way Gapminder does so admirably? Unfortunately this is not an easy task given the enormity of the problems they are trying to address. It’s not a user experience like an interactive Atlas, it’s a high quality visual-rich publication (as it was designed to be). This is a must see resource for anyone interested in data that matters, and it is absolutely authoritative.</p><p>Score: 9.5/10 Encarta Love Meter</p><h4><strong>Do we really need a World Atlas in light of the existing great resources?</strong></h4><p>Maybe we don’t, I accept, but I will say this. Over the last 18 months I have traversed much of this rugged landscape in my quest to access and unlock important insights and rekindle the magic of the Encarta experience. I’ve hiked up snow capped mountains to gaze upon some of the best vistas we have to offer (Our World in Data) and I’ve had to cut the rope and let some baggage go along the journey. I’ve also had a go at creating something myself, exposing me directly to many thousands of raw datasets currently available.</p><p>As I reflect on this journey, what I think is still missing in the best of our public resources (Our World in Data, Gapminder, Visual Capitalist) is they are each a <em>collection</em> of great tools and resources. None of them is an actual complete <em>product</em> that delivers a consistent user experience, like Encarta’s World Atlas. Of the more product-type resources (such as ArcGIS mapviewer and Google Earth) these also don’t cut the mustard in my view because they don’t provide the interesting statistics that matter for broad appeal (biodiversity, literacy rates, GDP etc). So I think we’re still left with an incomplete and haphazard way for the public to discover this important data. And that’s a problem.</p><p>I would bet that less than 1% of those reading this article have ever heard of Our World In Data, or Visual Capitalist. Have you? I hadn’t, nor did I even know about Gapminder when I started. Conversely, I would bet that 100% of you have heard of Wikipedia and use it regularly. And this is the very essence of the problem: most great resources are simply lost in the noise of an ever expanding internet.</p><p>As the internet expands, the signal to noise ratio of quality to crap is going in the wrong direction, leaving the highest quality parts of the internet more and more obscured by the tidal waves of shit. Every day, it’s getting worse. The only way I can think of to combat this is to improve the ratio of quality material (good luck), or have a household name that everyone knows and trusts. Wikipedia is that for the encyclopaedia. I’d like to think we can also have a household name for an Atlas type experience, one day too.</p><p>After weighing up my arguments, does this whole concept of a modern World Atlas have social utility? Is <a href="https://worldatlas.org/Access-to-electricity-(percent-of-population)/2018/map">this</a> kind of information, presented in <a href="https://worldatlas.org/Cell-phones-(per-100-people)/x/line">this</a> kind of way, useful to you? Do you think it would be useful to others? Would it be useful in education? If the answer is yes, then my flawed experiment might have legs. If the answer is no, I’ve missed the mark and it’s time for me to let it go.</p><h4><strong>Who should pay for something like this?</strong></h4><p>If, for the sake of argument, we decided a community World Atlas would be something we wanted, this brings us to the question of funding. We would need to ensure it’s done in the most independent manner, so as to never corrupt the intent of the mission. I would opt for the Wikipedia foundation model of zero advertising, community and volunteer sponsored and curated as the only way to fund a tool like this without being influenced by companies, sovereign nations or capitalism. Our World in Data has a grant based model that I think would also work well. To stay true to the public education intent, it would need to be zero advertising and free for all, forever. With no exceptions.</p><h4><strong>The many flaws and caveats of my prototype</strong></h4><p>Please remember <a href="https://worldatlas.org/">worldatlas.org</a> is a working prototype made with love, red wine and zero seed funding, so it’s slow and it’s riddled with bugs and things that could be better. I’m a hacky data scientist, UX designer and software developer so most of this project has been minimum effort to get running, and done around full-time work.</p><h4><strong>Finale</strong></h4><p>If I asked you, right now, to compare literacy standards in your country with that of another, where would you start?</p><p>A resourceful person would get there through Google searching, Wikipedia, and perhaps digging up a single-use website, or an old report from somewhere. You’d get an answer eventually but it would be an erratic one-off work around, and the data you find would probably be restricted to a single year and limited list of countries. Others might take a different path to you and end up with different and conflicting data. If you wanted to compare any other statistics, you would start the whole process again. I think we can do sooooo much better than that.</p><p>That same problem took 5 seconds using my site. I rapidly located literacy rates for adults in 2018 via the search, and I can <a href="https://worldatlas.org/Literacy-rate--adult-total-(percent-of-people-ages-15-and-above)/2018/map">display it on a map</a> or <a href="https://worldatlas.org/Literacy-rate--adult-total-(percent-of-people-ages-15-and-above)/2018/bar">rank each country</a> in order. I can also chart how this measure has <a href="https://worldatlas.org/Literacy-rate--adult-total-(percent-of-people-ages-15-and-above)/x/line">changed over time</a> in a line graph, and tag in any available countries I want to see. I can download the raw data in .csv or .xlsx format, view it’s source information, and look at related data (literacy rates for different age groups and sex, for example, or the AVG years spent in school for different age groups) which are grouped together in the navigation and search results. I can also export any of the charts I looked at as PDF, SVG, or JPG files in full form (e.g. all available countries ranked in order).</p><p>To me, this is immensely powerful for learning.</p><p>The level of functionality outlined above would have taken teams of developers to build a few years ago, but the latest tools in data science (such as <a href="https://plotly.com/dash/">Plotly Dash</a>) have massively reduced the effort to bring data to life. I am convinced it’s this new generation of visualisation tools in data science that can punch a hole through the wall of apathy and bring quality data insights to the mainstream public; literally slap them in the face with it. We need to show the public what a ‘front-end’ to the World’s best data could look like.</p><h4><strong>Closing comment (the dream)</strong></h4><p>I’d love there to be a public resource that was free and as fun to use as Encarta that allowed for the visual exploration of important data in an Atlas style way. A family household name and trusted place to visit if you are interested in things like population data, carbon emissions, deforestation, fish stocks, biodiversity, health, economy etc. A learning and education hub, free from advertising, and equally valuable to all ages from children to data professionals. Ever evolving it would be bolstered with new and interesting datasets, and cutting-edge ways of representing the data. There would be something for everyone here.</p><p>This is the dream I have tried to capture with my <a href="https://worldatlas.org/">World Atlas</a> prototype. I can’t see why something like it couldn’t be built properly on the cheap with the open-data and open-source tools we have today. We just need a coordinated effort from a group of talented volunteers, a plan to do it, and $12M USD (jokes).</p><p>As a tech nerd who now works in product, I can categorically say the best product I’ve ever used in my life is Microsoft Encarta. I experienced this product when I was 12 years old and I’ll never forget it. I’ve yet to use anything since that has taken my breath away in the way Encarta did, and I doubt anything ever will. I challenge us all to do better.</p><p><strong>Let’s build products that should exist for the good of all, not just those that make money.</strong></p><p>If you’d like to discuss the Atlas project, or provide feedback and ideas, please contact me at dan@worldatlas.org or <a href="https://www.linkedin.com/in/dan-baker-59441a58">linkedin.com/in/dan-baker-59441a58</a>. This project is open source on Github at <a href="https://github.com/danny-baker/atlas">https://github.com/danny-baker/atlas</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b9ae799e345" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ive-built-a-public-world-atlas-with-2-500-datasets-to-explore-8b9ae799e345">I’ve Built a Public World Atlas with 2,500 Datasets to Explore</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy a beast virtual machine]]></title>
            <link>https://medium.com/data-science/deploy-a-beast-virtual-machine-420b8756190e?source=rss-e5e839f0817c------2</link>
            <guid isPermaLink="false">https://medium.com/p/420b8756190e</guid>
            <category><![CDATA[virtual-machine]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[making-sense-of-big-data]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <dc:creator><![CDATA[Dan Baker]]></dc:creator>
            <pubDate>Mon, 29 Mar 2021 19:59:42 GMT</pubDate>
            <atom:updated>2022-01-07T20:44:14.757Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/making-sense-of-big-data">Making Sense of Big Data</a></h4><h3>Deploy a beast virtual machine for data science</h3><h4>A guide to build a custom data science vm in Azure with Jupyter Hub running out-of-the-box, in a single command.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*-aEIxmqlrHWoVacR5HXb8Q.png" /><figcaption>(Image from <a href="https://www.pixilart.com/">pixilart.com</a> Source: <a href="https://www.pixilart.com/art/little-laptop-1b5bef298dcee31">https://www.pixilart.com/art/little-laptop-1b5bef298dcee31</a>)</figcaption></figure><p>Do you dream of one day having a power beyond your clunky laptop J-Hub or Google Colab session? Fantasize about dedicated hardware with 64+ cores and 500+ GB of RAM for your M/L experiment? What about direct access to Nvidia Tesla T4, M60, V100, and AMD Radeon MI25 GPU rigs, or internet speeds of 1800+ MBit/second to rip down data for processing? (Bottom line: if any of this has your pupils dilating, like me, you probably need to get out more.)</p><p><strong>Today I’m going to turn that nerdy dream into a reality. I’ll show you how to deploy any virtual machine in Azure, preconfigured for data science, with a running Jupyter Hub server, in a single command.</strong></p><p>If you have always wanted to crack into the mysterious world of dedicated cloud hardware, now is the time. There is a new language on the block called <a href="https://github.com/Azure/bicep">Bicep</a>, from Microsoft (launched August 2020). It provides a simple and elegant way to describe Azure cloud components, which you can deploy effortlessly at command-line.</p><p>I’ve built a <a href="https://github.com/danny-baker/deploy-beast-datascience-vm">Bicep template set on GitHub</a> specifically for data scientists. Hand pick your VM specs including model (cores and ram options), choose an OS disk size (up to 4TB), deploy the script with a single command and your virtual machine is up in 90 seconds. It comes with a custom Linux Ubuntu 18.04 OS image specially setup for data science with 150GB of goodies preloaded including native support for Python, R, Julia, SQL, C#, Java, Node.js, F#.</p><p>If you don’t know Linux, don’t worry. Out of the box, this machine runs a Jupyter Hub server giving you instant (secure) access to the machine over HTTPS (with self-signed TLS certificates). As soon as the machine is up, you can access it via JHub. Zero setup. Zero hassle. Zero jokes.</p><p>For the Python JHub fans, I’m sure many of you are frustrated by hardware limitations running from your local machine, or indeed limitations from cloud Notebook services like Google Colab Premium.</p><p><strong>To be crystal clear: this VM is your own private JHub server, with all the horsepower and storage you are willing to pay for.</strong></p><p>Need 500GB Ram, 4TB storage, and 64 CPU cores for 6 hours? Deploy the exact VM you need in seconds and get the job done like a pro. Maybe you and some buddies need a medium-spec rig with 128GB RAM and an Nvidia Tesla T4 for a month; share the cost and get collaborating. Or dip your toes in the water with the Azure 30 day <a href="https://azure.microsoft.com/en-gb/free/">Free Trial </a>(200USD credit) and experiment with a basic 4 core machine, 32GB ram and 1TB of storage. The choice is yours.</p><p>Your virtual machine will run, uncompromisingly, unconstrained, exclusively for you. Listen, and understand. This machine is out there. It can’t be bargained with. It can’t be reasoned with. It doesn’t feel pity, or remorse, or fear. And it absolutely will not stop, ever, until your [credit card is] dead. (Sorry, Terminator. Couldn’t resist.)</p><p>In the following sections I’ll attempt to summarise and contrast the hardware options, and limitations, for the modern data scientist. I’ll provide a quick and dirty crash course in cloud infrastructure (based on my imperfect understanding) and I’ll also profile Azure’s entire VM catalogue through the lens of data science. My goal is to demystify cloud infrastructure (at least for Azure) and paint a picture of what options are out there, when you might want to use them, and how much you can expect to pay.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ro_Qfz_1TnfkKUtlpijF7Q.png" /><figcaption>If only cloud tech was like this. (Image from <a href="https://www.pixilart.com/">pixilart.com</a>. Source: <a href="https://www.pixilart.com/art/rainbow-cloud-c07afa838edaa20">https://www.pixilart.com/art/rainbow-cloud-c07afa838edaa20</a>)</figcaption></figure><p><strong>The Problem: Cloud is scary</strong></p><p>I think the reality is there are tons of data scientists who are frustrated by the limitations of local hardware or free cloud services and yearn to try dedicated virtual hardware, but just don’t know how to start easily. (i.e. The intellectual entry cost is too high)</p><p>Who wouldn’t be scared off by all the ridiculous vendor specific terminology, and difficult-to-compare pricing? And that’s on-top of other scary things associated with ‘cloud’ remote machines like linux, git, ssh-keygen public/private encryption, networking protocols, TLS certificates and VPN security considerations. It’s a literal nightmare and that’s why there is an entire industry forging itself out of abstracting away all this pain to deliver notebook-as-a-service type offerings.</p><blockquote>It’s amusingly fitting that ‘cloud’ technology is just as opaque as the atmospheric clouds from which it derives its name.</blockquote><p>The sad truth is that if you gathered a team of top engineers, locked them in a room for two weeks, and asked them to design the most convoluted, incoherent and unintelligible cloud ecosystem they could imagine, you would get what we have today.</p><p>Instead of naming things after their function, Vendors are incentivised to make up corporate buzz words like ‘EFA fabric adapter’ or ‘infiniband’ (these are real, by the way) to lock in customers. In the short term this might protect market and customer share, but in the long run, I think the lack of functional standardisation of terms is really damaging to the industry. It just shouldn’t be this complicated when fundamentally the hardware is doing the same basic thing. In the automotive industry, every single mechanic in the world calls a ‘fuel pump’ a ‘fuel pump’. It pumps fuel, ergo it is named accurately after its function. Sure there are different brands of fuel pump and some pump at a higher pressure than others, but they are fundamentally doing the same thing. Sadly, this has not happened in the cloud world and I think that obfuscation of reality is really sad. It’s a hard enough abstraction to imagine how cloud virtual hardware works in principle, let alone understand it when every component is named differently by different ‘manufacturers’.</p><p>The second problem is this: hardware isn’t free. And the modern data scientist is almost always hitting hardware constraints. Be it on your local machine, frustrating session time outs in Kaggle Kernals or Google Colab, and constant pressure on ram and storage. Do you upgrade your local machine? Go with managed services (like <a href="https://colab.research.google.com/signup#">Google Colab Premium</a> or the <a href="https://cloud.google.com/ai-platform">Google AI platform</a>) or go forth bravely into the final frontier that is dedicated virtual hardware? How do costs and limitations compare, and how steep will the learning curve be to get in?</p><p>If you need better specs you do need to be prepared to pay for it. Period. But I think if done sensibly Cloud is a more cost effective option than upgrading your local machine, and you can scale well beyond even the best money can buy in terms of local hardware in your bedroom.</p><p>My hope is this essay will give you a broad overview of these issues, and a simple option to get an Azure data science vm up super fast with minimal setup.</p><p><strong>Take Home Messages</strong></p><ol><li>Setting up a dedicated virtual machine for data science is not as terrifying as it used to be, opening the doorway to access monster hardware specs you have always dreamt of.</li><li>Microsoft has released a new language called <a href="https://github.com/Azure/bicep">Bicep </a>which helps you elegantly describe and build their infrastructure. They also released a custom data science <a href="https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/dsvm-ubuntu-intro">OS Image</a>. (Very cool).</li><li>The combination of Bicep and the data science OS image mean you can essentially push a button to build a data science VM in 90 seconds flat, and connect to it via JHub. I’ve <a href="https://github.com/danny-baker/deploy-beast-datascience-vm">built templates</a> to help you do just this: hand pick your vm specs and deploy infrastructure-as-code like a boss.</li><li>Whilst dedicated hardware usually must be paid for, you can experiment using $200 USD credit for 30 days on the Free Trial, with some important caveats.</li></ol><h3><strong>Quickstart</strong></h3><p>If you know what you are doing with deploying Azure resources using ARM templates (and Bicep) clone my <a href="https://github.com/danny-baker/deploy-beast-datascience-vm">GitHub repo</a>, open the appropriate Bicep template file, set your VM specs, create a resource group, and deploy in the Azure CLI with a single command.</p><p><strong>Deploy with a single command</strong></p><p>az deployment group create -f vmtemplate.bicep --resource-group &lt;RESOURCE GROUP NAME&gt; --parameters adminUsername=&quot;USERNAME&quot; adminPassword=&quot;PASSWORD&quot;</p><p>This will use the default template VM specs optimised for Free Account (E-series, 4 cores, 32GB RAM, 64GB temp SSD, 1TB SSD OS disk):</p><p>Or specify vm specs with up to six parameters:</p><p>az deployment group create -f vmtemplate.bicep --resource-group &lt;RESOURCE GROUP NAME&gt; --parameters adminUsername=&quot;USERNAME&quot; adminPassword=&quot;PASSWORD&quot; vmModel=&quot;Standard_E4s_v3&quot; osDiskSize=1000 osDiskType=&quot;Premium_LRS&quot; projectName=&quot;myproject&quot;</p><p><strong>Access vm via JHub</strong></p><p>Once deployed, grab the public IP address of the new vm (from Portal or CLI) access the Jupyter Hub server in the browser via <a href="https://xxx.xxx.xxx.xxx:8000/">https://xxx.xxx.xxx.xxx:8000</a></p><p><strong>Access vm via Secure Shell (SSH)</strong></p><p>Access directly over terminal: ssh USERNAME@xxx.xxx.xxx.xxx</p><p>Notes:</p><ul><li>Bicep templates deploy just like ARM .json templates (you just need to install Bicep first)</li><li>username and password should be wrapped in “quotations” or special characters don’t detect properly</li><li>password needs to be strong (1 capital, 1 number, 1 special char)</li><li>For JHub and other services exposed to the internet, the vm creates a self-signed certificates for HTTPS (TLS/SSL). When you try to connect, modern browsers will still throw a tantrum. Just click through the security warnings and you can connect ok, and be confident that you are accessing the services over an encrypted protocol.</li><li>If you don’t <em>need</em> JHUB and want more secure SSH access to your VM using keys, I have a template specifically for that vmtemplate_ssh.bicep.</li><li>For advanced users, I also have a template for putting your VM behind a premium VPN gateway vmtemplate_vpn.bicep. See detailed instructions later in document or in the GitHub readme.</li><li>Beware that having a VM with a public IP and exposed ports does introduce vulnerabilities to attacks. It’s probably ok for short-term tasks but best practice is to put the VM privately behind a VPN. I’ve provided a template for this in my repo, although note it is for advanced users as it requires manual steps to create root/client certificates. I think the risks of attack are pretty low for short-term jobs and it should be fine to play around with the default setup, but be aware there are risks.</li></ul><h3><strong>Guide for Beginners</strong></h3><p>If you are new to cloud infrastructure and pretending you’re not scared at this point, worry not. Follow this guide and step-by-step instructions below and you will be up and running in no time with a fully dedicated VM.</p><p>Here’s what you’re going to do: in a single command you will compile a Bicep file (high level language describing the cloud infrastructure) into an Azure Resource Manager (ARM) template (a lower level abstraction of the infrastructure in JSON format) and then build the resources needed in Azure. It takes about 2 mins to deploy and you are up and running. If you are new to Microsoft cloud, I’d recommend getting a <a href="https://azure.microsoft.com/en-gb/free/">free Microsoft Azure account</a>, so you can play around within the limitations of the trial at no cost.</p><p>What is really important to stress is that the OS image we’re using for this deployment is fully setup (by Microsoft) for data science. This means literally zero setup (or Linux skills) are needed. It ships with support for all the common languages, and it natively runs an Jupyter Hub server and R Studio lab server as services out of the box! This means you can connect instantly to the VM as soon as it is up, securely over HTTPS.</p><h4>Hardware choices for the modern Data Scientist</h4><p>When choosing hardware, to my mind there are really three main options:</p><ol><li>Local Machine</li><li>Managed Service (e.g. Notebook-as-a-service offerings)</li><li>Dedicated remote hardware (i.e. Virtual Machine)</li></ol><p>Each has it’s advantages and limitations and there many other factors like the ability to collaborate and share work, which lend themselves better to some options. Below I’ve provided a quick snapshot comparison of the well known notebook tools vs dedicated cloud hardware. It’s difficult to know exacts from Kaggle and Colab as many limitations are not exactly specified or published, so I’ve gone from accounts from others etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2eADvYvRwzBdexLbBb0ZyQ.png" /><figcaption>A quick comparison of Kaggle vs Colab vs a cloud virtual machine (in this case Azure).</figcaption></figure><h4>Why virtual machines?</h4><ul><li>Scalability and choice: access hundreds of cores, thousands of GBs of RAM and massive storage.</li><li>Pay for just what you use (billed per second)</li><li>Full control (exclusive control over your hardware)</li><li>Insane internet speeds (I’ve clocked 1,854 MBit/second download speed with a typical 4 core VM)</li></ul><h4>Alternatives to a dedicated virtual machine</h4><ul><li>Your local hardware</li><li><a href="https://colab.research.google.com/notebooks/intro.ipynb#recent=true">Google Colab</a></li><li><a href="https://www.kaggle.com/kernels">Kaggle Kernals</a></li><li><a href="https://notebooks.azure.com/">Azure Notebooks</a> (although I think this is now defunct)</li><li><a href="https://deepnote.com/">Deepnote</a></li><li><a href="https://curvenote.com/">Curvenote</a></li><li><a href="https://www.dataschool.io/cloud-services-for-jupyter-notebook/">Etc.</a></li></ul><h4><strong>How are Google Colab (and similar services) free??</strong></h4><p>The reality is, they are not really. Hardware isn’t free. But hardware can be underutilised and this, I suspect, is exactly how these services can be provided at zero cost to you. What you are getting access to on Kaggle Kernals and Google Colab will be spare capacity from the spot market that is just sitting there unused; the dregs at the bottom of the beer pint. The reason you can’t get a lock on hardware for more than 6–12 hours is probably because this is the cadence that components are typically unused before enterprise customers wake up in the morning and spin up hardware on demand at full price. When this happens you will be kicked off your hardware in favour of paying customers. I don’t know this for sure, but it’s my best guess. The point I’m trying to make here is when you want proper access to hardware you need to pay for it, either through a kick ass local rig, a managed service, or dedicated virtual machine in the cloud. The free services are great, but severely limited for good reason.</p><h4>Can I access Jupyter Hub securely on my virtual machine?</h4><p>YES…well <em>reasonably </em>securely. In fact this is probably the most important thing about this particular setup. Microsoft has done all the work building a special data science linux OS image that runs a Jupyter Hub server automatically. No setup required. They have also handled self-signed TLS certificates which means you can connect to your VM’s JHub server using HTTPS. You will need to click through a security warning in most browsers but this is just because you are using self-signed certificates between the vm and your pc. What this means is you can instantly have JHub up and running on serious horsepower, and collaborate in real-time with others on your new hardware if you wish. I’ll stress again here that there are risks with having a machine with ports open to the public internet. The best option is to put the machine on a private network behind a VPN but this is a lot more complex and is harder to get JHub working (I have a dedicated template for in my GitHub repo for advanced users). For beginners, I think you are fine to play around with a the default VM setup and public IP. Just be aware of the risks.</p><h4>What is infrastructure-as-code?</h4><p>This example demonstrates how to build cloud infrastructure-as-code, which is a way of describing the components you need as ‘code’. That’s it really. Many (or all) of the big players have some kind of API that they use to interpret the infrastructure to deploy. Whether you are building a machine from the browser or direct from <em>code</em>, it’s all being turned into a domain specific format for the vendor to ingest, like a blueprint, in order to build the components and wire them up. Microsoft Azure use things called ARM Templates, which are .json representation of all the infrastructure you want to build. AWS use a .json and .yaml like interface. I’m not sure about Google and the others. Hot off the press, Microsoft has released a new open-source language called <a href="https://github.com/Azure/bicep">Bicep</a> (August 2020) which drastically simplifies the way you describe their infrastructure. It is really awesome and we’re going to be using it!</p><h4>What is actually built when I deploy a virtual machine??</h4><p>It’s important to understand that when you provision a virtual machine there are other cloud resources that are also needed; it’s not just a magical VM that gets provisioned in isolation. To deploy a VM with some ports exposed to the internet, for example, what you are doing in reality within Azure is building a virtual network, subnet within the network, virtual network interface card, network security group (controls things like which ports to open/close), storage account with persistent disk (preloaded with an operating system), the virtual machine itself (which is really the compute cpu-ram component) and a public facing IP address to bind to the network interface card so you can access the VM over the internet. Before you throw yourself off the nearest balcony I promise it’s not too bad once you get the basics. All this magic happens in one clean step so you don’t need to worry about the complexity and can focus on what you do best: the <em>science</em> of data :)</p><p>Here is the network topography just to give you a picture of the end product that you will build in Azure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/0*GXALDwwxsOSIDMGs.PNG" /><figcaption>These are the components you will build, with a single command, to create your data science vm.</figcaption></figure><h4>Making sense of VM Machine Types in Azure</h4><p>I think most heavy-weight data science applications require high in-memory processing, and parallel processing either with CPU or GPU. If you are a hack data scientist like me, you will know that all of life’s problems can be solved with more RAM. As a result I think the VM types of most interest are D/E/M/N Series from Azure.</p><p>My quick and dirty profile of Azure machine types and what to care about for data science applications is outlined in the table below. There are many subvariants so this is just a flavour to give you an idea of specs and cost ranges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCearFdpw6inwZHW-WHo-A.png" /><figcaption>A quick fly-over of the Azure VM classes in respect to data science</figcaption></figure><p>An important thing to note: RAM quantity is scaled based on CPU cores. So you can’t just have a 4 core machine with 256GB RAM, just in case you were wondering. Different classes have different core:memory ratios that is worth understanding. The E-series have the highest memory:core ratio.</p><p>More info <a href="https://azure.microsoft.com/en-gb/pricing/details/virtual-machines/linux/">here</a>. Useful website <a href="https://azureprice.net/">azurepricenet</a> allows you to quickly find the VM family and specific model you need, with pricing.</p><p>For non-GPU applications in data engineering and M/L, I think the D/E series get you a solid all-rounder setup with up to 96 cores and 672GiB RAM in a single instance, plus many options to suit a specific project. For example, an ‘E16as_v4’ will get you 16 x 2.35Ghz cores, 128GiB RAM and 256GB of temporary SSD storage for about $1USD/hr.</p><p>If you are doing something crazy, the M series are straight out beasts and single instances can clock out to 416 cores and 11,400Gb RAM. I mean I don’t know what you would use these for in data science. These are (to be fair) more suited to hardcore enterprise applications. But they are there.</p><p>And for the rapidly evolving deep-learning folk, the new <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu?context=/azure/virtual-machines/context/context">N-series</a> are for you. There are a number of variants and classes within but in essence you can customise the VM to get fractional access to GPU (e.g. 0.25GPU core to 4 dedicated GPU cores per node). You have direct access to Nvidia Tesla T4 and M60, Volta V100, AMD Radion MI25 bolted to vms with latest generation CPU core banks from 4–64 cpu cores. I should stress these are not available in the Free Trial, you <em>must </em>go to a paid plan. These are serious and I imagine what many of you might want to try.</p><p>It’s also worth mentioning here that Microsoft isn’t known as the big player specifically in GPU cloud offerings. It’s fair to say this is Google and others. Bottom line is most of them offer GPU options and if this is critical to your work, then it’s worth researching options. In any case, I think Azure is a great option for trying out dedicated cloud hardware because of the ease of deployment now with the new Bicep language.</p><p>Another important thing worth noting is that on a standard PAYG account you won’t be able to provision a beast out of the box. All Azure accounts have soft and hard vcpu (core) quotas, so if you want anything beyond about 4 cores you will need to lodge a service desk request for a quota increase, which can take 48 hrs to process. You can check your current limit based on a given region for all VM types in the Az CLI with az vm list-usage --location &quot;East US&quot; -o table. More info <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/quotas">here</a>.</p><h4>Where does my VM live?</h4><p>Microsoft has data centres across the world which you can visualise on this <a href="https://azure.microsoft.com/en-gb/global-infrastructure/geographies/">map</a>. Your VM will live in a data centre of your choosing based on the location of the ‘resource group’ that you will create. An Azure resource group is simply a convenient bucket to put all your resources in, much like a folder on a your desktop computer. You can control access at the ‘folder’ level and remove all its contents by deleting it in one go. There are marginal price differences between regions, but for this use case, the most important factor is choosing the location for your resource group that is closest to your present location. This will minimise latency between you and the machine. For example “Central US” or “UK South”.</p><h4>Can I just build a data science VM using the Azure portal in browser?</h4><p>YES. In fact, I’d recommend you build your first VM using the <a href="http://portal.azure.com/">Azure portal</a>, selecting the <a href="https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/dsvm-ubuntu-intro">data science OS image</a>. This is exactly the same OS image as I’m using in this build template. There are a few limitations to using the portal so you can’t specify as many options but you can definitely get it up and access your vm on JHub etc. I hope this guide shows you how easy it can be to deploy infrastructure as code which is what is actually happening behind the scenes when you deploy from the Azure Portal anyway.</p><h4>How am I billed for my VM?</h4><p>You pay by the second. And yes, leaving the VM on will rack up your credit card faster than tequila shots on a Friday night (you are protected on the Free Account, don’t worry).</p><p>Most of the other infrastructure is essentially free (the virtual network, subnet, public IP, etc). The key costs are the COMPUTE (the virtual machine) and STORAGE (the persistent disk attached to it).</p><p>Note the hourly rate is PAYG and if you have an ongoing demand for a vm, you can usually reduce the on-demand price by 60–70% on a 3yr reservation (which you can cancel at any time). Many VMs are also available on spot pricing which is alarmingly attractive. Don’t fall for this. I don’t think this is a good idea for data science applications because you typically require long uninterrupted processing pipelines. On the spot market, your VM can be pulled without warning and ruin all your work. PAYG is the only way to get guaranteed exclusivity while using the resource.</p><p>Golden rule is: always remember to turn you vm off (deprovision) or tear down all resources (delete everything) You can temporarily deprovision it, preserving the OS disk which is just like switching off your PC or you can delete the whole resource group when you are finished.</p><h4>What is the best VM available on the free trial?</h4><p>The 30 day free trial gets you 200 USD of credit which is great, but note some important limitations below:</p><ul><li>Max cores = 4 per region (meaning no big bad wolf VMs on free account)</li><li>No access to GPU VM series (upgrade to PAYG account to access N-Series GPU optimised vms, starting at about 1USD/hr)</li></ul><p>Most beastly setup on free account:</p><ul><li>E-series ‘Standard E4s_v3’ (the default in my template)</li><li>4 cores (Intel Xeon Platinum 8272CL base core clock 2.5GHz which can increase to 3.4Ghz all cores)</li><li>32GB ram</li><li>1TiB Premium SSD disk (Fastest OS disk type)</li><li>Insane internet speeds (Typically 1000+ MBit/second)</li></ul><p>This package will burn ~$10USD credit/day and you can run it full throttle 24–7, uninterrupted with no cpu constraints for 20 days until your free credit depletes. You can set the OS disk size to anything up to 4,095GB (4TiB) but 1TiB maximises storage vs credit for the 30 day trial.</p><h4>How does storage work with vms?</h4><p>All VMs need a managed persistent disk for the OS image. You can attach additional disks (several usually) and mount them on the filesystem but note this is fidly if you are not comfortable with linux. By far, the quickest and easiest option is to just beef up the OS disk size (up to 4TiB SSD) to what you need for the task at hand as this is mounted automatically.</p><p>It’s also worth nothing that many VM classes offer temporary high speed storage. This is usually located geographically close to your compute hardware (cpu cores) in the data centre and is basically the fastest storage you can get. I think this is sometimes referred to as ‘scratch’ storage. Note it’s ephemeral and only lasts a few days so is useful for data processing stages. The temporary storage is automatically mounted on your VM at location /mnt if your VM class supplies it.</p><h4>How do I transfer data to and from my VM?</h4><p>This is a proper remote Linux machine so it’s not as straight forward as copy-pasting documents to your windows file system running JHub on your laptop etc. But, the beauty is that you can do most of your uploading/downloading directly through the Jupyter Hub server by way of the browser, which should cover most people’s needs. Don’t forget you can rip down data from the internet at lightening speeds from your VM. Because your VM is sitting in a data centre, it typically had D/L speeds ten times faster (1800+ Mbit/s) than on your home internet or Colab.</p><p><strong>Getting data into your VM:</strong></p><ul><li>From your pc: use the JHub upload feature in browser or tools like rsync from a linux terminal (requires you to have WSL in windows)</li><li>From the internet: wget/curl/github (just like you would in cloud notebook tools)</li></ul><p><strong>Getting data out of your VM:</strong></p><p>This is a little trickier than you think because the machine is running Linux and you probably are running Windows, but I have few standard recommendations. And it also depends where you plan to store the data and how much data you have.</p><ul><li>small (&lt;30GB): use the JHub file download feature in browser (the simplest)</li><li>medium: (&lt;1TB): use a linux data transfer application like rsync (requires linux on your local machine, native for MacOS but Windows you will need <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">WSL</a>)</li><li>large: (+1TB): transfer direct to a cloud data storage service like Azure Blob, or Network File System (NFS). This will allow you to rapidly get data out of the VM and shut it down (to save $$), then you can connect directly to the cloud storage service to either download or store the data more long term.</li></ul><h3>Instructions</h3><p>Now that the crash course is complete, we can start with the step-by-step guide to deploy your vm infrastructure as code. Note this is all detailed in the my <a href="https://github.com/danny-baker/deploy-beast-datascience-vm">GitHub repo</a> README.md</p><h4>Prerequisites</h4><ul><li>Microsoft Azure account (e.g. <a href="https://azure.microsoft.com/en-gb/free/">Free Trial</a> or pay-as-you-go)</li></ul><h4>1. Install VS Code and Bicep/ARM extensions (OPTIONAL)</h4><p>This is preparing us for opening and editing the project files, primarily the vmtemplate.bicep file or similar templates. Microsoft Visual Studio code is great for this project as it is open source and has downloadable extensions for bicep and ARM templates, meaning it colours the code really nicely to make it more readable. Download and install <a href="https://code.visualstudio.com/">VS Code</a> with <a href="https://marketplace.visualstudio.com/items?itemName=msazurermtools.azurerm-vscode-tools">ARM Tools</a> and <a href="https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-bicep">Bicep</a> extensions. Note the extensions can be easily installed from within VS Code once it&#39;s running. Of course this is optional. You could use any editor (e.g. Notepad++, VIM, etc.) And you don’t need to edit the files at all because you can pass vm specs as parameters in the build command.</p><h4>2. Install Azure CLI</h4><p>What is the Azure CLI? It’s a dedicated program that provides a command line interface (i.e. CLI) for interacting directly with Azure resources. This means you can build and tear down infrastructure at command line like a boss, rather than doing it from the web browser portal. The most straight forward way is to <a href="https://docs.microsoft.com/en-us/cli/azure/install-azure-cli">install the Azure CLI</a> which can run on Windows, MacOS, and Linux.</p><p>Do I absolutely need to use the Azure CLI? For this complete example, Yes. Because there are some limitations with the portal interface. In general: No. If you just want to build a VM using a username/password with the data science OS image, you can do it all from the portal as a one-off if you need to.</p><h4>3. Install Bicep</h4><p>Bicep is a cool new domain-specific-language for deploying Azure resources in a much more simplified way than using ARM templates, which are in JSON format and can be painful to read. Bicep files compile into Azure Resource Manager (ARM) templates, which can then be directly ingested by Azure to build infrastructure. Bicep scripts, therefore are a high-level abstraction on top of ARM templates that simplify the description of the infrastructure you want to build. It only takes a few mins to install Bicep. Follow the installation guide <a href="https://github.com/Azure/bicep">here</a>.</p><p>From within the Azure CLI latest version it is basically:</p><p>az bicep install</p><p>az bicep upgrade</p><h4>4. Clone this repo</h4><p>Copy this project repo to your local machine either from Github over browser as a straight download, or via git clone etc. If you are not familiar with GitHub and git, really, you just need to get the vmtemplate.bicep file to your local machine where you can access it from within the Azure CLI.</p><h4>5. Configure VM specs (the fun part)</h4><p>Open the vmtemplate.bicep file in visual studio code.</p><p>At the beginning of the file I’ve summarised all the knobs and dials you might want to turn to tweak your VM settings. By default, I’ve chosen the toughest setup I could find within the limits of a Free Account (This is an E4s_v3 with 32GB RAM, 64GB temp SSD storage, and a 1TB Premium SSD OS Drive). But of course you can mix and match practically any VM up to 4 cores on the free account, and experiment with different HDD sizes and types.</p><p>You can either modify the default vm spec values in the .bicep file itself, or override the default values by passing the desired values in as parameters in the build command. Totally up to you. Examples provided further on.</p><p><strong>Key decision points:</strong></p><ul><li>VM Model — This is critical as it determines the number of cores, RAM, temporary storage, and other limitations in relation to I/O. There are literally hundreds of options. Lookup what you want either on <a href="https://azure.microsoft.com/en-gb/pricing/details/virtual-machines/linux/">Azure docs</a> or on <a href="https://azureprice.net/">azurenet</a>. You just need the text string, such as “Standard_E4s_v3”.</li><li>OS disk size — Default to 1TB premium SSD, but you can choose anything up to 4TiB (4095GB) as a single disk.</li><li>OS disk type Take special note there are 3 distinct classes of storage ‘Premium_LRS’ which is SSD, ‘StandardSSD_LRS’ which is constrained SSD media, then the good old fashioned hard disk drive ‘Standard_LRS’. Standard SSD is half the price of premium ssd, and standard HDD is 1/4 the price of premium ssd. Refer to docs <a href="https://azure.microsoft.com/en-gb/pricing/details/managed-disks/">here</a>. In all datascience applications, I’d use nothing other than “Premium_LRS” for maximum performance.</li></ul><h4>6. Login to Azure</h4><p>From the azure CLI:</p><p>az login --use-device-code</p><p>This should open the browser and get you to punch in a code displayed in the CLI</p><p>Check accounts available</p><p>az account list --output table</p><p>Set subscription (if default account is not correct)</p><p>az account set --subscription &lt;name&gt;</p><p>List resource groups (to view any existing resource groups in your subscription)</p><p>az group list --output table</p><p>OPTIONAL: Permanently set Azure CLI output format to table which is way more human readable than JSON, which is the default. I highly recommend doing this.</p><p>az configure (and follow prompts. Ensure you select output as table format)</p><h4>7. Create Azure resource group</h4><p>Create a new resource group in a region that is geographically close to your current location.</p><p>To view available regions:</p><p>az account list-locations</p><p>In this example I’m creating a resource group called “beast” in the “Central US” region. It usually takes a few seconds.</p><p>az group create --name beast --location &quot;Central US&quot;</p><p>Check resource group is created</p><p>az group list --output table (if you have changed your Az CLI configuration, you don&#39;t need to append the --output table every time)</p><p>After a few seconds, it should appear. You can check in portal.azure.com directly by searching for ‘resource groups’ or typing az group list in CLI.</p><h4>8. Setup access</h4><p>We are almost ready to construct the resources. The final thing we need to do is setup how you will access the machine. There are two main ways you can do this: username/password credentials OR SSH public/private key encryption. I have a separate bicep file for each option. I’d recommend user/pass default for now, as this is the option that JHub is compatible with.</p><p>Choose a username and password</p><ul><li>Note passwords must contain at least 1 uppercase, 1 number and 1 special character.</li><li>I’d suggest you generate a strong password with something like <a href="https://keepass.info/">keepass</a>.</li><li>For the default template we will be using you must create a username and password. This is because Jupyter Hub requires a user/pass and does not support SSH keys. And because I assume most people will want to run notebooks on their VM, I’ve setup the template to allow a username/password, which is not really the most secure way to connect to a Linux host.</li></ul><p>OPTIONAL: Create SSH key-pair</p><p>Only applicable if you do not plan to use JHub and want the most secure way to access your VM. If you are running linux, WSL in Windows or MacOS and you have basically a linux terminal, you can create public/private key encryption files for secure shell access (SSH) to the VM. This is the safest way to access it, although note that Jupyter Hub does not support it. So no matter what, if you are planning to use JHub mainly, you will need to use the recommended vmtemplate.bicep template. From JHUB you can access a full root terminal to do whatever you need. So this is really only for more hardcore ppl that want to be able to directly SSH into the VM with encrypted keys. If you definitely want this option, create SSH keypair have have public key ready to be copied.</p><h4>9. THE GREAT BUILD</h4><p>This is the moment you have been waiting for: we are ready to build the infrastructure. From the Azure CLI ensure you navigate to the current working directory where the vmdeploy.bicep file(s) reside. You must also be logged into your Azure account via the Az CLI (step 6). Build the VM by running the appropriate command below, passing in the parameters for username/password or public key.</p><p>Let&#39;s assume you decide on username: jamesbond / password: G0|den3y3 and have already created a resource group called &#39;beast&#39;.</p><p><strong>Deploy with username/password</strong></p><p>Build using default VM specs in the .bicep file:</p><p>az deployment group create -f vmtemplate.bicep --resource-group beast --parameters adminUsername=&quot;jamesbond&quot; adminPassword=&quot;G0|den3y3&quot;</p><p>Build passing custom VM specs as parameters:</p><p>az deployment group create -f vmtemplate.bicep --resource-group beast --parameters adminUsername=&quot;jamesbond&quot; adminPassword=&quot;G0|den3y3&quot; vmModel=&quot;Standard_D2s_v3&quot; osDiskSize=180 osDiskType=&quot;Premium_LRS&quot; projectName=&quot;myproject&quot;</p><p><strong>Deploy with SSH public key</strong></p><p>az deployment group create -f vmtemplate_ssh.bicep --resource-group beast --parameters adminUsername=&quot;jamesbond&quot; adminPublicKey=&quot;INSERT FULL ASCII PUBLIC KEY HERE&quot;</p><p>Notes</p><ul><li>For SSH, you are calling a different bicep template file called vmtemplate_ssh.bicep.</li><li>Always encase the username and password (and public key) in inverted commas to ensure special characters parse properly. Sometimes you will get an error without them.</li></ul><p>If it worked you should see something that looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/0*1mqDr13-QC9Roycy.PNG" /><figcaption>If you see something like this in the Azure CLI after patiently waiting for 2 minutes you are looking good.</figcaption></figure><p>Login to Azure Portal and take a look at your new infrastructure!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DLSzSHwco_pS9UjG.PNG" /><figcaption>Navigate to the resource group in portal, and you can see all the new components created (on right)</figcaption></figure><p>Grab the public IP address of your VM</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*79U9OXFtK2-KevIN.PNG" /><figcaption>Click on the VM itself, to view it’s Public and Private IP address (top right)</figcaption></figure><h4>10. Connect to the machine over the browser via Jupyter Hub!</h4><p>Your new VM has a bunch of services preconfigured. After deployment, it immediately runs a range of containerised services (via Docker) including a Jupyter Hub service exposed on port 8000. Jupyter Hub is (in part) a webserver and so you can directly connect to it from any browser over the internet.</p><p>First get the IP address of your new vm:</p><ul><li>From Azure Portal: search for resource group, navigate to the correct group, click on the VM (and you can see the public IP top right)</li><li>From Azure CLI: az vm show -d -g &lt;RESOURCE GROUP NAME&gt; -n &lt;VM NAME&gt; --query publicIps -o tsv</li></ul><p>Open a browser and access Jupyter Hub webserver (which is running on your vm exposed via port 8000), where the IP address is substituted for the x’s.:</p><p><a href="https://xxx.xxx.xxx.xxx:8000/">https://xxx.xxx.xxx.xxx:8000</a></p><p>Navigate past the browser security warning. Your vm has generated it’s own self signed SSL certificate to allow encrypted browser traffic (HTTPS). However, as this is not a public certificate, the browser will often kick up a warning when you first connect. Don’t worry and you can usually click accept the risk, and ‘go to site’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SHZRqKNdmeaEoieW.PNG" /><figcaption>You are still transferring securely data via HTTPS it’s just you are using a self-signed certificate from the Linux VM which is not publicly recognised by the browser.</figcaption></figure><p>Once through this, you should then see a Jupyter hub login screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HzbaJH9MeKrGdOWP.PNG" /><figcaption>If you get to this screen. Start celebrating.</figcaption></figure><p>Login to Jupyter Hub with the username and password you supplied for the VM at deployment</p><p>If it has worked, you will see the Jhub session that looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JmfWbmiUtIr9jln3.PNG" /><figcaption>Jupyter Hub session hosted by your remote VM running in your local machine browser!</figcaption></figure><h4>And you are IN. At this point you can start playing with notebooks or collaborate with buddies to data science it up!</h4><p>I sincerely hope this has been useful to someone; it certainly would have been useful to my past-self a year ago. Please note in my <a href="https://github.com/danny-baker/deploy-beast-datascience-vm">Github repo</a> I’ve got extra details at the end of the READEME.md for extra nerd things like how to put your vm behind a VPN (if security is paramount) and how to test out some basic specs of your VM like internet speed, check processor and RAM utilisation if you are unfamiliar with Linux etc.</p><p>Final caveat is I am not an authority on cloud, I just happen to have spent six months working with various Azure VMs and felt it was time to share some of what I’ve learned along the way.</p><p>I welcome corrections and clarifications, and I’d love to see a similar write up from someone who knows AWS and Google Cloud.</p><p>Enjoy ❤</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=420b8756190e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deploy-a-beast-virtual-machine-420b8756190e">Deploy a beast virtual machine</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying your Dash App to Heroku — THE MAGICAL GUIDE]]></title>
            <link>https://medium.com/data-science/deploying-your-dash-app-to-heroku-the-magical-guide-39bd6a0c586c?source=rss-e5e839f0817c------2</link>
            <guid isPermaLink="false">https://medium.com/p/39bd6a0c586c</guid>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dash]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[gunicorn]]></category>
            <dc:creator><![CDATA[Dan Baker]]></dc:creator>
            <pubDate>Mon, 23 Nov 2020 00:50:19 GMT</pubDate>
            <atom:updated>2026-05-09T07:18:14.967Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XUI3DmnqWVQgBQNTgCnyEQ.png" /><figcaption>(Image by author, inspired by <a href="https://www.youtube.com/watch?v=CsGYh8AacgY">Charlie the Unicorn</a> and <a href="https://gunicorn.org/">Gunicorn</a>)</figcaption></figure><h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h3>Deploying your Dash App to Heroku — THE MAGICAL GUIDE</h3><p>Update: In the years since writing this article, Plotly has built its own solution for deploying and sharing Dash apps in seconds, which requires zero configuration. Check out <a href="https://plotly.com/cloud?utm_source=towardsdatascience&amp;utm_medium=external_blog&amp;utm_campaign=studio_cloud_community&amp;utm_content=deploying_your_dash_app_to_heroku_magical_guide">Plotly Cloud</a>. It’s favoured by the warm and welcoming <a href="https://community.plotly.com/">dash community forum</a> for 1-button deployments and avoiding the <em>pain </em>that inspired this article. Read on for some comic relief on my Dash journey as well as some other useful fundamentals.</p><p>So you have your <a href="https://plotly.com/dash/">Dash </a>app running on your local machine and you’re finally ready to share it with the world on a public site.</p><p>The problem is: words like like Git, Flask, Gunicorn and Heroku sound like strange mythical creatures, even after a few drinks. Worry not: having just been through the process of deploying Dash to Heroku myself for the first time, I’ll share what I’ve learned along the way. I’ll outline some surprising pitfalls and solutions I found, in the hope this will save you time and effort. This is the guide I wish I had when I started.</p><p><strong>Quickstart<br></strong>I’ve built a <a href="https://github.com/danny-baker/dash-heroku">Dash-Heroku starter pack</a> on Github. This is a barebones Dash app fully setup and tested that runs on Heroku. It has all the necessary special files and tweaks so you can serve static files (such as images) which is the most common pain point. For first timers, I’d recommend you get this demo running on Heroku first, and then transfer your locally running Dash project code carefully over to it :)</p><p><strong>Background</strong></p><p>The process to successfully deploy a Dash app on Heroku is not trivial for many first timers. For example, simply serving static files (like documents, audio, images, video) doesn’t work out of the box with Heroku like it does from your local machine. I didn’t realise this, along with a few other quirks.</p><p>This essay is designed to supplement the existing documentation, and attempts to fill in some gaps, explain the quirks, and provide a very brief fly-over of each component and how everything fits together. I’ll share notes from my personal experience. I’ll also attempt to explain the technology stack based on my imperfect understanding of how it all works. No doubt I’ve got some things wrong, so I welcome corrections from the community.</p><p><strong>Assumptions</strong></p><ul><li>You have a running Dash app locally hosted (with a requirements.txt)</li><li>You’re relatively new to Python</li><li>You have a strong cup of coffee (better yet, a glass of red)</li><li>You’ve never deployed a public web app before</li><li>Words like Heroku and Gunicorn scare you.</li></ul><p><strong>The Problem</strong></p><p>In my research I scanned many blog articles and guides about deploying Dash to Heroku, but found most to be a little lacklustre or not specific to Dash. YouTube videos, too, are often super quick (do it in 15 minutes!!) and don’t explain the core concepts. The majority of the guidance I’ve seen so far is bare minimum, light on explanation, and doesn’t outline key issues you’ll encounter in the detail you need. If you’re working with Dash, chances are you are new to Flask as well, so many concepts are not well understood. I’ve yet to read a guide that outlines the core concepts and pitfalls for <em>specifically </em>deploying a Dash app to Heroku. I believe this lack of comprehensive written guidance is a (solvable) barrier to entry for many potential Dash users.</p><p><strong>How much pain is this going to be? (about 10 hours)</strong></p><p>How hard really is it to Deploy to Heroku as a first timer? How much pain is involved? What problems will you encounter? I’m glad you asked. In short, I’d say optimistically it can be done in a few hours ground-up, but realistically about 10hrs for a first timer. Pain meter: medium spicy.</p><p>This is because it takes <em>time</em> to create the accounts and setup the environments like GitHub and Heroku command line interface, add special new files to your project directory (repository), modify the code in a few spots, and get used to the commands you need in order to see what’s going on and make it all work. It can be a little daunting at first, but once you’ve done the initial setup, you’re on easy street. Deploying code running on your laptop to your live public web app is a few mouse clicks and single command in a terminal, which is super cool. (Caveat: this does not cover security and authentication — this is purely to deploy a hobby app).</p><p><strong>Why Heroku?</strong></p><p>Like myself, you’ve probably heard of Heroku as a well-loved platform for deploying hobbyist web applications for free. Of course, it’s not the only one — there are many, and it is scalable to enterprise-level deployments. But it is universally known and liked by the community, so I chose to go this route and see what all the fuss was about. Verdict so far: <em>loving it</em>. Key reasons the community loves Heroku (as far as I know):</p><ul><li>It’s free for hobby apps, which is great to get started and for demos</li><li>It’s got clear, concise documentation</li><li>It natively supports Python web apps (read: minimal config needed)</li></ul><p><strong>What is Heroku?</strong></p><p>It’s a platform-as-a-service (PaaS) for deploying and hosting web applications. In the context of your Dash app, this means Heroku provides the physical hardware (storage, compute), software services (linux/unix/sql), and dependencies (packages), in a containerised environment to deploy and host your application on a publicly accessible URL (end-point). It does this through provision of virtualised linux containers called “Dynos” which essentially act as your personal linux webserver, with ram, cpu, and linux ‘installed’. (It’s not quite like this in reality but a good analogy).</p><p>Dynos come in a variety of shapes and sizes and can be scaled vertically (more ram, compute, storage per instance) or horizontally (duplicate dynos in parallel) as your specific project requirements demand. You can hot swap dyno types instantaneously at command line while your app is live, which is mint. The free version gets you one dyno with up to 500MB storage and 500MB ram. It sleeps after 30 minutes of inactivity, presumably so Heroku resources are not drained. So the catch with the free version is that your website can take a good 30–60 seconds to load initially, as your free Dyno is provisioned on demand. If you go to a paid plan, starting at about $7USD/month, your dyno(s) stay on and ready 24hrs/day.</p><p>The biggest constraint with Heroku dynos is RAM; you don’t get much. At least in my experience, I’ve found that Python (Dash) apps often require a decent amount of RAM as you are reading in Pandas dataframes and doing all sorts of fun stuff. It’s fine for a single instance of your app for a select small group of users, but if you want to scale out you will need to run duplicate parallel instances of your app, and this will cost you $$ in RAM to either run more dynos in parallel, or multiple instances of your app in larger tier dynos.</p><p><strong>Why GitHub?</strong></p><p>In short — Heroku natively supports deploying repositories that reside in GitHub. This is good news. Basically it means if your project is already in a GitHub repository (free for private/public repos) then you can easily deploy it on Heroku AFTER you have added a few additional files that are outlined in the deployment guides, and in a section further below.</p><p>If you’ve been developing your pet project on a local machine, before you can deploy on Heroku you first need to get your project onto GitHub. (Note you don’t have to use GitHub, but it’s an easy option.) This is an important step to take your code to a public (or private) cloud repository. It’s a good move anyway because you have full versioning history, you are protected from hard drive failure, and you can share publicly or privately. It does, however, come with intellectual debt, with some interesting concepts and terminology to get your head around, like clone, fork, merge, push, pull, and commit.</p><p>Yet another barrier to newcomers is the issue with security and how this affects you accessing and changing the code in your cloud repository on GitHub or similar. Basically, GitHub wants a secure connection between your computer and its servers before it will happily accept code changes. There are two main ways it achieves this: using credential authentication over HTTPS (requiring a username/pass every time a connection is made), or via SSH public/private key encryption, which is not natively supported by Windows. This extra complication, combined with the other scary words like clone, fork, and merge can be a little overwhelming at first. Fear not: there’s a desktop app that can setup a secure connection between itself and your GitHub repo, facilitating seamless easy updates to your code repository.</p><p>If you’re developing on a windows/mac machine (which I assume the majority of first timers are), I’d highly recommend getting the <a href="https://desktop.github.com/">Github Desktop </a>application. This just makes the process of cloning, fetching and pushing changes back to your repository on Github <em>MUCH</em> easier without the need for any command line. It’s not that I’m against command line, it’s just that this particular process can be clunky on windows, requiring either user credential authentication or SSH keys mentioned above.</p><p>(Special note: if you are more hardcore you can of course install <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">Windows Subsystem for Linux</a> which is a way to have a fully functioning Linux system on Windows without the need for dual boot options etc. If you do this you can setup SSH keys and enjoy the benefits of linux for managing your code repo but it’s really not necessary for first timers.)</p><p>In short, you can avoid a <em>lot</em> of hassle just by using the Desktop app, and setting up the security within the app — then it’s single click of the mouse to commit changes and push updated code to your remote repo on GitHub.</p><p><strong>What is Git?</strong></p><p>No one really knows.</p><p><strong>What is Gunicorn?</strong></p><p>When you figure it out, let me know. <a href="https://gunicorn.org/">Gunicorn</a>, to the best of my understanding, is a production-ready HTTP server specifically for Python web applications which runs natively in Unix; it’s the thing that actually <em>serves</em> the web browser request. If you’ve been developing your Dash app purely on your local machine @ ‘localhost:8080’ or ‘http://127.0.0.1:8050/’ you will be running a light weight HTTP server that is shipped with your Python installation. This is not Gunicorn. It’s likely you have not yet glimpsed this rare and mythical creature of the forest.</p><p>The local HTTP server (shipped with your Python installation) is automatically run by your Python Kernel when your dash app is executed on your local machine. The issue is that it’s not designed for handling incoming traffic from a production website and so when you deploy to the web, you need a production-ready HTTP server. A popular one is Gunicorn. Notably, Heroku provides native support for Gunicorn, which makes things easy. It’s all outlined in the guides, but just to clarify, all you need to do is add a single line of code to your dash app (‘server = app.server’), add Gunicorn into your requirements.txt so it is installed as a package on your local machine (and by Heroku during deployment), and reference it in a special file you will create called the Procfile. More on this later but I think it’s worth briefly touching on the HTTP server as it’s all a bit mysterious the first time.</p><p><strong>What is Flask?</strong></p><p>Before cups and bottles, in the medieval times people used to drink out of flasks. This of course has no bearing whatsoever on what Flask is in the context of Python, but hold onto the drinking analogy. Flask is a micro-framework for Python web applications. If you are not familiar with frameworks and why they matter, essentially they provide a structure (a pattern) for you to develop your project in a way that supports growth and multiple collaborators. Some are rigid and some are more flexible, but the ultimate goal is to help a complex project become manageable by separating functional components of code (and other things) into logical places, as well as providing tools and libraries to assist with common tasks (e.g. setting up sql databases, accessing the underlying host operating system, user authentication etc.)</p><p>Popular frameworks in Python include Flask and <a href="https://www.djangoproject.com/">Django</a>. In Ruby, you have probably heard of <a href="https://rubyonrails.org/">Ruby on Rails</a>, which is a well known framework for developing Ruby web apps. If you are sticking to a bare minimum Dash app, you don’t need to worry about frameworks, but just note they are for proper, serious, software development and they are important. Your Dash app is in reality still a Flask app (Dash sits on top), Flask has just been hidden away behind the scenes. The reason Flask is popular is it is non-rigid, and does not enforce strict directory structures and file names. It can also be used piecemeal, so parts of it can be plucked out as needed, keeping the project as simple as it can be while retaining features. This is probably why Plotly chose it for Dash! To return to the ‘drinking analogy’: unlike many other frameworks, Flask as a framework can be ‘sipped’ as it is needed to support just the features that are what is needed for a given project, and no more.</p><p><strong>Web is hard</strong></p><p>This is a simple truth. Web is multi-layer, multi-language, multi-protocol, multi-platform, and multi-user. It’s a mind-boggling chain of infrastructure bolted to other infrastructure to make a modern (scalable) web application run. For many non-IT people (and IT people for that matter), even the concept of a locally hosted webserver takes a bit of abstract thought, let alone understanding the true technology stack that lies underneath a real client-server architecture. It’s also worth reflecting on just how new some of this technology really is, so I’ve indicated the year these tools were created in the table below.</p><p><strong>The simplified technology stack</strong></p><p>This is imperfect, so please help me to correct it. But it’s useful, I think, to see some of the layers required to get your code deployed onto the web. We start with your actual code at the very top of the stack, and drill down layers all the way to Heroku.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0019bce6919d7a8bceb70c5a1274ff80/href">https://medium.com/media/0019bce6919d7a8bceb70c5a1274ff80/href</a></iframe><p>The point I’m trying to make here is that this grossly oversimplified web technology stack is still far from simple; to say nothing about front end layers such as Javascript, CSS etc. Web is hard because of the sheer number of abstraction layers.</p><blockquote>Dash, to me, is a beautiful abstraction at the very top of the technology stack — building upon everything below it — in order to simplify what is actually an insanely complex machine: the modern data-rich web application.</blockquote><p><strong>Dash-Heroku deployment, in a nutshell</strong></p><p>What actually needs to be done:</p><ol><li>Dash app running on localhost</li><li>Install <a href="https://git-scm.com/downloads">Git</a></li><li>Setup GitHub account (+ recommend install GitHub Desktop)</li><li>Setup Heroku account (+ install the command line interface)</li><li>Add dependencies and special files (i.e. install and import Gunicorn, create Procfile and runtime.txt)</li><li>Clone repo from GitHub to local machine (only once)</li><li>Create Heroku app linked to your repo (only once, ref deployment guides, Heroku CLI)</li><li>Commit and push your code changes to GitHub repo (repetitively)</li><li>Deploy/Re-deploy Heroku app by pushing changes (“git push Heroku main”)</li></ol><p>Just to be super clear here for those that are very new to this process. Once you’ve created a Heroku app linked to your repo, this means that when changes are pushed to your Heroku remote repo on Github, this will actually trigger a Heroku build of your app. What that means is, when you type git push heroku mainthis is pushing your local code to your remote heroku repository on github (on a branch called main, which is the default primary). As soon as this happens, this will trigger a build and you should see this in your terminal (console) immediately after you type the command.</p><p><strong>Deployment Guides</strong></p><p>The guides below are concise and useful, and I would of course start with these. If I’m honest, I think they are a little light on detail for newcomers and would benefit greatly by having a supplementary <em>explanatory guide</em> akin to something like this essay.</p><ul><li>Plotly’s <a href="https://dash.plotly.com/deployment">Dash deployment guide</a></li><li><a href="https://devcenter.heroku.com/articles/getting-started-with-python">Heroku’s guide</a></li><li>Recommendations: Install <a href="https://devcenter.heroku.com/articles/heroku-cli">Heroku command line interface</a> (CLI) and <a href="https://desktop.github.com/">GitHub Desktop</a> if a windows/mac user</li><li>Also, this YouTube <a href="https://www.youtube.com/watch?v=b-M2KQ6_bM4&amp;feature=youtu.be">tutorial</a> from a fellow <a href="https://community.plotly.com/">Plotly Community Forum member</a>.</li><li>My <a href="https://github.com/danny-baker/dash-heroku">Dash-Heroku </a>starter pack on GitHub is a running base to build on.</li></ul><h3>The magical ingredients to add to your project</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*UDbTISUcXgNXNpwTEV130Q.png" /><figcaption>(Image from <a href="https://www.pixilart.com/">pixilart.com</a>, Source: <a href="https://art.pixilart.com/6977a966cd0f6a7.png">https://art.pixilart.com/6977a966cd0f6a7.png</a>)</figcaption></figure><p>A quick note on the special files you need uniquely to get your python project deployed to Heroku. This is outlined in the deployment guide, so I’ve just provided a few notes from my experience:</p><p><strong>Ingredient 1: Procfile</strong></p><p>This strange extensionless file must reside in your project root, and tells Heroku how to handle web processes (in our case using Gunicorn HTTP server) and the name of your Python application.</p><p>Typically the Procfile would contain a single line:</p><pre>web: gunicorn app:server</pre><p>Where:</p><ul><li>‘web:’ tells Heroku the dyno main process is a web process</li><li>‘gunicorn’ tells heroku that the HTTP server to use is Gunicorn (for which it has native support for)</li><li>‘app’ references the <em>filename</em> of the main python file without the .py extension. So if you follow the convention of ‘app.py’ you would use ‘app’ here. But note if your main python file is ‘anything.py’, you would have ‘anything’ in place of ‘app’.</li><li>‘server’ references the underlying flask app. Commonly you would define a <em>variable</em> ‘server = app.server’ and this references that variable, I believe. To be more confusing, the ‘app’ in this variable declaration actually refers to the dash instantiation variable in the snippet below:</li></ul><pre>app = dash.Dash(__name__)<br>server = app.server</pre><p>Yes I know what you’re thinking, this is finicky and it’s really easy to misunderstand with all these ‘app’ references everywhere. Take home is: as long as you’re using an app.py main file, as is the convention, and you declare a ‘server = app.server’ line of code after your Dash declaration, you can use the example Procfile and it should work. If you get anything with the Procfile wrong, pain and suffering will ensue.</p><p>To make the Procfile in Windows, you can just create a text file, and enter the single line. Then strip out the extension. This worked for me and I don’t need to have a secondary Procfile.win, which is sometimes talked about in the documentation. Not, Heroku is case-sensitive and the Procfile filename <em>MUST</em> be with a capital ‘P’. Lower case ‘p’ will not be detected properly by Heroku.</p><p><strong>Ingredient 2: runtime.txt</strong></p><p>This file (which must also be in your root project folder) simply tells Heroku which Python runtime to use. Currently it can contain a single line, e.g.:</p><pre>python-3.7.8</pre><p>Just create this as a notepad .txt file in your project folder, and commit-push to your remote GitHub repo. Done.</p><p>That’s really it. It’s mainly these two files (Procfile, runtime.txt) that Heroku needs in your repo project directory in order to work. As long as you have followed the basics, and added Gunicorn to your requirements.txt etc, in theory you are good to go.</p><p><strong>Ingredient 3: perseverance</strong></p><p>Not to be underestimated. Dogged… stubborn… raw perseverance is a key ingredient to the potion.</p><h3>It’s time for magic</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xVwkKNCRXO_vXpqBFf-SAA.png" /><figcaption>(Image from <a href="https://www.pixilart.com/">pixilart.com</a>, Source: <a href="https://www.pixilart.com/art/green-forrest-3b0fcf2469d413f">https://art.pixilart.com/6977a966cd0f6a7.png</a>)</figcaption></figure><p>You’ve got your code in a GitHub repository, with the required tweaks and files created. You have Heroku CLI installed and have created a Heroku app linked to your GitHub repo. It’s 4am and the sun is coming up soon. The scene is grim with pizza boxes on the floor and red wine stains on the keyboard. It’s show time.</p><p>Cast the magic spell:</p><pre>git push heroku main</pre><p>These four words are the spell that makes the magic happen. Type them into the terminal in the right conditions, sit back smugly, and enjoy the show.</p><p>For those new to Heroku, if everything has worked after your “git push Heroku main” from the terminal, your app will be deployed to a Heroku subdomain like:</p><p><a href="http://blah.herokuapp.com/">http://blah.herokuapp.com/</a></p><p>If this is the case, recommend a little dance.</p><p>Copy-paste the URL displayed in the terminal into your browser and get ready…to be disappointed. Chances are the first time you will see “APPLICATION ERROR” in the browser or something like that. Don’t panic.</p><p>The first thing you should do is bring up the log (which is effectively your python console) and see what’s going on, from the terminal. Any print statements, or logger outputs from your code will display here just as they do in the console on your local machine. But most importantly, you also see Heroku system outputs such as Dyno restarts, crashes, and error messages. This is the critical way to see what’s happening with your app at a low level. I usually keep a dedicated CMD window open running Heroku logs, which is my console output.</p><p>View logs:</p><pre>heroku logs --tail</pre><p>Check for things like “Module not found errors” and simple things like that. The most common problems I’ve found are forgetting to add packages to my requirements.txt file because I frantically installed them to my local machine with conda/pip to get something working. If you’ve found some obvious problems, fix them, commit-push your code to GitHub, and then redeploy from Heroku CLI with git push heroku main.</p><p>Notably though, the first time is indeed the hardest. Errors in your Procfile, for example, can still cause Heroku to deploy successfully, but the dynos will crash or fail to start, so definitely check the Procfile.</p><p>Special note if there are no changes to your remote repo on GitHub, Heroku will not deploy. (Which makes sense.) So if you have a GitHub repo cloned onto your local machine, and you are making changes, be sure to commit and push changes back to your remote GitHub repo first (either with command or with GitHub desktop), then in the Heroku CLI terminal, just type in the “git push heroku main” deploy command.</p><h3><strong>Cheat Sheet: Useful commands from Heroku CLI</strong></h3><p>Below is my cheat-sheet of important tips, pitfalls, and pitfall solutions when using Heroku.</p><p>Explicitly referencing your app name:</p><p>Note that Heroku can sometimes be funny about requiring you to explicitly specify your app in the command. If you just have a single Heroku app, often you can avoid it. But sometimes (without any apparent reason) you may need to append “-a &lt;yourapp&gt;” to the Heroku command.</p><p>Display current apps:</p><pre>heroku apps</pre><p>Display current dynos:</p><pre>heroku ps</pre><pre>heroku ps -a &lt;yourapp&gt;</pre><p><a href="https://devcenter.heroku.com/articles/scaling">Scale dynos</a>:</p><pre>heroku ps:scale web=2:standard-1x</pre><p>In this case, we are provisioning two standard-1x dynos to run concurrently. Special note, if WEB_CONCURRENCY=4, this means each Dyno can serve 4 simultaneous HTTP incoming requests, meaning your whole Dash application can serve 8 concurrent requests — the benefit of horizontal scaling. More on this later.</p><p>Run bash terminal:</p><pre>heroku run bash -a &lt;yourapp&gt;</pre><p>Restart dynos:</p><pre>heroku dyno:restart</pre><p>Add additional log metrics:</p><pre>heroku labs:enable log-runtime-metrics</pre><p>View logs:</p><pre>heroku logs --tail</pre><h3><strong>HEROKU TIP: Add runtime metrics to log</strong></h3><p>From the Heroku CLI (once logged in) when you have deployed your app, you can view a live log tail by typing</p><pre>heroku logs --tail</pre><p>Repeated just in case you missed it. This is mission critical. It essentially gives you your console output. One thing I’d suggest is adding in a new feature that outputs resources statistics of your dyno(s) timestamped every ~20 seconds, like memory levels, cpu load etc, which is very useful. Type this in the Heroku CLI, to permanently add it:</p><pre>heroku labs:enable log-runtime-metrics</pre><h3><strong>HEROKU PITFALL: Serving static files does not work</strong></h3><p>I repeat: serving static files <em>DOES NOT WORK</em>. Something of paramount importance that is not obvious, is that out-of-the-box, Heroku (I think more correctly: Gunicorn itself) does not natively support serving static files. This means that while your python application itself can access files in any subfolder in your project folder (such as .csv files etc.), it’s a very different story to actually serve them via http in the client browser.</p><p>Any images, documents, video, audio, anything you are currently serving from your ‘localhost’ webserver will fail on deployment with Heroku. I believe this is a quirk of the PaaS model in that files themselves are not stored in the traditional way you would imagine them to be on a file system, so there are issues with low level connection/packet headers that are attached to files, and/or Gunicorn itself does not natively support serving static files. In any case, there’s magic under the hood.</p><p>As an aside, if you don’t already know from the docs, it’s important to understand that the Heroku file system is not persistent. Like many of my past relationships, Heroku’s file system is ephemeral or transient; it lasts about as long as a one-night stand. With the exception of the files you deploy with your project repo (e.g. csv, json files etc.), any new files created at runtime will disappear after a few days, like that fleeting love interest that wasn’t to be, or that person who never called.</p><p>Anyway, to store and serve persistent static files, as I said, any files uploaded to Heroku as part of your project file suite will be fine, persistent, and accessible by your dash app internally. BUT the moment you want to serve static files externally to browser, you will rapidly run into problems. There are two main solutions I know of. One is simple and fast.</p><p>Solutions:</p><ol><li>Host your files on a 3rd party like <a href="https://aws.amazon.com/s3/">S3</a>, <a href="https://aws.amazon.com/cloudfront/">Cloudfront </a>and link the URL in your dash app (Worth doing if you will be hosting a serious footprint of files)</li><li>Use the <a href="http://whitenoise.evans.io/en/stable/">Whitenoise </a>library. Quick and easy. A few lines of code and you’re serving files just like in your localhost setup.</li></ol><p>Personally I found Whitenoise to be a life saver. Literally “pip install whitenoise” (and make sure it’s in your requirements.txt) and you’re almost there. A few lines of code needed in your dash app:</p><pre>from whitenoise import WhiteNoise<br>server = app.server<br>server.wsgi_app = WhiteNoise(server.wsgi_app, root=‘static/’)</pre><p>You should already have the “server=app.server” anyway, as this is needed by Gunicorn and for the Procfile.</p><p>What this essentially does is wrap Whitenoise around your underlying Flask app. You can then have a folder (which you must create) called “static/” in your root. Everything contained within this (including subfolders) can be statically served by Heroku. Images, videos, pdfs, whatever you want.</p><p>Special note: the ‘static’ folder effectively becomes root when you are serving files, so you would use ‘image.blah’ in your code rather than ‘static/image.blah’ even though your actual file is in /static/image.blah.</p><p>Special note: Heroku is <em>file extension case-sensitive</em>. So blah.png is different to blah.PNG.</p><p>Also, don’t try to get smart and change the ‘static’ folder name in the Whitenoise code declaration to some arbitrary name or ‘assets’ or anything like that: it must be ‘static’ due to an underlying Flask constraint.</p><p>This seems like a pretty major issue that I don’t think much documentation exists on. I spent a long time on Stack Overflow looking it up.</p><p>Also, the Whitenoise documentation is not specific to Dash — it is more focused on general Python apps which are typically Flask apps. This means that it’s still not obvious what you need to do, and the code snippets will not work without modification. For example Whitenoise states that, for Flask apps, you must add the following code to your app:</p><pre>app.wsgi_app = WhiteNoise(app.wsgi_app, root=‘static/’)</pre><p>This won’t work for your dash app. In this case ‘app’ is the Flask app. So in a Dash app (which sits on top of Flask) you actually need to replace the ‘app’ references (both of them) with ‘app.server’ in the snippet above to reference the underlying Flask app and for Whitenoise to work. Or simply define a variable such as ‘server = app.server’ and use the code snippet I outlined at the beginning of this section.</p><p>Again, lots of these things are a two second fix if you know how. But can cost you literally HOURS AND HOURS……AND HOURS of time if you don’t know. Trivial for Flask developers. Not trivial <em>at all</em> for newcomers.</p><h3><strong>HEROKU PITFALL: Favicon may not work</strong></h3><p>For some reason, I had lots of trouble with this. Anyway, I managed to get it going by simply having a:</p><pre>/assets/favicon.ico</pre><p>From my root project directory. Special note that no other static files are served from here; it’s a stand-alone folder. In fact, don’t be lulled into thinking you can serve static files from your /assets folder on Heroku: you can’t. (see Whitenoise section).</p><p>Others have had problems with Heroku changing the extension name of the favicon causing it to fail. One failsafe option to note is that you can log into a Heroku Bash shell after you have deployed, and navigate to all your project folders/files to see what Heroku sees. See <a href="https://community.plotly.com/t/display-favicon-in-heroku-dash-app/44013/3">this </a>post.</p><p>From heroku CLI:</p><pre>heroku run bash -a &lt;yourappname&gt;</pre><p>This will provision a new Dyno container running a Bash shell. Basically, it’s a terminal to your deployed app.</p><h3><strong>HEROKU PITFALL: Web concurrency is important and can be configured</strong></h3><p>There is lots of ‘worker’ and ‘web’ terminology that gets confusing. Out of the box when using Gunicorn as your Python HTTP server, Heroku essentially guesses how many concurrent web-worker-processes to run for each dyno instance running your web app. Typically this is 1–6 concurrent ‘gunicorn — web-worker-processes’ per dyno for the commonly used hobby to standard 2-x dynos. This is how many client requests (i.e. from a web browser) can be simultaneously served by your app at an instantaneous point in time.</p><p>A Gunicorn web-worker-process is a process capable of serving a <em>single </em>HTTP request at a time. So if you only had one, this means your website becomes quite unresponsive with a few users making simultaneous requests, and having to wait for these requests to be actioned from a queue. Essentially this is the magic of what Gunicorn does: it forks the main web process running on its Dyno into multiple processes so that it can serve simultaneous HTTP requests from each given Dyno resource.</p><p>Web concurrency, in Heroku, therefore, essentially allows each dyno instance to carve up it’s resources to serve multiple concurrent HTTP requests, which it terms WEB_CONCURRENCY. Unfortunately this can sometimes lead to Heroku (by default) underestimating resources needed, and running over Dyno memory limits, causing failure, restarts, massive slow-downs due to disk swap having to be used, etc. At least, this is what I experienced. Basically, you don’t want to have too much web concurrency because it might break your dyno, and the default setting chosen by Heroku may be too high.</p><p>You don’t need to worry about this on day one — your app will work. But as you start load testing it, you may find you run into memory overrun issues and all sorts of things like that. If you have a high horsepower python application that chews resources, I suggest you manually set your WEB_CONCURRENCY variable in Heroku command line.</p><p>For example:</p><pre>heroku config:set WEB_CONCURRENCY=3</pre><pre>heroku config:set WEB_CONCURRENCY=3 -a &lt;herokuappname&gt;</pre><p>The above statement variations tell Heroku to carve up Dyno resources to support 3 concurrent HTTP requests per Dyno, for all Dynos running a Web process for your app. (So if you have multiple Dynos serving your app in parallel, it automatically sets them all to this same setting).</p><p>If performance is not compromised, you can increase web concurrency to increase the number of clients you can serve in parallel, while minimising Dyno cost. If you need to serve more, you can scale Dyno’s horizontally knowing that each one can serve an explicit number of concurrent HTTP requests that you have set.</p><p>And of course, you can monitor this with “heroku logs --tail” or in the Heroku dashboard ‘metrics’ section.</p><h3><strong>HEROKU PITFALL: Hard limit 30 second request timeout</strong></h3><p>It’s important to be aware that Heroku has an immutable 30 second timeout for serving HTTP requests. This is a common problem especially encountered by Dash users because many of the data science applications have long load times — see <a href="https://community.plotly.com/t/dash-heroku-timeout/36517">this</a> post. These might work fine running on your local host, but be aware that your Heroku deployed app must be able to serve within 30 seconds or it will time out. Heroku <a href="https://devcenter.heroku.com/articles/request-timeout">docs</a> state a few work arounds but take special note of this problem.</p><h3><strong>HEROKU PITFALL: Develop on the master/main branch of your GitHub repository</strong></h3><p>If you’re new to GitHub, just know that you can have multiple ‘branches’ of your project, as you might take it in different directions. These can be merged or left as separate branches. The central branch by default is called ‘master’ or ‘main’ in GitHub. When you create your Heroku app it interacts with your GitHub repository to create a kind of Heroku mirror image behind the scenes. If you’re developing your current code on a branch that is not master or main, prepare for pain. It’s not that it can’t be done, I just had a lot of trouble with this when trying to deploy to Heroku. I found the best rule of thumb is to just develop all my code on the default ‘main/master’ branch in my GitHub repository.</p><h3>Useful Stuff</h3><p>A quick run down of other useful stuff.</p><p><strong>Custom Domain</strong></p><p>It’s not too difficult to set up a custom domain for your Heroku app. Obviously, you need to purchase a domain first. Once you’ve done that, the provider will typically have a portal where you can login and adjust settings.</p><p>Heroku will generate a unique DNS target in the SETTINGS area of the Heroku web portal dashboard, once logged in. Such as:</p><pre>Animate-salamander-8duwlndghfqbtj0t90uep8bmu.herokudns.com </pre><p>What you need to do is copy this (your own) DNS target from the Heroku web portal (settings page) and then login to your domain provider portal and for your domain, create a new “CNAME record” with host “www” value “Animate-salamander-8duwlndghfqbtj0t90uep8bmu.herokudns.com” (your unique Heroku DNS target).</p><p>If it worked ok, in a few hours your new domain should work!</p><p>Essentially all this is doing is redirecting to the Heroku DNS target when someone types your actual domain name. If your new domain is <a href="http://www.blah.com/">www.blah.com</a> it now has a CNAME record to redirect the incoming HTTP request to Heroku infrastructure, which then serves the actual page (as if you’d typed in <a href="http://blah.herokuapp.com/">blah.herokuapp.com</a>). It’s a tricky thing to set this stuff up because it’s not done very often and you don’t know if it has worked for hours (because it takes time for DNS servers around the world to replicate the new domain list). But there is <a href="https://devcenter.heroku.com/articles/custom-domains">good documentation</a>.</p><p><strong>Flask Caching on Heroku</strong><br>If you have Flask Caching running on your local machine, it’s straight forward to setup on Heroku with a free Memcachier account. And the <a href="https://devcenter.heroku.com/articles/memcachier#python">docs</a> are good. You can cache to the ephemeral Heroku file-system without Memcachier, noting you might max out your 500MB of Dyno storage, otherwise you can get 100MB free high performance cache via Memcachier.</p><p><strong>Getting Fancy with security and autoscaling etc.</strong></p><p>When you want to go to the next level and setup auto-scaling of machines, proper security/authentication etc, I think this is when it starts becoming worth considering <a href="https://dash.plotly.com/dash-enterprise">Dash Enterprise</a>, upgrading to top-tier Dynos with Heroku (gets $$) OR going down the path of setting up custom infrastructure manually. If you are going manual you could, for example, provision your own virtual machines, set up containerised pipelines using Docker and Kubernetes, manage autoscaling/self healing with Rancher, etc. It’s well and truly DevOps and cloud engineering territory. I’m sure there are lots of other midrange steps you can take like run Docker to build your own containers to deploy, but I want to keep this guide to the bare minimum you need to get on Heroku.</p><p><strong>Closing thoughts</strong></p><p>For the newbies and hobbyists out there, I sincerely hope this short novel has been useful to help get your project up and running faster with less pain. Don’t forget I have a fully running <a href="https://github.com/danny-baker/dash-heroku">starter pack</a> on Github that you can use as a guide to see EXACTLY what you need as a bare minimum.</p><p>I think Dash is a game-changing tool that is helping to bring data science literacy (and data visualisation technology) to the mainstream public and business world. This can only be a good thing, and this is why I’ve taken the time to write this piece.</p><p>Final caveat: I’m not an expert on web, but I do think there is a major gap in the documentation for new starters. I’ve done my best to research all the facts, so this content is to the best of my understanding. I certainly invite corrections and clarifications by the experts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/72/0*mTDIB04BC7HQsE1E" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39bd6a0c586c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deploying-your-dash-app-to-heroku-the-magical-guide-39bd6a0c586c">Deploying your Dash App to Heroku — THE MAGICAL GUIDE</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>