<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dan Lester on Medium]]></title>
        <description><![CDATA[Stories by Dan Lester on Medium]]></description>
        <link>https://medium.com/@dslester?source=rss-b6bb58cd796c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OhP0MUjv3bVvKGj2o92WzQ.jpeg</url>
            <title>Stories by Dan Lester on Medium</title>
            <link>https://medium.com/@dslester?source=rss-b6bb58cd796c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:31:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dslester/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Structuring GitHub Actions Safely]]></title>
            <link>https://levelup.gitconnected.com/structuring-github-actions-safely-5309e5ee54de?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5309e5ee54de</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[slackbot]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Tue, 25 May 2021 14:49:06 GMT</pubDate>
            <atom:updated>2022-09-22T08:45:30.057Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to design GitHub Actions workflows with a beginning, middle, and end, so you can tear down your environment safely and notify your team accurately.</h4><p>GitHub Actions allows you to automate any software workflow by including a simple YAML file in your git repository spelling out what you want to execute and when. If you’re new to it, take a look <a href="https://github.com/features/actions">here</a>.</p><p>Recently, my team built a GitHub workflow to test deployment of our software to the cloud — set up an environment on Amazon Web Services, run some tests, then tear it down again. This started life as one job with a long series of steps. At the end, we used an if: failure() <a href="https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#job-status-check-functions">status check function</a> to make sure that the final step always runs after any earlier step fails — and that final step sent a Slack notification to let us know something went wrong.</p><p>That seemed to work, but let’s say a failure happens during one of the testing steps. In that case, we jump straight to sending a (correct) failure notification, but that skips over all of the tear down steps. So the resources are still live on the cloud until someone picks up the notification and manually stops everything on AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UPediNWyNgOjjJiK" /><figcaption>Photo by <a href="https://unsplash.com/@kar111?utm_source=medium&amp;utm_medium=referral">Karine Avetisyan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the end, we refactored the workflow by dividing the long list of steps into a chain of three separate ‘jobs’. This gave a logical division but threw up some unexpected gotchas when it comes to sending the success/failure notification.</p><p>As a reminder, jobs are the higher level collections for steps in GitHub Actions workflows. Jobs may be run in parallel by default, and individual jobs do not share the same workspace or virtual machine, so you cannot automatically share state between jobs (compared to ‘steps’ which all run in the same environment as their siblings).</p><p>Whether it makes sense to split your workflow into multiple jobs depends on what you are doing, but the important thing for our example is to know that it is possible to set up a dependency tree between jobs. So we can run job 1 (set up the environment), then job 2 (run the tests), before finally running job 3 (tear down the environment and send notifications).</p><h3>Notifications</h3><p>In my case I’m using the <a href="https://github.com/marketplace/actions/slack-templates">marketplace action Slack Templates</a> to send notifications to Slack, but you can use any number of free GitHub Actions to send directly to the Slack channel of your choice, or Microsoft Teams, or whatever you want to use.</p><p>Either way, you’ll need some form of authentication token to fire your notification, and you should save this as an organization or repository secret:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ihexy3ICgwnmAqtJdq6ifw.png" /><figcaption>Setting token as a secret in the GitHub repo (Image by the author)</figcaption></figure><h3>Jobs in Serial</h3><p>Here was my first attempt at splitting the workflow into three distinct jobs. In fact, the middle job in my real workflow used a different runner to the other two (not <em>ubuntu-latest</em>) so this <em>had</em> to be in a separate job. In this example, I simulate the main steps with exit 0 — which can be easily changed to exit 1 if I want to pretend that step failed, for example due to a test failure. Any non-zero exit code will stop the workflow by default.</p><pre>name: &quot;Test Multiple Jobs&quot;</pre><pre>on:<br>  workflow_dispatch:<br>  <br>jobs:<br>  startupjob:<br>    name: Start Up<br>    steps:<br>      - name: Set Everything Up<br>        run: exit 0</pre><pre>  testsjob:<br>    name: Run Tests<br>    needs: startupjob<br>    steps:<br>      - name: Run the test suite<br>        run: exit 0</pre><pre>  teardownjob:<br>    name: Tear Down<br>    needs: testsjob<br>    if: always()<br>    steps:<br>      - name: Tear Down<br>        run: exit 0<br>        <br>      - name: Notify via Slack Templates<br>        uses: ScribeMD/slack-templates@0.6.7<br>        if: always()<br>        with:<br>          bot-token: ${{ secrets.SLACK_TOKEN }}<br>          channel-id: ${{ secrets.SLACK_CHANNEL_ID }}<br>          template: result</pre><p>In this example, I’ve used on: workflow_dispatch which means it is manually triggered through GitHub’s website or API, but maybe your workflow will be an on: push event or something else.</p><p>By default, GitHub may choose to run multiple jobs in parallel, so we use the needs keyword to set up the dependency tree.</p><p>The first job (startupjob) has no dependencies, but then the second job (testsjob) has a needs: startupjob entry, meaning it needs startupjob to finish before it will run.</p><p>Finally, teardownjob has needs: testsjob to ensure it only runs after the tests have finished. So we have a linear dependency graph like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8RAy0QUJxiLx-RHfQcQjdg.png" /><figcaption>Serial job dependency graph (Image by the author)</figcaption></figure><h3>Always</h3><p>You’ll notice there are two if: always()lines in the third ‘tear down’ job. They work at different levels.</p><p>The first means that the entire teardown job should always run. Especially if the second job (tests) fails — which is likely occasionally if we are testing anything worthwhile — we still want to tear down the environment to conserve resources, and ensure the state is reset for the next time it is needed.</p><p>The second if: always() applies at the step scope, within the job. This means that, within the teardown job, even if the tear down itself fails, we still want to send the notification.</p><p>To be clear, the second if: always()alone is not enough to ensure the notification step is always run. It only applies within the teardown job. Without the job-level if: always() we would find that a failure in an earlier job would finish the entire workflow early, and teardownjob would be skipped despite one of its steps being intended to run ‘<em>always’</em>.</p><p>So in the correct setup (with two <em>always</em> entries, one for the teardown job and one for the notification step) we can simulate a failure in startupjob (change to exit 1 for example) and see the following state:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJwDh5h1aISlPuwup3QKPg.png" /><figcaption>Failure, Skipped, and Success results for different jobs (Image by the author)</figcaption></figure><p>That’s great — the tear down still runs to reset our environment (and it should be coded in such a way that it can cope with the situation where the startup job didn’t run entirely).</p><p>But there is a complication in how we interpret status for our notification.</p><h3>Job Status forgets the Past</h3><p>Passing through status: ${{ job.status }} to the Slack notification action feels about right, and it will be correct in the situations where either everything goes to plan or when a failure happens in a step earlier in the teardown job.</p><p>But if you look at the job dependency graph above you can see a big red cross next to ‘Start Up’ and a big green tick next to ‘Tear Down’.</p><p>Remember that job is the context only of the current teardownjob, not the entire workflow, so job.status will actually equal success throughout the teardown job. And then the status passed to the Slack notification is success even though our startup or tests job failed — both cases where we want to note the failure to ensure someone takes a look.</p><p>Let’s look a bit harder at the logic required. We want to notify of failure if job.status does not equal success (meaning an earlier failure in the current tear down job itself) OR if an earlier step failed.</p><p>You can’t actually access the status of other jobs directly <em>unless</em> the current job has a direct dependency on those jobs. In that case you can use the <a href="https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#needs-context">needs context</a> to access the dependent jobs (remember we specified needs: testsjob as a field in the teardownjob).</p><p>So needs.testsjob provides access to the previous job, and in particular needs.testsjob.result will contain the string success, failure, cancelled, or skipped.</p><p>My conclusion was to use this expression to determine the overall workflow outcome to present in our Slack notification:</p><pre>${{ job.status == &#39;success&#39; &amp;&amp; needs.testsjob.result == &#39;success&#39; }}</pre><h4>Back to the Beginning</h4><p>Is that enough? Do we not also need to access the result of the firstjob , ‘startupjob’?</p><p>It turns out we do not as long as we maintain the linear dependency tree. This is because, if startupjob fails it causes the entirety of testsjob to be skipped — and needs.testsjob.result equals skipped (so does not equal success).</p><p>So here is the final workflow file, including the corrected status check for sending the Slack notification:</p><pre>name: &quot;Test Multiple Jobs&quot;</pre><pre>on:<br>  workflow_dispatch:<br>  <br>jobs:<br>  startupjob:<br>    name: Start Up<br>    steps:<br>      - name: Set Everything Up<br>        run: exit 0</pre><pre>  testsjob:<br>    name: Run Tests<br>    needs: startupjob<br>    steps:<br>      - name: Run the test suite<br>        run: exit 0</pre><pre>  teardownjob:<br>    name: Tear Down<br>    needs: testsjob<br>    if: always()<br>    steps:<br>      - name: Tear Down<br>        run: exit 0<br>        <br>      - name: Notify via Slack<br>        uses: ScribeMD/slack-templates@0.6.7<br>        if: always()<br>        with:<br>          bot-token: ${{ secrets.SLACK_TOKEN }}<br>          channel-id: ${{ secrets.SLACK_CHANNEL_ID }}<br>          template: result<br>          results: &quot;${{ job.status }} ${{ join(needs.*.result, &#39; &#39;) }}&quot;</pre><p>(Note the final results entry should all be on one line — it wraps above)</p><p>We see the correct notification in our Slack notification now.</p><h3>Summary</h3><p>This article showed how to design GitHub Actions workflows in a linear dependency graph of separate jobs.</p><p>We used if: always() on the final job to ensure it is run even when the earlier setup and core testing jobs have failed, and this means that our cloud environment can be dismantled correctly in all cases.</p><p>However, this complicates the expression used to determine whether to notify of ‘success’ or ‘failure’ via our Slack integration, and we have to understand our precise dependency tree in order to create an accurate expression.</p><p>We used the public action <a href="https://github.com/marketplace/actions/slack-templates">Slack Templates</a> to handle the notification into Slack.</p><p>If you do need to share state between GitHub Actions workflow jobs, <a href="https://levelup.gitconnected.com/github-actions-how-to-share-data-between-jobs-fc1547defc3e">here is an excellent summary</a> of the options available.</p><p>Please let us know if different workflow structures or status expressions make sense for your particular workflows!</p><h3>Endid</h3><p>To make the Slack notifications much easier, take a look at <a href="https://endid.app/">Endid, my Slack app for monitoring Github Actions</a>. This offers an instant install to monitor all GitHub Actions from the repos, so you don’t need to worry about adding fiddly notification code to your workflows at all!</p><p>Furthermore, it offers intelligent monitoring where it only sends notifications when a workflow suddenly fails, or to report success after it has been fixed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5309e5ee54de" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/structuring-github-actions-safely-5309e5ee54de">Structuring GitHub Actions Safely</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why GitHub’s default branch keeps ghosting you]]></title>
            <link>https://medium.com/@dslester/why-githubs-default-branch-keeps-ghosting-you-2c2e91b713b1?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c2e91b713b1</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Fri, 16 Apr 2021 11:50:54 GMT</pubDate>
            <atom:updated>2021-04-16T11:50:54.032Z</atom:updated>
            <content:encoded><![CDATA[<h4>Renaming ‘master’ to ‘main’ is harder than you think because there is no such thing as a default branch in git</h4><p>Many GitHub projects have recently performed the task of changing their ‘master’ branch to ‘main’. And GitHub has even set the worldwide default branch as ‘main’ for new repos. There are many great guides to help with the migration such as <a href="https://stevenmortimer.com/5-steps-to-change-github-default-branch-from-master-to-main/">this one</a>, and the procedure can happen pretty smoothly for an existing repo.</p><p>But things start to go wrong if you create new repositories, especially if you create lots of new repos, perhaps using GitHub’s API. We are doing this on the <a href="https://github.com/Quansight/qhub">QHub project</a> — it’s a tool to easily set up the full continuous-deployment pipeline, including GitHub actions, for your own managed Kubernetes cluster running a JupyterHub environment. So we automate the creation of lots of new repos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lmXhIZNgDD9rYhfc" /><figcaption>Photo by <a href="https://unsplash.com/@heftiba?utm_source=medium&amp;utm_medium=referral">Toa Heftiba</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Sure, there is an option in GitHub to set the default branch for a new repo, and there is even more control within an organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vKCgEghbpghVSqrybvQgqw.png" /></figure><p>But even when this is set to ‘main’ it doesn’t seem to help. Repos keep appearing with ‘master’ as the default branch.</p><p>The reason is that there is actually no such thing as a ‘default branch’ in git.</p><h4>Empty GitHub repos are really very empty</h4><p>Let’s run some tests to see what’s really going on. First, create a completely empty repo (don’t add any automatic README files etc or that will ruin it).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RBvQ6atYMISZM8iwN9VcpQ.png" /></figure><p>So GitHub knows this is an empty repo and shows us some ideas for populating it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wYj2zanuh_OIb5PaLjv6aw.png" /></figure><p>But keep it empty and see what happens on the command-line.</p><p>Clone the repo and take a look at the HEAD file in the .git folder. This is where git stores its pointer to your current HEAD (basically to say which branch is currently active and where new commits will go).</p><pre>git clone <a href="https://github.com/danlester/branchtest.git">https://github.com/danlester/branchtest.git</a><br>cd branchtest<br>cat .git/HEAD</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1UzM2XhQ3kJdpVwVd06P_w.png" /></figure><p>It says refs/heads/master — that can’t be right! We have set the GitHub default branch to main…</p><p>This is because the GitHub repo is genuinely completely empty. It is git clone’s best attempt to set up this empty git repo as a local git repo, without being quite so bare.</p><p>But in fact ‘master’ still doesn’t really exist yet anyway.</p><pre>git checkout master</pre><p>Gives:</p><pre>error: pathspec &#39;master&#39; did not match any file(s) known to git</pre><p>As soon as we commit something, master will exist:</p><pre>echo &quot;Hello&quot; &gt; README.md<br>git add README.md<br>git commit -m&quot;Commit test readme&quot;<br>git push</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aYDkRSHP3OJdEVwQzvHcKQ.png" /></figure><p>And back on GitHub, things are just as bad:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3paAeGtmg2Ub6kDv31K7Pg.png" /></figure><p>There is only one branch (master). And in Settings / Branches for the repo we can see the default branch is master:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mqb1ki9gFYt2cr7O0muHMw.png" /></figure><p>So why are we back in the 19th century? It looks like GitHub has completely ignored our default branch setting!</p><h3>GitHub is a social network for Git</h3><p>There is really no concept of a ‘default branch’ in git (as in, plain old command-line git). Only GitHub seems to have this concept as the “default branch for pull requests and code commits”. Remember, PRs are a GitHub-only feature, and code commits can be made using the UI on GitHub’s site.</p><p>The only related feature in command-line git is that when you do a basic git clone it decides which branch to checkout locally, and bases that simply on what HEAD points to on the remote repo.</p><p>When it clones an empty repo, as we’ve seen, command-line git wants to set HEAD to something and it chooses ‘master’ even though that doesn’t exist.</p><p>If you create another new repo in GitHub, but this time choose for a README file to be auto-generated, that’s the point at which GitHub’s default branch setting comes into play:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OGVXSTq7AE5dJJi0Gx43Q.png" /></figure><p>You can see at the bottom: <em>This will set main as the default branch</em>. And indeed it does — our new repo has only a main branch (which of course must be the ‘default’ in GitHub language).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6_5eM-gzVoQMFi-xGzXvQ.png" /></figure><p>So GitHub hasn’t reverted back to last year’s default of master at all. If, in our first test above, we created another branch before making our very first commits:</p><pre>git checkout -b anotherbranch</pre><p>and then pushed that, we’d find that GitHub only knows about ‘anotherbranch’ and has no knowledge of main or master (and of course has to set ‘anotherbranch’ as its default since there is no other branch).</p><p>Again, GitHub really only has the concept of a default branch for its ‘social networking’ features — Pull Requests and code commits made via the GitHub website.</p><h4>Does GitHub’s Default Branch affect git clone?</h4><p>Let’s make a new branch on our new repo, so we have ‘main’ and ‘newbranch’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6Y5ayamxpuHa3IGxyw7Bg.png" /></figure><p>And then switch the default branch to ‘newbranch’ in GitHub:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uMqkoITozFJzc8wjWXkEiQ.png" /></figure><p>When we do a fresh git clone, we are in fact on ‘newbranch’!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j3J3UvRPBC5pbciohWGEpA.png" /></figure><p>And we see that the remote origin has its HEAD ref set to newbranch. So GitHub does have some recollection that we changed the default branch, after all…</p><p>This also shows that git clone does take a look at the remote repo’s HEAD if it is set, and ‘borrows it’ as a sensible default. But the point here is that ‘default branch’ is a very lightweight concept in command-line git.</p><p>Other than with git clone, another time this comes into play is with git init where we make a new local repo directly (not cloning from GitHub).</p><h3>git init</h3><p>When we git init a local repo, we have the same state as we did when we cloned an empty repo from GitHub.</p><p>There is some concept of ‘master’ as the default, but the branch still doesn’t really exist until we make a commit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jd6fpuaOnFFbTAPMXELT9Q.png" /></figure><p>You can also see I’m using git version 2.24 here. Something different happens with git version 2.28+.</p><h4>Politically Correct git</h4><p>I’ve now upgraded to the latest git, and git init has a new ‘initial-branch’ option:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kj3cPIzi30k687zQWRWOUw.png" /></figure><p>We now have the same situation as before, but with ‘main’ as the first attempt at a branch ref. As before, the branch doesn’t really exist until we commit something (or manually create the branch). But, finally, the first commits will now go into the main branch!</p><p>How does this help GitHub? We still can’t change GitHub’s empty repos, but we can change the default behavior for empty repos, and this also influences the empty repo we clone from GitHub.</p><p>Set the ‘initial branch’ to ‘main’ in your computer’s git config:</p><pre>git config --global init.defaultBranch main</pre><p>Now when you git clone an empty repo from GitHub we at least get some attempt to point to main by default:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ocGUkugYDgIGEj3q-1Ml5w.png" /></figure><p>And if we commit files, the main branch will be created automatically.</p><p>To be clear, the fact we have a main branch locally is nothing to do with the ‘default branch’ option you set in GitHub for new repos!</p><h3>Summary</h3><p>GitHub’s ‘default branch’ setting really only applies to its ‘social’ features — Pull Requests and commits made on their website.</p><p>It does also set the ‘HEAD’ of the remote git repo to that branch, and this has some influence over the behavior of your local git clone — but definitely not when the GitHub repo is completely empty.</p><p>There isn’t really any concept of a default branch in git command-line, other than as a starting point when you perform a git init (to create a new local repo) or a git clone (on a remote repo).</p><p>In git version 2.28+ you can set the initial branch for git init, and this also sets the initial ‘HEAD’ for a git clone of an empty repo:</p><pre>git config --global init.defaultBranch main</pre><p>In earlier git versions, you always need to control the initial branch after git init (for a new local repo) or git clone of an empty repo. The easiest thing is to run:</p><pre>git checkout -b main</pre><p>This shouldn’t fail even if the main branch already exists and/or you are on it. And it shouldn’t fail even if the repo is completely empty, whereas git branch -M main to attempt a rename would fail on an empty repo with no initial branch to rename.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c2e91b713b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero to Bitcoin Mogul in one Afternoon]]></title>
            <link>https://medium.datadriveninvestor.com/zero-to-bitcoin-mogul-in-one-afternoon-7e5ed14273?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e5ed14273</guid>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[investment]]></category>
            <category><![CDATA[bitcoin]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Mon, 16 Nov 2020 09:03:51 GMT</pubDate>
            <atom:updated>2020-11-17T02:26:33.334Z</atom:updated>
            <content:encoded><![CDATA[<h3>Become a Bitcoin Mogul in One Afternoon</h3><h4>Buying your first cryptocurrency. All you need is a credit card and a driving license.</h4><p>As a technology entrepreneur, maybe you would expect me to be a bit more entrepreneurial with my investments.</p><p>I like to think I’ve read enough to understand “blockchain” conceptually, and occasionally in the past I’ve casually looked into owning some cryptocurrency. The problem has always been that enthusiasts who write on this topic are often too far down the rabbit hole — they can’t help but caution that you absolutely need a “cold storage hardware wallet”, and to discuss the history and merits of different cryptocurrencies (not to mention, different “forks” of those currencies too).</p><p>So I’ll keep it simple and just run through a first attempt at buying <em>some</em> cryptocurrency. I’ll write it up straight away so that any other beginner can follow the same steps without having to fight through the display of impressive jargon that I’ll inevitably insist upon once I have more experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*COhdhyM9oEyXzHHU" /><figcaption>Photo by <a href="https://unsplash.com/@wildbook?utm_source=medium&amp;utm_medium=referral">Dmitry Demidko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’m peripherally aware of some other cryptocurrencies, but we’ve all heard of Bitcoin (the original and best, one presumes) so let’s buy some of that. We won’t invest more than $10 in this experiment so surely we don’t need to worry too much about security. If the hackers take the whole lot, we can pretend that was an intentional part of our learning experience in the first place.</p><p>I’ll use a popular and trustworthy-looking exchange called Coinbase (I’m sure I’ve read about it in the mainstream press) to buy my first Bitcoins — we won’t worry about exorbitant transaction fees for this first attempt.</p><p>However, I will make one slightly philosophical objection. I’m pretty sure that holding cryptocurrency in Coinbase means you don’t really hold it at all. In fact, Coinbase now lets you hold balances in USD and other major currencies too, so it’s starting to sound just like a bank… how is that a valid part of the decentralized crypto revolution? It sounds like it isn’t, so as our final bold step towards our high tech future we’ll withdraw our (tiny fraction of a) Bitcoin onto our own computers and look after it for ourselves thank you.</p><p>It’s starting to sound like I’m being sucked in… there’s every danger that by the time you read this, I’ll have come back and edited this article, adding crypto-insider warnings and insisting that the only real way to enter into the world of cryptocurrency is to “mine” your own bespoke currency in the cloud. We really need to get a move on before it’s too late.</p><h4>Why?</h4><p>I’m sure there are many different reasons why people get into cryptocurrency. Traders might try to make money as prices go up and down; criminals might want a way to launder money; and increasingly, normal(ish) people use it as an investment vehicle to diversify from the usual savings accounts, stocks, and bonds.</p><p>There is an <a href="https://bitlift.com/learn/introduction-to-bitcoin/">excellent introduction to the concepts and history of cryptocurrencies</a>, written by a company called BitLift. It explains the fundamentals of using gold or US Dollars as currencies historically and gives some arguments as to why you might not entirely want to rely on the US Federal Reserve indefinitely. I’d highly recommend reading, but their final suggestion is to simply try buying some cryptocurrency to see where it takes you… and that’s where the trail goes cold, leaving us to fend for ourselves.</p><p>For our purposes, we won’t try to come up with a reason for buying Bitcoin, other than as an experiment to feel a little more comfortable with the idea of owning some cryptocurrency. Then, if someone suggests a crypto investment in the future we can consider it honestly rather than dragging our feet and feeling that there’s too much to learn to get involved safely.</p><p>We’ll dive just deep enough into the technical details to be able to believe that we do indeed have completely independent and secure ownership of our Bitcoins in the end.</p><p>To be clear, this article does not represent investment advice for your $10 in any shape or form. Quite the opposite: be prepared to squander it in the interest of learning something.</p><h3>Buying like a Regular Consumer</h3><p>As I’ve suggested, buying our first cryptocurrency from Coinbase is embarrassingly easy and we hardly get involved. It’s just like a bank. We provide a credit card and they turn a dollar amount into Bitcoin somehow, then hold on to it for us on their website.</p><p>We could skip this step if we knew someone who already owned some Bitcoin and who was willing to transfer some to us. But we’ll use Coinbase as an easy starting point, and as a source of coins that we can transfer to our computer somehow in the next step.</p><p>A couple of years ago, governments generally tightened up regulation around cryptocurrencies — and that’s why Coinbase is just like a bank. You’ll need some ID such as a driving license or passport so they can verify who you are, just as any bank needs to “know your customer” to comply with anti-money laundering and terrorism laws.</p><p>Go to <a href="https://www.coinbase.com/join/lester_ds">coinbase.com</a> and set up an account (that’s an affiliate link — we both get $10 if you trade enough!). I won’t lead you through that — it’s straightforward. Just follow through to the point where it seems ready for you to click on the ‘Buy’ button (and it’s already asked you for your ID), and I’ll see you below.</p><p>Click on the big blue Trade button in Coinbase. In the Buy tab, make sure ‘Bitcoin’ is selected and enter £10 or $10 in the amount field, or something sensible for your local currency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*IGl9G1aWp0tuq4um0zKK_g.png" /></figure><p>I’ve chosen “10” because it optimizes the transaction fee. There seems to be a flat £0.99 fee up to £10, and then it rises a little bit, so it makes sense to use up our full £10 allowance within that lowest band. I believe the thresholds are the same numbers if you’re working in USD.</p><p>Click ‘Preview Buy’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*3uzgTmJZxiM_RNfWnyQCpA.png" /></figure><p>Click ‘Buy now’ and you’re done! You are now the proud owner of 0.0007 Bitcoins! Actually, it might well not be as instant at that… I had the transaction declined by my credit card company a couple of times and had to let them know that, yes, I’ve finally succumbed and gone all revolutionary. I should send this article to the customer service agent I spoke to — he sounded most intrigued.</p><p>Here is my new-found wealth in my portfolio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4th9L9jw0IsVL-DyGo71A.png" /></figure><p>The eagle-eyed amongst you might notice the Bitcoin amounts don’t quite match the previous screenshots — that’s because I had to recreate the earlier screenshots at a later time after the Bitcoin price had changed slightly.</p><p>Anyway, now we’re rich! For the time being, we will leave our extensive crypto portfolio in Coinbase, to return later in this article when we want to snatch these funds from under the noses of the banking giants and onto our own computer.</p><h3>Financial Independence</h3><p>That was all a bit too mainstream. Moving forward, we want to find our own way to dive into the Bitcoin network and store some kind of digital key that means that we, and we alone, own and control our Bitcoin holding.</p><p>All we need to fundamentally create a secure way to hold and trade Bitcoin is to generate a random 256-bit number and keep it secret. We can use that to generate public addresses where other people can send Bitcoin. If they send to one of those addresses in a transaction, the transaction is stored publicly on the “blockchain”. It doesn’t reach us directly.</p><p>But because we know the private key that generates the address, we are able to initiate other transactions with any ‘balance’ attached to that address — so the Bitcoins belong to us in the sense that we are able to send it somewhere else. We can also look at the balance attached to the address, and so can everyone else. The blockchain is entirely public, which it needs to be in order to be maintained by consensus.</p><p>As you may be aware, the decentralized nature of Bitcoin relies on multiple ‘nodes’ (computers running the core Bitcoin software), each incentivized to faithfully maintain the blockchain so far — and the idea is that if the network is formed of enough independent nodes it is impossible for them to collude and lie about the state of the blockchain since there will still be a majority of nodes that present the real truth.</p><p>We won’t need to run our own node on this network, but we do need a piece of software on our laptop that will:</p><ol><li>Help us create and work with our private key</li><li>Connect to the Bitcoin network to look up any Bitcoins sent to us and create transactions if we want to send to someone else</li></ol><p>This piece of software is known as a ‘wallet’. I will go with the open-source Electrum software, available for all major operating systems.</p><h4>Installing the Electrum Wallet</h4><p>Download and install the application for your operating system from the <a href="https://electrum.org/#download">Electrum website</a>. They are pretty keen that you verify the download as described on their downloads page.</p><p>When you start Electrum, it will guide you through the process of setting up a new wallet. Basically, a wallet is just a new private key allowing Electrum to generate everything else it needs to interact with that key: Bitcoin addresses, mainly. Choose a filename for the wallet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*qCLUpxWRyxiA-67CEw8yQQ.png" /></figure><p>The next few steps take you through some options:</p><ul><li>Select a ‘Standard Wallet’</li><li>Create a new seed</li><li>Choose ‘Legacy’ seed type</li></ul><p>Finally, you will be presented with your ‘seed’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*6g_kNavmnSJJkZCvJI7eAg.png" /></figure><p>This phrase <em>is</em> your private key! You need to make a note of it somewhere safe. The recommendation is to write it down on a piece of paper and <strong>not</strong> to <strong>ever</strong> store it electronically — which is a little hypocritical since Electrum itself is about to write it onto your hard disk after you click Next.</p><h4>Seeds for Keys</h4><p>So how is this seed phrase your new private key? To simplify things, assume that Electrum has a standard list of 2,048 words that it uses. This means that any 11-bit number (0–2,047) can be represented by one of the words. So 12 of these words in order can correspond to a 12*11 = 132-bit number.</p><p>Although this seems to fall short of the 256-bit key we were expecting, it is ‘stretched’ by Electrum. This is done in such a way that it as is strong as a random 256-bit key in terms of the difficulty of making a brute-force attack. More <a href="https://electrum.readthedocs.io/en/latest/seedphrase.html">details are here</a>, and it is also interesting to read that Electrum doesn’t in fact (any longer) rely on a fixed list of words.</p><h4>The Wallet File</h4><p>The next step, after Electrum has presented you with your seed phrase, is for it to ask you what that phrase was. It asks you to type it in again to be sure you’ve written it down correctly.</p><p>After that, you will be asked if you want to enter a password. This is to encrypt the wallet file on your disk to protect it from other users of your computer, or in case someone is able to obtain the wallet file from a backup somehow.</p><p>This would be a sensible move in general, but for our experiment I would suggest not using a password. That way, it is easier to inspect the wallet file to understand what is going on.</p><p>Either way, the password is not fundamentally a ‘Bitcoin thing’. Even if password-protected at this stage, you can still restore a wallet based only on the seed phrase you’ve written down; and the password is not related to any part of the Bitcoin send/receive process.</p><p>That’s useful to understand — everything important can be recreated from your seed phrase, input into Electrum on a new computer perhaps, along with Electrum looking up the latest status on the Bitcoin network. Electrum will also store additional information in your wallet file, such as notes and descriptions you write for your own purposes, that can’t be restored from the seed phrase alone. But the point is that these aren’t fundamentally of value anyway.</p><h3>Paying Your Own Computer</h3><p>It’s time to transfer your Bitcoins out of Coinbase! To do this, you only need a Bitcoin address from your wallet. Based on your Bitcoin private key, Electrum generates a number of addresses, any one of which can be used to receive a transaction. The recommendation is not to use the same address more than once, but in fact it is perfectly possible to receive multiple donations to one address.</p><p>Click on the Receive tab. Enter a Description for your transaction (‘to first wallet’, for example), and the amount of Bitcoins you intend to send in mBTC (0.001 of a Bitcoin) — perhaps enter “0.01” mBTC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eE7bYpFJw68Ex1b_8yLK4w.png" /></figure><p>In practice, these details don’t matter. They are only for your own tracking purposes within Electrum so you know when you’ve handed out a particular address to someone already, and want to monitor it and see when the transfer has been made.</p><p>It is possible to construct a ‘signed request’ with all this extra information so that someone can know it’s genuinely you who is asking for this transfer to be made, but that’s not something we need here.</p><p>All we need is the address, so copy that from the ‘Address’ box on the right — the long string of letters and numbers.</p><h4>Back to Coinbase</h4><p>In Coinbase, click Trade and then Send.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCQoTfNBORPJpgL9DNlCYg.png" /></figure><p>Paste the Bitcoin address in the ‘To’ box, and enter a number of Bitcoins in the box at the top (0.0001 BTC is a sufficiently small amount, but large enough to be accepted, at the time of writing).</p><p>Make sure ‘Pay with’ shows ‘BTC Wallet’.</p><p>Click ‘Continue’ then the funds should be on their way!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7dOMXixRbVOtoieCclC8QA.png" /></figure><p>Coinbase will also take a small transaction fee, not for their own purposes but to offer to the Bitcoin network at large. This acts as an incentive for the ‘Bitcoin miners’ running the nodes to actually choose to include it the next block of the blockchain.</p><h4>Checking For Arrival</h4><p>After ten minutes or so, you should see the ‘Paid’ status appear in the Receive tab of Electrum:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qBDUa4oLIG7E2KqWSnu1Qw.png" /></figure><p>Actually, if you’re watching carefully in the Electrum ‘History’ tab, the transaction will first show up as ‘unconfirmed’ before resolving to ‘Paid’. Remember that the Bitcoin network is based on consensus. Electrum wants to see the transaction included in the new blockchain block by multiple nodes, and also to see its parent block used as the input block to subsequent blocks in the blockchain. As more blocks are added on top of your block, and this is confirmed by more nodes, the more sure you can be that your transaction is firmly embedded in Bitcoin’s blockchain history. After a relatively small threshold of checking for this, Electrum signals to you that the transaction is definitely complete.</p><h3>Your Place in History</h3><p>In the History tab, you can right-click on the completed transaction and select ‘View on block explorer’. This takes you to a website (blockstream.info seems to be Electrum’s choice, but there are other explorer websites including blockchain.com and explorer.bitcoin.com). There, you can see your transaction as displayed publicly on the Bitcoin blockchain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1CdgX7Ay72IYvMwqEzRfjg.png" /></figure><p>This shows some metadata about the finished transaction.</p><p>Let’s look a bit further down on the transaction page. You might expect the transaction to be simple: our Coinbase account on one side and our Electrum wallet on the other. Maybe there should also be something related to transaction fees. But as you can see, my destination Bitcoin address (highlighted) is one of many addresses in this transaction that I thought was simply from me to myself:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e98UGUrK0cphd987Joi-FA.png" /></figure><p>The reason for this is obvious in retrospect: Coinbase is a large operator serving many customers. It has large Bitcoin holdings of its own and groups together multiple customers who ask to send funds around the same time. So their approx $20,000 USD Bitcoin on the input side is split into maybe hundreds of smaller outputs. We can probably never know who owns the other addresses in our transaction, but in this case we can be fairly sure they are Coinbase customers or partners.</p><p>The metadata at the top of the page includes a link to the block that contains our transaction, and we can click through to find out more. The block itself will contain around 2,000 transactions: ours is only one of these. A lot of other transactions are more obviously balanced — one input being sent to one or two output addresses (sometimes the second address will actually belong to the sender and is intended to collect change from the transaction).</p><p>If you’re even more advanced than me, you could try creating a second wallet in Electrum and sending funds from one to the other. I presume in that case you will see a simpler transaction.</p><h3>Congratulations!</h3><p>Well done — you are now fully immersed in the world of cryptocurrency. At the very least, you can smile knowingly when friends talk about Bitcoin at a dinner party. If they notice and ask you about it, just tell them that as a crypto insider anonymity is a very important value, and you have no desire to leave any clues by exposing your vast crypto knowledge to mere beginners.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e5ed14273" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/zero-to-bitcoin-mogul-in-one-afternoon-7e5ed14273">Zero to Bitcoin Mogul in one Afternoon</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Data Dashboards Automatically, Reliably, and Securely]]></title>
            <link>https://medium.com/data-science/deploying-data-dashboards-automatically-reliably-and-securely-372ef802ca3c?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/372ef802ca3c</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[jupyterhub]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Fri, 11 Sep 2020 15:52:27 GMT</pubDate>
            <atom:updated>2020-09-11T15:52:27.906Z</atom:updated>
            <content:encoded><![CDATA[<h4>New open source software to securely automate deployment of data scientists’ interactive visualizations in all languages and frameworks</h4><p>Today’s data projects require interactive visualizations for them to stand out and impress clients or decision-makers in your organization. There is a growing list of open source dashboarding frameworks that allow data scientists to build user interfaces without having to learn Javascript or HTML.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTTssiv9TTjOtkg6JhG4Mw.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@thisisengineering?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">ThisIsEngineering</a> from <a href="https://www.pexels.com/photo/engineers-in-sound-studio-3861947/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>For example, Volià, described in its simplest form, displays a Jupyter notebook as a user-friendly web app — code cells are hidden, and the notebook is run automatically from top-to-bottom without the user having to shift+enter their way through it. Python-based widget libraries provide simple user controls such as sliders and dropdowns.</p><p>Data Scientists and Analysts can start using these frameworks by following a short tutorial. And they clearly have the technical ability to find a way to host the resulting web apps, perhaps on their own laptop temporarily, or by deploying through an AWS account.</p><p>But this simple deployment step turns out to be the most common blocker for the adoption of dashboarding frameworks in an organization.</p><p>Although the process can be straightforward, it is unrewarding in itself — some tedious commands to run just when the data scientist proudly finished their brand-new analysis. Even worse, it is error-prone and can leave sensitive data exposed.</p><p>Just as automated testing and continuous integration make releasing new features fun rather than a chore in traditional software development, any barrier to sharing the data dashboard stifles innovation and discourages new iterations.</p><p>Leaving the deployment choices to your data scientists is unwise, as technically astute as they undoubtedly are. They aren’t going to have time to secure the servers, and will likely choose the easiest authentication system. For secure regular deployment, you really need a unified approach to hosting across your data team that can be approved wholesale by your information security department.</p><p>These dashboards are technically web-apps. They don’t need IT to spend three months auditing them for security, but there needs to be an approved method for deploying them that wouldn’t horrify the IT department as much as the ad-hoc methods that your data scientists will use if left to their own devices.</p><p>You need to know where all your dashboards are running and how they are authenticated. Otherwise, there may be outdated and insecure servers running out there in the wild exposing your networks and sensitive data. When employees leave your organization, you need to be able to terminate their access to these dashboards. If the employee happens to be the one running a handful of dashboards on their personal AWS account, you don’t want to rely on them to remember what was running where so you can turn them off or transfer ownership!</p><h3>The ContainDS Product Suite</h3><p>Free open source ContainDS software products can provide a unified deployment platform for your data scientists, allowing them to share dashboards based on open source frameworks in an automated, secure, and reproducible way.</p><p>Any open source dashboarding framework can be utilized. Supported as standard are Voilà, Streamlit, Plotly Dash, Bokeh, Panel, and R Shiny. Those should be a good starting point for any projects your data science team is likely to face!</p><p><a href="https://containds.com/">ContainDS</a> is a collection of two main products and related open source technologies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*33kR2CPLzDJh4AfCytxn9A.png" /><figcaption>ContainDS Dashboards documentation (image by author)</figcaption></figure><p><em>ContainDS Dashboards</em> is a platform for hosting and sharing dashboards over the internet or an internal network, with named authenticated users — perhaps specific colleagues or clients.</p><p>Sometimes even this is too open, so where dashboards need to be shared offline due to lack of internet or for contractual reasons why the data can’t be accessed over a network, <em>ContainDS Desktop</em> is an app for your Windows or Mac computer allowing you to run dashboards on your local machine and share them with others as single flat files.</p><p>Here we’ll focus on the online dashboards software.</p><h4>Online Dashboards through JupyterHub</h4><p>ContainDS Dashboards is an extension for the popular <a href="https://jupyter.org/hub">JupyterHub software</a>. This makes it especially easy to install if you already have a JupyterHub in use, but setting one up for the first time is not too complicated and it can be useful to have anyway.</p><p>JupyterHub is a way to centrally manage Jupyter notebook environments for your whole team. The standard installation allows each user to spin up their own Jupyter notebook, and the ContainDS Dashboards extension allows them to directly start user-friendly dashboards instead, sharing them with other authenticated users.</p><p>Different ‘distributions’ of JupyterHub provide different approaches to maintenance and scalability. There are lots of bespoke options, but the two main paths are <a href="https://zero-to-jupyterhub.readthedocs.io/">Zero 2 JupyterHub</a> which runs on Kubernetes, allowing seamless scaling of resources over multiple machines for large numbers of users or projects; and <a href="https://tljh.jupyter.org/">The Littlest JupyterHub</a> to set up a single VM to run JupyterHub (given the range of VMs available on cloud providers these days, this can still support surprisingly heavy usage!).</p><h4>Loading your app’s files</h4><p>There are two main ways to choose app files for deploying a dashboard: your ‘file source’ can either be from your existing Jupyter server tree or from a Git repo (public or private).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*04szXPBraaMN2ofvwJpgQg.png" /><figcaption>Creating a Voilà dashboard from a GitHub repo (image by author)</figcaption></figure><p>If you are already heavy users of Jupyter notebooks, and perhaps just want to deploy notebooks as Voilà or Panel apps, then it might make sense to use the first option — Jupyter Tree. You can edit your notebooks as normal, then once you’re happy head to the Dashboards menu in JupyterHub to enter the path to your notebook and see it deployed automatically as a new dashboard.</p><p>Even better, there is now a companion Jupyter extension so you can create a dashboard directly from JupyterLab or Notebook with one click.</p><p>Alternatively, if you are used to editing your Streamlit, R Shiny, Plotly Dash apps etc on your local machine, it might be more convenient to check your code into a Git repo and then instruct ContainDS Dashboards to pull it straight from your repo and deploy it. You can use public or private Git repos, and GitHub integration means you can one-click login to JupyterHub through your GitHub account and automatically grant access to your repos in the process.</p><p>Using either file source method, you can also select from multiple Conda environments if you’ve made them available to your JupyterHub users.</p><p>It’s also important to select the correct ‘framework’ from the dropdown to ensure the right mechanism is used to serve the dashboard. As already listed, Voilà, Streamlit, Plotly Dash, Bokeh, Panel, and R Shiny are currently supported out-of-the-box, but it is easy to add any custom framework that works as a web app.</p><p>Once deployed, dashboards are really just like separate Jupyter servers, but instead of running Jupyter notebook they run directly the server software of your chosen framework. If you’ve ever tried the Voilà Preview button in a Jupyter notebook, you will be familiar with the end result — but in the case of ContainDS Dashboards the deployment has no Jupyter front-end at all. Your apps will be deployed as pure web apps. This is exactly what you need for apps that you are going to share with others… the end users should not be able to run arbitrary code on your server.</p><h4>Sharing with Other Users</h4><p>The new Dashboards menu that is added to JupyterHub is not only used to register a new dashboard for deployment, but also serves as a list of contents for any dashboards that have been shared with you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7DKEu-sGrH0xYSobQtiYUQ.png" /><figcaption>Dashboards main menu screen (image by author)</figcaption></figure><p>When you create the dashboard, you can choose whether to make it available to all users in your JupyterHub, or just to selected named users. JupyterHub allows a wide range of authentication methods — so, for example, using LDAP or Google Single-sign-on, all your colleagues can easily access your dashboards through an account that will be automatically created for them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*ssFxgy6zBCBnTfxnf2p8uA.png" /><figcaption>Selecting named users for dashboard access (image by author)</figcaption></figure><p>Authorized users can click into any dashboard that has been shared with them, click to confirm the OAuth consent screen, then immediately start interacting with the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Dy-jnVTvEYSfT_PNSOBUg.png" /><figcaption>Example dashboard visualizations running inside ContainDS Dashboards (image by author)</figcaption></figure><h4>Extendable and Configurable</h4><p>Everything about JupyterHub is highly configurable: from where you host it (Kubernetes, on a cloud VM, or on your internal network) to how users authenticate at login.</p><p>The same applies to ContainDS Dashboards — you have full control over the way it behaves, and you can even plug in your own dashboarding visualization frameworks (e.g. Flask-based web apps) just by editing the configuration files.</p><h3>Conclusion</h3><p>That was a quick overview of ContainDS Dashboards, explaining how easy it is for a data scientist to deploy a new interactive visualization to share with clients or colleagues.</p><p>Your data scientists are already experimenting with the new visualization frameworks that have appeared on the open source landscape over the last few years. Their apps work great on their development machine, but it’s always a pain when they need to deploy it.</p><p>Often, they revert to exporting a PDF or just copy-and-pasting graphs into emails instead. This is a real missed opportunity to allow decision-makers to truly immerse themselves in the data models.</p><p>If the dashboard does end up being deployed, it’s often not in an IT-approved manner, with simple authentication steps and hosting on arbitrary cloud servers.</p><p>For medium-to-large data science teams, different projects have different needs — and data scientists want to choose the open source frameworks that make sense to their own skills and the project’s requirements.</p><p>To overcome these problems, ContainDS Dashboards provides a unified deployment and sharing model that can be administered by IT and used effortlessly by data science teams whatever technologies they are using to drive their analyses.</p><p>For installation details see <a href="https://cdsdashboards.readthedocs.io/">ContainDS Dashboards documentation</a>.</p><p><em>Dan Lester is a co-founder of </em><a href="https://ideonate.com/"><em>Ideonate</em></a><em>, making tools for data scientists, including </em><a href="https://containds.com/"><em>ContainDS</em></a><em>, a data science deployment platform for teams working on discrete projects.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=372ef802ca3c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deploying-data-dashboards-automatically-reliably-and-securely-372ef802ca3c">Deploying Data Dashboards Automatically, Reliably, and Securely</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Looking under the hood at the Dashboarding frameworks]]></title>
            <link>https://medium.com/data-science/looking-under-the-hood-at-the-dashboarding-frameworks-f87d8b4f3f7a?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f87d8b4f3f7a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Thu, 23 Jul 2020 17:30:27 GMT</pubDate>
            <atom:updated>2020-07-23T17:30:27.056Z</atom:updated>
            <content:encoded><![CDATA[<h4>Code simple experiments that highlight the different execution models behind the popular dashboarding frameworks Voila, Streamlit, and Plotly Dash</h4><p>Data scientists today have a wide range of open source tools that will allow them to build user interfaces for their models without having to learn any Javascript — or even having to think about the UI at all beyond declaring which variables should be selectable by the user.</p><p>Once they believe the promise that this is achievable, the next questions are around the performance and programming details:</p><ul><li>Will each user on my dashboard need to wait during the initial data processing each time?</li><li>If I want to move beyond a simple UI in the future, how extensible is the framework I’ve chosen? Or will I run into a brick wall trying to make it do more?</li><li>When two users access the app at the same time, will they start to see each others’ data, and the calculations will go out-of-sync?</li></ul><p>Sometimes all you need is to run some simple code tests to understand how the frameworks operate. This article will demonstrate the execution models of three popular frameworks: Voila, Streamlit, and Plotly Dash. We’ll write a version of a basic UI in each framework, let two different users access it at the same time, and then we’ll observe how this affects our simulated data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8lzlkK7vG2flLNuN" /><figcaption>Photo by <a href="https://unsplash.com/@thevanegmond?utm_source=medium&amp;utm_medium=referral">Nathan Van Egmond</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The Tests</h3><p>We’ll simulate two main components of a dashboard: startup calculations, and a user interaction.</p><p>Often, a dashboard will need to read in or compute a predictive model when it first starts up. A common complaint when data scientists have started to build out a dashboard is “every time a new user accesses the dashboard, they have to wait for the model to recalculate”. We want to understand how easy it will be for our code to cache our model for each new user.</p><p>Instead of actually calculating a model, all we’ll do is store the current time in a variable. If that keeps showing the same time on each run, we’ll know that the ‘model’ has not been recalculated — we’re using the time variable that was cached on the initial server startup. If it changes every time a button is pressed, we’ll know that everything is being recalculated from scratch.</p><p>The user interaction will be real (there is no need to simulate it). All that we will offer is an ‘Increment’ button that will increase the value of a counter and display it again. We can observe this to understand if multiple users see the same value as each other, or if they are running isolated environments. Does the counter increment successfully for each user, or does it reset each time the button is pressed because the framework is recalculating the entire script from scratch?</p><p>We will access the dashboard in Chrome and also in Firefox to simulate two completely different users accessing the web app at the same time.</p><h3>Voila</h3><p><a href="https://voila.readthedocs.io/">Voila</a> is a solution to display your Jupyter notebooks as standalone web apps. Essentially, it hides code cells by default and simply runs your notebook from top-to-bottom, then finally allows the user to interact with any widgets that were defined on the page. Here is the code from our test notebook:</p><pre>from datetime import datetime<br>start_time = datetime.now()<br>print(start_time)</pre><pre>from ipywidgets import Button, Output, VBox<br>count = 0</pre><pre>button = Button(description=&#39;Increment&#39;)<br>out = Output()</pre><pre>def on_click(_):<br>    global count<br>    with out:<br>        count += 1<br>        print(&#39;Clicked! Start Time {}; Count {}&#39;.format(start_time, count))<br>        <br>button.on_click(on_click)</pre><pre>VBox([button,out])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QMdUtwM8Je89jo-rrUqouw.png" /><figcaption>Jupyter notebook ready to be Voila-ized</figcaption></figure><p>Running the Jupyter notebook is similar to using Voila as the server, and you can see above that the notebook is doing what we expected: the start_time variable stays the same once calculated, and the counter increments on each button click.</p><p>Let’s try running in the Voila server. We’ll open up in Chrome and Firefox separately. We’ll click ‘Increment’ in each browser alternately to see if the counter works as expected:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tp4Z4rqQ1aZ3TIbma3oHxg.png" /><figcaption>Voila notebook running in Chrome and Firefox</figcaption></figure><p>The counter works as hoped — each user gets an independent value. The time also still stays the same throughout each notebook’s run. But you can see that the time in each browser is different, meaning that the ‘startup calculations’ were run independently for each user. That implies that each user would have to sit through any long-running model calculation on each access of the web app.</p><p>This allows us to understand the Voila execution model and to think through its limitations and advantages for any project. There is (intentionally) a similarity with the Jupyter notebook model; it’s as though each user has loaded the notebook in a new kernel, and shift+entered their way through all the cells. Once at the bottom, we end up with a Python kernel state — i.e. some variables set in memory. Nothing more will happen really… unless we’ve hooked up some widgets to run Python code as a callback. This is what we’ve done with the Button widget. But the notebook code cells are disregarded at this point — everything that happens is based on the Python memory state at this point, including the button click callback function now just being run from memory rather than the notebook cell.</p><p>The details aren’t important to understand if you’re new to this, but hopefully, the results will make more sense when compared against the other two frameworks.</p><p>Some workarounds for the ‘startup calculation cost’ problem in Voila will be discussed later.</p><h3>Streamlit</h3><p>Here is a similar dashboard built with <a href="https://streamlit.io/">Streamlit</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0360ea7a29f11be8b5f72545c724c962/href">https://medium.com/media/0360ea7a29f11be8b5f72545c724c962/href</a></iframe><p>This time the results are very different:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYGIekhjXucgrlszgmYkRA.png" /><figcaption>Streamlit concurrency test 1</figcaption></figure><p>Not only do we see a different time in each browser initially, but every time we click on the Increment button, a new time is generated and the counter never progresses beyond 1!</p><p>This is because the execution model in Streamlit is different: the whole script is rerun from top-to-bottom every time the user interacts with it at all. This sounds inefficient, and it certainly can be, but it allows the simple declarative scripting style to work. In the Streamlit code, you can see the button’s click functionality is just inside an ‘if’ statement. There is no real callback mechanism needed at all, compared to the Voila widgets where we hook up a callback function to be executed on button click.</p><p>To work around this potential inefficiency, Streamlit provides a cache decorator. Here is an updated Streamlit example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7038ac473c89f107c39748b5eda3d403/href">https://medium.com/media/7038ac473c89f107c39748b5eda3d403/href</a></iframe><p>Now, we find that the ‘startup calculation’ of obtaining the current time is run only one time ever.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4mDEp2Ph0P6vj6FKfcTWQ.png" /><figcaption>Streamlit concurrency test 2</figcaption></figure><p>The time in both browsers is exactly the same, and this never changes when either ‘Increment’ button is pressed. While the cache decorator is simple and useful, it is limited to caching objects that can sensibly be ‘pickled’ in Python — i.e. converted into a format that can be loaded from or saved to disk.</p><p>As before, the count never increases. This omission of per-user state is something that Streamlit is looking to address, and a <a href="https://discuss.streamlit.io/t/two-people-on-same-session-state/3211">workaround hack</a> is available. But actually it isn’t that common to really need it: in most dashboards, state is normally held in widgets anyway which <em>are</em> maintained on a per-user basis by Streamlit. For example, a slider can be dragged by the user to change an input to the model, and its new value <em>does</em> survive the code rerun.</p><h3>Plotly Dash</h3><p>Here is a similar script for the Plotly Dash framework:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e9445b8ad6092e9daf5f644d2e56fff/href">https://medium.com/media/3e9445b8ad6092e9daf5f644d2e56fff/href</a></iframe><p>The code here is a bit more involved from the start, but that does lead to greater control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d2VX7nUhLpG9GWYNPeEbxg.png" /><figcaption>Plotly Dash concurrency example</figcaption></figure><p>Here the ‘startup calculation’ (the time) remains the same across both browsers and doesn’t change when the Increment button is clicked. In addition, the counter does increment successfully and independently per user. This uses the n_clicks property of Plotly Dash’s button component which it ties to each physical HTML button that is rendered.</p><p>Note that in the Plotly Dash example, the output is overwritten each time the button is pressed. In Voila, each button press appended new output. That behavior could easily be changed in either framework.</p><h3>Conclusion</h3><p>If you look again at the code for Plotly Dash, you can see that the web server is much more obviously exposed to your own code. Our code’s app.run_server call is where the server ‘loop’ is actually running. By contrast in Voila and Streamlit, the server was very much external to our own script — it was something on the outside ‘loading in’ our script to process it for itself.</p><p>For Plotly Dash this highlights that we <em>are</em> building a traditional web app, and the real bonus that the Dash framework provides is in exposing user interface components that can be used without writing any Javascript or HTML.</p><p>Generally, Voila and Streamlit are easier for non-programmers to start off with, but extensibility may become limited further down the line if you do want to build out more traditional ‘web app’ functionality.</p><p>This isn’t to say that any one of these frameworks is inherently ‘better’ than any other. They all have different designs that lend themselves to different scenarios.</p><p>Ultimately, Plotly Dash’s ease at working with per-user state and maintaining ‘startup calculations’ comes because it does not attempt to remove us from a standard Python programming environment. The flip side is that coding has a higher learning curve for someone coming from a Jupyter-only background -where Voila or Streamlit would allow them to add a few widgets to their usual top-to-bottom procedural code.</p><p>The truth is that command of the underlying programming language should enable you to overcome any of the problems discussed. For example, Voila will easily allow us to write our own code to save our ‘startup calculations’ to disk, and read them in if available on the next run. But this doesn’t feel natural when coming from a notebook environment.</p><p>Realistically, for any of these frameworks, basic working dashboards are built using copy-and-paste code snippets found from similar examples in the docs or out there on the web! To get started building dashboards, you might want some idea how quickly it might need to evolve into a larger web app, but the best way to choose a framework to try out is to understand the environment and coding style where you’re already most comfortable.</p><p><em>The sample code for this article is </em><a href="https://github.com/danlester/exec-model-tests"><em>available on GitHub</em></a><em>.</em></p><p><em>Dan Lester is a co-founder of </em><a href="https://containds.com/"><em>ContainDS</em></a><em>, a data science platform for teams working on discrete projects.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f87d8b4f3f7a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/looking-under-the-hood-at-the-dashboarding-frameworks-f87d8b4f3f7a">Looking under the hood at the Dashboarding frameworks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are Dashboards for Me?]]></title>
            <link>https://medium.com/data-science/are-dashboards-for-me-7f66502986b1?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f66502986b1</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Tue, 07 Jul 2020 14:57:41 GMT</pubDate>
            <atom:updated>2020-07-07T14:57:41.917Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding whether your Jupyter notebooks would be more useful as interactive dashboards</h4><p>As a data scientist or analyst, you often create data visualizations for yourself in Jupyter notebooks, copying graphs into emails and presentations. Your friends talk about the latest ‘interactive dashboards’ they produce for their colleagues, but it sounds like a lot of work, and would probably just confuse your clients anyway.</p><p>This article is not a tutorial to teach you anything in-depth about building dashboards. It takes you through building and deploying the simplest possible dashboard based on a Jupyter notebook, but only to expose you to the concept. By the end of the article, you’ll know what your dashboard friends are talking about and you’ll be in a better position to decide if interactive dashboards are something you want to pursue further.</p><p>I’ll talk through the choices for hosting dashboards, and also give an overview of the popular dashboarding frameworks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SMTEZRHFogxvYZ9S" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>The Rise of Open Source Visualizations</h4><p>Business Intelligence tools have long been used to create charts based on internal company data. Think Google Analytics reports, maybe used by a marketing department to understand the effectiveness of their campaigns and direct their future efforts.</p><p>These dashboards are ingrained into monolithic BI software suites, usually put together using a drag-and-drop interface.</p><p>As a data scientist or quantitative developer, you analyze a different set of data in Jupyter, and you don’t have any similar tools to build dashboards around your data. It sounds like you’d need to build an entire web app including front-end Javascript to introduce something like that.</p><p>Over the last few years, some great open source frameworks have appeared. They allow you to easily turn your Python (or R etc) analyses into interactive web apps without having to think about the front-end at all. Your language of choice is all you need to instruct the framework to take care of the user interface for you.</p><p>These frameworks will introduce widgets (sliders, checkboxes, drop-downs etc) for the end-user to input parameters, and they may also provide interactive graph or table functionality — being able to zoom into your charts, or select data ranges that update the graphics in related chart components.</p><p>I’ll talk a bit more about the available frameworks later, but first I’ll describe the kind of dataviz scenarios that already have you thinking that there must be a better way.</p><h3>You are the Widget</h3><p>When you’re working on a Jupyter notebook, it makes sense for you to update parameters yourself while you’re trying things out.</p><p>Here’s your first attempt at a graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAKX0v5m-q8Tf5xMt3hKwg.png" /><figcaption>Graph with t = 1.0</figcaption></figure><p>In the second cell, you’ve manually entered t=1.0. You copy the graph into an email, and then your client comes back and asks what would happen if you change t to 2.0.</p><p>So you change the cell and run again.</p><p>That’s no big deal a couple of times, and is very easy if the client is in the room with you. But at some point, it might make sense to take a step back and see if you can automate this process. We’re assuming that you’re not going to ask your non-technical client to shift+enter through the notebook themselves of course.</p><h3>At Home in Jupyter</h3><p>I promised this wouldn’t be an in-depth tutorial of a visualization framework, so I’ll pick the easiest possible way to turn our notebook into something interactive.</p><p>In the first instance, we’ll add a widget to the notebook itself, so you get to slide a slider in Jupyter (just instead of updating the <em>t</em> parameter in the code cell).</p><p>That’s still not user-friendly enough for our non-technical client though, who won’t want to see the code cells. So the next step will be to run the notebook in something called the ‘voila’ server instead of in the Jupyter server. It’s very similar to the Jupyter server, but instead of letting the user shift+enter through the code cells, it hides the code cells and just runs everything from top to bottom — showing only the output components, including our slider widget.</p><h4>Interact</h4><p>Jupyter has an important extension package called <a href="https://ipywidgets.readthedocs.io/en/latest/user_install.html">ipywidgets</a>. You might need to install that first (run !pip install ipywidgets in a Jupyter cell, then restart and reload the notebook).</p><p>Then all we need to do is refactor our graph-drawing code cell into a function, and apply the ‘interact’ decorator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9eCePGJ-0k8QA64uKu0twg.png" /><figcaption>Using interact from ipywidgets</figcaption></figure><p>The ‘interact’ function automatically detects that our function has a float argument (t) and provides a slider automatically. You can slide the slider and immediately see the different graphs based on the new input of <em>t</em>.</p><p>Great! Next, we’ll make this fully user-friendly.</p><h4>Viewing in Voilà</h4><p>Install the <a href="https://voila.readthedocs.io/">Voila</a> python package withpip install voila or <a href="https://voila.readthedocs.io/en/stable/install.html">similar for Conda</a>. You’ll need to restart your Jupyter server to pick this up. When you reload the notebook page, you’ll now have a Voila button in the toolbar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*jE2NjJ-B5Rf1VT1Mpks6SA.png" /><figcaption>The Voila preview button in the Jupyter toolbar</figcaption></figure><p>Click that and you’ll jump to a new page showing the Voila preview:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pI31_1j4OqreLYwoBADWFA.png" /><figcaption>Voila Preview of the notebook</figcaption></figure><p>As promised, the code cells are hidden, so all we’re left with is the interactive component. Add a few markdown cells as an introduction — and maybe change the default Voila template — then you have a fully functioning web app!</p><p>The Voila preview that we’ve seen so far is a useful extension to jump straight into our new dashboard, but it’s piggy-backing on the Jupyter server, and anyone who has access to the Voila preview would be able to change the URL and get back to the notebook view.</p><p>So to run our notebook safely, we can use the voila server directly, where previously we used the Jupyter server.</p><p>Where you ran jupyter notebook to run Jupyter before, stop that with Ctrl+C, and instead run voila ./Dashboard.ipynb (or provide the correct path to your notebook file).</p><p>This time you can see our dashboard is at the root (and only) URL of this voila-only server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*xvF2jPOHPBc4buyaxOa18A.png" /><figcaption>The Voila Dashboard</figcaption></figure><h3>Sharing Your Dashboard</h3><p>Exciting! You can already imagine your clients and colleagues believing you’ve turned into a full-stack web developer overnight. So how do you share your web app with your non-technical audience?</p><p>Unfortunately, this is where you may face a hurdle. To be clear, there is nothing particularly complicated technically about hosting Voila (or similar) dashboards. But one way or another you (or someone close to you) will have to maintain or buy some form of hosting service.</p><p>In some services there are free options for dashboards that can be public, or for low-usage tiers.</p><p>If your organization is already running a JupyterHub (or wants to) then my own ContainDS Dashboards project will allow you to automatically host your dashboards within your JupyterHub, available securely to authenticated users.</p><p>In general, there are two important aspects to consider:</p><ul><li>Where the hosting will exist</li><li>How to authenticate users</li></ul><p>Let’s assume we are still using Voila to run our dashboard — but most of the advice about hosting applies to any of the open source frameworks.</p><h4>Do It Yourself</h4><p>You can of course just run the voila server on any VM or physical server you happen to have.</p><p>This could be on cloud services such as AWS, Google Cloud, Heroku, or Azure. Generally, the big cloud providers will require you to configure your own server, and certainly by the time you’ve added HTTPS certificates you’ll need to run your dashboard server behind a standard web server like nginx or apache — i.e. web traffic reaches nginx and then reverse proxies to the voila server.</p><p>Heroku and Google App Engine make things a bit easier as everything is configuration-based, and HTTPS is generally a default.</p><p>However, authentication is a huge problem. How do you ensure that only your intended users can access the dashboard? Your web server can be configured with a password, and maybe even a list of allowed username/passwords, but this is cumbersome to update.</p><p>Again, none of this is technically difficult, but it requires details that you might want your IT department to manage in the longer term, especially for anything sensitive.</p><p>Also in the ‘do it yourself’ category is just running the voila server on your own computer, as you did for your own trials, and open up a ‘tunnel’ for your user. Again, authentication is missing, and you’ll need to keep your laptop open! For a quick share of non-sensitive data this could still be useful — see the <a href="https://voila.readthedocs.io/en/stable/deploy.html#sharing-voila-applications-with-ngrok">ngrok section</a> of the Voila docs.</p><h4>JupyterHub and Binder</h4><p>Where it’s OK for your dashboard to be public, there is a free service called <a href="https://mybinder.org/">MyBinder.org</a> that will launch a Jupyter notebook (plus access to Voila via its own URL) based on the contents of a Git repo. This requires you to update the Git repo of course, and takes a while for the environment to build the first time it starts after a new commit. You can send your client a direct link that will start things up and then jump straight to the Voila URL — but the Jupyter server will also be running at a different URL if they know how to enter it, just as when we were trying out the Voila preview earlier. MyBinder is essentially a public and open deployment of the open source JupyterHub project (a website that provides Jupyter notebook environments to multiple users).</p><p>Especially if your organization is already running its own JupyterHub, my own <a href="https://cdsdashboards.readthedocs.io/">ContainDS Dashboards</a> extension fits in neatly. Since a JupyterHub is already running (for regular Jupyter notebook servers), no extra infrastructure is required at all. JupyterHub has highly configurable authentication modules, so for example you can require login through your corporate email single sign-on.</p><p>ContainDS Dashboards adds an extra ‘Dashboards’ menu to JupyterHub. You simply specify the path to your notebook or script and select which framework you are using (e.g. ‘voila’, ‘streamlit’, etc as discussed later). Straight away, any user authenticated into your JupyterHub can access your dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*papGQizAZL4HV6WC5Vs0Ng.png" /><figcaption>ContainDS Dashboards</figcaption></figure><p>If your organization is not currently running a JupyterHub then it’s easy enough to start one, and it is popular enough that help is available for a range of different installation scenarios.</p><h4>Managed Services</h4><p>The easiest way to deploy a dashboard, without any infrastructure of your own, is through a managed service. These are generally tied to particular frameworks.</p><p>If you’re using the R Shiny framework for your dashboards (described later), then the <a href="https://www.shinyapps.io/">shinyapps.io</a> cloud service is an easy choice for a one-off deployment. Once you’ve configured an access token in your R environment, you only have to run deployApp() to automatically upload your app.</p><p>On the free and basic plans, dashboards are public. For Standard plans and above ($99 per month or higher), authentication is included. On the shinyapps.io site you can enter email addresses to invite to your app. They can login with GitHub or Google single sign-on or just set up a shinyapps account based on that email address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7yF1wCG8IA1DB1AUJ2ERNg.png" /><figcaption>shinyapps.io</figcaption></figure><p>A similar service exists for the <a href="https://plotly.com/dash/">Plotly Dash</a> framework (R or Python). Dash Enterprise allows you to connect a Git repo so commits are automatically deployed to this cloud service. They have a wider range of authentication methods, again including basic email login.</p><p>Streamlit (a Python framework) is also working on a Teams product.</p><p>Most of these managed services are ‘tribal’ — only catering for the framework they’ve been built around. But they definitely take care of the hosting and authentication!</p><p>A new service called <a href="https://datapane.com/teams/">Datapane</a> attempts to be framework agnostic, although requires you to add some extra deployment Python code directly to your script. Published reports appear within Datapane’s own-style web interface, and generally the end-user needs to understand that they have to run the report. They don’t appear to launch straight into it as they do for Voila. It’s worth a look for regular sharing within teams.</p><h4>Offline Deployment</h4><p>Another component of my own ContainDS suite is the <a href="https://containds.com/desktop/">Desktop</a> product. This is a desktop app that both you and your client can install locally. You can then export your entire Jupyter environment as a single flat file to send to your client who can import it and see it running through Voila as intended.</p><p>This is useful for very sensitive situations, or where you don’t want to involve servers at all. However, both parties need to be running Docker or VirtualBox, so there is some overhead to get started…</p><p>The principle is that this is much more reliable than trying to get your client to install Conda and Python environments!</p><h3>Overview of Frameworks</h3><p>You might have noticed that the ‘Sharing Your Dashboard’ section wasn’t entirely conclusive. Deployment is still a painful problem — again, not technically insurmountable, but it always needs infrastructure in place.</p><p>Sorry if this destroyed your original excitement about building your own dashboards! But I think it’s important to tell you the truth…</p><p>If you still believe that dashboards could transform your data story-telling abilities, here is a brief overview of some popular open source dashboarding frameworks.</p><p>I think they can be roughly broken down by: language(s) supported; visualization features; and ease-of-use versus UI control.</p><h4>Streamlit</h4><p><a href="https://streamlit.io/">Streamlit</a> is an interesting new framework based on simple Python declarative scripting. It’s easy to create great-looking web apps using widgets (e.g. sliders and dropdowns) in a Python script that makes intuitive sense to the developer without having to worry about ‘events’ — how the code should react when something is clicked.</p><p>Whereas other frameworks may require you to think through the user flow yourself and connect up events, Streamlit by default takes care of this logic for you. As long as you’ve declared UI components, when the user interacts with it, Streamlit will take care of re-running the appropriate parts of your script.</p><p>This simplistic paradigm comes back to make some things more complicated of course — such as having to add caching decorators if too much of your code is being run too often.</p><p>Streamlit scripts must be run through the streamlit run command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2SY2Z0QloXorZhj29Yw4RQ.jpeg" /><figcaption>Data Explorer in Streamlit by <a href="https://twitter.com/CWerner76/status/1236777905893576704">Christian Werner</a></figcaption></figure><h4>Plotly Dash</h4><p><a href="https://plotly.com/dash/">Plotly Dash</a> is a ‘full-stack’ framework whereby you can use Python, R, or Julia to control not only the back-end data processing side of things, but also the front-end UI. Compared to Streamlit, you more often have to think through the way the user interface flow will work, connecting events. That’s not necessarily a bad thing, and ultimately gives you more control earlier on.</p><p>Similarly, you have more options for how you can work a Plotly Dash app as a sub-app into a larger application. The underlying server mechanism is more clearly exposed.</p><h4>Bokeh and Panel</h4><p><a href="https://bokeh.org/">Bokeh</a> grew out of Anaconda Inc (the virtual environment company) and is supported by the NumFOCUS non-profit which also supports Jupyter and many other scientific computing packages such as NumPy.</p><p>It is a Python framework, and its strength is in really detailed control of interactive graphs.</p><p>As a direct comparison, most people say they find Plotly Dash easier to get started and ultimately provides a similar level of control. But Bokeh can dive a bit deeper into more complex visualizations.</p><p>There is a highly extensive range of open source components that sit on top of Bokeh.</p><p><a href="https://panel.holoviz.org/">Panel</a> is sometimes seen as a separate visualization framework, but originally at least, it extended Bokeh. It provides a more flexible widget system, not just focusing on graphics. And it has a design goal of being able to run almost identically in a Jupyter notebook as it can as a standalone app.</p><p>Panel allows you to easily incorporate other open-source visualization components into your app. It is itself part of the wider <a href="https://holoviz.org/">HoloViz</a> effort to simplify data visualization in Python.</p><p>The kind of thing that sets Panel and HoloViz apart is the ability to connect multiple graphs together, so that interacting with one (e.g. selecting a range of points) updates the other graphs in a corresponding way.</p><p>There are academic fields that have specialized ecosystems built around HoloViews, e.g. Geographic; and toolsets for dealing with visualizations of large datasets.</p><p>My feeling is that if you have an unusual or detailed type of visualization, it is more likely to be possible to achieve with the Bokeh / HoloViz stack than other frameworks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wBfDsQUKiBLBxOnm7wlFXw.png" /><figcaption><a href="https://holoviz.org/">HoloViz website</a></figcaption></figure><h4>R Shiny Server</h4><p>If you’re an enthusiast of the R language you’ll already know about the <a href="https://shiny.rstudio.com/">Shiny framework</a> for interactive apps… and if you’re firmly a Pythonista you’re not about to switch now!</p><p><em>Shiny user interfaces can be built entirely using R, or can be written directly in HTML, CSS, and JavaScript for more flexibility.</em></p><p>If you’re only building a dashboard, and R is perfect for your analysis, Shiny should be perfect for your UI too. But it’s worth bearing in mind that if your code is going to develop into a larger web app in the future, you may end up missing the database and general software engineering power brought by a more general-purpose language like Python.</p><p>Shiny is part of a powerful suite of data scientists’ tools brought to you by RStudio.</p><p>Note that Voila of course supports R notebooks. If you have a simple Jupyter notebook written in R then Voila may be the quickest way to make it interactive.</p><h4>Voilà</h4><p>We already demonstrated Voilà earlier. Its strength is being ‘Jupyter native’. Any languages supported by Jupyter can work with Voilà too. The interactive widgets provided natively are basic, but this is an easy way to go from notebook to web app.</p><p>You don’t need to be limited to Jupyter’s widgets either. It is entirely possible to build a notebook using Panel components, and this can be deployed through Voilà (as well as Bokeh/Panel directly).</p><p>To understand how ipywidgets works, here is an <a href="https://towardsdatascience.com/bring-your-jupyter-notebook-to-life-with-interactive-widgets-bc12e03f0916">excellent tutorial</a>.</p><h3>Summary</h3><p>Maybe your clients only ever wanted a PDF anyway. It doesn’t always make sense to turn your analysis into an interactive dashboard, but hopefully I’ve given you some ideas to help you recognize when the back-and-forth email exchanges with your customers could move up a level.</p><p>The fun part, for a data scientist or analyst, is undoubtedly in using the frameworks to build a satisfying user interface. I hope the overview of the open source frameworks will help you find a suitable framework to try out.</p><p>Hosting and authentication is often a sticking point, though. I’ve given you an overview of the challenges there to save you investing time in learning a dashboarding framework only to realize that you don’t have the resources or mandate to get your dashboard into the hands of your clients or colleagues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f66502986b1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/are-dashboards-for-me-7f66502986b1">Are Dashboards for Me?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Roundup of Data Visualization Frameworks]]></title>
            <link>https://medium.com/@dslester/a-roundup-of-data-visualization-frameworks-c254fdd33108?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c254fdd33108</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[jupyterhub]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Fri, 19 Jun 2020 13:21:14 GMT</pubDate>
            <atom:updated>2020-07-02T10:08:07.235Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://containds.com/dashboards/">ContainDS Dashboards</a> is an extension for JupyterHub allowing you to publish interactive visualizations for non-technical colleagues to try out. There are now a number of different open-source visualization frameworks supported out-of-the-box, and this post summarises those available, helping you decide which might be right for your project.</p><h3>Voila</h3><p><a href="https://voila.readthedocs.io/">Voila</a> is very closely related to Jupyter itself — it’s really just a user-friendly and secure version of the Jupyter Notebook or Lab server that you might use to build out your prototype in the first place. Basically, it runs the notebook from top-to-bottom, hiding code cells by default, and blocking arbitrary code from being executed.</p><p>The problem with showing Jupyter notebooks directly to colleagues is that the code cells will get in the way — either by clouding their reading experience, or because the user may make unexpected or even unsafe changes. Voila solves this in a way that closely mirrors the work you’re already doing in Jupyter. Add in some <a href="https://ipywidgets.readthedocs.io/">interactive widgets</a> and you can very easily build a web app!</p><p>Since apps are just Jupyter notebooks, you can write them in any language that works for you — Python, R, or <a href="https://github.com/jupyter/jupyter/wiki/Jupyter-kernels">anything supported</a> by Jupyter.</p><p>Our <a href="https://cdsdashboards.readthedocs.io/en/stable/chapters/userguide/voila.html">Voila user guide</a> shows the simple deployment of a Voila dashboard on your JupyterHub using ContainDS Dashboards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RCaVMYghiQWURJS7aKXJ1w.png" /><figcaption>Example from <a href="https://voila-gallery.org/">Voila Gallery</a></figcaption></figure><h3>Streamlit</h3><p><a href="https://streamlit.io/">Streamlit</a> is a really interesting new framework based on a simple Python declarative scripting paradigm. It’s easy to create great-looking web apps using widgets (e.g. sliders and dropdowns) in a Python script that makes intuitive sense to the developer without having to worry about ‘events’ — how the code should react when something is clicked.</p><p>Whereas other frameworks may require you to think through the user flow yourself and connect up events, Streamlit by default takes care of this logic for you. As long as you’ve declared a slider, when the user interacts with it, Streamlit will take care of re-running the appropriate parts of your script.</p><p>This simplistic paradigm comes back to make some things more complicated of course — such as having to add caching decorators if too much of your code is being run too often.</p><p>Streamlit scripts must be run through the ‘streamlit run’ command.</p><p>See our Medium article <a href="https://medium.com/@dan_19973/streamlit-apps-deploy-instantly-with-zero-configuration-7729944c649c?source=friends_link&amp;sk=1f9b765b9b3d3c1501c4815c078f2962">Streamlit apps: Deploy Instantly with Zero Configuration</a> or our <a href="https://cdsdashboards.readthedocs.io/en/stable/chapters/userguide/streamlit.html">Streamlit user guide</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2SY2Z0QloXorZhj29Yw4RQ.jpeg" /><figcaption>Streamlit Covid-19 Explorer by <a href="https://twitter.com/CWerner76/status/1236777905893576704">Christian Werner</a></figcaption></figure><h3>Plotly Dash</h3><p>Plotly Dash is a ‘full-stack’ framework whereby you can use Python, R, or Julia to control not only the back-end data processing side of things, but also the front-end UI. Compared to Streamlit, you more often have to think through the way the user interface flow will work, connecting events. That’s not necessarily a bad thing, and ultimately gives you more control earlier on.</p><p>Similarly, you have more options for how you can work a Plotly Dash app as a sub-app into a wider-ranging application. The underlying server mechanism is more obviously exposed.</p><p>According to <a href="https://plotly.com/dash/">their website,</a> Plotly Dash is <em>the most popular framework for building ML and data science apps</em>. It is well commercialized too (much like R Shiny — see below). That may give you more confidence in the development teams’ sustainability and ability to offer commercial options when needed, or, depending on your outlook, you may have more confidence in a crowd-built open-source framework that has been around for a long time (such as Bokeh).</p><p>See our Medium article <a href="https://medium.com/analytics-vidhya/plotly-dash-apps-deploy-instantly-with-zero-configuration-fe1e5730763d?source=friends_link&amp;sk=f9160aaedf1b0ac7b70053684cfb5f31">Streamlit apps: Deploy Instantly with Zero Configuration</a> or our <a href="https://cdsdashboards.readthedocs.io/en/stable/chapters/userguide/plotlydash.html">Plotly Dash user guide</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7ET7e2Clv2GwaAmD9l2Yw.png" /><figcaption>Example from <a href="https://dash-gallery.plotly.host/Portal/">Plotly Dash Gallery</a></figcaption></figure><h3>Bokeh</h3><p>This is a long-standing framework that, in the Python world at least, popularised the idea of a visualization framework.</p><blockquote><em>Bokeh is an interactive visualization library for modern web browsers. It provides elegant, concise construction of versatile graphics, and affords high-performance interactivity over large or streaming datasets. Bokeh can help anyone who would like to quickly and easily make interactive plots, dashboards, and data applications.</em></blockquote><blockquote><a href="https://docs.bokeh.org/en/latest/index.html"><em>The Bokeh website</em></a></blockquote><p>Bokeh grew out of Anaconda Inc (the virtual environment company) and is supported by the NumFOCUS non-profit which also supports Jupyter and many other scientific computing packages such as NumPy.</p><p>Its strength is in really detailed control of interactive graphs.</p><p>As a direct comparison, most people say they find Plotly Dash easier to get started and ultimately provides a similar level of control. But Bokeh can dive a bit deeper into more complex visualizations.</p><p>See our <a href="https://cdsdashboards.readthedocs.io/en/stable/chapters/userguide/bokehpanel.html">Bokeh user guide</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9GCAlsBk6NTPo3wb" /><figcaption>Example from the <a href="https://demo.bokeh.org/">Bokeh Gallery</a></figcaption></figure><h3>Panel</h3><p>Panel builds on top of Bokeh. It provides a more flexible widget system, not just focusing on graphics. And it has a design goal of being able to run almost identically in a Jupyter notebook as it can as a standalone app.</p><p>As a result of this background, ContainDS Dashboards doesn’t actually have a ‘panel’ presentation type for you to choose directly. You can run Panel ipynb files through either the Voila or Bokeh deployment options! (Bokeh must be used for Panel in py scripts.)</p><p>Panel allows you to easily incorporate other open-source visualization components into your app. It is itself part of the wider <a href="https://holoviz.org/">HoloViz</a> effort to simplify data visualization in Python.</p><p>If you like open-source and grant-supported projects that somehow seem to tie together the broader scientific computing movement, Panel is as pure as it gets!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vvnnmsx5gZQnd8www_tKcQ.png" /><figcaption>Example from <a href="https://gapminders.pyviz.demo.anaconda.com/gapminders">Panel Gallery</a></figcaption></figure><h3>R Shiny Server</h3><p>If you’re an enthusiast of the R language you’ll already know about the <a href="https://shiny.rstudio.com/">Shiny framework</a> for interactive apps… and if you’re firmly a Pythonista you’re not about to switch now!</p><p><em>Shiny user interfaces can be built entirely using R, or can be written directly in HTML, CSS, and JavaScript for more flexibility.</em></p><p>If you’re only building a dashboard, and R is perfect for your analysis, Shiny should be perfect for your UI too. But it’s worth bearing in mind that if your code is going to develop into a larger web app in the future, you may end up missing the database and general software engineering power brought by a more general purpose language like Python.</p><p>Shiny is part of a powerful suite of data scientists’ tools brought to you by RStudio.</p><p>Note that Voila of course supports R notebooks. If you have a simple Jupyter notebook written in R then Voila may be the quickest way to make it interactive.</p><p>See our <a href="https://cdsdashboards.readthedocs.io/en/stable/chapters/userguide/rshiny.html">R Shiny Server user guide</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wlHVLZZn8hPV2Qj7" /></figure><p>All these frameworks have their strengths and weaknesses. If you have a team of data scientists working on discrete bespoke projects then you’ll find that different frameworks will make sense for different problems.</p><p>That’s why the ContainDS Dashboards JupyterHub extension allows your team to use your existing JupyterHub authentication and hosting infrastructure to deploy web apps built in <em>any</em> open source framework.</p><p><em>Originally published at </em><a href="https://containds.com/2020/06/19/a-roundup-of-data-visualization-frameworks/"><em>https://containds.com</em></a><em> on June 19, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c254fdd33108" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Plotly Dash apps: Deploy Instantly with Zero Configuration]]></title>
            <link>https://medium.com/analytics-vidhya/plotly-dash-apps-deploy-instantly-with-zero-configuration-fe1e5730763d?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe1e5730763d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[dash]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Wed, 03 Jun 2020 13:22:01 GMT</pubDate>
            <atom:updated>2020-09-08T12:13:40.607Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new workflow allowing data scientists to deploy their own Plotly Dash apps to be shared automatically and securely with colleagues</h4><p>Building prototype visualizations with <a href="https://plotly.com/dash/">Plotly Dash</a> is easy — getting things running on your own computer, at least. Sharing with a wider audience is a bit more complicated. It’s possible to configure cloud servers to host your app, but it takes a bit of time to secure properly under HTTPS, and it then becomes yet another service for you to monitor.</p><p>The investment in deployment infrastructure makes sense if you are building a long-term web app that you want to share permanently with the entire planet. But what about smaller throw-away apps that illustrate a proof-of-concept for your colleagues or clients, and are destined for speedy iteration throughout the life of a finite project?</p><p>Wouldn’t it be better to deploy any new Plotly Dash app automatically, reliably, and securely?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C60iPuRQIXKbyGh5yKf0YQ.png" /><figcaption>An example Dash app from Plotly’s gallery</figcaption></figure><h4>A New Use for your JupyterHub</h4><p>Many data science teams are already running a JupyterHub for easy experimentation with Jupyter notebooks. If not, it’s easy enough to get one started — and once running, it can serve as the center point for all your data science prototypes and dashboards going forward.</p><p>The latest release of a JupyterHub extension called <a href="https://cdsdashboards.readthedocs.io/">ContainDS Dashboards</a> adds a simple ‘Dashboards’ menu to your JupyterHub control panel. It allows data scientists to simply register a dashboard for any new Plotly Dash script to automatically launch it as a secure and user-friendly web app accessible to any other user of your JupyterHub.</p><p>The Dash app is run and managed using the same underlying infrastructure as your regular Jupyter servers, so this means that security and authentication can be managed using JupyterHub’s usual range of options (e.g. single sign-on through corporate email, or LDAP). No extra management of the Dash apps is required — the data scientist responsible for the app can remove it or redeploy it, entirely through the JupyterHub website.</p><p><strong>UPDATE: You can now deploy dashboards direct from a Git repo, and share only with named users. See </strong><a href="https://cdsdashboards.readthedocs.io/en/stable/"><strong>ContainDS Dashboards docs</strong></a><strong> for the latest info.</strong></p><h3>Simple Dash Deployment Workflow</h3><p>Let’s take a look at how easy it is for a data scientist to deploy their own Dash app as a ‘dashboard’ for their colleagues to try out.</p><p>Logged into their Jupyter server on JupyterHub, they upload their Dash script (plotlyapp.py in this example) plus any image or CSS/JS assets. Our example <a href="https://github.com/ideonate/cdsdashboards/tree/master/examples/sample-source-code/plotlydash">source code is here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ROSvbzHOgxNZ4T8n9F4vg.png" /><figcaption>Uploading our Dash app source code to the Jupyter tree</figcaption></figure><p>Click ‘Dashboards’ in the JupyterHub Control Panel’s menu bar. You will see the page showing any dashboards created by you, or shared with you by colleagues. As we’re new to ContainDS Dashboards, we don’t see any existing dashboards:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RCmQd_X52wRx3kxFbYMdmw.png" /><figcaption>Dashboards page in JupyterHub</figcaption></figure><h4>Register a Dashboard</h4><p>Click ‘New Dashboard’ so we can register a dashboard for our Plotly Dash app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ajg7kOhL5YebBQJwKcXfhw.png" /><figcaption>New Dashboard for our Dash app</figcaption></figure><p>Fill in a name (‘Plotly Dash Test’) and optionally a description.</p><p>The default Jupyter notebook server (i.e. ‘My Server’ from the JupyterHub home page) should already be selected as the source. If you have other named servers they should be available here.</p><p>Select the framework required. For our example, we need to change this to <em>plotlydash</em>.</p><p>Specify the URL-path to the Python file of our Dash app, relative to the home folder. In our case, plotlyapp.py was at the top level in our Jupyter tree, so we just enter plotlyapp.py.</p><p>Click ‘Save’.</p><h4>Run the Dashboard</h4><p>When you click Save, the dashboard will be built automatically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0xhRS3ulELu0qn_glWg2vQ.png" /><figcaption>Dashboard building</figcaption></figure><p>Click ‘Go to Dashboard’ and you’ll see your Dash app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFj_BRMKdwPO-XWmRhdljw.png" /><figcaption>The Dash app running in JupyterHub</figcaption></figure><p>That was easy!</p><h3>Sharing with Colleagues</h3><p>Once you’ve registered your dashboard as above, you don’t really have to do anything to share with your colleagues. Dashboards are automatically available to anyone logged in to the JupyterHub. Just ask them to visit the Dashboards menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oe9heQhvrRLZq_1X9F0z7A.png" /><figcaption>Dashboards page lists our app to another user</figcaption></figure><p>Another user can see our dashboard in the list above. They can click on ‘Plotly Dash Test’ to reach the dashboard summary page, where they’ll be able to click ‘Go to Dashboard’ just as you did.</p><p>Before accessing your app they’ll need to click to confirm they authorize this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lQ07tJj6uLtjFRRuBPBKWQ.png" /><figcaption>Authorization for our app</figcaption></figure><p>This is because your app has access to their API tokens which can be used to identify their JupyterHub account.</p><h3>Working with Multiple apps</h3><p>If you want to deploy more Plotly Dash apps, just upload them to your JupyterHub home folder as you did for the first one, and then register a new Dashboard corresponding to it.</p><p>Of course, you will need any dependencies for your app to be installed in the JupyterHub user environment — this includes Dash itself.</p><p>To force a rebuild of your Dashboard, click ‘Edit’ next to it in the Dashboard list in JupyterHub. When you click ‘Save’ the Dashboard will always be rebuilt.</p><h3>Moving Forward with JupyterHub</h3><p>If you already have a JupyterHub, or think it’s a useful investment to <a href="https://jupyterhub.readthedocs.io/en/stable/index.html">start one</a>, I really hope you’ll see the benefit of using that as the basis for your Plotly Dash deployments too.</p><p>I think it makes a lot of sense for iterative and experimental apps that are intended to be shared within your organization.</p><p>Especially in corporate settings where data is sensitive, it becomes difficult to keep track of random deployments by data scientists to different cloud servers and to ensure that appropriate data security measures are in place. Using JupyterHub means the infrastructure can be established once, safely within the organization’s private network or in securely monitored cloud locations.</p><p>If you or other data scientists are Jupyter notebook users, ContainDS <a href="https://cdsdashboards.readthedocs.io/">Dashboards</a> can also be used to present user-friendly and secure versions of Jupyter notebooks in exactly the same way. Instead of using the ‘Plotly Dash’ framework when you create a Dashboard, just select ‘Voila’ instead and specify the path to an ipynb file (a Jupyter notebook). <a href="https://github.com/voila-dashboards/voila">Voila</a> is really just a user-friendly and safe version of the Jupyter server — it hides code cells, and only lets the user interact with any widgets or output on the page.</p><p>ContainDS Dashboards also supports the <a href="https://www.streamlit.io/">Streamlit</a> framework — a great new way to make interactive data science apps using a simple declarative scripting interface.</p><p>Please <a href="https://cdsdashboards.readthedocs.io/">try out ContainDS Dashboards</a> for yourself, and let me know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe1e5730763d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/plotly-dash-apps-deploy-instantly-with-zero-configuration-fe1e5730763d">Plotly Dash apps: Deploy Instantly with Zero Configuration</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Scientists are Dangerous]]></title>
            <link>https://pub.towardsai.net/data-scientists-are-dangerous-31279f2d1d0c?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/31279f2d1d0c</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-security]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[opinion]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Fri, 15 May 2020 18:01:01 GMT</pubDate>
            <atom:updated>2020-05-15T18:01:01.193Z</atom:updated>
            <content:encoded><![CDATA[<h4>Opinion</h4><h4>How data scientists’ priorities and workflows may be the biggest data security threat to your organization</h4><p>There is a struggle over data in the modern enterprise: for good offensive strategy, employees need access to as much data as possible. But the defensive side pulls in the opposite direction — carefully restricting data access to avoid costly data breaches and to comply with regulations¹.</p><p>Data scientists are the heroes unlocking the potential of your organization’s data, but without processes to help them protect the data they wield so skilfully, there is a danger that the mixture of skills, pressures, and toolsets found in your data science teams are a hidden simmering volcano of data security nightmares.</p><p>Worries exist throughout the data preparation stage, but in project-based data science, problems really surface when it’s time to share prototypes with clients or colleagues.</p><p>The average cost of a data breach is $8 million. If you’re manipulating a large client’s data and you suffer a data breach, you could cost them over $100 million…²</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wUakpnOyTKkXYEIbFOUrJA.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@oidonnyboy?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Nick Wehrli</a> from <a href="https://www.pexels.com/photo/bird-s-eye-view-photo-of-volcano-3762284/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><h3>Trust Me I’m a Scientist</h3><p>No CEO today can afford to say “nothing” when asked what projects they are undertaking to apply machine learning or data science in their company. Data science projects are often started (in part) as a response to a trend, and also with high hopes that they can bring some sort of magic results to the company. For those reasons, employees may lower the security bar when sharing information and files with the data science team. They are <em>data</em> scientists, after all!</p><p>For discrete one-off projects, the data scientist struggles to obtain the data they need to investigate their brief. Once it arrives, it’s not complete.</p><p>“How do you expect me to perform any analysis without customer spending data and ZIP codes?”</p><p>Rather than frustrate the data scientist for another two weeks, after a little hesitation, the reply comes: “Oh, I’ll just send you all the data I’ve got.”</p><p>After it’s been emailed over, maybe now it’s safe on the data scientist’s laptop.</p><p>Until the analysis is complete.</p><h3>Web Deployment under the Radar</h3><p>Most open-source data science frameworks have a user interface that works through a web browser. Often the web isn’t inherently important to the project — but web browsers provide a universal and powerful front-end. And sometimes, the web <em>is</em> a suitable mechanism for sharing the final result.</p><p>So maybe the data scientist has produced an interactive model in a Jupyter notebook (or as a Streamlit, Dash, or Flask app). The client or decision-makers who commissioned the project are going to love moving the sliders to see the predictions change!</p><p>If you are the CISO or IT manager, look away now.</p><h4>Disconnected Laptops</h4><p>The data scientist is not necessarily a computer scientist (but knows enough, should they have time to try).</p><p><em>I’ll run a secure tunnel to my laptop so I can just email the client the link!</em> (Didn’t IT have a policy about that? Oh, never mind — I’ll take it down at the end of the day anyway.)</p><p><em>Let’s spend a few minutes (hours?) installing Python on the sales guy’s laptop so he can demonstrate the dashboard at the client’s site!</em> (Must remember to at least remove the ALL_CONFIDENTIAL_DATA.CSV file when we’ve finished.)</p><p>Of course, it’s not beyond the data scientist to carefully control these scenarios and ensure compliance with your organization’s IT security requirements. But it’s not their real priority, and in their high-pressure job, it’s not something that <em>should</em> deserve their brainpower.</p><h4>Unsecured and Unapproved Cloud Hosting</h4><p>Well, it <em>is</em> a web app, so our prototype dashboard surely belongs to web hosting. Cloud hosting might even be in line with corporate data policy for this client’s data.</p><p>The problem is that IT needs a form to be completed before you are allowed to host a new application. Your code and deployment configuration need to be reviewed and formally tested for vulnerabilities.</p><p>Well, it’s not <em>really</em> a web app, certainly not one that needs to find its way into the long-term maintenance schedule. So it falls through the cracks.</p><p>The data scientist uses his personal AWS account; or just runs this tiny Jupyter web server as a separate process on an existing VM. Or something else that sounds innocent enough but ultimately would never be IT-approved for the long term deployment of a standard web app.</p><p>By the time the client gets to see the finished link, either it’s not secure, <em>or</em> the data scientist has wasted half a day guessing again how to configure the cloud provider’s VMs. Neither sounds great for your business. (Don’t worry, the server had SSL. Self-signed because Let’s Encrypt was harder, but the client knows it’s from us because they’re expecting the link.)</p><p>(Oh — must set a reminder that the server needs to be stopped in the morning before I forget where it’s hosted.)</p><p>Misconfigured cloud servers cause 43% of all exposed records, according to a study by IBM³.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YhF6t1g_UTWH0qnPeD2_dw.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@eberhardgross?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Eberhard Grossgasteiger</a> from <a href="https://www.pexels.com/photo/down-angle-photography-of-red-clouds-and-blue-sky-844297/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><h3>Does this happen in the real world?</h3><p>Of course, this story is fabricated, and the data scientist is deliberately painted as careless just to make the point — I’m fully aware that most data scientists have the skills and awareness to control these scenarios. The dangers highlighted in this article have all been raised by data scientists themselves in conversations with them during research for the data science platform <a href="https://containds.com/">ContainDS</a>.</p><p>But the fact remains that in many cases, IT policies and procedures have not adapted to cater to discrete project-based data science.</p><p>AI startups may have the incentive to adopt wholesale data science platforms to manage everything from experiments to final deployment of their core product. But project-based data science is still best performed in small teams who should be free to use the ideal environment for the brief. In one department, you might have different data scientists working with Jupyter, Streamlit, Dash, and R Studio — all independently, but ultimately with the same range of requirements for deployment to the decision-makers who need the results.</p><h3>Who needs to move this forward?</h3><p>Data science team leaders need to make the most noise about the fact that, although their output is software, their workflows and requirements are not the same as those of other software teams. Data engineers might be best placed to make technology suggestions and to maintain the infrastructure required.</p><p>The IT department, empowered by the Chief Information Security Officer, needs to drive change and prepare the policies and infrastructure that will help remove repetitive sysadmin guesswork from data scientists’ lives. By reading this, you’ve become aware of a new security hole in your organization. You have a job to do now.</p><p>Especially for organizations that want the flexibility brought to them by open-source data science frameworks, at <a href="https://containds.com/">ContainDS</a>, I’m helping build a suite of products for sharing data science visualizations with decision-makers, either locally or through authenticated web hosting. We believe that the publishing infrastructure needs to be ready-and-waiting for new project prototypes and dashboards to slot into as soon as they can be shared. Completing a project should not trigger a scramble to find a deployment solution.</p><h4>References</h4><p>[1]: Harvard Business Review (May 2017). The 2 Types of Data Strategies <a href="https://hbr.org/2017/05/whats-your-data-strategy">https://hbr.org/2017/05/whats-your-data-strategy</a></p><p>[2]: CSO Online. What is the cost of a data breach? <a href="https://www.csoonline.com/article/3434601/what-is-the-cost-of-a-data-breach.html">https://www.csoonline.com/article/3434601/what-is-the-cost-of-a-data-breach.html</a></p><p>[3]: Computer Weekly. Data breach costs on the rise, IBM study shows <a href="https://www.computerweekly.com/news/252467190/Data-breach-costs-on-the-rise-IBM-study-shows">https://www.computerweekly.com/news/252467190/Data-breach-costs-on-the-rise-IBM-study-shows</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31279f2d1d0c" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/data-scientists-are-dangerous-31279f2d1d0c">Data Scientists are Dangerous</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Streamlit inside JupyterHub]]></title>
            <link>https://medium.com/swlh/running-streamlit-inside-jupyterhub-17cbe8c8aaaf?source=rss-b6bb58cd796c------2</link>
            <guid isPermaLink="false">https://medium.com/p/17cbe8c8aaaf</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[jupyterhub]]></category>
            <dc:creator><![CDATA[Dan Lester]]></dc:creator>
            <pubDate>Sun, 10 May 2020 19:08:04 GMT</pubDate>
            <atom:updated>2020-05-11T15:15:40.919Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to run a Streamlit app in place of the regular Jupyter notebook server, so your JupyterHub provides an authentication layer around the Streamlit app</h4><p>Streamlit is a great framework for building beautiful data science web apps and prototypes, but the data scientist is largely left to their own devices when it comes to deploying the app to non-technical users.</p><p>Many organisations already have the capability to run a JupyterHub — which brings many options for authentication, such as single-sign-on through corporate email accounts. That makes JupyterHub a great base for hosting Streamlit apps. The flexibility of JupyterHub means that, with the right configuration, it doesn’t really mind what you run as the underlying ‘notebook’ server!</p><h3>Installing JupyterHub</h3><p>There are a few different ways to install JupyerHub for your organisation. The <a href="https://jupyterhub.readthedocs.io/en/stable/index.html">distributions are described here</a>. For a nice standardised single-server setup, you will do well with <a href="http://tljh.jupyter.org/en/latest/">The Littlest JupyterHub</a> (TLJH). This has a simple command line tool to easily set up HTTPS certificates and other configuration, but has some important <a href="http://tljh.jupyter.org/en/latest/topic/requirements.html">base requirements</a> on your server.</p><p>I’ll mostly assume you are going with TLJH, but steps for your own manual installation will be similar.</p><p>The initial installation and setup of TLJH is described for various cloud providers (or your own server) in <a href="http://tljh.jupyter.org/en/latest/install/index.html">TLJH’s Installation docs</a>.</p><h4>Streamlit and Helper packages</h4><p>We need to install Streamlit and also a small component called <a href="https://github.com/ideonate/jhsingle-native-proxy">jhsingle-native-proxy</a> into the Python environment that’s usually used to run the Jupyter notebook server.</p><p>SSH into your server and run:</p><pre>source /opt/tljh/user/bin/activate<br>sudo -E python3 -m pip install streamlit jhsingle-native-proxy</pre><p>You will also need to pip install any other dependencies that your Streamlit app may have. See TLJH docs for more details about <a href="http://tljh.jupyter.org/en/latest/howto/env/user-environment.html">the user environment</a>.</p><h3>Wrapper to manage Streamlit process</h3><p>We can’t use the streamlit command as a direct drop-in replacement for the usual Jupyter singleuser-notebook command. This is mainly because it expects different command line parameters, and also expects web traffic at / rather than e.g. /user/dan so needs to be told more about where it’s running.</p><p>The jhsingle-native-proxy component, installed in the last section, takes care of the authentication into JupyterHub, and proxies valid users’ web traffic further downstream to the Streamlit process which it manages internally.</p><p>So we need to write a Bash script to translate from the command-line inputs normally supplied for running the Jupyter Server into a command to run jhsingle-native-proxy.</p><p>We need to copy some text into a new script file, so you might need to install a text editor:</p><pre>sudo apt-get install vim</pre><p>And then paste the contents below into a new file called /usr/local/bin/streamlit-wrapper.</p><p>(For example, sudo vi /usr/local/bin/streamlit-wrapper to edit the file. To use vi: press ‘i’ when it opens; paste the text; press escape then ‘:wq’ and enter.)</p><pre>#!/bin/bash<br>export LC_ALL=C.UTF-8<br>export LANG=C.UTF-8</pre><pre>port=&quot;8888&quot;<br>delim=&#39;=&#39;</pre><pre>for var in &quot;$@&quot;<br>  do<br>    echo &quot;$var&quot;</pre><pre>    splitarg=${var%%$delim*}</pre><pre>    if [ &quot;$splitarg&quot; == &quot;--port&quot; ]; then<br>      # Yes, port was given in one arg e.g. --port=8888</pre><pre>      port=${var#*$delim}<br>      echo &quot;Setting external port $port&quot;<br>    fi</pre><pre>done</pre><pre>jhsingle-native-proxy --destport 0 streamlit hello {--}server.port {port} {--}server.headless True {--}server.enableCORS False --port $port</pre><p>The script above does two simple things:</p><ul><li>Iterate through all command line arguments supplied to the script. Extract the one that reads something like --port=1234 into the variable $port</li><li>Run jhsingle-native-proxy telling it to run <em>itself</em> on port $port. In turn, it runs Streamlit on a random free port (that’s what --destport 0 means) and passes <em>that</em> port number through to streamlit as --server.port {port}.</li></ul><p>For simplicity, we are just running streamlit hello rather than streamlit run &lt;myscript.py&gt; which is what you’d probably want.</p><p>The script needs to be executable by all users so:</p><pre>sudo chmod +x /usr/local/bin/streamlit-wrapper</pre><h4>Tell JupyterHub to run the Wrapper Script</h4><p>Finally, we need to configure TLJH to invoke our new wrapper script rather than the Jupyter notebook server when it starts a new server. That goes in the jupyterhub_config.py file, which for TLJH can be assembled from any py files in a special folder.</p><pre>sudo vi /opt/tljh/config/jupyterhub_config.d/streamlit.py</pre><p>And paste:</p><pre>c.Spawner.cmd = &#39;/usr/local/bin/streamlit-wrapper&#39;</pre><p>Restart your JupyterHub server to read this in. For TLJH:</p><pre>sudo tljh-config reload</pre><p>When you next restart your server within JupyterHub you’ll see the Streamlit demo app!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Shj88cTupqtA2jvnOI48wA.png" /><figcaption>Streamlit demo running in JupyterHub</figcaption></figure><h3>Authentication by JupyterHub</h3><p>As promised, jhsingle-native-proxy takes care of restricting access to the Streamlit server through internal OAuth provided by JupyterHub. If you try to access the user’s server as a logged-out user, e.g. in an incognito web browser window… first of all you’ll hit the main JupyterHub login page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZsT6wkpnY9XOK2acWhAF9w.png" /></figure><p>And then if you login as a user (other than the one who started the server in question), you’ll get an OAuth connection page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DICiuobiTBlhI-Fhy7DCjw.png" /></figure><p>If you Authorize access, you will actually end up with a <strong>403: Forbidden</strong> message. That’s because, by default, only the original user is allowed to access it.</p><p>Should you want any logged-in user of your JupyterHub to access the server (regardless of who started it) then update your /opt/tljh/config/jupyterhub_config.d/streamlit.py file so it reads:</p><pre>c.Spawner.cmd = &#39;/usr/local/bin/streamlit-wrapper&#39;</pre><pre>c.Spawner.environment = {&#39;JUPYTERHUB_ANYONE&#39;: &#39;1&#39;}</pre><p>The JUPYTERHUB_ANYONE environment variable tells jhsingle-native-proxy to extend authentication to any logged-in user of JupyterHub, forwarding all their web traffic to the Streamlit process.</p><h3>Conclusion</h3><p>To make this useful, you will probably want to run something other than the ‘streamlit hello’ demo of course…</p><p>You could ensure different users’ Streamlit python scripts are in their own home folder, perhaps always named app.py. The wrapper script can then run streamlit run ~/app.py instead of streamlit hello. The ability to run different scripts — maybe displaying a whole folder of scripts for the user to choose from — is a ‘<a href="https://github.com/streamlit/streamlit/issues/543">known feature request</a>’ for Streamlit…</p><p>Getting the scripts into the users’ folders in the first place is another problem. There are some basic ideas for <a href="http://tljh.jupyter.org/en/latest/howto/content/share-data.html">sharing user data in TLJH</a>, but really the configuration depends on your setup.</p><p>At ContainDS we think that JupyterHub is a natural base for hosting any data science web apps, whether built with Jupyter, Streamlit, Dash, Flask, or any other web framework. The <a href="https://containds.com/dashboards/">ContainDS Dashboards</a> platform is developing streamlined workflows to allow data scientists to automatically, reliably, and securely deploy their Streamlit web apps so they can be managed by JupyterHub.</p><p>Please <a href="https://containds.com/support/">get in touch</a> if you are interested in talking through the requirements for your organisation. We are always happy to advise, whether or not you are using any of our software.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17cbe8c8aaaf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/running-streamlit-inside-jupyterhub-17cbe8c8aaaf">Running Streamlit inside JupyterHub</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>