<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shah Newaz Khan on Medium]]></title>
        <description><![CDATA[Stories by Shah Newaz Khan on Medium]]></description>
        <link>https://medium.com/@shahnewazk?source=rss-375467dbf668------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0bfOaB9vDz73Uw9F.</url>
            <title>Stories by Shah Newaz Khan on Medium</title>
            <link>https://medium.com/@shahnewazk?source=rss-375467dbf668------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 15:45:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shahnewazk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Passing the baton, lessons learned from handing projects over to clients]]></title>
            <link>https://medium.com/commit-engineering/passing-the-baton-lessons-learned-from-handing-projects-over-to-clients-6b761a5ec73b?source=rss-375467dbf668------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b761a5ec73b</guid>
            <category><![CDATA[contractors]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[handover]]></category>
            <category><![CDATA[startup-lessons]]></category>
            <category><![CDATA[client-relationship]]></category>
            <dc:creator><![CDATA[Shah Newaz Khan]]></dc:creator>
            <pubDate>Tue, 30 Jun 2020 16:06:35 GMT</pubDate>
            <atom:updated>2020-06-30T16:11:06.891Z</atom:updated>
            <content:encoded><![CDATA[<h3>Passing the baton, lessons learned from handing projects back to teams</h3><p>Change is the norm now: new companies, new teammates, new roles every couple of years. To be effective at our software engineering jobs, we have to be comfortable learning new systems and processes quickly, but we also need to learn to leave positions gracefully. The last thing we want as software engineers is for our hard work to be for naught because the people picking up where we left off can’t decode our code. It’s up to us to work out loud: to document and share our logic and thinking to ensure a smooth transition when we’re gone.</p><p>This is especially true for <a href="https://commit.dev/">Commit</a> engineers. Commit’s model is to connect engineers with startups, offering opportunities for both groups to know each other and work together before either commits to a long-term opportunity. By its very nature, this creates many transitions in and out of organizations. As I’ve progressed in my career, working with many different teams and startups, I’ve given thought to how best to pass the baton to the team members I’m leaving behind, so there is no loss of knowledge when I go. Here are the ways I work to ensure a seamless transition when I’m gone.</p><p><strong>Documentation</strong></p><p>Believe it or not, it starts with your on-boarding. When I start at a new position, I like to “work out loud,” to document what I’m doing and why, right from the get-go. I set up folders in the organization’s tool of choice (for example, Confluence or Notion) and start writing <a href="https://en.wikipedia.org/wiki/Request_for_Comments">RFC documentation</a> for the high-level components I’m designing, whether it’s front end, back end, DevOps, etc. I also like to map out data flow diagrams to visualize the systems we’re planning and building as it makes it easier to build a mental map when talking about the systems with colleagues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*gb3O_w7V3-7JJkh7-j8Zxg.png" /><figcaption>Dataflow diagram included in RFC documentation and linked to the repository README.md</figcaption></figure><p>From here I link my design/POC tickets back to these documentations. The key is to document what I’m thinking and the logic behind my decisions, so that when I leave, someone can step in and continue the work with little or no time getting up to speed.</p><p>Roughly two weeks before the end of a fixed-term contract, I’ll review the documentation I have in place and formalize what’s there. But I strategically leave gaps in it. Then I meet with team members to discuss the hand-off process and have them review the documentation and fill in the gaps. This is a way for them to get their hands on the work, instead of just reading about it, and to gauge their understanding of the components. If they can fill in the gaps, great, they clearly understand the processes. If not, it’s an indication that we need to sit down and discuss it in more detail.</p><p><strong>Tooling</strong></p><p>Another way to help smooth the process of exiting from an organization is to use tooling to automate processes. It’s almost like you’re still getting work done, but you’re not there to do it.</p><p>When my time in a particular position is coming to an end, I review the internal tools I’ve helped configure and write shell scripts to automate as many processes as possible. Examples include packaging CLI tools such as <a href="https://docs.confluent.io/current/app-development/kafkacat-usage.html">kafkacat</a>, <a href="https://aws.amazon.com/cli/">AWS CLI</a> and <a href="https://octant.dev/">Octant</a> into a Docker container, so it’s easy for others to start using them without having to deal with installation for Windows, OSX or Linux environments. In addition, I make non-sparing use of <a href="https://www.gnu.org/software/make/manual/make.html">Makefiles</a> and shell scripts to encapsulate steps like introspecting a cluster, deploying a local dev environment, and setting up a database.</p><p><strong>Training</strong></p><p>The third approach to setting a company up for success when you’re on your way out is to train the people who will take over from you. This is especially relevant for complex processes and systems, where documentation and tooling provide a great start, but more direct knowledge-sharing is necessary.</p><p>On one of my recent contracts, we introduced <a href="https://kubernetes.io/">Kubernetes</a>, known to be a rather complex, heavyweight application. Soon after we got it running, my contract was coming to an end. We had set up documentation describing the processes and practices involved, and automated as many processes as possible through tooling, but we realized that direct training would be needed to ensure the organization would be up to speed on operating the system. We held two four-hour training sessions, introducing the entire 25-member engineering team to Kubernetes, including hands-on sections on how to perform end-to-end deployments and operate the system using the tooling we created to deploy applications and perform rollbacks and disaster recovery.</p><p><strong>Build bridges that last</strong></p><p>Of course, leaving a position or a company isn’t just about preserving the knowledge you’ve gained and the processes you’ve helped build. The relationships I’ve built are usually the most meaningful part of every job I’ve had: the people I’ve met, the communities I’ve been part of. Preserving those relationships is important to me, not as a career-development strategy (though that’s admittedly relevant), but simply because these personal connections are what makes the work worth doing.</p><p>There are ways to keep your friendships and professional relationships strong even if you stop working with people. You can join Slack channels and other communication networks, you can set up monthly Zoom calls or social events. One thing I like to do is write LinkedIn recommendations for my colleagues. It’s a perfect pairing of professional promotion and personal positivity that proves productive for all parties.</p><p>And it’s just a nice thing to do.</p><p>Do you have tips for leaving a role in great shape when you leave? Let me know in the comments below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b761a5ec73b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/commit-engineering/passing-the-baton-lessons-learned-from-handing-projects-over-to-clients-6b761a5ec73b">Passing the baton, lessons learned from handing projects over to clients</a> was originally published in <a href="https://medium.com/commit-engineering">Commit Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dockerizing Airflow]]></title>
            <link>https://medium.com/data-science/dockerizing-airflow-58a8888bd72d?source=rss-375467dbf668------2</link>
            <guid isPermaLink="false">https://medium.com/p/58a8888bd72d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Shah Newaz Khan]]></dc:creator>
            <pubDate>Wed, 25 Jul 2018 06:23:53 GMT</pubDate>
            <atom:updated>2020-07-08T00:50:30.279Z</atom:updated>
            <content:encoded><![CDATA[<h4>Apache Airflow on Docker for local workloads</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_gjHN5eycqXc_RMz" /><figcaption>Photo by <a href="https://unsplash.com/@koushikc?utm_source=medium&amp;utm_medium=referral">Koushik Chowdavarapu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://airflow.apache.org/">Airflow</a> is the de facto ETL orchestration tool in most data engineers tool box. It provides an intuitive web interface for a powerful backend to schedule and manage dependencies for your ETL workflows.</p><p>In my day to day work-flow, I use it to maintain and curate a data lake built on top of <a href="https://aws.amazon.com/s3/">AWS S3</a>. Nodes in my Airflow DAGs include multi-node <a href="https://aws.amazon.com/emr/">EMR Apache Spark</a> and <a href="https://aws.amazon.com/fargate/">Fargate</a> clusters that aggregate, prune and produce para-data from the data lake.</p><p>Since these work-flows are executed on distributed clusters (20+ nodes) and have heavy dependencies (output from one ETL is fed in as input to the next) it made sense to orchestrate them using Airflow. However it did not make sense to have a central Airflow deployment as I will be the only one using it.</p><p>I therefore chose to <a href="https://www.docker.com/">Dockerize</a> Airflow so that I could spin up a container and easily run these <a href="https://airflow.apache.org/concepts.html#workflows">work-flows</a> without having to worry about the Airflow deployment.</p><p>In this post I will go over how I achieved this along with some brief explanation of design decisions along the way.</p><h3>Airflow Components</h3><p>In Airflow ETL <a href="https://airflow.apache.org/concepts.html#workflows">work-flows</a> are defined as <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">directed acyclic graphs</a> (<a href="https://airflow.apache.org/concepts.html#dags">Airflow DAG</a>) where each node is a self-contained ETL with each downstream node being dependent on successful completion of the upstream node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*C0LeRf-Wolb9FV7l5cmASQ.png" /><figcaption>Simple Airflow DAG</figcaption></figure><p>Airflow has three deployment components:</p><ul><li>Webserver ( <a href="http://flask.pocoo.org/">Flask</a> backend used to trigger and monitor DAGs)</li><li>Scheduler ( A daemon process to schedule and run the DAG executers )</li><li>Database ( A presistance layer for DAG &amp; DAG instance definitions )</li></ul><h3>Airflow Quickstart</h3><p>It is quick and easy to get <a href="https://airflow.apache.org/start.html">started</a> with airflow:</p><pre># airflow needs a home, ~/airflow is the default,<br># but you can lay foundation somewhere else if you prefer<br># (optional)<br>export AIRFLOW_HOME=~/airflow</pre><pre># install from pypi using pip<br>pip install apache-airflow</pre><pre># initialize the database<br>airflow initdb</pre><pre># start the web server, default port is 8080<br>airflow webserver -p 8080</pre><p>Upon running these commands, Airflow will create the $AIRFLOW_HOME folder and lay an airflow.cfgfile with defaults that get you going fast. You can inspect the file either in $AIRFLOW_HOME/airflow.cfg, or through the UI in the Admin-&gt;Configuration menu. The PID file for the webserver will be stored in $AIRFLOW_HOME/airflow-webserver.pid or in /run/airflow/webserver.pid if started by systemd.</p><p>Out of the box, Airflow uses a sqlite database, which you should outgrow fairly quickly since no parallelization is possible using this database backend. It works in conjunction with the SequentialExecutor which will only run task instances sequentially. While this is very limiting, it allows you to get up and running quickly and take a tour of the UI and the command line utilities.</p><p>Here are a few commands that will trigger a few task instances. You should be able to see the status of the jobs change in the example1 DAG as you run the commands below.</p><pre># run your first task instance<br>airflow run example_bash_operator runme_0 2018-01-01<br># run a backfill over 2 days<br>airflow backfill example_bash_operator -s 2018-01-01 -e 2018-01-02</pre><h3>Dockerizing Airflow</h3><p>A container’s main running process is the ENTRYPOINT and/or CMD at the end of the Dockerfile. It is generally recommended that you separate areas of concern by using one service per container.</p><p>However since we want to have the Airflow Webserver &amp; the Airflow Scheduler processes both running, we will usesupervisord as a process manager.</p><p>This is a moderately heavy-weight approach that requires you to package supervisord and its configuration in the docker image (or base your image on one that includes supervisord), along with the different applications it manages.</p><p>Then you start supervisord, which manages your processes for you. First we will need to define supervisord.conf :</p><pre>[supervisord]                       <br>nodaemon=true</pre><pre>[program:scheduler]                       <br>command=airflow scheduler                       stdout_logfile=/var/log/supervisor/%(program_name)s.log                       stderr_logfile=/var/log/supervisor/%(program_name)s.log                       autorestart=true</pre><pre>[program:server]                       <br>command=airflow webserver -p 8080                       stdout_logfile=/var/log/supervisor/%(program_name)s.log                       stderr_logfile=/var/log/supervisor/%(program_name)s.log                       autorestart=true</pre><p>Then we will use supervisord as the ENTRYPOINT to our Dockerfile:</p><pre>FROM python:3.6.3</pre><pre># supervisord setup                       <br>RUN apt-get update &amp;&amp; apt-get install -y supervisor                       COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf</pre><pre># Airflow setup                       <br>ENV AIRFLOW_HOME=/app/airflow</pre><pre>RUN pip install apache-airflow                       <br>COPY /dags/response_rate_etl.py $AIRFLOW_HOME/dags/</pre><pre>RUN airflow initdb</pre><pre>EXPOSE 8080</pre><pre>CMD [&quot;/usr/bin/supervisord&quot;]</pre><p>Build the Docker image:</p><pre>docker build . -t airflow</pre><p>Run the container:</p><pre>docker run -d -p 8080:8080 --rm \<br>   --name airflow_container \<br>   airflow</pre><p>Launch a DAG:</p><pre>docker exec airflow_container airflow trigger_dag example_bash_operator</pre><p>Monitor DAG run:</p><p>Open a browser and navigate to http://localhost:8080 to monitor the DAG instance run.</p><p>That’s all folks, stay tuned to future posts where I will go into defining AWS EMR dags, defining custom Airflow Operators, injecting AWS Credentials and more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58a8888bd72d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dockerizing-airflow-58a8888bd72d">Dockerizing Airflow</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Coloring Plotly Box-Plots]]></title>
            <link>https://medium.com/data-science/applying-a-custom-colormap-with-plotly-boxplots-5d3acf59e193?source=rss-375467dbf668------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d3acf59e193</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[dash]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[matplotlib]]></category>
            <dc:creator><![CDATA[Shah Newaz Khan]]></dc:creator>
            <pubDate>Mon, 23 Jul 2018 07:31:01 GMT</pubDate>
            <atom:updated>2020-07-08T22:08:38.840Z</atom:updated>
            <content:encoded><![CDATA[<h4>Applying a custom color scale to Plotly Boxplots</h4><p>I was recently asked by my boss to apply a color scale to a series of <a href="https://plot.ly/">Plotly</a> <a href="https://plot.ly/python/box-plots/">Box plots</a> I was working on for a <a href="https://plot.ly/products/dash/">Dash</a> application. The notion was to plot multiple box plots on the same graph and have each box plot be darker or lighter based on its distance from the overall median in the dataset.</p><h3>The problem</h3><p>The issue here is that Plotly only supports <a href="https://plot.ly/python/colorscales/">color scales</a> for heatmaps, scatter plots &amp; contour plots.</p><p>I will demonstrate the default behavior by plotting some generated data for Canucks players +/- per game all time as follows:</p><blockquote>Note: Complete code at the end of the post</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*3TnVapNTl9A9x-dKu6gtQQ.png" /></figure><p>The dashed green line represents the NHL league average +/- per game at +2.0, what my boss wanted was to color each box plot darker or lighter based on how far away each players median +/- value was from the league average .</p><h3>The solution</h3><p>Given that I was already in a Python environment, I chose to use the colormap &amp; colormap normalizer features from the <a href="https://matplotlib.org/">Matplotlib</a> Python module.</p><p>First step was to import Matplotlib:</p><pre>import matplotlib.pyplot as plt<br>import matplotlib<br>import pandas as pd</pre><p>Then determine the minand maxvalues in the dataset:</p><pre>x_data = [&#39;Henrik Sedin&#39;, &#39;Daniel Sedin&#39;,<br>          &#39;Brock Boeser&#39;, &#39;Elias Pettersen&#39;,<br>          &#39;Bo Horvat&#39;, &#39;Pavel Bure&#39;,]</pre><pre># Generate random data<br>y0 = np.random.randn(50)+4<br>y1 = np.random.randn(50)+4<br>y2 = np.random.randn(50)+2<br>y3 = np.random.randn(50)+2<br>y4 = np.random.randn(50)+1<br>y5 = np.random.randn(50)+3</pre><pre>y_data = [y0,y1,y2,y3,y4,y5]</pre><pre>df = pd.DataFrame(y_data)<br>vmin, vmax = df.min().min(), df.max().max()</pre><p>Initialize the Matplotlib cmap &amp; normalizer:</p><pre>norm = matplotlib.colors.Normalize(vmin=vmin, vmax=vmax)<br>cmap = matplotlib.cm.get_cmap(&#39;GnBu&#39;) # green to blue color way</pre><p>Find the median of each y_data and use the normalizer to map it to an appropriate color based on the median value:</p><pre>for xd, yd in zip(x_data, y_data):<br>        <br>        median = np.median(yd) # find the median<br>        color = &#39;rgb&#39; + str(cmap(norm(median))[0:3]) # normalize<br>        <br>        traces.append(go.Box(<br>            y=yd,<br>            name=xd,<br>            boxpoints=&#39;all&#39;,<br>            jitter=0.5,<br>            whiskerwidth=0.2,<br>            fillcolor=color,      # add the box plot color<br>            marker=dict(<br>                size=2,<br>            ),<br>            line=dict(width=1)<br>        ))</pre><h3>VIOLA!</h3><p>The resulting plot will render a darker fillcolorthe further positive the players median +/- is from the league average and vice versa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*QpX-B-2vZhRcPkari11IHA.png" /></figure><h3>BONUS</h3><p>Here is the complete code to get the result above:</p><pre>from plotly import __version__<br>from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot<br>init_notebook_mode(connected=True)<br>import plotly.graph_objs as go<br>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import matplotlib</pre><pre>x_data = [&#39;Henrik Sedin&#39;, &#39;Daniel Sedin&#39;,<br>          &#39;Brock Boeser&#39;, &#39;Elias Pettersen&#39;,<br>          &#39;Bo Horvat&#39;, &#39;Pavel Bure&#39;,]</pre><pre>y0 = np.random.randn(50)+4<br>y1 = np.random.randn(50)+4<br>y2 = np.random.randn(50)+2<br>y3 = np.random.randn(50)+2<br>y4 = np.random.randn(50)+1<br>y5 = np.random.randn(50)+3</pre><pre>y_data = [y0,y1,y2,y3,y4,y5]</pre><pre>traces = []</pre><pre>df = pd.DataFrame(y_data)<br>vmin, vmax = df.min().min(), df.max().max()</pre><pre>norm = matplotlib.colors.Normalize(vmin=vmin, vmax=vmax)<br>cmap = matplotlib.cm.get_cmap(&#39;GnBu&#39;)</pre><pre>for xd, yd in zip(x_data, y_data):<br>        median = np.median(yd)<br>        color = &#39;rgb&#39; + str(cmap(norm(median))[0:3])<br>        <br>        traces.append(go.Box(<br>            y=yd,<br>            name=xd,<br>            boxpoints=&#39;all&#39;,<br>            jitter=0.5,<br>            whiskerwidth=0.2,<br>            fillcolor=color,<br>            marker=dict(<br>                size=2,<br>                color=&#39;rgb(0, 0, 0)&#39;<br>            ),<br>            line=dict(width=1),<br>        ))</pre><pre>layout = go.Layout(<br>    title=&#39;Canucks all time +/- per game&#39;,<br>    yaxis=dict(<br>        autorange=True,<br>        showgrid=True,<br>        zeroline=True,<br>        dtick=5,<br>        gridcolor=&#39;rgb(255, 255, 255)&#39;,<br>        gridwidth=1,<br>        zerolinecolor=&#39;rgb(255, 255, 255)&#39;,<br>        zerolinewidth=2,<br>    ),<br>    margin=dict(<br>        l=40,<br>        r=30,<br>        b=80,<br>        t=100,<br>    ),<br>    paper_bgcolor=&#39;rgb(243, 243, 243)&#39;,<br>    plot_bgcolor=&#39;rgb(243, 243, 243)&#39;,<br>    showlegend=False,<br>    shapes= [{<br>      &#39;type&#39;: &#39;line&#39;,<br>      &#39;x0&#39;: -1,<br>      &#39;y0&#39;: 2,<br>      &#39;x1&#39;: 6,<br>      &#39;y1&#39;: 2,<br>      &#39;line&#39;: {<br>        &#39;color&#39;: &#39;rgb(50, 171, 96)&#39;,<br>        &#39;width&#39;: 4,<br>        &#39;dash&#39;: &#39;dashdot&#39;<br>      }<br>    }]<br>)</pre><pre>fig = go.Figure(data=traces, layout=layout)<br>iplot(fig)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d3acf59e193" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/applying-a-custom-colormap-with-plotly-boxplots-5d3acf59e193">Coloring Plotly Box-Plots</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connecting to a Jupyter Notebook on a remote Linux machine with an SSH tunnel]]></title>
            <link>https://medium.com/data-science/connecting-to-a-jupyter-notebook-on-a-remote-linux-machine-277cef04abb7?source=rss-375467dbf668------2</link>
            <guid isPermaLink="false">https://medium.com/p/277cef04abb7</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[remote-working]]></category>
            <dc:creator><![CDATA[Shah Newaz Khan]]></dc:creator>
            <pubDate>Sun, 15 Jul 2018 04:55:39 GMT</pubDate>
            <atom:updated>2020-08-02T02:49:06.805Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7VGKTEdVXm_c87i4" /><figcaption>Photo by <a href="https://unsplash.com/@florianolv?utm_source=medium&amp;utm_medium=referral">Florian Olivo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>My desktop at work is a powerful machine I use for exploratory data analysis and other machine learning work-flows.</h4><p>In this post I will go over how I connect to my work machine and run <a href="http://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html">Jupyter Notebook</a> workloads when I am working remotely.</p><h3>SSH into remote machine</h3><p>Step 1 is to ssh into your remote machine and launch Jupyter Notebook to a local port with the --no-browser option.</p><pre>user@local_machine$ ssh user@remote_machine <br>user@remote_machine$ jupyter notebook --no-browser --port=8889</pre><h3>Setting up an SSH tunnel</h3><p>Step 2 is to set up an <a href="https://www.ssh.com/ssh/tunneling/">SSH tunnel</a> from your local machine to port 8889 on the remote machine where the Jupyter Notebook is being served.</p><pre>user@local_machine$ ssh -N -L localhost:8888:localhost:8889 user@remote_mahcine</pre><p>Here is a break down of the ssh options</p><ul><li>-N Do not execute a remote command. This is useful for just forwarding ports</li><li>-L local_socket:remote_socket<br>Specifies that connections to the given TCP port or Unix socket on the local (client) host are to be forwarded to the given host and port, or Unix socket, on the remote side. This works by allocating a socket to listen to either a TCP port on the local side, optionally bound to the specified bind_address, or to a Unix socket. Whenever a connection is made to the local port or socket, the connection is forwarded over the secure channel, and a connection is made to either host port hostport, or the Unix socket remote_socket, from the remote machine.</li><li>Port forwardings can also be specified in the configuration file. Only the superuser can forward privileged ports. IPv6 addresses can be specified by enclosing the address in square brackets.</li><li>By default, the local port is bound in accordance with the GatewayPorts setting. However, an explicit bind_address may be used to bind the connection to a specific address. The bind_address of “localhost” indicates that the listening port be bound for local use only, while an empty address or ‘*’ indicates that the port should be available from all interfaces.</li></ul><p><strong>Note</strong> that running the SSH tunnel with -N will not log any outputs, as long as you do not get an error this means you have established a tunnel.</p><h3>Load Jupyter Notebook on a local browser</h3><p>On the local machine browser load localhost:8888 and the Jupyter Notebook from your remote machine will load out as expected.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=277cef04abb7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/connecting-to-a-jupyter-notebook-on-a-remote-linux-machine-277cef04abb7">Connecting to a Jupyter Notebook on a remote Linux machine with an SSH tunnel</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>