<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Ellis on Medium]]></title>
        <description><![CDATA[Stories by Daniel Ellis on Medium]]></description>
        <link>https://medium.com/@dan.ellis?source=rss-4cb5669be321------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qwiSpjpuFiJLA7m-kkfrow.jpeg</url>
            <title>Stories by Daniel Ellis on Medium</title>
            <link>https://medium.com/@dan.ellis?source=rss-4cb5669be321------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:45:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dan.ellis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Basic Emotions I Feel]]></title>
            <link>https://medium.com/@dan.ellis/basic-emotions-i-feel-ae38e37bf6e?source=rss-4cb5669be321------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae38e37bf6e</guid>
            <category><![CDATA[imposter-syndrome]]></category>
            <category><![CDATA[life-lessons]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Daniel Ellis]]></dc:creator>
            <pubDate>Wed, 16 Nov 2016 14:29:13 GMT</pubDate>
            <atom:updated>2016-11-16T23:55:46.327Z</atom:updated>
            <content:encoded><![CDATA[<p>The other day, I stumbled upon <a href="http://www.corporateprogrammer.com/blog/the-imposter-within/">yet another article about imposter syndrome</a> as it relates to programming. It ended up making its way to /r/programmingcirclejerk, where I took the opportunity to make <a href="https://www.reddit.com/r/programmingcirclejerk/comments/5ct7sx/imposter_syndrome_part_%E2%84%95_or_how_i_stopped/d9z89a8/?context=3">this smarmy comment</a>:</p><blockquote>Sometimes, I think I’m really good at X. Other times I think I’m just OK at X. Then on some days, I’m not very productive, and I think I’m bad at X.</blockquote><blockquote>What the hell is wrong with me? Does anyone know of a way to extinguish these basic human emotions? Or at least could someone tell me what javascript framework I need to learn?</blockquote><p>I got a bit of karma out of it, which of course justifies my actions, but I also wanted to develop my thoughts in a less sarcastic way and make a serious point: <strong>everyone feels this way, and it isn’t going away any time soon</strong>. Furthermore, the ways we choose to cope with this feeling tend to further our sense of isolation and insecurity.</p><p>Let’s take an example.</p><h3>Describing A Technical Thing</h3><p>Think about every verbal description you’ve ever heard of a technical problem you aren’t familiar with. At some point, inevitably, the describer of the problem loses you. Now you <em>might</em> guess that their over-familiarity with the problem is hindering their ability to communicate with you effectively. You <em>might </em>let that realization embolden into frequently asking followup questions. But you don’t. You don’t say anything at all. Instead, you start <strong>the nod</strong>.</p><p>You know <strong>the nod</strong>, because you’ve seen it in others. Your face becomes a little softer — maybe a little more flush. It’s a bit deer-in-the-headlight-ish. You don’t <em>really</em> understand what they just said, but you don’t want to interrupt, and you certainly don’t want to look stupid. Maybe they used a term so casually that you’re now assuming you’re supposed to know it. But you don’t know it, so you let it slide.</p><p>Only now they’re using that tidbit as a jumping off point for <em>another</em> piece of information. By the end of their spiel, you’re at 50% comprehension at best. But rather than accept this as normal given the aforementioned factors (your hesitance to speak up and their communication issues), you assume the problem is your general inadequacy, and you internalize it.</p><h3>News Flash: The Depth is Infinite</h3><p>You are <strong>always</strong> going to be coming across some new thing you’ve never heard of. The variations on the technical acronyms alone you’ll run into over the course of your career will be endless. Here’s a few:</p><blockquote>ACL, AMP, AMQP, APT, CAP, CSRF, DOM, DKIM, DNS, ETL, FAT, FQDN, HA,ICMP, IDL, IP, IOPS, LAMP, KVM, LVM, MX, NUMA, NTFS, OOM, ORM, PCB, PCI, PID, PPA, PSU, RDP, RHEL, RPM, RSS, SATA, SCSI, SMTP, SOA, SPF, SSRF, SSH, SSI, SVG, TCP, TTL, UAC, URI, URL, UDP, UTF, VCL, VM, VPN, WYSIWYG, XML, XP, XSS, YAML.</blockquote><p>Each one of these invokes some kind of image in my head. I’d guess if you scrolled through <a href="https://en.wikipedia.org/wiki/List_of_computing_and_IT_abbreviations">this wikipedia page</a> you could generate a similarly lengthy list of things you’re familiar with. And yet still, after amassing all this knowledge, you and I both walk around with a nagging sense of insecurity.</p><h3>But What Then Should We Do?</h3><p>Back to that original article. The author gives some couple dozen “techniques” to help mitigate the problem, all of which are quite reasonable. But still, they’re numerous — and holding numerous things in the head is hard.</p><p>They also happen to be focused on making <em>you</em> feel better. That’s fine, but there’s a lot more of everyone else than there is of you. So it seems to me more valuable for the world to flip it around and focus on how to help out others. I won’t tell you what <em>you</em> should do, but I’ll tell you what I’m going to do.</p><p>I’m going to encourage “stupid” questions. I’m going to frequently check the understanding of the people I’m explaining things to, keeping in mind there’s no way they’ll get it all on the first pass. Because I know what it’s like to be on the receiving end of this stuff.</p><p>I know that despite my best attempts, the next time I hear someone casually and confidently throw out a 3 letter acronym I haven’t heard before, I’ll <em>still</em> shrink a bit inside — I’ll still feel insecure. Of course, I’ll try not to show it, because I’m human.<strong> </strong>After all, hiding our humanity is one of the most human traits of all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae38e37bf6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[forkstat’s Pretty Cool]]></title>
            <link>https://medium.com/@dan.ellis/forkstats-pretty-cool-2d4fcffda431?source=rss-4cb5669be321------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d4fcffda431</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Daniel Ellis]]></dc:creator>
            <pubDate>Thu, 28 Jul 2016 03:28:21 GMT</pubDate>
            <atom:updated>2016-07-28T16:19:18.061Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the last few days, I’ve been working automating my team’s server building process and speeding up launch and provision times. Just when I thought I was nearly done with the process, I ran into a couple of gnarly issues.</p><p>First, I started getting emails every time one of the new servers would boot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Qvm0bRrLCJopJT87FMqFQ.png" /></figure><p>Even worse, it was going to my coworkers, who didn’t hesitate to let me know:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NjY4IfpDoedWJL5DUj4gYg.png" /><figcaption>Identities hidden to protect the guilty.</figcaption></figure><p>This was at the pre-provisioning stage, so the email made sense, but I had no clue what was running sudo at this stage. On top of this, I discovered puppet was being run on startup, even after <a href="https://docs.puppet.com/puppet/latest/reference/man/agent.html#OPTIONS">disabling puppet directly</a> as well as disabling it with upstart. I needed some other things to happen in the provisioning process before the puppet run could take place, so this was no good.</p><p>I double checked our <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html">user data</a> script (one way of passing initialization commands to <a href="https://cloudinit.readthedocs.io/en/latest/">cloud init</a> on boot) — nope nothing running sudo there. These shenanigans had to be running somewhere else.</p><p>I wondered what syslog might have to say:</p><pre>sSMTP[1400]: Sent mail for <a href="/service/mailto:root@reddit.com">foo@bar.com</a> (221 2.0.0 Bye) uid=0 username=root outbytes=440</pre><p>Super useful.</p><p>How was I going to figure this out? I sat and stared at my monitor for a bit, but that didn’t work. After googling around, I came across <a href="http://manpages.ubuntu.com/manpages/wily/man8/forkstat.8.html">forkstat</a>. forkstat uses <a href="http://man7.org/linux/man-pages/man7/netlink.7.html">netlink connector</a>, a special socket API provided by the kernel to get information on processes launching and exiting, among other things (a cool minimal python example that uses this directly is <a href="https://gist.github.com/drdaeman/1488608">here</a>). Because it doesn’t use anything fancy like ptrace, it’s overhead is low enough that I decided to give it a shot.</p><p>I created the following <em>/etc/init/forkstat.conf</em> file, baked it into the server image, and launched a new server:</p><pre>start on startup<br>task<br>exec forkstat</pre><p>After startup, I checked the newly created forkstat logs and found a couple of juicy tidbits:</p><pre>11:17:13 fork  1447 child           /bin/sh /etc/init.d/rc 2<br>11:17:13 exec  1447                 /bin/sh /etc/rc2.d/S21puppet start<br>11:17:13 fork  1468 child           /bin/sh /etc/rc2.d/S21puppet start<br>11:17:13 exec  1468                 start-stop-daemon --start --quiet --pidfile /var/run/puppet/agent.pid --startas /usr/bin/puppet -- agent</pre><p>Ah! So there was an rc.d script launching puppet. One google search later and <em>update-rc.d -f puppet remove</em> did the trick.</p><p>Then, I found this:</p><pre>11:17:13 fork  1419 child           ruby /usr/bin/foo -c /etc/foo.conf<br>11:17:13 exec  1419                 sudo /usr/local/sbin/foo-script<br>11:17:13 exec  1419                 sudo /usr/local/sbin/foo-script<br>11:17:13 fork  1419 parent          sudo /usr/local/sbin/foo-script<br>11:17:13 fork  1424 child           sudo /usr/local/sbin/foo-script<br>11:17:13 fork  1424 parent          sudo /usr/local/sbin/foo-script<br>11:17:13 fork  1425 child           sudo /usr/local/sbin/foo-script<br>11:17:13 exit  1424      0    0.000 sudo /usr/local/sbin/foo-script<br>11:17:13 fork  1425 parent          sudo /usr/local/sbin/foo-script<br>11:17:13 fork  1426 child           sudo /usr/local/sbin/foo-script<br>11:17:13 exec  1426                 sendmail -t</pre><p>Ah, our <em>foo</em> script was launching the sudo subprocess, and ultimately calling sendmail. I realized I didn’t actually want that script to be running on boot, since the machine wasn’t provisioned yet, so I disabled it launching on startup.</p><p>So yeah, forkstat is a pretty cool, simple investigative tool. I have a feeling I’ll be using it again in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d4fcffda431" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving a Tricky Problem]]></title>
            <link>https://medium.com/@dan.ellis/solving-a-tricky-problem-d1a8766f5347?source=rss-4cb5669be321------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1a8766f5347</guid>
            <dc:creator><![CDATA[Daniel Ellis]]></dc:creator>
            <pubDate>Fri, 24 Apr 2015 16:14:57 GMT</pubDate>
            <atom:updated>2015-04-24T16:14:57.227Z</atom:updated>
            <content:encoded><![CDATA[<h3>What to do when Google fails you</h3><p>Every once in a while when you go to Google an issue you’re having, you don’t find any good results. After closing the tab, re-opening it, and searching for the same thing 2 or 3 more times before realizing what you’re doing, you might be tempted to give up.</p><p>Should you quit? Hell no! Unless you don’t feel like it. Then you should quit. But this time I didn’t. So here’s the story of a time when I didn’t quit. It even has a happy ending.</p><h3>The Problem</h3><p>A couple of days ago, I ran into a problem like this. I was trying to debug an SSL connection issue that only showed up sporadically. I used tcpdump combined with a looping bash script to try to find the issue and get a network capture of it.</p><pre>for i in $(seq 60); do<br>    gtimeout 45 ./run.sh &gt; /tmp/capture-$i.log &amp;<br>    gtimeout 45 tcpdump -i any -w /tmp/capture-$i.pcap ‘src 1.1.1.1 || dst 1.1.1.1’ &amp;<br>     wait<br>done</pre><p>Basically, I was just running the script I was having trouble with along with tcpdump for 45 seconds, waiting for both to end, and logging to a file that would show the error if it happened, as well as a pcap.</p><p>After I finally managed to reproduce the problem and get a capture, I was disappointed to see this in wireshark:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5emI7bjqDq3L3PDZQb881A.png" /><figcaption>The capture file appears to be damaged or corrupt. (pcapng: interface index 1 is not less than interface count 1)</figcaption></figure><p>Awesome. Well maybe I could take a look at the data in tcpdump at least:</p><pre>reading from PCAP-NG file capture-41.pcap<br>tcpdump: pcap_loop: a packet arrived on interface 1, but there’s no Interface Description Block for that interface</pre><p>Unacceptable. This was a rare capture; I needed to figure out a way to take a look at it. Hell, even if I managed to capture it again, there was no guarantee it wouldn’t just be corrupt again.</p><h3>The Tricky Road to a Solution</h3><p>The first thing I noticed was tcpdump thought this file was a “pcap-ng” file, rather than a pcap file. I looked up <a href="https://ask.wireshark.org/questions/1891/pcap-vs-pcap-ng">the difference between the two</a> and found that pcap-ng files contain extra meta information about things like the interface! Perhaps if I could convert to the less-information pcap format, it would simply strip this extra information out and then be readable.</p><p>According to T<a href="https://wiki.wireshark.org/Development/PcapNg#File_format_conversion">he Wireshark Wiki</a>, there are a couple of tools that can be used to do this conversion. So I tried the first one:</p><pre>dan@Dans-MacBook-Pro:/tmp$ editcap -F libpcap -T ether capture-41.pcap out</pre><pre>(process:4447): GLib-CRITICAL **: g_hash_table_lookup: assertion `hash_table != NULL’ failed<br>editcap: An error occurred while reading “capture-41.pcap”: The file appears to be damaged or corrupt.<br>(pcapng: interface index 1 is not less than interface count 1)</pre><p>What a bunch of gibberish! What a bunch of crap! What a bunch of — okay, I needed to stay calm. Maybe the second tool would help:</p><pre>dan@Dans-MacBook-Pro:/tmp$ tcpdump -r capture.pcap -w out<br>reading from PCAP-NG file sc-41.pcap<br>tcpdump: pcap_loop: a packet arrived on interface 1, but there’s no Interface Description Block for that interface</pre><p>Well it was still tcpdump, so I wasn’t sure what I expected.</p><p>My next step was to see if there were any libraries for parsing pcapng files I could leverage. Since I had a vague notion of what the problem might be (I had a feeling it was related to the filtering I added to tcpdump), maybe I could use an abstraction over the file format to tinker with it and fix the issue.</p><p>I ended up finding <a href="https://github.com/rshk/python-pcapng">python-pcapng</a>. I started by just loading up the file and looking around:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b16abf5e7d38e060b2172047721d031d/href">https://medium.com/media/b16abf5e7d38e060b2172047721d031d/href</a></iframe><p>Looking at the first block (presumably the header), I found this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ea96e39efa11504e76f971d8c6ecc64/href">https://medium.com/media/7ea96e39efa11504e76f971d8c6ecc64/href</a></iframe><p>Okay, one interface with an index of 0. That made sense based on the error. Next, I found the first packet:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc0d70cd8fedf796100073d7f4f4088c/href">https://medium.com/media/cc0d70cd8fedf796100073d7f4f4088c/href</a></iframe><p>Ah! It was referencing the interface as 1, rather than 0. Trying to access the interface information for the packet confirmed my suspicion:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22ed0b9b161e430deb92eaacd92fe150/href">https://medium.com/media/22ed0b9b161e430deb92eaacd92fe150/href</a></iframe><p>If I could just change this on all of the packets and re-write the file, perhaps that would fix the problem. Unfortunately, I went back to the python-pcapng page and saw <a href="https://github.com/rshk/python-pcapng#pcap-ng-write-support">something unfortunate</a>:</p><blockquote>Support for writing pcap-ng files is “planned”; [..] I didn’t add that part (yet) as I currently don’t need it, and I’m wondering whether anybody might.</blockquote><p>Damn. Still, it wasn’t the time to give up. Thankfully, earlier in the readme for the project the <a href="http://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html">pcapng file format spec</a> is referenced. Maybe I would be able to find what I needed there.</p><p>Indeed, after looking around a bit, I stumbled upon <a href="http://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html#sectionepb">a nice diagram for the so-called “Enhanced Packets”</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pp_MMN4gmlveLJYp5KLWcA.png" /><figcaption><strong>Enhanced Packet Block format.</strong></figcaption></figure><p>I must confess, I didn’t anticipate the day when a block ascii diagram would get me really excited, but it doesn’t get a lot more self-explanatory than this. My next step was to open up the capture in a hex editor and see if I could by hand find the packet that python-pcapng had showed me.</p><p>After searching for the payload shown in the above example, I was able to find the start of the packet (referenced in the block diagram as being 4 bytes, starting with 0x06):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*6jx2jbj1gajh_Afj4FY_Cw.png" /></figure><p>Looking at the second block from the start of the beginning of the packet (4 bytes later), I saw something that both lined up with “Interface ID” in the block diagram and contained the data:</p><pre>01 00 00 00</pre><p>Bingo!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*kns1--ehT8f6_USnwgyFTg.gif" /></figure><p>The next step was to replace that 01 with a 00 throughout the file. The only problem was that block in the middle. According to the block diagram, it specifies “<em>Block Total Length</em>,” which sounds pretty important. Unfortunately, my hex editor didn’t have any support for capturing groups or wildcards for search and replace, so it was back over to Python.</p><p>I popped open a shell and whipped up a little big of regex to patch the file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3f53bf7b35e3b62e91534a42cdcccb6/href">https://medium.com/media/a3f53bf7b35e3b62e91534a42cdcccb6/href</a></iframe><p>Now for the moment of truth. Would it work?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wTQUrMBwTja4I7A12UFt_A.png" /></figure><p>Success! That felt <em>good</em>.</p><h3>What Next?</h3><p>Maybe now I’ll try to reproduce the problem again with tcpdump and file an issue. Maybe I’ll try to fix it myself. Maybe others running into this issue will now be able to find it on Google. Who knows? Like they always say:</p><blockquote>Once you’ve hand-patched a corrupted pcapng file using a mixture of regexes, python, and a hex editor, the world is your oyster.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1a8766f5347" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Don’t Need $1MM for a Distributed System]]></title>
            <link>https://medium.com/@dan.ellis/you-dont-need-1mm-for-a-distributed-system-70901d4741e1?source=rss-4cb5669be321------2</link>
            <guid isPermaLink="false">https://medium.com/p/70901d4741e1</guid>
            <dc:creator><![CDATA[Daniel Ellis]]></dc:creator>
            <pubDate>Tue, 17 Feb 2015 16:28:09 GMT</pubDate>
            <atom:updated>2015-02-17T16:28:09.715Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up containers, load balancing, and service discovery on light hardware</h3><p>The biggest problem in software is a human one.</p><p>If programming were just a matter issuing commands once a day down from on high, we could write our fancy new app in machine language, order a pizza, and go back home to binge out on <em>Netflix</em>. Unfortunately, in the real world, someone is going to have to come back to maintain that mess of code, and they’re going to be pissed when they find out you’re watching <em>House of Cards</em> for the third time.</p><p>The real problem we deal with, then, isn’t whether or not the machine is <em>capable</em> of doing what we want—it is. The real problem is that <strong>we have terrible memories and poor attention to detail</strong>. Our biggest challenge is still how to stop ourselves from screwing everything up.</p><p>If this is true for software, it’s doubly true for distributed systems. The biggest challenge in a distributed system for the average jo(e) <em>won’t</em> be a <a href="http://research.microsoft.com/en-us/people/mickens/thesaddestmoment.pdf">Byzantine Generals problem</a>—it’ll be whether or not they can even find the damn computer that request X was processed on for Y date and <em>oh did you remember that box Z was taken out for maintenance because we ran out of letters in the alphabet and we only allocated one character for the names?</em></p><blockquote>But wait!</blockquote><p>I hear you say.</p><blockquote>I can’t really learn those skills without having a lot of hardware!<br>It’s a catch 22!</blockquote><p>First, stop talking so loudly. You’re taking up more space than I am. Second, it’s simply not true.</p><p>Sure, experience with Big Fat Boxes <em>can</em> teach those skills, but only in the same way owning Jimi Hendrix’s 1.3 million dollar Stratocaster <em>can</em> make you a great guitarist. You <em>might</em> start playing 64th note solos, but more than likely you’re just going to make the same mistakes you would have made with a $100 guitar (and who are we kidding, the thing will be so expensive you’ll probably just be too afraid to touch it).</p><p>My point: once you get over a handful of anything, you have to start thinking differently. Whether it’s 10 or 100 or 1000 servers — <strong>a large number is a large number. </strong>If you can’t coordinate <em>10</em> <em>servers </em>and handle the problems associated therein, what makes you think you’ll be any better when <em>a few thousand </em>are involved?</p><p>I say this not to discourage. My point is that <strong>you can still get some of the most important skills on commodity hardware —</strong> skills such as managing complexity across systems and gracefully handling when a node inevitably fails.</p><h4>Where We’re Going</h4><p>My aim with this piece is to begin to show the possibilities that are available to you today. All of the below was done in a single afternoon, on a <em>$20/month</em> Digital Ocean instance (i.e. about $0.50 worth of computing time).</p><p>We’ll go through creating containers with <a href="https://www.docker.com/">Docker</a>, setting up service discovery with <a href="https://www.consul.io/">Consul</a>, and dynamically balancing the load with Nginx. We’ll start with creating a simple web application to use as our example.</p><h3>Making the Application</h3><p>For our application, let’s do something simple: create an HTTP server that does some work lasting a few seconds. The first thing to do (after <a href="https://docs.docker.com/installation/">installing Docker</a>) is to launch a container. I’m working on Ubuntu, so I’m using that as my base:</p><pre>sudo docker pull ubuntu<br>sudo docker run -i -t ubuntu /bin/bash</pre><p>This should drop you into a fresh shell as root into your new container. Now, let’s install Python:</p><pre>apt-get install -y vim python python-dev python-distribute python-pip</pre><p>As well as <a href="http://flask.pocoo.org/">Flask</a>, for a simple server:</p><pre>pip install flask</pre><p>Next, let’s create our Flask application, making it calculate a prime number and return back the result (I’m aiming for ~5 seconds of processing time here):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86fe6c3c8a1208aa0e012b873aa9b201/href">https://medium.com/media/86fe6c3c8a1208aa0e012b873aa9b201/href</a></iframe><p>Now let’s get the container’s IP:</p><pre>root@dockbox:/home/ubuntu# docker inspect 6b0262f0fe5c<br>[...]<br>        “IPAddress”: “172.17.0.13&quot;,<br>[...]</pre><p>Now, simply <em>curl</em>ing the endpoint should give us a fresh, randomly generated prime number:</p><pre>root@dockbox:/home/ubuntu# time curl 172.17.0.13:5000<br>63386263<br>real 0m5.948s<br>user 0m0.000s<br>sys 0m0.009s</pre><p>And it took about 5 seconds — the sweet spot I was aiming for.</p><h3>Cranking Up the Containers</h3><p>Let’s kick it up a notch and launch 20 more of these things. First, let’s commit our container to an image so we can launch more:</p><pre>docker commit 6b0262f0fe5c prime:v1</pre><p>Next, a quick sanity check, launching from our new image:</p><pre>root@dockbox:~# docker run -d prime:v1 python /root/work.py<br>root@dockbox:~# docker inspect 09a72800be9e | grep &quot;IPAddress&quot;<br>        &quot;IPAddress&quot;: &quot;172.17.0.16&quot;,<br>root@dockbox:~# curl 172.17.0.16:5000<br>55098829</pre><p>Perfect. Now launch the rest:</p><pre>for i in $(seq 1 20); do docker run -d prime:v1 python /root/work.py; done</pre><p>And just like that, we have 20 web servers ready for handling our application. Let’s move on to actually serving requests to our new, distributed application.</p><h4>Statically Load Balancing the Containers</h4><p>First, we’re going to need nginx on our host:</p><pre>apt-get install nginx</pre><p>Then, we’ll need our containers’ IP addresses:</p><pre>for i in $(docker ps -q); do docker inspect $i | grep &quot;IPAddress&quot; | cut -d&quot; &quot; -f 10 | sed &#39;s/[^0-9\.]*//g&#39;; done</pre><p>Finally, we’ll use the output to create an nginx config (<em>/etc/nginx/nginx.conf</em> on my system):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a8d2e24293763af1ed07fedc90f1859/href">https://medium.com/media/3a8d2e24293763af1ed07fedc90f1859/href</a></iframe><p>In a bit, we’ll come up with a way to register this with service discovery, but for now this works. After reloading nginx:</p><pre>service nginx reload</pre><p>And loading it up in the browser, we see our beautiful prime number:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gnKdkKMZZeMeDGp-moH3pA.png" /><figcaption>If you didn’t know any better, you’d think it was just an every day number.</figcaption></figure><p>Not too shabby. Keep in mind that without a firewall, <strong>this puts whatever you are load balancing to open to the public</strong>. Right now I don’t really care, but if you’re opting for something more sensitive, you should set up some iptable rules or have your server only listen locally.</p><h4>What now?</h4><p>One of the most frequent problems you have to deal with in distributed systems is how to <strong>manage where traffic is sent</strong>. This sounds like a great use case for service discovery.</p><h3>Running Consul in the Containers</h3><p>If you aren’t familiar with Consul, or service discovery, check out <a href="https://www.consul.io/intro/index.html">this nice introduction</a>. Otherwise, I’m assuming you have a cursory understanding. Let’s dive right in.</p><h4>Setting Up the Server</h4><p>I personally am using <a href="https://launchpad.net/~bcandrea/+archive/ubuntu/consul">this ppa</a> for convenience’s sake.</p><pre>apt-add-repository ppa:bcandrea/consul</pre><p>By default, this repo starts consul in <em>agent</em> mode, and I’d rather have the host be a <em>server</em> for this project. It also binds to <em>eth0</em>, and I <strong>don’t</strong> particularly want this to be open to the public. Let’s change those things.</p><p>Here’s my modified <em>/etc/init/consul.conf</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c78b2c86b449b03ced2ba7ff060352e5/href">https://medium.com/media/c78b2c86b449b03ced2ba7ff060352e5/href</a></iframe><p>Now, after stopping and starting the server, we’re in business:</p><pre>service stop consul<br>service start consul</pre><h4>Setting up the Agents</h4><p>Let’s kill our running containers from before, and modify our base image to set up the consul agent. You can stop a container by simply calling:</p><pre>docker stop <em>container_id</em></pre><p>Now, let’s launch our image again:</p><pre>docker run -i -t prime:v1</pre><p>And inside the container, follow the previous steps for installing Consul, with an extra step to get access to <em>apt-add-repository </em>and <em>curl</em> for doing health checks:</p><pre>apt-get install software-properties-common curl<br>apt-add-repository ppa:bcandrea/consul<br>apt-get update<br>apt-get install consul</pre><p>Since the container doesn’t have a fully functioning init system, we’re going to write our own incantation to start the Consul agent. Here’s what I came up with (saved to <em>start-consul.sh).</em></p><pre>BIND=`ifconfig eth0 | grep &quot;inet addr&quot; | awk &#39;{ print substr($2,6) }&#39;`<br>/usr/bin/consul agent -config-dir=&quot;/etc/consul.d&quot; -retry-join 172.17.42.1 -bind=$BIND</pre><p>We’ll want to register a service to take full advantage of Consul, so let’s create a file called<em> /etc/consul.d/web.json</em> (this will allow its config to be automatically picked up, since we specified its directory as the config directory):</p><pre>{<br>    &quot;service&quot;: {<br>        &quot;name&quot;: &quot;web&quot;,<br>        &quot;tags&quot;: [&quot;nginx&quot;],<br>        &quot;port&quot;: 5000,<br>        &quot;check&quot;: {<br>            &quot;script&quot;: &quot;curl localhost:5000 &gt;/dev/null 2&gt;&amp;1&quot;,<br>            &quot;interval&quot;: &quot;30s&quot;<br>        }<br>    }<br>}</pre><p>Next, since Docker has a single entry point for containers, let’s make a generic <em>run.sh</em> file that will launch both the consul agent, as well as the web server:</p><pre>/root/start-consul.sh &amp;<br>python /root/work.py</pre><p>Give the file execution permissions, exit the container, and commit it to a new image:</p><pre>docker commit 18feadc515f7 prime:v2</pre><p>Start it up:</p><pre>docker run -d prime:v2 /bin/bash /root/run.sh</pre><p>And check that our consul server is aware of the container:</p><pre>root@dockbox:~# consul members<br>Node          Address            Status  Type    Build  Protocol<br>dockbox       172.17.42.1:8301   alive   server  0.4.1  2<br>ea66b40276c3  172.17.0.131:8301  alive   client  0.4.1  2</pre><p>Smooth. Let’s launch more!</p><pre>root@dockbox:~# for i in $(seq 1 19); do docker run -d prime:v2 /bin/bash /root/run.sh; done<br>root@dockbox:~# consul members | wc -l<br>21</pre><p>That includes the header line, but you get the idea. The new containers were automatically picked up by Consul. Pretty cool, eh?</p><h3>Dynamic Load Balancing</h3><p>Now let’s take the information we now have from service discovery and use it to automatically modify our nginx config. We’re going to use <a href="https://github.com/hashicorp/consul-template#installation"><em>consul-template</em></a> to do this.</p><h4>Preparing the Nginx Template</h4><p>Once you’ve followed the instructions for <a href="https://github.com/hashicorp/consul-template#installation">installing consul-template</a>, create a template file for consul-template<em> </em>to work off of:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/56b2915792ea67697e57899cafa400c3/href">https://medium.com/media/56b2915792ea67697e57899cafa400c3/href</a></iframe><p>I stored this as <em>/etc/nginx/nginx.ctmpl</em>. This will serve as the base that consul template will use to generate the nginx config every time it detects changes to running services.</p><h4>Starting Consul Template</h4><p>To start consul template watching the file, run:</p><pre>./consul-template -template &quot;/etc/nginx/nginx.ctmpl:/etc/nginx/nginx.conf:service nginx reload&quot;</pre><p>You should see that <strong>the config is automatically reloaded</strong>:</p><pre>* Reloading nginx configuration nginx                       [ OK ]</pre><p>Looking at the new config file should confirm that it has been updated with any running containers you have. After a few seconds, having launched a few more containers and given their service checks time to pass, here’s what my nginx config shows:</p><pre>upstream primes {<br>   server 172.17.0.152:5000;<br>   server 172.17.0.154:5000;<br>   server 172.17.0.155:5000;<br>   server 172.17.0.163:5000;<br>   server 172.17.0.152:5000;<br>   server 172.17.0.161:5000;<br>   server 172.17.0.156:5000;<br>   server 172.17.0.158:5000;<br>   server 172.17.0.168:5000;<br>   server 172.17.0.167:5000;<br>   server 172.17.0.157:5000;<br>}</pre><p>Wahoo! This is exciting stuff. Loading up the website shows that my load balancing still works correctly. If you continue to launch containers and monitor your nginx config, you should start to see more of your containers being <strong>automatically added into rotation</strong> once their service checks start to pass.</p><h3>Setting Up the Web UI</h3><p>The web UI is essentially just a wrapper around all of the <a href="http://www.consul.io/docs/agent/http.html">HTTP API</a> endpoints, but is <strong>much</strong> less cumbersome for humans to use. If you’re following along (using the ppa I mentioned earlier), you can easily install the web-ui to get a little better insight into what’s going on:</p><pre>apt-get install consul-web-ui</pre><p>When you do this, a new file will be created in <em>/etc/consul.d/</em> called <em>10-web-ui.json</em>. <strong>If you don’t want the whole world peeking in</strong>, you’ll want to change the default binding of 0.0.0.0 to 127.0.0.1.<em> </em>Once you do that, you can restart consul and tunnel traffic from your machine to the web UI over SSH:</p><pre>ssh <a href="/service/mailto:sauce@dan.dev.saucelabs.net">root@</a>dockbox -L 8500:localhost:8500 -N</pre><p>Now you can load up the web UI (<a href="http://localhost:8500/ui/"><em>localhost:8500/ui/</em></a>) and check out your containers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-AQTweBjkHChzOUJlkfNKg.png" /><figcaption>*Sniff* they grow up so fast</figcaption></figure><p>All in all, not bad for an afternoon’s work!</p><h3>Where to Go from Here</h3><p>I hope you see the possibilities of what we’ve explored here. There are so many more things we could start to do on this box to build skills a bit more, such as:</p><ul><li>Simulate network outages and see how our system responds.</li><li>Randomly take down containers while under a stress test and see how many requests fail. How can this number be reduced? What might be an acceptable compromise?</li><li>What is the maximum number of requests our system can handle at a time? Could we tweak some operating system settings to make this higher?</li><li>How could we tag and release different versions of our software (e.g. <em>development</em>, <em>staging</em>, and <em>production</em>), while having them all co-exist peacefully in their own containers?</li><li>One box can only go so far. How can we get multiple boxes running a set of containers and manage them easily?</li></ul><p><strong>The possibilities are endless.</strong> If you’re creative and curious enough, you can easily find a hundred more things to test out. Get a database cluster involved. Try some caching layers. Let your imagination go wild! You don’t need crazy hardware. Just your brain.</p><h3>Helpful Commands</h3><p>Here are some commands I found myself running regularly throughout this process.</p><h4>List all containers</h4><pre>docker ps -a</pre><h4>Launch 15 containers</h4><pre>for i in $(seq 1 15); do docker run -d prime:v6 /bin/bash /root/run.sh; done</pre><h4>Stop all running containers</h4><pre>for i in $(docker ps -q); do docker stop $i; done</pre><h4>Delete all containers</h4><pre>for i in $(docker ps -a -q); do docker rm $i; done</pre><h4>Remove a container after it’s stopped</h4><pre>docker run --rm [...]</pre><h3>Other Resources</h3><ul><li><a href="https://docs.docker.com/userguide/dockerimages/">Working with Docker Images</a></li><li><a href="https://www.digitalocean.com/community/tutorials/docker-explained-how-to-containerize-python-web-applications">Docker Explained: How To Containerize Python Web Applications</a></li><li><a href="https://docs.docker.com/userguide/usingdocker/">Working with Containers</a></li><li><a href="https://github.com/wsargent/docker-cheat-sheet#exposing-ports">Docker Cheat Sheet</a></li><li><a href="http://godoc.org/github.com/hashicorp/consul-template/dependency">consul-template structs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70901d4741e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Experience Setting Up Consul]]></title>
            <link>https://medium.com/@dan.ellis/my-experience-setting-up-consul-b78fd67e1b78?source=rss-4cb5669be321------2</link>
            <guid isPermaLink="false">https://medium.com/p/b78fd67e1b78</guid>
            <dc:creator><![CDATA[Daniel Ellis]]></dc:creator>
            <pubDate>Sat, 14 Feb 2015 23:45:40 GMT</pubDate>
            <atom:updated>2015-02-14T23:45:40.606Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve spent the last couple of days off and on setting up Consul v0.4.1 in a development environment with about ~20 total nodes. Here were some of the things I ran into.</p><h3>Your cluster could appear to “work” but still be misconfigured.</h3><p>If your agents can reach the server, they’ll come up as alive. But if your server can’t get back to the agents, they will frequently drop out. You might see something like this in your system logs:</p><pre>memberlist: Push/Pull with xxx failed: dial tcp 100.200.100.200:8301: i/o timeout</pre><p>The server will try over both TCP and UDP on port 8301, so make sure you have both allowed in your firewall rules.</p><h3>If you’re bootstrapping, you’re gonna have a lesser experience</h3><p>They recommend a minimum server cluster size of 3, and it’s clear that the bootstrap experience just hasn’t been tested all that much. I personally just wanted to get a setup going under a development environment, so I had little interest in starting with a 3 node cluster. Failover wasn’t particularly important to me, given the development setup.</p><p>I frequently got into a state where <a href="https://groups.google.com/forum/#!topic/consul-tool/ZRxr0YuJm48">my bootstrapped, single node server decided it didn’t want to be leader</a>, and <a href="https://groups.google.com/forum/#!topic/consul-tool/Aqfa0OUpUbU">an agent inexplicably decided it wanted to be a server, and wouldn’t join the rest of the nodes</a>.</p><h4>Fixing the hesitant leader</h4><p>The first problem was caused by my changing the bind address for my server, and solved by removing the invalid nodes from the <em>peers.json</em> in my data directory.</p><pre>[“172.22.74.140:8300&quot;, <strong>“bad.bad.bad.bad”</strong>]</pre><h4>Convincing the rogue server it was an agent</h4><p>The second one ended up being resolved by deleting a few lines from one of my config files in <em>/etc/consul.d/</em></p><p>Somehow, the original contents:</p><pre>{<br>    “data_dir”: “/var/lib/consul”<br>}</pre><p>Were magically overwritten to this:</p><pre>{<br>    “bootstrap”: true,<br>    “server”: true,<br>    “data_dir”: “/var/lib/consul”,<br>    “log_level”: “INFO”,<br>    “enable_syslog”: true<br>}</pre><p>I have no idea how these happened. Apparently the first is a known bug in 4.1 and is fixed in the upcoming (and previous) release. I’m not sure if the second was really Consul’s fault, since I’m using an <a href="https://launchpad.net/~bcandrea/+archive/ubuntu/consul/+packages">Ubuntu package</a> that might be doing some weird things I’m unaware of.</p><h3>Consul template is awesome</h3><p>I haven’t explored it too much yet, but it seems to work about as awesome as I had imagined. I set up a basic template generator, and within a few seconds of manually removing a node, the template was updated. Adding the node back appeared to update the template instantly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b78fd67e1b78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Slacking Off]]></title>
            <link>https://medium.com/@dan.ellis/slacking-off-7356a2fa3037?source=rss-4cb5669be321------2</link>
            <guid isPermaLink="false">https://medium.com/p/7356a2fa3037</guid>
            <dc:creator><![CDATA[Daniel Ellis]]></dc:creator>
            <pubDate>Thu, 12 Feb 2015 06:52:16 GMT</pubDate>
            <atom:updated>2015-02-12T06:52:16.351Z</atom:updated>
            <content:encoded><![CDATA[<h3>Slacking Off: Hacking Slack and Annoying My Co-Workers</h3><p><em>Disclaimer: I use “hack” in the non-breaking-into-computers way here.</em></p><p>A few weeks ago, my company changed from using campfire to Slack as its communication tool of choice. Overall, I’ve been impressed with Slack, and I’m not surprised with the success they’ve seen. A few nights ago a friend asked me a deep question:</p><blockquote>What’s so great about it?</blockquote><p>My response was equally deep:</p><blockquote>I don’t know.</blockquote><p>Seriously though, I really don’t. It just feels right in every single place. I almost never feel as though I’m fighting against the UI. It just… works. Anyway, I’m digressing (and swooning a bit too).</p><p>A few weeks ago, I decided to start poking around with Slack to see what all I could make it do programmatically. I was inspired after quite a few abuses of <strong>@channel</strong> (if you’re a Slack user, you know how controversial these can be):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqxmZcsVKbP9YBL1ICcr9Q.jpeg" /></figure><p>Yeah, people take this stuff seriously. And where people take things seriously, opportunity for mischief arises. (Granted, it is annoying.)</p><h3>Reversing the Communication</h3><p>I began brainstorming a few things I might be able to do, but the first step was to start to learn how the internals actually worked. Thankfully, the client itself is just a wrapper around the web interface, so I was able to use Chrome’s developer tools, which were familiar with me, to explore.</p><p>My first question: how was the chat data being transmitted? I opened up the network panel, refreshed the page, and saw this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gyw0HqfyR7YdxHyREH2WXg.png" /><figcaption>I may have gotten a bit obsessive with the blurring…</figcaption></figure><p>Nice! The data was basically just json. How about some other actions? What happened if I tried to edit a message?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jatMEOj3ljshV2N91iTJWg.png" /></figure><p>Turns out that request, along with a few other “maintenance-y” things, are still sent over standard HTTP requests.</p><p>Cool. So it was time to start hacking. My first step was to see if I could re-use an existing web-socket URL in Python to try to inject commands into. Unfortunately, it appeared that these websocket URLs were only valid for the initial connection.</p><p>I started digging around the Slack JavaScript source code, and found that although the websocket URL was embedded in the page on initial load, if the connection was lost, a request was made with the user’s API token to get the URL to a new session!</p><p>Thankfully, tokens are stored right in the page, so it was pretty easy to find:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zocf6jGyTTlGMG_YVt7lKA.png" /></figure><p>After a brief search for a websocket library for Python, I combined it with the output from Chrome’s handy <em>Copy as cURL </em>feature, parsed the response, and started a websocket connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*c2uWHbFzzQEvkt2N1tGY0Q.png" /></figure><p>This was basically all I needed to get up and running (though this process was lengthy, and took a bit longer than I’m letting on). Here are some of the things I created.</p><h3>The Hacks</h3><h4>Animated Text</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*W8wxHRMb6CU9mDmCiEuu_A.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*t7yks8PqqvXut3ELOm1Heg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*T6LzWzLP0HuClRsaMntKKA.gif" /></figure><p>I had a Zalgo one, but a change in the last couple of months appears to have broken it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*ijnCYayIA3nA0UvSP2vN0w.gif" /><figcaption>Slack killed Zalgo :-(</figcaption></figure><h4>Animated Icons</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*nIjS-KUdAeit92IqtboaTQ.gif" /></figure><p>Since I did this, Slack has now added support for animated icons. Whatever. I did it before it was cool.</p><h4>Animated Avatars</h4><p>It has an annoying flashing effect, but using the “crop” option of uploaded images, we can easily animate our avatars as well.</p><blockquote>…</blockquote><p>Dammit! It looks like they changed something, and my script isn’t working. I’ve learned my lesson about writing blog posts 2 months after I’ve done the thing I’m writing about. I leave this one as an exercise to the reader.</p><h4>Changing Notification Text</h4><p>This one is a bit more tricky and nefarious.</p><p>Using <a href="http://hakshop.myshopify.com/products/usb-rubber-ducky-deluxe">a USB rubber ducky</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*ZSLplCbkCmY4raq7-IAWkQ.png" /></figure><p>I was able to create a USB thumb drive that would type a JavaScript command to send a user’s API token to my web server when inserted into a user’s computer (functioning essentially like a really fast-typing keyboard). Then, I could use that to change their notification text to be on something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*lvzImjbjo3RMgcJ0lb40MA.png" /></figure><p>Every time someone said “the”… well, you can guess what happened.</p><p>Also, I just want to say this one made me feel uneasy. I’m not sure about the ethics of it. I wasn’t getting anyone’s password at least, and I wasn’t installing anything on their computer.</p><h3>Other Ideas</h3><h4>Channel Spammer</h4><p>Given that you can automate the deletion of your messages, you could easily spam <em>@channel</em> and then immediately delete it, giving people notifications without being able to track down where they came from.</p><p>The only reason I didn’t follow this through was that my username still pops up in the OSX notification box, so they’d still know what was going on.</p><h3>Techniques That Helped</h3><h4>JavaScript De-obfuscation</h4><p>I spent a while searching the internet for a de-obfuscator for JavaScript, before I realized Chrome has one built in! It’s nice and un-assuming:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*xQ9v9MWVQw84cub1aE7Xcg.png" /></figure><p>Yup, those two brackets there. They turn this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xJX6uYlxiiLbacvFeE0jXA.png" /></figure><p>Into this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEXt1tnk04ZkGiSjeMkLiQ.png" /></figure><p>Ah, much better!</p><h4>Breakpointing Inside Closures</h4><p>After digging through the source code for a while, I found a couple of objects I wanted to play with in the javascript console. The problem? They were defined in anonymous closures. The solution? Breakpoint, and make the objects global!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6HergWKKonrw7E58xdUCg.png" /></figure><p>At this point, I can easily grab any variable accessible to that closure and save it for later use, continuing execution.</p><p>If you look closer at that image, you might have a couple of questions. TS stands for <a href="http://tinyspeck.com/">TinySpeck</a>. As for that <em>DELEEEEETETEEESTTTT</em> thing, well…</p><h3>Funny Things I Found</h3><h4>An Easter Egg</h4><p>If you type in “DELEEEEETETEEESTTTT” to any channel and press Enter, a funny thing happens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pYgnHVk_bLbYQSig08Dwog.png" /></figure><p>You can’t send it! I’m assuming this is / was used for debugging, and it just so happened to be useful for figuring out the websocket connection process mentioned earlier, since this causes the client to reconnect.</p><h4>Colorful Comments</h4><p>Slack oozes style, and the comments in their JavaScript don’t disappoint:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*reFIgLLJ3gZWDfARdV6CXg.png" /></figure><h3>Hacking as Play</h3><p>You may wonder why I did all of this. There is certainly an element of mischievousness to it all, but that doesn’t explain everything. At it’s core: it’s play. Hacking is play. Creativity is play. In my opinion, the essence of hacking is getting a system to do something that it otherwise wasn’t meant to do. Extending the bounds of it.</p><p>A lot of times this overlaps with mischief, but mischief is just the general expression rather than the end goal itself. Doing all of this play without having a target is like a tree falling in the forest with no-one else around to hear it. Having someone hear it in many ways brings the hack to life.</p><p>People sometimes laugh, pigeon holing it as a quirky (err, frustrating?) personality trait. But in my mind, it’s the essence of learning, the essence of curiosity. What greater curiosity than to dig deep into a system that you didn’t build, just to learn its inner workings and have fun with it?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7356a2fa3037" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>