<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kent Gruber on Medium]]></title>
        <description><![CDATA[Stories by Kent Gruber on Medium]]></description>
        <link>https://medium.com/@KentGruber?source=rss-67a458d49f6e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*gIKiM6KL-T30zbS3-36CWQ.png</url>
            <title>Stories by Kent Gruber on Medium</title>
            <link>https://medium.com/@KentGruber?source=rss-67a458d49f6e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:13:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@KentGruber/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Glint — Extensible, declarative linter for Golang]]></title>
            <link>https://medium.com/@KentGruber/glint-extensible-declarative-linter-for-golang-6e05e4386deb?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e05e4386deb</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[linting]]></category>
            <category><![CDATA[command-line]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 01:16:09 GMT</pubDate>
            <atom:updated>2019-09-13T23:32:52.227Z</atom:updated>
            <content:encoded><![CDATA[<h3>Glint — Extensible, declarative linter for Golang</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9AA_DsAGwxx6n3--vhrKiQ.jpeg" /><figcaption>📸Credit for the photo goes to <a href="https://unsplash.com/@veeterzy"><strong>Veeterzy</strong></a></figcaption></figure><p>I’ve become fascinated with <a href="https://golang.org/pkg/go/ast/"><strong>go/ast</strong></a> — a library that allows you to inspect the <a href="https://en.wikipedia.org/wiki/Abstract_syntax_tree">abstract syntax tree</a> of a golang program in its <em>native tongue</em>. While playing around with it over this past weekend after watching a “<a href="https://www.youtube.com/watch?v=YRWCa84pykM">just for func</a>” episode, I had a idea:</p><blockquote>What if there was a <a href="https://en.wikipedia.org/wiki/Declarative_programming">declarative</a> way of configuring a golang <a href="https://en.wikipedia.org/wiki/Lint_(software)">linter</a>?</blockquote><p>I began sketching out the initial program in my editor, creating a JSON file that would serve as the base for all the rules that could be defined.</p><pre>{<br>    &quot;rules&quot;: [<br>        {<br>            &quot;type&quot;: &quot;&quot;,<br>            &quot;comment&quot;: &quot;&quot;,<br>            ...<br>        },<br>        ...<br>    ]<br>}</pre><p>After a little bit more toiling around with the backend engine I had something that could unmarshal the JSON-defined rules, walk the abstract syntax tree of a program, and apply rules to each node of the tree… <em>I had a linter!</em></p><h3>Glint</h3><p>Since all software needs some sort of name, I dubbed this project “<a href="https://github.com/picatz/glint"><strong><em>glint</em></strong></a>”. ✨</p><p>I wrote three different rule types which glint currently supports:</p><ul><li><strong>import</strong> to inspect imported packages</li><li><strong>struct</strong> to inspect the fields of <a href="https://gobyexample.com/structs">struct</a> types</li><li><strong>method</strong> to inspect <a href="https://gobyexample.com/methods">method</a> calls</li></ul><p>For any of the aforementioned rule types, a comment is required which is displayed in the lint message. In order for someone to extend the available rule types in the core of glint, they would need to create a golang type that satisfies the following interface:</p><pre>type <strong>Rule</strong> interface { <br>    <strong>Action</strong>(fs *token.FileSet, node ast.Node)<br>}</pre><blockquote><strong>What’s this mean?</strong>: <em>A rule must perform some sort of action, which is performed on a node found in the AST. Rule types can be added later as necessary, or even created by outside contributors.</em></blockquote><p>However, the primary goal for glint <em>isn’t</em> really to be easily extensible at the engine source code level. Instead, the focus is to be extensible at the configuration level, describing rules for the linter in something I like to call a Glintfile.</p><p>Three example rules that can be described:</p><pre>{    <br>  &quot;<strong>rules</strong>&quot;: [<br>    {<br>      &quot;<strong>type</strong>&quot;: &quot;import&quot;,<br>      &quot;<strong>comment</strong>&quot;: &quot;we don&#39;t use <a href="https://godoc.org/-/subrepo">golang.org/x/*</a> packages&quot;,<br>      &quot;<strong>match</strong>&quot;: [<br>        &quot;golang.org/x/\\w+&quot;<br>      ],           <br>    },<br>    {<br>      &quot;<strong>type</strong>&quot;: &quot;struct&quot;,<br>      &quot;<strong>comment</strong>&quot;: &quot;always set ReadTimeout to something not 0&quot;,<br>      &quot;<strong>name</strong>&quot;: &quot;<a href="https://golang.org/pkg/net/http/#Server">http.Server</a>&quot;,<br>      &quot;<strong>field</strong>&quot;: &quot;ReadTimeout&quot;,<br>      &quot;<strong>match</strong>&quot;: [<br>        &quot;nil&quot;, &quot;0&quot;<br>      ]<br>    },<br>    {<br>      &quot;<strong>type</strong>&quot;: &quot;method&quot;,<br>      &quot;<strong>comment</strong>&quot;: &quot;use 2048 bits or more when generating RSA keys&quot;,<br>      &quot;<strong>call</strong>&quot;: &quot;<a href="https://golang.org/pkg/crypto/rsa/#GenerateKey">rsa.GenerateKey</a>&quot;,<br>      &quot;<strong>argument</strong>&quot;: 1,<br>      &quot;<strong>less_than</strong>&quot;: 2048<br>    }<br>  ]<br>}</pre><p>Using this — I really hope — an operator shouldn’t necessarily need to be a golang AST expert to create or customize the behavior of a rule. Instead they can declaratively describe, share, and collaborate on rules at a higher-level presented in this mini DSL (<a href="https://en.wikipedia.org/wiki/Domain-specific_language">domain specific language</a>) using a familiar syntax that is both human <em>and</em> machine readable (JSON).</p><p><em>So, how do you run the linter?</em> <strong>Great question!</strong></p><p>It is primarily meant to be used from the command-line. Though, perhaps in the future it can integrated as a plugin within your text editor of choice.</p><h4>Installation</h4><pre><strong>$ </strong>go get -u <a href="https://github.com/picatz/glint">github.com/picatz/glint</a></pre><h4>Inspect a Single File</h4><pre>$ glint /path/to/source/code.go<br>...</pre><h4>Inspect a Package</h4><pre>$ glint github.com/picatz/backdoor<br>...</pre><h4>Specify the Glintfile</h4><pre>$ glint -rules=/path/to/Glintfile code.go</pre><h3>Conclusion</h3><p><a href="https://github.com/picatz/glint">You can check out documentation, source code, and examples for </a><a href="https://github.com/picatz/glint">glint on GitHub</a>. Contributions are of course welcome if you would like to help improve the core rule types. Otherwise trying it out or attempting to build rules with it to give feedback on the <a href="https://github.com/picatz/glint/issues">issue tracker</a> would be just as welcome. It’s still in an early stage with some room to improve, but I think it should be quite useful — or at the very least, <em>usable</em> — in its current state.</p><p>👏 Finally, I want to give a big <strong><em>thank you</em> </strong>to both<strong> </strong><a href="https://medium.com/justforfunc/understanding-go-programs-with-go-parser-c4e88a6edb87">Fransec Campoy</a> and <a href="https://arslan.io/2017/09/14/the-ultimate-guide-to-writing-a-go-tool/">Fatih Arslan</a> for making such wonderful blog posts about go/ast and related packages for analyzing golang source code.</p><p>Until next time, that’s all folks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e05e4386deb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build your own Intrusion Detection System]]></title>
            <link>https://medium.com/@KentGruber/build-your-own-intrusion-detection-system-e652f574037d?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e652f574037d</guid>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[intrusion-detection]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Sat, 06 Apr 2019 15:12:05 GMT</pubDate>
            <atom:updated>2019-04-07T04:55:12.271Z</atom:updated>
            <content:encoded><![CDATA[<p>Intrusion detection systems monitor network traffic in hopes of identifying malicious activity. Often times pre-built solutions are deployed such as <a href="https://www.zeek.org/">Zeek</a> (<a href="https://blog.zeek.org/2018/10/renaming-bro-project_11.html">formerly Bro</a>), <a href="https://suricata-ids.org/">Suricata</a>, or <a href="https://www.snort.org/">Snort</a> — but, how can we build our own?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CJkyfpn9cKGiyV07JBqW0A.png" /><figcaption>📸Credit for the photo goes to <a href="http://www.directorjakobowens.com/"><strong>Jakob Owens</strong></a></figcaption></figure><h4>Processing Packets with PacketGen</h4><p>One of the <em>best</em> packet analysis libraries out there is <a href="https://github.com/sdaubert/packetgen"><strong>PacketGen</strong></a>. It offers simple — but powerful — ways to generate, send and capture network packets. With a <a href="https://github.com/sdaubert/packetgen/wiki/Supported-Protocols">wide variety of supported protocols</a> it provides a wonderful base to build our engine upon.</p><p>Since it’s written in Ruby, a language that makes it<a href="https://thoughtbot.com/blog/writing-a-domain-specific-language-in-ruby"> easy to craft domain specific languages</a>, we can slap on some <a href="https://blog.appsignal.com/2018/02/20/ruby-magic-syntactic-sugar-methods.html">syntactic sugar </a>to wrap PacketGen’s functionality into a flexible IDS — in less than 25 lines-of-code. Seriously.</p><h4>⚙️ Engine DSL</h4><p>The following is all of the required code for our IDS engine (don’t feel the need to read it all unless you really want to):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a51143d4a685ef15dc779edfd9a093b4/href">https://medium.com/media/a51143d4a685ef15dc779edfd9a093b4/href</a></iframe><p>💁‍♀️ Quick explanation of what’s going on:</p><ul><li><a href="http://rubylearning.com/satishtalim/including_other_files_in_ruby.html">require</a> &quot;packetgen&quot; loads the PacketGen library to use.</li><li><a href="http://rubylearning.com/satishtalim/writing_our_own_class_in_ruby.html">class</a> IDS contains our intrusion detection engine.</li><li><a href="http://ruby-for-beginners.rubymonstas.org/writing_classes/initializers.html">def initialize</a> is called when we create a .new instance of an IDS object allowing us to do all the required setup necessary, which is basically just loading the rules for our engine and applying them to a live packet capture.</li><li><a href="https://www.howtogeek.com/howto/programming/ruby/ruby-function-method-syntax/">def rule</a> actually registers a new rule in the engine (which will be placed in a <a href="https://www.tutorialspoint.com/ruby/ruby_hashes.htm">key-value pair</a>).</li><li><a href="http://rubylearning.com/satishtalim/ruby_blocks.html">&amp;block</a> is used as an argument to these functions which allows for do...end blocks — also known as <a href="https://scotch.io/tutorials/understanding-ruby-closures">closures</a> — to be passed as arguments and evaluated by our engine. This becomes a bit more clear later on when you see how it is used. So, keep an eye out for those do...end blocks in a minute in the example code snippets.</li></ul><p>Frankly, if you’re not all that familiar with Ruby, the &amp;block stuff might seem a little intimidating — but, as an operator using this library to develop rules you might not need to intimately understand <em>everything</em> in the engine to write rules for it.</p><p>In fact, the engine could abstracted away (hidden) from an operator and presented as a command-line application that reads configuration files with a few more lines of code.</p><p>For this blog post, we’ll just keep it really bare-boned.</p><h4>✍️ Writing Rules</h4><p>Now, let’s try writing some rules. Without rules/signatures to match against, a <a href="https://en.wikipedia.org/wiki/Intrusion_detection_system#Detection_method">signature-based IDS</a> wouldn’t do you any good. First we’ll manually convert the following snort rule to our engine’s syntax as a starting point:</p><pre><strong>alert tcp any any -&gt; any any (content: &quot;cgi-bin/phf&quot;; flags: PA; msg: &quot;CGI-PHF probe&quot;;)</strong></pre><p>The main features of the rule are that it checks both the PSH and ACK TCP flags are set and that <a href="https://techterms.com/definition/payload">packet’s payload</a> must contain the string <strong>&quot;cgi-bin/phf&quot;</strong>.</p><p>When converted, it ends up being <em>very</em> readable:</p><pre><strong>rule</strong> &#39;TCP&#39; <strong>do</strong> |packet|<br>   <strong>next unless</strong> packet.tcp.flag_psh? <strong>and</strong> packet.tcp.flag_ack?<br>   <strong>next unless</strong> packet.body.contains? <strong>&quot;cgi-bin/phf&quot;</strong><br>   puts <strong>&quot;CGI-PHF probe&quot;</strong><br><strong>end</strong></pre><p>Perhaps we want to know whenever there’s <strong>any</strong> communication going to Google’s DNS server 8.8.8.8:</p><pre><strong>rule</strong> &#39;IP&#39; <strong>do</strong> |packet|<br>   <strong>next unless</strong> packet.ip.dst == <strong>&quot;</strong><strong>8.8.8.8&quot;</strong><br>   puts <strong>&quot;Talking to Google&#39;s DNS server&quot;</strong><br><strong>end</strong></pre><p>We could narrow it down to simply DNS communication:</p><pre><strong>rule</strong> &#39;DNS&#39; <strong>do</strong> |packet|<br>   <strong>next unless</strong> packet.ip.dst == <strong>&quot;</strong><strong>8.8.8.8&quot;</strong><br>   puts <strong>&quot;Talking to Google&#39;s DNS server using DNS&quot;</strong><br><strong>end</strong></pre><p>Or, similarly, just ICMP communication:</p><pre><strong>rule</strong> &#39;ICMP&#39; <strong>do</strong> |packet|<br>   <strong>next unless</strong> packet.ip.dst == <strong>&quot;</strong><strong>8.8.8.8&quot;</strong><br>   puts <strong>&quot;Talking to Google&#39;s DNS server using ICMP&quot;</strong><br><strong>end</strong></pre><p>If we wanted to catch FTP root login attempts (case sensitive match):</p><pre><strong>rule</strong> &#39;TCP&#39; <strong>do</strong> |packet|<br>   <strong>next unless</strong> packet.tcp.dport == <strong>21</strong><br>   <strong>next unless</strong> packet.body.include?(<strong>&quot;USER root&quot;</strong>)<br>   puts <strong>&quot;FTP root login attempt&quot;</strong><br><strong>end</strong></pre><p><em>The examples could go on-and-on from here. But I think that gives you a pretty good idea of what is possible using this engine.</em></p><p>You can essentially create any rule you can imagine. Moreover, you could run <em>any valid Ruby code</em> which gives you the ability to do centralized logging, e-mail alerting, slack notifications — again,<em> pretty much anything</em> you can imagine.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66885e5bc4d78d70a0c7dcacbd57773a/href">https://medium.com/media/66885e5bc4d78d70a0c7dcacbd57773a/href</a></iframe><h3>Conclusion</h3><p>There are advantages and disadvantages to writing your own solution to a problem. While I don’t expect any organization to deploy this exact code in production, I really do love the simplicity of it, especially as a starting point for others interested in this topic.</p><p><em>Until next time, that’s all folks!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e652f574037d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build your own Ad Blocker]]></title>
            <link>https://medium.com/@KentGruber/build-your-own-ad-blocker-90230436e364?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/90230436e364</guid>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[ad-blocking]]></category>
            <category><![CDATA[chrome-extension]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Fri, 14 Dec 2018 14:16:37 GMT</pubDate>
            <atom:updated>2020-01-04T18:57:49.817Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve always wondered if I could make my own ad blocker — to my surprise, in about five lines of JavaScript, we can!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJrYth91KZGNlvDHLmj-6Q.png" /><figcaption>This is what we’ll see when chrome blocks a URL with our custom Ad Blocker</figcaption></figure><h3>The Source Code</h3><p>The <a href="https://www.google.com/chrome/">chrome browser</a> exposes a <a href="https://developer.chrome.com/extensions/webRequest">webRequest</a> API that enables developers to observe and analyze traffic and to intercept, block, or modify requests in-flight.</p><p>Using this API, we can develop a <a href="https://en.wikipedia.org/wiki/URL">URL</a> blocking extension that’ll stop any requests to doubleclick.net (but we could use any URL in practice):</p><pre>chrome.webRequest.onBeforeRequest.addListener(<br>  function(details) { return {cancel: true}; },<br>  { urls: [&quot;*://*.doubleclick.net/*&quot;] },<br>  [&quot;blocking&quot;]<br>);</pre><p>The onBeforeRequest event allows us to perform some custom action before a request occurs. In fact, it’s called before any <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a> connection is made.</p><p>By registering a listener with a function that returns {cancel: true} allows us to stop the request. We can also specify which URL(s) it will block, which can be defined block both HTTP as well as HTTPS, and the even the subdomains of doubleclick.net by using “*://*.doubleclick.net/*&quot;.</p><blockquote><strong><em>Note: </em></strong><em>More than one URL could be targeted, but for the sake of brevity in this post, I’ve decided to just focus on </em><em>doubleclick.net which is a popular source of web advertisements. A broader list of advertisement sources would be necessary to be most effective.</em></blockquote><p>🧙‍✨Believe it or not, that’s all of the “real” code you need to write for this to work.</p><p>Now we just need to create an <a href="https://developer.chrome.com/extensions/manifest">manifest.json</a> file to add some extra metadata about the extension and give it the necessary <a href="https://developer.chrome.com/extensions/declare_permissions">permissions</a>:</p><pre>{<br>  &quot;name&quot;: &quot;cube&quot;,<br>  &quot;version&quot;: &quot;1.0&quot;,<br>  &quot;description&quot;: &quot;Chrome URL Blocking Extension&quot;,<br>  &quot;permissions&quot;: [&quot;webRequest&quot;, &quot;webRequestBlocking&quot;, &quot;&lt;all_urls&gt;&quot;],<br>  &quot;background&quot;: {<br>    &quot;scripts&quot;: [&quot;background.js&quot;]<br>  },<br>  &quot;manifest_version&quot;: 2<br>}</pre><blockquote><strong><em>Note:</em></strong><em> the </em><em>background.js file contains the previously shown JavaScript code that contains the core functionality for the extension.</em></blockquote><p>That’s it! We’ve created all of the necessary files to load our custom chrome extension. They should be sitting in their own directory, somewhere on your local filesystem so they can now be loaded as an extension.</p><h3>Loading the Extension</h3><ol><li>In your browser, open the Extension Management page by navigating to chrome://extensions or by clicking on the Chrome menu, hovering over More Tools, and then selecting Extensions.</li><li>Enable Developer Mode by clicking the toggle switch next to Developer mode.</li><li>Click the LOAD UNPACKED button and select the extension directory, wherever it exists on your local filesystem.</li><li>Go to doubleclick.net to check that it’s actually being blocked!</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QmxHAZHv5lwytKHquww84g.png" /><figcaption>We’ll see these new errors in our console, which is what we want!</figcaption></figure><h3>CUBE (Chrome URL Blocking Extension)</h3><blockquote><em>You can also find a slightly different version of the code shared in this post on GitHub: </em><a href="https://github.com/picatz/cube"><strong><em>github.com/picatz/cube</em></strong></a></blockquote><p>Please feel free to improve and customize as you like. It’s just a start, but is exactly what I wish I had when I began looking into this topic.</p><p><em>Until next time, that’s all folks!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90230436e364" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracking down a Golang memory leak with grmon]]></title>
            <link>https://medium.com/@KentGruber/tracking-down-a-golang-memory-leak-with-grmon-74569a00a177?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/74569a00a177</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[memory-leak]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Wed, 22 Aug 2018 14:03:28 GMT</pubDate>
            <atom:updated>2018-08-22T14:11:02.872Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9o4NHx1Hddjc9BfmAy7TVg.png" /><figcaption>🎉Finally fixing this on Sunday evening was such a relief! 🎉</figcaption></figure><p>Discovering there’s a memory leak in your application is the easy part. Diagnosing where the problem is to fix it can be tricky. Using the open-source <a href="https://github.com/bcicen/grmon"><strong>grmon</strong></a> command-line tool, I was recently able to quickly track down leaking <a href="https://golangbot.com/goroutines/">goroutines</a> in a <a href="https://subfinder-v2.appspot.com/">web application I deployed a few days ago</a>.</p><blockquote>A “<strong>leaking goroutine</strong>” is a light-weight thread that runs in the background of our program, unintentionally <strong>forever</strong>.</blockquote><h3>💧Oops, I’m Leaking Goroutines</h3><p>In short-lived applications like command-line tools, leaking goroutines tend to have a less noticeable effect on performance and memory.</p><p>However, if not implemented properly in long-lived applications like a web server, these improperly managed goroutines can degrade performance or reliability and raise the costs associated with your application over time.</p><p>While memory is <em>relatively</em> cheap, it’s certainly <strong><em>not</em></strong> free.</p><h3>Example Application</h3><p>The memory leak in my application boiled down to the fact I had a bunch of goroutines that were getting stuck sending data down a channel, even after the HTTP request that spawned those goroutines had finished.</p><p>To help illustrate this problem, here is an example “command-line” version of the application:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e439c17626cf2e19f099eedf74fd8fc4/href">https://medium.com/media/e439c17626cf2e19f099eedf74fd8fc4/href</a></iframe><p>This application is centered around the newProducer function. For the most part, it seems harmless — it even looks like it cleans up after itself with defer close(results)!</p><p>Problems won’t start to bubble up until you decide you want to turn this command-line program into an API endpoint for a web application to use.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24516a28f942e7fa99215b24e8948681/href">https://medium.com/media/24516a28f942e7fa99215b24e8948681/href</a></iframe><p>For the exampleEndpoint function, <em>to prevent abuse let’s pretend</em>, we’ve added a limit in the form of a counter which we will increment upon each result from the newProducer. After three results have been written to the client connection, we stop the function with a return.</p><p>So, now we feel confident, and ran some local tests — we’re ready to deploy this baby to the cloud!</p><p>Depending on the amount of traffic your application receives, you might start noticing a steady increase in memory usage that just <em>doesn’t</em> stop almost right away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p9ZsvQuBJolwVrfemyLF_g.png" /><figcaption>This is the exact graph I shared with <a href="https://github.com/orgs/subfinder/people">the team</a> to let them know there’s a problem.</figcaption></figure><p>👩🏽‍💻Back on our localhost, we want to start debugging this issue.</p><h3>👁‍🗨 grmon</h3><p>Setting up <a href="https://github.com/bcicen/grmon">grmon</a>— a command-line goroutine monitor — in our application to help with debugging is pretty straightforward:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1699197ea14ca1f1e0d3354a090d4c6/href">https://medium.com/media/f1699197ea14ca1f1e0d3354a090d4c6/href</a></iframe><p>The only thing about the application that has changed is the grom.Start() called before http.ListenAndServe() and the import to bring in our grmon agent.</p><p>Now, when we go to run our server, we can monitor our goroutines by running grmon in another terminal window. For good measure, and because I want to start making requests at the server, I’ll open a third terminal window to <a href="https://curl.haxx.se/">curl</a> the endpoint.</p><p>We can use <a href="https://en.wikipedia.org/wiki/Tmux">tmux</a> to easily see everything running at once:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kxq4awmN1DMqRdJQ7WKadw.png" /></figure><p>By default, we see five default goroutines that are running when we fire up our server. This is fine. If we didn’t touch the server from here, we shouldn’t see too much activity going on at all. It’s not until we start interacting the server that the problems will occur.</p><p>Now, let’s curl the server’s HTTP endpoint to see what happens with those goroutines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yvkbWF1w5oHuQZ71m2a7_A.png" /></figure><p>Do you see that <strong>chan send</strong> that is associated with the main.newProducer function in the lower half of the screenshot?</p><p>If we didn’t touch the server from here, that <em>won’t ever</em> go away or change state.</p><p>Let’s go ahead and curl the endpoint a second time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VPaXsjsDxNEQ215U-OqEDg.png" /></figure><p>Again, one of those main.newProducer functions is stuck in the <strong>chan send</strong> state. For the record, if we were to use a tool like <a href="https://golang.org/pkg/net/http/pprof/">pprof</a>, we would see similar indicators — but not as interactively.</p><p>🤔<em>How do we fix this?</em></p><h3>Setting a Channel Send Timeout</h3><p>We can fix this problem by implementing a channel send timeout. This is essentially a select statement with 2 different cases: first, where the send succeeds, and second, where a timeout occurs.</p><p>We can adjust our newProducer function to accept a first argument as a time.Duration and use that with a select upon each iteration of the given products:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31f4569570b9c7771157bd12fa514339/href">https://medium.com/media/31f4569570b9c7771157bd12fa514339/href</a></iframe><p>Now, even if a consumer of the client where to cut the the producer short for any reason, we have the &lt;-time.After(readTimeout) value to protect us from leaking an extended period of time; and we can tune this value or modify it based on the request if we wanted to.</p><blockquote><strong><em>Note</em></strong>: this is just one way to fix problem. Another — perhaps more idiomatic way — to solve this problem would be to use the <a href="https://golang.org/pkg/context/">context</a> package as <a href="https://rakyll.org/leakingctx/">gohpers like Rakyll have shown before</a>.</blockquote><p>For this application, we’ll set a default timeout of 1 second stored in the timeout variable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e7bcccd1ce7385b56f6287b15a05fe2/href">https://medium.com/media/2e7bcccd1ce7385b56f6287b15a05fe2/href</a></iframe><p>Monitoring the application with grmon, we can confirm we no longer have the stuck <strong>chan send</strong> state. Instead, we will see a <strong>select</strong> state which will go away after about a second.</p><p>You can increase the timeout value to a large value to be able to easily identify it when monitoring with grmon like so (looking for the select stat for main.newProducer):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLAdApi3du9JzcWuHovmNg.png" /></figure><h3>Conclusion</h3><p>Leaking goroutines aren’t always obvious to diagnose. Using <a href="https://github.com/bcicen/grmon">grmon</a> gave me the exact tooling I needed to find and fix my issue, and I hope it can be of some use to you in the future too.</p><p>⛵️Now my web application is cruising along nicely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0iBUbIPc8vdHN3nbTtjo2w.png" /><figcaption>Look at that beautiful, low memory usage.</figcaption></figure><p><em>Until next time, that’s all folks!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74569a00a177" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a High Performance Port Scanner with Golang]]></title>
            <link>https://medium.com/@KentGruber/building-a-high-performance-port-scanner-with-golang-9976181ec39d?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9976181ec39d</guid>
            <category><![CDATA[asynchronous]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[infosec]]></category>
            <category><![CDATA[multithreading]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Tue, 17 Jul 2018 16:45:29 GMT</pubDate>
            <atom:updated>2018-07-18T02:46:36.064Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://golang.org/">Golang</a> has a fantastic multi-threading API which allows for all sorts of opportunities for building high performance tools, including one of my favorites, a port scanner!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hu8uNkzsZJgH4OrJgxZscA.png" /></figure><blockquote><strong><em>Side Note: </em></strong><em>If you also like </em><em>async/await style programming and </em><strong><em>Ruby</em></strong><em>, </em><a href="https://medium.com/@KentGruber/building-a-high-performance-asynchronous-port-scanner-with-ruby-f2a335a3d44">you might like this</a>.</blockquote><p>I’ve written about building a port scanner several times before; and this implementation is the fastest one I’ve been able to create thus far. Utilizing the multi-threaded runtime and the baked-in concurrency model provided by Golang, asynchronous I/O performance is just beautiful.👌</p><p>What does that end up looking like to <em>actually code</em>, and how fast does it <strong><em>go</em></strong>?</p><h3>📦 Assembling our Package</h3><p>Like any other language, we’ll pull in some libraries that have encapsulated the stuff we’re interested in orchestrating into a useful program.</p><p>In the world of Golang, we’ll import a bunch of packages — and at the very top of our script, we’ll declare it as our “<em>main</em>” package.</p><pre><a href="https://www.golang-book.com/books/intro/11"><strong>package</strong></a> <a href="https://stackoverflow.com/questions/42333488/package-main-and-func-main">main</a></pre><pre><a href="https://tour.golang.org/basics/2"><strong>import</strong></a> (<br>    <a href="https://golang.org/pkg/context/">&quot;context&quot;</a><br>    <a href="https://golang.org/pkg/fmt/">&quot;fmt&quot;</a><br>    <a href="https://golang.org/pkg/net/">&quot;net&quot;</a><br>    <a href="https://golang.org/pkg/os/exec/">&quot;os/exec&quot;</a><br>    <a href="https://golang.org/pkg/strconv/">&quot;strconv&quot;</a><br>    <a href="https://golang.org/pkg/strings/">&quot;strings&quot;</a><br>    <a href="https://golang.org/pkg/sync/">&quot;sync&quot;</a><br>    <a href="https://golang.org/pkg/time/">&quot;time&quot;</a></pre><pre>    <a href="https://godoc.org/golang.org/x/sync/semaphore">&quot;golang.org/x/sync/semaphore&quot;</a><br>)</pre><pre><em>// more code ...</em></pre><p>Unlike <a href="https://en.wikipedia.org/wiki/Object-oriented_programming">object-orientated</a> languages, there’s no concept of a <a href="https://en.wikipedia.org/wiki/Class_(computer_programming)"><em>class</em></a> wherein we add methods inside of. Instead, we can just attach <a href="https://tour.golang.org/basics/4">functions</a> to <em>any </em><a href="https://www.golang-book.com/books/intro/3">type</a>, ultimately giving us a <a href="https://tour.golang.org/methods/1">method</a>.</p><p>So, let’s make a type called PortScanner which will be <a href="https://tour.golang.org/moretypes/2">struct</a> type containing two fields: ip and lock.</p><pre><strong>type</strong> PortScanner <strong>struct</strong> {<br>    ip   string<br>    lock *semaphore.Weighted<br>}</pre><p>The ip will be the <a href="https://en.wikipedia.org/wiki/IP_address">IP address</a> of the host on the network we’re interested in scanning. The lock will act as a threshold that will limit the number of <a href="https://gobyexample.com/goroutines">go routines</a> that will be running at any given time.</p><p>To set our lock based on the limits of the operating system, we may opt to use the value of the ulimit command. This can be captured to be parsed using the <a href="https://golang.org/pkg/os/exec/">os/exec</a> package.</p><blockquote><strong><em>Note: </em></strong><a href="http://wiki.linuxquestions.org/wiki/Ulimit">ulimit</a><strong><em> </em></strong>is a <a href="http://wiki.linuxquestions.org/wiki/Builtin">builtin</a> shell command used to manage various resource restrictions. The -n flag will reveal the maximum number of open files allowed; we can capture the output of this command and convert it to the right type.</blockquote><p>To abstract that whole process away of getting the value from the command-line, I decided to put it in a function called Ulimit.</p><pre><strong>func</strong> Ulimit() int64 {<br>    out, err <strong>:=</strong> exec.<a href="https://golang.org/pkg/os/exec/#Command">Command</a>(&quot;ulimit&quot;, &quot;-n&quot;).<a href="https://golang.org/pkg/os/exec/#Cmd.Output">Output</a>()</pre><pre>    <strong>if</strong> err <strong>!=</strong> nil {<br>        panic(err)<br>    }</pre><pre>    s <strong>:=</strong> strings.<a href="https://golang.org/pkg/strings/#TrimSpace">TrimSpace</a>(string(out))<br>    i, err <strong>:=</strong> strconv.<a href="https://golang.org/pkg/strconv/#ParseInt">ParseInt</a>(s, 10, 64)<br>    <br>    <strong>if</strong> err <strong>!=</strong> nil {<br>        panic(err)<br>    }</pre><pre>    <strong>return</strong> i<br>}</pre><p>Since we’re building a port scanner, we’re going to need another function that can actually, ya’ know, <em>scan the port. </em>The <a href="https://golang.org/pkg/net">net</a> package includes a convenient method to make a connection with an allotted timeout value.</p><pre><strong>func</strong> ScanPort(ip string, port int, timeout time.Duration) {<br>    target <strong>:=</strong> fmt.<a href="https://golang.org/pkg/fmt/#Sprintf">Sprintf</a>(&quot;%s:%d&quot;, ip, port)<br>    conn, err <strong>:=</strong> net.<a href="https://golang.org/pkg/net/#DialTimeout">DialTimeout</a>(&quot;tcp&quot;, target, timeout)<br>    <br>    <strong>if </strong>err <strong>!=</strong> nil {<br>        <strong>if</strong> strings.<a href="https://golang.org/pkg/strings/#Contains">Contains</a>(err.Error(), &quot;too many open files&quot;) {<br>            time.<a href="https://golang.org/pkg/time/#Sleep">Sleep</a>(timeout)<br>            ScanPort(ip, port, timeout)<br>        } <strong>else</strong> {<br>            fmt.<a href="https://golang.org/pkg/fmt/#Println">Println</a>(port, &quot;closed&quot;)<br>        }<br>        <strong>return</strong><br>    }<br>    <br>    conn.Close()<br>    fmt.Println(port, &quot;open&quot;)<br>}</pre><p>As you probably noticed, most of the ScanPort function deals with the err from net.<a href="https://golang.org/pkg/net/#DialTimeout">DialTimeout</a>. This is because, as with any port scanner, we have some errors to juggle:</p><h4>🤹🏻‍ Three Essential Errors to Juggle:</h4><ul><li>🛑<strong>connect:</strong> connection refused is an error that will occur when a port is not open to connect to. This is a pretty typical response for most ports, and this response that is enforced with most <a href="https://serverfault.com/questions/157375/reject-vs-drop-when-using-iptables">REJECT</a> firewall rules.</li><li>⏱<strong>dial:</strong> i/o timeout is an error produced by net.<a href="https://golang.org/pkg/net/#DialTimeout">DialTimeout</a> when an operation didn’t complete within the allotted timeout value. This may happen for various reasons including a <a href="https://serverfault.com/questions/157375/reject-vs-drop-when-using-iptables">DROP</a> firewall rule where the target doesn’t explicitly tell you a port isn’t open from the scanner’ perspective. We really don’t want to wait around forever to get nothing.</li><li>📑<strong>socket:</strong> too many open files is an error that will occur when we try to connect to a port, but we have way too many connections (files, really) already open and need to schedule the execution to happen again, but just a little later, continuously delaying its execution until it’s able to be processed.</li></ul><p>Based on the result of the error for the connection attempt, if the attempt failed as a result of too many files being open on the system, we’ll delay the execution of that port scan. Once the delay returns, we can again retry it until the connections fails because of a timeout or, better yet, a successful connection.</p><p>To use the ScanPort method for a range of ports, we’ll implement a method on the PortScanner struct that will scan a given range of ports (from the <em>first</em> f to the <em>last</em> one l) along with a Duration to use as a timeout. We’ll call this function Start, as it will kick-off the actual connection attempts later on.</p><pre><strong>func</strong> (ps *PortScanner) Start(f, l int, timeout time.Duration) {<br>    wg <strong>:=</strong> sync.WaitGroup{}<br>    <strong>defer</strong> wg.Wait()<br>    <br>    <strong>for</strong> port <strong>:=</strong> f; port <strong>&lt;=</strong> l; port<strong>++</strong> {       <br>        wg.Add(1)<br>        ps.lock.Acquire(context.TODO(), 1)</pre><pre>        <strong>go func</strong>(port int) {<br>            <strong>defer</strong> ps.lock.Release(1)<br>            <strong>defer</strong> wg.Done()<br>            ScanPort(ps.ip, port, timeout)<br>        }(port)<br>    }<br>}</pre><h4>Limiting Go Routines for Performance</h4><p>While it is be easy to just spawn off a new go routine for every connection attempt, this isn’t the most efficient way of utilizing our computer’s resources. Instead, it’s much better to bound our connections attempts to a set size based on our shell’s limits. The lock in our port scanner “locks down” the concurrent connection attempts, and thus wastes significantly less time by not trying to exceed its threshold.</p><h3>The Main Function</h3><p>All that’s left is to write our program’s main function to use the code we just implemented to scan all of the ports on our local host.</p><pre><strong>func</strong> main() {<br>    ps <strong>:=</strong> <strong>&amp;</strong>PortScanner{<br>        ip:   &quot;127.0.0.1&quot;,<br>        lock: semaphore.NewWeighted(Ulimit()),<br>    }</pre><pre>    ps.Start(1, 65535, 500<strong>*</strong>time.Millisecond)<br>}</pre><h3>🔨 <strong>Compiling the Program</strong></h3><p>Compiling our port scanner is as trivial as running the following command:</p><pre><strong>$</strong> go build <a href="https://gist.github.com/picatz/9c0028efd7b3ced3329f7322f41b16e1.go">port_scanner.go</a></pre><p>Which provides us with a binary which we can execute to perform our port scan now:</p><pre><strong>$</strong> ./port_scanner</pre><p><em>But, how fast does it </em><strong><em>go</em></strong><em>?</em> Let’s use the time command on unix-like systems to give us a quick benchmark for an idea:</p><pre><strong>$</strong> time ./port_scanner<br>...<br>real    0m<strong>31.384s</strong><br>user	0m4.766s<br>sys	0m7.653s</pre><p>Let’s do the math on that: 65535 ports / 31 seconds is about 2000 ports a second. That’s incredibly fast, and <a href="https://medium.com/@KentGruber/building-a-high-performance-asynchronous-port-scanner-with-ruby-f2a335a3d44">2x faster than the single-threaded Ruby version I’ve made before</a>! 🚀</p><h3>📃 Source Code</h3><p>Below is the full source code for the scanner (copy+paste friendly for testing):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23c2c3b7f92d6c4ced75df99c4aada22/href">https://medium.com/media/23c2c3b7f92d6c4ced75df99c4aada22/href</a></iframe><h3>👋 Conclusion</h3><p>The multi-threaded runtime in Golang coupled with an asynchronous design allows for some incredible performance for a simple port scanner. Feel free to hack on it some more or let me know where it could be improved.</p><p>Until next time, that’s all folks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9976181ec39d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a High Performance, Asynchronous, Port Scanner with… Ruby?]]></title>
            <link>https://medium.com/@KentGruber/building-a-high-performance-asynchronous-port-scanner-with-ruby-f2a335a3d44?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2a335a3d44</guid>
            <category><![CDATA[infosec]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[asynchronous]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[how-to]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Fri, 08 Jun 2018 16:13:35 GMT</pubDate>
            <atom:updated>2018-06-09T19:25:06.763Z</atom:updated>
            <content:encoded><![CDATA[<p>Asynchronous, high performance, <em>and</em> Ruby — you might not think that makes a lot of sense. However, the <a href="https://github.com/socketry/async"><strong>async</strong></a> Ruby future is here; and it’s <strong><em>fast</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iosc8K07qzfPUg634vJ3tQ.png" /><figcaption>Sadly couldn’t find the original link to this piece of original art to credit the artist directly.</figcaption></figure><p>To see just how fast, let’s try building an <em>asynchronous</em> <a href="https://en.wikipedia.org/wiki/Port_scanner">port scanner</a>; meaning a port scanner that can look for many open ports at a time. To qualify as “<em>fast</em>”, at least for the sake of this blog post, means it should be able to scan hundreds of ports a second.</p><p><em>Just how do we do that?</em></p><h3>🔧 Single-Threaded Design</h3><p>The first option you might jump to in order to increase performance in languages like Ruby or Python is <em>usually</em> multi-threading. This isn’t the absolute worst choice but quickly becomes a bottle-neck in and of itself due to a <a href="https://en.wikipedia.org/wiki/Global_interpreter_lock">global lock</a> these languages enforce on their runtime ensuring only one thread runs at a time, even if you have many.</p><p>Effectively, this means your “multi-threaded” programs are really more like a confusing single-threaded program — sometimes actually performing <em>worse </em>than their truly single-threaded variants. If this is the first time you’ve heard this, it can be kind’of tricky to really understand.</p><p>The main advantage of using multi-threading is providing a means to deal with real world situations like network or disk I/O (input/output) operational latency. So, programmers have come up with abstractions to deal with this. With multi-threading APIs in Ruby or Python, we have an abstraction as a programmer to splice up our work into time segments. On I/O blocking operations, the language will decide that another thread can run, usually providing a speed boost for things like varying network connection speeds. This doesn’t come cheap though.</p><p>Threads will often end up “fighting” each-other for control of the runtime causing CPU context-switching which eats up your performance. This can be a really annoying thing to wrap your head around and provide a rather erratic runtime behavior that is a pain to debug or code complex interactions.</p><p>To avoid the headache of orchestrating multiple threads, we will be building a single-threaded port scanner. But, like in a multi-threaded program, we still want to be able to weave together tasks <strong><em>asynchronously</em></strong>.</p><blockquote>If a program can be broken up into a series of tasks that can be executed independently, it can be considered asynchronous. It is a property of how a program can be executed, not what it does. —<a href="https://www.codeotaku.com/journal/2018-06/asynchronous-ruby/index"> Samuel Williams</a></blockquote><h3>⚡️ Modern Async Support for Ruby</h3><p>For sometime now, <a href="https://github.com/ioquatix">Samuel Williams</a> and <a href="https://gitter.im/socketry/async">friends</a> have been working away on bringing modern, asynchronous and timeout-capable <a href="https://en.wikipedia.org/wiki/Input/output">I/O</a> for Ruby to the masses under the <a href="https://github.com/socketry"><strong>Socketry</strong></a> project.</p><h3>Yukihiro Matsumoto on Twitter</h3><p>link: socketry/async: Composable asynchronous I/O for Ruby.: https://t.co/4r0cuf3ZA7</p><blockquote><strong>Side Note:</strong> You can read more about it <a href="https://www.codeotaku.com/journal/2018-06/asynchronous-ruby/index">here</a>.</blockquote><p>Using <a href="https://github.com/socketry/async-await">async-await</a> will provide us with a clean API to describe asynchronous tasks. To ensure our network connections are asynchronous, we will use <a href="https://github.com/socketry/async-io">async-io</a>. Using async Gems within the Socketry project provides a plethora of options to build <a href="https://en.wikipedia.org/wiki/Non-blocking">non-blocking I/O</a> programs with great performance. The underlying design of async is based on a <a href="https://en.wikipedia.org/wiki/Reactor_pattern">Reactor Pattern</a> for concurrency backed by <a href="https://ruby-doc.org/core-2.5.0/Fiber.html">Fibers</a>.</p><blockquote><strong><em>Side Note:</em></strong> Another project you might want to check out is <a href="https://github.com/socketry/falcon"><strong>Falcon</strong></a>, a <a href="https://www.codeotaku.com/journal/2018-06/improving-ruby-concurrency/index">high performance</a> asynchronous multi-process web-server built on async libraries.</blockquote><h3>👩🏽‍💻 Implementing the Port Scanner with Async</h3><p>To get started, let’s write the first part of the script to <strong>require</strong> the async libraries we want to use.</p><pre><strong>require</strong> &quot;async/await&quot;<br><strong>require</strong> &quot;async/io&quot;<br><strong>require</strong> &quot;async/semaphore&quot;</pre><pre><em># more code ...</em></pre><p>Now, let’s create a <em>PortScanner</em> class, including the libraries within it to provide clean async semantics and create our <a href="http://ruby-for-beginners.rubymonstas.org/writing_classes/initializers.html">#initialize</a> method:</p><pre><em># previous code ...</em></pre><pre><a href="http://ruby-for-beginners.rubymonstas.org/writing_classes/definition.html"><strong>class</strong></a> <em>PortScanner<br>  </em><strong>include</strong> Async::Await<br>  <strong>include</strong> Async::IO</pre><pre><strong>  def</strong> <a href="http://ruby-for-beginners.rubymonstas.org/writing_classes/initializers.html">initialize</a>(<strong>host:</strong> &quot;127.0.0.1&quot;, <strong>ports:</strong> (1..1024))<br>    <strong>@host</strong>      = host<br>    <strong>@ports</strong>     = ports<br>    <strong>@semaphore</strong> = <em>Async::Semaphore</em>.new(<strong>`</strong><a href="http://wiki.linuxquestions.org/wiki/Ulimit">ulimit</a> -n<strong>`</strong>.<a href="https://apidock.com/ruby/String/to_i">to_i</a>)<br>  <strong>end</strong></pre><pre>  <em># more code...</em><br><strong>end</strong></pre><p>By default, if no <strong>host:</strong> argument is given when creating a new instance of our scanner, our localhost will be the target to scan. If not given any <a href="https://ruby-doc.org/core-2.5.0/Range.html">Range</a>/<a href="https://ruby-doc.org/core-2.5.0/Array.html">Array</a>/<a href="https://ruby-doc.org/core-2.5.1/Enumerable.html">Enumerable</a> for <strong>ports:</strong> , then ports <a href="https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html">1...1024</a> will be used, very similar to <a href="https://nmap.org/">nmap</a>.</p><p>The <a href="https://en.wikipedia.org/wiki/Semaphore_(programming)"><strong>@semaphore</strong></a> will be used to lock down the number of scans going on at a time to whatever the max number of available file descriptors are on the system. For every open connection, there’s an open file on the system, and we want to limit this to what <a href="http://wiki.linuxquestions.org/wiki/Ulimit">ulimit</a> tells us that is, at least specifically on unix-like operating systems including macOS or Linux.</p><blockquote><strong>Note: </strong><a href="http://wiki.linuxquestions.org/wiki/Ulimit">ulimit</a><strong> </strong><em>is a </em><a href="http://wiki.linuxquestions.org/wiki/Builtin"><em>builtin</em></a><em> shell command used to manage various resource restrictions. The </em><em>-n flag will reveal the maximum number of open files allowed; </em><em>#to_i on the output from that command in the code will turn it into a </em><a href="https://ruby-doc.org/core-2.5.0/Integer.html">Integer</a>. We cache this result and use it to set the maximum number of open connections we’ll try at a time, since there’s no reason to push past that threshold.</blockquote><p>Now, let’s implement a method which can scan a single port in our <em>PortScanner</em> class:</p><p>⚠️ This is the code to scan a <em>single</em> port. Moreover, this is will be the most complicated part of the code because of all the things we need to <em>juggle</em>, which I’ll explain in a bit more detail in just a moment.</p><pre># inside of PortScanner, under initialize ...</pre><pre><strong>def</strong> scan_port(port,<strong> timeout:</strong> 0.5)<br>  timeout(timeout) <strong>do<br>    </strong><em>Async::IO::Endpoint</em>.tcp(<strong>@host</strong>, port).connect <strong>do</strong> |peer|<br>      peer.close<br>      puts &quot;#{port} open&quot;<br>    <strong>end<br>  end<br>rescue</strong> <em>Errno::ECONNREFUSED, Async::TimeoutError</em><br>  puts &quot;#{port} closed&quot;<br><strong>rescue</strong> <em>Errno::EMFILE</em><br>  sleep timeout<br>  <strong>retry<br>end</strong></pre><pre><em># more code...</em></pre><p>To break down what’s happening, the method #scan_port will attempt to connect to the target on the given port — I know, that’s the obvious part.</p><p>If any error occurs —<em> and that’s going to happen</em>— we can <a href="http://rubylearning.com/satishtalim/ruby_exceptions.html"><strong>rescue</strong></a> that method and do something based on the cause of the error, right inside of the method itself.</p><p>🤔 <em>Why do we do this again?</em></p><p>During a port scan, especially for very large ranges, say thousands, or tens of thousands, our scanner may try to open up a connection, but — due the the limitations of our operating system — we may not be able to. We could be browsing <a href="https://twitter.com/KentGruber">Twitter</a> or <a href="https://github.com/picatz">GitHub</a> while our scanner runs super large batches after-all. Remember ulimit and the <strong>@semaphore</strong> from earlier inside of #initialize?</p><h4>🤹🏻‍ Three Essential Errors to Juggle:</h4><ul><li>🛑<em>Errno::ECONNREFUSED</em> is an error that will occur when a port is not open to connect to. This is is a pretty typical response for most ports, and this response that is enforced with most <a href="https://serverfault.com/questions/157375/reject-vs-drop-when-using-iptables">REJECT</a> firewall rules.</li><li>⏱<em>Async::TimeoutError</em> is an error produced by timeout(timeout) when an operation didn’t complete within the allotted <strong>timeout:</strong> value ( default of 0.5 ). This may happen for various reasons including a <a href="https://serverfault.com/questions/157375/reject-vs-drop-when-using-iptables">DROP</a> firewall rule where the target doesn’t explicitly tell you a port isn’t open from the scanner’ perspective. We really don’t want to wait around forever to get nothing.</li><li>📑<em>Errno::EMFILE</em> is an error that will occur when we try to connect to a port, but we have way too many connections (files, really) already open and need to schedule the execution to happen again, but just a little later, continuously delaying its execution until it’s able to be processed.</li></ul><p>Keeping track of the appropriate errors helps prevent missing open ports on larger scan ranges and allows us to handle things according to the error. In certain situations, multi-threaded port scanners can end up missing open ports simply because the scanner is counting <em>any and all</em> errors as a “closed port” when it probably should be retrying the ports that error’d out because of the operating system’s limitations. This goes for basically every port scanning implementation I’ve created and/or used.</p><p>In order to scan all the ports the <em>PortScanner</em> class is initialized with, let’s implement a #start method which will iterate over each port and calls the #scan_port method we just created. But, this time we’ll use the <strong>async</strong> keyword in-front of the method definition.</p><pre># inside of PortScanner, under scan_port ...</pre><pre><strong>async def </strong>start(timeout: 0.5)<br>  <strong>@ports</strong>.map <strong>do</strong> |<strong>port</strong>|<br>    <strong>@semaphore</strong>.async <strong>do</strong><br>      scan_port(port, timeout: timeout)<br>    <strong>end</strong><br>  <strong>end</strong>.collect(<strong>&amp;:</strong>result)<br><strong>end</strong></pre><p>Within the #start method, each port in the range that the PortScanner is initialized with will be scanned, running asynchronously — and resource bound to the operating system’s limitations with the <strong>@semaphore</strong>. The #collect(<strong>&amp;:</strong>result) near the very end of the method will allow us to collect the result of each port scan attempt.</p><h3>⚙️ Using the Scanner</h3><p>To use the port scanning API we just created could look something just like the following:</p><pre><em># previous code ...</em></pre><pre><em>PortScanner</em>.new(<strong>host:</strong> &quot;127.0.0.1&quot;, <strong>ports:</strong> (1..1024)).start</pre><p><em>There aren’t any open ports to test with?</em> Using <a href="https://en.wikipedia.org/wiki/Netcat">nc</a>, it’s pretty easy to create some open ports. The following will setup a <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a> listener on your localhost on port 1024:</p><pre><strong>$</strong> nc -v -l -p 1024 </pre><p>Now, let’s run our port scanner, using <a href="https://en.wikipedia.org/wiki/Grep">grep</a> to filter for the open ports:</p><pre><strong>$</strong> ruby async_port_scanner.rb <strong>|</strong> grep &quot;open&quot;<br>1024 open</pre><p>💥<strong><em>Bam!</em></strong> Just like that, we have a port scanner! <em>How fast does it go? </em>Using <a href="https://en.wikipedia.org/wiki/Time_(Unix)">time</a> on Unix-like operating systems can help us get an idea:</p><pre><strong>$</strong> time ruby async_port_scanner.rb <strong>|</strong> grep &quot;open&quot;<br>1024 <strong>open</strong></pre><pre>real 0m<strong>2.345s</strong><br>user 0m0.531s<br>sys  0m0.130s</pre><p>That’s roughly 2–3 seconds to scan all those ports (1024 of them ) on a single thread! What if we wanted to scan all the ports (1…65535), how long would that take?</p><pre>real <strong>1m12.693s</strong><br>user 1m5.933s<br>sys  0m5.157s</pre><p>So, <em>if I can do the calculation right</em>: 65535 ports/(1min + 12secs = 72secs) would be roughly ~910 ports a second. That’s pretty fast! 🚀</p><h3>📃 Source Code</h3><p>Below is the full source code for the scanner (copy+paste friendly for testing):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1907906e10bf76115e85eaae7b8e104/href">https://medium.com/media/e1907906e10bf76115e85eaae7b8e104/href</a></iframe><h3>👋 Conclusion</h3><p>Building a high performance, asynchronous port scanner with Ruby is actually pretty awesome, and very possible! By embracing single threaded performance backed by <a href="https://ruby-doc.org/core-2.5.0/Fiber.html">Fibers</a> provides really interesting opportunities, and great performance for I/O-bound operations ( like a port scanner ) that should only get better over time.</p><p>If you like what you see, be sure to give the <a href="https://github.com/socketry">Async/Socketry gem(s)</a> a star 🌟on <a href="https://github.com/socketry"><strong>GitHub </strong></a>— or, make a pull request to make them better! If you see where I did something wrong, want to improve the implementation, or have any questions, feel to chat on <a href="https://gitter.im/socketry/async"><strong>Gitter</strong></a>!</p><p>Until next time, that’s all folks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2a335a3d44" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[雲道場 Kumo Dōjō: Terraform, Docker and a Slackbot]]></title>
            <link>https://medium.com/@KentGruber/%E9%9B%B2%E9%81%93%E5%A0%B4-kumo-d%C5%8Dj%C5%8D-terraform-docker-and-a-slackbot-91bbb77a5e4e?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/91bbb77a5e4e</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Mon, 12 Mar 2018 22:57:45 GMT</pubDate>
            <atom:updated>2018-03-12T22:57:45.859Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I’ve been curious about building cybersecurity infrastructure, focusing on competition practice environments. Using cloud technologies like Amazon Web Services (AWS) one can easily spin up infrastructure — so I decided to see what I could build on a Sunday.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7k0rlqYPe9opukS5ViFttw.png" /></figure><h4>AWS</h4><p>My main problem with AWS is the GUI — albeit a bearable experience when necessary, I’d much rather keep myself happily at the command-line. The <a href="https://github.com/aws/aws-cli">aws-cli</a> and <a href="https://github.com/awslabs/aws-shell">aws-shell</a> projects help with that quite a bit, but there’s another tool that’s really caught my eye: <a href="https://www.terraform.io/">terraform</a></p><h4>Terraform</h4><p>With terraform, one can easily describe and work on their infrastructure “as code” which provides neat opportunities for really helping automate the building of every piece comprising our infrastructure.</p><h4>Docker</h4><p>Where does <a href="https://www.docker.com/">docker</a> come into this? I love using docker in order to run practically any and all of my applications now, and today’s example is no exception. Specifically we will be using docker as a way for users to login to a ubuntu:16.04 flavored bash shell for them to practice administering a ubuntu box. Later it could be cool if we could orchestrate specific tasks that could be monitored on the host running docker allowing the users to gain points for completing a specific task. That will be out of scope for this blog post at least though.</p><p>One other interesting about about Docker is the health-checks which provide a really convenient way of ensuring your instance is relatively healthy with something as simple as a shell command that can run every (I dunno, like) 10 seconds. Instances can easily be setup so that they’ll continuously restart on failure — which can be good and bad. Either way, there’s some really nice tooling in a pretty decent command-line DSL ( if docker-compose isn’t your thing at least ).</p><h4>Slackbot</h4><p>While probably any chat-ops platform would do, my team(s) tend to like to use <a href="https://slack.com/">slack</a>. So, building slackbots is common for all sorts of weird things to add to the team’s workflow.</p><h4>authorized_keys</h4><p>When using <a href="https://en.wikipedia.org/wiki/Secure_Shell">ssh</a>, the ~/.ssh/authorized_keys file is generally used for just storying your public ssh keys that are able to logon to the server. There’s some special syntactic sugar that is supported here to do things like specify a command that linked to that ssh key making it only able to run that single command.</p><pre>command=&quot;echo &#39;Hello there!&#39;&quot; ssh-rsa AAAAB3NzaC1yc2EAAAADAQ...</pre><p>Using the command=&quot;&quot; syntax, I can dole out private ssh keys to users in my team with a slackbot to allow them access, limiting what they can do ( getting inside a docker container ) and for monitoring / inevitable abuse attempt correlation things later.</p><h3>Building the Slackbot ( 雲道場 Kumo Dōjō )</h3><p>I’m sure this could be done better. Maybe I’ll even fix it up sometime. But, for the sake of prototyping and enjoying making this for the hell of it on a Sunday, I think this is awesome:</p><p>For whatever reason I’ve called the slackbot the “<em>Kumo Dojo</em>” ( Cloud Dojo ):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ceed35f26e1d04537b6eed300f3bb97/href">https://medium.com/media/5ceed35f26e1d04537b6eed300f3bb97/href</a></iframe><p>The following is the Dockerfile used:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/853a5bf7103772fddadd6e4c52e33c27/href">https://medium.com/media/853a5bf7103772fddadd6e4c52e33c27/href</a></iframe><p>With the Dockerfile we can build and run the application. But, first we need to setup our host on AWS.</p><h3>Using Terraform to build a Ubuntu Box with Docker and our Slackbot Provisioned</h3><p>Now that we have our Slackbot, let’s throw it up on the cloud. I’ll be using a simple t2.micro AWS instance with Ubuntu 16.04 and then it’ll be provisioned will all the things we need:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb5db4a3043338eddaa2a9abf1a7c6da/href">https://medium.com/media/eb5db4a3043338eddaa2a9abf1a7c6da/href</a></iframe><p>Now we just need to terraform -auto-approve and within roughly 3 minutes we should have a slackbot that provides a way to ssh into our freshly made AWS instance into their own isolated Docker container to start practicing setting up services and installing packages.</p><p>When they’ve messed up or don’t know what happened, they can just exit their ssh session and re-connect to get a fresh container to start from scratch.</p><p>This provides a really interesting jumping-off point for working on expanding the slackbot or really just gamifying the whole experience with a points system and perhaps different challenges that could orchestrated all through docker and some ssh key magic.</p><p>Until next time, that’s all folks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91bbb77a5e4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding which(1)]]></title>
            <link>https://medium.com/@KentGruber/understanding-which-1-9e0e2446ff83?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e0e2446ff83</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Sun, 21 Jan 2018 20:04:23 GMT</pubDate>
            <atom:updated>2018-01-21T20:04:23.541Z</atom:updated>
            <content:encoded><![CDATA[<p>To lookup the full path of a shell command, you’ll probably use <a href="https://en.wikipedia.org/wiki/Which_(Unix)">which</a>. But, how does it actually work?</p><p>To find that out I decided to build my own, analyze, and just look at the history of this interesting little command-line application that I’ve used for years, but never really understood what it did.</p><p>Consulting the <a href="https://en.wikipedia.org/wiki/Man_page">man</a> pages is probably a good first step for anyone looking to get <em>a little bit</em> more information about a particular command. So, that’s what I did; and I stumbled across this interesting bit of trivia — at least hidden within the macOS version of which:</p><pre><strong>HISTORY<br></strong>The which command first appeared in FreeBSD 2.1.</pre><pre><strong>AUTHORS<br></strong>The which utility was originally written in Perl and was contributed by Wolfram Schneider &lt;wosch@FreeBSD.org&gt;.  The current version of which was rewritten in C by Daniel Papasian &lt;dpapasia@andrew.cmu.edu&gt;.</pre><p>The version of which on macOS also only supports two command-line flags: -a ( for all found paths, not just the first ) and -s ( for no output ).</p><p>If we were to look, for example, a CentOS 7 box, we’d see something quite different: flags like --skip-dot, --show-dot, --read-alias and plenty more that would be tedious to go over I assure you, but are very cool nonetheless! If you’re on Ubuntu you’ll see that which only supports -a out of the box.</p><p>From what I’ve been able to tell, -a is the only flag supported by most unix like operating systems for which that you can even rely on.</p><p>I had no idea which supported so many different command-line flags depending on the operating system you’re working on. There’s even a windows port or two out there that exist.</p><p>But, I still haven’t answered: how does it work? Well, the macOS man page actually spells it out pretty clearly:</p><pre>which -- locate a program file in the user&#39;s path</pre><p><strong>To clarify further: </strong>What they mean by the “<em>user’s path</em>” is the $PATH environment variable of the shell.</p><h3>Searching for cats in your $PATH</h3><p>You see, when working in the context of a shell and you type in:</p><pre><strong>$</strong> cat cool.txt</pre><p>If felt so inclined, we could call the cat program directly using its full path, wherever it is on your system:</p><pre><strong>$</strong> /bin/cat cool.txt</pre><p>Our system can find the full path to cat to call without us having to know the full path to the cat file. This is a feature that is widely used and easily overlooked, at least in terms of its implementation details.</p><p>Unix-like shells have a $PATH environment variable ( amongst many other environment variables, which can even be manipulated by the user ) : and when we try to run something, our shell can go looking for it in all of the directories available in $PATH — that can be any number of directories depending on your system and what you’ve installed.</p><p>On a Ubuntu <a href="https://en.wikipedia.org/wiki/Vagrant_(software)">vagrant</a> box that I spun up, the default $PATH variable was this:</p><pre>/home/ubuntu/bin:/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin</pre><p>To make that a bit cleaner, we could split up the available directories by the “:“ character in whatever programming of choice you’d like, I’ll use <a href="https://en.wikipedia.org/wiki/AWK">awk</a> :</p><pre><strong>$</strong> echo $PATH | awk &#39;BEGIN{RS=&quot;:&quot;}{$1=$1}1&#39;<br>/home/ubuntu/bin<br>/home/ubuntu/.local/bin<br>/usr/local/sbin<br>/usr/local/bin<br>/usr/sbin<br>/usr/bin<br>/sbin<br>/bin<br>/usr/games<br>/usr/local/games<br>/snap/bin</pre><p>All of those paths could have files stored in them that the shell could use. Moreover, order matters in many cases ( even with the which command )! So, fun-fact: this can be a really nasty thing to mess with.</p><h3>Brewing up our own which(1)</h3><p>With all the inconsistencies of which that I discovered, I figured I could make my own! If all these other people made their own, with their weird flavor added to it — I could do the exact same thing and make it weirder!</p><p>I’ll make one that, basically, always runs the -a flag to show all available paths.</p><p>We could arbitrarily pick any language for this task really. Ruby, Python, Crystal, JavaScript — but I ended up implementing this in Go!</p><p>With Go, I can easily build many cross-platform tools that can compile to a neat little binary for whatever system you’d like: mac, linux, windows — easy!</p><p>The first thing we can do is write a simple function that’ll give us a <a href="https://tour.golang.org/moretypes/7">slice</a> of strings much like we did earlier with echo and a bit of awk :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4f0cc5cd0f547a4c975da980196992f/href">https://medium.com/media/e4f0cc5cd0f547a4c975da980196992f/href</a></iframe><p>We’ll also need to be able to check if the program we’re looking for exists in the path at some point:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/84dbbb86568029a99dc0bd418ad8d0d5/href">https://medium.com/media/84dbbb86568029a99dc0bd418ad8d0d5/href</a></iframe><p>Then we’re going to want loop over the possibilities and then we’ll go ahead and consume the results through a channel ( which I’ll call “messages” ):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e78d1b722c4ea8f877c234d21a36980/href">https://medium.com/media/9e78d1b722c4ea8f877c234d21a36980/href</a></iframe><p>Now, within our main() function, we just need to grab the content from the command-line, check it; and then pass the arguments as a slice to our FindInPath() function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5fb21ed023a4dace506136e5313e8a0/href">https://medium.com/media/e5fb21ed023a4dace506136e5313e8a0/href</a></iframe><p>All together, we have our full program!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c21dc4dff45b0aad13c0c9daaaf0ac0/href">https://medium.com/media/6c21dc4dff45b0aad13c0c9daaaf0ac0/href</a></iframe><p>In just over 50 lines of code, we’ve implemented a working example of the which command-line utility ( that <strong><em>always</em></strong> runs the -a option )!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*zVStHikOlB6ti3Ap_o4o-Q.gif" /><figcaption>Just imagine, with a bit more code — we could even implement the other flags!</figcaption></figure><p>Moving forward when I find the time, I think I’d like to implement the rest of the flags and make it fully compatible with the actual which command-line utility; and hopefully support windows? That’d be really neat.</p><p>So, I think that’s where I’m going to end this post — hope you learned something!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e0e2446ff83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Packet Analysis in Color]]></title>
            <link>https://medium.com/@KentGruber/packet-analysis-in-color-860597310149?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/860597310149</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[crystal-lang]]></category>
            <category><![CDATA[information-security]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Sat, 02 Dec 2017 20:09:18 GMT</pubDate>
            <atom:updated>2017-12-02T20:09:18.077Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine <a href="https://en.wikipedia.org/wiki/Network_packet">network packets</a> flying in the air or running along your computer’s ethernet cable — <em>what </em><strong><em>color</em></strong><em> are they</em>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ba-BQx9OB5NSFEzMxLi4pw.jpeg" /><figcaption>Maybe, something like this?</figcaption></figure><p>To see a possible spectrum of <a href="https://en.wikipedia.org/wiki/8-bit_color#/media/File:256colour.png">256 colors</a> available to us in our packets, we could write a tiny bit of <a href="https://crystal-lang.org/">Crystal</a> code to use <a href="https://github.com/the-tcpdump-group/libpcap">LibPcap</a> to capture them live and represent each byte (8-bits) within it as an individual block of color on the <a href="https://en.wikipedia.org/wiki/Command-line_interface">command-line</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/59801e30fdc99be4e3cda562b85b9d45/href">https://medium.com/media/59801e30fdc99be4e3cda562b85b9d45/href</a></iframe><p>You will need to have <a href="https://github.com/picatz/packetz">Packetz</a>, a network packet capturing library built with <a href="https://github.com/the-tcpdump-group/libpcap">LibPcap</a>, included in your program’s <a href="https://github.com/crystal-lang/shards">shard.yml</a> file:</p><pre><strong># example shard.yml</strong><br><strong>name:</strong> color_live_packets<br><strong>version:</strong> 0.1.0<br>  <strong>dependencies:</strong><br>    <strong>packetz:</strong><br>      <strong>github:</strong> picatz/packetz</pre><p>To install the dependencies now with the shard.yml file, just run:</p><pre>$ <strong>shards install</strong></pre><p>Now we can simply compile the program with crystal :</p><pre>$ <strong>crystal</strong> <strong>build</strong> color_live_packets.cr <strong>--release</strong></pre><p>We now have a binary that we can run to see network packets in <a href="https://en.wikipedia.org/wiki/8-bit_color#/media/File:256colour.png">256 colors</a> on the command-line to help visualize patterns and provide a different insight into what’s going on with our network traffic.</p><pre>$ <strong>./color_live_packets</strong><br>██████████████████████████████████████████████████████████████████^C<br>Stopping!</pre><p>In 256 color:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ba-BQx9OB5NSFEzMxLi4pw.jpeg" /></figure><p>You can obviously tune the code to your needs to improve it for your situation. I just wanted to be able to do this — and with Crystal, it was incredibly easy to build my own simple packet analysis command-line tool.</p><p>I think this really just provides a great point to get started with other ways of coloring packets.</p><h4>GitHub Code</h4><p>This blog post’s code has been adapted from a command-line tool I wrote called <a href="https://github.com/picatz/niji">niji</a> on GitHub which essentially does exactly the same thing — but you can specify a network interface to listen on! Options, ‘yo!</p><h4>More Crystal</h4><p>To learn more about Crystal, you can check out the project’s <a href="https://github.com/crystal-lang/crystal">GitHub</a>, <a href="https://crystal-lang.org/docs/">Documentation</a>, <a href="https://twitter.com/crystallanguage">Twitter</a>, or even the <a href="https://gitter.im/crystal-lang/crystal">Gitter</a> chatroom! I think you’ll find a wonderful, helpful community that is doing really awesome things with a beautiful language.</p><h3>Closing Thoughts</h3><p>Maybe I should make an application with electronJS or something to visualize packets not just on the command-line. Like a rainbow <a href="https://www.wireshark.org/">Wireshark</a>. Rainbow shark. 🌈 🦈</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*BivnuZyfCZOJpBLqg62MOQ.gif" /><figcaption>I promise there’s applications to this color packet analysis practice other than just rainbow vomit. 😂</figcaption></figure><p><strong>Are there applications for this practice?</strong> I <a href="https://www.youtube.com/watch?v=pWdd6_ZxX8c"><em>think</em></a> there very well could be. Specifically in helping determine patterns visually outside of simply parsing packets into a known structure like Wireshark or <a href="https://github.com/sdaubert/packetgen">PacketGen</a> does. I think is really more of an aide to traditional packet analysis than a replacement.</p><p>It’s also <em>really</em> cool to look at.</p><p><em>Until next time, that’s all folks!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=860597310149" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VMware on the Command-line]]></title>
            <link>https://medium.com/@KentGruber/vmware-on-the-command-line-b9a28a5aa16c?source=rss-67a458d49f6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9a28a5aa16c</guid>
            <category><![CDATA[vagrant]]></category>
            <category><![CDATA[vmware]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[virtualization]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Kent Gruber]]></dc:creator>
            <pubDate>Sat, 11 Nov 2017 19:11:20 GMT</pubDate>
            <atom:updated>2017-11-11T19:11:20.213Z</atom:updated>
            <content:encoded><![CDATA[<p>When first introduced into the world of V<a href="https://en.wikipedia.org/wiki/Virtual_machine">irtual Machines</a>, one might find themselves working with the popular VM solution <a href="https://www.vmware.com/">VMware</a>.</p><p>Practically every intro book will show you how to poke around the <a href="https://en.wikipedia.org/wiki/Graphical_user_interface">GUI </a>to start, stop, and generally manage the various different VMs you have running.</p><p>This is fine, but not ideal for everything.</p><p>Compared to the workflow that tools like <a href="https://www.vagrantup.com/">Vagrant</a> provide, I tend to find VMware to be far less command-line workflow focused. What I mean by this is: it’s typical that you <em>won’t</em> find many people scripting their VMware workflow… or genuinely taking an absurd amount of pride in manually doing this tedious work. I’ve seen this in the classroom so many times, <em>it hurts.</em></p><p>It hurts because I really like automation. No. <strong><em>I love automation.</em></strong></p><p>So, while I understand Vagrant and VMware<strong><em> are not</em></strong> competing products ( <a href="https://www.vagrantup.com/vmware/index.html">you can even use VMware with Vagrant</a> ) — I wanted to bring that Vagrant-like, scriptable workflow to my VMware experience <em>without </em>even using Vagrant.</p><p>Why not, right?</p><h3>vmrun</h3><p><em>No one</em> ever told me about the <a href="https://www.vmware.com/support/developer/vix-api/vix112_vmrun_command.pdf"><strong>vmrun</strong></a> command. I wish someone had. Because it has quickly become one of my most-used commands. On your specific system, you will need to find where the command is installed to use it.</p><p>I’ve opted to make a simple <a href="http://tldp.org/LDP/abs/html/aliases.html">BASH alias </a>( on <a href="https://en.wikipedia.org/wiki/MacOS">macOS</a>, in my case ) to use it without referencing its full path. The <a href="https://www.vmware.com/support/developer/vix-api/vix112_vmrun_command.pdf">official documentation</a> has support for Linux, macOS and Windows on how to find and use the vmrun command.</p><p>alias vmrun=&quot;/Applications/VMWare\ Fusion.app/Contents/Library/vmrun&quot;</p><p>This command allows you to do all sorts of nifty things. Let’s go over just a few functions to get you started!</p><blockquote><strong><em>Note:</em></strong><em> some </em><em>vmrun commands require </em><a href="https://www.vmware.com/support/ws55/doc/new_guest_tools_ws.html"><em>VMware tools</em></a><em> or </em><a href="https://github.com/vmware/open-vm-tools"><em>open-vm-tools</em></a><em> to be installed on the VM. This </em>may not <em>be ideal for certain situations, but doesn’t hurt in many others. Others commands may require a username/password to interact with the VM.</em></blockquote><h4>Starting a VM (without a GUI)</h4><p>To simply start a VM you’ve provisioned through the GUI. All you will need to do is point to the <strong>.vmx</strong> file that exists in the VM’s directory contents. You’ll find this nestled along a bunch of other <em>.vmdk</em>, <em>.log</em> and <em>.vmxf</em> files wherever you saved your VM. To disable the GUI, simple add the nogui option on the same line.</p><p><strong>vmrun</strong> start /path/to/example.vmx nogui</p><h4>Listing Running VMs</h4><p>To a get list of the paths to the currently running VMs, you can use the list command option. When there are no VMs running, it’ll also let you know. This is a simple, awesome command and helpful for wrangling a bunch of VMs that are running without a GUI to stop’em.</p><p><strong>vmrun</strong> list</p><h4>Check VM Tools</h4><p>Maybe you don’t know if you’ve installed something like open-vm-tools and don’t want to use <a href="https://en.wikipedia.org/wiki/Yellow_Dog_Updater,_Modified">yum</a> or whatever on the VM to check it. The checkToolsState command is what you want.</p><p><strong>vmrun</strong> checkToolsState /path/to/example.vmx</p><p>When the VM tools are installed on the box, you’ll get a little message “installed” printed to <a href="https://en.wikipedia.org/wiki/Standard_streams">STDOUT</a> that lets you know you’re good to go.</p><p>What kind of things can we easily do with the VM when these tools are installed? <em>Seriously, just go read the documentation.</em></p><h4>List Running Processes</h4><p>You can lists all processes running in the VM with the listProcessesInGuest option. This requires VM tools and a valid guest login on the target VM. The login can be passed at the command-line or you can interactively type in the parameters as they are required at the command-line very simply.</p><p><strong>vmrun</strong> listProcessesInGuest /path/to/example.vmx</p><p>When prompted, type in your username and password. With valid credentials, this command will give you simple output with is formatted like so, separated by newlines for each process:</p><p>pid = 31337, owner=user, cmd=/usr/bin/vmtoolsd</p><h4>What’s the IP Address?</h4><p>You’ve got a VM running. Cool, but, you don’t know the IP address because it was dynamically assigned with something like DHCP. Moreover, you’ve started the box without a GUI, <em>because you’re now a gangster that can do that </em>— so you can’t just login with the VMware graphical terminal you would normally use to run <a href="https://en.wikipedia.org/wiki/Ifconfig">ifconfig</a>. The getGuestIPAddress command is what you need to quickly do this.</p><p><strong>vmrun</strong> getGuestIPAddress /path/to/example.vmx</p><p>This’ll spit back out the first IP address available on that box, or will error out on you if you don’t have VM tools installed. To ensure this is the IP address of the interface you can SSH into, for example, you’ll want to setup the first interface on that box to be a shared network interface between you guest and host systems. This is just one way to do this.</p><h4>Stopping the VM</h4><p>We’ve done all the work we needed to do for the day on our VM and we’re ready to shut it down. Remember the start command? It’s very similar, but stop instead. You <em>won’t</em> need to specify if the VM started with or without a GUI to keep it simple.</p><p><strong>vmrun</strong> stop /path/to/example.vmx</p><h3>Conclusion</h3><p>Just like that, we’ve started to build up a pretty dynamic, scriptable VM experience that could be wrapped up in your <a href="https://crystal-lang.org/">favorite programming language</a> of <a href="https://www.ruby-lang.org/en/">choice</a> or used directly without much fuss.</p><p>I recommend looking into VMware’s products some more to learn all about its (<em>sometimes less documented</em>) command-line tools which you can use to make your lives easier and, of course, <em>automated!</em></p><p>Until next time, that’s all folks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9a28a5aa16c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>