<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Szymon Szypulski on Medium]]></title>
        <description><![CDATA[Stories by Szymon Szypulski on Medium]]></description>
        <link>https://medium.com/@szymon.szypulski_40581?source=rss-b0bc876177a9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*LHj0x08x7yXKq1YP.</url>
            <title>Stories by Szymon Szypulski on Medium</title>
            <link>https://medium.com/@szymon.szypulski_40581?source=rss-b0bc876177a9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:59:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@szymon.szypulski_40581/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Who watches the watcher — infrastructure monitoring heartbeat]]></title>
            <link>https://medium.com/ragnarson/who-watches-the-watcher-infrastructure-monitoring-heartbeat-ddf709f190b2?source=rss-b0bc876177a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddf709f190b2</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Szymon Szypulski]]></dc:creator>
            <pubDate>Thu, 01 Feb 2018 00:00:00 GMT</pubDate>
            <atom:updated>2018-02-01T12:17:59.100Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/144/0*UCQ4Y9sGxhlc6EVb." /></figure><p>February 1, 2018 — <a href="https://blog.ragnarson.com/authors/szymon-szypulski.html">Szymon Szypulski</a> in <a href="https://blog.ragnarson.com/categories/development.html">development</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*IM7tFpg4yxKJSiCG.png" /></figure><p>At Ragnarson we believe that monitoring every aspect of the infrastructure is crucial for a long-term maintenance of infrastructure projects. This is extremely important, it is unlikely for us to take on projects with no budget for the monitoring. But even if you set up alerts for every service, there is one aspect of the monitoring which is overlooked by many beginner engineers.</p><p>I have seen and heard many times about infrastructure failing together with monitoring. The app goes down, but the same issue that causes the app to fail also takes down the monitoring service. No one noticed until the customer called and asked why the website is down. It happened to me in the early days of my career. Now setting up additional guards for the monitoring is a mandatory feature of any monitoring system we set up.</p><p>This important topic is rarely discussed and difficult to find resources for using Google or any other search methods. In this blog post, I will discuss how you to prevent situations when your infrastructure is down and you do not notice.</p><p>There are two approaches, each has its own trade-offs. First is more expensive, but it is simpler to deploy. However, you may violate data protection laws choosing external monitoring provider. The second option will cost you less long-term, it is more complex and harder to deploy and you will have full control over your data location.</p><h3>First case — external monitoring</h3><p>Where a budget allows or time to deliver is short, we are using <a href="https://www.datadoghq.com/">DataDog</a>. It is all-in-one monitoring and alerting solution, it can monitor every aspect of your infrastructure.</p><p>It can be easily integrated with external services like <a href="https://www.pagerduty.com/">PagerDuty</a>. It can keep you awake if something bad happens to your infrastructure — you will hate it and love it at the same time.</p><p>Without getting too much into technical details, DataDog is an agent-based, push solution. This means, there is a process running on each server, it is gathering data from all the services every minute and sends them to DataDog HQ.</p><p>Below you can see the default ‘Process’ type alert. If there is no agent process reporting for a given host for longer than 2 minutes — trigger an alert. In our case, the alert will be passed to PagerDuty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*AgwmxXt5NT__BGpb.png" /></figure><p>I am cheating a little bit here and I have done nothing exceptional. DataDog has a built-in watchdog if it detects there is no new data.</p><p>If you are running critical systems and you have to be an extra paranoid you can also check DataDog <a href="https://status.datadoghq.com/">status page</a>. There is an option to subscribe to their status in various ways — phone, webhook, email, RSS.</p><p>If you use something else than DataDog. A tool with the status page from the past, without convenient notification options, you can always try to <a href="https://zapier.com/zapbook/pagerduty/rss/">integrate it over Zapier</a>.</p><h3>Self-hosted monitoring</h3><p>DataDog requires some hefty budget and you may be restricted by various national or health-related data laws. Or you simply prefer in-house solutions.</p><p>Internally, we are experimenting with an on-premise monitoring solution based on Prometheus. From our point of view, it seems to be the best tool suited for metrics-based monitoring and alerting. It has a big community <a href="https://prometheus.io/blog/2017/11/08/announcing-prometheus-2-0/">and version 2.0 was released recently. With some great improvements, especially in resource usage ‘department’.</a></p><p>In case of an in-house monitoring, there is no upstream status page, we are the single user of the service. If a monitoring node goes down you will not even know. If a service goes down (or is not operating as supposed to) there will be no alert.</p><p>I did not find any obvious agent check-ups, like in case of DataDog. So I have started with simple google searches for generic ideas how people may handle ‘monitoring the monitoring’. Picks from the top of my head were:</p><ul><li>monitoring heartbeat</li><li>verify monitoring system</li><li>ensure monitoring software is running</li><li>monitoring dead man switch (Do not ask me how I have connected monitoring with bomb trigger. This is not a search phrase which will be used by most of the people)</li></ul><p>Unfortunately, my google-fu was poor. Most of the results were about ‘real life’/heart monitoring. I have even learned a new trick — the <a href="https://searchresearch1.blogspot.se/2010/10/around-has-always-been-around.html">AROUND keyword</a>. However, in conjunction with various self-hosted monitoring solutions did not give me any good results. Only the last phrase gave me some ideas how to ensure my monitoring is up and running. Yet, it was still far from a ready-made solution.</p><p>I left the idea for a few days to grow on me. It was not urgent since we are just experimenting with Prometheus.</p><p>After a few days, I have stumbled on <a href="https://docs.google.com/presentation/d/1xMnqKHNfBS1DWcnMQ6S3iTS10_DMISoDfNt6If4ihe4/pub?start=false&amp;loop=false&amp;delayms=3000&amp;slide=id.g16d63954a7_0_68">this short five-step overview</a> on #monitoring at <a href="https://signup.hangops.com/">Hangops slack</a> (a highly recommended community). It describes how to configure <a href="https://prometheus.io/">Prometheus</a> with <a href="https://www.pagerduty.com/">PagerDuty</a> and <a href="https://deadmanssnitch.com/">DeadManSnitch</a>. It was the biggest eye-opener. The best approach to my issue I was able to find. I do not know who is the author, but he/she deserves a medal.</p><p>It is not my goal to copy the slides, see it by yourself. The basic idea applies almost to any on-premise monitoring. Fire an automatic alert periodically. If there are no alerts, our alerting software does not work, call on-duty team. Since we are using PagerDuty anyway it is almost a perfect fit. Of course, there is no ideal solution which covers all the corners, even this has some strong and weak points:</p><p>Advantages:</p><p>Disadvantages:</p><ul><li>You are adding another point of failure to the infrastructure.</li><li>Requires extra services, which adds a complexity and an implementation/maintenance cost.</li><li>If you run <a href="https://www.pagerduty.com/blog/failure-friday-at-pagerduty/">‘Failure Friday’</a>, it is another point on the checklist.</li><li>It is not HA, it may fail as well.</li></ul><p>Overall I think it adds a reasonable level of the complexity for what it gives. I would not go anywhere further — the watcher who watches the watcher who watches the watcher is enough ;-)</p><p>I hope now you have a better overview how to ‘watch the watcher’ in your infrastructure. Anyway, if something seems to be unclear, please leave us a comment. Yet, if you would like us to set it up for you, <a href="https://calendly.com/macias/30min/">give us a call</a> or <a href="https://blog.ragnarson.com/2018/02/01/sales@ragnarson.com">send us an email</a>.</p><p>At Ragnarson we help companies deliver great products. We take care of development and deployment so that they can focus on growing the product and working with customers.</p><p><a href="https://ragnarson.com/?utm_source=ragnarson&amp;utm_medium=blog&amp;utm_campaign=work-with-us#contact">Work with us</a></p><p><em>Originally published at </em><a href="https://blog.ragnarson.com/2018/02/01/who-watches-the-watcher.html"><em>blog.ragnarson.com</em></a><em> on February 1, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddf709f190b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ragnarson/who-watches-the-watcher-infrastructure-monitoring-heartbeat-ddf709f190b2">Who watches the watcher — infrastructure monitoring heartbeat</a> was originally published in <a href="https://medium.com/ragnarson">Ragnarson</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[InSpec — Inspect Your Infrastructure]]></title>
            <link>https://medium.com/ragnarson/inspec-inspect-your-infrastructure-18debd45f0db?source=rss-b0bc876177a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/18debd45f0db</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Szymon Szypulski]]></dc:creator>
            <pubDate>Mon, 13 Mar 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-03-14T17:59:06.490Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/144/0*FGQeLCDr39ZG3Hk4." /></figure><p>March 13, 2017 — <a href="https://blog.ragnarson.com/authors/szymon-szypulski.html">Szymon Szypulski</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f8FmS4Q7LkHw3ku4.png" /></figure><p>We don’t have many infrastructure related posts here. We have a few full-time DevOps, it’s a part of our normal offer. But, we’re busy most of the time. When an opportunity to write a blog post emerged, I knew I’ve had to write about a thing which changed our workflow. During last two years, there were few ideas which helped us providing better and more reliable services — one of them is infrastructure integration testing.</p><p><em>Title is stolen from </em><a href="http://inspec.io/"><em>inspec.io</em></a><em>, I hope they don’t mind.</em></p><p>Before we start, few notes:</p><ul><li>This is only an introduction to the idea of infrastructure integration testing.</li><li>It’s opinionated.</li><li>It’s based on my two years experience with <a href="http://serverspec.org/">Serverspec</a> and few weeks with InSpec.</li><li>By infrastructure, I mean everything, from your local workstation to your &gt;1024 machine cluster.</li><li>Some ideas in this blog post may be closer to acceptance or functional testing. I’m aware of this, it’s a thin line in some cases.</li><li>I will omit some InSpec features.</li><li>Tests are hard.</li></ul><h3>Why not unit testing?</h3><p>In our case, unit testing is strongly coupled with Chef and ChefSpec, which means this blog post would apply to the much narrower audience.</p><p>Besides, it’s easy to <a href="https://coderanger.net/overtesting/">overtest</a> in the case of unit tests, where only true logic should be verified.</p><p>Integration tests are closer to the infrastructure and I’ll focus on InSpec for now.</p><h3>What is InSpec?</h3><p><em>InSpec is an open-source testing framework for infrastructure with a human-readable language for specifying compliance, security and other policy requirements.</em></p><p>This sentence is borrowed from <a href="http://inspec.io/">inspec.io</a>. Is it true? Let’s start with short code sample:</p><pre>describe file(&#39;/etc/myapp.conf&#39;) do<br>  it { should exist }<br>  its (&#39;mode&#39;) { should cmp &#39;0644&#39; }<br>end</pre><pre>describe port(8080) do<br>  it { should be_listening }<br>end</pre><p>It consists two resources (file and port) with basic matchers verifying file existence, proper mode and listening port. Resources are classes which help with verifying various aspects of infrastructure.</p><p>And the output of execution:</p><pre>$ inspec exec foo.rb</pre><pre>Target: local://</pre><pre>File /etc/myap.conf<br>  ✔ should exist<br>  ✔ mode should cmp == &quot;0644&quot;<br>Port 8080<br>  ∅ should be listening<br>  expected `Port 8080.listening?` to return true, got false</pre><pre>Test Summary: 2 successful, 1 failures, 0 skipped</pre><p>Is it readable? Well, it is, with basic English knowledge it shouldn’t be too hard to understand what source code describes and what is the result. It can specify compliance rules and policy requirements.</p><p>Also, the code is very similar to RSpec, which is used as the underlying foundation of InSpec. One <em>novelty</em> may be the <em>cmp</em> matcher. It helps to compare file modes, string, single element array strings and numbers. You can read more about cmp in <a href="http://inspec.io/docs/reference/matchers/">InSpec documentation</a>.</p><p>The tool itself doesn’t depend on any particular operating system. You can develop your code on almost anything which runs Ruby. For the best experience, you may want to use <a href="https://github.com/chef/chef-dk">ChefDK</a>. Most of the resources will work with common tools available on any system.</p><p>I should mention that InSpec is developed by Chef Software Inc., but it’s isn’t tied to Chef it can be used without it. You don’t even have to like Chef (but why wouldn’t you?).</p><h3>Why InSpec?</h3><p>Why not $(another framework)? I assume you have a favorite integration testing tool, so I’ll try to convince you to use InSpec.</p><ul><li>It’s open source. I know most of the tools are, but it’s a good start.</li><li>Its development is supported by Chef Software Inc., it probably won’t disappear like some nodejs libs in the past and it won’t be abandoned.</li><li>Awesome community. If you have any issues, you can always ask for help on <a href="http://community-slack.chef.io/">Chef slack’s InSpec channel</a>, core maintainers are frequently the first responders.</li><li>Resource-rich. When I’m writing this, there are <a href="http://inspec.io/docs/reference/resources/">over 60 resources</a> available and this number is growing. If you need something unusual you can always use file/command resources or contribute to InSpec.</li><li>Can run anywhere. From your local workstation, you can verify machine over ssh, docker or winrm.</li></ul><pre>$ inspec exec test.rb -t ssh://user@hostname<br>$ inspec exec test.rb -t winrm://Administrator@windowshost --password &#39;your-password&#39;<br>$ inspec exec test.rb -t docker://container_id</pre><ul><li>It has an interactive shell. I’m so used to Chef Shell, I don’t know how I was able to write integration tests without inspec shell.</li></ul><pre>$ inspec shell<br>Welcome to the interactive InSpec Shell<br>To find out how to use it, type: help<br>inspec&gt; describe file<strong>(</strong>&#39;/home&#39;<strong>)</strong> <strong>do       <br></strong>inspec&gt;   it <strong>{</strong> should exist <strong>}</strong> <br>inspec&gt; end   </pre><pre>Profile: inspec-shell <br>Version: unknown </pre><pre> File /home <br>    ✔  should exist </pre><pre>Test Summary: 1 successful, 0 failures, 0 skipped</pre><ul><li><a href="https://github.com/chef/inspec#supported-os">It works on most popular operating systems</a>. On less popular too!</li><li>Ruby. As Serverspec/RSpec derivative, it’s written in Ruby. If I need to, I can easily jump into the source and verify any doubts.</li></ul><h3>How to use InSpec?</h3><p>First, we’ll focus on possible scenarios.</p><ol><li>We have this awesome automation tool, it does things. Does it? Most basic usage, verify your provisioner — Chef, Puppet, Salt, Ansible, shell scripts.</li><li>Our app/code has to run in various environments in a mix of conditions. Multiple Linux distributions, Docker, Windows.</li><li>Operating system migration. It’s similar to previous one, but it’s worth separate mention. Ubuntu 14.04 LTS will be supported until April 2019. Get ahead and start testing your provisioning on next LTS before it will be too late.</li><li>Security compliance. There are sets of InSpec profiles linked on <a href="https://supermarket.chef.io/tools?type=compliance_profile">Chef Supermarket</a> which can help you verify if your operating system is running under sane security settings.</li><li>Cheap reporting. Describe your whole infrastructure in InSpec, run it periodically and store the reports for better times. Another department may ask for it, someone may ask if your environments are really the same, you may need to prepare for an external audit.</li></ol><h3>Examples</h3><p>Time for some coding. I would like to show very basic examples of usage. Everything I’ll show is available at <a href="https://gitlab.com/szymon.szypulski/inspec-blog-post">a Git repository</a>.</p><h4>Tools</h4><p>For running this demo I’ll use a small cookbook which will install web/database server, everything will be tied with <a href="http://kitchen.ci/">Test Kitchen</a>. For running the code you will need Ruby with Bundler or you can use ChefDK.</p><p>As mentioned, the whole thing will be driven by Test Kitchen. Everything is configured in <a href="https://gitlab.com/szymon.szypulski/inspec-blog-post/blob/master/.kitchen.yml">.kitchen.yml</a>.</p><pre>---<br>driver:<br>  name: vagrant<br>  provision: true<br>  vm_hostname: inspec.ragnarson.com<br>provisioner:<br>  name: chef_zero<br>  client_rb:<br>    node_name: inspec.ragnarson.com<br>  require_chef_omnibus: &quot;12.18.31&quot;<br>verifier:<br>  name: inspec<br>platforms:<br>  - name: ubuntu-14.04<br>  - name: ubuntu-16.04</pre><p>Vagrant will be our VM provider. Chef will provision the recipes. Everything will be verified by InSpec and we’ve two versions of Ubuntu at our disposal. List of all instances used for testing is available after executing kitchen list:</p><pre>$ kitchen list<br>Instance                   Driver   Provisioner  Verifier  Transport  Last Action    Last Error<br>web-ubuntu-1404            Vagrant  ChefZero     Inspec    Ssh        &lt;Not Created&gt;  &lt;None&gt;<br>web-ubuntu-1604            Vagrant  ChefZero     Inspec    Ssh        &lt;Not Created&gt;  &lt;None&gt;<br>database-ubuntu-1604       Vagrant  ChefZero     Inspec    Ssh        &lt;Not Created&gt;  &lt;None&gt;<br>ssh-hardening-ubuntu-1604  Vagrant  ChefZero     Inspec    Ssh        &lt;Not Created&gt;  &lt;None&gt;</pre><h4>Web</h4><p>Let’s start with ‘web’ server. <a href="https://gitlab.com/szymon.szypulski/inspec-blog-post/blob/master/recipes/web.rb">Code is very simple, get me nginx</a>.</p><pre>package &#39;nginx&#39;</pre><p>First, we’ve to provision instance by running kitchen converge web-ubuntu-1604. Instance names are a combination of test suites and available platforms. Under suites in <em>.kitchen.yml</em> we&#39;ve a definition of <em>web</em> suite.</p><pre>suites:<br>  - name: web<br>    run_list:<br>    - recipe[inspec-blog-post::web]</pre><p>While the machine is provisioning we can take a look at test file located in test/integration/&lt;suite name&gt;/web_spec.rb (there may be multiple test files for one suite). We&#39;ll verify few things.</p><ul><li>Is package installed?</li></ul><pre>describe package(&#39;nginx&#39;) do<br>  it { should be_installed }<br>end</pre><ul><li>Is service running? If we would have monit installed we could also check if it’s monitored.</li></ul><pre>describe service(&#39;nginx&#39;) do<br>  it { should be_running }<br>  <em># it { should be_monitored.by(&quot;monit&quot;) }</em><br>end</pre><ul><li>Is Nginx configuration file present and it has proper ssl_protocols set.</li></ul><pre>describe file(&#39;/etc/nginx/nginx.conf&#39;) do<br>  it { should exist }<br>  its(:content) { should match(/ssl_protocols TLSv1 TLSv1.1 TLSv1.2;/) }<br>end</pre><ul><li>Our special CVE-2016–1247 case, on some systems it was possible to escalate from www-data to root user when log directory was owned by the www-data user. We can test against this in InSpec.</li></ul><pre><em># CVE-2016-1247</em><br>describe file(&#39;/var/log/nginx&#39;) do<br>  it { should_not be_owned_by(&#39;www-data&#39;) }<br>end</pre><p>To verify if our server does follow the rules run kitchen verify web-ubuntu-1604. You should see all green output, our instance state is matching our expectations.</p><pre>System Package <br>    ✔  nginx should be installed <br> Service nginx <br>    ✔  should be running <br> File /etc/nginx/nginx.conf <br>    ✔  should exist <br>    ✔  content should match /ssl_protocols TLSv1 TLSv1.1 TLSv1.2;/ <br> File /var/log/nginx <br>    ✔  should not be owned by &quot;www-data&quot; </pre><pre>Test Summary: 5 successful, 0 failures, 0 skipped</pre><p>Let’s see what will happen if we’ll run the same test suite on Ubuntu 14.04. Execute kitchen verify web-ubuntu-1404. If the instance isn&#39;t converged it will be, as part of verify task.</p><pre>...<br>    ∅  content should match /ssl_protocols TLSv1 TLSv1.1 TLSv1.2;/<br>...<br>Test Summary: 4 successful, 1 failures, 0 skipped</pre><p>The command should fail, because Ubuntu 14.04 doesn’t have default SSL protocols list, well it doesn’t even have SSL enabled. I know it’s a trivial test, but it shows how you can use InSpec.</p><h4>Database</h4><p>Our <em>database</em> recipe is quite simple as well, it should just install PostgreSQL.</p><pre>package &#39;postgresql&#39;</pre><p>Lets verify our test suite kitchen verify database-ubuntu-1604. In the meantime, look at our test suite code.</p><ul><li>As previously, is package installed?</li></ul><pre>describe package(&#39;postgresql&#39;) do<br>  it { should be_installed }<br>end</pre><ul><li>Does it run?</li></ul><pre>describe service(&#39;postgresql&#39;) do<br>  it { should be_running }<br>end</pre><ul><li>For PostgreSQL, there is the <em>postgres_conf</em> resource which is more convenient than regular expression used for <em>nginx.conf</em>.</li></ul><pre>describe postgres_conf do<br>  its(&#39;max_connections&#39;) { should eq &#39;100&#39; }<br>end</pre><ul><li>Is daemon running on proper host and port?</li></ul><pre>describe port(5432) do<br>  it { should be_listening }<br>  its(&#39;addresses&#39;) { should_not include &#39;0.0.0.0&#39; }<br>  its(&#39;protocols&#39;) { should include(&#39;tcp&#39;) }<br>end</pre><ul><li>In this case, there should be no service running on ports between 22 and 80. Block can be passed to resource and code looks clean.</li></ul><pre>describe port.where { protocol <strong>=~</strong> /tcp/ <strong>&amp;&amp;</strong> port <strong>&gt;</strong> 22 <strong>&amp;&amp;</strong> port <strong>&lt;</strong> 80 } do<br>  it { should_not be_listening }<br>end</pre><p>The summary should be <em>all green</em>.</p><pre>Test Summary: 7 successful, 0 failures, 0 skipped</pre><h4>Profiles</h4><p>There is one InSpec feature I didn’t write much about. There are <a href="https://supermarket.chef.io/tools?type=compliance_profile">publicly available InSpec profiles</a> which describe best configuration practices for specific services. In our test suites as an example, I’ve used [ssh-hardening] developed as part of the dev-sec group.</p><p>If you are using external profile, Test Kitchen configuration is a little bit different.</p><pre>...<br>- name: ssh-hardening<br>  includes:<br>    - ubuntu-16.04<br>  verifier:<br>    inspec_tests:<br>      - name: ssh-hardening<br>        url: https://github.com/dev-sec/tests-ssh-hardening/archive/master.tar.gz<br>...</pre><p>If you’ll run verify, you will see how ssh and sshd are configured by default on Ubuntu 16.04.</p><pre>$ kitchen verify ssh-hardening-ubuntu-1604</pre><p>You’ll see lots of warnings, mostly because Ubuntu relies on sane defaults and it doesn’t specify most of the options explicitly. But you can find some interesting things in kitchen output.</p><pre>...<br>×  sshd-06: Server: Do not permit root-based login or do<strong> </strong>not allow password and keyboard-interactive authentication <strong>(</strong>expected &quot;prohibit-password&quot; to match /no|without-password/<br>   Diff:              <br>   @@ -1,2 +1,2 @@<br>   -/no|without-password/<br>   +&quot;prohibit-password&quot;<br>   <strong>)</strong>              <br>   ×  SSH Configuration PermitRootLogin should match /no|without-password/<br>   expected &quot;prohibit-password&quot; to match /no|without-password/<br>   Diff:              <br>   @@ -1,2 +1,2 @@<br>   -/no|without-password/<br>   +&quot;prohibit-password&quot;<br>...</pre><p>By default there is no root password on Ubuntu, therefore it isn’t possible to log in as root remotely. In case anyone (or a tool) sets the password, it’s good practice to double check if password-based logins are disabled.</p><p>To cleanup after our examples, run kitchen destroy.</p><h3>Limitations</h3><p>It’s easy to notice I like InSpec. However, there are few things which could be improved:</p><ol><li>Code sharing is less than ideal. Custom resources are stored in the <em>libraries</em> directory. If you want to share a resource across multiple cookbooks, you have to create a separate profile.</li><li>Some resources are still missing from Serverspec, but it isn’t a big problem. Maintainers are re-implementing/moving resources upon user request. You just may need to wait a while for release.</li></ol><h3>Alternatives</h3><p>To achieve some level of objectivity, I would like to mention alternatives to InSpec.</p><h4>Serverspec</h4><p>It’s probably a most common alternative to InSpec, migration between any of them <a href="http://inspec.io/docs/reference/migration/">isn’t complicated</a>. In my opinion, it’s closer to RSpec. It does support shared groups, which helps if you made some bad decisions and you are keeping too many cookbooks in a single repository. Unfortunately, it won’t run so easily over docker/winrm protocols. In our case, it’s also slower to re-run test after changes. Test files upload process can take over 60 seconds, where in the case of InSpec our tests start almost immediately.</p><h4>Goss</h4><p>Maybe it isn’t a full blown alternative. It’s a small infrastructure validation tool which I found in <a href="https://www.cronweekly.com/">Cron Weekly newsletter</a>. It’s written in Go, it doesn’t have many dependencies, it’s self-contained, it’s fast. Tests are written in YAML if you don’t like Ruby. It doesn’t have so many resources like InSpec or Serverspec, but there is one awesome feature — it can expose a health endpoint over HTTP. You can validate your infrastructure remotely. I wonder If it can be ported to InSpec or Serverspec. It would be so much easier to just run single query without any dependencies. It may be in a conflict with Chef Compliance tho.</p><h3>Summary</h3><p>How integration tests or InSpec are helping us?</p><ol><li>We had a highly experimental environment — sandbox. It was used during the very early stage of infrastructure development, where things were checked manually. We don’t need it anymore, we just write integration tests and run them in Test-Kitchen. It saved us some money because we no longer need additional AWS instances.</li><li>InSpec is a little bit faster than Serverspec. During development, when changes to test suite are made. Serverspec has to re-upload the code to run it on an instance. InSpec handles it much faster. This saves us 15–30 seconds on each test suite. With over 20 suites in our main repo, it helps a lot.</li><li>It helps to mix wrapper cookbooks. Sometime in the past, we’ve had a rule: never mix services on single node. Interactions with operating system were sometimes complex. However on the early stage of the project it’s a resource waste to have one/two (replicated) instances for each service. Integration tests help with this, a lot.</li><li>We’re gradually preparing for new Ubuntu LTS. With each bigger change we’re running tests against newer Ubuntu and we’re fixing code which wasn’t universal enough.</li><li>It helped us nail multiple smaller and bigger bugs, most notably:</li></ol><ul><li>We were loosing HTTP2 at some stage across our HTTP stack serving Kibana, Ubuntu 14.04 had outdated OpenSSL. We didn’t know about it until integration tests were introduced.</li><li>Multiple cookbooks were overwriting the same sysctl value. We’ve had different sysctl settings for different services. When two or more wrapper cookbooks were put on the same node it came to our attention that cookbooks were interfering with each other.</li></ul><p>That’s all. I know I’ve just scratched the surface of integration testing. I hope, if you didn’t do integration testing, you’ll at least reconsider introducing them into your workflow.</p><p><em>Originally published at </em><a href="https://blog.ragnarson.com/2017/03/13/inspec-inspect-your-infrastructure.html"><em>blog.ragnarson.com</em></a><em> on March 13, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18debd45f0db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ragnarson/inspec-inspect-your-infrastructure-18debd45f0db">InSpec — Inspect Your Infrastructure</a> was originally published in <a href="https://medium.com/ragnarson">Ragnarson</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>