<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dennis Webb on Medium]]></title>
        <description><![CDATA[Stories by Dennis Webb on Medium]]></description>
        <link>https://medium.com/@denniswebb?source=rss-ff8b05bce729------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gAae7bDeROWnmgSZC1ptRA.jpeg</url>
            <title>Stories by Dennis Webb on Medium</title>
            <link>https://medium.com/@denniswebb?source=rss-ff8b05bce729------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:45:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@denniswebb/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fast and Secure Backups to S3 with Restic]]></title>
            <link>https://medium.com/@denniswebb/fast-and-secure-backups-to-s3-with-restic-49fd07944304?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/49fd07944304</guid>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[restic]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Tue, 14 Aug 2018 11:01:01 GMT</pubDate>
            <atom:updated>2018-08-14T11:01:01.193Z</atom:updated>
            <content:encoded><![CDATA[<p>We all know how important backups are, but even today we seldom have a reliable and secure backup solution. Some people have a small NAS device at home that they backup their data to, but how useful is that in the case of theft or fire?</p><p><a href="https://restic.net/">Restic</a> is an open source backup application written in <a href="https://golang.org/">Go</a>. It runs on all modern operating systems including Windows, Mac, and Linux. There is no graphical interface as it is a console application, but this guide will teach you how to install and configure it for regular backups to the cloud.</p><h3>Installation</h3><p><strong>Note: </strong>This guide is tailored towards Mac and Linux users. I may mention some Windows specific steps for installation, but the rest of the guide assumes a Unix style OS.</p><p>To get the latest stable release of Restic, visit their <a href="https://github.com/restic/restic/releases/latest">Github releases page</a>. On Linux and Mac, extract the downloaded archive and move the Restic binary to /usr/local/bin as restic. On Windows move the application into your C:\Windows\System32 directory. This will put Restic in your path, making using it as simple as typing restic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*HgZG8Dh9ky96ophKoztAfw.png" /><figcaption>Installing Restic from Github on Ubuntu Linux</figcaption></figure><h4>Mac</h4><p>You can also quickly install by running brew install restic.</p><h4>Debian/Ubuntu</h4><p>sudo apt-get install restic -y will install, but the version provided might be many versions behind the latest release.</p><h3>S3 Bucket and IAM Configuration</h3><p>Start off creating an S3 bucket to store your backups. Log into your <a href="https://console.aws.amazon.com/console">AWS console</a> and create a new bucket to store your backups. Feel free to choose whatever name you want for your bucket. I highly recommend enabling <strong>versioning</strong> and <strong>encryption</strong>. Restic also encrypts your data, but you can’t be too safe. And triple check to make sure your bucket is not public.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*xRdFC7c_ntVu_zqvGlnWVw.png" /><figcaption>Review of my new bucket for backups</figcaption></figure><p>Now that we have a bucket to store our data, now we need to create a dedicated AWS user for Restic. Go ahead and do this now, don’t use your personal keys even though it might be quicker to do.</p><p>Create a new user name <strong>restic</strong> and grant it programmatic access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*8w_fUalDVIKN5jdkAuOBNA.png" /></figure><p>Create a custom JSON policy for the user. Use the snippet below, replacing BUCKET_NAME with your bucket’s unique name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*N57lw1VMvar0FBczXQtYlA.png" /><figcaption>Click <strong>Create policy</strong> to customize the permissions for the new user</figcaption></figure><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:GetBucketLocation&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::BUCKET_NAME&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:DeleteObject&quot;,<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::BUCKET_NAME/*&quot;<br>        }<br>    ]<br>}</pre><p>Once created, store the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in a safe place. We’re gonna need it soon.</p><h3>Restic Configuration</h3><h4>Generate a Repository Key</h4><p>Restic protects your data by encrypting it. This way even if somebody gains access to your S3 bucket, the still cannot read your data. Generate a strong 24+ character password using <a href="https://www.lastpass.com/password-generator">LastPass’s password generator</a>. Store a copy of this password in a secure location because without it, your backups will be useless.</p><h4>Configuration File</h4><p>Using your <a href="https://neovim.io/">favorite text editor</a> create a file named .restic.env in your home directory. Inside this file, we are going to add lines for each of the 4 required environment variables. Use the example block below for yours, replacing the values with your own.</p><pre>export AWS_ACCESS_KEY_ID=&quot;AKIAJNFRE7YAHGDUMD&quot;<br>export AWS_SECRET_ACCESS_KEY=&quot;EtE7mGPqcbjyPT4KLXX1qVDUkAVhA&quot;<br>export RESTIC_PASSWORD=&quot;!emP*#IWFEoy8%@zp9qQpIHo&quot;<br>export RESTIC_REPOSITORY=&quot;s3:<a href="https://s3.amazonaws.com/backup-dennis">https://s3.amazonaws.com/</a>BUCKET_NAME&quot;</pre><p>Let’s protect this file from other users by changing the permissions on the file to read-only by your user with chmod 400 ~/.restic.env. To activate these environment variables into your current bash session run source ~/.restic.env. I don’t like having credentials in my shell all the time. This method allows me to only have them active when I need them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*bJ9_OX5anoMr5HTCFqR8Ig.png" /><figcaption>Creating, securing, and testing my .restic.env file</figcaption></figure><h3>Initializing Your Repository</h3><p>Before Restic can backup anything, you must first initialize its repository. This only has to be done once. To do so run source ~/.restic.env ; restic init.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*yh8E0ZN9-UoUjKNYnQ8gng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*-76NspI7HGOOJPBg6zSpXw.png" /><figcaption>Contents of an empty initialized repository</figcaption></figure><h3>Your First Backup</h3><p>Let’s create a text file to backup and go through a few basic Restic commands to demonstrate backup and restore procedures.</p><ul><li>Create a sample file name <strong>backuptext</strong> by running echo &quot;My first file to backup&quot; &gt; backuptext.</li><li>We’re going to set our required environment variables with source ~/.restic.env</li><li>Backup the file by executing restic backup backuptext.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*BkE-I_dV52NOQqBQpW2MRA.png" /><figcaption>Creating and backing up our test file</figcaption></figure><ul><li>Overwrite <strong>backuptext</strong> with echo &quot;My modified file&quot; &gt; backuptext.</li><li>Backup the modified file. restic backup backuptext.</li><li>List the snapshots in your repository. restic snapshots.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*wYTsYMhk54SAEmxpbqgVBg.png" /></figure><ul><li>Confirm the contents of <strong>backuptext</strong>. cat backuptext</li><li>Restore from the first snapshot with restic restore SNAPSHOTID -t restored-data replacing <strong>SNAPSHOTID</strong> with the first snapshot listed when you ran restic snapshots. The file will be restored into a directory named <strong>restored-data</strong>.</li><li>Confirm contents of restored file. cat restored-data/backuptext</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*7cu55sGBH3RySnctaJ8gWA.png" /><figcaption>Restoring file from backup</figcaption></figure><h4>Clean Up Test Snapshots</h4><p>We can cleanup our test snapshots with the restic forget --keep-tag=&quot;whatever&quot; --prune command. Since we haven’t tagged any of our backups, it will delete all of these because the --keep-tag flag tells Restic to only keep backups with the tag <em>whatever</em>. The --prune flag tells Restic to also delete the data from S3, which isn’t done by default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*JoZ7nGYvoTrzX8xH-sWRaA.png" /><figcaption>Deleting all snapshots</figcaption></figure><h3>Real Backups</h3><p>Now that we have a working Restic configuration, it’s now time to create a more useful setup. This walk-through will backup <em>/etc</em>, /usr and <em>/root</em> directories. You typically would not backup /usr, but I’m using it as a source for lots of data. This backup will be scheduled to run every 15 minutes. Since Restic only uploads changed files, subsequent jobs will run fast after the initial upload is done.</p><h4>Specifying Files to Backup</h4><p>We’re going to create a file called restic.files in our home directory. This file will list all the files and directories we want to back up. We could specify them on the command-line like we did earlier, but a file makes it easier to work with once the list starts expanding. List each directory on a separate line in the file.</p><pre>/etc<br>/root<br>/usr</pre><h4>Running Our Initial Backup</h4><p>Once again we’ll setup our environment variables with source ~/.restic.env. To run our backup using our restic.files list we’ll use the --files-from /root/restic.files flag. Restic has issues using the ~ shorthand, so we have to type the full path to the file. The --tag seed flag tags the snapshot as a seed. This is a freestyle label that can be used for organization. The full command is restic backup --files-from=/root/restic.files --tag seed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*itQAuUD8z-h37n_wnTGf6g.png" /><figcaption>Our seed backup</figcaption></figure><p>We’ll run the command again. Notice how quickly it runs this time. 3 seconds compared to 50 seconds for our initial backup. This is because Restic can quickly determine if a file is already stored in S3 and only upload new and changed files. It also does chunking, which means if parts of a very large file are changed, only the changed chunks need to be uploaded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*j7Oq3GBsB6U93iYXKmQCpw.png" /><figcaption>Subsequent backups are very quick</figcaption></figure><h4>Automating Our Backups with Cron</h4><p>Since Restic backups run fast after the initial seed, it’s very useful to run backups very frequently. For our example, let’s have cron run backups every 15 minutes. We’ll also configure the job to write output and errors to restic.log and restic.err in your home directory.</p><ul><li>Run crontab -e to edit your user’s crontab.</li><li>Add the following line */15 * * * * . ~/.restic.env ; /usr/local/bin/restic backup --files-from=/root/restic.files --tag automated 2&gt;&gt; ~/restic.err &gt;&gt; ~/restic.log. Notice we use . instead of source. This is a nuance of cron using a POSIX shell and not bash.</li><li>After the next quarter-hour, check your snapshots, restic snapshots, and confirm the task completed as expected.</li><li>If no new snapshot is created, check the restic.err file for errors.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*Ij_EFq0SjHjxHL98X19ivw.png" /><figcaption>Automated snapshot listed</figcaption></figure><h3>Conclusion</h3><p>You now have a nice automated backup solution to S3. For most users, simply backing up your <em>home</em> directory will be all you need. On a business system, backing up /etc, /var, and specific data directories will be needed.</p><p>If you would like an estimate for how much S3 storage you are using, restic status will return that information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*Hptg3BRzK3LbIquINHgoLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/1*sU4hyFmLDqnR8yAzWZiAVw.png" /><figcaption>Root directory structure of an active repository</figcaption></figure><p>In upcoming articles, I will expand on this one by giving instructions on adding an automated backup to a <a href="https://www.wdc.com/products/personal-cloud-storage.html">Western Digital My Cloud</a> NAS and for backing up <a href="https://aws.amazon.com/efs">AWS Elastic File System (EFS)</a> shared filesystems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49fd07944304" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Passing the Certified Jenkins Engineer Exam]]></title>
            <link>https://medium.com/@denniswebb/jenkins-cje-e82dc00fc640?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/e82dc00fc640</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[certification]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Sat, 11 Aug 2018 01:57:33 GMT</pubDate>
            <atom:updated>2018-08-11T01:57:48.778Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*vSp3urPX-HmoYKlc" /></figure><p>I work with quite a few software development teams, spanning multiple companies. One thing that 80% of them have in common is <a href="https://jenkins.io/">Jenkins</a>. Jenkins is an automation server used to help build, test, and deploy your software. And although that’s what it’s designed to do, it can handle any software task that you need automated and can script.<br>When I learned that <a href="https://www.cloudbees.com/">CloudBees</a> offered the <a href="https://www.cloudbees.com/jenkins/jenkins-certification">Certified Jenkins Engineer certification</a>, I thought it’d be a great way to learn more about what Jenkins could really do. I knew I was not taking advantage of its benefits. Once I really dove in, I learned there was so much more to Jenkins than simple build jobs.</p><h3>What I Learned</h3><p>I discovered a lot about Jenkins I didn’t know. Almost all of my Jenkins experienced focused on only using the standard <a href="https://wiki.jenkins.io/display/JENKINS/Building+a+software+project"><strong>Freestyle Project</strong></a>. Authentication was just credentials stored in the Jenkins database with all users having full run of the system. Only a few systems used folders to help organize jobs, most had 100 or more disorganized jobs at the root.</p><h4>Pipelines</h4><p><a href="https://jenkins.io/doc/book/pipeline/">Pipelines</a> are amazing. This is a feature that I definitely benefitted the most from learning. I am a strong believer of everything being stored in source control. And with my existing Jenkins experience, that wasn’t the case. All of the jobs were hand-rolled and a backup of $JENKINS_HOME was the only thing protecting us from a disaster.</p><p>The idea with Pipelines is you create a <a href="https://jenkins.io/doc/book/pipeline/jenkinsfile"><strong>Jenkinsfile</strong></a> in the root of your code projects repository. Your Jenkins job is located in source control, and all modifications to it are tracked. Inside your Jenkinsfile you define your pipeline using Jenkin’s Pipeline domain specific language(DSL). Practically everything you can define for a Freestyle Project in the GUI, you can configure in a Jenkinsfile, plus more.</p><p>Jenkinsfiles can describe a declarative pipeline where you define what you want, and Jenkins handles making it happen per your definition. Think of this as being similar to <a href="https://terraform.io/">Terraform</a> templates or <a href="https://www.ansible.com/">Ansible</a> playbooks. The other option is scripted, where every action and the execution order is defined by the file. Declarative is becoming the most popular format. For the test, I recommend writing a few pipelines using both formats without using any documentation or assistance. Knowing most of the DSL and what the functions do is critical to passing the exam.</p><h4>Security</h4><p>Jenkins offers a lot more authentication options other than using its built-in database for users. I learned there are plugins for <a href="https://plugins.jenkins.io/saml">LDAP</a>, <a href="https://plugins.jenkins.io/saml">SAML</a>, <a href="https://plugins.jenkins.io/github-oauth">Github</a>, and <a href="https://plugins.jenkins.io/google-login">Google</a> authentication along with many others. Although I assumed this was possible, I had never worked with any of these plugins.</p><p><a href="https://wiki.jenkins.io/display/JENKINS/Matrix-based+security">Matrix-based Security</a> was another great feature to learn about. Being able to restrict groups and users to just read permissions to certain parts of the system is a lot better than the default setting of <em>Logged-in users can do anything</em>. This is something you definitely need to have experience with when taking the exam.</p><h4>Artifacts and Fingerprints</h4><p>Although I knew about artifacts with Jenkins, I didn’t know anything about fingerprints. Once again it was something I had never dealt with. I learned how fingerprints are simply md5 hashes of the artifacts themselves. That was simple enough, but then I realized that once you enable them, you can track all the uses of a distinct artifact across multiple jobs. That’s a pretty useful feature.</p><h3>Study Guides</h3><p>You can’t mention passing a certification test without answering the next question everybody asks, “How did you study?” Below is a list of the resources I used. I crammed for this test in a little over a week. I had Jenkins experience and many years of in-the-weeds DevOps experience, so a lot of the concepts were old to me. I just had to learn <em>The Jenkins Way.</em></p><h4>Linux Academy’s CJE Course</h4><p><a href="https://linuxacademy.com/">Linux Academy</a> has an almost 20-hour course to prepare you for the CJE exam. They cover a lot of the material and assume you have no previous Jenkins knowledge. Their labs were very useful. I have to say I enjoy using their lab servers more than just spinning up instances in my AWS account. For one the cost is built-in to my monthly membership fee, and it’s very simple. They also offer practice quizzes and flashcards (some community written) to study from.</p><p>The course covers almost all the topics needed to pass the exam. The problem is, they only scratch the surface of a lot of topics such as pipeline scripts, global libraries, the REST API, and security. The exam covers these topics much deeper than what’s covered here. I still think this is the perfect place to start. You can watch the videos at 1.5–2 times normal speed to get through the material faster.</p><p>Also be sure to go through some of the flashcards on the site. Some of the community ones cover topics that are on the exam, but not covered by the course. This is a good way to find your weak spots.</p><h4>Jenkins Tutorials</h4><p>Go through all of the <a href="https://jenkins.io/doc/tutorials">tutorials</a> available the Jenkins documentation site. It’s one thing to read about the Pipeline DSL, but it’s another to actually write it from scratch. Little details about when to use parentheses over curly-braces are missed when just reading. When you are doing the tutorials, you will find sections where the instructions say, “Copy the following Declarative Pipeline code and paste it…”. Please do not do this. Let’s do this <a href="https://learnpythonthehardway.org/">the hard way</a>. Type out the code by hand. It will help you learn and retain the language details better for the exam.</p><h4>Certified Jenkins Engineer 2017 Certification Exam Study Guide</h4><p>This is the <a href="https://www.cloudbees.com/sites/default/files/cje-study-guide-2017.pdf">official study guide</a> put out by CloudBees. This is not a comprehensive book to teach you what you need to know. It’s more like an exam blueprint listing the topics covered by the exam. It also provides some very useful links to online resources you should read. It also contains 5 sample questions that resemble the type of questions covered on the exam. And no, none of the sample questions appeared on my exam, so no free points there.</p><h4>Jenkins Documentation Website</h4><p>Another resource to cover is the <a href="https://jenkins.io/doc/">Jenkins Documentation Website</a>. I know that the tutorials above are a part of this site, but I feel that hands-on experience is very important and a must for the exam. That’s why I gave them a separate section. The rest of the documentation is still important. And if you actually read the links in the CloudBees Study Guide, you’ve already read it. If you haven’t followed those links, then read this for sure if nothing else. It isn’t as long as you would think. It covers the concepts learned in the Linux Academy course with more detail.</p><h4>jenkins-cli and REST API</h4><p>These 2 are a huge part of the test. There aren’t any great tutorials, guides, or labs to guide you through using them. You do need to know them very well. I recommend setting up a pipeline, folders, and a couple of freestyle jobs using each of these utilities. Practice running jobs from the CLI and API, with and without parameters. Trigger jobs to poll SCM for changes without waiting on a webhook or the cron schedule to come around. Download a config.xml file from one of your freestyle jobs. Read the file returned. Understand it. Doing these things are very important.</p><h3>Final Thoughts</h3><p>The exam is challenging. Jenkins is a very narrow subject to be tested on. So they go deep into the details to make sure you know what you’re doing. Unlike an AWS exam where the questions can barely scratch the surface of most topics, the CJE exam is focused on a single product and all aspects of it.</p><p>If you have no Jenkins experience, but dedicate 1–2 hours a day to study, you can cover all of the material above in a month. I recommend covering it in the order I laid it out. Be sure to follow along with the Linux Academy instructor, and do the work a second time with their individual hands-on labs. Complete all the tutorials in the Jenkins docs also. You can use your Linux Academy lab servers to do those also. As you can tell, I believe hands-on experience is the best.</p><p>Good luck. If you take the exam, let me know how it went for you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e82dc00fc640" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Back to Running After Injury]]></title>
            <link>https://medium.com/@denniswebb/back-to-running-after-injury-c4ba40163e94?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4ba40163e94</guid>
            <category><![CDATA[plant-paradox]]></category>
            <category><![CDATA[injury]]></category>
            <category><![CDATA[marathon]]></category>
            <category><![CDATA[marathon-training]]></category>
            <category><![CDATA[running]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Tue, 07 Aug 2018 00:01:01 GMT</pubDate>
            <atom:updated>2018-08-07T00:01:01.888Z</atom:updated>
            <content:encoded><![CDATA[<p>I love running. Absolutely love it. I’m not what you’d call fast, but I do ok. Last year I got back to serious running. I ran the Memphis St. Jude marathon on December 2, 2017, with a finishing time of 4:11:29, my slowest ever, but faster than my goal of 4:15.</p><p>At the first of this year, I had a goal of running 2018 miles. I was running around 50 miles a week and was feeling great. 8 miles was a normal run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rr92AEwkwcKweHAgUHT2bA.jpeg" /><figcaption>Mileage over past 12 months</figcaption></figure><p>Mid-February everything changed. I started getting a severe pain in what felt like my upper hamstring. I go to the doctor and get X-rays. Nothing conclusive can be seen. Over the next 6 weeks, I go through 2 dose packs, a steroid shot, and lots of at home physical therapy with stretching. Nothing helped. Every time I tried to run, the pain was immediate. I got to where I had a limp when I walked and had completely stopped running. The doctor wanted me to get an MRI to help diagnose, but United Healthcare decided to not to approve it.</p><p>After months of arguing with the insurance company, I finally got the MRI approved. Unfortunately, in the meantime, I decided to learn to longboard. That last 12 minutes before I broke my wrist and had to have surgery. So the MRI had to be delayed a couple of weeks until I was feeling better. When the MRI results came back, the diagnosis was a stress fracture to the pelvis. I hadn’t attempted to run in 8 weeks. The doctor said after 4–6 additional weeks of rest I should be fully healed. The morning of my follow-up I got up early and attempted a short 2 mile run. For the first time in 4 and a half months, I was able to run pain-free. The doctor confirmed that morning I was fully healed.</p><p>So I was ready to start running again. But with such a long hiatus, my fitness had taken a nose-dive. The broken wrist didn’t help as I could no longer swim or bike either. I had let myself go. I gained 20+ lbs. during this period.</p><p>Any runner knows building mileage too quickly is where injuries happen. I don’t want to get stupid and do too much too soon. So I started with running 3 miles. My speed was almost 2 minutes per mile slower than before the injury. I struggled to finish. For the next 4 weeks, I decided to run 0.9 miles and walk 0.1 miles to give myself a break. I would run 3 miles 3–4 times per week. The good news is no pain at all. I feel great after running which is great.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*G_YvN-lTOjNAXuCndjJsBw.png" /></figure><p>I don’t run to run just 3 miles though. I like running long. I like my daily runs to be 7 miles, mediums around 10 miles, and long runs 13+ miles. So once again I am going to register to run the full marathon on December 1, 2018. That’s 17 weeks away. Just the right amount of time to build to 26.2. I don’t have a PR goal of under 4 hours this year, I’ll save that for 2019. This year is all about just finishing — hopefully under 4 and a half hours.</p><p>For my plan, I am going to go with <a href="https://cdn.shopify.com/s/files/1/1754/7753/files/Beginner_Marathon_-_3.15.pdf?12975293061006073263">the beginner plan</a> from <a href="https://aerbook.com/maker/productcard-2183034-476.html">Hansons Marathon Method book</a>. I used the same plan last year for training and really enjoyed it. Starting on week 2 this week, I’ll continue 3 mile dailies, and my long will go up to 4 miles. This is right in line with my current fitness level. Mileage does jump pretty radically from week 5 to week 6, but I’ll just have to pay attention to my body and adjust as necessary. I don’t think I will have much of an issue.</p><p>I also started the <a href="https://youtu.be/2h0VhF9ad78">Plant Paradox diet</a> to help change my eating habits and lose the extra weight. If everything goes as planned, by race day I will be down near my goal weight of 165 pounds.</p><h3>Training Plan for the Week of August 6</h3><p>Mon, Wed, Fri, and Sat: 3 miles. My long run on Sunday will be just 4 miles. Total weekly mileage: 16 miles</p><p>Nothing special just light easy running. On off days I plan to bike and still get in 3 <a href="https://www.lesmills.com/us/workouts/fitness-classes/bodypump">BodyPump</a> classes in during the week.</p><p>You’re welcome to <a href="https://www.strava.com/athletes/21910716/training/log?feature=public-training-log">follow me on Strava</a>. I’ll take all the kudos I can get on there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4ba40163e94" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing for EC2 Per Second Billing]]></title>
            <link>https://medium.com/the-blue-sentry-blog/designing-for-ec2-per-second-billing-e044bc9b2042?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/e044bc9b2042</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Wed, 04 Oct 2017 15:31:23 GMT</pubDate>
            <atom:updated>2017-10-04T15:38:02.673Z</atom:updated>
            <content:encoded><![CDATA[<p>Amazon’s new <a href="https://aws.amazon.com/about-aws/whats-new/2017/10/announcing-amazon-ec2-per-second-billing/">per second billing for EC2 usage</a> is good news. Taking utility billing down to the second, which was previously per hour, is a huge change. Organizations with long-running workloads on AWS won’t necessarily see a benefit from this change. For companies who run a lot of job or worker instances, the savings could be huge.</p><h3>Transient EMR Clusters</h3><p>When you need to do batch processing of data, this is typically done using <a href="https://aws.amazon.com/emr/">Elastic Map Reduce(EMR)</a> clusters. These clusters run on EC2 and are gaining the benefit of per second billing. With per hour billing, the typical goal is to size the cluster so the job runs as close to an hour without going over. Allocate too much power, and the job will complete in 20 minutes but billed for the full hour. Allocate too little, and it could run 63 minutes and now your paying for 2 hours. This is a fine balance that requires monitoring, and if the amount of data needing to be processed varies, it’s almost impossible to get right.</p><p>With per second billing, you can size your cluster with plenty of power, and only pay for the try processing you use. If it runs 63 minutes, no big deal, pay for 63 minutes. If it completes in 32 minutes, then you only pay for 32 minutes. Companies can now run their EMR jobs more frequently also, keeping their data warehouses more up to date.</p><p>Remember to take into consideration the bootstrapping time of an EMR cluster. This takes on average 10 minutes, running more frequent transient EMR clusters will result in paying for that bootstrap time.</p><p>Also be aware that per second billing also applies to spot instances also.</p><h3>SQS/ASG Configuration</h3><p>Textbook <a href="https://aws.amazon.com/sqs/">Simple Queue Service(SQS)</a> examples involve using an <a href="https://aws.amazon.com/autoscaling/">AutoScaling Group(ASG)</a> to spin up EC2 instances to handle the queue size. When the jobs have been completed and removed from the queue, the ASG will delete some of the instances leaving just enough to handle the current load.</p><p>Prior to per second billing, people were very conservative in their ASG scale-out/scale-in rules. This caused queues to get backed up more than optimal. Companies didn’t want to pay to spin up an instance for an entire hour if after 15 minutes the jobs in the queue fell back to normal levels.</p><p>ASGs were also configured to not terminate instances quickly. Nothing worse than running an instance for 15 minutes and terminating it, just to have another surge of jobs cause another instance to be started for another 15 minutes. Now you’ve had to pay for 2 hours while only using 30 minutes.</p><p>Now with per second billing, architects should take a new look at the ASG configurations. Don’t be afraid to scale up and service your queues better. Also, don’t be afraid to scale back. If there’s no work to be done, don’t pay for idle machines because it used to be the smart thing to do.</p><h3>When Lambda’s 5 Minute Limit is Not Enough</h3><p>When deciding if a new project is going to run on <a href="https://aws.amazon.com/lambda/">Lambda</a> or <a href="https://aws.amazon.com/ec2/">EC2</a>, a major point to consider is, “Will this job run for over 5 minutes?” If the answer is, “Yes”, then Lambda is out of the question due to its 5-minute running limit. Lambda offers per 1/10 second billing, which makes it very attractive. Now with per-second EC2 billing, running jobs on EC2 isn’t nearly as expensive as it previously was.</p><p>A common use-case I’ve seen are cron jobs. These are processes that you want running on a scheduled basis: every 15 minutes, every hour, etc. If the job could run in under 5 minutes, it was natural to run it on Lambda using a <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html">CloudWatch Event</a> cron job to start the process. If it took more than 5 minutes, then it was common to have an always-on EC2 instance.</p><p>Sure you could still use CloudWatch Events to start a transient EC2 instance on a scheduled basis that did the processing then terminated itself, but you were getting billed for the entire hour no matter if the instance was only on for 10 minutes. So if the job ran frequently, it was easier to leave the instance running all the time.</p><p>Now you can spin up an instance and only pay for the time you are really using that machine.</p><h3>Conclusion</h3><p>There are many different situations where per-second billing will save your company money. Reassess your current architecture and alway look for new ways to save. I’ve made plenty of design decisions for customers based on costs and savings. Sometimes the game changes and new opportunities come up to save even more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e044bc9b2042" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-blue-sentry-blog/designing-for-ec2-per-second-billing-e044bc9b2042">Designing for EC2 Per Second Billing</a> was originally published in <a href="https://medium.com/the-blue-sentry-blog">The Blue Sentry Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Importing GitHub Repositories into Terraform]]></title>
            <link>https://medium.com/devops-complete/importing-github-repositories-in-terraform-a04ae24ad4b2?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/a04ae24ad4b2</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Tue, 03 Oct 2017 21:02:23 GMT</pubDate>
            <atom:updated>2017-10-03T21:14:40.873Z</atom:updated>
            <content:encoded><![CDATA[<p>Once your organization goes from 5 <a href="https://github.com/">GitHub</a> repositories to hundreds, it gets really difficult to keep them under control. Some have branch protection setup, others don’t. Some require CircleCI or Travis to finish successfully before merging, others don’t. You can see where this is going. These obstacles can be conquered by managing your repositories using <a href="https://terraform.io/">Terraform</a>.</p><p>Start by building a Terraform module that you will re-use for all of your repositories. Below is a gist of a very generic one. The only required input variables is <strong>repository_name</strong>. Without any additional input, it defaults to protecting the <strong>master</strong> branch, requires all pull requests against <strong>master</strong> be reviewed, and that <a href="https://circleci.com/">CircleCI</a> status checks pass before merging.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f652f9c532c61dfd78e6e86ef3632e82/href">https://medium.com/media/f652f9c532c61dfd78e6e86ef3632e82/href</a></iframe><p>Put the file in a module directory named <strong>github</strong> for this example. If you are unfamiliar with how Terraform modules work, the <a href="https://www.terraform.io/docs/modules/create.html">online documentation</a> explains it very well.</p><p>Now create a root Terraform file that will call this module to create your repositories. The gist below is a guideline for building this file. In this file I am creating 3 repositories: <em>datadog-openvpn, os2cw, </em>and<strong> </strong><em>wait4mysql.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d76872442ccd3bea6116267d38eba87/href">https://medium.com/media/5d76872442ccd3bea6116267d38eba87/href</a></iframe><p>Please notice the GitHub provider at the top of this file. This is how Terraform learns your GitHub credentials and organization name. You will need a GitHub token to authenticate with. Instructions on generating one can be found in the <a href="https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/">GitHub documentation</a>.</p><h3><em>Greenfield vs Brownfield</em></h3><h4>Importing Existing Repositories</h4><p>If you have the foresight to know this is the path you want to take before you have created any repositories, then you simply apply the above configuration and everything is good. But when you are trying to bring existing resources under the control of Terraform, you need to<strong> </strong><a href="https://www.terraform.io/docs/import/index.html"><strong>import</strong></a> these resources. This adds the information about a resource into Terraform’s <a href="https://www.terraform.io/docs/state/index.html">state</a> so it doesn’t think it needs to create a resource that already exists.</p><p>To import an existing GitHub repo, run terraform import &lt;resource_path&gt; &lt;repository_name&gt; from the directory containing your Terraform files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*2OGwrzfzbGWXB3RrV_sFMg.png" /><figcaption>Importing repository datadog-openvpn from the above example</figcaption></figure><p>After importing all the existing repositories, the next run of terraform plan will only want to add branch protections to these repositories and not attempt to recreate them.</p><h4>What if we have a lot of repositories?</h4><p>If you have a lot of repositories, use the following <a href="https://curl.haxx.se/"><strong>curl</strong></a> and <a href="https://stedolan.github.io/jq/"><strong>jq</strong></a> statements to automate not just importing your repositories, but also writing the Terraform code itself.</p><p>This snippet below will query GitHub for all your repositories and output to your terminal the Terraform code to create all of your current repositories. It will also sort the repositories alphabetically so my OCD doesn’t start acting up.</p><pre>curl -s &quot;<a href="https://api.github.com/orgs/PotomacInnovation/repos?access_token=$GITHUB_TOKEN&amp;per_page=10&amp;page=1">https://api.github.com/orgs/<strong>&lt;Organization_Name&gt;</strong>/repos?access_token=$GITHUB_TOKEN&amp;per_page=100</a>&quot; | jq -r &#39;sort_by(.name) | .[] | &quot;module \&quot;\(.name)\&quot; {\n  source = \&quot;modules/github\&quot;\n\n  repository_name=\&quot;\(.name)\&quot;\n  description=\&quot;\(.description)\&quot;\n  private=\(.private)\n}\n&quot;&#39;</pre><p><strong>Note: </strong>If you are doing this for your personal GitHub repositories and do not have an organzation, modify the above URL replacing /orgs/<a href="https://api.github.com/orgs/PotomacInnovation/repos?access_token=$GITHUB_TOKEN&amp;per_page=10&amp;page=1"><strong>&lt;Organization_Name&gt;</strong></a><a href="https://api.github.com/orgs/PotomacInnovation/repos?access_token=$GITHUB_TOKEN&amp;per_page=10&amp;page=1">/repos</a> with /users/<a href="https://api.github.com/orgs/PotomacInnovation/repos?access_token=$GITHUB_TOKEN&amp;per_page=10&amp;page=1"><strong>&lt;GitHub_Username&gt;</strong></a><a href="https://api.github.com/orgs/PotomacInnovation/repos?access_token=$GITHUB_TOKEN&amp;per_page=10&amp;page=1">/repos</a>.</p><p>When I do this for my personal GitHub repository, I get the following output:</p><pre>module &quot;datadog-openvpn&quot; {<br>  source = &quot;modules/github&quot;</pre><pre>repository_name=&quot;datadog-openvpn&quot;<br>  description=&quot;OpenVPN DataDog Agent&quot;<br>  private=false<br>}</pre><pre>module &quot;ddbkv&quot; {<br>  source = &quot;modules/github&quot;</pre><pre>repository_name=&quot;ddbkv&quot;<br>  description=&quot;ddbkv simply stores key/value pairs to a DynamoDB table&quot;<br>  private=false<br>}</pre><pre>module &quot;go-license&quot; {<br>  source = &quot;modules/github&quot;</pre><pre>repository_name=&quot;go-license&quot;<br>  description=&quot;Checks directory tree for all licenses detected.&quot;<br>  private=false<br>}</pre><pre>module &quot;os2cw&quot; {<br>  source = &quot;modules/github&quot;</pre><pre>repository_name=&quot;os2cw&quot;<br>  description=&quot;Push disk and memory metrics to cloudwatch.&quot;<br>  private=false<br>}</pre><pre>module &quot;wait4mysql&quot; {<br>  source = &quot;modules/github&quot;</pre><pre>repository_name=&quot;wait4mysql&quot;<br>  description=&quot;Waits for a MySQL server to respond to requests before exiting.&quot;<br>  private=false<br>}</pre><p>All of that from using a simple curl/jq command.</p><p>If you copy that into your Terraform configuration, then all you’re left to do is import the resources. Below is the curl/jq statement to accomplish that.</p><pre>curl -s &quot;<a href="https://api.github.com/users/denniswebb/repos?access_token=$GITHUB_TOKEN&amp;per_page=100">https://api.github.com/</a><a href="https://api.github.com/orgs/PotomacInnovation/repos?access_token=$GITHUB_TOKEN&amp;per_page=10&amp;page=1">orgs/<strong>&lt;Organization_Name&gt;</strong></a><a href="https://api.github.com/users/denniswebb/repos?access_token=$GITHUB_TOKEN&amp;per_page=100">/repos?access_token=$GITHUB_TOKEN&amp;per_page=100</a>&quot; | jq -r &#39;sort_by(.name) | .[] | &quot;terraform import module.\(.name).github_repository.main \(.name)&quot;&#39;</pre><p>And there’s the import lines you can paste into your shell or add to a bash script.</p><h3>Conclusion</h3><p>The above should get you started in managing your GitHub repositories using Terraform. Being able to script the code makes the job a lot less of a undertaking. Expand on the basic module to make it work for you and your organization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a04ae24ad4b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devops-complete/importing-github-repositories-in-terraform-a04ae24ad4b2">Importing GitHub Repositories into Terraform</a> was originally published in <a href="https://medium.com/devops-complete">DevOps Complete</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[S3 Antivirus Scanning with Lambda and ClamAV]]></title>
            <link>https://medium.com/upside-engineering/s3-antivirus-scanning-with-lambda-and-clamav-7d33f9c5092e?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d33f9c5092e</guid>
            <category><![CDATA[antivirus]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Thu, 21 Sep 2017 13:44:59 GMT</pubDate>
            <atom:updated>2017-09-21T13:44:59.522Z</atom:updated>
            <content:encoded><![CDATA[<p>There have been many stories over the past months about S3 buckets being left unsecured. Terabytes of sensitive data available for the whole world to download. CyberSecurity 101 teaches, <em>“Don’t leave private data open to the public.”</em>, but somehow many major companies have let this one slip through the cracks.</p><p>Another lesson that everybody knows is, <em>“Never open a file that has not been scanned for viruses.”</em> So how do you protect yourself and scan the files stored on S3? Amazon doesn’t have a built-in antivirus tool for the task. At <a href="https://web.upside.com/?utm_source=medium&amp;utm_medium=medium-blog&amp;utm_campaign=EngineeringBlog&amp;utm_content=BSIUpsideDennis&amp;utm_promo=0">Upside</a>, we’ve created our own solution with Lambda and S3 events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*74lJOPyU4L_hrMQAt_-Cuw.png" /></figure><h3>Overview</h3><p>Using <a href="http://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-event-notifications.html">S3 Event Notifications</a>, a<a href="http://docs.aws.amazon.com/lambda/latest/dg/with-s3.html"> Lambda function is invoked</a> to scan the newly uploaded file. The function will download the object from S3 and scan it for viruses using the open-source antivirus software <a href="https://www.clamav.net/">ClamAV</a>. Once scanning is complete, the function will add 2 tags to the S3 object, av-status and av-timestamp. av-status can have a value of either CLEAN or INFECTED. S3 bucket policies prevent anybody from reading a file where the status is INFECTED.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*QVJad0fRCRmqgocg82UfxA.png" /></figure><h3>The Details</h3><h4>Working with ClamAV and Lambda</h4><p>If you’ve ever worked with Lambda, you’re probably wondering how we got ClamAV installed. Lambda functions are simply a <a href="https://en.wikipedia.org/wiki/Zip_(file_format)">ZIP archive</a> containing your source code (Python for our scanner) and any other files required by your function. When we are making our deployment archives for the antivirus function, we also include the binaries for ClamAV. To know the binaries will work on Lambda, we leverage the <a href="https://hub.docker.com/_/amazonlinux/">amazonlinux Docker image</a> to download the RPMs using yum. After downloading, we only include the required files: <a href="https://linux.die.net/man/1/clamscan">clamscam</a>, <a href="https://linux.die.net/man/1/freshclam">freshclam</a>, and libclamav.so.7.</p><p>We do not include the definition files in our Lambda archives as those are over 100MB in size and become outdated quickly. Instead, we store the most current version of the definitions in a separate secure S3 bucket for fast downloading. The definitions are updated using freshclam from a separate Lambda that runs every hour from a <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html">CloudWatch Event</a>. Another benefit of a bucket for definitions is all 4 of our AWS accounts use a single bucket for retrieving definitions.</p><h4>S3 Configuration</h4><p>There are 2 configuration steps for S3. One is the event to invoke the scan, and the other is a bucket policy that prevents accessing files tagged as infected.</p><p>S3 events can be configured in the AWS S3 console under bucket properties. We scan on all ObjectCreate events. AWS will handle assigning permissions for S3 to invoke the antivirus lambda function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*azoCRZoVa66VcvFW4W8vEg.png" /></figure><p>Configuring <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html">bucket policies</a> is also done in the S3 console. We add the following to all of our policies to prevent anybody from accessing files tagged infected. It is also recommended to not allow any of your users to modify the object tags with a name of av-status.</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Deny&quot;,<br>      &quot;Action&quot;: [&quot;s3:GetObject&quot;, &quot;s3:PutObjectTagging&quot;],<br>      &quot;Principal&quot;: &quot;*&quot;,<br>      &quot;Resource&quot;: [&quot;arn:aws:s3:::my-bucket/*&quot;],<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;s3:ExistingObjectTag/av-status&quot;: &quot;INFECTED&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><h4>SNS Notification</h4><p>The Lambda has optional support for SNS notification of scan results for every file scanned. This has many uses such as notifying a service that a new file has been received and marked as clean. Another use is triggering another Lambda function to handle infected files, such as moving to a quarantine bucket or deleting. We created the scanner with a single purpose and decided that different buckets would want different actions taken after scanning. SNS allows us to separate these responsibilities.</p><h4>DataDog</h4><p>DataDog metrics and events are published if the DATADOG_API_KEY environment variable is configured. DataDog is able to alert our NOC when infected files are received and to alert them to an attack if many infected files are being uploaded in a short period of time.</p><h3>In Action</h3><p>Our back-end customer support software relies on the antivirus status tags when sharing files and photos with customers. During chat conversations, both customers and our support staff, referred to as Navigators, can share files to assist in communication. These attachments are placed in a S3 bucket and are only made accessible to the users after being scanned and confirmed to be safe. For the record, users upload and download the files from S3 by using <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html">pre-signed URLs</a>. This allows access to the specific files for a limited time and prevents others from accessing the content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsBVpWmbWUO9dlvteJfktQ.png" /></figure><h3>Large Object Limitation</h3><p>There is one limitation with this setup. <a href="http://docs.aws.amazon.com/lambda/latest/dg/limits.html">AWS Lambda limits</a> your function to 512MB of /tmp storage space. Since the S3 object must be copied locally before being scanned, if the object is &gt; 400MB (ClamAV definitions are over 100MB in size), then the scan job will fail. For our use case, this size limitation has not been an issue.</p><h3>Open Source</h3><p><a href="https://web.upside.com/?utm_source=medium&amp;utm_medium=medium-blog&amp;utm_campaign=EngineeringBlog&amp;utm_content=BSIUpsideDennis&amp;utm_promo=0">Upside</a> Travel is releasing this solution under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2 license</a>. Full source is available from <a href="https://github.com/upsidetravel/bucket-antivirus-function">GitHub</a>. You will find more detailed installation instructions there to help with deployment into your own environment. If you have an improvement you’d like to contribute, please submit a pull request.</p><p>If this post piqued your interest, check out our open positions at Upside and apply today: <a href="http://upsd.io/2wP1OSJ">http://upsd.io/2wP1OSJ</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d33f9c5092e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/upside-engineering/s3-antivirus-scanning-with-lambda-and-clamav-7d33f9c5092e">S3 Antivirus Scanning with Lambda and ClamAV</a> was originally published in <a href="https://medium.com/upside-engineering">Upside Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running AWS CLI Commands from Terraform Enterprise]]></title>
            <link>https://medium.com/devops-complete/running-aws-cli-commands-from-terraform-enterprise-9782d30e6447?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/9782d30e6447</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Tue, 12 Sep 2017 16:50:59 GMT</pubDate>
            <atom:updated>2017-09-12T16:51:01.549Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re not familiar with <a href="https://www.hashicorp.com/products/terraform/">Terraform Enterprise</a>(formerly known as Atlas), it <a href="https://www.hashicorp.com/">HashiCorp’s</a> hosted solution to allow easy collaboration between teams using <a href="https://terraform.io/">Terraform</a> to provision their infrastructure. Commit your Terraform templates to GitHub, and Terraform Enterprise pulls down your code and automatically runs terraform plan against it. All changes are presented with the option to <strong>Apply</strong> or <strong>Discard</strong> the changes. It also offers more features such as <a href="https://www.terraform.io/docs/enterprise/state">Remote State Storage</a> and an <a href="https://www.terraform.io/docs/enterprise/artifacts">Artifact Registr</a>y.</p><p>Like all hosted solutions, you lose some flexibility with Terraform Enterprise. The virtual machine that Terraform Enterprise uses for plans and applies is a very minimal Linux installation. A lot of the tools you have on your machine such as <a href="https://www.python.org/">Python</a> and the <a href="https://aws.amazon.com/cli/">AWS CLI </a>are not available. This isn’t an issue if every resource you manage with Terraform is fully supported by a provider. If you’re like me though, you’ve come to rely on the <a href="https://www.terraform.io/docs/provisioners/null_resource.html">null_resource provider</a> when you need to add unsupported functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eyyuJ-Svck4NsmJHUNiUjg.png" /><figcaption>Terraform Enterprise Plan Output</figcaption></figure><p>Recently I was assigned a ticket to create 2 new SNS mobile endpoints for iOS and Android for our application. Like all infrastructure, these would be provisioned and managed using Terraform. Unfortunately Terraform currently has no support for <a href="http://docs.aws.amazon.com/sns/latest/dg/SNSMobilePush.html">AWS SNS Mobile Push</a>. When faced with this issue, the common solution is to use a <strong>null_resource</strong> to create the resource using the AWS CLI. As stated in the intro, since we use Terraform Enterprise, I first need to make the AWS CLI available.</p><p>It sounds real simple to just download the install package and install it during the <strong>apply</strong> phase. And while that will work if you are only trying to use it for a single resource, if you have multiple <em>null_resources</em> relying on the CLI, they can conflict with each other and cause issues. To prevent conflicts, I install the cli separately for each resource. Each installation is done into a temporary folder named /tmp/${uuid()} where <a href="https://www.terraform.io/docs/configuration/interpolation.html#uuid-">uuid()</a> is the built-in <a href="https://www.terraform.io/docs/configuration/interpolation.html">Terraform function</a>. This allows an almost guaranteed unique location like /tmp/d8d36c50–155b-9389-da21–407e79b40d44<em>. </em>I store this directory name in a bash variable named WORKDIR.</p><p>The next step is to download the AWS CLI installation bundle into WORKDIR. After extracting the contents, install using install -i &quot;$WORKDIR&quot;/aws to specify that the installation occurs into our unique temporary location.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfa4629c549f984924b902eec6bae46e/href">https://medium.com/media/dfa4629c549f984924b902eec6bae46e/href</a></iframe><p>Once the CLI is installed and available, the null_resource is able to execute &quot;$WORKDIR&quot;/aws/bin/aws to use the commands. As you can see above, I have a module created solely for installing the CLI. When building resources, I call the rendered text from that module for each null_resource as show below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10c06624a04f2f772be8f2072677158d/href">https://medium.com/media/10c06624a04f2f772be8f2072677158d/href</a></iframe><p>The installation will only happen during the creation of the resource, not with every run of terraform plan or apply.</p><p>I know this information is only useful to a small percentage of Terraform users, I wanted to share to hopefully help somebody. This technique can be used for installing other utilities that are required when using Terraform Enterprise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9782d30e6447" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devops-complete/running-aws-cli-commands-from-terraform-enterprise-9782d30e6447">Running AWS CLI Commands from Terraform Enterprise</a> was originally published in <a href="https://medium.com/devops-complete">DevOps Complete</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Custom DataDog Agent Check]]></title>
            <link>https://medium.com/devops-complete/writing-a-custom-datadog-agent-check-7367c98ffc5a?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/7367c98ffc5a</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[openvpn]]></category>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Fri, 25 Aug 2017 22:15:14 GMT</pubDate>
            <atom:updated>2017-08-28T12:57:25.705Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*McPwOS7qWVUoehP7YGcsjQ.png" /></figure><p><a href="https://www.datadoghq.com/">DataDog</a> users already know they have agent checks available for a variety of different products and services. But what if there’s something you need to monitor that’s not currently supported? Then it’s time to write your own.</p><p>Complete source from this article available from <a href="https://github.com/denniswebb/datadog-openvpn">https://github.com/denniswebb/datadog-openvpn</a></p><h4>Use Case</h4><p>I have multiple customers using <a href="https://openvpn.net/index.php/access-server/overview.html">OpenVPN AS</a> from the <a href="https://aws.amazon.com/marketplace/seller-profile/ref=srh_res_product_vendor?ie=UTF8&amp;id=aac3a8a3-2823-483c-b5aa-60022894b89d">AWS </a>Marketplace for remote access to the AWS VPC. It’s a great solution because setting up a server requires very little configuration, and the client software is simple to setup on both Windows and Mac clients. When launching an instance, you choose how many licenses you need. Of course as customers grow, so does the number of licenses used. Monitoring available licenses is a must because users not being able to connect because they unexpectedly outgrew the connection count is just poor management. To prevent this, I wanted a DataDog alarm setup to warn us when available licenses crossed a threshold.</p><h4>Before We Start Writing Code</h4><p>The first step was discovering an easy way to get the license statistics. The sacli<strong> </strong>command has a parameter named LicUsage<strong> </strong>that does that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*HMtsy-2YC6EN-treo4ETPQ.png" /><figcaption>sacli LicUsage showing 30 of 50 licenses used</figcaption></figure><p>The command outputs a string that represents a list. The first element is the number of licenses in use, and the second is the total number of connection licenses. That’s all the info we need as a basic subtraction will give us the number of available licenses.</p><h4>Writing the Code</h4><p>DataDog Agent Checks are made up of 2 files, &lt;project_name&gt;.py and &lt;project_name&gt;.yaml. These files should be saved in /etc/dd-agent/checks.d and /etc/dd-agent/conf.d respectively.</p><p>The first file we’ll create is /etc/dd-agent/conf.d/openvpn.yaml. Agent config files have 2 sections: init_config and instances. Below is our configuration file where we’ll set 2 parameters in the init_config section. metric_prefix will be the prefix for the metric name and sacli_path is the full path to the sacli tool.</p><p>The instances section defines one or more instances to check. If we were reporting file sizes, the ability to specify more than one file would be nice. This is what this section is good for. Since it is required, I put a single empty object to satisfy the requirement.</p><pre><em>init_config:<br></em>  metric_prefix: openvpn<br>  sacli_path: /usr/local/openvpn_as/scripts/sacli<br><br>instances:<br>    [{}]</pre><p>Now we’ll create the actual agent file in /etc/dd-agent/checks.d/openvpn.py. The file extension already tells you this is a <a href="https://www.python.org/">Python</a> module. Start by adding the imports section and declaring our new class that comes from DataDog’s AgentCheck module.</p><pre>from checks import AgentCheck</pre><pre>class OpenVPN(AgentCheck):<br>    def check(self, instance):</pre><pre>if __name__ == &#39;__main__&#39;:<br>    check.check(instance)</pre><p>This is boilerplate that does nothing yet. The class’s check function is what’s called by the DataDog service regularly. To use the parameters in our configuration file, we call init_config.get. This returns the stored value, and if none is found, the second parameter is the fallback value.</p><pre>def check(self, instance):<br><strong>    metric_prefix = self.init_config.get(&#39;metric_prefix&#39;, &#39;openvpn&#39;)</strong></pre><p>Next up we’ll add a function to return a dictionary of the various OpenVPN license usage statistics. For this we’ll use Pythons ast and subprocess<strong> </strong>modules. Add these import lines to the top of your file. The check_output<strong> </strong>function will return the output of sacli LicUsage, while ast will be used to convert the stringified list outputted to a real Python list.</p><pre>import ast<br>from subprocess import check_output</pre><p>The get_license_usage function is added to the OpenVPN<strong> </strong>class. The function returns a dictionary of the license usage statistics returned by sacli<strong>. </strong>You’ll notice I used sudo to run sacli with elevated permissions. Unfortunately the command must be run with root privileges to work. Later i will cover adding the dd-agent user to /etc/sudoers to allow this.</p><pre>def get_license_usage(self):<br>    sacli_path = self.init_config.get(&#39;sacli_path&#39;, &#39;/usr/local/openvpn_as/scripts/sacli&#39;)<br>    lic_usage = ast.literal_eval(check_output([&quot;sudo&quot;, sacli_path, &quot;LicUsage&quot;]))<br>    return {<br>        &#39;used&#39;: lic_usage[0],<br>        &#39;total&#39;: lic_usage[1],<br>        &#39;available&#39;: lic_usage[1] - lic_usage[0]<br>}</pre><p>Adding more functionality to our check function, we add a variable to hold the results of the get_license_usage function. The final 3 lines call the AgentCheck<strong> </strong>class’s gauge function to push the 3 metrics to DataDog. The format is gauge(&lt;metric.name&gt;, &lt;metric-value&gt;). In addition to <em>gauge</em> type, DataDog also support <em>count</em> and <em>increment</em>. For a complete list of metrics you can send, see <a href="https://help.datadoghq.com/hc/en-us/articles/206955236-Metric-types-in-Datadog">this article</a>. When submitting metrics you can add tags by using the named parameter tags and passing a list of tags to it.</p><pre>def check(self, instance):<br>    metric_prefix = self.init_config.get(&#39;metric_prefix&#39;, &#39;openvpn&#39;)<br><strong>    license_usage = self.get_license_usage()<br>    self.gauge(&#39;%s.licenses_used&#39; % metric_prefix, license_usage[&#39;used&#39;])<br>    self.gauge(&#39;%s.licenses_available&#39; % metric_prefix, license_usage[&#39;available&#39;])<br>    self.gauge(&#39;%s.licenses_total&#39; % metric_prefix, license_usage[&#39;total&#39;])</strong></pre><h4>The Completed Agent Check</h4><p>The complete project is available on GitHub at <a href="https://github.com/denniswebb/datadog-openvpn">https://github.com/denniswebb/datadog-openvpn</a></p><pre>import ast<br>from checks import AgentCheck<br>from subprocess import check_output<br><br>class OpenVPN(AgentCheck):<br><br>    def check(self, instance):<br>        metric_prefix = self.init_config.get(&#39;metric_prefix&#39;, &#39;openvpn&#39;)<br>        license_usage = self.get_license_usage()<br>        self.gauge(&#39;%s.licenses_used&#39; % metric_prefix, license_usage[&#39;used&#39;])<br>        self.gauge(&#39;%s.licenses_available&#39; % metric_prefix, license_usage[&#39;available&#39;])<br>        self.gauge(&#39;%s.licenses_total&#39; % metric_prefix, license_usage[&#39;total&#39;])<br><br>    def get_license_usage(self):<br>        sacli_path = self.init_config.get(&#39;sacli_path&#39;, &#39;/usr/local/openvpn_as/scripts/sacli&#39;)<br>        lic_usage = ast.literal_eval(check_output([&quot;sudo&quot;, sacli_path, &quot;LicUsage&quot;]))<br>        return {<br>                    &#39;used&#39;: lic_usage[0],<br>                    &#39;total&#39;: lic_usage[1],<br>                    &#39;available&#39;: lic_usage[1] - lic_usage[0]<br>                }<br><br>if __name__ == &#39;__main__&#39;:<br>    check.check(instance)</pre><h4>Final Steps</h4><p>We’re still not done. Remember we need to run sacli with elevated privileges. The easiest way to accomplish this is to add the following line to your /etc/sudoers file using the visudo tool. This allows dd-agent to run sacli as root without being prompted for a password.</p><pre>dd-agent ALL=NOPASSWD: /usr/local/openvpn_as/scripts/sacli</pre><p>We need to test that our custom check runs without giving errors. The command to test a DataDog check is sudo -u dd-agent dd-agent check openvpn. If no errors are returned, restart the DataDog agent with sudo /etc/init.d/datadog-agent restart.</p><p>You can find your 3 new metrics in the DataDog app as openvpn.licenses_available, openvpn.licenses_used, and openvpn.licenses_total.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*j27BwyvhUiZ8pj3n.png" /></figure><p>And that’s all there is to it. I hope this tutorial helps anybody looking to create their own DataDog agent check. DataDog has an article titled <a href="https://docs.datadoghq.com/guides/agent_checks/">Writing an Agent Check</a> that has another example. Also be sure to discover the different types of metrics you can send along with events.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7367c98ffc5a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devops-complete/writing-a-custom-datadog-agent-check-7367c98ffc5a">Building a Custom DataDog Agent Check</a> was originally published in <a href="https://medium.com/devops-complete">DevOps Complete</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I Learned — 2017/04/21]]></title>
            <link>https://medium.com/@denniswebb/what-i-learned-2017-04-21-c3e0725053ee?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3e0725053ee</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[vim]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Sat, 22 Apr 2017 04:41:42 GMT</pubDate>
            <atom:updated>2017-04-22T04:41:42.022Z</atom:updated>
            <content:encoded><![CDATA[<h3>Yarn with Root</h3><p>After updating our Docker build process to use <a href="https://yarnpkg.com/">yarn</a> v0.23.2, we started getting 401 errors when pulling from <a href="https://www.jfrog.com/artifactory/">Artifactory</a>. The issue was open on GitHub for a week, and although others had the same issue, nobody had a solution.</p><p>Today I decided to take the time to investigate the problem. Turns out that when yarn is ran by root, it users /usr/local/share as root’s home instead of /root. This caused yarn to fail authentication with Artifactory since our .npmrc file was located in /root/.npmrc. The solution for us was to simply place the file in the root of the project before telling yarn to pull down packages during our docker build process.</p><h3>base16 Color Schemes</h3><h4>.Xresources</h4><p>Copy the 256 scheme from <a href="https://github.com/chriskempson/base16-xresources">base16-xresources</a> to your ~/.Xresources</p><h4>base16-shell</h4><p>To get 256 working correctly with URxvt and other terminals, install the scripts from <a href="https://github.com/chriskempson/base16-shell">base16-shell</a>. Be sure to add the scripts to your shell startup script.</p><p>Also includes the useful colortest script at ~/.config/base16-shell/colortest</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*DWpo1yZC34vAteH1Uqr-OQ.png" /><figcaption>colortest output</figcaption></figure><h4>base16-vim</h4><p>Get your vim colors right by adding this vim plugin and configuration.</p><pre>call plug#begin(&#39;~/.local/share/nvim/site/plugged&#39;)<br>Plug &#39;chriskempson/base16-vim&#39;<br>call plug#end()</pre><pre>let base16colorspace=256<br>colorscheme base16-default-dark</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3e0725053ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building fio from Source on Amazon Linux]]></title>
            <link>https://medium.com/devops-complete/building-fio-from-source-on-amazon-linux-68e0ac0bb95b?source=rss-ff8b05bce729------2</link>
            <guid isPermaLink="false">https://medium.com/p/68e0ac0bb95b</guid>
            <category><![CDATA[fios]]></category>
            <category><![CDATA[be]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[storage]]></category>
            <dc:creator><![CDATA[Dennis Webb]]></dc:creator>
            <pubDate>Mon, 17 Apr 2017 21:20:43 GMT</pubDate>
            <atom:updated>2017-04-17T21:20:43.199Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve recently been doing some storage performance tests on AWS EC2 instances. <a href="https://github.com/axboe/fio">Fio</a> has been my primary tool for this testing. Fio can be installed inside of Amazon Linux using sudo yum install -y fio, but the version provided is out of date. This short article will show you how to build fio from source instead.</p><h3>Building fio from source</h3><h4>1. Install build tools</h4><p>To build software from source on Linux, you need a few tools to help you with this (gcc, make, git, etc.). The easiest way to install these tools is to install the <em>Development Tools</em> yum group.</p><p>sudo yum groupinstall “Development Tools&quot; -y</p><h4>2. Install libaio sources</h4><p>Fio also requires the libaio header files to build. To install the sources to almost any package append -devel to the package name. For libaio the command is:</p><p>sudo yum install libaio-devel -y</p><h4>3. Clone master repository</h4><p>The source files for fio are located on multiple mirrors, but the master git repo location is git://git.kernel.dk/fio.git. Clone this repo to your instance with:</p><p>git clone git://git.kernel.dk/fio.git</p><h4>4. Configure, make, make install</h4><p>The build process itself is handled via a configure script and Makefile. You don’t need to know any details about this. Running the 3 commands will build the binaries and install them into /usr/local/bin. Configure and make can be ran by any user, but make user will need write permissions to /usr/local/bin.</p><pre>cd fio<br>./configure<br>make<br>sudo make install</pre><h4>5. Verify installation success</h4><p>The easiest way to verify that everything worked is to run fio -version.</p><pre>fio -version<br>fio-2.19-12-gb94d</pre><h4>Complete bash script</h4><pre>#!/usr/bin/env bash</pre><pre>sudo yum groupinstall <strong>&quot;Development Tools&quot; </strong>-y<br>sudo yum install libaio-devel -y<br>git clone git://git.kernel.dk/fio.git<br>pushd fio<br>./configure<br>make<br>sudo make install<br>popd<br>rm -rf fio<br>fio -version</pre><h3>Basic Usage</h3><p>Using fio is a lot more involved than building it. Below is a sample command to get you started. Check out the fio manpage for more information.</p><h4>To test random read/write performance</h4><p>fio --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --iodepth=32 -- size=4G --readwrite=randrw --rwmixread=75</p><p>This will perform a test of the storage where the file specified with --filename flag. You might want to play with the value of --rwmixread to specify the percentage of reads vs. writes. In this example, 75% of the operations will be reads and 25% writes.</p><h3>Conclusion</h3><p>There’s a lot to learn about using fio. I have a lot to use myself. I’m planning on posting a few articles on my fio results with various AWS storage configurations in the upcoming days.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68e0ac0bb95b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devops-complete/building-fio-from-source-on-amazon-linux-68e0ac0bb95b">Building fio from Source on Amazon Linux</a> was originally published in <a href="https://medium.com/devops-complete">DevOps Complete</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>