<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mike Perttula on Medium]]></title>
        <description><![CDATA[Stories by Mike Perttula on Medium]]></description>
        <link>https://medium.com/@mperttula?source=rss-bd6cbd98fbb1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sLcvxlLGt99rzowDsoNi_w.jpeg</url>
            <title>Stories by Mike Perttula on Medium</title>
            <link>https://medium.com/@mperttula?source=rss-bd6cbd98fbb1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:45:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mperttula/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Compliance at Velocity with AWS Config]]></title>
            <link>https://medium.com/signiant-engineering/compliance-at-velocity-with-aws-config-8edcc3a3d04d?source=rss-bd6cbd98fbb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8edcc3a3d04d</guid>
            <category><![CDATA[amazon-ec2]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[configuration-management]]></category>
            <category><![CDATA[config]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Mike Perttula]]></dc:creator>
            <pubDate>Tue, 13 Feb 2018 14:37:18 GMT</pubDate>
            <atom:updated>2018-02-20T14:11:14.475Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*XmL6XoaeClJ6wfwrAB2TSA.jpeg" /><figcaption>..in most cases</figcaption></figure><p>When it’s time to lay down the law, you better make sure you close the loopholes. This is what we had to do recently when we noticed that our <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html">EC2 Security Groups</a> were skirting compliance in our development environments. The rule in this case is don’t allow any open internet connections (0.0.0.0/0) on SSH port 22 or RDP port 3389. Any service or developer wanting an instance to be accessible via SSH or RDP had to make sure that the connection was locked down to a known <a href="https://www.signiant.com/">Signiant</a> network. We found that there were outlaws skirting these rules and thats where we sprang into action.</p><h4>Creating Compliance</h4><p>At re:Invent in 2014, AWS introduced the <a href="https://aws.amazon.com/config/">Config service</a>. The service works as watchdog, auditing changes in your AWS account and alerting on whether your service configurations fall into a compliance with the standards of the environment. This is determined by rules that you create to track down what you consider compliant. A couple of years ago we decided to make use of this service to lock down EC2 Security to groups in an automated fashion. As mentioned earlier, we wanted to make sure that anyone creating Security Groups would need to make sure that SSH and RDP was locked down. To do this we created 2 config rules.</p><p>First we would tackle SSH. The setup of a rule is trivial with AWS Config’s libary of Managed Rules. When Clicking the “Rules” link in the Config dashboard and choosing “Add rule”, we are given the option of viewing managed rules. These rules are provided by AWS as common configuration audits us to use. There is already a restricted-ssh option for us to use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*x6MYzU5W7f_Ns8G7uwPuxA.png" /><figcaption>A library of rules at your fingertips</figcaption></figure><p>Choosing this rule gives us everything we wanted and will alert on non-compliance should a Security Group be created with a wide open SSH inbound rule. The default behaviour for the rule’s frequency is to check for compliance every time a configuration change occurs on a Security Group within the account. From here we need a compliance officer to action non compliant groups and making everything on the level.</p><h4>Reaping Rewards</h4><p>The offenders have been identified by AWS Config but we still need to action that intel. AWS documentation suggests <a href="https://docs.aws.amazon.com/config/latest/developerguide/notifications-for-AWS-Config.html">streaming config changes to SNS</a> and then subscribing to that topic via email or any other alerting method of your choice. This is a good method to be informed of the issue but we needed to go further and handle the alert in automated fashion, shutting down the rule. For that we created a solution called the <a href="https://github.com/Signiant/aws-config-rule-port-reaper">AWS Config Rule Port Reaper</a>, a docker container that is ready to do the dirty work of shutting down rogue open ports. It is scheduled to run periodically in each of our AWS accounts, finding out what resource isn’t compliant and then removing the rule in question based on the type of infraction.</p><p>We had originally developed this before Lambda was being widely used at Signiant. Today this would most likely be a Lambda function from the get go, weeding out trouble and returning us to a fairly secure state.</p><h4>Catching Them All</h4><p>Until recently everything was working well with our security group Compliance. Users were more mindful on what ports they left opened and we rarely had the reaper doing working. But then we found a loophole.</p><p>We received an alert that the reaper ran and that there was a Security Group not in compliance. We expected to the Reaper to work as per normal but something was amiss. We continued to receive warnings about the same security group being not in compliance for SSH. This should have been handled by our ‘Lawbringer’ but it was not making the cut. Taking a closer look at the security group we found not every case was being considered. The group looked as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*uG6hyo9M1RlAJZt8mex6pQ.png" /><figcaption>The Loophole</figcaption></figure><p>SSH was wide open, and the Config rule had deemed it not compliant. It seems our Reaper tool was not robust enough to handle a port range over a port that was being monitored. Further to this, we had used the managed AWS config rule, <a href="https://console.aws.amazon.com/config/home?region=us-east-1#/rules/rule-details/restricted-rdp">restricted-rdp, </a>to alert on RDP ports that open to the web. We learned this rule did not take into account that RDP can be over TCP and UDP, alerting only on TCP cases. The TCP security group rule was squashed by the reaper but the UDP rule was left behind in a state of non conformity. The oversight in the Reaper would require a quick change in checking for a range condition, but the restricted-rdp over UDP was going to require a Custom Rule.</p><p>There are a few ways to create a custom Config rule. You can go the manual route where you follow the AWS Config console creation wizard which includes you developing a Lambda from scratch ( or using an existing one ). It has to handle the Config event that gets fired by the rule for checks. The event includes a “Configuration Item” object that will includes all the configuration of a single AWS Resource in json. In our case it’s a Security Group configuration. As well you will be able to pull any parameters specified in the Config rule parameters section that can optionally be used with the evaluating code. The Lambda must return either COMPLIANT, NON_COMPLIANT, or NOT_APPLICABLE (for those times you are way off in your evaluation). Figuring out a manual deploy and linking the ARN back to the rule is cumbersome and we also have to think about how we are going to roll this out to multiple accounts and multiple regions. Luckily I was able to come across a speedy solution to my woes.</p><h4>Ruling at Velocity</h4><p>The stranger that came into town to clean up the mess ended up being the <a href="https://github.com/awslabs/aws-config-rdk">AWS Config Rules Development Kit</a> provided by <a href="https://github.com/awslabs">AWS Labs</a>. The python based tool, available on to install on pip, allows you to ramp up rule creation in a flash. Once installed, you initialize your working directory where you’ll be writing the rule which also prepares an S3 bucket for the region you’ll be deploying the rule:</p><pre>rdk -p dev -r us-east-1 init</pre><p>Your lambda code will also be stored in this bucket. Next you create a rule:</p><pre>rdk -p dev create restrict-rdp-udp --runtime python2.7 --resource-types AWS::EC2::SecurityGroup --input-parameters &#39;{&quot;PORT&quot;:&quot;3389&quot;}&#39;</pre><p>This automatically generates stub code in my language of choice (java, nodejs, or python), in this case python, and will be checking the SecurityGroup resource type. We also introduce the PORT input parameter. All that’s left to do is to look through the Configuration object and parameters to determine what I think would be complaint. The stub has a section in the evaluate_complaince method for you to determine what is complaint vs non complaint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*IT9KvqLm2Q2emsN6piOhxQ.png" /><figcaption>Stub code generated by the RDK</figcaption></figure><p>The code that I ended up using is detailed in my <a href="https://bitbucket.org/signiant/custom_config_rules">Custom Config Rule repository</a>. You can test your code with an example resouce, test_event_template.json, that they provide. Once you’re satisfied, it’s time to deploy:</p><pre>rdk -p dev -r us-east-1 deploy restrict-rdp-udp</pre><p>This command uploads a compressed Lambda to the initialized Config bucket, and runs a CloudFormation Template to create the rule and deploy the Lambda. The use of this suite help me with the majority of the heavy lifting and the deploy was very fast and repeatable to multiple regions and/or accounts. I had started with the manual processes but easily abandoned it once I found this tool.</p><p>The AWS Config RDK is considered to be in “Open Beta” right now so you may want to evaluate it for your purposes. What’s locked down in the Lambda role creation may be something you want to review for a production environment. Otherwise I recommend it for the speed of compliance it allowed me for something that was missing.</p><p>A future tool I plan to evaluate is <a href="https://github.com/iagcl/watchmen">Watchmen</a>, which allows you to centrally manage Config Rules in one account instead of having to deploy them to multiple regions and accounts. This looks like an ideal solution to speed up where your rules are deployed saving you some time on making sure your environments are on lockdown.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8edcc3a3d04d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/compliance-at-velocity-with-aws-config-8edcc3a3d04d">Compliance at Velocity with AWS Config</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Highly Available Downloads Across Clouds]]></title>
            <link>https://medium.com/signiant-engineering/highly-available-downloads-across-clouds-23a9ff6df03d?source=rss-bd6cbd98fbb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/23a9ff6df03d</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[replication]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Mike Perttula]]></dc:creator>
            <pubDate>Tue, 04 Jul 2017 18:24:23 GMT</pubDate>
            <atom:updated>2017-07-04T18:24:23.246Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*yzIbm1pw19IMuhJVXA8txQ.jpeg" /></figure><p>We’re always told never to put all your eggs in one basket. If you have used Amazon’s Simple Storage Service (S3) in the past year then you may know why. On the morning of February 28th, 2017 Amazon had a large outage in their S3 subsystem that handles all their GET, LIST, PUT, and DELETE requests in the US-East-1 region. If you used Object storage in that region you were out of luck until the outage was resolved which, in this case, took hours. At Signiant, we have a redundancy strategy for our user-facing web applications but during the S3 outage, a flaw was highlighted in how we distribute our installable components. We went into into action to address this.</p><p>In doing our post mortem we determined our ability to distribute software and binaries to customers can be affected by an S3 outage. S3 has its hooks in much of the functionality of AWS and specifically Cloudfront, Amazon’s CDN solution, was affected by the outage. We use Cloudfront to serve up a number of our software assets such as the <a href="https://shuttle.support.signiant.com/customer/portal/articles/2469753-download-signiant-app">Signiant App</a> (which allows customers to make use of our acceleration technology in <a href="https://www.signiant.com/products/media-shuttle/?cc=homepage">Media Shuttle</a>). Not having new versions available, or being possibly offline if nothing could be served by Cloudfront, is not a risk we wanted to take.</p><h3>X - CDN</h3><p>The simplest way to make sure you are highly available for object storage is to make sure assets are served from more than one region. This is a staple at Signiant and we already have these assets served in multiple regions. With the S3 outage though, we found that all of Cloudfront was unavailable for a period of time so we decided to go cross CDN to ensure our object storage for downloadable assets is not dependant on a single CDN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*A_N0cEv6l_xXfDaH9eNv9w.png" /></figure><p>Signiant has a great relationship with AWS but also with Azure. Our <a href="https://www.signiant.com/products/media-shuttle/?cc=homepage">Media Shuttle</a> and <a href="https://www.signiant.com/products/flight/?cc=homepage">Flight</a> products both allow uploading and downloading from both AWS and Azure cloud storage. It was a no brainer in using Azure for our cross cloud redundancy given we already run infrastructure there today. Azure’s CDN service makes use of Akamai and Verizon for serving content so we get a bonus of possibly using up to 3 CDNs at one time if needed.</p><h3>Object storage Copy</h3><p>When releasing our downloadable software, we make use of our <a href="https://medium.com/signiant-engineering/a-clone-army-of-the-jenkins-republic-4b6568caf33">Jenkins </a>promotion tools to get the downloadable artifacts to S3. From there Cloudfront will serve them to its edge nodes for end user download. Instead of investing time in modifying the promotion tools to also upload to Azure blob storage, we created a small service called the “CDN Replication Service”. This service polls an Amazon SQS Queue for changes to an s3 bucket and uploads them to Azure blob storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*CugtJsKiPo3QA2GF6DVWHw.png" /><figcaption>S3 Events to SQS</figcaption></figure><p>S3 has a handy ability to send S3 events such has PUT/GET/COPY/DELETE to multiple Amazon Services. By forwarding the upload events to a queue, our replication service is free to pull from that queue and replicate files as they are added. Since our downloadable artifacts are public, we can send a command to Azure’s <a href="https://docs.microsoft.com/en-us/rest/api/storageservices/copy-blob">Copy Blob</a> REST call to pull the new or changed files directly from S3 to a storage location on Azure without incurring any middleman transfer charges from the service. Any transfer failures are left on the queue to retry on the next poll.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*J-eYh-zCY-tON9teQisQIg.png" /></figure><p>Currently the service only handles new or changed content. A future enhancement will be to handle messages related to deletion (not required at this time as we don’t prune old versions often).</p><h3>How available do you need to be?</h3><p>Our failover strategy for downloadable assets may not be for everyone. Here are some things to consider when determining your failover strategy:</p><ul><li>Cost : Is hosting across multiple CDNs and cloud storage worth the return you will receive in availability for customers?</li><li>Complexity : Are you adding additional layers of complexity and another point of failure with your HA solution?</li></ul><p>Any cloud solution is bound to have downtime. In Signiant’s case the cost and complexity was minimal and we felt adding this change does provide enough benefit so that our software is available for download 24/7.</p><p>If you are interested in our solution, you can find s3queue2blob code on <a href="https://github.com/Signiant/s3queue2blob">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23a9ff6df03d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/highly-available-downloads-across-clouds-23a9ff6df03d">Highly Available Downloads Across Clouds</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>