<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jorge Quintás on Medium]]></title>
        <description><![CDATA[Stories by Jorge Quintás on Medium]]></description>
        <link>https://medium.com/@jorgeqr?source=rss-877973a05568------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sduFwRJZ_ZWPH0rziIKfGg.jpeg</url>
            <title>Stories by Jorge Quintás on Medium</title>
            <link>https://medium.com/@jorgeqr?source=rss-877973a05568------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:06:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jorgeqr/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fostering security with Hashicorp Vault]]></title>
            <link>https://medium.com/jobandtalenteng/fostering-security-with-hashicorp-vault-4e45ec4eb399?source=rss-877973a05568------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e45ec4eb399</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[vault]]></category>
            <dc:creator><![CDATA[Jorge Quintás]]></dc:creator>
            <pubDate>Wed, 28 Feb 2018 10:30:31 GMT</pubDate>
            <atom:updated>2019-05-07T13:36:23.084Z</atom:updated>
            <content:encoded><![CDATA[<p>A standard monolithic stack uses a couple of secrets such as database passwords, certificates, tokens, Cloud Service Provider credentials, etc. But when using microservices you might end up having a lot more secrets, increasing not only the amount of work needed to carefully manage them but also the risks of getting into trouble in case the work is not properly done.</p><p>Ask yourself the following questions:</p><ol><li>Do you know which users have access to each secret?</li><li>Do you have a policy regarding the secrets’ life cycle? Do you enforce that policy?</li><li>Can you trace when a secret has been accessed (read, written, renewed, etc.) and by whom?</li><li>Do you have plaintext secrets in your VCS?</li></ol><p>If your answer was <em>no</em> to any of the first three questions and/or <em>yes</em> to the last one, this post is for you. If not, congratulations on your good work! But you might find it interesting regardless.</p><p>We’ve chosen Hashicorp Vault (Vault from now on) to help us follow the best practices for handling secrets, and to improve the overall security of our platform without having more workload every time the number of secrets we handle increases.</p><h3>So, what is Vault?</h3><p>Vault is a tool that provides safe access to secrets. It’s formed by multiple backends (with some glue between them) responsible for specific tasks.</p><p>The Vault’s main components are: <em>storage</em>, <em>secret</em>, <em>audit</em> and <em>auth</em> backends.</p><h3>Storage backend</h3><p>This backend is responsible for interacting with a <a href="https://www.vaultproject.io/docs/configuration/storage/index.html">large list of supported storage backends</a>. Storage is not trusted, so all the data leaving the Vault for storage will be encrypted. If you want to have a High Availability cluster of Vault check which storage backends support HA operation.</p><h3>Secret Backend</h3><p>Secret backends either store or generate secrets. Some secret backend examples are:</p><ul><li><strong>kv</strong>: stores key-value pairs</li><li><strong>database</strong>: (if correctly configured) can be used to dynamically generate users for multiple db engines.</li><li><strong>transient</strong>: can be used to easily encrypt/decrypt texts.</li></ul><p>Secrets are managed as if they were Virtual File Systems. You can:</p><ul><li><strong>Mount</strong>: so you could read or write the specified mount point</li><li><strong>Unmount</strong>: make the path unavailable, revoking all secrets belonging to the unmounted path</li><li><strong>Remount</strong>: to change the path, this operation revokes secrets belonging to the old path</li></ul><p><a href="https://www.vaultproject.io/docs/secrets/index.html">Full list of secret backends</a></p><h3>Audit Backend</h3><p>Audit Backends are responsible for tracing every interaction with Vault. Sensitive information will appear hashed with a salt using HMAC-SHA256, so analysis can be performed without leaking any secrets.</p><p>Multiple audit backends can be configured simultaneously, but bear in mind that at least one must be able to persist the data for the operation to be completed by Vault. If all audit backends are blocked (i.e. full disk when a local file is configured) Vault will remain unresponsive until the problem is fixed. This guarantees that no operation is done without leaving a trace.</p><h3>Auth backend</h3><p>Vault supports several authentication methods that cover a broad set of use cases.</p><p>You can find the full list <a href="https://www.vaultproject.io/docs/auth/index.html">here</a>.</p><p>Some of them are intended to authorize users while others are better suited being used by applications. For example, you might want to use your organization LDAP to authenticate users, but find it more convenient to use AWS’s instance profiles for your applications.</p><p>Once the authentication is successful, a token is returned to the client. This token could be understood as a session cookie; this way Vault’s client will be able to use it on your requests. If you are using the API instead of the binary client, you should send said token in X-Vault-Token HTTP header. Furthermore, with a successful authentication, the applicable policies are added to the token.</p><p>Depending on your current setup you’ll need to find the most suitable authentication backends among the available ones. For now, the information to take into account is:</p><ol><li>If authentication is successful, you will receive a token that will be used in further requests.</li><li>A list of configured policies will be attached to the token defining the operations that can be done while using it.</li></ol><p>All users who are granted authentication will have a default policy with limited capabilities, like management of their own token and access to a private secret endpoint called cubbyhole.</p><p>Policy files are described using <a href="https://github.com/hashicorp/hcl">HCL</a> format. A simple read-only policy for ansible mountpoint:</p><pre>path “ansible” {</pre><pre>capabilities = [“read”, “list”]</pre><pre>}</pre><pre>path “ansible/*” {</pre><pre>capabilities = [“read”, “list”]</pre><pre>}</pre><p>Policies are deny-by-default, so you’ll have to explicitly grant access to the paths. A token with the above policy will have read-only access to <em>/ansible</em>.</p><p>Lists of policies can be attached to users or groups so that the effect of each policy is combined.</p><h3>Practical example: Our setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ozgN6agrYX_jKlmjVg-pLw.png" /><figcaption>Our Vault design, diagram made with <a href="https://cloudcraft.co/">https://cloudcraft.co</a></figcaption></figure><p>Now let’s get down to the fun part, a real life example: our own case scenario. We asked ourselves the 4 questions presented at the beginning of the post and didn’t like the answers. It’s not that we had the worst answers possible (4 wrong answers!), but we had enough wrong answers to realize that it was time to get back on track to better and sustainable security policies.</p><p>Our main concerns, other than what Vault provided straight out of the box, were:</p><p><strong>High Availability</strong>: our infrastructure is completely automated and it can scale without manual intervention. In case more capacity is needed, at least one Vault server must be up and running in order to send secrets to our <em>provisioners</em> (name we gave to the instances in charge of automatically deploy new servers from scratch to production ready)</p><p><strong>Disaster and Recovery</strong>: we store a copy of Vault secrets in S3. The files are encrypted using GnuPG, the recipients are the members of DevOps guild. In case of data loss in Vault, we have an internal tool that configures it back again (authorization methods, policies, and secrets).</p><h3>Steps</h3><p>We started with a small concept test using Vault in development mode and executed in docker, in order to get a feeling of what we had learned that far.</p><p>We had been using docker in production for more than 2 years, and not regarding what Vault’s <a href="https://www.vaultproject.io/guides/production.html">production hardening guide</a> said, we deployed Vault in one of our ECS clusters with the purpose of further learning, and to start testing in High Availability using DynamoDB as the storage backend. Remember that it’s the storage backend’s responsibility to provide HA capabilities (and so DynamoDB does)</p><p>For the final setup we decided to use EC2 instances behind an internal ABL. With this setup the ALB sees the active node up and the inactive one down (remember the return HTTP status 429 when <a href="https://www.vaultproject.io/docs/concepts/seal.html">unsealed</a>); which fits perfectly, since requests will be routed only to the active Vault instance. The HA in Vault is intended only to better the availability, not to increase capacity. By being assembled this way, just one Vault instance delivers requests while the rest are hot replicas. If the active node fails, one of the standby nodes becomes automatically active.</p><p>We have created a privileged IAM role assigned to Vault instances’ profile, which gives access to dynamoDB allowing Vault to configure itself. Once Vault is able to start, more restrictive permits may be assigned to that role. Bear in mind that Vault uses some values that may be acceptable by default, but that might not suit your needs for this particular case. Review DynamoDB read and write capacities before production use.</p><p>We took all these steps using <a href="https://www.terraform.io/">Terraform</a> and <a href="https://www.ansible.com/">Ansible</a>. We wrote a custom Systemd service unit configuration for Vault to ensure that a configuration exists and that the <em>cap_ipc_lock</em> Linux capability is set in Vault’s executable, so that Vault’s process memory is not swapped to disk (if the swap partition is not encrypted a malicious user could read the dumped memory content of the Vault’s process!).</p><p>So far so good. We have a highly available Vault server living inside our VPC. What now? Let’s start using it.</p><p>We added another tool to our tool belt to help us establish ssh tunnels through a bastion server in order to access our Vault’s leaders (yes, plural. We have the same setup in each of our different environments). We handle all of our environments as if they were the production environment: they are deployed, configured and monitored in much the same way. The only difference between them is in the scale. This way they all deserve the same security standards. When we talk about the “Vault server” we are referring either to “this specific environment Vault server”, or to “the Vault servers in any environment”, indistinctively.</p><p>Now, we have to set Vault according to our needs and begin storing secrets. In order to accomplish this, we need to:</p><ul><li>Configure the authentication backend</li><li>Mount the secret backends that we need</li><li>Configure the policies for these assembly points</li><li>Write secrets into Vault</li></ul><p>We soon realized that the best option was to build our own tool to make the bootstrap and to open the Vault server in order to get it to a functional state. Besides helping out with these four tasks, this tool also stores an encrypted copy of the secrets with GnuPG in S3; which helps us not only to manage secrets more easily, but also we get a versioned, encrypted and updated copy of Vault data. Furthermore, all this allows the tool to easily carry out recovery in the face of a “disaster”; it’s as easy as bootstrapping a fresh and unsealed Vault server, and feeding it the desired revision of the S3 backed up secrets.</p><p>For the integration with Ansible we used <a href="https://github.com/jhaals/ansible-vault">ansible-vault</a>, with some modifications made to our inventories to guarantee that secrets from different environments don’t cross paths.</p><p>We use the same bootstrap tool previously mentioned to interactively add, remove or modify existing secrets.</p><h3>Conclusions</h3><p>We found out that Vault is a powerful tool, one that will “force” us to follow the path of best security practices. We can now answer correctly the four questions at the beginning of the post:</p><ol><li>Do you know which users have access to each secret? <strong>yes</strong></li><li>Do you have a policy regarding the secrets’ life cycle? Do you enforce that policy? <strong>yes</strong></li><li>Can you trace when a secret has been accessed (read, written, renewed, etc.) and by whom? <strong>yes</strong></li><li>Do you have plaintext secrets in your VCS? <strong>not anymore</strong></li></ol><p>And even better, Vault allows us to easily apply the same policies across our platform.</p><h4>We are hiring!</h4><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/jobandtalenteng/status/1125743791196049409&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/7171964a0e0f5cd8313526de1e33393f/href">https://medium.com/media/7171964a0e0f5cd8313526de1e33393f/href</a></iframe><p>If you want to know more about how is work at Jobandtalent you can read the first impressions of some of our teammates on this<a href="https://jobandtalent.engineering/our-first-impressions-working-at-jobandtalent-part-1-991a48eac2a4"> blog post</a> or visit our<a href="https://twitter.com/jobandtalentEng"> twitter</a>.</p><p><em>Thanks to </em><a href="https://twitter.com/spejman"><em>Sergio Espeja</em></a><em>, </em><a href="https://twitter.com/dagoonline"><em>David Gonzalez</em></a><em> and </em><a href="https://twitter.com/victorcoder"><em>Victor Castell</em></a><em> for reviews and comments, and </em><a href="https://www.linkedin.com/in/ximena-guzman-1318b9a4/"><em>Ximena Guzman</em></a><em> for language corrections</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e45ec4eb399" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobandtalenteng/fostering-security-with-hashicorp-vault-4e45ec4eb399">Fostering security with Hashicorp Vault</a> was originally published in <a href="https://medium.com/jobandtalenteng">Job&amp;Talent Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>