<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kristian Lesko on Medium]]></title>
        <description><![CDATA[Stories by Kristian Lesko on Medium]]></description>
        <link>https://medium.com/@kristian.lesko_19209?source=rss-19d72ede76f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ghyDdp7MbJzkgOi3</url>
            <title>Stories by Kristian Lesko on Medium</title>
            <link>https://medium.com/@kristian.lesko_19209?source=rss-19d72ede76f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:02:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kristian.lesko_19209/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From FreeIPA to Okta: Replacing GoodData’s Identity Management System]]></title>
            <link>https://medium.com/gooddata-developers/from-freeipa-to-okta-replacing-gooddatas-identity-management-system-cda51ec77cbd?source=rss-19d72ede76f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cda51ec77cbd</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ssl-certificate]]></category>
            <category><![CDATA[okta]]></category>
            <dc:creator><![CDATA[Kristian Lesko]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 12:43:11 GMT</pubDate>
            <atom:updated>2023-07-25T12:43:11.731Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="From FreeIPA to Okta: Replacing GoodData’s Identity Management System" src="https://cdn-images-1.medium.com/max/1024/0*D31ECekzBtfb4vwN" /></figure><p>Identity management (IdM) lies at the core of the security architecture and processes in most software companies. It is responsible for ensuring that only authorized actors are allowed access to valuable protected resources, including our customers’ data.</p><p>For multiple years, our infrastructure had relied on <a href="https://freeipa.org/">FreeIPA</a> as the identity management system of choice. It governed user access to virtual machines through secure shell (SSH), facilitated third-party application logins, and served other use cases. However, several reasons led to its ultimate removal from our stack and replacement by other systems.</p><p>This article builds upon a <a href="https://www.gooddata.com/blog/creating-robust-ssh-with-hashicorp-vault-certificates/">previous blog post</a>, which focused specifically on the rework of our SSH access management. Here, we document the broader motivations for our IdM changes, as well as the specific steps and approaches the infrastructure team at <a href="https://www.gooddata.com/?utm_medium=blogpost&amp;utm_source=medium.com&amp;utm_campaign=gooddata_okta_freeipa_072023&amp;utm_content=autor_patrik">GoodData</a> took during this complex migration endeavor.</p><h3>The Old Approach</h3><p>FreeIPA was introduced into GoodData in 2012 in response to the growing size of our infrastructure and the subsequent need for <em>single sign-on</em>, i.e. avoiding the need for users to authenticate separately against every internal service. With FreeIPA, engineers would simply need to authenticate once at the start of their day, obtaining a <a href="https://freeipa.org/page/Kerberos">Kerberos ticket</a>. All subsequent access to internal web applications would then reuse this ticket and be transparent to the user.</p><p>One of the primary use cases for FreeIPA in our stack was governing SSH logins to virtual machines; we have covered this in more detail in <a href="https://www.gooddata.com/blog/creating-robust-ssh-with-hashicorp-vault-certificates/">the previous post</a>. FreeIPA also provided useful components covering several other use cases. Notably, its <a href="https://freeipa.org/page/NTP_Server">NTP server</a> provided an easy way of time synchronization for the enrolled virtual machines, while the bundled <a href="https://freeipa.org/page/PKI">certificate infrastructure</a> enabled SSL certificate management for our internal services.</p><h3>The Shortcomings</h3><p>However, as the years went on, multiple aspects of the IdM setup based on FreeIPA proved to be insufficient for effective operations.</p><p>What we lacked the most was integration between FreeIPA and many of the third-party systems we utilized. While internally deployed web applications using <a href="https://httpd.apache.org/">httpd</a> as a frontend were easily extended with Kerberos-based authentication via FreeIPA, this did not hold true for many external services, which typically only provide the <a href="https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language">SAML</a> or <a href="https://openid.net/developers/how-connect-works/">OIDC</a> methods of authentication with identity providers.</p><p>The missing link between FreeIPA and many other applications we used meant that a unified approach to user management could not be adopted. This resulted in complicated processes related to user access management, as well as onboarding and offboarding. For instance, the team responsible for user management had to manually verify if a terminated employee had been provisioned in numerous applications, and manually remove their user account.</p><p>Operating the FreeIPA service itself also caused headaches to our infrastructure team on multiple occasions. We had <a href="https://freeipa.org/page/V4/Replica_Setup">replication</a> set up between FreeIPA servers in multiple different locations, but it proved fragile in case a network issue was encountered. Additionally, the FreeIPA deployment itself presented a <em>single point of failure</em>; if a regression was introduced by our operations, most of the engineers would be rendered unable to login to the services they needed.</p><h3>The Replacement</h3><p>All the aforementioned pain points ultimately led to a company-wide decision to migrate to <a href="https://www.okta.com/">Okta</a> as the centralized IdM solution. This spelled an imminent end to our usage of FreeIPA, but at the same time, created many new challenges for the infrastructure team to resolve.</p><p>Most importantly, a direct one-to-one replacement of FreeIPA components by Okta would not be possible; for example, Okta doesn’t provide governance of SSH logins or SSL certificate management out of the box. Therefore, while we would transition to using Okta as the central directory of user accounts, we would have to figure out how exactly to employ various open-source tools and adopt different approaches for each specific use case we needed to replace.</p><p>We already covered the replacement for SSH login management in <a href="https://www.gooddata.com/blog/creating-robust-ssh-with-hashicorp-vault-certificates/">the previous post</a>. Now, let’s delve into how we approached the replacement of the other necessary use cases.</p><h4>Web Applications</h4><p>The one aspect of our IdM ecosystem that can be considered migrated in a “one-to-one” fashion was the authentication for internally deployed web applications. Where we previously authenticated users by httpd’s LDAP or Kerberos modules, we moved on to using the <a href="https://github.com/OpenIDC/mod_auth_openidc">mod_auth_openidc</a> module instead, without any large architectural changes being required.</p><p>Additionally, as implied above, a massive advantage of migrating to Okta was also the support for single sign-on into multiple third-party applications. No more manual management of user accounts in each app!</p><h4>Certificates</h4><p>Replacing the infrastructure of SSL certificates used by our virtual machines, on the other hand, required more attention. With FreeIPA, every enrolled server could obtain an SSL certificate fairly easily using <a href="https://freeipa.org/page/Certmonger">certmonger</a>. However, this capability would no longer be available with Okta.</p><p>Before starting to seek an equivalent replacement, we took a step back and considered the exact use cases we had for certificates. We identified two distinct ways in which we utilized SSL certificates:</p><ul><li>protecting user-facing endpoints of internal web applications;</li><li>machine-to-machine authentication between internal services.</li></ul><p>For the former case, the solution was to use publicly trusted certificates issued by <a href="https://letsencrypt.org/">Let’s Encrypt</a>. We only needed to figure out the lifecycle and distribution of the certificates. In the end, we selected <a href="https://cert-manager.io/">cert-manager</a> running in our service Kubernetes cluster to handle obtaining the certificates (including DNS validation), and a simple <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/">CronJob</a> to store the certificates into our <a href="https://www.vaultproject.io/">HashiCorp Vault</a> instance, where all the consuming machines can access them.</p><figure><img alt="A schema of the new certificate management solution." src="https://cdn-images-1.medium.com/max/1024/0*oA_GFbly54QPrHJV" /></figure><p>The Vault service also played a key role in replacing our machine-to-machine authentication mechanism; we opted for creating a <a href="https://developer.hashicorp.com/vault/docs/secrets/pki">private certificate authority</a> (CA) to cover this use case. Since all of our virtual machines had already been integrated with Vault, it was then relatively simple for some servers to obtain a client SSL certificate and for other servers to verify it against Vault’s CA.</p><figure><img alt="A schema for new API access solution making use of client SSL certificates issued by Vault." src="https://cdn-images-1.medium.com/max/1024/0*s-hUx3fgZSaF6Plj" /></figure><h4>Time Server</h4><p>Replacing the FreeIPA-provided time server proved to be straightforward. Since most cloud providers provide their own time servers nowadays (for example, Amazon has the <a href="https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/">Time Sync Service</a>), we simply redirected our NTP configuration to use these instead of FreeIPA.</p><h3>The In-between</h3><p>With a suitable replacement for all of FreeIPA’s components identified, we focused on designing a rollout plan for the migration to Okta. We recognized that there would have to be a longer time window for the transition; we simply would not be able to migrate all of the users and application integrations from FreeIPA to Okta at the same time and without introducing an unacceptably long downtime for the employees.</p><p>To provide a bridge between Okta and FreeIPA, and to enable a smooth switch of one use case after another, we decided to introduce a synchronization mechanism between Okta and FreeIPA. We utilized our already existing internal tool called <a href="https://github.com/gooddata/freeipa-manager">freeipa-manager</a> for this purpose, which supported managing FreeIPA entities by their YAML representations stored in a Git repository. Initially, we created accounts for all users in Okta, and then extended this tool by adding support for creating users based on a response from Okta API.</p><p>This transition period was not ideal for the employees, since they had to remember two separate passwords for both of our IdM systems, as well as needing to keep track which applications are authenticated by which system. We focused heavily on cross-team communication to make this time as painless as possible for all the users involved.</p><h3>The Conclusion</h3><p>All in all, our migration from FreeIPA to Okta took slightly less than two years, starting in early 2021 and ending with removing the FreeIPA servers themselves from our environment in the second half of 2022.</p><p>It was an immense learning experience that required a wide cooperation between the company’s departments, as well as learning the various deeper internals of the authentication technologies involved. In retrospect, we can confidently conclude that this massive change was well worth the effort, bringing our infrastructure security and the related user experience to a higher level.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cda51ec77cbd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gooddata-developers/from-freeipa-to-okta-replacing-gooddatas-identity-management-system-cda51ec77cbd">From FreeIPA to Okta: Replacing GoodData’s Identity Management System</a> was originally published in <a href="https://medium.com/gooddata-developers">GoodData.AI Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating Robust SSH with HashiCorp Vault Certificates]]></title>
            <link>https://medium.com/gooddata-developers/creating-robust-ssh-with-hashicorp-vault-certificates-589f49df6b2?source=rss-19d72ede76f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/589f49df6b2</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[vault]]></category>
            <dc:creator><![CDATA[Kristian Lesko]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 11:24:11 GMT</pubDate>
            <atom:updated>2023-03-09T11:24:11.152Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAlZqCyeP4pdzPvc3_d2sg.png" /></figure><p>Secure shell (SSH) is a standard way of accessing server infrastructure in the IT industry. By contrast, <strong>no single standard solution exists</strong> for managing users’ SSH access in an automated and secure manner. Let us explore how we tackled this challenge at GoodData.</p><h3>Our Requirements</h3><p>Given the diverse and globally distributed platform infrastructure we operate at GoodData, secure shell (SSH) is a core way of accessing our systems in order to perform necessary operations and debugging. It is therefore vital for us to have an SSH authorization method that is:</p><ul><li><strong>secure</strong> in accordance with the latest industry standards;</li><li><strong>robust</strong> in its performance; and</li><li><strong>easy to use </strong>for our employees.</li></ul><p>Having utilized <a href="https://www.freeipa.org/">FreeIPA</a> to provide SSH access management for multiple years, a strategic decision was made in late 2020 to switch to <a href="https://www.okta.com/">Okta</a> as the company-wide identity management solution. This transition presented us with an opportunity to refresh our access control setup and ensure its consistency with current security standards.</p><h3>Searching for a Solution</h3><p>We identified <strong>insufficient scalability</strong> as the primary shortcoming of our previous SSH setup. With FreeIPA, every SSH login attempt requires the user’s public SSH key to be verified against the server, rendering FreeIPA a <em>single point of failure</em> and preventing any user logins to target machines in case our FreeIPA servers experienced an outage or maintenance.</p><p>We set out to find and implement a solution that would avoid this pitfall; one that would be robust enough to support daily operations on the scale we require, i.e. managing access to hundreds of servers distributed across the globe.</p><p>Additionally, deprecating long-lived SSH credentials from the ecosystem was another goal we deemed important. There was an inherent security risk stemming from static SSH keys being present on employees’ devices, as they could possibly be stolen and used for access to our infrastructure. While we had partially mitigated this risk using multi-factor authentication, we were striving for an ideal solution that would only rely on short-lived, dynamic credentials. This way, any leaked credential would not pose a security risk, because it would expire very quickly.</p><p>After extensive research, we selected <a href="https://www.vaultproject.io/">HashiCorp Vault</a> to provide SSH access management via its <a href="https://developer.hashicorp.com/vault/docs/secrets/ssh/signed-ssh-certificates">dynamic SSH certificates engine</a>.</p><h3>Technical design</h3><p>Since Vault had already been present in our infrastructure as a store for static key-value credentials, extending its role by adding the SSH certificate signing backend was fairly straightforward.</p><h4>User Authentication</h4><p>For user access, Vault has been integrated with Okta using the <a href="https://developer.hashicorp.com/vault/tutorials/auth-methods/oidc-auth">OIDC authentication method</a>. We defined several <strong>user login roles</strong> corresponding to varying levels of access granted to distinct groups of users; each role is tied to a specific OIDC claim value provided from Okta. Every Vault login role, in turn, allows signing SSH certificates by a corresponding SSH backend role. The <a href="https://developer.hashicorp.com/vault/api-docs/secret/ssh#key_id_format">SSH certificate key ID configuration</a> ensures that user role information is preserved in the resulting signed certificate.</p><blockquote>For example, if a user <strong>first.last </strong>belongs to <strong>group1</strong>, they can login to Vault using login role <strong>group1</strong>. Vault policy will also allow them to sign a certificate using SSH role <strong>group1</strong>, and the resulting certificate’s key ID will be <strong>okta-first.last:group1</strong>.</blockquote><figure><img alt="High-level schema of our SSH authorization workflow, describing how users obtain their SSH certificates from Vault, login to the jump stations using their own accounts and login to other servers using a shared system account." src="https://cdn-images-1.medium.com/max/1024/1*78v2JnnUvJIGZOJXMWzn3A.png" /><figcaption>High-level schema of our SSH authorization flow.</figcaption></figure><h4>SSH User Provisioning</h4><p>In absence of a centralized SSH authorization server, a challenge remains: how to determine which users are allowed to connect to a specific server? To resolve this, we have defined two distinct SSH login scenarios we needed to support:</p><ol><li>login to externally exposed <strong>jump stations</strong>; and</li><li>further hops to internally accessible servers.</li></ol><p>On jump stations, we require the users to login under a personal non-privileged account, in order to provide a safe environment to access systems using personal credentials (such as <a href="https://aws.amazon.com/iam/identity-center/">AWS IAM Identity Center</a> logins or reading secrets from Vault itself). Therefore, a simple Python script is periodically run on the jump station servers to fetch a list of relevant users from Okta API and create a system account for each of them.</p><p>On all other servers, a single shared user account is sufficient for all users logging in via SSH; therefore, no periodic user synchronization from Okta is required.</p><p>By this splitting of use cases, the need to deliver Okta credentials is limited to a handful of jump station nodes while we still keep the level of user access separation we require.</p><h4>User Authorization</h4><p>The process of authorizing users utilizes the sshd <strong>AuthorizedPrincipalsCommand</strong> directive; three pieces of setup are pre-delivered by our configuration management tool to facilitate this:</p><ul><li>Vault SSH certificate authority public key;</li><li>list of Vault SSH roles allowed to login to a given server; and</li><li>an authorization script to be called upon user login.</li></ul><p>When a user attempts to login to a server, AuthorizedPrincipalsCommand is called, with the <strong>%i</strong> (certificate’s key ID) and <strong>%u</strong> (username) parameters provided to our authorization script. The script parses the user’s Vault role from the key ID (making use of its static format described above) and compares it against the list of roles allowed to login. On jump stations, it additionally verifies whether the user is logging under their own personal account name. A second factor verification is also hooked into the login process for the jump stations case.</p><h3>Benefits and Lessons</h3><p>Using Vault’s SSH certificates engine, we have been able to transition to a much more robust and secure login system than before. Users across the company appreciate the system’s stability; since authorization is done with information statically available to SSH servers and no live communication with Vault is needed at login time, user experience is much smoother than previously. Security has also been improved by eliminating static, long-lived SSH keys in favor of quickly expiring SSH certificates.</p><p>We have encountered minor challenges of the new setup. Having a shared system account for all users on a majority of servers means that auditing user actions is slightly harder; we have worked around this by parsing the certificate key ID into an environment variable upon login. Commands such as <strong>who</strong> are not useful since introducing the new setup. Retraining users to the new Vault-based SSH login workflow was an initial hurdle we have been able to overcome with time.</p><p>Overall, though, we conclude that the new way of doing SSH at GoodData provides more security and a better user experience than the past solution based on static SSH keys.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=589f49df6b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gooddata-developers/creating-robust-ssh-with-hashicorp-vault-certificates-589f49df6b2">Creating Robust SSH with HashiCorp Vault Certificates</a> was originally published in <a href="https://medium.com/gooddata-developers">GoodData.AI Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>