<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Cider Security - Medium]]></title>
        <description><![CDATA[Blog posts from Cider Security’s R&amp;D and leadership teams. Topics include: CI/CD Pipelines, DevOps, DevSecOps, Vulnerabilities, Cybersecurity, Entrepreneurship, and more. - Medium]]></description>
        <link>https://medium.com/cider-sec?source=rss----a827f45f8a43---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Cider Security - Medium</title>
            <link>https://medium.com/cider-sec?source=rss----a827f45f8a43---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:29:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/cider-sec" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploiting Jenkins build authorization]]></title>
            <link>https://medium.com/cider-sec/exploiting-jenkins-build-authorization-22bf72926072?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/22bf72926072</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[pipeline]]></category>
            <dc:creator><![CDATA[Asi Greenholts]]></dc:creator>
            <pubDate>Thu, 17 Feb 2022 15:08:34 GMT</pubDate>
            <atom:updated>2022-02-17T15:08:34.244Z</atom:updated>
            <content:encoded><![CDATA[<h4>Are you aware of the risks lurking in your default Jenkins configuration?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FtkZXZNDnR30H_9vyEmSrA.jpeg" /></figure><h3>TL;DR</h3><p>The default build authorization configuration in Jenkins — controlling the permissions allocated to pipelines — is insecure and is often left unmodified in production environments. To address this issue, you should use the “Authorize Project” and the “Role-Based Authorization Strategy” plugins to define secure build authorization configurations.</p><h3>Introduction</h3><p>Jenkins is the most widely used CI/CD automation platform. Its great flexibility and extensibility helps thousands of organizations solve many of their engineering challenges. Having said that, Jenkins is also probably one of the most complex CI/CD automation platforms; While its advanced flexibility and extensibility has many advantages, it also has some significant pitfalls that must be acknowledged and addressed.</p><p>One significant pitfall is the platform’s insecure default configurations. The default configurations are designed to allow functionality on initial setup of the system and are meant to be replaced after initial installation, as advised by <a href="https://www.jenkins.io/doc/book/security/">Jenkins’ documentation</a>. However, these insecure configurations often remain in production environments unbeknownst to DevOps and security engineers.</p><p>In this blog we explore the potential dangers arising from one of these default configurations — the <strong>build authorization configuration</strong>.</p><h4>What is “build authorization” and why are its default settings problematic?</h4><p>Running builds require authorization to perform actions in Jenkins, so every build is associated with a user account. This association is called “build authorization.” The permissions are used to perform the <a href="https://www.jenkins.io/doc/book/security/access-control/permissions/">various actions</a> that the build needs, such as allocating an agent to run the build or storing build results in the system.</p><p>The authorization model in Jenkins consists of permissions that control the access to different resources and actions on various objects in the system. Permissions are verbs, grouped by different system components.</p><p>Some notable permissions are:</p><ul><li>Overall/Administer — Allows users to do everything.</li><li>Job/Configure — Allows users to configure jobs.</li><li>Job/Build — Allows users to trigger builds.</li><li>Agent/Build — Allows users to execute builds on an agent.</li></ul><p>All other descriptions can be viewed by hovering over the permission names in Jenkins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QOTpmerDPA3VrAo5RaLWBQ.png" /></figure><p>There is one more special permission that is not visible in this table, which is called <a href="https://javadoc.jenkins-ci.org/hudson/security/ACL.html#SYSTEM2">SYSTEM</a>. This is the permission used by Jenkins itself and it can access any resource and perform actions on any object in the system</p><p>The most important thing to note in this context is that the Jenkins default settings assign <strong>every build</strong> to “run as system.” In other words, they assign it to the all-powerful SYSTEM user, meaning any action executed during the build has permission to do whatever it wants.</p><p>This can become a major problem when an attacker obtains permissions to a repository linked with a pipeline — by compromising developer credentials, access token, SSH key, or any other method. If the hijacked user account has the ability to modify the Jenkinsfile and, practically, the Jenkins pipeline, the attacker can take advantage of it to run malicious commands in the pipeline by executing a <a href="https://medium.com/cider-sec/ppe-poisoned-pipeline-execution-34f4e8d0d4e9">PPE (Poisoned Pipeline Execution)</a> attack. With the default build authorization settings, these malicious commands will run with the almighty SYSTEM user. Some <a href="https://www.jenkins.io/doc/book/security/build-authorization/">examples</a> of actions that can be performed from within builds are: triggering other builds, creating new jobs and running on different nodes.</p><h3>How exactly does the default build authorization leave us vulnerable?</h3><p>To demonstrate why the default configurations are so bad, we’re going to simulate a common situation in which attackers have gained initial access to an organization’s repositories and can push code (either to new or existing branches) that triggers a pipeline defined by a Jenkinsfile in their control — to which they will add malicious commands. As regular users, the attackers in our scenario have only Overall/Read and Job/Read permissions on Jenkins.</p><p>When attacking Jenkins through manipulation of the Jenkinsfile, the attack surface that’s available consists of two elements: (1) <a href="https://www.jenkins.io/doc/book/pipeline/#step">pipeline steps</a> that can be invoked — dependant on which plugins are installed, and (2) Groovy scripts that can run within the Groovy Sandbox whitelist, assuming it’s turned on as defined in the default configurations. Here we will focus on element #1.</p><p>To map all available pipeline steps, the attackers can do multiple things, for example:</p><ul><li>Try to use steps from the <a href="https://www.jenkins.io/doc/pipeline/steps/">Pipeline Step Reference</a> and see if they work. Most likely, the set of plugins suggested for installation by Jenkins during its installation will be available.</li><li>Look for steps used inside the organization’s Jenkinfiles stored in accessible SCM (source code management) repositories.</li><li>Write a script to parse all build outputs that they can read and look for more available steps.</li></ul><p>Let’s look at some example attack scenarios:</p><h4>Attack scenario 1: Running builds on agents</h4><p>Users with Overall/Read permissions can view all agent names and their labels. Another option for attackers to discover agent labels is to look for them in accessible SCM repositories. Using this knowledge, attackers can execute a Direct-PPE attack and change the agent label in the Jenkinsfile. The attackers can then run builds on any agent they choose — due to the fact that the default Jenkins configuration is to run builds as the SYSTEM user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NZBSt-CxauUDjc7p" /></figure><p>The following is an example of the change made to the Jenkinsfile from the diagram above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/0*VtQWk4g82WVFpqbA" /></figure><p>By running their build on different agents, attackers can perform multiple actions, such as (a) finding valuable files created by previous jobs running on that agent; (b) extracting sensitive environment variables that were loaded by other builds; (c) gaining extensive network access, which can be used for lateral movement through the organization’s assets; (d) leveraging the agent’s identity, like in the case of an agent running on an AWS EC2 instance which can be used to obtain the permissions of the EC2 instance role in order to perform actions against the AWS account.</p><h4>Attack scenario 2: Triggering builds</h4><p>Attackers can exploit the default build authorization permissions to trigger any build job they choose using the <strong>build</strong> step (and provide it with any parameter they want if it accepts parameters). This step requires the Job/Build permission and can be used in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SXGX25Cic5hDR3az" /></figure><p>The only requirement for this is knowing the target job name, a likely scenario in most environments as global Job/Read permissions are in many cases given to all users. Another means of discovering job names is to check pull requests in the organization’s SCM repositories, which may contain the job name that a request triggers.</p><p>An attacker that is able to run build jobs can then carry out malicious deployments using one of the following:</p><ul><li>Deploying malicious code that was previously pushed to the codebase</li><li>Triggering a deployment pipeline with parameters controlled by the attacker that will ultimately change the behavior of the application.</li></ul><h4>What should we do to protect ourselves?</h4><p>To protect against attacks such as those described above, two things need to happen. <strong>First</strong>, we need to make sure that new builds do not automatically run as SYSTEM, but rather by a specific user. <strong>Second</strong>, we must see to it that each user has only the minimum possible permissions that it needs to execute builds.</p><p>1) To achieve the first goal, we can use the <strong>“Authorize Project” plugin</strong>.</p><p>The “<a href="https://plugins.jenkins.io/authorize-project/">Authorize Project</a>” plugin allows us to define both <strong>default</strong> and <strong>per-project build authorization</strong>. It offers the following four options:</p><ul><li>Run as the user who triggered the build</li><li>Run as anonymous</li><li>Run as the specified user</li><li>Run as SYSTEM</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d4sWLB2fojTaQICn" /></figure><p>Builds can be triggered either manually, or by a series of “automatic” triggers, such as: “build periodically,” “discover pull requests,” “poll SCM” and so on. When triggered manually, by pressing the play button for example, the build will run as the user who manually initiated it. Every time a build is triggered by an automatic method, it will run as whatever user has been defined by the system as the “default” build authorization user.</p><p>However, if “per-project build authorization” has been defined for that project, the build will run as the designated user regardless of whether it was triggered automatically or manually.</p><p>The plugin should therefore be used not only to ensure that the default user is no longer defined as SYSTEM, but also to reduce the attack surface by configuring per-project build authorization, with different users that have explicit permissions for specific projects, whenever possible.</p><p>Per-project build authorization can be configured for Pipelines, Freestyle and Multi-configuration projects in the project screen under “Authorization”. Note, however, that it cannot currently be configured for Multibranch Pipelines. Since Multibranch Pipelines are a widely used feature of Jenkins, not being able to configure per-project build authorization poses a major security issue. If you’d like to encourage the contributors of Jenkins to provide a solution, feel free to vote on the issue <a href="https://issues.jenkins.io/browse/JENKINS-50559?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel">here</a>.</p><p>2) To achieve our second goal (making sure each user has the minimum possible permissions that it needs to execute builds), we can use the “<a href="https://plugins.jenkins.io/role-strategy/"><strong>Role-Based Authorization Strategy</strong></a><strong>” </strong>plugin.</p><p>This plugin allows system administrators to create roles with particular permissions and then assign them to users or groups. The roles are divided into three categories:</p><ul><li>Item roles — allow users to perform the actions specified, but only on the items (such as projects, credentials) that match the specified regex.</li><li>Node roles — allow users to perform the actions specified, but only on the agents that match the specified regex.</li><li>Global roles — allow users to perform the specified action on all objects in the system.</li></ul><p>Once the roles have been configured, they can then be assigned to users or groups, according to the authentication method in use.</p><p>“<a href="https://plugins.jenkins.io/matrix-auth/">Matrix Authorization Strategy</a>” is another potentially useful plugin to address this issue.</p><h3>So what exactly should I do?</h3><p>Now that we’re familiar with the threat and with the tools we can use to address it, here are few specific steps you should follow:</p><ol><li>To prevent builds from running as SYSTEM, configure an alternate default build authorization strategy and per-project build authorization strategy.</li><li>To minimize user privileges, assign human users and build users specific, limited roles using the “Role-Based Authorization Strategy” plugin.</li><li>To minimize the attack surface, remove any unused plugins to reduce available steps.</li></ol><h4>How to configure build authorization strategies</h4><ol><li>Install the “Authorize Project” plugin.</li><li>Because the “Authorize Project” plugin doesn’t support configuring per-project authorization for Multibranch Pipelines, configure the “Project Default Build Authorization” under “Configure Global Security” to use a specific user for all Multibranch Pipelines. That user should not be used by a human and it should usually have only Agent/Build for specific nodes and Job/Read for all Multibranch Pipelines configured using the “Role-Based Authorization Strategy” plugin.</li><li>Configure “Per-project configurable Build Authorization” under “Configure Global Security” to enable one or more of the following: “run as specific user”, “run as anonymous” and “run as user who triggered build”. Note that the first is the best option of the three (in terms of both security and function).</li><li>For Pipelines or Freestyle jobs, configure each job’s “Authorization” in the job’s screen to “run as specific user” with a user specially created for that job or group of jobs using an item and node roles that consist of minimal privileges such as Agent/Build and Job/Read only.</li></ol><h4>Best practices for configuring user permissions</h4><p>Use the “Role-based Authorization Strategy” plugin to configure the following settings:</p><ol><li>Add all users except administrators to a global role that has only the Overall/Read permission. If more global roles are needed, assign them to a minimum number of users.</li><li>Because roles are defined using regular expressions, it is recommended to enforce project naming conventions to allow easily configuring granular roles.</li><li>Create specific roles for both items (e.g. Jobs, Credentials) and agents with patterns matching the projects and nodes they need access to.</li><li>Avoid assigning users with the Job/Configure, Job/Delete, Job/Create, Job/Move, Run/Delete, Run/Replay, Agent/Configure, Agent/Connect, Agent/Delete, Agent/Disconnect and Agent/Provision permissions, since these pose more risk.</li><li>Consider removing the Job/Build permission from users to prevent a scenario where a specific pipeline was manipulated/tampered with, and then executed by unsuspecting privileged users, allowing the compromised build to have elevated privileges.</li></ol><p>If all of these actions seem a bit daunting and complex, note that you can use the <a href="https://www.jenkins.io/projects/jcasc/">Configuration as Code plugin</a> to simplify the process considerably. Jenkins Configuration as Code provides the ability to define Jenkins configuration as a simple, yaml syntax. You can see examples of how to do that <a href="https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos/role-strategy-auth">here</a>.</p><p>Seeing that Jenkins configurations are dynamic we need to ensure that these recommended configurations remain unchanged and that we are able to detect any major change that might have a negative influence on our attack surface. In case your environment is connected to the Cider platform, you can view any such changes under the “Pipeline Risks” page.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22bf72926072" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/exploiting-jenkins-build-authorization-22bf72926072">Exploiting Jenkins build authorization</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PPE — Poisoned Pipeline Execution]]></title>
            <link>https://medium.com/cider-sec/ppe-poisoned-pipeline-execution-34f4e8d0d4e9?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/34f4e8d0d4e9</guid>
            <category><![CDATA[application-security]]></category>
            <category><![CDATA[hacking]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[supply-chain]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Omer Gil]]></dc:creator>
            <pubDate>Tue, 08 Feb 2022 13:05:53 GMT</pubDate>
            <atom:updated>2022-02-09T04:58:36.661Z</atom:updated>
            <content:encoded><![CDATA[<h3>PPE — Poisoned Pipeline Execution</h3><h4>Running malicious code in your CI, without access to your CI</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gbnKew_FPKefRORFB3tWig.jpeg" /></figure><h3><strong>Authors</strong></h3><p><a href="https://twitter.com/omer_gil">Omer Gil</a>, Head of Research @ <a href="https://medium.com/u/78e7218f52ba">Cider Security</a><br><a href="https://twitter.com/Dkrivelev">Daniel Krivelevich</a>, CTO @ <a href="https://medium.com/u/78e7218f52ba">Cider Security</a></p><h3>Intro</h3><p>Dev environments have become a major part of today’s attack surface. And within them, the most lucrative assets are the systems responsible for CI and CD — those that build, test, and deploy code — and typically possess the secrets and access to the most critical assets of the organization. So it’s only natural that attackers are continuously on the lookout for novel ways to gain access to these systems.</p><p>The target is clear — code execution. Attackers that are able to execute code in the CI, are likely to have a clear path all the way to production.</p><p>For an attacker, gaining access to the CI can be done using various techniques — like exploiting known vulnerabilities in the service or the underlying host, or abusing inadequate identity and access management configurations.<br>However, as more and more security controls are being adopted around CI and CD systems, these techniques become more challenging to carry out: internet access to these systems is restricted, they are regularly patched, and strong authentication and authorization controls are continuously implemented.</p><p>So while direct access to CI is the easiest way for attackers to execute their malicious code in the CI, today’s attackers realize that succeeding to achieve that is a task that is far from trivial.<br>But, What if attackers had the ability to execute malicious code in the CI without ever having any access to it?</p><h3>Defining a CI pipeline execution flow</h3><p>To understand the possibilities attackers have to execute code in the CI, we first need to be familiar with how the execution flow of a CI pipeline is defined.</p><p>The most common way (and in certain systems, the only way) to define a pipeline, is by using a CI configuration file hosted in the repository the pipeline builds. This file describes the order of executed jobs, conditions that affect the flow, and build environment settings. These files typically have a consistent name and format, for example — Jenkinsfile (Jenkins), .gitlab-ci.yml (GitLab), .circleci/config.yml (CircleCI), and the GitHub Actions YAML files located under .github/workflows.<br>When triggered, the pipeline job pulls the code from the selected source (e.g. commit / branch), and runs the commands specified in the CI configuration file against that code.<br>The commands executed within the pipeline are either invoked directly by the CI configuration file, or indirectly by a script, code test, or linter that resides in a separate file referenced from the CI configuration file.</p><p>Attackers that are able to manipulate the commands executed — either directly or indirectly — by the pipeline, can leverage this ability to achieve their target of executing code in the CI, using an attack vector called <strong>PPE — Poisoned Pipeline Execution</strong>.</p><h3>PPE — Poisoned Pipeline Execution</h3><p>The Poisoned Pipeline Execution (PPE) vector abuses permissions against an SCM repository, in a way that causes a CI pipeline to execute malicious commands.<br>Users that have permissions to manipulate the CI configuration files, or other files which the CI pipeline job relies on, can modify them to contain malicious commands, ultimately “poisoning” the CI pipeline executing these commands.</p><p>For an attacker to be able to successfully carry out a PPE attack, the following criteria must be met:</p><ul><li>The attacker obtains permissions against an SCM repository — through user credentials, access token, SSH key, OAuth token or other methods. In some flavors of PPE anonymous access to a public repo will also suffice.</li><li>Changes to the repository in question — either through pushing directly to remote branches or suggesting changes through a PR from a remote branch or fork — trigger a CI pipeline — without additional approvals/review (a pipeline linked to repositories which are triggered through periodic polling of the repo are also relevant in this context).</li><li>The permissions obtained by the attacker allow triggering the events that cause the pipeline to be executed.</li><li>The files the attacker is able to change define the commands that are executed (directly or indirectly) by the pipeline.</li><li>The pipeline node has access to non-public resources (e.g. secrets, other nodes, compute resources)</li></ul><p>Pipelines executing unreviewed code, for example those which are triggered directly off of pull requests or commits to arbitrary repository branches, are more susceptible to PPE. The reason is that these scenarios, by design, contain code which has not undergone any reviews or approvals.</p><p>Once able to execute malicious code within the CI pipeline, the attacker can conduct a wide array of malicious operations, all within the context of the pipeline’s identity, including:</p><ul><li>Access to any secret available to the CI job, such as secrets injected as environment variables or additional secrets stored in the CI. Being responsible for building code and deploying artifacts, CI/CD systems typically contain dozens of high-value credentials and tokens — such as to a cloud provider, to artifact registries, and to the SCM itself.</li><li>Access to external assets the job node has permissions to, such as files stored in the node’s file system, or credentials to a cloud environment accessible through the underlying host.</li><li>Ability to ship code and artifacts further down the pipeline, in the guise of legitimate code built by the build process.</li><li>Ability to access additional hosts and assets in the network/environment of the job node.</li></ul><p>Our research around PPE draws inspiration from several publications around attacks using similar vectors published in recent years, combined with analysis of dozens of CI/CD environments, through which we witnessed how widespread it is.<br>PPE combines multiple unique characteristics that drove us to dive deep into researching it and share our conclusions and insights: its critical impact, the amount of environments potentially susceptible to the attack, the low detectability, and the existence of multiple flavors — each unique in relation to both its execution techniques as well as the appropriate preventative measures.<br>The objective of this blog post is to assist security teams, engineers, and red teamers in becoming better acquainted with PPE, the different exploitation techniques, and the potential countermeasures.</p><p>As mentioned, PPE consists of three primary flavors, which we cover in this post:</p><ul><li>Direct (D-PPE)</li><li>Indirect (I-PPE)</li><li>Public (P-PPE, or 3PE)</li></ul><h3>Direct PPE (D-PPE)</h3><p>The most important prerequisite¹ for Direct PPE is that the CI configuration file — which defines the build — resides together with the code being built. This effectively means that the author of the code also controls the build definition.</p><p>In a D-PPE scenario, the attacker modifies the CI config file in a repository they have access to, either by pushing the change directly to an unprotected remote branch on the repo, or by submitting a PR with the change from a branch or a fork. Since the CI pipeline execution is triggered off of the “push” or “PR” events, and the pipeline execution is defined by the commands in the modified CI configuration file, the attacker’s malicious commands ultimately run in the build node once the build pipeline is triggered.</p><h4>D-PPE attack flow</h4><p>The diagram below demonstrates the flow of a D-PPE attack, in which credentials to an AWS hosted production environment are exfiltrated by the attacker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EjtiHqh-39nK4gV2BLc9g.png" /></figure><ol><li>An attacker creates a new remote branch in the repository, updating the pipeline configuration file with malicious commands intended to access AWS credentials stored in the GitHub organization and send them to a remote server.</li><li>The push of code triggers a pipeline, which fetches the code from the repository, including the malicious pipeline configuration file.</li><li>The pipeline runs based on the configuration file “poisoned” by the attacker. As per the attacker’s malicious commands, AWS credentials stored as repository secrets are loaded into memory.</li><li>The pipeline proceeds to execute the attacker’s commands that send the AWS credentials to a server controlled by the attacker.</li><li>The attacker is then able to use the stolen credentials to access the production environment.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*OW5mpdRTTOGF_6q8r4jFxw.gif" /></figure><p>Let’s see some examples.<br>(Note: All code snippets below were shortened for simplicity, removing init commands, installations, etc.)</p><h4>Example 1: AWS secret exfiltrated from the Jenkins credential store</h4><p><a href="https://www.jenkins.io/doc/book/pipeline/getting-started/#defining-a-pipeline">The best practice</a> when defining a Jenkins pipeline is to commit a Jenkinsfile to the source control, from which Jenkins can load it directly when executing a job.</p><p>In a similar fashion to many other CI configuration files, the Jenkinsfile is constructed of different sections, such as the stages and steps describing the pipeline flow, a configuration of the executing agent, and a definition of environment variables.</p><p>As part of this example, a <a href="https://www.jenkins.io/doc/book/pipeline/multibranch/">multibranch pipeline</a> is configured and used on a Jenkins instance, while the declarative Jenkinsfile is stored in the repository. When a pipeline job is triggered, the code is pulled from the repository along with the Jenkinsfile. Our Jenkinsfile looks as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbd21df00c2b0a2ffa5219e96aa511c8/href">https://medium.com/media/fbd21df00c2b0a2ffa5219e96aa511c8/href</a></iframe><p>The attacker creates a pull request, modifying the Jenkinsfile to contain the following content:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48dfd62b55f162fef9b84c8e42105140/href">https://medium.com/media/48dfd62b55f162fef9b84c8e42105140/href</a></iframe><p>In their PR, the attackers modify the Jenkinsfile to load AWS credentials using the <a href="https://www.jenkins.io/doc/pipeline/steps/pipeline-aws/#withaws-set-aws-settings-for-nested-block">“withAWS” plugin</a>. In this example, the attacker is able to load the “AWS_key” secret as it is stored on the Jenkins credential store with the “global” scope, making it available to any pipeline running on the instance. Then, the pipeline job exfiltrates the environment variables, which include AWS credentials, to a remote server.</p><p>Attackers can gain knowledge of the names of secrets they can potentially load into memory by analyzing existing CI configuration files (in this example — Jenkinsfile) in repositories they have access to and detecting names of credentials the organization loads as part of the pipeline configuration. The same technique could be applied to loading many more secrets, in addition to the AWS key in this example.</p><h4>Example 2: Jenkins OS commands on privileged node</h4><p>Pipelines are executed on nodes, which are the containers or machines executing the commands specified in the pipeline configuration file. Given the sensitivity of the nodes, the principle of least privilege must be applied around the nodes, both in relation to the users/applications that have permissions to access the node, as well as to the permissions the node has to various resources in the CI/CD environment. Our use case demonstrates this principle — the ‘agent’ block allocates a simple, lean image to the pipeline, containing a the relevant golang utilities require to run the pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*Gi2oVEvgypx2UYb2juLuBA.png" /></figure><p>The attackers can modify the ‘agent’ block to allocate another node available on the Jenkins instance to run the pipeline.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/72f8ee5feeef2e81848d09f2314c9247/href">https://medium.com/media/72f8ee5feeef2e81848d09f2314c9247/href</a></iframe><p>In the example above, the attacker modifies the ‘agent’ block to allocate a node based on a specific EC2 instance with an attached IAM role to run the pipeline. In a similar manner to how potential secret names were identified in the first example, potential node names are relatively easy to obtain by searching Jenkinsfiles in other repositories, or by opportunistically changing the agent specifier to ‘any’ in hope for the CI/CD wheel of fortune to draw a highly privileged node.</p><p>Since the pipeline is able to execute OS commands on the underlying host, the attacker can access the instance metadata service to get temporary credentials and exfiltrate them. Theattacker can choose to be a bit more stealthy and directly run commands using the AWS CLI as in the example above, in which secrets stored on the Secret Manager service are listed, fetched and exfiltrated.</p><h4>Example 3: GitHub Actions credential theft</h4><p>In GitHub Actions, custom pipelines, more commonly referred to as “workflows” can be created in any branch in the repository, and then access any secret stored across the entire organization when executed.</p><p>When using GitHub Enterprise, secrets can be protected from direct access by assigning them to environments. Users can configure these environments to be triggered only by specific branches, while the branches themselves can be protected by branch protection rules to restrict unreviewed code to be pushed to them.</p><p>However, these controls aren’t available in licenses other than GitHub Enterprise, such as Free or Team (which is the most popular license). Therefore, any user account with Write permission on a repository in a Free or Team organization, is able to create a workflow that can access any secret in the repository and in the organization. However, GitHub Enterprise implementation alone is not sufficient; Even when the controls become available, implementing them in an effective manner across the entire ecosystem is a tedious task which is far from trivial.</p><p>In the following example, a GitHub repository is connected with a GitHub Actions workflow that fetches the code, builds it, runs tests, and eventually deploys artifacts to AWS.<br>The workflow executes a file named <em>pipeline.yml</em>, which is the workflow configuration file describing the execution steps. When new code is pushed to a remote branch in the repository, the code is fetched by the runner (the workflow node), including the configuration file — which defines the job’s flow.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac48fd951867ca875e1e512a80744b2c/href">https://medium.com/media/ac48fd951867ca875e1e512a80744b2c/href</a></iframe><p>Attackers with repo permissions can create a remote branch, and push a modified version of the workflow configuration file that contains malicious commands:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a61a721bac409e04b5c72b132904d221/href">https://medium.com/media/a61a721bac409e04b5c72b132904d221/href</a></iframe><p>When executed, the workflow will load AWS credentials stored in the repository or organization as environment variables. Then, the credentials are exfiltrated to the attacker controlled remote server.</p><h3>Indirect PPE (I-PPE)</h3><p>As previously mentioned, the most important prerequisite for Direct PPE is that the CI configuration file — which defines the build — resides together with the code being built, granting the attackers with access to the repository full control over the build definition. However, in certain cases, that prerequisite is not met:</p><ul><li>If the pipeline is configured to pull the CI configuration file from a separate, protected branch in the same repository.</li><li>If the CI configuration file is stored in a separate repository from the source code, without the option for a user to directly edit it.</li><li>If the CI build is defined in the CI system itself — instead of in a file stored in the source code.</li></ul><p>In such a scenario, the attacker can still poison the pipeline by injecting malicious code inside files invoked indirectly by the pipeline configuration file.</p><p>Numerous commands that run as part of the pipeline for the purpose of testing, building and sometimes deploying the code, are defined in separate files residing in the SCM repositories for example:</p><ul><li><em>make</em>: Executes commands defined in the “Makefile” file.</li><li>Scripts executed by the pipeline which are stored in the same repository as the source code itself (e.g. <em>python myscript.py</em> — where myscript.py would be manipulated by the attacker).</li><li>Code tests: Various testing frameworks exist for running tests on code within the build process. These frameworks are typically specific for each language and framework, and rely on dedicated files containing the commands executed against the application’s code as part of the test. Attackers that are able to manipulate the code responsible for testing are then able to run malicious commands inside the build.</li><li>Automatic tools: Linters and security scanners used in the CI, are also commonly reliant on a configuration file residing in the repository. Many times these configurations involve loading and running external code from a location defined inside the configuration file.</li></ul><p>So rather than poisoning the pipeline by inserting malicious commands directly into the pipeline definition file, In I-PPE, an attacker injects malicious code into files the aforementioned tools rely on, ultimately running the malicious code in the pipeline node once the pipeline is triggered and invokes the tools in question.</p><p>The challenge with I-PPE is that even if we were to identify all the files that are referenced from within all the CI configuration files in our environment, and place the appropriate protections around them — a daunting task on its own — that would not necessarily be enough; each one of these files can potentially contain reference to another file that could also be susceptible to I-PPE.</p><h4>I-PPE attack flow</h4><p>The diagram below demonstrates the flow of an I-PPE attack in Jenkins, in which credentials stored in the repository are exfiltrated by the attacker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VP6SdR4vMW8c02ZQDnZftw.png" /></figure><ol><li>An attacker creates a pull request in the repository, appending malicious commands to the <em>Makefile</em> file.</li><li>A Jenkins pipeline is triggered, fetching the code from the repository, including the malicious <em>Makefile</em>.</li><li>The pipeline runs based on the configuration file stored in the main branch. It gets to the <em>build</em> stage, and loads the AWS credentials into environment variables — as defined in the original Jenkinsfile. Then, it runs the <em>make build</em> command, which executes the malicious command that was added into <em>Makefile</em>.</li><li>The malicious <em>build</em> function defined in the Makefile is executed and sends the AWS credentials to the attacker.</li><li>The attacker uses the stolen credentials to access AWS.</li></ol><p>An awesome example for an I-PPE attack is documented in a <a href="https://sprocketfox.io/xssfox/2021/02/18/pipeline/">blog post</a> by <a href="https://twitter.com/xssfox">@xssfox</a>, where an AWS CodeBuild pipeline of a website belonging to AWS allowed anonymous attackers to modify a script executed by the build configuration file with the creation of a pull request, resulting in the compromise of deployment credentials.</p><h4>Examples: Secret exfiltration on Jenkins</h4><p>The following examples are presented on Jenkins for the sake of simplicity, as they are focused on showcasing the potential impact of the I-PPE vector. However, these vectors are agnostic to the CI solution, and are equally exploitable in other environments vulnerable to I-PPE.</p><p>In Jenkins, controlling the Jenkinsfile allows running various functions made available by the system, like loading credentials stored in the instance credential store as environment variables. Without the ability to modify the CI configuration file, attackers cannot call these functions directly, significantly reducing the amount of potential vectors for stealing credentials.</p><p>However, credentials can be obtained directly from the available environment variables. For a successful exfiltration of credentials from environment variables, these variables must be loaded before the step running the command manipulated by the attacker is executed².</p><p>In all of the following examples we’ll use the Jenkinsfile presented below, focusing on a different stage in each example.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ae1ca640e40137551610b6b6f486727/href">https://medium.com/media/7ae1ca640e40137551610b6b6f486727/href</a></iframe><h4>Example 1: Makefile</h4><p>In the “build” stage, we can see that the <em>make build</em> command is running. The “make” command reads and executes commands defined in the “Makefile” file residing in the same working directory — and therefore will exist in the same repository as the Jenkinsfile itself. Let’s have a look at the contents of the makefile:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdc9af6e6fb74edc3acd8ad8185a6599/href">https://medium.com/media/cdc9af6e6fb74edc3acd8ad8185a6599/href</a></iframe><p>The <em>build</em> function runs commands having to do with building the code (again, redacted for brevity). The actual contents of the build block don’t have any actual significance for the attackers, all they care about is that they are able to change the function content to include their payload.</p><p>The malicious command below sends the environment variables to a remote server controlled by the attacker, including the Docker Hub credentials which are loaded in memory.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3300998b147148efcef641bd9de97bba/href">https://medium.com/media/3300998b147148efcef641bd9de97bba/href</a></iframe><h4>Example 2: Code tests</h4><p>In the “test” stage we can see the “go test” command, which runs tests on application code written in Go, based on test files stored in the repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lBvIt2316dOs95iIlIABvg.png" /></figure><p>GO’s testing package runs on files having the “_test.go” suffix which reside in the repository, like the example test file below³.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cb087fb45de494798eafe1ab0a12723/href">https://medium.com/media/3cb087fb45de494798eafe1ab0a12723/href</a></iframe><p>Code tests run code written in the language they test. The attackers can modify this test file to execute OS commands, and exfiltrate all environment variables to a remote server:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ceff19554c1866e1a561cd2a17dd4ed7/href">https://medium.com/media/ceff19554c1866e1a561cd2a17dd4ed7/href</a></iframe><p>Note that in the original Jenkinsfile, no credentials are loaded in memory during the “test” stage. However, this does not mean there is no potential damage to be done here — The attacker runs an OS command that extracts environment variables. These variables are often juicy and are likely to contain tokens available for all pipeline stages or other sensitive information.</p><h4>Example 3: Terraform</h4><p>The next stage in the pipeline runs terraform commands, in this case — <em>terraform plan</em>, which is used for creating the execution plan and presenting the changes that are about to be made by terraform. Execution of this command requires read only access to the cloud environment, as it doesn’t make any actual changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USw8xl4J1XFbmPhh2ywQng.png" /></figure><p>Usually, the <em>terraform apply</em> command follows (in deployment pipelines), executing the created plan against the environment — and it is at this point that permissions to modify the environment are needed. For this reason, a recommended best practice for running the <em>apply</em> command is to avoid running it over unreviewed code.</p><p>In contrast to <em>apply</em>, the <em>plan</em> command typically does run over unreviewed code, as a kind of a simulation of the planned infrastructure changes. The thing is that although it looks harmless, <em>plan </em>can be abused to execute OS commands. This vector was researched and published by <a href="https://twitter.com/rung">Hiroki Suezawa</a> (<a href="https://github.com/rung/terraform-provider-cmdexec">https://github.com/rung/terraform-provider-cmdexec</a>) and <a href="https://twitter.com/_alxk">Alex Kaskasoli</a> (<a href="https://alex.kaskaso.li/post/terraform-plan-rce">https://alex.kaskaso.li/post/terraform-plan-rce</a>).</p><p>In Terraform, it is possible to make use of various Providers, which are basically plugins that assist with interacting with cloud providers and APIs. Providers declared in terraform code are installed and then used as part of the commands it runs, including <em>terraform plan</em>.</p><p>The security challenge with using providers in <em>terraform plan</em> stems from a relatively unfamiliar fact that providers can execute OS commands. A custom provider can be added to the repository, ultimately running malicious commands when <em>terraform plan</em> is executed in the “terraform” pipeline stage. Another option is to fetch the provider from the Terraform registry, which stores providers allowing execution of OS commands. This will allow an attacker to exfiltrate the AWS credentials loaded as part of this stage.</p><p>Unfortunately, since <em>terraform apply</em> usually follows the <em>plan</em> command, it’s common to see the same set of credentials provided for both commands, despite the fact that Write permissions against the Cloud environment aren’t needed for <em>terraform plan</em>. This makes <em>terraform plan</em> susceptible to high impact attack scenarios, despite the original intent of <em>plan</em> for read-only purposes.</p><p>To summarize the potential impact, an example attack flow taking advantage of <em>terraform plan</em>:</p><ol><li>The attacker obtains access to a repository linked with a pipeline. The pipeline configuration file is defined to execute the <em>terraform plan</em> command, based on the terraform code present in the repository.</li><li>The attacker proceeds to add code that uses a terraform provider which is designed to execute OS commands. The attacker’s code instructs the provider to execute a set of OS commands that use the AWS CLI to conduct malicious operations against the cloud environment.</li><li>The attacker creates a PR originating from their remote branch, which triggers the pipeline.</li><li>The pipeline runs and executes the <em>terraform plan</em> command, which uses the provider to execute the malicious code against the cloud environment.</li></ol><h3>Public PPE (3PE)</h3><p>Execution of a PPE attack requires access to the repository hosting the pipeline configuration file, or to files invoked by pipeline commands. In most cases, the permission to do so would be given to organization members — mainly engineers. Therefore, attackers would typically have to be in possession of an engineer’s permission to the repository to execute a direct or indirect PPE attack.</p><p>However, in some cases poisoning build pipelines is available to anonymous attackers on the Internet. Public repositories (for example Open Source projects) oftentimes allow any user to contribute — usually by creating pull requests, suggesting changes to the code. These projects are commonly automatically tested and built using a CI solution, in a similar fashion to private projects.</p><p>If the CI build pipeline of a public repository runs unreviewed code suggested by anonymous users, it is susceptible to a Public PPE attack, or in short — 3PE. This also exposes internal assets, such as secrets of private projects, in cases where the pipeline of the vulnerable public repository runs on the same CI instance as private ones.</p><p><a href="https://twitter.com/spaceB0xx">Tyler Welton</a> has conducted a great research on exploiting build servers using Direct-3PE at <a href="https://www.youtube.com/watch?v=mpUDqo7tIk8">DEF CON 25</a>.</p><h3>PPE — The defenders perspective</h3><p>PPE is definitely a highly sophisticated and — potentially — a highly impactful attack vector. Which begs the question — how do we secure our environments against PPE? There is a diverse set of preventative measures that optimize our posture against PPE. But an equally important task in relation to securing our environment against PPE — is knowing whether or not, and where, our environments are susceptible to PPE. While doable, this task is highly complex and requires obtaining the answers to these questions:</p><ul><li>Which SCM repos are linked to which CI/CD pipelines?</li><li>Which humans and applications have permissions against SCM repositories that are linked to CI/CD pipelines?</li><li>From where is each pipeline configured to pull the CI configuration file, and which accounts (humans / applications) have permissions to make changes to the configuration file?</li><li>What events trigger the execution of the pipeline?</li><li>Which blocks in the pipeline configuration file execute commands that are declared outside of the CI configuration file itself — in other files in the same repository or even in other repositories?</li><li>What secrets and permissions — within the CI/CD environment as well as on the underlying host — does each block within the CI configuration file have access to?</li></ul><p>Possessing the answers to these questions is mandatory for accurately painting our attack surface and understating which PPE vectors can an attacker carry against our environments, and which areas require the most immediate attention on the preventive measures front.</p><p>As far as the prevention, multiple settings and protections spanning across both SCM and CI systems need to be taken into consideration:</p><ul><li>Ensure that pipelines running unreviewed code are executed on isolated nodes, not exposed to secrets and sensitive environments.</li><li>Evaluate the need for triggering pipelines on public repositories from external contributors. Where possible, refrain from running pipelines originating from forks, and consider adding controls such as requiring manual approval for pipeline execution.</li><li>For sensitive pipelines, for example those that are exposed to secrets, ensure that each branch that is configured to trigger a pipeline in the CI system has a correlating branch protection rule in the SCM.</li><li>To prevent the manipulation of the CI configuration file to run malicious code in the pipeline, each CI configuration file must be reviewed before the pipeline runs. Alternatively, the CI configuration file can be managed in a remote branch, separate from the branch containing the code being built in the pipeline. The remote branch should be configured as protected.</li><li>Remove permissions granted on an SCM repository from users that do not need them.</li><li>Each pipeline should only have access to the credentials it needs to fulfill its purpose. The credentials should have the minimum required privileges.</li></ul><h3>Summary</h3><p>Access to SCM organizations and repositories is obtained by attackers all the time. Credentials, access tokens, and SSH keys are stolen, by any of the classic attack methods such as phishing, credential stuffing, or lateral movement in a company’s internal network.</p><p>PPE is a vector allowing attackers to leverage that access to execute malicious code in CI pipelines, leading the way for accessing production environments in a matter of minutes, or even seconds.</p><p>Engineers need to ask themselves: if your SCM token or user account gets compromised, how far can the attacker go?</p><p><em>[1] A pipeline can be vulnerable to D-PPE even if the CI configuration file resides on another repository, if a user has permissions to edit it and trigger the pipeline.</em></p><p><em>[2] The attacker might have additional options, but let’s keep it simple.</em></p><p><em>[3] Yeah, we know it doesn’t do any meaningful test, unless you’re not sure about the value of 1.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34f4e8d0d4e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/ppe-poisoned-pipeline-execution-34f4e8d0d4e9">PPE — Poisoned Pipeline Execution</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secret Diver — Searching for deeply hidden secrets]]></title>
            <link>https://medium.com/cider-sec/secret-diver-searching-for-deeply-hidden-secrets-6c7a05b54bea?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/6c7a05b54bea</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[secrets]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Wed, 19 Jan 2022 12:56:19 GMT</pubDate>
            <atom:updated>2022-01-19T15:51:02.250Z</atom:updated>
            <content:encoded><![CDATA[<h3>Secret Diver — Searching for deeply hidden secrets</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gc19A-Fw2CERIcgx" /></figure><p>We are glad to introduce Secret Diver, a tool that allows you to search inside Docker layers for secrets.</p><h3><strong>TL;DR:</strong></h3><blockquote>Docker images are composed of layers. These containers, even after modifications and updates, may have secrets hiding in previous layers. One often overlooked but vital practice should be to check and verify that these layers don’t expose your secrets. <br>We have built a tool that searches the different layers in a fun and easy way.</blockquote><p>Docker images are actually OCI images that are comprised of multiple layers. When docker builds an image, a new layer is created for each command in the docker file. The end result (docker image) is a tar file that consists of all the layers in tar file format and an additional manifest.</p><p>You can read more about how <a href="https://jessicagreben.medium.com/digging-into-docker-layers-c22f948ed612">Docker layers </a>work in this excellent article.</p><p>A very useful utility to analyze the different layers of a docker image is “<a href="https://github.com/wagoodman/dive">Dive</a>” which allows to analyze container structures</p><h4>Short example:</h4><p>Shown below is an example analysis of a docker image consisting of 3 layers:</p><ul><li>The first layer imports the base image (in this case alpine)</li><li>The second layer creates a .pypirc file with our secret inside</li><li>This layer with the secret is needed only when building the image</li><li>In a real-life scenario, between the second and third layers, you would typically find the logic supporting the building of the image, such as fetching internal Python packages, running installation scripts, and so on.</li><li>The third layer deletes the .pypirc file</li></ul><pre><strong>dive</strong> cidersecurity/secret-deleted-in-container</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z9VHSGC2yhxc1GmX" /><figcaption>Running dive on a container</figcaption></figure><p>The final image built is uploaded to the container registry and becomes available for download. Once someone downloads the image and runs it in a docker environment, only the squashed file system is visible <strong>(without the secret in layer 2).</strong></p><pre><strong>docker run</strong> -it --entrypoint=/bin/sh \<br>               cidersecurity/secret-deleted-in-container</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uzdVkcqVx-NbAGun" /><figcaption>Opening shell inside the container</figcaption></figure><p>So a question we often ask ourselves, when we are trying to be security-minded and ensure there are no overlooked exploitable oversights, is whether someone can still obtain our secret, despite it being deleted.</p><blockquote>Regretfully, no surprises here. The answer is YES!!</blockquote><h4>We’ll now demonstrate how you too can replicate the method to obtaining deleted secret.</h4><p>If we take the sample scenario above, there are essentially two ways to do so:</p><ol><li>If you look closely at the “dive” above, we can view the commands that were run and, because we wrote the password as one of the commands, it is written to the manifest. The manifest contains all the commands and all the arguments that were used in each stage.</li><li>As mentioned above, the final image contains all the layers squashed together. An advanced user can extract any layer they wish from the image and read the file system contents of the layer extracted.</li></ol><h4>Let’s get our hands dirty:</h4><ol><li>We pull and save the container image by saving it as a tar file and then extracting the layers.</li></ol><pre><strong>docker pull</strong> cidersecurity/secret-deleted-in-container</pre><pre><strong>docker save</strong> cidersecurity/secret-deleted-in-container -o \<br>            secret-container.tar</pre><pre><strong>tar xvf</strong> secret-container.tar</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SPRQ37ErGaea2uoG" /><figcaption>Extracting the image locally</figcaption></figure><pre><strong>tar xvf </strong>7fb8df49962686...06d733/layer.tar<br>ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X70gMMSg6AzWld8E" /><figcaption>Extracting the layer</figcaption></figure><p>3. When we view the contents, we see the deleted .pypirc we were looking for. <strong>Great Success!!</strong></p><pre><strong>ls -alh tmp</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PfMqpN9EP9PbEuWX" /><figcaption>Password found in file</figcaption></figure><p>Now the fun part begins — how do we now hack this PoC for scale and repeatability?</p><p>This got us thinking about automation. What if we could do this automatically and combine a secret scanner to scan each layer independently for secrets?</p><h3>We are Proud to Introduce Secret Diver</h3><p>We developed the Secret Diver tool, specifically to prevent the scenario described above. We often forget that when we delete a secret from the repository, it still exists in the log history of the repository. But hackers don’t forget that — it’s the first thing they check for. That’s why we have you covered with Secret Diver.</p><h4>How does it work?</h4><p>Steps to use the tool:</p><p><strong>Manually</strong></p><ol><li>Install secret-diver from here — <a href="https://github.com/cider-rnd/secret-diver">https://github.com/cider-rnd/secret-diver</a></li></ol><pre>go install github.com/cider-rnd/secret-diver@v0.0.1</pre><p>2. Run secret-diver in<strong> human mode</strong> (machine mode is the default)</p><pre><strong>secret-diver </strong>-image=cidersecurity/secret-deleted-in-container -human</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4uClSYEGaz_E77De" /><figcaption>Running secret diver in human mode</figcaption></figure><p><strong>Inside your CI</strong></p><p>Run Secret Diver as part of the build process so that newly created images are scanned immediately after creation:</p><pre><strong>docker build</strong> -t &lt;image-tag&gt; .<br><strong>secret-diver</strong> -image=&lt;image-tag&gt;</pre><p>It is recommended to use the default machine mode which outputs to friendly <a href="https://docs.oasis-open.org/sarif/sarif/v2.0/sarif-v2.0.html">SARIF</a> output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ElyZg2_oa34__6Z2" /></figure><p>2. All done, you are good to go. Pass this sarif file to your favourite issue collection system or just view it <a href="https://sarifweb.azurewebsites.net/">here</a></p><blockquote>P.S. Going forward we recommend, from the security perspective, using multi-stage docker files which will upload only the final stage and automatically squash the previous stages into the base stage.</blockquote><p>Have fun with the tool and <strong>do no harm!!!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c7a05b54bea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/secret-diver-searching-for-deeply-hidden-secrets-6c7a05b54bea">Secret Diver — Searching for deeply hidden secrets</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizing CI/CD from an attacker’s perspective]]></title>
            <link>https://medium.com/cider-sec/visualizing-ci-cd-from-an-attackers-perspective-22dfa38c9d09?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/22dfa38c9d09</guid>
            <category><![CDATA[graph-data-model]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[ci]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[dc]]></category>
            <dc:creator><![CDATA[Leon Goldberg]]></dc:creator>
            <pubDate>Mon, 10 Jan 2022 14:03:50 GMT</pubDate>
            <atom:updated>2022-01-10T14:03:50.111Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kvu6oAH7A3665uKNSCrjdQ.png" /></figure><p><strong>Lessons learned and insight gained from a year of modeling and engineering CI/CD graphs</strong></p><p>CI/CD environments and processes are increasingly becoming a key area of focus for both hackers and — consequently — for defenders.</p><p>At <a href="https://medium.com/cider-sec">Cider Security</a>, we believe that visualizing these environments and modeling the relationships between the different objects in them is a key guiding principle for anyone interested in building the necessary capabilities to protect CI/CD ecosystems.</p><p>I decided to take the opportunity to share some insights around the elaborate task of engineering a solution that represents CI/CD in graphs, after having spent a year doing so.</p><h3>Intro</h3><p>One of our primary goals at Cider is to help organizations protect their CI/CD ecosystem.</p><p>Attackers are continuously focusing on hacking CI/CD systems. 2021 alone brought us — among many others — <a href="https://www.zdnet.com/article/codecov-breach-impacted-hundreds-of-customer-networks/">codecov</a>, the <a href="https://fossa.com/blog/embedded-malware-npm-coa-rc-ua-parser/">npm coa/rc/ua-parser hijacking</a>, and the recent and <a href="https://logging.apache.org/log4j/2.x/security.html">highly disruptive log4j vulnerability</a>.</p><p>Once an attacker gains a foothold within an organization’s CI/CD ecosystem, they will look to advance their position within, en route to eventually finding and exfiltrating valuable assets. To move within the ecosystem (e.g., SCM to CI, or CI to an artifact repository, etc.), the attacker will strive to perceive a graph representation of the ecosystem, looking for the relationships between the different systems.</p><p>From the attacker’s perspective, the ecosystem is not a group of isolated &amp; siloed systems but a graph of highly interconnected systems, which contrasts the traditional <a href="https://github.com/JohnLaTwC/Shared/blob/master/Defenders%20think%20in%20lists.%20Attackers%20think%20in%20graphs.%20As%20long%20as%20this%20is%20true%2C%20attackers%20win.md">defender list mindset</a>, where systems and assets are secured individually in a silo.</p><p>It’s my experience that both dev and security share similar concerns about the role that CI/CD plays in the organization’s attack surface. I vividly remember a moment in one of my previous companies where I became concerned with the inadequate visibility we had into our CI/CD ecosystem. Ultimately, we had to deploy pipelines that had access to, and credentials of, the most sensitive environments storing our customers’ most lucrative data.</p><p>Even though this was the reality, I did not feel we understood the blast radius of a security breach in some of the systems, and whether or not, ultimately, the assets that our customers were entrusting us with — were safe.</p><p>At Cider, we understood that to effectively and holistically secure CI/CD ecosystems, we must match the attacker’s perspective and think in graphs rather than lists.</p><p>Making this decision is the trivial part — modeling &amp; engineering is where the challenge comes in;</p><h3>CI/CD graph modeling &amp; engineering</h3><p>CI/CD ecosystems are complex, typically consisting of multiple systems, where each system is often complex by its own merit. In addition, each type of system is usually offered by several different vendors (with both overlapping and distinct features and properties), and each system can often integrate with several other systems by more than one means.</p><p>Here’s a simplified overview of a modern CI/CD ecosystem for demonstration:</p><ul><li>Source control management: A Github organization will consist of numerous users, collaborating on multiple repositories, sending webhook events on a variety of triggers, with endless branches and commits correlating to various branch protecting rules and conditions — and that’s just the tip of the iceberg.</li><li>Dependency/artifact management: Organizations typically use multiple package registries simultaneously (e.g., npm/JFrog), potentially containing millions of packages each, where each package is nesting its own dependency tree.</li><li>Integration: Jenkins pipelines can be executed by both pull and push; be defined inline or by manifests located in arbitrary repositories and branches. Jenkins’ marketplace consists of more than a thousand plugins that can extend or alter pipeline behavior.</li><li>Deployment: the OCI distribution specification helps standardize the distribution of container images; however, there are multiple overlay vendors with different authentication schemes (e.g., ECR, GCR, DockerHub, etc.).</li><li>Production: K8s consists of around ten baseline objects of varying complexity, but is also highly extendable. Understanding the structure of the underlying infrastructure (whether it’s cloud or on-premise) is important for a comprehensive view of the production environment.</li></ul><p>The above is a partial and low-resolution overview of a basic CI/CD ecosystem, but it does demonstrate the key resource families that can be modeled in a CI/CD graph, and I’ll refer to them throughout.</p><p>Here’s a more in-depth view of some of the interesting considerations and challenges we encountered at Cider while modeling and implementing CI/CD graphs:</p><p><strong>1. Naive vs. intelligent relationships</strong></p><p>Some intra-system relationships become apparent by simple observation. For example, an SCM repository that contains an SCM branch is easily observable, and a simple GraphQL response transformation can generate the appropriate nodes and edges.</p><p>In contrast, determining the effective permissions of an SCM user against an SCM branch is a more involved effort. It requires processing the user’s permissions against the relevant SCM organization &amp; against the relevant SCM repository, as well as the branch protection rules of the SCM repository.</p><p>Inter-system relationships are also often complex. For example, can you try to accurately, in high resolution, model whether a developer (or, more likely, a DevOps engineer) in your organization can push code directly to production? What about indirectly?</p><p><strong>2. Variance in graph resolution</strong></p><p>There’s good correlation between the resolution of the data in the graph and the insight an organization can derive from it. The aforementioned basic overview is a low-resolution one — although it covers a lot of ground, it contains little insight.</p><p>In contrast, consider a potential subgraph of a container registry. The most basic and low-resolution subgraph consists of a single node that represents the registry as a whole. However, increasing the resolution, nodes representing the registry’s repositories emerge, followed by the images of each repository… the layers of each image in each repository… the content of each layer… and so on, and so forth.</p><p>With these high-resolution nodes in mind, not only are we already able to gain interesting insight into the container registry (e.g., images’ SBOM), but we have also discovered opportunities to create intelligent relationships between the higher resolution nodes to nodes of different systems (for example a package in a container image that is identical to a package in JFrog repository — or even more interestingly, a package in a container image that has an identical name but different content in comparison to its JFrog counterpart).</p><p><strong>3. Variance in raw data (frequency, volume, interfacing)</strong></p><p>Raw data that can be transformed to graph nodes and edges comes in all shapes, sizes, and frequencies.</p><p>Roughly speaking, the higher resolution data will be naturally more granular and change more frequently. Following up on the container registry example, the lower resolution nodes (the actual registry and the contained repositories) will change infrequently, whereas the higher resolution nodes (the images and the underlying layers) will change frequently — and the changes will often carry greater deltas (e.g., imagine the changes to a package-lock.json when new packages are introduced).</p><p>Additionally, the higher resolution data will often require more advanced extraction and processing methods, while the lower resolution data is often retrievable by simple REST or GraphQL requests.</p><p>Following up on the container registry example: listing the images of some container registry repository is easily achievable via a REST request — however, deriving the full SBOM graph (with both OS-level packages and application-level packages) of a new image in the aforementioned repository is a significantly more involved process.</p><p><strong>4. Data integrity, versioning, and resiliency</strong></p><p>Besides the interesting challenge of modeling CI/CD graphs, retrieving the required data, transforming it, and storing it in a way that’s performant, resilient, and consistent is likewise an interesting challenge.</p><p>As previously mentioned, the relevant raw data comes in in all shapes, sizes, and frequencies. Some data is more appropriate to be pulled periodically, whereas other data is more efficiently retrieved via registration to push events or streams.</p><p>The synchronization of all the different data avenues is critical to ensure consumers of the graph are consuming consistent data that is not in a transient state.</p><p>It is not enough to rely on ACID stores, as the interoperability between the CI/CD systems is great, where a single change in a high-resolution node may lead to a cascade of altered intelligent edges and nodes.</p><p>The process of determining the new relationships in such an event often involves a sequence of operations that have to be performed asynchronously due to both a potentially high number of hops, as well as prolonged execution durations.</p><p>In the interim, the graph must remain in a consistent state — albeit not necessarily accurate in terms of real-time representation. How accurate your graph must be in terms of real-time representation is derived from your SLA considerations, which may indeed lead to very different architectures.</p><p><strong>5. Graph presentation and interaction: user-friendly nodes/edges vs. algorithmic nodes/edges</strong></p><p>Another interesting topic in the realm of CI/CD graphs is user-friendly nodes and edges. Although high-resolution nodes and edges are great for deriving in-depth insights, they’re not all necessarily adequate for the presentation layer, often being noisy or just not interesting for observation in many cases.</p><p>In particular, intelligent edges between high-resolution nodes meant for vector calculations are often not intuitive or user-friendly. Therefore, derivative user-friendly nodes and edges should sometimes be generated to present the insights in a user-friendly and intuitive way.</p><p>Furthermore, user interaction is another interesting challenge. Users may want to, for example, simulate how an altered configuration of a resource affects blast radius. A translation layer between the user-friendly nodes/edges and the algorithmic nodes/edges can be implemented to achieve the required bi-directionality.</p><p><strong>6. Data normalization</strong></p><p>Each type of CI/CD system is often offered by several different vendors. Between the various vendors, there are varying degrees of overlap in terms of features and properties. When building generic CI/CD graphs, our motivation is to be able to present a vendor-agnostic representation of the ecosystem; otherwise, each integration of a system into our graph will be tedious.</p><p>Under ideal circumstances where the overlap is great (e.g., container registries), adding integration with a new vendor will merely require a thin transformation layer, and all relevant deliverables will work without much additional effort.</p><p>Typically, systems based on standardized protocols or specifications (e.g., Git, OCI, etc.) will contain substantial overlap between the different vendors and will be easier to normalize, whereas the non-standardized ones will be more difficult.</p><p>As such, it’s important to strive for normalized nodes and edges — but to also know when to “throw in the towel” and define vendor-specific ones. Some properties and relationships are entirely vendor-specific, where attempts for generic expression are futile.</p><h3>Derived insight from CI/CD graphs</h3><p>So we’ve modeled and engineered a graph. We took care of every one of the considerations mentioned above — but that is just the first half of our journey. Now it’s time to capitalize on the data;</p><p>Here are some prominent examples:</p><p><strong>1. Enhanced observability at scale</strong></p><p>The lower to medium resolution nodes and edges represent the major resources in a CI/CD ecosystem, such as SCM repositories, SCM users, CI pipelines, container registry repositories, etc. Each one of these nodes is potentially connected to several of the other nodes-making a tabular representation of the ecosystem somewhere between highly confusing and not possible.</p><p>For example, imagine the average medium-sized organization, maintaining a few hundred SCM repositories. Each repository is potentially accessible by various means; username/password, SSH key, access token, OAuth token, and more. Repositories trigger builds off of various conditions — considering the source branch, branch protection rules, and other factors.</p><p>Ultimately, understanding which humans — and which applications — have the permissions to trigger which CI pipeline — and under what conditions — relies upon complex queries and conditions that cannot be adequately calculated by means that are based on tabular data representation.</p><p><strong>2. Basic attack vector/blast radius discovery</strong></p><p>Basic attack vectors become observable in higher resolution data, and the likelihood of correct identification of a vector increases with resolution and volume. The more auxiliary components and guardrails represented in the graph, the more false vectors can be eliminated, and the more refined blast radiuses can be derived.</p><p>For example, imagine a subgraph of a K8s cluster. Some possible nodes in such a subgraph are pods, containers, image layers, layer SBOM, etc. Additionally, let’s suppose we identified a vulnerable version of log4j in one of the layers’ SBOM.</p><p>On the one hand, with enough volume and a high enough resolution, this can allow an attack scenario that leads all the way to the underlying cloud infrastructure (e.g., via lateral movement to a privileged pod) or to a database with customer data (e.g., reading a mounted secret).</p><p>Alternatively, the same scenario can lead to absolutely nowhere, e.g., the library isn’t loaded to the memory, or no significant lateral movement is possible.</p><p>The latter is nonetheless a security risk — however, the difference in severity from the former is significant, and it is essential to be able to make the distinction.</p><p><strong>3. Advanced attack vector/blast radius discovery</strong></p><p>The log4j scenario described above is potentially disastrous — however, its graph discovery and representation are reasonably straightforward.</p><p>At Cider, we’re engaged in researching advanced CI/CD vectors, often involving tens of high-resolution nodes with multiple different inter-system edges between the various nodes.</p><p>A nice example of what we consider an advanced vector is a vector, or rather a combination of several different vectors, that accurately describe whether a CI/CD pipeline will be executed on a pull request or a pushed commit to an SCM repository — and if so, what code will be executed in the pipeline, and what sensitive information is potentially accessible to that code.</p><p>Sensitive information may indeed be exfiltrated (e.g., codecov), but it doesn’t necessarily end there. CD pipelines often have access to credentials that, unsurprisingly, facilitate continuous delivery. These pipelines will often contain access to write credentials to your container registry, your K8s API, and other production-related integrations.</p><p><strong>4. Attack vector/blast radius simulations</strong></p><p>I’ve briefly touched on simulations before, and indeed, simulating different high-resolution CI/CD graph states based on simulated modifications to a live CI/CD system is a powerful discoverability measure.</p><p>The relationship between security, DevOps, and dev teams involves a continuous cycle of give and take — applying the most comprehensive and strict security measures for every feature or component in every system might severely hinder development velocity, and consequently, hinder GTM or TTM.</p><p>On the other hand, disregarding security will eventually lead to the accumulation of significant security debt, which will become progressively more challenging to account for.</p><p>It is therefore imperative to find the right balance — however that is often easier said than done. As evident, CI/CD ecosystems are complex, and understanding the full effect on the blast radius of lacking a security measure or control is difficult without at least a moderately high-resolution CI/CD graph.</p><p>By employing a high-resolution CI/CD graph, different states based on different simulated security controls can be derived algorithmically and visualized in a user-friendly presentation.</p><p><strong>5. Discoverability</strong></p><p>The dynamic and fast-paced nature of today’s engineering ecosystem has caused managing our environments with configuration/infrastructure-as-code, rather than manually, inevitable — it is imperative to achieve maximal replicability and restorability.</p><p>However, for various reasons, exceptions are often made — resources are not deployed via IaC; permissions are granted ad-hoc; code is not templated, etc.</p><p>Machine learning for graphs can help discover patterns and anomalies in CI/CD ecosystems. Applying unsupervised clustering algorithms on graphs can uncover both lower-resolution anomalies (e.g., unusual SCM user permissions, unusual SCM repository branch protection rules, etc.), as well as higher-resolution anomalies (e.g., unusual Dockerfile base images, unusual K8s manifests, unusual SBOM, etc.).</p><h3>Closing thoughts</h3><p>Graph structures and algorithms have excellent synergy with CI/CD ecosystems. Dramatically improved low-resolution visibility and high-resolution blast radius discovery are just some of the possible benefits of representing CI/CD ecosystems with graphs.</p><p>However, successfully doing so at scale is a challenging yet fascinating R&amp;D endeavor. CI/CD systems have hundreds of potential graph nodes and edges to be modeled, requiring the orchestration of multiple, diverse, and often complex data sources.</p><p>Fortunately, the task is as fascinating as it is challenging. There’s virtually no limit to how complex &amp; high-resolution your graph may be.</p><p>From exploring and modeling esoteric or extremely high-resolution data through making sure your infrastructure &amp; application tiers support its retrieval and presentation in a way that’s scalable and efficient, CI/CD graphs are an R&amp;D oasis.</p><p>These capture the essence of the challenges, dilemmas, and thoughts processes that make up for our day-to-day at Cider when working on CI/CD graphs. If you find this topic as fascinating as I do, I am always looking to exchange thoughts and ideas. Feel free to reach out to me at my <a href="/service/mailto:leon@cidersecurity.io">email</a> or DM me on <a href="https://www.linkedin.com/in/leon-goldberg/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22dfa38c9d09" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/visualizing-ci-cd-from-an-attackers-perspective-22dfa38c9d09">Visualizing CI/CD from an attacker’s perspective</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NPM might be executing malicious code in your CI without your knowledge]]></title>
            <link>https://medium.com/cider-sec/npm-might-be-executing-malicious-code-in-your-ci-without-your-knowledge-e5e45bab2fed?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/e5e45bab2fed</guid>
            <category><![CDATA[npm]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Mon, 03 Jan 2022 16:03:29 GMT</pubDate>
            <atom:updated>2022-01-03T16:11:34.553Z</atom:updated>
            <content:encoded><![CDATA[<p>How to tell if you are using NPM safely within your CI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*64JTMmJNwI8NxwRKKtbNsg.jpeg" /></figure><p>The JavaScript ecosystem is highly reliant on dependencies. <em>And all I wanted was a method to safely download my desired dependencies from the internet.</em></p><p>The industry standard for doing so in Javascript is “NPM” or “Node Package Manager”. As a developer, when installing node.js software, I usually run “npm install” to download all the necessary packages for the software to run correctly.</p><p><strong>Running “npm install” does two things automatically:</strong></p><ol><li>It downloads required dependencies from the internet.</li><li>It runs all scripts necessary to properly build the software.</li></ol><p><strong>Both of these actions come with security risks.</strong> You can read about the risks associated with the first one in my article on <a href="https://medium.com/cider-sec/this-time-we-were-lucky-85c0dcac94a0">dependency attacks</a>, <strong>but</strong> <strong>this article is all about potential problems with the second — running external scripts.</strong></p><p>In its first phase, NPM goes to the package repository and fetches the needed packages. To authenticate against this package repository, it usually uses secrets/credentials which are accessible to the NPM client when accessing the package repository.</p><p>When running the second phase, NPM executes scripts, <strong>but even though access to the package repository is no longer required</strong>, <strong>NPM still has access to the same credentials</strong> while doing so.</p><h3><strong>Why is this a problem?</strong></h3><p>When NPM installs and runs scripts, it does so for all of the package dependencies, their dependencies, and so on. <strong>Because the developer doesn’t control all the sub-dependencies and their scripts, unverified and potentially malicious scripts will gain automatic access</strong> to the credentials and secrets used in the first action stated above.</p><p><strong>To mitigate this problem, NPM allows you to use the command “ignore scripts”</strong>, to disable all scripts associated with dependencies from running. <strong>But now we have a different problem: there are packages that <em>need</em> the ability to run scripts</strong> for purposes such as compiling, fetching resources, etc. These desired scripts can be executed separately, after the install phase. This would prevent the scripts from gaining access to secrets that are necessary only at the installation phase.</p><p>For example:</p><pre><strong># Install phase with access to secrets</strong></pre><pre>(<br>    NODE_AUTH_TOKEN=your-secret-token-here<br>    npm ci --ignore-scripts<br>)</pre><pre><strong># Build phase with no access to secrets</strong></pre><pre>(<br>   npm rebuild<br>   npm run install --if-present<br>   npm run prepare --if-present<br>)</pre><p>For more details about how to do this see this great article on a <a href="https://wildwolf.name/secure-way-run-npm-ci/">secure way to run NPM CI</a> by Wild Wild Wolf.</p><h3><strong>However, this solution on its own is not enough</strong></h3><p><strong>In my previous article about </strong><a href="https://medium.com/cider-sec/malicious-code-analysis-abusing-sast-mis-configurations-to-hack-ci-systems-13d5c1b37ffe"><strong>malicious code analysis</strong></a><strong>, I described how security scanning tools can be affected by the environment they are running in.</strong> For example, when a scanner is executed inside a certain directory, it will begin by picking up any configuration files it needs from its current directory.</p><p>These configuration files are supposed to tell the tools how to behave (e.g., to skip certain directories, skip checks, ..). Some, however, can also be used to instruct the tool to execute commands, thereby using the scanning tool itself to perform the unwanted actions it is meant to prevent.</p><p><strong>The same method can be applied to NPM.</strong> By adding a <strong>.npmrc</strong> file to the intended repo, the developer can manipulate the configuration which is picked up by the “<strong>npm</strong>” command.</p><p>For example, in NPM version 6, adding the command<br>“onload-script=${PWD}/evil-script” to the .npmrc file will execute the script “evil-script” (also added to the repository by the malicious developer) even if NPM has been given the “ignore scripts” command. In version 6, this will happen when executing any npm command, including the security function “npm audit,” “npm ci — ignore-scripts” and any other command such as “npm whoami”.</p><p>A similar action can be taken in the later npm versions 7 and 8 by replacing the git command in .npmrc, adding the following line to the configuration file: “git=${PWD}/evil-script.sh”. In this case, this would only affect the “npm ci” and “npm i” functions, however, the scope of potential harm is exactly the same.</p><p><strong>This means that any developer is able to execute code on almost any pipeline </strong>in your organization if it is using “npm”. Running a simple script to extract environment variables can seriously damage your environment. Therefore, obtaining developer permissions (whether by an attacker or a rogue developer looking to hack pipelines) can be used to cause real harm.</p><h3><strong>Disclosing the flaw to NPM:</strong></h3><p>I contacted NPM (GitHub) through their official bug bounty program and got the following response:</p><blockquote>Essentially ignore-script is designed to prevent the execution of malicious pre/post-install scripts of packages defined inside dependencies. CI Triggered on a pull request is intended to execute any code pushed as part of PR, however, this class of issue is usually<a href="https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions"> prevented by running CI jobs in an isolated environment</a> which GitHub Actions and Travis CI employs.</blockquote><p>I agree in principle that isolating CI Jobs and giving them minimal permissions is a wise preventative measure, but it can be very hard to implement in practice.</p><p>In my experience, <strong>I have seen many different CI/CD solutions — and have encountered many pipelines that still allow too much access to internal resources.</strong> While steps are being taken in the right direction, I still believe that the safest option would be a “defense in depth” approach. This means that <strong>we need to both isolate the environments of our CI/CD and adjust the tools we run to prevent unintended code execution.</strong></p><p>To continue pursuing the subject, I have opened an issue to NPM which is still under triage:</p><p><a href="https://github.com/npm/cli/issues/4101">https://github.com/npm/cli/issues/4101</a></p><h3><strong>What should I do?</strong></h3><ul><li>Run CI/CD functions in isolated environments which contain the minimal needed permissions to both resources on the host as well as to external resources.</li><li>Execute NPM securely by separating the script execution from the install phase (See above).</li></ul><h4><strong>Possible Workarounds for ensuring the trustworthiness of npm configurations</strong></h4><ul><li>Remove project configuration file (from a cloned repository) before using npm.</li><li>Copy project configuration file from external trusted resources (not controlled by developers).</li><li>Implement measures to detect potentially malicious changes in configurations (such as in .npmrc files) before triggering the build process.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5e45bab2fed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/npm-might-be-executing-malicious-code-in-your-ci-without-your-knowledge-e5e45bab2fed">NPM might be executing malicious code in your CI without your knowledge</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing your resilience against Log4Shell]]></title>
            <link>https://medium.com/cider-sec/optimizing-your-resilience-against-log4shell-ba5178e2078a?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/ba5178e2078a</guid>
            <category><![CDATA[application-security]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[dependency-injection]]></category>
            <category><![CDATA[log4j]]></category>
            <category><![CDATA[log4shell]]></category>
            <dc:creator><![CDATA[Cider Security]]></dc:creator>
            <pubDate>Mon, 13 Dec 2021 19:49:21 GMT</pubDate>
            <atom:updated>2021-12-13T20:06:37.379Z</atom:updated>
            <content:encoded><![CDATA[<p>Collection of actionable measures — across Prevention, Mitigation, Detection and assessment — for coping with the Log4Shell chaos</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fTvFsSgUR44UKY142GBEIQ.png" /></figure><p>As more information continues to unravel about the nature and characteristics of the Log4Shell vulnerability, it is clear that coping with this vulnerability is a complex task, and upgrading Log4j across your entire ecosystem is far from trivial. Defenders are therefore required to implement measures across multiple verticals — Prevention, Mitigation, Assessment and Detection — in order to optimize their posture in relation to the Log4Shell scenario.</p><p>Below is a list of actionable measures we’ve compiled — spanning across all 4 verticals— that can assist you in optimizing your resilience against this vulnerability:</p><h3><strong>Prevent</strong></h3><p>Upgrade version of all log4j packages to 2.15.0 — <a href="https://logging.apache.org/log4j/2.x/security.html">https://logging.apache.org/log4j/2.x/security.html</a></p><h3><strong>Mitigate</strong></h3><p>While the different mitigations may not provide a full-proof solution for the vulnerability, they are easier to implement and significantly reduce the attack surface, and therefore recommended as temporary measures until all relevant log4j packages are upgraded.</p><ol><li><strong><em>Mitigations in Java configuration:</em></strong></li></ol><ul><li>In Log4j releases &gt;=2.10, the vulnerability can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true.</li><li>For releases &gt;=2.7 and &lt;=2.14.1, all PatternLayout patterns can be modified to specify the message converter as %m{nolookups} instead of just %m.</li><li>For releases &gt;=2.0-beta9 and &lt;=2.10.0, the mitigation is to remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class</li><li>AWS have released a <a href="https://aws.amazon.com/blogs/opensource/hotpatch-for-apache-log4j/">hotpatch</a> that is to hotpatch a running JVM using any Log4j 2.0+.</li></ul><p><strong>2. <em>Server hardening:</em></strong></p><ul><li>Restrict egress access (LDAP/HTTP/HTTPS/DNS) from all relevant servers to prevent the malicious user-controlled payload from being able to communicate with the attacker’s servers and download malicious code.</li></ul><p><strong>3. <em>WAF</em>:</strong></p><ul><li>General recommendation is to add a rule to your WAF searching and blocking “jndi:ldap://” for any input from the outside world.</li><li>Many of the WAF vendors have updated their rulesets with specific detections for Log4Shell. Make sure to follow and implement the WAF-specific recommendations.</li></ul><h3><strong>Detect</strong></h3><p>Your attack surface consists of a combination of your organic assets and assets which exist in 3rd parties you’ve implemented inside your environment, therefore detection efforts should focus on both:</p><p><strong>Organic assets </strong>— Search for usage of Log4j within your codebase (log4j-api and log4j-core) by looking at the following artifacts -</p><ul><li>Pom.xml and other Java package manager files (ensure you inspect not only the root hierarchy — analysis should be done against the entire dependency tree)</li><li>Search in your codebase for imports of log4j code by looking for the following string: “org.apache.logging.log4j” (for example, in Github — <a href="https://github.com/search?q=org:apache+org.apache.logging.log4j&amp;type=code">https://github.com/search?q=org:apache+org.apache.logging.log4j&amp;type=code</a>)</li></ul><p><strong>3rd parties</strong> — Assess the resilience of 3rd party products within your environment and follow vendor specific patching/mitigation recommendations. <a href="https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592">This</a> resource contains many vendor specific bulletins around Log4Shell.</p><h3><strong>Assess/Investigate</strong></h3><p>Several measures exist to assess and validate whether your environment is vulnerable and/or if the mitigations implemented are effective:</p><ol><li>IOCs / YARA signatures useful for detecting compromised assets -</li></ol><ul><li><a href="https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b">https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b</a></li><li><a href="https://github.com/Neo23x0/signature-base/blob/master/yara/expl_log4j_cve_2021_44228.yar">https://github.com/Neo23x0/signature-base/blob/master/yara/expl_log4j_cve_2021_44228.yar</a></li></ul><p>2. <a href="https://github.com/projectdiscovery/nuclei">Nuclei</a> Payload to detect whether your assets are exploitable:</p><pre>Wget <a href="https://raw.githubusercontent.com/numanturle/Log4jNuclei/main/log4j-detect.yaml">https://raw.githubusercontent.com/numanturle/Log4jNuclei/main/log4j-detect.yaml</a> <br>nuclei -c 200 -t log4j-detect.yaml -u $URL_YOU_WANT_TO_SCAN</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba5178e2078a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/optimizing-your-resilience-against-log4shell-ba5178e2078a">Optimizing your resilience against Log4Shell</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our dependencies are under attack, and this time we were lucky…]]></title>
            <link>https://medium.com/cider-sec/this-time-we-were-lucky-85c0dcac94a0?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/85c0dcac94a0</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[npm]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[dependency-injection]]></category>
            <category><![CDATA[supply-chain]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Wed, 24 Nov 2021 04:42:50 GMT</pubDate>
            <atom:updated>2021-11-24T05:02:42.842Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-I-GT76TpuSDIJHL3DH7Ow.jpeg" /></figure><h4>How can we protect ourselves when our world infrastructure is under attack?</h4><p><strong>In the past few weeks, our world infrastructure has been under attack. </strong>The attack is very simple: Find a dependency package most of the world is using, hack into the developer’s account and update to a new version with some malware inside. Why was this possible? How can we protect ourselves? What are the potential problems and how can we address them?</p><h3><strong>Why is this possible?</strong></h3><p>When installing a package in Node.js, NPM (node package manager) automatically downloads all the package dependencies. Those dependencies then download additional dependencies, and so on, and so on, until all the needed dependencies have been acquired/downloaded.</p><p>Because keeping all of these dependencies up to date would be difficult and time consuming, NPM gives developers the option of automatically updating to new major/minor versions using special characters prefixed to the version. However, this option opens up the possibility that new dependencies will be upgraded automatically in the next build, which makes it possible to attack dependency owners by adding malicious content in the form of a new updated version.</p><p>Each version of a package<strong> </strong>has 3 sections: Major, Minor and Patch. This means, for example, that version 4.0.12 means:</p><ul><li>Major — 4</li><li>Minor — 0</li><li>Patch — 12</li></ul><p>When defining a version we can give different kinds of instructions, which will determine how the dependency will be upgraded. For example, “5.3.1” means “Use the exact version 5.3.1,” while simply writing “*” means “always use the latest version.” Other, intermediate options include:</p><ul><li><strong>4.x</strong> =&gt;Use all upgrades included in major version 4</li><li><strong>&gt;=4</strong> =&gt; Major version 4 and above</li><li><strong>~4.1.2</strong> =&gt; Major version 4, Minor version 1, Upgrade only to patch versions 2 and above</li><li><strong>^4.1.2</strong> =&gt; Major version 4, Upgrade to any Minor version above 1 and patch version 2 and above</li></ul><p>To get a better feel of how versioning works, you can try out the semgrep version calculator here: <a href="https://semver.npmjs.com/">https://semver.npmjs.com</a></p><p>The default option when installing a new dependency is the last example shown on this list <strong>(^4.1.2)</strong> which means that installing automatically upgrades the package to the next minor and patch versions.</p><p>Because of this, when hackers gained access to an account that has permissions to update the COA and RC packages and added malware to the package, any new installations downloading the latest “COA” or “RC” version automatically got this compromised version. Luckily for the world, this particular attack was not as stealthy or as comprehensive as it could have been, causing installations in pipelines to begin failing worldwide. This triggered an investigation by npm, and the malwared versions were quickly removed.</p><blockquote><strong>This time we were lucky</strong>, but next time a more targeted attack, running in stealth under the radar could be performed by taking advantage of this same weakness. This means that one mistake in a commonly used dependency or a calculated malicious act could leave most companies in the world compromised.</blockquote><h3><strong>How can we protect ourselves?</strong></h3><p>One important first step is using “npm ci” instead of “npm install.” Both are ways to install, but npm ci is much more strict in its dependency fetching policy.<strong> </strong>When we use “<strong>npm install</strong>,” we are telling the system to download<strong> </strong>whichever new packages the developer requested in the<strong> package.json</strong> file, including any automatic updates defined by the default semantic versioning.</p><p><strong>Npm ci</strong>, in contrast, <strong>requires</strong> the developer to upload a <strong>package-lock.json, </strong>which<strong> </strong>contains all exact versions of dependencies and sub-dependencies, thus telling the package manager exactly which dependencies to download. The developer then needs to periodically update the package-lock to reflect valid updates in the dependencies required. When using <strong>“npm ci”</strong> we are telling the system to <strong>download only </strong>the specific packages stated in the <strong>package-lock.json.</strong></p><p>A quick search on github finds <strong>11,294,073</strong> code instances of <strong>“npm install”</strong> versus <strong>279,920</strong> instances of <strong>“npm ci”</strong> which means the process in automatic scripts that downloads new repositories without thinking is 40 times more common than the stricter, safer one.</p><p>In order to create <strong>trusted/reproducible</strong> builds we have to use <strong>“npm ci”</strong> in all stages of the build systems (qa, testing, security, production) to be sure we are aligned with the same versions and nothing changed during the progression through the CICD pipeline.</p><h3><strong>Unfortunately npm ci comes with problems of its own…</strong></h3><p><strong>(a)</strong> <strong>Updating packages becomes a more complex process: </strong>Using<strong> “npm ci”</strong> introduces a new problem, since for each change in the dependency tree the developer would need to update dependency version changes, validate them and upload the newly created <strong>package-lock.json.</strong></p><p>Adopting this protocol can be hard for large organizations, but they can educate their developers to continually update the package-lock file or use solutions such as “<strong>dependabot</strong>” or “<strong>renovate”</strong> which can assist with creating <strong>PR </strong>to help.</p><p><strong>(b)</strong> <strong>Developers’ machines can still be vulnerable: </strong>Updating dependencies on the developers’ machines moves the vulnerability from the CI environments to the developers’ machines. <em>Anyone</em> who needs to run<strong> “npm install”</strong>, will be vulnerable and can be targeted for dependency attacks on their local machine.</p><p>To minimize this risk, organizations can add some preventive measures when developers run <strong>“npm install”</strong> by proxying their package registries and working only with versions which are above a certain age, for instance only allowing packages older than 1 week.</p><p>Furthermore, if they do have the need to run and update in the CI environments, organizations can make sure their systems are protected by preventing access to secrets, preventing access to sensitive systems, hardening pods and killing pods at first sight of suspicious behaviour.</p><p><strong>(c)</strong> <strong>The</strong> <strong>latest version of npm ci does not verify the lock file correctly:</strong></p><blockquote>We recently discovered that NPM 7 does not properly validate the package-lock as expected:</blockquote><blockquote>The npm ci command in npm 7.x and 8.x through 8.1.3 proceeds with an installation even if dependency information in package-lock.json differs from package.json. This behavior is inconsistent with the documentation, and makes it easier for attackers to install malware that was supposed to have been blocked by an exact version match requirement in package-lock.json.<br><a href="https://nvd.nist.gov/vuln/detail/CVE-2021-43616"><strong>https://nvd.nist.gov/vuln/detail/CVE-2021-43616</strong></a></blockquote><p>When writing this blog and researching best practices to protect yourselves we found out that in recent versions of npm, the “npm ci” command does not work as written in the documentation and does not verify the package-lock properly</p><h3><strong>What should security/devops teams do now?</strong></h3><h4><strong>Check if you have been attacked</strong></h4><ol><li><strong>Check if you have “COA” or “RC” versions in the package-lock versions<br></strong>Vulnerable Coa versions: <em>2.0.3, 2.0.4, 2.1.1, 2.1.3, 3.0.1, 3.1.3</em><br>Vulnerable RC versions: <em>1.2.9, 1.3.9, 2.3.9</em></li><li><strong>Verify Build logs<br></strong>Make sure the CI build logs did not execute any malicious code</li><li><strong>Confirm that vulnerable versions did not go to production<br></strong>If you don’t support reproducible builds (Using npm install in CI), you need to go to container/server, which uses a repo with the affected package and confirm that they don’t have these versions and/or the <a href="https://www.virustotal.com/gui/file/26451f7f6fe297adf6738295b1dcc70f7678434ef21d8b6aad5ec00beb8a72cf?nocache=1">malware signatures.</a></li><li><strong>Check the status of developer machines<br></strong>Check that your developers don’t have these <a href="https://www.virustotal.com/gui/file/26451f7f6fe297adf6738295b1dcc70f7678434ef21d8b6aad5ec00beb8a72cf?nocache=1">malware signatures</a><strong> </strong>on their machines.</li></ol><h4><strong>Employ preventative measures</strong></h4><ol><li>Use NPM v6 in your CI as a workaround.</li><li>Force usage of package-lock.json in your CI by running “npm ci — ignore-scripts” in your dockerfiles and build scripts.</li><li>Validate that all repos have a package-lock.json submitted to the repo.</li><li>We have created a quick program to check for instances of insecure npm usage and identify where the less secure “npm install” is being used.</li></ol><blockquote>1. Download CIMatch — “<strong>go get -v github.com/cider-rnd/cimatch”.<br></strong>2<strong>. </strong>Run “<strong>cimatch --human</strong>” in all your repos and ci configurations<br>3. Add “<strong>cimatch</strong>” to your security checks when scanning for new dockerfiles and jenkinfiles (More rules to come).</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85c0dcac94a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/this-time-we-were-lucky-85c0dcac94a0">Our dependencies are under attack, and this time we were lucky…</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Malicious code analysis: Abusing SAST (mis)configurations to hack CI systems]]></title>
            <link>https://medium.com/cider-sec/malicious-code-analysis-abusing-sast-mis-configurations-to-hack-ci-systems-13d5c1b37ffe?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/13d5c1b37ffe</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[hacker]]></category>
            <category><![CDATA[malicious-code-analysis]]></category>
            <category><![CDATA[sast]]></category>
            <category><![CDATA[vulnerability-management]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Wed, 03 Nov 2021 15:45:51 GMT</pubDate>
            <atom:updated>2021-11-03T15:45:51.375Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWPVXBYB55UM6xQQktK7xA.png" /></figure><h4>What happens when SAST tools do more than just scanning? What if security scanners abuse their privileges?</h4><h3>TL;DR</h3><p><em>I recently found a new method that allows secure code analysis mechanisms to be bypassed and even worse — abused to execute malicious code on their host. This allows anyone with access to a source control repository to run malicious code in sensitive environments — which may be abused to steal credentials, access tokens and more.</em></p><p>This is a result of ongoing research around exploiting SAST scanners that was first presented at <a href="https://www.youtube.com/watch?v=Jl-CU6G4Ofc">DEF CON 29</a> this summer.</p><h3>Part 1 — Prequel</h3><blockquote>The number one goal of any hacker is and always will be one thing: running their own {malicious} code on the target computer, or in more professional wording, remote code execution (RCE).</blockquote><p>Hackers want to hack production systems. That is where the crown jewels are. Where the confidential data is stored. Where mission-critical tasks are running. There are various ways to gain access to production and numerous different motives for doing this, but they all have one uniting goal: running unauthorized code on production environments.</p><p>For a long time now, CI systems, and engineering systems in general, are sought after by hackers due to the ease of getting to production through them. They store sensitive credentials used for deployment, and they have a red carpet paved to production for carrying out the actual deployments.</p><p>The attack vector I want to introduce is a new method of attack that allows running malicious code on CI systems, allowing sensitive production credentials to be exfiltrated and malicious artifacts to be deployed.</p><blockquote><strong>“Malicious Code Analysis” attack<br></strong>A method which abuses code analysis software configurations to run code on the host running the code analysis</blockquote><h3><strong>Part 2 — How it was discovered</strong></h3><p>Before going deep into the technical areas, I want to share details on what brought me to research the behavior of code analysis software. Developers and builders want to develop as quickly and freely as possible. In a world full of chaos and security issues, we want to be able to develop code in a rapid and frictionless manner, while using various automations such as security guardrails to protect us from making mistakes along the way.</p><p>This is where static code analysis (or in short — SAST) solutions come in.</p><p>They get code as input, and no matter how malicious or harmful the code is, it will never be executed or cause any harm. The software statically analyses the code and identifies potential flaws. And for this reason, both defenders and developers can safely run this static analysis software against any piece of code without being concerned about the code being executed.</p><p>But as a veteran hacker, I know these type of statements are rarely true. There is always an exception.</p><p>So, one day I built a simple code scanning solution. Running this solution on random code gave me my desired results — 99% of the time — but every once in a while, the service crashed. Interesting. This means the service was somehow impacted by the actual input (code) it analyzed. So I decided to go down the rabbit hole and debug my scanner and voila — I found out that a certain character combination brought the whole service to its knees.</p><blockquote>This character combination was a self-evaluating form in Clojure, meaning that when the <strong>scanner read the code it actually tried to execute it.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*1pqd2pCX5nlDqOx-ckC52g.png" /><figcaption>Self evaluating code in Clojure</figcaption></figure><p>One hour later, I had a shiny new code execution vector that runs whenever my code gets scanned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFfp0zjyINnZXLhRxrnykQ.png" /><figcaption><a href="https://github.com/jonase/kibit/issues/235">https://github.com/jonase/kibit/issues/235</a></figcaption></figure><p><em>So I had a working POC of a scanner that went far beyond the scope of statically analyzing the code it received as input. This started a chain of investigations for what else I can break.</em></p><h3>Part 3 — Features evolving into security bugs</h3><blockquote>Remember Shellshock?</blockquote><blockquote>Shellshock was born in 1989, but was only found in 2014. The big mystery here is: how does a 25-year-old bug hide so well?</blockquote><p>The answer is quite simple: when the bug was created it wasn’t critical. In short, the feature of assigning a function and executing it from the environment variables was not critical because in order to create an environment variable you had to have local physical access.<strong><br></strong><a href="https://unix.stackexchange.com/questions/157381/when-was-the-shellshock-cve-2014-6271-7169-bug-introduced-and-what-is-the-pat/157495#157495"><strong>Read more here</strong></a><strong>.</strong></p><p>Then came the internet. Developers needed a way to interact with processes running on remote computers. The best way to pass data for starting processes in bash was to send them through environment variables. As it is only data it was pretty safe.</p><p>Except in bash as we saw earlier we can create functions, and these functions can self execute, just like our self executing functions in Clojure in the previous part.</p><h3>Part 4- Connecting the dots</h3><p>Code scanning has similar concepts; from the dawn of development, we had different methods and techniques to scan code. The majority of them involved running the scanner locally on the machine of the developer writing the code. So there hasn’t really been any need to secure the tools themselves.</p><blockquote>But wait, we have since connected these tools to the internet, and running them daily in our automations.</blockquote><p>The same tools now run as part of our deployment systems, systems to which we grant lots of power and permissions to deploy our most sensitive code. This is a big responsibility to put on tools that were built a decade ago with no intention of running on the internet…</p><blockquote><em>Game on…</em></blockquote><h3>Part 5- The Malicious Code Analysis attack</h3><p>Code pushed into production has to be verified. We have to make sure it doesn’t contain any security flaws or misconfigurations. The more code we have in our organization, the more we need to rely on automatic scanning and safety procedures to make sure everything is running as it should.</p><p>The benefits of scanning are great; we can put in guardrails pretty easily — starting from the linting of our code to standardize the way we write code to adding security measures that help react in real-time to new bugs introduced into the code before they actually get deployed to our production systems.</p><p>However, like all good things, there are often downsides. Before detailing the risks, let’s align on what code scanning actually is.</p><p><strong>What is a code scanner? (i.e. static analysis program)</strong></p><blockquote>Static program analysis is the analysis of computer software that is performed <strong>without actually executing programs. </strong><a href="https://en.wikipedia.org/wiki/Static_program_analysis"><em>https://en.wikipedia.org/wiki/Static_program_analysis</em></a></blockquote><p>I’ll skip the detailed explanation of how scanners work and save this for a future blog. If you want a detailed explanation, you can watch my talk from <a href="https://www.youtube.com/watch?v=Jl-CU6G4Ofc"><strong>Defcon29</strong></a> here.</p><p>As head of Marketplace integrations, a big part of my job is getting intimately familiar with SAST scanners — including what they do and how they behave. So I found myself thinking: if I can get one scanner to execute code, there is a high probability that I can get other scanners to misbehave.</p><p><strong>My research question became very specific:</strong></p><blockquote>Assuming most scanners do intend to perform static analysis, how can I exploit their own behavior to bypass them and even exploit them?</blockquote><p><strong>Abusing SAST configuration</strong> — Most scanners generate a lot of noise (i.e. findings), with one of the main reasons for this being that they have many preconfigured rules that most certainly don’t apply to all code repositories universally. So by design, each one in its own way has a configuration used to disable different subsets of these rules according to the users’ requirements.</p><h3>Bypassing the scanners</h3><p>The first mission I defined was to bypass the scanner. Security puts a lot of effort into automating processes for scanning R&amp;D code to make sure it doesn’t go into production. I wanted to create code that would bypass all those protections, but didn’t want to work hard or rely on getting caught by some custom rulesets of the security team. I wanted to bypass it all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vUDJFRQ2M5IeIB9rX05krg.png" /></figure><p><strong>Here is a walkthrough of the process and research of one of my targets — Checkov:</strong></p><p><a href="https://github.com/bridgecrewio/checkov"><strong>Checkov</strong></a><strong> </strong>is an open-source tool designed to audit “infrastructure as code” files. The tool is super effective in finding flaws and easily shows problems found in our code.</p><p>In our example, I wanted to create a privileged pod using terraform. Usually Checkov will scan and fail my deployment (if executed inline inside the CI) or at least alert me.</p><p>Reading the documentation shows us that when the scanners are executed, they look for a configuration file inside its target directory.</p><p>Adding a <strong><em>.checkov.yml</em></strong> allows us to configure how we want the scanner to behave, including what rules to exclude from the scan. In this case, we want to add a configuration that will never fail any code, and use only my own custom check</p><pre><strong>soft-fail:</strong> true<br><strong>check:</strong><br>  - THIS_NOT_THE_CHECK_YOUR_ARE_LOOKING_FOR</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xWNWTQzG3ky0b9Od4uCDpA.png" /></figure><p>Once I add this configuration file, I “pass” the Checkov inspection even though my code has security flaws;</p><p><strong>Before adding a config file:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*L5_o8a_i5WGXthtzNEoRMg.jpeg" /></figure><p><strong>After adding our “custom” config file</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*1vggb78HXqem170MFTadhQ.png" /></figure><p>Essentially, this means that anyone who wants to bypass the SAST security inspection, for any reason (for example — to release to production faster without any ‘obstacles’ put in place by the security team, can follow these three simple steps:</p><ol><li>Create a config file disabling all rules</li><li>Put it into the code directory that is to be scanned</li><li>Never again deal with failing pipelines</li></ol><blockquote>This is not proper advice. Do not try this in you CI/CD pipelines.</blockquote><blockquote><strong>Important notice:<br></strong>We are not blaming or putting responsibility on OSS creators. We should not expect OSS to provide the same level of security as their commercial alternatives. These tools were not build to protect the systems they running in. That is the user’s responsibility.</blockquote><blockquote>We should assume OSS projects could potentially contain security flaws and make sure they are properly configured and running in a safe environment.</blockquote><blockquote>With this being said: we did reach out to Palo-Alto as they support <strong>Checkov</strong> commercially. The feedback was very fast and professional and they added a disclaimer to their README regarding this behavior. In addition, they added support for setting the config file, which was previously not possible.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xKwwZt5j8fP8fegqHfP6aA.png" /><figcaption><a href="https://github.com/bridgecrewio/checkov/blob/master/README.md">https://github.com/bridgecrewio/checkov/blob/master/README.md</a></figcaption></figure><p><strong>Why did this happen?</strong></p><p>Most SAST Engines rely on configuration, usually defining which rules we want to use, setting operation flags, or just even to enable debug mode.</p><p>Most scanners use the same convention of searching for a configuration in the working directory or target (scanning) directory. Because the most common use case for running scanners is from the working directory, it is common usage to add a scanning configuration there to define our custom policy.</p><p>Because of this behavior, and given access to a repo that is to be scanned, we can “abuse” the scanner logic and tell it to skip all rules, move to debug mode, or worse scenarios <em>(See next chapter)</em></p><p><strong>Is it common?<br></strong>Below, we can see a list of scanners that accept configuration files in the current working directory. For each one here, adding the “evil” configuration will disable all security testing.</p><pre><strong>phpstan.neon<br>--<br></strong>paths:<br>  - /var/empty</pre><pre><strong>kics.config<br>--<br></strong>queries-path: /var/empty</pre><pre><strong>.rubocop.yml<br>--<br></strong>&lt;%= exit! %&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UWUYAU5whSdmHrNjyFTQdA.png" /></figure><p>You can find our opensource repo we released with more researched configuration files here: <a href="https://github.com/cider-rnd/cicd-lamb"><strong>https://github.com/cider-rnd/cicd-lamb</strong></a></p><p>Detailed explanations can be found in my <a href="https://youtu.be/Jl-CU6G4Ofc?t=954">YouTube</a> video from Defcon.</p><h3><strong>Attacking the Scanners Right Back</strong></h3><p>So anyone who has gotten this far is probably asking themselves the same question: if the scanner really picked up my rule configuration, can I do something more interesting than just disable scanner features?</p><p>Because lots of scanners are basically frameworks for scanning code, they allow us developers, security engineers or DevOps to add our own custom rules.</p><p>These rules are sometimes created as code or have other ways of dynamic configuration loading. This leaves us with the possibility to <strong>execute code</strong> if and when we are able to override the configuration file.</p><p>Going back to the documentation, we can learn that Checkov allows us to load external checks, which is a great feature on its own. But when allowing configuration files to be loaded from the target directory, things can get pretty interesting.</p><p>So when we create a folder with python scripts in our target directory (in our example, we named the folder “scripts”), coupled with creating a configuration file that tells the scanner to pick up our custom python code from the requested folder, will result in executing our custom code when running the scan:</p><pre><strong>external-checks-dir:</strong><br>  - scripts</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNKKPuIs4cKltqVMBmEiKA.png" /></figure><p>So essentially this means that if I am a developer, <strong>or any human or application with permission on a source code repo</strong>, and I want to run malicious code on the CI, I can achieve my goal by doing the following:</p><ol><li>Create my own ‘malicious’ branch of the repo (knowing that when I issue a PR or commit code to this branch it will trigger a scan in the CI).</li><li>Add the following payloads to the branch —</li></ol><ul><li>A custom malicious configuration file</li><li>malicious files/folders triggered by the configuration file</li></ul><p>3. Push / issue a PR (or whatever action will end up triggering the build and running the scan)</p><p>4. Drink your favorite drink and wait for the shell to pop</p><h3>Conclusions</h3><p>Continuing our research, we created a list of scanners researched where we could find our <strong>Malicious Code Analysis</strong> attack. I am sure this is just the beginning; there are many other tools that weren’t checked. But now every time I install a new tool I can’t stop myself from thinking: is it vulnerable?</p><p>For more detailed examples checkout our CICD Lamb repo<br><a href="https://github.com/cider-rnd/cicd-lamb">https://github.com/cider-rnd/cicd-lamb</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0lQGGLphn6vFCGAuVmUmg.png" /></figure><p>Most security scanning is supposed to find security issues and flaws in code uploaded to our systems. The thought that the code itself will try to target the scanner itself usually is not thought of when designing scanners.</p><p>Today in 2021, we are enabling so much automation — with developers, DevOps, SecOps, AppSec engineers each trying to automate away as much of our work as possible in order to meet the continuously growing volume and demand by different organizational stakeholders. It is crucial to understand that this automation can be vulnerable to attacks as well and is being attacked</p><p>These automations affect every area of our software development lifecycle (SDLC) — a crucial part of our CI/CD environments — areas that have access to our most sensitive integrations and systems. These areas are the jackpot of all attacks. In this place, silently a shell pops without any understandable reason.</p><p><strong>We need to prepare for that day</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LKj5kmKRHSFUOZINwEFbVA.png" /></figure><h3>AfterMath (Fixing our problems)</h3><p>The first stage of fixing is understanding there is a problem. If you got this far, I hope you are starting to understand, that this is only the tip of the iceberg.</p><p>A few tips which will help you in the following quest to prevent these problems:</p><ul><li>Always scan your code in an ISOLATED environment. Block all internet access; isolate resources such as disk, CPU, network access and finally destroy after use.</li><li>Create a fixed configuration file, set the scanner to use it and not the default</li><li>Sanitize your CI/CD environment, and align it with the principle of least privilege. Your scanners need to run in an environment with the minimal possible privilege</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zpLOs_umzn11N5l3M7QCiw.png" /></figure><h3><strong>Acknowledgments</strong></h3><p>I want to thank all that supported and helped me with this research.</p><p><strong>Cider Security</strong> — Which gave me the time and resources to research this finding<br><strong>Daniel Krivelevich</strong> — Who pushed me into doing more thorough research<br><strong>Sharone Revah Zitzman</strong> — Thank you for the enormous impact on the presentation and the blog post<br><strong>Palo Alto, Barak Schoster</strong> — For the quick response and great feedbacks</p><p><strong>Special thanks to all the open-source SAST developers and contributors out there, Each one of you is helping the world become a safer place</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13d5c1b37ffe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/malicious-code-analysis-abusing-sast-mis-configurations-to-hack-ci-systems-13d5c1b37ffe">Malicious code analysis: Abusing SAST (mis)configurations to hack CI systems</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bypassing required reviews using GitHub Actions]]></title>
            <link>https://medium.com/cider-sec/bypassing-required-reviews-using-github-actions-6e1b29135cc7?source=rss----a827f45f8a43---4</link>
            <guid isPermaLink="false">https://medium.com/p/6e1b29135cc7</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[bug-bounty]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[appsec]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Omer Gil]]></dc:creator>
            <pubDate>Tue, 12 Oct 2021 14:47:37 GMT</pubDate>
            <atom:updated>2022-01-12T13:06:14.137Z</atom:updated>
            <content:encoded><![CDATA[<h4>Not using GitHub Actions? You’re also vulnerable.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eF7Dba5AO-cCg7WVwDADsg.jpeg" /></figure><h3>TL;DR</h3><p>A newly discovered security flaw in GitHub allows leveraging GitHub Actions to bypass the required reviews mechanism and push unreviewed code to a protected branch, potentially allowing malicious code to be used by other users or flow down the pipeline to production.</p><h3>The risk of a compromised user account</h3><p>GitHub is the most popular source control management system, serving millions of users and companies who use it to host their codebases. A GitHub organization can include any number of members — from several to hundreds or even thousands of members, with varying permissions. Write permissions are commonly granted to many users, as that is the base permission needed to directly push code to a repo.</p><p>As GitHub organization owners are aware of the constant need to protect their code against different types of threats, one attack vector that is always of great concern is that of a compromised user account. Because if an attacker is able to take control of an account with Write permissions (by obtaining their password, personal access token, or an SSH key), they can directly push code to the repo, which might be used by other software and users. This code can also go down the CI/CD pipeline, run unreviewed in the CI, or find itself in the company’s production environment.</p><p>So does a compromise of a single user account mean the attacker can push code down the pipeline without restrictions?</p><h3>Required reviews for merge</h3><p>To avoid this exact scenario (and for quality considerations, obviously), branch protection rules were created, and are used by nearly all engineering organizations today to provide baseline protection against such attack vectors. For sensitive branches (such as the default one — or any other branch we’d want to protect), we can set rules to limit an account with Write permissions to directly push code to it by requiring the user to create a pull request. Once a pull request is created, it needs to be approved by a preset number of approvers before it can be merged to the target branch.</p><p>For obvious reasons, a user cannot approve their own pull request, meaning that a requirement of even one approval, forces another organization member to approve the merge request in the codebase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X_oiSaLH6mdDkEhw" /></figure><p>Creating these protection rules that require one approval on a pull request by another organization member significantly reduces the risk of compromising an account, as the code needs to be manually reviewed by another user¹. This also prevents developers from pushing unreviewed code to sensitive branches.</p><h3>GitHub Actions</h3><p>GitHub has evolved significantly since its inception — and continues to add features, products, and tools for code management and shipment. One such tool is GitHub Actions — GitHub’s CI service — which is used to build, test, and deploy GitHub code by building and running workflows from development to production systems.</p><h3>Why is GitHub Actions relevant to bypassing protected branch configurations?</h3><p>Our research has exposed a flaw that leverages GitHub Actions to bypass protected branch restrictions reliant on the multiple reviews control.</p><p>So if your organization uses GitHub, but doesn’t use GitHub Actions for CI, you obviously have no reason to be concerned about this flaw, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/0*9pP2Sc-sSGKo62Du" /></figure><p>This begs the question, if you are an organization using GitHub, but haven’t yet gotten started with GitHub Actions, should you be worried about GitHub Actions’ attack surface, even if you never installed or used it in your organization? Let’s see.</p><ol><li>GitHub Actions is installed by default on any GitHub organization, and on all of its repositories.</li><li>Any user that can push code to the repo (Write permissions or higher), can create a workflow that runs when code is pushed.</li><li>With each workflow run, GitHub creates a unique GitHub token (GITHUB_TOKEN) to use in the workflow to authenticate against the repo. These permissions have a default setting, set in the organization or repository level. This setting allows granting the token with restricted permissions — <em>Read</em> permission on the contents and metadata scopes, or permissive permissions — <em>Read/Write</em> permissions on various scopes, such as contents, packages, and pull requests.</li><li><a href="https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#setting-the-permissions-of-the-github_token-for-your-repository">However</a>…</li></ol><blockquote>Anyone with write access to a repository can modify the permissions granted to the GITHUB_TOKEN, adding or removing access as required, by editing the permissions key in the workflow file.</blockquote><h3>Getting to the point</h3><p>So, what does a typical GitHub organization look like?<br>It generally has:</p><ol><li>Branch protection rules that can be set by organization owners to require pull request approvals before merge, where a user cannot approve their own pull request.</li><li>GitHub Actions installed by default for all GitHub organizations, on all repositories.</li><li>Permission for any user with Write access to run a workflow in the repo.</li></ol><p>Practically, this means an attacker that hijacks a user account and wants to push code to a protected branch, can simply push their malicious code to a new remote branch, along with a workflow with the following content:</p><ul><li>Workflow code is aimed to approve the PR using the GitHub API.</li><li>Workflow is configured to run on <em>pull_request</em> events. That includes creations and updates of PRs.</li><li>Workflow is granted with Write permissions on the pull requests API endpoint.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/129625274214d8e90bd3ed4b04c5b259/href">https://medium.com/media/129625274214d8e90bd3ed4b04c5b259/href</a></iframe><p>Then, the attacker creates a pull request, with the intent to merge their malicious code to a protected branch. As the PR is created, it cannot be merged since approval is required. However, the workflow immediately runs and the PR is approved by the <em>github-actions</em> bot, which the GITHUB_TOKEN belongs to.</p><p>It’s not an organization member, but counts as PR approval, and effectively allows the attacker to approve their own PR, basically bypassing the branch protection rules with the result of pushing code to a protected branch without any other organization member’s approval.</p><p>If the attacker wants to make the process even faster, they could also merge the PR through the workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zctFXYRdnN7kI5RH" /><figcaption>The PR is successfully approved with “github-actions” as the reviewer, and the code can be merged</figcaption></figure><h3>PoC video</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMWdsaxha6GQ&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMWdsaxha6GQ&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0cec58b0881740f965123a02ae68f53/href">https://medium.com/media/c0cec58b0881740f965123a02ae68f53/href</a></iframe><p>Any organization using GitHub as its codebase repository, trusting the security mechanism of required reviews to protect against direct push of code to sensitive branches, actually lacks this protection by default, even if GitHub Actions was never installed or used in the organization.</p><h3>Report to GitHub</h3><p>This security issue was reported to GitHub through their bug bounty program. They accepted it, wrote that it’ll be tracked internally until resolved, and approved to publish a write-up.</p><h4>Timeline</h4><p>15/09: Reported to GitHub bug bounty program<br>15/09 : First response from GitHub<br>22/09: Triage<br>22/09: Payout<br>23/09: Approval for write-up</p><h3>Mitigation</h3><ul><li>If you’re not using GitHub Actions, disable it for the entire organization or for specific repositories where it’s not required.</li><li>If GitHub Actions is in use in the organization, you can do one of the following:<br>- Require a review approval in pull requests from Code Owners.<br>- Increase the required number of approvals to 2 or more.<br>- Wait until the issue is resolved by GitHub.</li></ul><h3>Update: January 2022</h3><p>Following this blog post, GitHub recently introduced a new setting to fix this vulnerability. Organization admins can now disallow GitHub Actions from approving pull requests.</p><p>This is an organization-wide setting, which by default allows Actions to approve pull requests in existing organizations, and disallows it in newly created orgs. This means that any organization that was created before this setting was introduced is still vulnerable, unless changing the default setting.</p><p>To disallow Actions from approving pull requests, browse to Actions under Organization Settings. Look for this setting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SQElhcWVS2f2KAkv5HrH8g.png" /></figure><p>Clearing this setting will prevent Actions from approving PRs. The text is a bit misleading, as it’s explained like Actions can approve a pull request — and it just won’t count as an approval for merge, while practically it prevents approvals entirely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uNU2vUQ6vURw0p3h" /></figure><p>We recommend you to use this new setting to disallow malicious actors from bypassing branch protection rules by approving their own pull requests.</p><p>Kudos to GitHub for fixing this security flaw.</p><p>[1] Obviously no one guarantees the approver actually reads the code, but at least now there’s who to blame, right?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e1b29135cc7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/bypassing-required-reviews-using-github-actions-6e1b29135cc7">Bypassing required reviews using GitHub Actions</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>