<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rotem Bar on Medium]]></title>
        <description><![CDATA[Stories by Rotem Bar on Medium]]></description>
        <link>https://medium.com/@irotem?source=rss-6abeb6c02ef1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8RzMLSxAr3-CrQPPOPwVcA.png</url>
            <title>Stories by Rotem Bar on Medium</title>
            <link>https://medium.com/@irotem?source=rss-6abeb6c02ef1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:03:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@irotem/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Secret Diver — Searching for deeply hidden secrets]]></title>
            <link>https://medium.com/cider-sec/secret-diver-searching-for-deeply-hidden-secrets-6c7a05b54bea?source=rss-6abeb6c02ef1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c7a05b54bea</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[secrets]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Wed, 19 Jan 2022 12:55:50 GMT</pubDate>
            <atom:updated>2022-01-19T15:51:02.250Z</atom:updated>
            <content:encoded><![CDATA[<h3>Secret Diver — Searching for deeply hidden secrets</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gc19A-Fw2CERIcgx" /></figure><p>We are glad to introduce Secret Diver, a tool that allows you to search inside Docker layers for secrets.</p><h3><strong>TL;DR:</strong></h3><blockquote>Docker images are composed of layers. These containers, even after modifications and updates, may have secrets hiding in previous layers. One often overlooked but vital practice should be to check and verify that these layers don’t expose your secrets. <br>We have built a tool that searches the different layers in a fun and easy way.</blockquote><p>Docker images are actually OCI images that are comprised of multiple layers. When docker builds an image, a new layer is created for each command in the docker file. The end result (docker image) is a tar file that consists of all the layers in tar file format and an additional manifest.</p><p>You can read more about how <a href="https://jessicagreben.medium.com/digging-into-docker-layers-c22f948ed612">Docker layers </a>work in this excellent article.</p><p>A very useful utility to analyze the different layers of a docker image is “<a href="https://github.com/wagoodman/dive">Dive</a>” which allows to analyze container structures</p><h4>Short example:</h4><p>Shown below is an example analysis of a docker image consisting of 3 layers:</p><ul><li>The first layer imports the base image (in this case alpine)</li><li>The second layer creates a .pypirc file with our secret inside</li><li>This layer with the secret is needed only when building the image</li><li>In a real-life scenario, between the second and third layers, you would typically find the logic supporting the building of the image, such as fetching internal Python packages, running installation scripts, and so on.</li><li>The third layer deletes the .pypirc file</li></ul><pre><strong>dive</strong> cidersecurity/secret-deleted-in-container</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z9VHSGC2yhxc1GmX" /><figcaption>Running dive on a container</figcaption></figure><p>The final image built is uploaded to the container registry and becomes available for download. Once someone downloads the image and runs it in a docker environment, only the squashed file system is visible <strong>(without the secret in layer 2).</strong></p><pre><strong>docker run</strong> -it --entrypoint=/bin/sh \<br>               cidersecurity/secret-deleted-in-container</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uzdVkcqVx-NbAGun" /><figcaption>Opening shell inside the container</figcaption></figure><p>So a question we often ask ourselves, when we are trying to be security-minded and ensure there are no overlooked exploitable oversights, is whether someone can still obtain our secret, despite it being deleted.</p><blockquote>Regretfully, no surprises here. The answer is YES!!</blockquote><h4>We’ll now demonstrate how you too can replicate the method to obtaining deleted secret.</h4><p>If we take the sample scenario above, there are essentially two ways to do so:</p><ol><li>If you look closely at the “dive” above, we can view the commands that were run and, because we wrote the password as one of the commands, it is written to the manifest. The manifest contains all the commands and all the arguments that were used in each stage.</li><li>As mentioned above, the final image contains all the layers squashed together. An advanced user can extract any layer they wish from the image and read the file system contents of the layer extracted.</li></ol><h4>Let’s get our hands dirty:</h4><ol><li>We pull and save the container image by saving it as a tar file and then extracting the layers.</li></ol><pre><strong>docker pull</strong> cidersecurity/secret-deleted-in-container</pre><pre><strong>docker save</strong> cidersecurity/secret-deleted-in-container -o \<br>            secret-container.tar</pre><pre><strong>tar xvf</strong> secret-container.tar</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SPRQ37ErGaea2uoG" /><figcaption>Extracting the image locally</figcaption></figure><pre><strong>tar xvf </strong>7fb8df49962686...06d733/layer.tar<br>ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X70gMMSg6AzWld8E" /><figcaption>Extracting the layer</figcaption></figure><p>3. When we view the contents, we see the deleted .pypirc we were looking for. <strong>Great Success!!</strong></p><pre><strong>ls -alh tmp</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PfMqpN9EP9PbEuWX" /><figcaption>Password found in file</figcaption></figure><p>Now the fun part begins — how do we now hack this PoC for scale and repeatability?</p><p>This got us thinking about automation. What if we could do this automatically and combine a secret scanner to scan each layer independently for secrets?</p><h3>We are Proud to Introduce Secret Diver</h3><p>We developed the Secret Diver tool, specifically to prevent the scenario described above. We often forget that when we delete a secret from the repository, it still exists in the log history of the repository. But hackers don’t forget that — it’s the first thing they check for. That’s why we have you covered with Secret Diver.</p><h4>How does it work?</h4><p>Steps to use the tool:</p><p><strong>Manually</strong></p><ol><li>Install secret-diver from here — <a href="https://github.com/cider-rnd/secret-diver">https://github.com/cider-rnd/secret-diver</a></li></ol><pre>go install github.com/cider-rnd/secret-diver@v0.0.1</pre><p>2. Run secret-diver in<strong> human mode</strong> (machine mode is the default)</p><pre><strong>secret-diver </strong>-image=cidersecurity/secret-deleted-in-container -human</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4uClSYEGaz_E77De" /><figcaption>Running secret diver in human mode</figcaption></figure><p><strong>Inside your CI</strong></p><p>Run Secret Diver as part of the build process so that newly created images are scanned immediately after creation:</p><pre><strong>docker build</strong> -t &lt;image-tag&gt; .<br><strong>secret-diver</strong> -image=&lt;image-tag&gt;</pre><p>It is recommended to use the default machine mode which outputs to friendly <a href="https://docs.oasis-open.org/sarif/sarif/v2.0/sarif-v2.0.html">SARIF</a> output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ElyZg2_oa34__6Z2" /></figure><p>2. All done, you are good to go. Pass this sarif file to your favourite issue collection system or just view it <a href="https://sarifweb.azurewebsites.net/">here</a></p><blockquote>P.S. Going forward we recommend, from the security perspective, using multi-stage docker files which will upload only the final stage and automatically squash the previous stages into the base stage.</blockquote><p>Have fun with the tool and <strong>do no harm!!!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c7a05b54bea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/secret-diver-searching-for-deeply-hidden-secrets-6c7a05b54bea">Secret Diver — Searching for deeply hidden secrets</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NPM might be executing malicious code in your CI without your knowledge]]></title>
            <link>https://medium.com/cider-sec/npm-might-be-executing-malicious-code-in-your-ci-without-your-knowledge-e5e45bab2fed?source=rss-6abeb6c02ef1------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5e45bab2fed</guid>
            <category><![CDATA[npm]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Mon, 03 Jan 2022 16:03:29 GMT</pubDate>
            <atom:updated>2022-01-03T16:11:34.553Z</atom:updated>
            <content:encoded><![CDATA[<p>How to tell if you are using NPM safely within your CI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*64JTMmJNwI8NxwRKKtbNsg.jpeg" /></figure><p>The JavaScript ecosystem is highly reliant on dependencies. <em>And all I wanted was a method to safely download my desired dependencies from the internet.</em></p><p>The industry standard for doing so in Javascript is “NPM” or “Node Package Manager”. As a developer, when installing node.js software, I usually run “npm install” to download all the necessary packages for the software to run correctly.</p><p><strong>Running “npm install” does two things automatically:</strong></p><ol><li>It downloads required dependencies from the internet.</li><li>It runs all scripts necessary to properly build the software.</li></ol><p><strong>Both of these actions come with security risks.</strong> You can read about the risks associated with the first one in my article on <a href="https://medium.com/cider-sec/this-time-we-were-lucky-85c0dcac94a0">dependency attacks</a>, <strong>but</strong> <strong>this article is all about potential problems with the second — running external scripts.</strong></p><p>In its first phase, NPM goes to the package repository and fetches the needed packages. To authenticate against this package repository, it usually uses secrets/credentials which are accessible to the NPM client when accessing the package repository.</p><p>When running the second phase, NPM executes scripts, <strong>but even though access to the package repository is no longer required</strong>, <strong>NPM still has access to the same credentials</strong> while doing so.</p><h3><strong>Why is this a problem?</strong></h3><p>When NPM installs and runs scripts, it does so for all of the package dependencies, their dependencies, and so on. <strong>Because the developer doesn’t control all the sub-dependencies and their scripts, unverified and potentially malicious scripts will gain automatic access</strong> to the credentials and secrets used in the first action stated above.</p><p><strong>To mitigate this problem, NPM allows you to use the command “ignore scripts”</strong>, to disable all scripts associated with dependencies from running. <strong>But now we have a different problem: there are packages that <em>need</em> the ability to run scripts</strong> for purposes such as compiling, fetching resources, etc. These desired scripts can be executed separately, after the install phase. This would prevent the scripts from gaining access to secrets that are necessary only at the installation phase.</p><p>For example:</p><pre><strong># Install phase with access to secrets</strong></pre><pre>(<br>    NODE_AUTH_TOKEN=your-secret-token-here<br>    npm ci --ignore-scripts<br>)</pre><pre><strong># Build phase with no access to secrets</strong></pre><pre>(<br>   npm rebuild<br>   npm run install --if-present<br>   npm run prepare --if-present<br>)</pre><p>For more details about how to do this see this great article on a <a href="https://wildwolf.name/secure-way-run-npm-ci/">secure way to run NPM CI</a> by Wild Wild Wolf.</p><h3><strong>However, this solution on its own is not enough</strong></h3><p><strong>In my previous article about </strong><a href="https://medium.com/cider-sec/malicious-code-analysis-abusing-sast-mis-configurations-to-hack-ci-systems-13d5c1b37ffe"><strong>malicious code analysis</strong></a><strong>, I described how security scanning tools can be affected by the environment they are running in.</strong> For example, when a scanner is executed inside a certain directory, it will begin by picking up any configuration files it needs from its current directory.</p><p>These configuration files are supposed to tell the tools how to behave (e.g., to skip certain directories, skip checks, ..). Some, however, can also be used to instruct the tool to execute commands, thereby using the scanning tool itself to perform the unwanted actions it is meant to prevent.</p><p><strong>The same method can be applied to NPM.</strong> By adding a <strong>.npmrc</strong> file to the intended repo, the developer can manipulate the configuration which is picked up by the “<strong>npm</strong>” command.</p><p>For example, in NPM version 6, adding the command<br>“onload-script=${PWD}/evil-script” to the .npmrc file will execute the script “evil-script” (also added to the repository by the malicious developer) even if NPM has been given the “ignore scripts” command. In version 6, this will happen when executing any npm command, including the security function “npm audit,” “npm ci — ignore-scripts” and any other command such as “npm whoami”.</p><p>A similar action can be taken in the later npm versions 7 and 8 by replacing the git command in .npmrc, adding the following line to the configuration file: “git=${PWD}/evil-script.sh”. In this case, this would only affect the “npm ci” and “npm i” functions, however, the scope of potential harm is exactly the same.</p><p><strong>This means that any developer is able to execute code on almost any pipeline </strong>in your organization if it is using “npm”. Running a simple script to extract environment variables can seriously damage your environment. Therefore, obtaining developer permissions (whether by an attacker or a rogue developer looking to hack pipelines) can be used to cause real harm.</p><h3><strong>Disclosing the flaw to NPM:</strong></h3><p>I contacted NPM (GitHub) through their official bug bounty program and got the following response:</p><blockquote>Essentially ignore-script is designed to prevent the execution of malicious pre/post-install scripts of packages defined inside dependencies. CI Triggered on a pull request is intended to execute any code pushed as part of PR, however, this class of issue is usually<a href="https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions"> prevented by running CI jobs in an isolated environment</a> which GitHub Actions and Travis CI employs.</blockquote><p>I agree in principle that isolating CI Jobs and giving them minimal permissions is a wise preventative measure, but it can be very hard to implement in practice.</p><p>In my experience, <strong>I have seen many different CI/CD solutions — and have encountered many pipelines that still allow too much access to internal resources.</strong> While steps are being taken in the right direction, I still believe that the safest option would be a “defense in depth” approach. This means that <strong>we need to both isolate the environments of our CI/CD and adjust the tools we run to prevent unintended code execution.</strong></p><p>To continue pursuing the subject, I have opened an issue to NPM which is still under triage:</p><p><a href="https://github.com/npm/cli/issues/4101">https://github.com/npm/cli/issues/4101</a></p><h3><strong>What should I do?</strong></h3><ul><li>Run CI/CD functions in isolated environments which contain the minimal needed permissions to both resources on the host as well as to external resources.</li><li>Execute NPM securely by separating the script execution from the install phase (See above).</li></ul><h4><strong>Possible Workarounds for ensuring the trustworthiness of npm configurations</strong></h4><ul><li>Remove project configuration file (from a cloned repository) before using npm.</li><li>Copy project configuration file from external trusted resources (not controlled by developers).</li><li>Implement measures to detect potentially malicious changes in configurations (such as in .npmrc files) before triggering the build process.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5e45bab2fed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/npm-might-be-executing-malicious-code-in-your-ci-without-your-knowledge-e5e45bab2fed">NPM might be executing malicious code in your CI without your knowledge</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our dependencies are under attack, and this time we were lucky…]]></title>
            <link>https://medium.com/cider-sec/this-time-we-were-lucky-85c0dcac94a0?source=rss-6abeb6c02ef1------2</link>
            <guid isPermaLink="false">https://medium.com/p/85c0dcac94a0</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[npm]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[dependency-injection]]></category>
            <category><![CDATA[supply-chain]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Wed, 24 Nov 2021 04:42:50 GMT</pubDate>
            <atom:updated>2021-11-24T05:02:42.842Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-I-GT76TpuSDIJHL3DH7Ow.jpeg" /></figure><h4>How can we protect ourselves when our world infrastructure is under attack?</h4><p><strong>In the past few weeks, our world infrastructure has been under attack. </strong>The attack is very simple: Find a dependency package most of the world is using, hack into the developer’s account and update to a new version with some malware inside. Why was this possible? How can we protect ourselves? What are the potential problems and how can we address them?</p><h3><strong>Why is this possible?</strong></h3><p>When installing a package in Node.js, NPM (node package manager) automatically downloads all the package dependencies. Those dependencies then download additional dependencies, and so on, and so on, until all the needed dependencies have been acquired/downloaded.</p><p>Because keeping all of these dependencies up to date would be difficult and time consuming, NPM gives developers the option of automatically updating to new major/minor versions using special characters prefixed to the version. However, this option opens up the possibility that new dependencies will be upgraded automatically in the next build, which makes it possible to attack dependency owners by adding malicious content in the form of a new updated version.</p><p>Each version of a package<strong> </strong>has 3 sections: Major, Minor and Patch. This means, for example, that version 4.0.12 means:</p><ul><li>Major — 4</li><li>Minor — 0</li><li>Patch — 12</li></ul><p>When defining a version we can give different kinds of instructions, which will determine how the dependency will be upgraded. For example, “5.3.1” means “Use the exact version 5.3.1,” while simply writing “*” means “always use the latest version.” Other, intermediate options include:</p><ul><li><strong>4.x</strong> =&gt;Use all upgrades included in major version 4</li><li><strong>&gt;=4</strong> =&gt; Major version 4 and above</li><li><strong>~4.1.2</strong> =&gt; Major version 4, Minor version 1, Upgrade only to patch versions 2 and above</li><li><strong>^4.1.2</strong> =&gt; Major version 4, Upgrade to any Minor version above 1 and patch version 2 and above</li></ul><p>To get a better feel of how versioning works, you can try out the semgrep version calculator here: <a href="https://semver.npmjs.com/">https://semver.npmjs.com</a></p><p>The default option when installing a new dependency is the last example shown on this list <strong>(^4.1.2)</strong> which means that installing automatically upgrades the package to the next minor and patch versions.</p><p>Because of this, when hackers gained access to an account that has permissions to update the COA and RC packages and added malware to the package, any new installations downloading the latest “COA” or “RC” version automatically got this compromised version. Luckily for the world, this particular attack was not as stealthy or as comprehensive as it could have been, causing installations in pipelines to begin failing worldwide. This triggered an investigation by npm, and the malwared versions were quickly removed.</p><blockquote><strong>This time we were lucky</strong>, but next time a more targeted attack, running in stealth under the radar could be performed by taking advantage of this same weakness. This means that one mistake in a commonly used dependency or a calculated malicious act could leave most companies in the world compromised.</blockquote><h3><strong>How can we protect ourselves?</strong></h3><p>One important first step is using “npm ci” instead of “npm install.” Both are ways to install, but npm ci is much more strict in its dependency fetching policy.<strong> </strong>When we use “<strong>npm install</strong>,” we are telling the system to download<strong> </strong>whichever new packages the developer requested in the<strong> package.json</strong> file, including any automatic updates defined by the default semantic versioning.</p><p><strong>Npm ci</strong>, in contrast, <strong>requires</strong> the developer to upload a <strong>package-lock.json, </strong>which<strong> </strong>contains all exact versions of dependencies and sub-dependencies, thus telling the package manager exactly which dependencies to download. The developer then needs to periodically update the package-lock to reflect valid updates in the dependencies required. When using <strong>“npm ci”</strong> we are telling the system to <strong>download only </strong>the specific packages stated in the <strong>package-lock.json.</strong></p><p>A quick search on github finds <strong>11,294,073</strong> code instances of <strong>“npm install”</strong> versus <strong>279,920</strong> instances of <strong>“npm ci”</strong> which means the process in automatic scripts that downloads new repositories without thinking is 40 times more common than the stricter, safer one.</p><p>In order to create <strong>trusted/reproducible</strong> builds we have to use <strong>“npm ci”</strong> in all stages of the build systems (qa, testing, security, production) to be sure we are aligned with the same versions and nothing changed during the progression through the CICD pipeline.</p><h3><strong>Unfortunately npm ci comes with problems of its own…</strong></h3><p><strong>(a)</strong> <strong>Updating packages becomes a more complex process: </strong>Using<strong> “npm ci”</strong> introduces a new problem, since for each change in the dependency tree the developer would need to update dependency version changes, validate them and upload the newly created <strong>package-lock.json.</strong></p><p>Adopting this protocol can be hard for large organizations, but they can educate their developers to continually update the package-lock file or use solutions such as “<strong>dependabot</strong>” or “<strong>renovate”</strong> which can assist with creating <strong>PR </strong>to help.</p><p><strong>(b)</strong> <strong>Developers’ machines can still be vulnerable: </strong>Updating dependencies on the developers’ machines moves the vulnerability from the CI environments to the developers’ machines. <em>Anyone</em> who needs to run<strong> “npm install”</strong>, will be vulnerable and can be targeted for dependency attacks on their local machine.</p><p>To minimize this risk, organizations can add some preventive measures when developers run <strong>“npm install”</strong> by proxying their package registries and working only with versions which are above a certain age, for instance only allowing packages older than 1 week.</p><p>Furthermore, if they do have the need to run and update in the CI environments, organizations can make sure their systems are protected by preventing access to secrets, preventing access to sensitive systems, hardening pods and killing pods at first sight of suspicious behaviour.</p><p><strong>(c)</strong> <strong>The</strong> <strong>latest version of npm ci does not verify the lock file correctly:</strong></p><blockquote>We recently discovered that NPM 7 does not properly validate the package-lock as expected:</blockquote><blockquote>The npm ci command in npm 7.x and 8.x through 8.1.3 proceeds with an installation even if dependency information in package-lock.json differs from package.json. This behavior is inconsistent with the documentation, and makes it easier for attackers to install malware that was supposed to have been blocked by an exact version match requirement in package-lock.json.<br><a href="https://nvd.nist.gov/vuln/detail/CVE-2021-43616"><strong>https://nvd.nist.gov/vuln/detail/CVE-2021-43616</strong></a></blockquote><p>When writing this blog and researching best practices to protect yourselves we found out that in recent versions of npm, the “npm ci” command does not work as written in the documentation and does not verify the package-lock properly</p><h3><strong>What should security/devops teams do now?</strong></h3><h4><strong>Check if you have been attacked</strong></h4><ol><li><strong>Check if you have “COA” or “RC” versions in the package-lock versions<br></strong>Vulnerable Coa versions: <em>2.0.3, 2.0.4, 2.1.1, 2.1.3, 3.0.1, 3.1.3</em><br>Vulnerable RC versions: <em>1.2.9, 1.3.9, 2.3.9</em></li><li><strong>Verify Build logs<br></strong>Make sure the CI build logs did not execute any malicious code</li><li><strong>Confirm that vulnerable versions did not go to production<br></strong>If you don’t support reproducible builds (Using npm install in CI), you need to go to container/server, which uses a repo with the affected package and confirm that they don’t have these versions and/or the <a href="https://www.virustotal.com/gui/file/26451f7f6fe297adf6738295b1dcc70f7678434ef21d8b6aad5ec00beb8a72cf?nocache=1">malware signatures.</a></li><li><strong>Check the status of developer machines<br></strong>Check that your developers don’t have these <a href="https://www.virustotal.com/gui/file/26451f7f6fe297adf6738295b1dcc70f7678434ef21d8b6aad5ec00beb8a72cf?nocache=1">malware signatures</a><strong> </strong>on their machines.</li></ol><h4><strong>Employ preventative measures</strong></h4><ol><li>Use NPM v6 in your CI as a workaround.</li><li>Force usage of package-lock.json in your CI by running “npm ci — ignore-scripts” in your dockerfiles and build scripts.</li><li>Validate that all repos have a package-lock.json submitted to the repo.</li><li>We have created a quick program to check for instances of insecure npm usage and identify where the less secure “npm install” is being used.</li></ol><blockquote>1. Download CIMatch — “<strong>go get -v github.com/cider-rnd/cimatch”.<br></strong>2<strong>. </strong>Run “<strong>cimatch --human</strong>” in all your repos and ci configurations<br>3. Add “<strong>cimatch</strong>” to your security checks when scanning for new dockerfiles and jenkinfiles (More rules to come).</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85c0dcac94a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/this-time-we-were-lucky-85c0dcac94a0">Our dependencies are under attack, and this time we were lucky…</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Malicious code analysis: Abusing SAST (mis)configurations to hack CI systems]]></title>
            <link>https://medium.com/cider-sec/malicious-code-analysis-abusing-sast-mis-configurations-to-hack-ci-systems-13d5c1b37ffe?source=rss-6abeb6c02ef1------2</link>
            <guid isPermaLink="false">https://medium.com/p/13d5c1b37ffe</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[hacker]]></category>
            <category><![CDATA[malicious-code-analysis]]></category>
            <category><![CDATA[sast]]></category>
            <category><![CDATA[vulnerability-management]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Wed, 03 Nov 2021 15:45:51 GMT</pubDate>
            <atom:updated>2021-11-03T15:45:51.375Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWPVXBYB55UM6xQQktK7xA.png" /></figure><h4>What happens when SAST tools do more than just scanning? What if security scanners abuse their privileges?</h4><h3>TL;DR</h3><p><em>I recently found a new method that allows secure code analysis mechanisms to be bypassed and even worse — abused to execute malicious code on their host. This allows anyone with access to a source control repository to run malicious code in sensitive environments — which may be abused to steal credentials, access tokens and more.</em></p><p>This is a result of ongoing research around exploiting SAST scanners that was first presented at <a href="https://www.youtube.com/watch?v=Jl-CU6G4Ofc">DEF CON 29</a> this summer.</p><h3>Part 1 — Prequel</h3><blockquote>The number one goal of any hacker is and always will be one thing: running their own {malicious} code on the target computer, or in more professional wording, remote code execution (RCE).</blockquote><p>Hackers want to hack production systems. That is where the crown jewels are. Where the confidential data is stored. Where mission-critical tasks are running. There are various ways to gain access to production and numerous different motives for doing this, but they all have one uniting goal: running unauthorized code on production environments.</p><p>For a long time now, CI systems, and engineering systems in general, are sought after by hackers due to the ease of getting to production through them. They store sensitive credentials used for deployment, and they have a red carpet paved to production for carrying out the actual deployments.</p><p>The attack vector I want to introduce is a new method of attack that allows running malicious code on CI systems, allowing sensitive production credentials to be exfiltrated and malicious artifacts to be deployed.</p><blockquote><strong>“Malicious Code Analysis” attack<br></strong>A method which abuses code analysis software configurations to run code on the host running the code analysis</blockquote><h3><strong>Part 2 — How it was discovered</strong></h3><p>Before going deep into the technical areas, I want to share details on what brought me to research the behavior of code analysis software. Developers and builders want to develop as quickly and freely as possible. In a world full of chaos and security issues, we want to be able to develop code in a rapid and frictionless manner, while using various automations such as security guardrails to protect us from making mistakes along the way.</p><p>This is where static code analysis (or in short — SAST) solutions come in.</p><p>They get code as input, and no matter how malicious or harmful the code is, it will never be executed or cause any harm. The software statically analyses the code and identifies potential flaws. And for this reason, both defenders and developers can safely run this static analysis software against any piece of code without being concerned about the code being executed.</p><p>But as a veteran hacker, I know these type of statements are rarely true. There is always an exception.</p><p>So, one day I built a simple code scanning solution. Running this solution on random code gave me my desired results — 99% of the time — but every once in a while, the service crashed. Interesting. This means the service was somehow impacted by the actual input (code) it analyzed. So I decided to go down the rabbit hole and debug my scanner and voila — I found out that a certain character combination brought the whole service to its knees.</p><blockquote>This character combination was a self-evaluating form in Clojure, meaning that when the <strong>scanner read the code it actually tried to execute it.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*1pqd2pCX5nlDqOx-ckC52g.png" /><figcaption>Self evaluating code in Clojure</figcaption></figure><p>One hour later, I had a shiny new code execution vector that runs whenever my code gets scanned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFfp0zjyINnZXLhRxrnykQ.png" /><figcaption><a href="https://github.com/jonase/kibit/issues/235">https://github.com/jonase/kibit/issues/235</a></figcaption></figure><p><em>So I had a working POC of a scanner that went far beyond the scope of statically analyzing the code it received as input. This started a chain of investigations for what else I can break.</em></p><h3>Part 3 — Features evolving into security bugs</h3><blockquote>Remember Shellshock?</blockquote><blockquote>Shellshock was born in 1989, but was only found in 2014. The big mystery here is: how does a 25-year-old bug hide so well?</blockquote><p>The answer is quite simple: when the bug was created it wasn’t critical. In short, the feature of assigning a function and executing it from the environment variables was not critical because in order to create an environment variable you had to have local physical access.<strong><br></strong><a href="https://unix.stackexchange.com/questions/157381/when-was-the-shellshock-cve-2014-6271-7169-bug-introduced-and-what-is-the-pat/157495#157495"><strong>Read more here</strong></a><strong>.</strong></p><p>Then came the internet. Developers needed a way to interact with processes running on remote computers. The best way to pass data for starting processes in bash was to send them through environment variables. As it is only data it was pretty safe.</p><p>Except in bash as we saw earlier we can create functions, and these functions can self execute, just like our self executing functions in Clojure in the previous part.</p><h3>Part 4- Connecting the dots</h3><p>Code scanning has similar concepts; from the dawn of development, we had different methods and techniques to scan code. The majority of them involved running the scanner locally on the machine of the developer writing the code. So there hasn’t really been any need to secure the tools themselves.</p><blockquote>But wait, we have since connected these tools to the internet, and running them daily in our automations.</blockquote><p>The same tools now run as part of our deployment systems, systems to which we grant lots of power and permissions to deploy our most sensitive code. This is a big responsibility to put on tools that were built a decade ago with no intention of running on the internet…</p><blockquote><em>Game on…</em></blockquote><h3>Part 5- The Malicious Code Analysis attack</h3><p>Code pushed into production has to be verified. We have to make sure it doesn’t contain any security flaws or misconfigurations. The more code we have in our organization, the more we need to rely on automatic scanning and safety procedures to make sure everything is running as it should.</p><p>The benefits of scanning are great; we can put in guardrails pretty easily — starting from the linting of our code to standardize the way we write code to adding security measures that help react in real-time to new bugs introduced into the code before they actually get deployed to our production systems.</p><p>However, like all good things, there are often downsides. Before detailing the risks, let’s align on what code scanning actually is.</p><p><strong>What is a code scanner? (i.e. static analysis program)</strong></p><blockquote>Static program analysis is the analysis of computer software that is performed <strong>without actually executing programs. </strong><a href="https://en.wikipedia.org/wiki/Static_program_analysis"><em>https://en.wikipedia.org/wiki/Static_program_analysis</em></a></blockquote><p>I’ll skip the detailed explanation of how scanners work and save this for a future blog. If you want a detailed explanation, you can watch my talk from <a href="https://www.youtube.com/watch?v=Jl-CU6G4Ofc"><strong>Defcon29</strong></a> here.</p><p>As head of Marketplace integrations, a big part of my job is getting intimately familiar with SAST scanners — including what they do and how they behave. So I found myself thinking: if I can get one scanner to execute code, there is a high probability that I can get other scanners to misbehave.</p><p><strong>My research question became very specific:</strong></p><blockquote>Assuming most scanners do intend to perform static analysis, how can I exploit their own behavior to bypass them and even exploit them?</blockquote><p><strong>Abusing SAST configuration</strong> — Most scanners generate a lot of noise (i.e. findings), with one of the main reasons for this being that they have many preconfigured rules that most certainly don’t apply to all code repositories universally. So by design, each one in its own way has a configuration used to disable different subsets of these rules according to the users’ requirements.</p><h3>Bypassing the scanners</h3><p>The first mission I defined was to bypass the scanner. Security puts a lot of effort into automating processes for scanning R&amp;D code to make sure it doesn’t go into production. I wanted to create code that would bypass all those protections, but didn’t want to work hard or rely on getting caught by some custom rulesets of the security team. I wanted to bypass it all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vUDJFRQ2M5IeIB9rX05krg.png" /></figure><p><strong>Here is a walkthrough of the process and research of one of my targets — Checkov:</strong></p><p><a href="https://github.com/bridgecrewio/checkov"><strong>Checkov</strong></a><strong> </strong>is an open-source tool designed to audit “infrastructure as code” files. The tool is super effective in finding flaws and easily shows problems found in our code.</p><p>In our example, I wanted to create a privileged pod using terraform. Usually Checkov will scan and fail my deployment (if executed inline inside the CI) or at least alert me.</p><p>Reading the documentation shows us that when the scanners are executed, they look for a configuration file inside its target directory.</p><p>Adding a <strong><em>.checkov.yml</em></strong> allows us to configure how we want the scanner to behave, including what rules to exclude from the scan. In this case, we want to add a configuration that will never fail any code, and use only my own custom check</p><pre><strong>soft-fail:</strong> true<br><strong>check:</strong><br>  - THIS_NOT_THE_CHECK_YOUR_ARE_LOOKING_FOR</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xWNWTQzG3ky0b9Od4uCDpA.png" /></figure><p>Once I add this configuration file, I “pass” the Checkov inspection even though my code has security flaws;</p><p><strong>Before adding a config file:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*L5_o8a_i5WGXthtzNEoRMg.jpeg" /></figure><p><strong>After adding our “custom” config file</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*1vggb78HXqem170MFTadhQ.png" /></figure><p>Essentially, this means that anyone who wants to bypass the SAST security inspection, for any reason (for example — to release to production faster without any ‘obstacles’ put in place by the security team, can follow these three simple steps:</p><ol><li>Create a config file disabling all rules</li><li>Put it into the code directory that is to be scanned</li><li>Never again deal with failing pipelines</li></ol><blockquote>This is not proper advice. Do not try this in you CI/CD pipelines.</blockquote><blockquote><strong>Important notice:<br></strong>We are not blaming or putting responsibility on OSS creators. We should not expect OSS to provide the same level of security as their commercial alternatives. These tools were not build to protect the systems they running in. That is the user’s responsibility.</blockquote><blockquote>We should assume OSS projects could potentially contain security flaws and make sure they are properly configured and running in a safe environment.</blockquote><blockquote>With this being said: we did reach out to Palo-Alto as they support <strong>Checkov</strong> commercially. The feedback was very fast and professional and they added a disclaimer to their README regarding this behavior. In addition, they added support for setting the config file, which was previously not possible.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xKwwZt5j8fP8fegqHfP6aA.png" /><figcaption><a href="https://github.com/bridgecrewio/checkov/blob/master/README.md">https://github.com/bridgecrewio/checkov/blob/master/README.md</a></figcaption></figure><p><strong>Why did this happen?</strong></p><p>Most SAST Engines rely on configuration, usually defining which rules we want to use, setting operation flags, or just even to enable debug mode.</p><p>Most scanners use the same convention of searching for a configuration in the working directory or target (scanning) directory. Because the most common use case for running scanners is from the working directory, it is common usage to add a scanning configuration there to define our custom policy.</p><p>Because of this behavior, and given access to a repo that is to be scanned, we can “abuse” the scanner logic and tell it to skip all rules, move to debug mode, or worse scenarios <em>(See next chapter)</em></p><p><strong>Is it common?<br></strong>Below, we can see a list of scanners that accept configuration files in the current working directory. For each one here, adding the “evil” configuration will disable all security testing.</p><pre><strong>phpstan.neon<br>--<br></strong>paths:<br>  - /var/empty</pre><pre><strong>kics.config<br>--<br></strong>queries-path: /var/empty</pre><pre><strong>.rubocop.yml<br>--<br></strong>&lt;%= exit! %&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UWUYAU5whSdmHrNjyFTQdA.png" /></figure><p>You can find our opensource repo we released with more researched configuration files here: <a href="https://github.com/cider-rnd/cicd-lamb"><strong>https://github.com/cider-rnd/cicd-lamb</strong></a></p><p>Detailed explanations can be found in my <a href="https://youtu.be/Jl-CU6G4Ofc?t=954">YouTube</a> video from Defcon.</p><h3><strong>Attacking the Scanners Right Back</strong></h3><p>So anyone who has gotten this far is probably asking themselves the same question: if the scanner really picked up my rule configuration, can I do something more interesting than just disable scanner features?</p><p>Because lots of scanners are basically frameworks for scanning code, they allow us developers, security engineers or DevOps to add our own custom rules.</p><p>These rules are sometimes created as code or have other ways of dynamic configuration loading. This leaves us with the possibility to <strong>execute code</strong> if and when we are able to override the configuration file.</p><p>Going back to the documentation, we can learn that Checkov allows us to load external checks, which is a great feature on its own. But when allowing configuration files to be loaded from the target directory, things can get pretty interesting.</p><p>So when we create a folder with python scripts in our target directory (in our example, we named the folder “scripts”), coupled with creating a configuration file that tells the scanner to pick up our custom python code from the requested folder, will result in executing our custom code when running the scan:</p><pre><strong>external-checks-dir:</strong><br>  - scripts</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNKKPuIs4cKltqVMBmEiKA.png" /></figure><p>So essentially this means that if I am a developer, <strong>or any human or application with permission on a source code repo</strong>, and I want to run malicious code on the CI, I can achieve my goal by doing the following:</p><ol><li>Create my own ‘malicious’ branch of the repo (knowing that when I issue a PR or commit code to this branch it will trigger a scan in the CI).</li><li>Add the following payloads to the branch —</li></ol><ul><li>A custom malicious configuration file</li><li>malicious files/folders triggered by the configuration file</li></ul><p>3. Push / issue a PR (or whatever action will end up triggering the build and running the scan)</p><p>4. Drink your favorite drink and wait for the shell to pop</p><h3>Conclusions</h3><p>Continuing our research, we created a list of scanners researched where we could find our <strong>Malicious Code Analysis</strong> attack. I am sure this is just the beginning; there are many other tools that weren’t checked. But now every time I install a new tool I can’t stop myself from thinking: is it vulnerable?</p><p>For more detailed examples checkout our CICD Lamb repo<br><a href="https://github.com/cider-rnd/cicd-lamb">https://github.com/cider-rnd/cicd-lamb</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0lQGGLphn6vFCGAuVmUmg.png" /></figure><p>Most security scanning is supposed to find security issues and flaws in code uploaded to our systems. The thought that the code itself will try to target the scanner itself usually is not thought of when designing scanners.</p><p>Today in 2021, we are enabling so much automation — with developers, DevOps, SecOps, AppSec engineers each trying to automate away as much of our work as possible in order to meet the continuously growing volume and demand by different organizational stakeholders. It is crucial to understand that this automation can be vulnerable to attacks as well and is being attacked</p><p>These automations affect every area of our software development lifecycle (SDLC) — a crucial part of our CI/CD environments — areas that have access to our most sensitive integrations and systems. These areas are the jackpot of all attacks. In this place, silently a shell pops without any understandable reason.</p><p><strong>We need to prepare for that day</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LKj5kmKRHSFUOZINwEFbVA.png" /></figure><h3>AfterMath (Fixing our problems)</h3><p>The first stage of fixing is understanding there is a problem. If you got this far, I hope you are starting to understand, that this is only the tip of the iceberg.</p><p>A few tips which will help you in the following quest to prevent these problems:</p><ul><li>Always scan your code in an ISOLATED environment. Block all internet access; isolate resources such as disk, CPU, network access and finally destroy after use.</li><li>Create a fixed configuration file, set the scanner to use it and not the default</li><li>Sanitize your CI/CD environment, and align it with the principle of least privilege. Your scanners need to run in an environment with the minimal possible privilege</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zpLOs_umzn11N5l3M7QCiw.png" /></figure><h3><strong>Acknowledgments</strong></h3><p>I want to thank all that supported and helped me with this research.</p><p><strong>Cider Security</strong> — Which gave me the time and resources to research this finding<br><strong>Daniel Krivelevich</strong> — Who pushed me into doing more thorough research<br><strong>Sharone Revah Zitzman</strong> — Thank you for the enormous impact on the presentation and the blog post<br><strong>Palo Alto, Barak Schoster</strong> — For the quick response and great feedbacks</p><p><strong>Special thanks to all the open-source SAST developers and contributors out there, Each one of you is helping the world become a safer place</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13d5c1b37ffe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cider-sec/malicious-code-analysis-abusing-sast-mis-configurations-to-hack-ci-systems-13d5c1b37ffe">Malicious code analysis: Abusing SAST (mis)configurations to hack CI systems</a> was originally published in <a href="https://medium.com/cider-sec">Cider Security</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developers, Please encode your URLs]]></title>
            <link>https://medium.com/appsflyerengineering/developers-please-encode-your-urls-e45922fb2a80?source=rss-6abeb6c02ef1------2</link>
            <guid isPermaLink="false">https://medium.com/p/e45922fb2a80</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[services]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Mon, 26 Oct 2020 17:32:03 GMT</pubDate>
            <atom:updated>2020-10-26T17:32:03.758Z</atom:updated>
            <content:encoded><![CDATA[<h4>If you like it, put a # on it!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*PWm9FJYDWstJ5a3ADn3rVA.png" /></figure><p>Uniform Resource Locators (URLs) are a funny thing. They seem so simple, but yet they have so many small complex rules to them, that all of a sudden, when you try to explain what a url is and the correct way to parse one, you find yourself in a sea of complexity.</p><h3>The inner workings of URLs</h3><p>Let’s start with the basics. A URL identifies a resource and tells you which method you should use to access it. An example of this is:</p><blockquote><a href="https://appsflyer.com/why-appsflyer">https://www.appsflyer.com/why-appsflyer/</a></blockquote><p>Here the resource is the <a href="https://appsflyer.com/why-appsflyer">https://appsflyer.com/why-appsflyer</a> page<br>The method used is https which means secure hyper-text transfer protocol.</p><p>To take this to the specifications, there are two main RFCs that govern our URL usage. Even though <a href="https://tools.ietf.org/html/rfc3986">RFC3986</a> is the update to <a href="https://tools.ietf.org/html/rfc1738">RFC1738</a>, both schemes are still widely used today by applications.</p><p><a href="https://tools.ietf.org/html/rfc3986"><strong>RFC3986</strong></a></p><pre>&lt;scheme&gt;//&lt;user&gt;:@&lt;host&gt;:&lt;port&gt;/&lt;url-path&gt;</pre><p><a href="https://tools.ietf.org/html/rfc1738"><strong>RFC1738</strong></a></p><pre>&lt;scheme&gt;//&lt;user&gt;:&lt;password&gt;@&lt;host&gt;:&lt;port&gt;/&lt;url-path&gt;</pre><p>Note, that in the updated <a href="https://tools.ietf.org/html/rfc3986">RFC3986</a>, using a password in the URL is deprecated — this will be important later on!</p><p><strong>A primer on URL encoding</strong></p><p>Web browsers request pages from web servers by using URLs. <br>URLs can only be sent over the internet using the <a href="https://www.w3schools.com/charsets/ref_html_ascii.asp">ASCII character-set</a>, however, URLs often contain characters outside of this set, therefore the URL has to be encoded into a valid format. <br>URL encoding replaces unsafe characters with a percentage sign (%) followed by two hexadecimal digits.</p><p>As an example, URLs cannot contain spaces.</p><blockquote>Therefore a space in a URL is encoded to %20.</blockquote><p>So the URL <strong>“www.appsflyer.com/hello everyone”</strong></p><p>would be encoded into <strong>www.appsflyer.com/hello%20everyone</strong></p><p>When the web server retrieves this url, they are then able to decode the URL and convert it back into <strong>“</strong><a href="http://www.appsflyer.com/hello"><strong>www.appsflyer.com/hello</strong></a><strong> everyone”</strong></p><p>For reference of all the URL encodings, visit <a href="https://www.w3schools.com/tags/ref_urlencode.ASP">https://www.w3schools.com/tags/ref_urlencode.ASP</a></p><h3>URLs within micro-services</h3><p>There are many different methods you can use for inter-service communication, such as REST, gRPC, and messaging brokers such as Kafka or RabbitMQ. In this story, we’ll be focusing on microservice communication using HTTP REST.</p><p>Here we have a service that receives a URL with two parameters:</p><ul><li>‘site’:controls which other microservice the query will get sent to;</li><li>‘user’: identifies a user in the system;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*3hxPksS8V0XyYbLX" /><figcaption>service which takes two parameters and builds a new URI</figcaption></figure><p>In this example we added some code that validates the user and creates a new url which gets sent to an internal micro-service. Validating a user from a url parameter that is controlled by the client is not a secure design practise, but this does happen in the real world. We’ve taken this idea as an example to help demonstrate why encoding URLs is so important.</p><p>Now in this case we did not explicitly write any code to carry out url decoding or encoding. However, the server engine when processing the request automatically decodes the parameters into the proper variables.<br>Let’s see the same example but with special characters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*SWFGhxL3HEJz0VOs" /><figcaption>Parameters are encoded in browser and decoding in code</figcaption></figure><h3>Url Tampering</h3><p>In the above examples, the site parameter is not validated and there is no encoding applied. This allows us to tamper with the generated url and control the request.</p><p>In the below request, we added a %23(#) in order to convert the following characters in the url to be a fragment. Read more about this <a href="https://en.wikipedia.org/wiki/URI_fragment">here</a>.</p><p>In order to identify an issue with no access to the source code, or internal servers, we first attempt to fetch data from our external server. Once we see our external server receive a request, we can let the games begin!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*apnlVaAr7wYpCZKa" /><figcaption>Injected a # to break url building</figcaption></figure><p>The next step is to bypass authorization checks by manipulating the URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*-FZSBQ7cVBJrPBbW" /><figcaption>Overridden second variable by injecting data into first parameter</figcaption></figure><p>In this case we are adding our payload to control all elements after https:// and finishing with a # sign which signals the request sender that this is a url fragment.</p><p>The result is that a request is sent with a different user to the one that was validated — bypassing the user validation check!</p><blockquote>Note that we couldn’t just change the user parameter because it is validated by our user validation function. We could assume also that the user parameter can come from our sso provider so it can’t be tampered with.</blockquote><h4>Another common scenario:</h4><p>In this example we have two dynamic query parameters — “key” &amp; “user” — which are controlled by the client. Note that the “user” parameter is validated properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*caCes_34hJBbtIpf" /></figure><p>With the same concept of parameter manipulation we can add data to the parameter which will inject new variables into requested url.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*JkP4lgtg_f5-kr5A" /><figcaption>Overriding the user variable in requested resource</figcaption></figure><h3>Fixing this</h3><p>Javascript has a feature called <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent">encodeURIComponent</a>, which allows us to safely put in a URI component into the URI without being afraid of users injecting content to break out of our parameter.</p><p>Other frameworks and languages have other solutions. You can lookup your language here <a href="https://rosettacode.org/wiki/URL_encoding">https://rosettacode.org/wiki/URL_encoding</a></p><p>Common feedback I receive from developers is that they will need to check with the receiving service that they decode it first. The answer is simple, in 99% of the cases the receiving service uses a common framework which automatically decodes their data. Of course full testing should be performed after doing some changes but it usually does not affect the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*inxyHQoKVJ_vhFn2" /><figcaption>Encoding made the key to be as requested with no injection possibilities</figcaption></figure><h3>To summarize</h3><p>When passing data between microservices or other external services it is vital to verify that you are building the target url in a safe way.</p><p>Going down memory lane, in the early days of software development twenty years ago, we had a similar problem with SQL. Developers would build SQL queries without the knowledge and understanding that they can be injected with custom queries. Today, most developers are familiar with SQL Injection, but not with URL Injection.</p><p>Please share this knowledge with all of your developer friends. If this is something new to you or you already know of it, we would love to hear your feedback.</p><blockquote>As for security researchers, you can add a %23(#) or %26(&amp;) to your payloads, weird activity usually is a good sign.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e45922fb2a80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/appsflyerengineering/developers-please-encode-your-urls-e45922fb2a80">Developers, Please encode your URLs</a> was originally published in <a href="https://medium.com/appsflyerengineering">AppsFlyer Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NGINX may be protecting your applications from traversal attacks without you even knowing]]></title>
            <link>https://medium.com/appsflyerengineering/nginx-may-be-protecting-your-applications-from-traversal-attacks-without-you-even-knowing-b08f882fd43d?source=rss-6abeb6c02ef1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b08f882fd43d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[bug-bounty]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Rotem Bar]]></dc:creator>
            <pubDate>Thu, 24 Sep 2020 08:59:08 GMT</pubDate>
            <atom:updated>2022-07-10T12:43:12.890Z</atom:updated>
            <content:encoded><![CDATA[<p>By <a href="https://medium.com/u/97c900ec3ddb">Danny Robinson</a> and <a href="https://medium.com/u/6abeb6c02ef1">Rotem Bar</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLbJddL3Osi4XNeZ0RaJSQ.png" /></figure><p>As a security team within a rapidly growing company, we encounter lots of different types of vulnerabilities. We have numerous in-house development teams, all running full speed to build new features into their products, which inevitably also leads to the introduction of security bugs.</p><p>Our mission is to prevent, detect and remediate all security issues within our environment. Our goal is to enable rather than block our developers’ work. In order to achieve this, our strategy has been to share our knowledge through building new processes and systems that help our developers release their latest feature safely without introducing vulnerabilities into our production systems.</p><p>This is a story about a traversal vulnerability, how we handled it and why it can also impact your organization.</p><h4><strong>BACKGROUND:</strong></h4><p>We found a <a href="https://owasp.org/www-community/attacks/Path_Traversal">Traversal Vulnerability</a> in one of our application services through our in-house scanner Jirachi (more on this in another post). Although it was a simple traversal attack and seemed easy to exploit, for some reason when testing through our staging site our Proof Of Concept kept failing.</p><p>We were able to exploit the directory traversal vulnerability when communicating directly with the service, and thought that this should be an easy Jira ticket to write up. However, when we went to write the full end-to-end POC, the exploit kept failing. We looked at the only other component that had changed between the two tests, and we found that the service was behind an AWS Application Load Balancer (ALB). However, there were no rules, no Web Application Firewall (WAF), nothing — so we couldn’t figure out why the load balancer was preventing our POC.</p><p>Investigating the AWS ALB further, we found an HTTP 400 error response that had similarities with an error response in <a href="https://www.nginx.com/">NGINX</a> that we found in a git commit. We quickly downloaded the latest version of NGINX, and digging through the commit history, we found that there was a configuration option called <em>merge_slashes</em>. When we set <em>merge_slashes</em> to “off”, we saw the same behaviour as we did with the ALB. We came to the realisation that the AWS ALB url parsing shares similar functionality as NGINX. Furthermore it was this functionality that was responsible for sending the http 400 error response and not allowing the request to reach the vulnerable application.</p><p>We couldn’t give up there though. We had to find a way to get our request to the vulnerable application!</p><h3>Technical POC: Traversal Attack through NGINX URL parser</h3><p>To demonstrate how we can get the malicious request through the NGINX url parser, we put together a sample application.</p><p>As you can see below, We created a sample web server code which has a Local File Inclusion vulnerability</p><pre>const express = require(&#39;express&#39;);<br>const app = express();<br>const port = 80;<br>const fs = require(&#39;fs&#39;);<br><br><br>app.get(&#39;*&#39;, (req, res) =&gt; {<br>    const data = fs.readFileSync(&#39;secure/&#39; + req.url)<br>    res.send(data);<br>})<br><br>app.listen(port, () =&gt; {<br>    <strong><em>console</em></strong>.log(`Vulnerable App listening at http://localhost:${port}`)<br>})</pre><p>The server accepts a GET request. The user-supplied url is concatenated with the ‘secure’ string and that file is then sent back to the user.</p><p>As an example sending a request to <a href="http://server/1.txt">http://server/1.txt</a> will cause the server to send back a file from &lt;CurrentWorkingDirectory&gt;/secure/1.txt</p><p>In order to demonstrate this, we send a GET request directly to the server and in the response we get the content of file 1.txt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*If86iSXAc26tXVul" /><figcaption>Vulnerable application</figcaption></figure><p>In order to exploit the traversal vulnerability in this app, we can request a file from a directory one level up by using the ‘../’ notation. Here we request the index.js file directly from the server .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*olxnjeVsMfxkQSYq" /><figcaption>POC —</figcaption></figure><p>Carrying out the same traversal attack through AWS ALB, we received a 400 bad request</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ETMe__1wmlFc12ii" /></figure><p>However, after some research we found that we can get our malicious request through the url parser by using multiple slashes (‘/’).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c8CARI431PbPFhzq" /></figure><p>In order to demonstrate this point further, the below image shows an attempt at retrieving the /etc/passwd file through an AWS ALB. Once again, we get a 400 bad request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BqN3xMT9iRb45EVq" /></figure><p>As before, we’re able to reach the vulnerable application by appending more slashes (‘/’).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OM0SlrUB3_TRqFjB" /></figure><h3>Root Cause Analysis</h3><h4><strong>How did we know AWS ALB shares similar functionality in url parsing as NGINX?</strong></h4><p>When the requests were blocked we saw a certain pattern.</p><p>The 400 bad request response from the AWS ALB above had a distinctive html signature:</p><pre>&lt;body bgcolor=”white”&gt; …</pre><p>After simple Google search and digging in GitHub we found that this signature is used by NGINX.</p><p>Specifically, we’re able to find which commit and version this error signature was removed in.</p><p><a href="https://github.com/nginx/nginx/commit/8117b3f5a06b68afb292ddd19bf6ee6000707232#diff-05d6b8ae21d192f78b349319edaeb9bf">https://github.com/NGINX/NGINX/commit/8117b3f5a06b68afb292ddd19bf6ee6000707232#diff-05d6b8ae21d192f78b349319edaeb9bf</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*fkiIeM4JpJZUDWsm" /></figure><p><strong>Fun fact:</strong></p><blockquote>body bgcolor=white was pretty common in the past as IE rendered pages by default in light gray, other browsers let the user select a default background such as black and there was large inconsistency between browsers. Putting bgcolor=white ensured that all browsers display the background as white.</blockquote><h4><strong>How did we determine that merge_slashes is ‘off’:</strong></h4><p>Within the NGINX documentation, we are able to see that the merge_slashes default value is “On”. Therefore two or more ‘/’ characters will be normalized into one ‘/’ character.</p><p><a href="http://nginx.org/en/docs/http/ngx_http_core_module.html#merge_slashes">http://NGINX.org/en/docs/http/ngx_http_core_module.html#merge_slashes</a></p><p>Enables or disables compression of two or more adjacent slashes in a URI into a single slash.</p><pre>Syntax: merge_slashes on | off;<br>Default: merge_slashes on;<br>Context: http, server</pre><p>When the <em>merge_slashes</em> configuration is turned on, using multiple slashes ‘///’ did not allow us to exploit that vulnerability successfully.</p><h4><strong>Trying out our bypass on latest version of NGINX:</strong></h4><p>The following shows the POC carried out on the latest stable version of NGINX (v1.18.0) with <em>merge_slashes</em> ‘off’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rHxMDmaQhjbzHp8q" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BfVm_YFblUkhdkL0" /></figure><h4><strong>Identification of NGINX issue:</strong></h4><p>In order to better understand how NGINX handles the url, we built a program to test the parsing function within the ngx_http_parse file:</p><pre>ngx_http_parse_complex_uri(ngx_http_request_t *r, ngx_uint_t merge_slashes)</pre><p>Within this function, the handling of state == sw_dot_dot at <a href="https://github.com/nginx/nginx/blob/master/src/http/ngx_http_parse.c#L1581">https://github.com/NGINX/NGINX/blob/master/src/http/ngx_http_parse.c#L1581</a> is what our tests focus on.</p><p>This shows the code written for testing the parsing function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/0*RtHov2j9pScRGxAP" /></figure><p>Here we can see when we run the test with a regular traversal attack, we get an error and return code 11.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/0*2hSgAtrNyFBzp5TQ" /></figure><p>Here we can see that when we run the test with extra slashes, we return 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/0*XAzrA_untf8ZWrPK" /></figure><h3>What is the Impact?</h3><h4><strong>The impact here is complicated</strong></h4><p>Looking at CVE’s we saw about 4000 known directory traversal CVE’s dating from 1999 to 2020. Directory Traversal attacks are here to stay!</p><p>Furthermore, companies testing themselves behind any ALB or NGINX solution configured with <em>merge_slashes</em> ‘off’ will probably not find this bug so easily. However, more knowledgable or dedicated attacks would know how to get their malicious request through the url parsing mechanism.</p><p>In order to get a better understanding of how wide-spread this issue is, we ran a search on github for <em>merge_slashes</em> ‘off’;. There are 549 Open Source Projects that use this configuration and unknowingly may be blocking malicious requests to applications that are vulnerable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y0d6QncunPnpGwXt" /></figure><h4><strong>Could this blocking behaviour be changed?</strong></h4><p>Not really as we see it.</p><p>If traversal is blocked within NGINX, then applications relying on “safe traversal as a feature” will start failing. This is critical for some businesses and therefore, we don’t see this behaviour being changed.</p><p><strong>Is this a vulnerability?</strong></p><p>NGINX as itself is not a protection solution and AWS ALB is not intended to protect your application. If you are looking for a protection solution, consider using a WAF.</p><p><strong>What do we recommend?</strong></p><p>If possible, run your tests on the application directly without any load balancers in the middle. If you must test through load balancers, you should ensure that your testing takes into account this behaviour so you don’t miss any possible vulnerabilities.</p><h3>Summary</h3><p>This exercise showed us why it is important to always look around for additional attack vectors, never rely on one protection mechanism and always continue ramping up our security methods.</p><p>We do not consider this a vulnerability within NGINX nor AWS Application Load Balancers. By taking this information public, we are hoping that we are able to make organizations more aware of the behaviour displayed in NGINX and ALB and allow them to better test their application to reduce their risks</p><p>Stay tuned for more information about our security efforts. In the following weeks, we will be releasing more security tools we built in house and would love to share with the community.</p><p><strong>Want to be part of our security adventures? We’re hiring! Contact us at </strong><a href="/service/mailto:security@appsflyer.com"><strong>security@appsflyer.com</strong></a></p><h4><strong>TL;DR</strong></h4><blockquote><em>Add forward slashes at the beginning of your requests when testing directory traversal attacks through load balancers.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b08f882fd43d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/appsflyerengineering/nginx-may-be-protecting-your-applications-from-traversal-attacks-without-you-even-knowing-b08f882fd43d">NGINX may be protecting your applications from traversal attacks without you even knowing</a> was originally published in <a href="https://medium.com/appsflyerengineering">AppsFlyer Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>