<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashwin Radhakrishnan on Medium]]></title>
        <description><![CDATA[Stories by Ashwin Radhakrishnan on Medium]]></description>
        <link>https://medium.com/@aradhakrishnan-MITRE?source=rss-c34c9e212bb4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Y3cP2ElcIqp6ZhvgZEeOEQ.png</url>
            <title>Stories by Ashwin Radhakrishnan on Medium</title>
            <link>https://medium.com/@aradhakrishnan-MITRE?source=rss-c34c9e212bb4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:43:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aradhakrishnan-MITRE/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Managed Services Evaluations — Round 2 (2023): Attribution and Speed and Efficiency, Oh My!]]></title>
            <link>https://medium.com/@aradhakrishnan-MITRE/managed-services-evaluations-round-2-2023-attribution-and-speed-and-efficiency-oh-my-59aa207641fa?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/59aa207641fa</guid>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Wed, 07 Jun 2023 21:28:08 GMT</pubDate>
            <atom:updated>2023-06-08T21:36:20.537Z</atom:updated>
            <content:encoded><![CDATA[<h3>Managed Services Evaluations — Round 2 (2023): Attribution and Speed and Efficiency, Oh My!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qXWP49iUI5ofWJcJdojTqg.png" /></figure><p>We are excited to announce some innovative updates to MITRE Engenuity’s upcoming <a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services-2/">ATT&amp;CK® Evaluations: Managed Services — Round 2 (2023)</a>. For the first full cycle ever, the Evaluations team followed a product management lifecycle process that has allowed us to maintain our foundation, while iterating on areas we learned needed improvement. Immediately after our <a href="https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-managed-services-oilrig-2022-and-the-top-10-ways-to-411b155c3af3">OilRig release on November 9, 2022</a>, we embarked on a structured “Feedback Gathering” stage, which was dedicated to listening to our end-user and vendor communities. This feedback was incorporated into our strategy and validated with deep dive sessions with whomever reached out to us. This release also serves as our roadmap update, so strap in, there is a lot to cover.</p><h3>Components of this Release Announcement</h3><p>First, let us cover the individual components of this release:</p><p>This article will be broken down into the following sections which will cover these key aspects of our upcoming Evaluation.</p><ol><li>Components of this Release Announcement — individual aspects of this release.</li><li>Solving the Problem — description of the problems relevant to Managed Services Evaluations.</li><li>Improvements in Round 2 — deep dive on the major improvements featured this round to address the problems and questions raised.</li><li>Next Steps — next steps and timelines for the remainder of this round.</li></ol><p>As always, please refer to our Overview Page to get a breakdown of the approach we are taking this year. The Overview Page includes:</p><ol><li><a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services-2/">Evaluation Summary</a> — the general purpose of this Evaluation.</li><li><a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services-2/">Methodology</a> — the methodology that will be used in this Evaluation.</li><li><a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services-2/#sn-environment">Environment</a> — the environment that will be used in this Evaluation.</li><li><a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services-2/#sn-target-organization-profile">Target Organization Profile </a>— an example of the target based on the publicly available CTI of the adversary or adversaries selected for this Evaluation.</li><li><a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services-2/#sn-reporting">Reporting </a>— this section describes the “Reporting Statuses” that will be assigned to the reports sent to the MITRE Engenuity Execution Team during the Evaluation.</li><li><a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services-2/#sn-additional-metrics">Additional Metrics</a> — this section describes the new metrics we are introducing in this Evaluation.</li><li><a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services-2/#sn-resources">Additional Resources</a> — this section contains all the published collateral for this round.</li></ol><p>If you are a managed service provider interested in hearing more, please fill out this <a href="https://mitre-engenuity.org/cybersecurity/attack-evaluations/get-evaluated/">Participation Form</a>, and we will reply shortly.</p><h3>Solving the Problem</h3><p>The hardest problem to solve in Evaluations is maintaining the principles of objectivity, consistency, and rigor within each participant’s engagement, while publishing results that are sufficiently nuanced, without leading to inadvertent editorialization. Sentence structure aside, if that statement was a lot to absorb, consider the complexity of cultivating an effective Evaluation methodology that checks every box in that very narrow niche. While Evaluations themselves have other intrinsic value, solving the problem as stated is incredibly important because it provides for our community’s collective ability to be proactive when fighting against adversaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zvr1Hm2WOoz5TOu7cbJDrA.png" /><figcaption>Goals for ATT&amp;CK Evaluations</figcaption></figure><p>As shared in our initial release article, the ATT&amp;CK Evaluations team intends on solving that problem in the pursuit of achieving these three goals:</p><ol><li><strong>Empower end-users with objective insights</strong> into leveraging specific commercial cybersecurity capabilities to address known adversary behaviors.</li><li><strong>Provide transparency around the true capabilities of commercial cybersecurity offerings</strong> to address known adversary behaviors.</li><li><strong>Drive the cybersecurity vendor community to enhance their offerings</strong> to better address known adversary behaviors.</li></ol><p>Each of our goals are structured to advance the cybersecurity community’s ability to address known adversary behaviors, which in turn allows all of us to be more strategic in preventing adversaries from committing data breaches, bringing down critical infrastructure, committing fraud, etc.</p><p>It will always be far easier to demolish than it will be to build. Even in Aristotelian logic, the validity of a universal affirmative proposition must be proven true categorically. To disprove a universal affirmative proposition, on the other hand, it is sufficient to demonstrate the truth of a single contradictory counterexample. This theory is held to be true in most regards, and it is doubly apparent when we begin to examine how breaches occur in the real-world. We are successful when we defend categorically, while adversaries are successful when they find just one way to breach our defenses. This is an inherent imbalance, and the unfortunate reality is that adversaries are more successful than we would like them to be. Nonetheless, we are tasked with preventing breaches, and a resilient strategy is the best way to defend against that imbalance because it allows us to proactively cover adversary tactics and techniques rather than reactively defend against the technical observables leveraged by adversaries. The MITRE ATT&amp;CK Framework serves as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community. Leveraging the ATT&amp;CK Framework to design and implement a threat-informed defense will allow you to chip away at that imbalance and fortify your defenses on a more strategic basis.</p><p>Though describing how to design and implement a <a href="https://mitre-engenuity.org/cybersecurity/center-for-threat-informed-defense/">threat-informed defense</a> is a deep conversation in and of itself, at its most abstracted level, any strategic threat-informed cybersecurity posture requires a tailored composition of people, process, and technology. The upcoming Managed Services Evaluation Round 2 is structured to evaluate commercial offerings regarding the people (and parts of the process) in that triangular structure. Any substantive Evaluation of managed services offerings, therefore, must be inherently different than a technology (aka product) Evaluation so that the elements of a holistic threat-informed defense not covered by our Enterprise Evaluations can be evaluated.</p><blockquote>The most consistent feedback we received from our community regarding the <a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig">OilRig</a> round was that it was far too focused on detection coverage, even though detection coverage is not what primarily contributes to the decision-making calculus of actual and prospective clients of managed services.</blockquote><p>If you are in this camp, I would urge you to re-read content from our <a href="https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-managed-services-oilrig-2022-and-the-top-10-ways-to-411b155c3af3">last release</a> because the published results dive far deeper than detection coverage. The insights our methodology enables us to discover in each Evaluation round are incredibly unique, and merely examining the colors of each sub-step within the Emulation Plan is an ineffective way to interpret the core of what is published. To mitigate this type of analysis, our strategy for this round is to emphasize far more elements than mere detection coverage by implementing significant changes to the Core Platform of ATT&amp;CK Evaluations. We prioritize consistent efforts to our Core Platform, and it is comprised of the Evaluations methodology, the technology we use to collect data during each Evaluation (called Arcade), and the website where we publish results.</p><h3>Improvements in Round 2</h3><p>Now let us readdress the problem.</p><blockquote>How do we publish results that are sufficiently nuanced, without leading to inadvertent editorialization?</blockquote><p>Our initial answer was to publish everything we collected and let the community interpret the results. As we learned, however, that has led into analyses that are broadly focused on detection coverage — or whatever else is presented at the surface — even if deeper analysis is achievable with the information that was published. To make this deeper analysis more accessible, our new and improved approach is to publish additional objectively and consistently collected qualitative and quantitative data that will help the community interpret the results in a way that can be made relevant to their own strategy for a threat-informed defense.</p><p>To capture those nuances in each participant’s Evaluation, it is first important for each managed service to be given the opportunity to define their unique approach. This way the participants can be measured against what they would like to emphasize to their prospective and existing clients. Once their approach has been defined, they must be evaluated against an extremely rigorous Emulation Plan so that they can truly be challenged. After the data has been collected in an objective and consistent manner, the results must then be published in a way that resonates with the end-users who want to use the results. To confirm that we were on the right path, we announced an open call for strategy validation sessions in our initial release article. After engaging with interested parties, the strategy validations sessions were in fact very validating, and the following are the conclusions we drew off the back of that exercise.</p><blockquote><strong>What are ways we can make the scenario more complex?</strong></blockquote><p>As we investigated how to make the Emulation Plan more challenging this round, we prioritized increasing the breadth and depth of the scenario. While breadth refers to the variety of adversary behavior that is emulated in the scenario, depth refers to the sophistication and technical complexity of those emulated behaviors. Though we cannot share the specifics of these improvements due to the black box nature of our Evaluation, significant resource has been put into improvement in both aspects. Additionally, since our Evaluation methodology is structured to discover how vendors can find the proverbial needle in the haystack, another way we are making the scenario more challenging is by increasing the size and realism of that haystack. To that end, this round will feature more realistic noise generation that will make it harder for participants to discern suspicious and malicious behavior.</p><blockquote><strong>How do we approach qualitative information collected in each Evaluation?</strong></blockquote><p>The key qualitative categories introduced in this round are attribution and remediation. Attribution is a key component of a strategic defense because it directs teams to allocate resources in a more concerted manner based on observed activity. Many of the managed services participants in the OilRig round had identified the adversary, and this round we are officially supporting the display of that content in the published results. We follow a black box methodology in Managed Services Evaluations, which means that we will not be publishing the Technique Scope until we publish the results from the round. This ensures that participants do not have any insight into which adversary behavior we are emulating. That said, our CTI research has yielded intriguing information, so we are introducing the Target Organization Profile. This target organization profile was generated based on the adversary behaviors that will be emulated in this round. Remember, for both end-users and managed services, we perform adversary emulation, not simulation. This means that the Evaluation and ensuing results are relevant for your organization regardless of whether this target organization profile represents your focus because the emulated behavior should be defended against even if it is not being used by the adversaries we choose. We will be releasing more content regarding the Target Organization Profile in the coming months, so stay tuned for a deeper dive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*WxwiwnsxvwCfTimpHwU4DQ.png" /><figcaption>Target Organization Profile</figcaption></figure><p>The next major improvement is capturing remediation guidance. We investigated a variety of different strategies to support implementing remediation, and ultimately determined that actual implementation of remediation suggestions provided by participants would prevent our ability to collect consistent data on each subsequent step in the Emulation Plan. For instance, if a participant told us to prevent “gosta” from downloading and opening the Word document with the malicious macro in the very first step of last year’s <a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/oilrig/Emulation_Plan">OilRig scenario</a>, we would not have been able to capture results for the remainder of the Evaluation. For this round we will allocate real estate on our results website to publish remediation suggestions sent to us in an organized format. While validating the efficacy of these remediations will be out of scope for this round, they will represent the managed service ‘value add’ to a client in the context of what is being emulated.</p><blockquote><strong>What are some quantitative metrics we can capture and publish this round?</strong></blockquote><p>There are four main categories of metrics we are looking to capture this round: reporting status, type of report, speed, and efficiency.</p><p>Reporting Statuses were purposefully basic in the OilRig round, and we learned that greater depth is required to describe reported activity. While we believe in the importance of elevating the tactics and techniques in most regards, clients of managed services do not require that level of depth in a standard engagement. For example, if you are a CIO who is tasked with cybersecurity and are onboarding a managed service to help, you likely won’t appreciate being told about every single technique/sub-technique an adversary used. This may get frustrating and become noisy in ways you were looking to avoid by onboarding a managed service. Moreover, every CIO would have different requirements for the tactics and techniques they would like in their reports. On the flipside, if you are a client who is augmenting your existing cybersecurity analyst team with a managed service, it is likely that the entire catalogue of tactics and techniques could be incredibly valuable. Evaluations are purposed with collecting data helpful in all circumstances, while also remaining relevant enough to allow for nuanced analysis in specific circumstances. Since each managed service vendor may have a specific approach that caters to a specific clientele, we do not want to restrict their strategy by defining which approach we think is best. To that end, we have introduced two descriptors of “Reported” behavior and the definitions are highlighted here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*sbPcJc5CIiFmxSmKzr04LQ.png" /><figcaption>Reporting Status</figcaption></figure><p>1. Reported — MITRE Engenuity was notified of the adversary behavior being evaluated and sufficient context was given to explain the activity.</p><p>a) General — activity was reported to the Evaluations team with sufficient context, though no additional information was provided regarding the tactic and/or technique of that behavior.</p><p>b) Tactic/Technique — activity was reported to the Evaluations team with sufficient context, and additional information was provided regarding the tactic and/or technique of that behavior.</p><p>2. Not Reported — MITRE Engenuity was not notified of the adversary behavior being evaluated or the information provided did not contain sufficient context to be considered “Reported.”</p><p>3. Not Applicable — Execution failed, so the managed service provider did not have the opportunity to report on the activity.</p><p>Managed services are diversified in many ways, and one of the most important characteristics of diversification is the form factors (instant message, email, PDF reports, dashboard access, etc.) each uses to conduct their services. Each client may only be able to support specific form factors, and even within client organizations, individual team members may have a variety of different preferences. We will highlight that diversity by categorizing the form factor of each report that is deemed to have a “Reported” status.</p><p>The participants will define the form factors to be used before the evaluation starts. For example, if a vendor participant uses email and daily PDF reports to conduct their service, each report given a “Reported” status will also include either “Email” or “PDF Report” as an additional level of context for end-users to see. The full list of supported form factors will be determined after the cohort closes, and it is our intent to include as many form factors as is possible. With these added metadata points, each individual report will have a far higher level of definition than last year’s evaluation.</p><p>Managed services offerings are also typically evaluated by more than just their ability to report adversary behavior. They are evaluated by the speed and efficiency of their service as well. Within the methodology of our Evaluation, the following is a graphical depiction of what can be objectively and consistently observed in each Evaluation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*stN4yr0-VPtCFhaimjeaNA.png" /><figcaption>Additional Metrics</figcaption></figure><ol><li>Activity Executed — the specific timestamp that our Red Team executed a specific sub-step within the Emulation Plan.</li><li>Activity Reported — the specific timestamp that we viewed the report of the executed activity sent by the participant.</li><li>Remediation Suggested — the specific timestamp that we receive the participant’s remediation suggestion.</li></ol><p>Note: Some managed services may decide to report and provide a remediation suggestion at the same time. In this scenario, the times to detect, investigate, and respond will be the same.</p><p>Since speed definitionally requires a value for “time elapsed,” this methodology allows us to support an objective and consistent way to display the speed of the managed service during the Evaluation. Due to the structure and breadth of the Emulation Plan, each measurement of the mean times to detect, investigate, and respond will be shown at the Emulation Plan step level and in aggregate for the entire Evaluation. These are the definitions of the key metrics that will be published:</p><ul><li><strong>Mean Time to Detect (MTTD)</strong>: average time between Activity Executed and when the Activity Reported content is viewed by MITRE Engenuity.</li><li><strong>Mean Time to Investigate (MTTI)</strong>: average time between when Activity Reported content is viewed and when the Remediation Suggestion is received by MITRE Engenuity.</li><li><strong>Mean Time to Respond (MTTR)</strong>: average time between Activity Executed and when the Remediation Suggestion is received by MITRE Engenuity.</li></ul><p>Even though we are not validating the remediation suggestions, we will still have the timestamp that those remediation suggestions were sent to our Execution Leads (who act as the pseudo-client in this format). That means that we can capture Time to Respond and calculate the MTTR throughout each Evaluation. Participants may be able to offer inaccurate remediation suggestions to lower their MTTR and MTTI. This is a known limitation of our methodology, but it is important to note that the remediations themselves are verifiable, whether or not the Evaluations team verifies the remediations during execution.</p><p>Measuring speed is unproductive without also considering efficiency. Spamming and other forms of trigger-happy alerting may lead to a lower MTTD, MTTI, and MTTR during an Evaluation, but would also not be valuable to a client in the real-world. There must be a control to influence efficient communication within the construct of this Evaluation. One of the key reasons a client would bring on a managed service rather than onboarding another product is because their strategy requires offloading relevant workflows to an external team. If a managed service is inundating your inbox with emails, peppering you with instant messages, creating hundreds of notifications on a dashboard, etc. the service is not being efficient, even if they may be quick to tell you something is wrong. To that end, beginning this round, we will curate a running count of each report sent to us during the evaluation based on the form factor categories defined by the participant. This running count will be shown at the Emulation Plan step level and in aggregate for the entire Evaluation.</p><p>With these improvements, we hope to detract from interpretations emphasizing mere detection count and empower deeper analyses of the unique offering of each managed service provider within the context of the Emulation Plan.</p><h3>Next Steps</h3><p>If you are interested in participating in this round, please <a href="https://info.mitre-engenuity.org/msr-2">fill out this form</a>. As an update to the approximate timelines we shared, here are the confirmed timelines for the next phases:</p><ol><li>Orientation Phase — onboarding all participants to the engagement to prepare them for subsequent phases. This phase will begin two weeks after closing the Call for Participation, and we will be reaching out to participants directly to coordinate.</li><li>Setup and Execution Phases — what most consider the actual Evaluation, the first black box engagement will commence with participants on January 15, 2024.</li><li>Results Phase — once we finish collecting the results for all participants in the cohort during execution, our team commences a processing and calibration phase. The start date and length will be determined by the size of the cohort.</li><li>Publication Phase — after results have been processed and shared back with the participants, we dedicate this phase for our stakeholders to prepare release collateral with access to all the finalized results. This date will also be determined by the size of our cohort, and the projected timeframe for final release is currently the third quarter of 2024.</li></ol><p>Thank you and we are looking forward to our most engaging Evaluation yet!</p><p>© 2023 MITRE Engenuity, LLC. Approved for Public Release. Document number AT0045</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59aa207641fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MITRE Engenuity ATT&CK® Evaluations: Managed Services — Round 2 (2023) Call for Participation]]></title>
            <link>https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-managed-services-round-2-2023-call-for-participation-d7fd5a85bcf9?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7fd5a85bcf9</guid>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 18:51:41 GMT</pubDate>
            <atom:updated>2023-03-16T18:51:41.196Z</atom:updated>
            <content:encoded><![CDATA[<h3>MITRE Engenuity ATT&amp;CK® Evaluations: Managed Services — Round 2 (2023) Call for Participation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GiXI9Jrdmekf1IRpg7bwHQ.png" /></figure><p>We are excited to officially open the Call for Participation for MITRE Engenuity’s ATT&amp;CK® Evaluations: Managed Services — Round 2 (2023)! As shared in our OilRig (2022) release, we will be holding these evaluations annually. As always, we prioritize providing as much public utility as possible with each Evaluation to help contribute to understanding the efficacy of the people, process, and technology relevant to security programs built to address adversary behavior.</p><h3>Goals for this Evaluation</h3><p>Our Evaluations are structured as research projects and are purposed with catering to the following goals:</p><ol><li><strong>Empower end-users with objective insights</strong> into how to leverage specific commercial cybersecurity capabilities to address known adversary behaviors.</li><li><strong>Provide transparency around the true capabilities of commercial security offerings</strong> to address known adversary behaviors.</li><li><strong>Drive the security vendor community to enhance their offerings</strong> to better address known adversary behaviors.</li></ol><p>If you are a security practitioner or leader who is constantly inundated with marketing material, it can become a burden to sift through the dense collateral and discern which security solution is the best fit for your organization. Ultimately, in the land of a million great options, which commercially available Managed Service will best help you realize your security strategy? We hope that Evaluations are one of the many data points contributing to your final decision. If this is the first time you have interacted with the Evaluations content, we suggest reading this <a href="https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-managed-services-oilrig-2022-and-the-top-10-ways-to-411b155c3af3">article</a> that was published alongside the Managed Services (2022), OilRig results on November 9, 2022.</p><p>If you are a Managed Services provider who is looking to help your current and prospective clients understand more about how your service addresses known adversary behavior, Evaluations can be a phenomenal platform to showcase your service and highlight the nuances of your offering to your audience. Moreover, as we deliver on our third goal, we hope to continue influencing the community to emphasize features and functionality that continuously improve and address known adversary behavior. Many of our participants join our Evaluation to discover where to prioritize enhancements in their roadmap. When you have multiple high priority feature requests from multiple high priority clients, participation in Evaluations may serve as a backdrop to understand priority as it contributes to strategic delivery. For the full scope of a released Evaluation, review the Managed Services (2022), OilRig round <a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig/">Overview Page</a> to learn more.</p><h3>Extensions to our Methodology</h3><p>In the interest of providing added value to our community, we are always looking for ways to extend our methodology. In this Evaluation, we are looking to answer the following questions as we march toward execution:</p><p>What are ways we can make the scenario more complex?</p><ul><li>As will always be the case, Managed Services Evaluations will be run in a black box format. The purpose of employing this methodology is to ensure that we genuinely evaluate the ability of the service, rather than the tools deployed to fulfill the service. Without sharing the specifics of the Emulation Plan, we are working on making the scenario used for the adversary emulation far more complex than the last round. This will allow us to capture and publish results that help the community understand the nuances of each Managed Services participant.</li></ul><p>What are some quantitative metrics we can capture and publish this round?</p><ul><li>In the OilRig round, our methodology included designations for three Reporting Statuses to provide context for each step of our Emulation Plan: Reported, Not Reported, and Not Applicable. We are looking to publish even more context in the results this round. Can we publish a volume metric on the content that participants send to us? Can we capture important metrics, like Mean Time to Detect, consistently and objectively? With these extensions to our methodology, we aim to equip our community with the tools they need to make purchasing decisions.</li></ul><p>How do we approach qualitative information collected in each Evaluation?</p><ul><li>Supporting qualitative information in Evaluations results is always tricky. Our foremost principle is objectivity, and we, therefore, ensure we do not editorialize throughout the Evaluation, especially in our published results. While that will always be true, we can collect data that may lend itself to qualitative information, which can be published upon release. For instance, did the participant appropriately attribute the reported activity to a specific adversary? Were there remediation suggestions provided in the content sent as part of the Evaluation? These types of data points will help the community understand the qualitative context of each participant’s performance in the Evaluation.</li></ul><p>How do the answers to these questions lead to improvements to our Evaluations Platform?</p><ul><li>Performing the best Evaluation possible is key; the results must also be presented in the most meaningful format so our community can digest them effectively. Simply put, the aforementioned areas we are extending to mean little if we cannot make the results more usable. To that end, we want to significantly improve our UI to support all the new areas we are investigating. This includes supporting the qualitative and quantitative data on the Evaluations Platform and improving existing tools for analysis.</li></ul><p>While we have well-formed answers to the questions above, we are eager to connect with our community to validate our strategies. To that end, if you would like to be part of the design process for these improvements, please contact us at <a href="/service/mailto:evals@mitre-engenuity.org">evals@mitre-engenuity.org</a>, and we will set up a time between March 27-April 14, 2023 to discuss. We will be engaging with as many community members as possible in that timeframe.</p><h3>Next Steps</h3><p>This Call for Participation will close on June 30, 2023, with an iterative update scheduled for May 10, 2023. We are approximating that the Execution Phase commences in Q4 of 2023, and the results will be published in Q2 or Q3 of 2024, depending on our cohort size. If you are interested in participating in this Evaluation, please fill out <a href="https://info.mitre-engenuity.org/msr-2">this form</a>, and we will get back to you with additional details as soon as possible.</p><p>Lastly, we have devoted many resources to learning from our community and ensuring we innovate in ways that resonate with our stakeholders. We will be sharing content regarding our progress on that front very soon. As always, we appreciate the collaboration, and look forward to another exciting Evaluation!</p><p><em>© 2023 MITRE Engenuity, LLC. Approved for Public Release. Document number AT0043</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7fd5a85bcf9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-managed-services-round-2-2023-call-for-participation-d7fd5a85bcf9">MITRE Engenuity ATT&amp;CK® Evaluations: Managed Services — Round 2 (2023) Call for Participation</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MITRE Engenuity ATT&CK® Evaluations: Managed Services — OilRig (2022) and the Top 10 Ways to…]]></title>
            <link>https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-managed-services-oilrig-2022-and-the-top-10-ways-to-411b155c3af3?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/411b155c3af3</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <category><![CDATA[managed-services]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Wed, 09 Nov 2022 17:31:26 GMT</pubDate>
            <atom:updated>2022-11-11T04:10:49.906Z</atom:updated>
            <content:encoded><![CDATA[<h3>MITRE Engenuity ATT&amp;CK® Evaluations: Managed Services — OilRig (2022) and the Top 10 Ways to Interpret the Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*UOTLb5bAOCH4QxKFWPT0pA.png" /></figure><p>We are proud to announce the <a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services/">completion</a> of our inaugural ATT&amp;CK® Evaluations: Managed Services round. There is a lot to unpack in this Evaluation, and this content should provide some much-needed context for you to consume the results effectively. With as much gusto as I can convey through a blog post, I urge every individual that seeks to apply the information gained from this round’s results to take the time to read this blog before trying to tackle the output. The results are far more effective with context, and we aim to provide as much as possible in this post.</p><h3>The Big Reveal: OilRig</h3><p>Unlike previous ATT&amp;CK® Evaluations: Enterprise rounds, in the Managed Services Evaluation we employed a <a href="https://medium.com/mitre-engenuity/att-ck-evaluations-for-managed-services-d8c60fda118b">“black box” (formerly referred to as “closed book”)</a> methodology and did not announce the adversary to the public or Evaluation participants at the outset. We will cover more about the general process in later sections, but for now, we’d like to announce that this Evaluation emulated behaviors inspired by <a href="https://attack.mitre.org/groups/G0049/">OilRig</a>, a threat actor with operations that align with strategic objectives of the Iranian government.</p><p>OilRig has a well-documented history of widespread impact, with campaigns appearing around the globe that are directed against financial, government, energy, chemical, telecommunications, and other sectors. The group continues to evolve its tactics, and leverages a combination of proprietary malware, customized versions of publicly available tools, and off-the-shelf, multi-purpose software. We selected OilRig based on their defense evasion and persistence techniques, their complexity, and their relevancy across industry verticals.</p><p>This round focused on OilRig’s use of custom web shells and defense evasion techniques. OilRig has demonstrated their sophistication in campaigns through organized resource development, unique data exfiltration methods, and use of customized toolsets to persistently access servers. While OilRig may leverage more common techniques compared to other threat actors, the group’s distinctive characteristics are rooted in their diverse arsenal of backdoors.</p><p>Our Emulation Plan combined relevant portions of a variety of OilRig campaigns. In order to support a new methodology for evaluating service providers, we sought to pace the Emulation Plan over the course of a five-day work week. Additionally, for the first time in an ATT&amp;CK Evaluations round, we simulated benign user noise within the target environment, which was particularly relevant due to the black box nature of this Evaluation.</p><p>At a high level, the OilRig Emulation Plan commenced with a pre-positioned email containing a link to download a macro-enabled document armed with our initial implant. From there, the plan used well-known techniques to pivot from machine to machine, discovering new hosts and dumping credentials along the way. Through this activity, our Red Team eventually reached the SQL server to exfiltrate database information.</p><p>In this general process, the Emulation Plan reflected techniques and sub-techniques that are highly relevant in the real world. For instance, spearphishing was used for initial access due to its prevalence in known attacks. On the other hand, web shells, lateral movement via RDP (Remote Desktop Protocol) tunnels and PsExec, pass-the-hash, credential dumping, and exfiltration over email are all still relevant and used by adversaries today. Our technique scope was executed over multiple machines, which provided a particularly relevant Evaluation for enterprise environments that are targets for similar behavior by threat actors operating in the wild.</p><h3>Components of this Evaluation Release</h3><p>Many of our community members are made aware of these results through the collateral (marketing, direct sales, etc.) prepared by participants, but the results that they reference are merely the tip of the iceberg. Below is a list of the release’s individual components:</p><ul><li>Updates to the <a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig/">Overview Page </a>— a summary of the entire Evaluation. This 10,000-foot view provides a great way to understand what was accomplished in this joint research project.<br>- ATT&amp;CK Description — added context about OilRig.<br>- Emulation Notes — high-level notes about the Emulation Plan.<br>- <a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig/#sn-operational-flow">Operational Flow</a> — a diagram to describe the general operational flow of the Emulation Plan with some associated qualitative content to describe the scenario.<br>- <a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig/#sn-technique-scope">Technique Scope</a> — a MITRE ATT&amp;CK Navigator representation of the specific techniques used in this emulation.<br>- <a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig/#sn-environment">Environment</a> — the Microsoft Azure environment leveraged in this Evaluation with some qualitative text to describe the infrastructure.<br>- <a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig/#sn-reporting-status">Reporting Status </a>— qualitative descriptions of the Reporting Statuses awarded for each step/sub-step within the Emulation Plan.<br>- <a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig/#sn-resources">Additional Resources</a> — ways to understand more about this Evaluation.</li><li><a href="https://attackevals.mitre-engenuity.org/managed-services/oilrig/#sn-results">Results</a> — each participants’ results from ATT&amp;CK® Evaluations: Managed Services — OilRig.<br>- Archive — on each participant’s results page, you can find an archived file of all the content that was sent to MITRE Engenuity in this Evaluation.<br>- <a href="https://attackevals.mitre-engenuity.org/managed-services/participants/demoparticipant?view=managed-services">Demo Participant</a> — for this inaugural round, we have also included a demo participant to help walk through the experience for the end user. Stay tuned, we will be publishing a walkthrough of the demo participant’s results in our next blog post.</li><li><a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/oilrig">Technical Collateral on GitHub</a> — technical collateral that can be leveraged to understand, recreate, and defend against the technical details of this Evaluation.<br>- <a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/oilrig/Emulation_Plan">Emulation Plan</a> — the steps and sub-steps used in this Evaluation.<br>- <a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/oilrig/Resources/Binaries">Binaries</a> — the specific technical artifacts that you can leverage to recreate this emulation in your own environment.<br>- <a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/oilrig/Resources/setup">Infrastructure Setup</a> — the technical details of the infrastructure utilized for each service provider that participated in this Evaluation.<br>- <a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/oilrig/YARA_Rules">YARA Rules </a>— signatures to help detect the activity described in the Emulation Plan.<br>- Caldera Port (to be released) — an automated implementation of the Emulation Plan for use in your own environment.</li></ul><p>As demonstrated above, there are many elements to this release. You should be utilizing the amalgam of resources most relevant to you, depending on the level of depth and breadth you are looking to reach with your analysis.</p><h3>How was this Evaluation Conducted?</h3><p>The following are goals for this and all our Evaluations:</p><ol><li>Empower end-users with objective insights into how to use specific commercial security capabilities to address known adversary behaviors.</li><li>Provide transparency around the true capabilities of security capabilities to address known adversary behaviors.</li><li>Drive the security vendor community to enhance their capability to address known adversary behaviors.</li></ol><p>These goals serve as our north stars when we are designing the methodology of our Evaluations. For this Evaluation, MITRE Engenuity employed a five-week process to evaluate each of the service providers who participated:</p><ul><li>Setup Phase (Weeks 1–3) — service providers worked with MITRE Engenuity to accomplish the activities required for execution.<br>- Upon gaining access to the Microsoft Azure range hosted by MITRE Engenuity, service providers had the opportunity to deploy the tools necessary for the Evaluation.<br>- In Week 3, the service provider had an opportunity to share how to access the content they would report for this Evaluation. This session was held with MITRE Engenuity, who acted as the pseudo-customer during execution.<br>- MITRE Engenuity performed multiple checks in Week 3 to ensure that the tools were deployed properly and were compliant with guidelines for this Evaluation.</li><li>Execution Phase (Weeks 4–5) — the MITRE Engenuity Red Team executed the Emulation Plan while the service providers worked with the MITRE Engenuity Execution Team.<br>- The emulation began on Week 4 — within business hours — at an unannounced time and cadence. Although service providers received information on the environment they were monitoring, they did not receive any information on when attacks would be occurring, what techniques were in scope, or what adversary MITRE Engenuity was emulating.<br>- After the emulation had concluded, the service provider had 24 hours to send the final package of content to MITRE Engenuity for consideration. This package (and the content that was collected by the Execution Team) served as the superset of what could possibly be considered for results.<br>- In Week 5, MITRE Engenuity presented information on the emulated adversary and associated details to the plan.</li></ul><h3>Reported vs Unreported</h3><p>If there is one thing that you take away from this blog, it should be this:</p><blockquote><strong>We did not expect (nor believe it is inherently valuable) for each service provider to report each technique/sub-technique evaluated in the Emulation Plan.</strong></blockquote><p>Therefore, any analysis that reflects counts or ratios of reported techniques vs the “total count” is antithetical to the purpose of this Evaluation. We are emphasizing this stance in the published results by not providing summaries of these counts as we have previously done in Enterprise Evaluation results. As we dive into the results interpretation, it is incredibly important to understand this distinction.</p><p>So, what does qualify as Reported versus Unreported? The threshold we used throughout this Evaluation, across each Emulation Plan sub-step was the presence of context. Did the service provider include enough context to describe the emulated activity? If MITRE Engenuity believed the answer was yes, they were awarded a “Reported Status” on the specific sub-step.</p><p>An “Unreported Status” for a specific sub-step could result from the following:</p><ul><li>The service provider believed the activity was not of significant importance and reporting to the customer would not add value.</li><li>The service provider believed the activity was implied or could be assumed through other content which they provided.</li><li>The service provider did not provide additional context to meet our threshold to be considered “Reported.”</li><li>The service provider missed or misinterpreted the activity.</li></ul><p><em>Note: We use the “Not Applicable” status when we could not execute the step/sub-step, in which case the results are therefore not applicable to the Evaluation.</em></p><h3>How do I Interpret These Results?</h3><p>Before we get into the fun stuff, let us clarify the role that ATT&amp;CK Evaluations intends on playing in the security space.</p><ul><li><strong>ATT&amp;CK Evaluations are a starting point.</strong> We hope that these results serve as one of many data points in your commercial capabilities assessment. Especially if you are interested in building a threat-informed defense, ATT&amp;CK Evaluations can help you determine what behaviors are actually addressed.</li><li><strong>There are no winners.</strong> Any claim by a participant that they “won” is an intrinsically flawed statement. As we walk through our Top 10, it should become clear why we cannot award a winner.</li><li><strong>This round was not a product Evaluation.</strong> There are inherent differences between service offerings and security products. It is, therefore, extremely important to ensure you do not reduce these complex results to what we typically include in the summaries of our Enterprise Evaluations.</li><li><strong>Before starting any analysis of technique coverage, it is important to determine which techniques are most relevant to your organization</strong> based on the adversary groups and threats that your organization faces. This information is outside the scope of ATT&amp;CK Evaluations; once you have this list, you can start analyzing the results of those techniques within ATT&amp;CK Evaluations. (You could use a variety of sources to get this information, whether it be the MITRE ATT&amp;CK Framework, public/commercial threat reporting and analysis, or your own cyber threat intelligence to determine this subset of techniques.)</li><li><strong>Not all techniques are created equal. </strong>A service provider reporting on Process Discovery might not have the same value as a service provider reporting on Credential Dumping due to the severity of the action.</li><li><strong>Not all procedures are created equal. </strong>Process Discovery (T1057) via Command-Line Interface (T1059) can be detected with most process monitoring. Process Discovery via API (T1106) would need API monitoring. A service provider could have reported one, but not the other.</li></ul><p>With that framework in mind, here are the top 10 ways a savvy security practitioner or leader should interpret these results.</p><ol><li>Look at the top-level Report Statuses of the service provider you are analyzing. This will serve as your first level of analysis to show you how the service provider performed against the Emulation Plan at the highest level possible.</li><li>Understand how the service provider would be relevant to your organization based on current gaps in your security controls. The following factors contribute to why techniques should not be assessed as absolutes, and why counts and ratios are inherently flawed.<br>a) For instance, the first technique (highlighted in Step 1.A.1) used in the Emulation Plan is Phishing. If a service provider did not report on that technique per the results, it may not be relevant to your organization if you have recently invested in an email security product. Alternatively, if you believe that phishing is a technique that needs better coverage in your organization, you may want to pay special attention to a service provider that reported it in this Evaluation.<br>b) While a malicious action may have multiple techniques associated with it, you may be primarily concerned with the malicious action being reported at all. For example, an implant communicating with its command-and-control server could be evaluated for Application Layer Protocol, Encrypted Channel, and Non-Standard Port. In this scenario, you may not be concerned with reports on all three techniques, so long as the service provider has reported the command-and-control traffic.</li><li>Look at how service providers presented their findings to their customers. Some provided links to alerts within a console that could be used to analyze threats and submit questions. In this scenario, they may be able to use the alert to pivot and find additional adversary behavior. Others used a more narrative-based approach with tickets or emails sent to the customer, explaining what happened without linking to alerts within a console. Every method is valuable in the right light, the trick is to determine which is most resonant with your team.</li><li>Determine if the service provider correctly attributed the adversary. This was not a requirement by MITRE Engenuity in this Evaluation, but adversary attribution could be helpful to your organization as you seek to explain your world to non-cybersecurity leadership. We often hear that adversary attribution allows SOC Analysts to summarize the technical aspect of incidents more effectively to their executive leadership.</li><li>Find out if the service provider recommended mitigations to the activity they detected. This was also not in scope for this Evaluation, but some service providers included some insight into how this activity should be remediated. Actual mitigations/remediations were not allowed in this Evaluation due to the impact they would have on the execution of our Emulation Plan.</li><li>Deeper analytics may be possible based on the data in the archive. For instance, you may be able to check timestamps in images to gauge response time for some results. This was also not stipulated in the MITRE Engenuity methodology, but this information may be possible based on the output of some of the service providers’ results. In future iterations of the Managed Services Evaluation, we are looking for fair and objective ways to include mean time to respond (MTTR) and other such metrics. The archive is the best way to get the content you need to perform these deeper analytics because you will find all the content the service provider offered MITRE Engenuity during the Evaluation, even if it didn’t meet the threshold for a “Reported” status.</li><li>Look at the report lengths and determine if they fit the requirements of your team. This is completely preference-based, but the published results provide in-depth insight into each service provider’s approach.</li><li>Determine if the language in the reports is relatable for you and your team. Again, this is completely subjective, but ultimately the language used by a services offering must resonate with your team for you to get value out of leveraging the service.</li><li>Review the archived information and determine the scope of the Evaluation at large. You will not only be able to understand the volume of information captured during the Execution Phase, but you may also find that the results of the service provider you are analyzing may meet your own threshold for reporting, even if they did not meet MITRE Engenuity’s (rather rigorous) threshold for reporting. By reviewing the archive, you can get a more accurate picture of the scope of engagement with the specific service provider and determine where their level of communication falls on your spectrum of “sufficient” versus “superfluous.”</li><li>Look at the releases published by the participants. Once you have reviewed the results and have a few service providers in mind, we recommend reviewing the associated releases published by these service providers regarding this Evaluation. They are often more detailed and dive deeper into the nuances of their own results.</li></ol><h3>Next Steps</h3><p>This joint research project was the first of its kind and required a massive collaborative effort from each participant and MITRE Engenuity. I want to take this opportunity to explicitly thank the ATT&amp;CK Evaluations team who has worked diligently over the past year to deliver this release. In the interest of iteratively improving on our process, we will be meeting with our <a href="https://info.mitre-engenuity.org/join-evaluations-community-advisory-board">Community Advisory Board</a> members and our participants to understand the good, the bad, and the ugly of this Evaluation with the intention of announcing Round 2 early next calendar year. We love feedback, so please do not hesitate to share it with us through the proper channels. Stay tuned and we appreciate your participation!</p><h3>A Big Thank You to Our Participants</h3><p>In this inaugural Evaluation round, we had the privilege of working with the following organizations: <a href="https://atos.net/en/">Atos</a>, <a href="https://www.bitdefender.com/">Bitdefender</a>, <a href="https://www.blackberry.com/us/en/services">BlackBerry</a>, <a href="https://www.bluevoyant.com/">BlueVoyant</a>, <a href="https://www.criticalstart.com/">Critical Start</a>, <a href="https://www.crowdstrike.com/services/managed-services/falcon-complete/">CrowdStrike,</a> <a href="https://partner.microsoft.com/en-au/solutions/managed-services">Microsoft</a>, <a href="https://www.nviso.eu/">NVISO</a>, <a href="https://www.opentext.com/solutions/managed-extended-detection-and-response">OpenText</a>, <a href="https://www.paloaltonetworks.com/partners/managed-security-service-providers">Palo Alto Networks</a>, <a href="https://www.rapid7.com/services/managed-services/">Rapid7</a>, <a href="https://redcanary.com/">Red Canary</a>, <a href="https://www.sentinelone.com/global-services/vigilance-respond/">SentinelOne</a>, <a href="https://www.sophos.com/en-us/partners/managed-service-providers">Sophos</a>, <a href="https://www.trendmicro.com/en_us/business/services/managed-xdr.html">Trend Micro</a>, and <a href="https://www.withsecure.com/en/partners/partner-offering/managed-service-provider">WithSecure</a>. Since there is inherent risk in anything inaugural, we cannot overstate our appreciation of our participants for working with the MITRE Engenuity team in this joint research project.</p><p><em>Note: Although Trend Micro participated and completed testing for this inaugural round, after an unintended situation, Trend Micro promptly and responsibly shared that their team had found sensitive information to MITRE Engenuity. Based on the agreement between MITRE Engenuity and Trend Micro, MITRE Engenuity did not publish Trend Micro’s results. Please contact Trend Micro directly for more information</em>.</p><p>© 2022 MITRE Engenuity, LLC. Approved for Public Release. Document Number AT0038</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=411b155c3af3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-managed-services-oilrig-2022-and-the-top-10-ways-to-411b155c3af3">MITRE Engenuity ATT&amp;CK® Evaluations: Managed Services — OilRig (2022) and the Top 10 Ways to…</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MITRE Engenuity ATT&CK® Evaluations: Enterprise — Turla Welcomes 31 Participants]]></title>
            <link>https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-enterprise-turla-welcomes-31-participants-857e48cd335d?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/857e48cd335d</guid>
            <category><![CDATA[product-evangelism]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[enterprise]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <category><![CDATA[evaluation]]></category>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Fri, 14 Oct 2022 23:44:28 GMT</pubDate>
            <atom:updated>2022-10-14T23:44:28.555Z</atom:updated>
            <content:encoded><![CDATA[<h3>MITRE Engenuity ATT&amp;CK® Evaluations: Enterprise — Turla Welcomes 31 Participants</h3><p>We’re pleased to introduce the newest cohort for ATT&amp;CK® Evaluations: Enterprise! This is the fifth iteration of our flagship Evaluation and the Emulation Plan will focus on Turla. As introduced previously, Turla has been active since at least the early 2000s. <a href="https://attack.mitre.org/groups/G0010/">Turla</a> is a sophisticated Russian-based threat group that has infected victims in over 45 countries. <a href="https://attack.mitre.org/groups/G0010/">[1]</a> The group is known to target government agencies, diplomatic missions, military groups, research, and media organizations. <a href="https://www.welivesecurity.com/wp-content/uploads/2020/05/ESET_Turla_ComRAT.pdf">[2]</a><a href="https://media.kasperskycontenthub.com/wp-content/uploads/sites/43/2018/03/07180251/Penquins_Moonlit_Maze_PDF_eng.pdf">[3]</a> <a href="https://attack.mitre.org/groups/G0010/">Turla</a> adopts novel and sophisticated techniques to maintain operational security, including the use of a distinctive command-and-control network in concert with their repertoire of using open source and in-house tools. <a href="https://adversary.crowdstrike.com/en-US/adversary/venomous-bear/">[4]</a> <a href="https://www.secureworks.com/research/threat-profiles/iron-hunter">[5]</a></p><p>Thank you to each of the following vendors for participating in the evaluation — AhnLab, Bitdefender, BlackBerry, Broadcom, Check Point, Cisco, Crowdstrike, Cybereason, Cynet, Deep Instinct, Elastic, ESET, Fortinet, HarfangLab, Malwarebytes, Microsoft, Palo Alto Networks, Qualys, Rapid7, ReaQta (IBM), Secureworks, SentinelOne, SOMMA, Sophos, Tehtris, Trellix, Trend Micro, Uptycs, VMware, WatchGuard, and WithSecure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LNgUz1cqMwEG1e_s1bxrPA.png" /></figure><p>This is our largest ATT&amp;CK® Evaluations cohort to date. Three new vendors have entered into the fold, with 25 vendors participating in both the Detections and Protections portions of our Evaluation and 6 vendors joining as Detection only participants. For reference, the Protections component of the Evaluation is entirely optional. If you are leveraging Evaluations to determine the fit of specific products to your security infrastructure, it’s important to understand that these portions of the Evaluation happen separately and should therefore be analyzed separately as well.</p><p>For newcomers to ATT&amp;CK® Evaluations, our <a href="https://attackevals.mitre-engenuity.org/methodology-overview/">Methodology Overview</a> is a great place to start learning more about the intent of this joint research project. The results for this round will be published in the third quarter of 2023, and we urge you to read through our <a href="https://medium.com/mitre-engenuity/wizard-spider-and-sandworm-att-ck-evaluation-results-data-encrypted-for-impact-t1486-b4cf47a2d5ca">previous content</a> to <a href="https://medium.com/mitre-engenuity/how-to-leverage-att-ck-evaluations-results-announcing-our-first-ever-community-advisory-board-ab195a349c1b">interpret</a> <a href="https://attackevals.mitre-engenuity.org/enterprise/participants/?adversaries=wizard-spider-sandworm">results</a> from the previous round, where we emulated Wizard Spider and Sandworm.</p><p>In advance of the Evaluation itself, we have updated the Evaluation <a href="https://attackevals.mitre-engenuity.org/enterprise/turla/">Overview Page</a>, which now includes information regarding:</p><ul><li>Turla (in the “ATT&amp;CK Description” section)</li><li>Unique aspects of this Emulation Plan (in the “Emulation Notes” section)</li><li>The <a href="https://attackevals.mitre-engenuity.org/enterprise/turla/#sn-participants">vendor participants</a> for this round</li><li>The superset of <a href="https://attackevals.mitre-engenuity.org/enterprise/turla/#sn-technique-scope">techniques/sub-techniques</a> that may be used in this Evaluation</li><li><a href="https://attackevals.mitre-engenuity.org/enterprise/turla/#sn-environment">The environment</a> that will be used in the execution of each Evaluation</li><li>The “<a href="https://attackevals.mitre-engenuity.org/enterprise/turla/#sn-detection-categories">Detection Category</a>” methodology</li></ul><p>As shared previously, we have also officially established a <a href="https://medium.com/mitre-engenuity/community-advisory-board-and-vendor-council-updates-for-att-ck-evaluations-enterprise-turla-fc18e5358c7f">Community Advisory Board and Vendor Council</a>. Over the course of 2 days at Black Hat/DEF CON, we held 2 sessions with each group to learn from our stakeholders. We intend to use that feedback to inform our innovations as we plan our roadmap for 2023 and will continue to have sessions to gather further insight. We are currently evaluating what we heard and will publish content regarding our learnings soon. We appreciate the engagement and are looking forward to sharing some exciting updates.</p><p>The MITRE Engenuity team is excited to begin this Evaluation and eager for the opportunity to engage with the community once again!</p><p>© 2022 MITRE Engenuity LLC. Approved for Public Release. Document number AT0034</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=857e48cd335d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-enterprise-turla-welcomes-31-participants-857e48cd335d">MITRE Engenuity ATT&amp;CK® Evaluations: Enterprise — Turla Welcomes 31 Participants</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community Advisory Board and Vendor Council + Updates for ATT&CK® Evaluations Enterprise — Turla]]></title>
            <link>https://medium.com/mitre-engenuity/community-advisory-board-and-vendor-council-updates-for-att-ck-evaluations-enterprise-turla-fc18e5358c7f?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc18e5358c7f</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <category><![CDATA[community-engagement]]></category>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Wed, 03 Aug 2022 20:27:43 GMT</pubDate>
            <atom:updated>2022-08-05T19:38:54.371Z</atom:updated>
            <content:encoded><![CDATA[<h3>Community Advisory Board and Vendor Council + Updates for ATT&amp;CK® Evaluations Enterprise — Turla</h3><p>After announcing our MITRE Engenuity ATT&amp;CK® Evaluations Community Advisory Board, we received an immense amount of support from security vendors seeking to participate. While reviewing early feedback from both end-users and vendors, we realized that the two groups provide inherently different feedback; the varied nature of this early feedback ultimately led to our creating the Vendor Council.</p><p>The purpose of this post is to provide additional information on how we aim to engage with the above-specified groups and to provide updates on ATT&amp;CK® Evaluations Enterprise-Turla. Without further ado, allow me to re-introduce the Community Advisory Board while also introducing — for the first time — the ATT&amp;CK® Evaluations Vendor Council.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tqlRXz4Eu70Eh45FKsxnTQ.png" /></figure><h3>Our Ecosystem</h3><p>In pursuit of conducting open and fair ATT&amp;CK®-based Evaluations for the public good, we have cultivated an ecosystem that help us secure our world:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*9-RvLkSAfUlf_5gEdFCZiQ.png" /></figure><p>Although we are committed to continuously deliver value to both sides of our ecosystem, our primary emphasis will always favor our end-user community. Nonetheless, our vendor participants’ products and services play a key role in determining how our end-users realize their security strategies. The objective for creating two distinct groups is to engage with our stakeholders directly and learn from those whom we impact. In true product management fashion, we intend to collect feedback from our stakeholders, innovate based on that feedback, and prioritize improving the aspects of our offering that resonate most with our user base.</p><p>The first session of both the Community Advisory Board and the Vendor Council will take place during Black Hat and DEF CON 2022 in Las Vegas. If you are interested in participating in the Community Advisory Board, please fill out <a href="https://info.mitre-engenuity.org/evals-community-advisory-board-blackhat?utm_campaign=Evals%20Community%20Advisory%20Board&amp;utm_medium=email&amp;_hsmi=2&amp;utm_content=2&amp;utm_source=hs_email">this form</a> to apply. Although few seats remain, we aim to support as many applicants as possible. Please note that our first Vendor Council meeting is restricted to organizations that have participated in an Enterprise Evaluation Round. We have already reached out directly to vendors that qualify.</p><h3>Updates for ATT&amp;CK® Evaluations Enterprise — Turla</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*NgM25g0I9j8GWSuKJKwf4g.png" /></figure><p>We are also excited to provide an update on our upcoming Evaluation, during which we will emulate Turla. As with every Evaluation, our <a href="https://attackevals.mitre-engenuity.org/enterprise/turla/">Overview Page</a> now has information regarding the <a href="https://attackevals.mitre-engenuity.org/enterprise/turla/#sn-technique-scope">Technique Scope</a> and <a href="https://attackevals.mitre-engenuity.org/enterprise/turla/#sn-detection-categories">Detection Categories</a> that constitute this Evaluation. (The Evaluations experts reading this blog post will notice that the Detection Categories have not changed this year.)</p><p>We would also like to update the community with a timeline for the upcoming round of ATT&amp;CK® Evaluations Enterprise: Turla. As shared in our previous post, the next major milestone is the closing of our Call for Participation on September 16, 2022. If you are interested in participating (or even just curious) please feel free to reach out to <a href="/service/mailto:evals@mitre-engenuity.org"><em>evals@mitre-engenuity.org</em></a>. Ensuing our ongoing, inaugural Managed Services evaluation, we intend to begin the Execution Phase of the Turla Evaluations in the first quarter of 2023 and publish results in the third quarter.</p><p>We eagerly await our group sessions at Black Hat and DEF CON and this new opportunity to connect with our community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc18e5358c7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/community-advisory-board-and-vendor-council-updates-for-att-ck-evaluations-enterprise-turla-fc18e5358c7f">Community Advisory Board and Vendor Council + Updates for ATT&amp;CK® Evaluations Enterprise — Turla</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ATT&CK® Evaluations: Enterprise — Call for Participation with Turla]]></title>
            <link>https://medium.com/mitre-engenuity/att-ck-evaluations-enterprise-call-for-participation-with-turla-f5fd53b033a5?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5fd53b033a5</guid>
            <category><![CDATA[enterprise]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[product-evaluations]]></category>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Tue, 28 Jun 2022 16:59:33 GMT</pubDate>
            <atom:updated>2022-06-28T16:59:33.491Z</atom:updated>
            <content:encoded><![CDATA[<h3>ATT&amp;CK® Evaluations: Enterprise — Call for Participation with Turla (2023)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z6WpNMI770_TRwMPG-ryIA.jpeg" /></figure><p>We are thrilled to announce Call for Participation for <a href="https://attackevals.mitre-engenuity.org/enterprise/turla/">Round 5 of MITRE Engenuity ATT&amp;CK® Evaluations: Enterprise</a>, which is officially open now and closes on September 16, 2022. We expect another strong cohort this year, and are eager to engage with vendor participants, both new and existing. This round will focus on evaluating product capabilities against adversary behavior inspired by <a href="https://attack.mitre.org/groups/G0010/">Turla (G0010)</a> and be executed in Q1 of 2023. Contact us at <a href="/service/mailto:evals@mitre-engenuity.org%20?subject=ATT&amp;CK%20Evaluations%20Enterprise%20Round%205%20-%20Turla&amp;CK%20Evaluations%20Enterprise%20Round%205%20-%20Turl&amp;CK%20Evaluations%20Enterprise%20Round%205%20-%20Tur&amp;CK%20Evaluations%20Enterprise%20Round%205%20-%20Tu&amp;CK%20Evaluations%20Enterprise%20Round%205%20-%20T&amp;CK%20Evaluations%20Enterprise%20Round%205%20-%20&amp;CK%20Evaluations%20Enterprise%20Round%205%20-%20t&amp;CK%20Evaluations%20Enterprise%20Round%205%20-%20&amp;CK%20Evaluations%20Enterprise%20Round%205%20-&amp;CK%20Evaluations%20Enterprise%20Round%205%20&amp;CK%20Evaluations%20Enterprise%20Round%205&amp;CK%20Evaluations%20Enterprise%20Round%20&amp;CK%20Evaluations%20Enterprise%20Round&amp;CK%20Evaluations%20Enterprise%20Roun&amp;CK%20Evaluations%20Enterprise%20Rou&amp;CK%20Evaluations%20Enterprise%20Ro&amp;CK%20Evaluations%20Enterprise%20R&amp;CK%20Evaluations%20Enterprise%20&amp;CK%20Evaluations%20Enterprise&amp;CK%20Evaluations%20Enterpris&amp;CK%20Evaluations%20Enterpri&amp;CK%20Evaluations%20Enterpr&amp;CK%20Evaluations%20Enterp&amp;CK%20Evaluations%20Enter&amp;CK%20Evaluations%20Ente&amp;CK%20Evaluations%20Ent&amp;CK%20Evaluations%20En&amp;CK%20Evaluations%20E&amp;CK%20Evaluations%20&amp;CK%20Evaluations&amp;CK%20Evaluation&amp;CK%20Evaluatio&amp;CK%20Evaluati&amp;CK%20Evaluat&amp;CK%20Evalua&amp;CK%20Evalu&amp;CK%20Eval&amp;CK%20Eva&amp;CK%20Ev&amp;CK%20E&amp;CK%20&amp;CK&amp;C&amp;">evals@mitre-engenuity.org</a> to learn more about this round.</p><h3><strong>Why Turla</strong></h3><p><a href="https://attackevals.mitre-engenuity.org/enterprise/turla/">Turla</a> is an extremely unique adversary that the ATT&amp;CK Evaluations team is thrilled to emulate. Active since at least the early 2000s, Turla is a sophisticated Russian-based threat group that has infected victims in over <a href="https://attack.mitre.org/groups/G0010/">45 countries</a>. The group is known to target government agencies, diplomatic missions, military groups, research, and media organizations. <a href="https://www.welivesecurity.com/wp-content/uploads/2020/05/ESET_Turla_ComRAT.pdf">[2]</a> <a href="https://media.kasperskycontenthub.com/wp-content/uploads/sites/43/2018/03/07180251/Penquins_Moonlit_Maze_PDF_eng.pdf">[3]</a> Turla adopts novel and sophisticated techniques to maintain operational security, including the use of a distinctive command-and-control network in concert with their repertoire of using open source and in-house tools. <a href="https://adversary.crowdstrike.com/en-US/adversary/venomous-bear/">[4]</a> <a href="https://www.secureworks.com/research/threat-profiles/iron-hunter">[5]</a> Turla is known for their targeted intrusions and innovative stealth. After establishing a foothold and conducting victim enumeration, Turla persists with a minimal footprint through in-memory or kernel implants. Turla executes highly targeted campaigns aimed at exfiltrating sensitive information from Linux and Windows infrastructure.<a href="https://www.welivesecurity.com/wp-content/uploads/2020/05/ESET_Turla_ComRAT.pdf">[2]</a> <a href="https://media.kasperskycontenthub.com/wp-content/uploads/sites/43/2018/03/07180251/Penquins_Moonlit_Maze_PDF_eng.pdf">[3]</a></p><h3><strong>What to Look For</strong></h3><p>Over the next month you can expect the ATT&amp;CK Evaluations team to deliver:</p><ol><li>A technique scope, which is the representation of every technique that may be included in this emulation plan</li><li>A summary of the Detection Categories that can be expected during this Evaluation consistent with the <a href="https://attackevals.mitre-engenuity.org/enterprise/wizard-spider-sandworm/detection-categories">Detection Categories that we highlighted in Round 4</a></li><li>A more detailed schedule that maps out the calendar for this round</li></ol><p>If this is your first-time hearing about ATT&amp;CK Evaluations, visit our <a href="https://attackevals.mitre-engenuity.org/methodology-overview/">website</a> for more information about our overall process. For reference, scheduling priority is set by the order in which contracts are signed and the last day for signing is September 16, 2022.</p><h3><strong>Community Contributions</strong></h3><p>We would also like to send a big thank you to our fantastic community who sent CTI <a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/wizard_spider#acknowledgements">contributions for Wizard Spider and Sandworm</a> for our last round. Similarly, we are excited to extend the opportunity to contribute for this upcoming round for <a href="https://attack.mitre.org/groups/G0010/">Turla (G0010)</a>. If you are interested in contributing, please follow these steps:</p><ul><li>Email us at <a href="/service/mailto:evals@mitre-engenuity.org">evals@mitre-engenuity.org</a> with your contribution (If you’d prefer secure means, email us at the above address, and we’ll get back to you with a secure sharing method). Your real name must be included for your information to be considered. Contributions from company accounts may add to the credibility of the information, but we are always happy to accept contributions from independent researchers.</li><li>We are looking for information about the group behaviors as well as the overall way they perform intrusions. Information structured using ATT&amp;CK tactics and/or techniques is helpful, but not required.</li><li>Tell us how you would like to be credited. You can choose to be credited with your name and/or company name, or alternatively, you can choose to remain anonymous. For any anonymous contributors, we will work with you to produce a short statement about the general visibility you have that led to you having access to the information.</li><li>We will not accept any leaked, proprietary, or sensitive information that was not released with the permission of the original source. Contributions are strictly on a voluntary basis for researchers and analysts who wish to share their own information.</li></ul><p>During the remainder of this year, we will put more content out via this Medium. To stay tuned for updates on this round and more, we encourage you to follow this account for notifications or reach out to us at <a href="/service/mailto:evals@mitre-engenuity.org?subject=ATT&amp;CK%20Evaluations%20Round%205%20Blog%20&amp;CK%20Evaluations%20Round%205%20Blog&amp;CK%20Evaluations%20Round%205%20Blo&amp;CK%20Evaluations%20Round%205%20Bl&amp;CK%20Evaluations%20Round%205%20B&amp;CK%20Evaluations%20Round%205%20&amp;CK%20Evaluations%20Round%205&amp;CK%20Evaluations%20Round%20&amp;CK%20Evaluations%20Round&amp;CK%20Evaluations%20Roun&amp;CK%20Evaluations%20Rou&amp;CK%20Evaluations%20Ro&amp;CK%20Evaluations%20R&amp;CK%20Evaluations%20&amp;CK%20Evaluations&amp;CK%20Evaluation&amp;CK%20Evaluatio&amp;CK%20Evaluati&amp;CK%20Evaluat&amp;CK%20Evalua&amp;CK%20Evalu&amp;CK%20Eval&amp;CK%20Eva&amp;CK%20Ev&amp;CK%20E&amp;CK%20&amp;CK&amp;C&amp;">evals@mitre-engenuity.org</a> to explore this or other evaluation programs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5fd53b033a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/att-ck-evaluations-enterprise-call-for-participation-with-turla-f5fd53b033a5">ATT&amp;CK® Evaluations: Enterprise — Call for Participation with Turla</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MITRE Engenuity ATT&CKⓇ Evaluations Results from Deception Trials]]></title>
            <link>https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-results-from-deception-trials-9f15d7a1f4d3?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f15d7a1f4d3</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[deception]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Wed, 25 May 2022 16:41:10 GMT</pubDate>
            <atom:updated>2022-05-25T17:08:56.179Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yt6QQkVvg3SlwaMHQERuAw.png" /></figure><p>Providers of non-detection-oriented security solutions need objective, transparent methodologies to evaluate and frame their value propositions. This need was the driver behind the launch of MITRE Engenuity <a href="https://medium.com/mitre-engenuity/introducing-att-ck-evaluations-trials-7f482cf4fa74">ATT&amp;CK® Evaluations Trials</a>, an exploratory new program designed to help niche solutions providers realistically describe their strengths in defending against known adversary behavior.</p><p>Results from the first Trials project, which focused on Deception solutions, are now available and published on the <a href="https://attackevals.mitre-engenuity.org/trials-deceptions/apt29-deceptions/">ATT&amp;CK Evaluations website</a>. In keeping with our commitment to transparency and trust, here’s a behind-the-scenes look at the participants, methodology, and findings.</p><h3>Participant Snapshot</h3><p>Deception product participants in our Deception Trials project were <a href="https://www.attivonetworks.com/">Attivo Networks</a> (recently acquired by SentinelOne) and <a href="https://www.countercraftsec.com/">CounterCraft Security</a>. As two of the foremost providers of Deception capabilities, our two vendor participants represent a fantastic snapshot of unique Deception products that have inherently different approaches. Accordingly, each have different views on how their solutions fit into a larger security architecture. Please note, we are directly citing responses to vendor participants who we’ve interviewed. The views reflected in the quotes are those of the vendors, and not verified or tested by MITRE Engenuity.</p><p>SentinelOne’s VP of Engineering, Srikant Vissamsetti explains their organization’s perception of how a Deception product fits into a larger security infrastructure.</p><blockquote>“Deception provides early detection of intruders with high fidelity detection and no false positives. Deception products misdirect intruders away from production assets and — study adversary behavior and motive with detailed forensics that helps in understanding the motivation of adversaries and the techniques and tools they use. The cloaking capability built into Deception products provides advanced protection by hiding credentials, Active Directory objects and data, and forces intruders into Deception systems. Deception, along with other security products like EDR and XDR, enhances and strengthens enterprise security.”</blockquote><p>David Barroso, Founder and CEO of CounterCraft, further explains the importance of a Deceptions solution in any security infrastructure,</p><blockquote>“Deception is the only way to provide advanced warning of how an organization is being targeted by their adversaries. It provides high-fidelity alerts on attack patterns, tactics, techniques, and procedures and is the only way to build a detailed profile of the who, what and why of an attack, as it relates directly to you. This detailed threat intelligence is timely, focused, and above all actionable because the intel being collected is directly related to your own organization and your attack surface.</blockquote><blockquote>Deception is the new kid on the block and CounterCraft has consciously designed our platform to fit seamlessly into an existing security infrastructure. With over 45 integrations to third-party security systems out of the box, fitting in with incumbent systems is simple, either to add valuable threat intel feeds to existing analysis systems or interacting with orchestration systems to provide automated cross-platform attack response.”</blockquote><p>With that frame of reference in mind, the next section will explore the process of our Trial Evaluation and identify each vendors’ interest in participating.</p><h3>Why Deception?</h3><p>As we embarked on our journey to design a process to evaluate Deceptions solutions, we first had to ensure that there was a need in the market for a Deceptions Evaluation. As is the case with all our offerings, this Trial virtually caters to a two-sided “marketplace.” On the demand side, we have our end-user community, and on the supply side, we have our vendor ecosystem.</p><p>Our end-user community consists of security practitioners: the folks who defend our data and privacy. We believe this to be the most important stakeholder persona in our Evaluations and we prioritize surfacing objective results to be consumed by the public. In fact, our first core goal of Evaluations is focused on empowering end-users with objective insights into how to use specific commercial security capabilities to address known adversary behaviors. To that end, we decided that Deception capabilities were a great place to start with Trials because there weren’t any other evaluations that focused on Deceptions specifically. The need existed and we decided to open up the <a href="https://medium.com/mitre-engenuity/introducing-att-ck-evaluations-trials-7f482cf4fa74">Call for Participation</a> for this round.</p><p>On the other side of the “marketplace” lies our vendor ecosystem, which consists of the organizations who develop and support products that help our security practitioners run their security program. Like many other cybersecurity providers, these Deception vendors sought emulation and other forms of evaluation to better describe their solution’s value and inform its evolution.</p><p>Srikant further explains his organization’s interest in participating in our first Trials Evaluations,</p><blockquote>“MITRE Engenuity ATT&amp;CK® Evaluations has a reputation for performing thorough and transparent evaluations of security technologies over the years. The testing is systematic and comprehensive and tests solutions’ ability to protect against adversary behavior. SentinelOne supports in its effort to provide transparent and fair methodologies to evaluate adversary behavior.”</blockquote><p>As is the case with the varied approach to Deceptions solutions themselves, David explains CounterCraft’s distinctive initial interest in joining Trials Evaluations,</p><blockquote>“CounterCraft participated to compare our Deception tech with other vendors in a clear and neutral way. Standard minimal functionality and performance criteria do not exist in the Deception space. This is confusing for end users when evaluating Deception technology. We hoped these trials would change that. Standard testing criteria would benefit end users and allow different vendor approaches to shine in different ways. We see MITRE as the best placed organization to create this common and fair testing framework and we are proud to be able to invest in the Trials to provide value to the whole Deception user and vendor community.</blockquote><blockquote>The CounterCraft expectation was to collaborate with MITRE Engenuity in the creation of the testing methodology to fully define metrics that measures the different benefits of a Deception platform and help provide a way of identifying key strengths of each technology. The goal of this is to inform potential users and help them understand how best to deploy Deception according to their needs. This expectation has not been reached yet, but this initial Trial is a start and provides many areas to build on.</blockquote><blockquote>At the forefront of our minds was the possibility of providing an open and impartial analysis of Deception technology, both to raise awareness and to showcase the CounterCraft approach to Deception deployment.”</blockquote><h3>Deception Evaluation Process</h3><p>The MITRE ATT&amp;CK Evaluations team were challenged to design a methodology that would fairly evaluate the varied, unique range of Deception-based defenses while also articulating key differences in products and strategy.</p><p>Ultimately, the <a href="https://attackevals.mitre-engenuity.org/enterprise/apt29/">APT29 Enterprise Evaluation</a> framework was updated, revised, and applied. This emulation is based on the APT29 adversary, distinguished by its stealth, sophistication, arsenal of custom malware, and dynamic operational cadences influenced by perceived intelligence value and/or infection method of victims.<em> </em>The evaluation itself followed a two-phase process:</p><p>● <strong>Phase 1 — Observe:</strong> <strong>Will the adversary encounter the Deception? </strong>During this first phase of the evaluation, emulations were run with the goal of capturing what each Deception solution deployed when faced with adversary behavior. Phase 1 results offer a valuable indicator of the degree and circumstances in which each Deception is capable of impacting the adversary.</p><p>● <strong>Phase 2 — Engage: Does the Deception affect the adversary?</strong> During the second phase of the evaluation, emulations were repeated. This time, the adversary technique engaged with the Deception, and the outcomes of those interactions were recorded. Phase 2 data reveals more about how each Deception performs against a variety of adversarial behaviors.</p><h3>Deception Evaluation Results</h3><p>Deception products are not as straightforward to evaluate as detection-oriented security systems, and they are far more nuanced than detection products. Before comparing results and weighing participants’ solutions against one another, note again that the two products have fundamentally different approaches.</p><p>Srikant describes some of the key Deception capabilities within the SentinelOne suite.</p><blockquote>“SentinelOne delivers comprehensive detection coverage with Deception authenticity and scalability across all attack surfaces (on-premises, cloud, and remote sites). Additionally, it goes beyond decoys and breadcrumbs by adding concealment and misdirections to its offering. SentinelOne uniquely creates decoys for everything from Windows endpoints to medical devices and energy substations.</blockquote><blockquote>SentinelOne protects against Active Directory enumeration and ransomware attacks. It hides files, AD objects, folders, mapped network and cloud shares, and removable drives so intruders can’t find the data or access it to steal or encrypt. SentinelOne gathers and automatically correlates data from the attack, including memory forensics. Extensive third-party integrations and playbooks provide automated incident response and negate the need for additional resources to analyze and respond to an incident.”</blockquote><p>On the other hand, CounterCraft has a different approach, which David dives into.</p><blockquote>“The CounterCraft approach to Deception deployment and the core of our technology is to provide high-fidelity, actionable threat intelligence in real-time. Our fully instrumented Deception environments and stealth communication channels allow monitoring of attacker behaviour, detecting activity, providing enriched threat intel, and allowing the defender to manage the attack and protect their assets.”</blockquote><p>Assessing these different approaches is extremely valuable as you determine how to read through the results and determine which solution may fit your security program. For those of you who are familiar with the Enterprise Evaluation results, keep in mind that we do not have Detection Categories for these results. Instead, we created Deception Categories that are based on the terminology found in the <a href="https://engage.mitre.org/">MITRE Engage framework</a>. For these reasons, It is even more important than usual to assess the gaps you are trying to fill before you look to examine the results.</p><p>Full results are available on the <a href="https://attackevals.mitre-engenuity.org/trials-deceptions/apt29-deceptions/">ATT&amp;CK Evalutations website</a><em>. </em>Note that we do not assign scores, rankings, or ratings; instead we offer unbiased evaluation results so that other organizations may provide their own analysis and interpretation. <a href="https://medium.com/mitre-engenuity/how-to-leverage-att-ck-evaluations-results-announcing-our-first-ever-community-advisory-board-ab195a349c1b">Learn how to best leverage and interpret ATT&amp;CK Evaluations results.</a></p><h3>Deception Trials Program Takeaway</h3><p>One intention behind the Trials program is to provide fresh insights that will contribute to the ongoing innovation and evolution of threat-informed defense. From a research point of view, this project revealed the complexity involved with comparing inherently different Deceptions products. It’s therefore unlikely that Deception evaluations will be reliable at scale any time soon.</p><p>We hope that these Evaluations help bring the vendor ecosystem forward in their prioritization of features and functionality for addressing known adversary behavior. It is for this reason we are looking to diversify our overall Evaluations offering with inclusions of both <a href="https://attackevals.mitre-engenuity.org/enterprise/wizard-spider-sandworm/">products</a> and <a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services/">services</a>. Accordingly, we hope that the vendors who participate reap the benefits of collaborating with us on these research projects.</p><p>Srikant explains SentinelOne’s intent on participating in this round,</p><blockquote>“SentinelOne is an enthusiastic supporter of what MITRE does, bringing transparent and open evaluation methodologies to the security industry to help define and continually expand a common cybersecurity language that describes how adversaries operate.</blockquote><blockquote>SeninelOne had tremendous success with the ATT&amp;CK Enterprise Evaluations, and participating in all the evaluations has become an essential practice that we have used to improve our products further. This also provides SentinelOne an opportunity to participate and pioneer the inaugural ATT&amp;CK Deception Trials and showcase the platform’s security diversity and depth.</blockquote><blockquote>SentinelOne Deception products helped hundreds of customers in preventing privilege escalation, lateral movement, and collecting threat intelligence using deception and identity cloaking technologies. While many customers benefitted from SentinelOne technology, it was never documented publicly for others to see the benefits of deploying deception products.</blockquote><blockquote>Customers understand that MITRE [Engenuity] has a very stringent testing methodology, and they can trust the results more than any claims made by a vendor. It is also challenging for customers for themselves to perform comprehensive evaluations as MITRE [Engenuity] does. We believe the results will accelerate the deployment of deception technology and let customers select the right vendor to work with.”</blockquote><p>David further describes a desire to collaborate on further work to improve the testing methodology,</p><blockquote>“CounterCraft expectation was that the testing would be a collaborative process allowing all deception vendors who participated to define the testing strategy based on our experience combined with the experience of MITRE in the deception field. The testing process still requires work to achieve this, but we are keen to further our collaboration with any future work.</blockquote><blockquote>CounterCraft hopes that in the future the evaluation process will provide an independent analysis, on common ground from a known impartial source, to help identify how best to deploy deception. It should provide a benchmark to measure all deception platforms equally and highlight strong areas and help identify the consumers of the results which is the best tool for their specific deception deployment.”</blockquote><p>The takeaway for organizations comparing Deception solutions is to appraise each one on the basis of its unique use case or scope. Subjective analysis is required to determine whether any Deception solution is the best addition to the diversity and depth of current cyber protections.</p><h3>More from ATT&amp;CK Evaluations</h3><p>Thank you to the vendors who participated in the initial round of our ATT&amp;CK Evaluation Trials program, and to the collaborating MITRE Engenuity teams working to advance the state of threat-informed defense.</p><p>The MITRE Engenuity ATT&amp;CK Evaluations program is on a mission to make a safer world with a threat-informed defense approach to security. To learn more or get involved, visit the <a href="https://attackevals.mitre-engenuity.org/">ATT&amp;CK Evaluations website</a>. For more about MITRE Engenuity, a tech foundation for public good, follow us on <a href="https://medium.com/mitre-engenuity">Medium</a>, <a href="https://www.linkedin.com/company/mitre-engenuity/">LinkedIn</a>, or <a href="https://twitter.com/MITREengenuity">Twitter</a>.</p><p>© 2022 MITRE Engenuity LLC. Approved for Public Release. Document number AT0033</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f15d7a1f4d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/mitre-engenuity-att-ck-evaluations-results-from-deception-trials-9f15d7a1f4d3">MITRE Engenuity ATT&amp;CKⓇ Evaluations Results from Deception Trials</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ATT&CK Evaluations: Managed Services Welcomes 17 Participants]]></title>
            <link>https://medium.com/mitre-engenuity/att-ck-evaluations-managed-services-welcomes-17-participants-3719cdc30f28?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/3719cdc30f28</guid>
            <category><![CDATA[managed-services]]></category>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Fri, 15 Apr 2022 16:27:20 GMT</pubDate>
            <atom:updated>2022-04-15T16:27:20.476Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*xxkL6WnSEwtCIV1BVvLNWA.png" /></figure><p>Off the back of our big <a href="https://attackevals.mitre-engenuity.org/enterprise/participants/?adversaries=wizard-spider-sandworm">release</a> at the end of March, we are excited to announce our cohort for our first ever ATT&amp;CK Evaluations: Managed Services. Thank you to each of the following vendors for participating in our inaugural <a href="https://attackevals.mitre-engenuity.org/managed-services/managed-services/">ATT&amp;CK Evaluation: Managed Services</a> — Atos, Bitdefender, BlackBerry, BlueVoyant, Critical Start, CrowdStrike, Cybereason, Microsoft, NVISO, OpenText, Palo Alto Networks, Rapid7, Red Canary, SentinelOne, Sophos, TrendMicro, and WithSecure (formerly known as F-Secure).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bACyWeqI1w_C8TSUaMEl0g.png" /></figure><p>As described in a <a href="https://medium.com/mitre-engenuity/att-ck-evaluations-for-managed-services-d8c60fda118b">previous post</a>, the goal for this Evaluation is to apply many of the same principles of our ATT&amp;CK Evaluation Enterprise to a new suite of offerings in the Managed Services space. While our traditional enterprise evaluations aim to evaluate technology, evaluations for managed services aim to evaluate the people of the security vendor space. We believe that a good security program is comprised of people, process, and technology, especially with regards to implementing a threat informed defense.</p><p>ATT&amp;CK Evaluations: Managed Services will employ a closed book version of adversary emulation, whereby the vendor participants will not know the emulated adversary until after the execution is complete, though it will be based upon publicly available threat intelligence. The emulation will be conducted in the Microsoft Azure Cloud, similar to the ATTACK Evaluations: Enterprise. Our team will execute the emulation, and participants will provide their analysis as if we were a standard customer. The evaluation will be focused entirely on understanding adversary activity, so remediation or prevention are both prohibited in this inaugural evaluation. During a post-mortem purple team exercise, we will disclose the adversary emulated, all behavior performed, and disclose how our team mapped participant provided analysis to that behavior. We will work with participants to enhance their detection capability during this period, as participants are encouraged to ask questions regarding the execution.</p><p>© 2022 MITRE Engenuity LLC. Approved for public release. Document number AT0030</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3719cdc30f28" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/att-ck-evaluations-managed-services-welcomes-17-participants-3719cdc30f28">ATT&amp;CK Evaluations: Managed Services Welcomes 17 Participants</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Leverage ATT&CKⓇ Evaluations Results + Announcing Our First Ever Community Advisory Board]]></title>
            <link>https://medium.com/mitre-engenuity/how-to-leverage-att-ck-evaluations-results-announcing-our-first-ever-community-advisory-board-ab195a349c1b?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab195a349c1b</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <category><![CDATA[enterprise]]></category>
            <category><![CDATA[product-evaluations]]></category>
            <category><![CDATA[evaluation]]></category>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Thu, 14 Apr 2022 21:58:34 GMT</pubDate>
            <atom:updated>2022-04-14T22:00:54.712Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/401/0*xmgcnanE3ATed-I3.png" /></figure><h3><strong>Introduction</strong></h3><p>Now that the dust is settling from the results we recently published for the <a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/wizard_spider">Wizard Spider</a> and <a href="https://github.com/center-for-threat-informed-defense/adversary_emulation_library/tree/master/sandworm">Sandworm</a> Enterprise Evaluation round, we wanted to take the opportunity to provide more insight into how to analyze the data we’ve published. Though the emulation and results analysis processes are standardized to minimize variability between each evaluation, each participant can choose portions of the test based on their product’s capabilities. For this scenario, vendors had the option to include or exclude steps involving Linux hosts and had an optional final day of testing related to protections in addition to the base model of detections.</p><p>By definition, we are not a product category (EDR, XDR, etc.) evaluation offering. We are a capability (Detections, Protections, etc.) evaluation offering. We help answer the question: what are the end-users getting from these products? Efforts in the market to homogenize our results and plot them over graphs and other visuals are not surprising, but they can be ineffective at making meaningful and actionable conclusions from the raw data. The question still remains: <strong>as an end-user who is looking to make purchasing decisions, how do I wade through the interpretations from vendor participants to find the tool that is right for my organization?</strong> It is the purpose of this post to answer that question and offer some direct advice towards the end of this post from the team who conducted the Evaluations themselves.</p><h3><strong>Some Background</strong></h3><p>As a precursor to making use out of the rest of this blog post, I would suggest reading the following content first:</p><ol><li><a href="https://medium.com/mitre-attack/dissecting-a-detection-part-1-19fd8f00266c">Dissecting a Detection: An Analysis of ATT&amp;CK Evaluations Data (Sources) Part 1 of 2</a></li><li><a href="https://medium.com/mitre-attack/dissecting-a-detection-part-2-74844e86e2b0">Actionable Detections: An Analysis of ATT&amp;CK Evaluations Data Part 2 of 2</a></li><li><a href="https://medium.com/mitre-engenuity/making-sense-of-att-ck-evaluations-data-42ca844940b9">Making Sense of ATT&amp;CK Evaluations Data: Who Really Won and How to Avoid Common Pitfalls</a></li></ol><p>Following the framework presented by those pieces should immediately help answer the question posed above, but here we will explore and extend these ideas with more recent context.</p><h3><strong>ATT&amp;CK Evaluations: Our Philosophy</strong></h3><p>For the community at large, here are a few guiding principles of our Evaluation:</p><ol><li>MITRE Engenuity will never rank vendors in the ATT&amp;CK Evaluations or stack them up against each other because we believe that an in-depth analysis is far more effective than simply looking at “scores” or “rankings.” There is no such thing as a one-size fits all security posture, and that is even more true when constructing a threat-informed defense that is going to actually defend you from any of the adversaries we emulate.</li><li>There are no “winners” in the ATT&amp;CK Evaluations, and vendor participants have been given guidance to avoid making such claims to that effect. Our team has extended the courtesy of reviewing content before publishing. If you happen across an interpretation of the data that implies a “winner” or “ranking,” it has not been endorsed, approved, or confirmed by our team.</li><li>We do not encourage “apples to oranges” comparisons of our Evaluations results that compare our Detection and Protection results. Though the emulation and results themselves are structured and non-variable, there are countless nuances that arise from both the opt-in nature of the Evaluation and differences in the various product suites. Aggregating them into a uniform analysis or ranking may be a flawed take, and this is especially true because the Protections Evaluation and participation in Linux were opt-in, and not required by MITRE Engenuity. These delineations are clearly observable in the data we published, and consumers of our data are encouraged to leverage our <a href="https://attackevals.mitre-engenuity.org/enterprise/participant-comparison/?adversary=wizard-spider-sandworm&amp;participants=&amp;cards_open=false">Vendor Comparison Tool</a> to compare achievements between various vendors.</li></ol><p>In the spirit of those guidelines, our team has reiterated the above to specific vendors, especially in cases of messaging that misrepresented the testing and data. Our guidance was for those vendors to remove content from websites, webinars, and other marketing materials. Our intent is to help the cyber community better understand the Evaluation and what the results imply. We want to apologize to the vendors who participated in this test and are dealing with a negative fallout from this marketing hype. We want to also say thank you to the 25+ vendors who followed our guidance and accurately represented their own results.</p><h3><strong>Some Pointers from the Team That Collected the Data</strong></h3><p>The reason I joined the Evaluations team at the beginning of this round was twofold. First, I am a MITRE ATT&amp;CK® Framework fanatic. The only way to truly defend against adversaries and protect data and provide for privacy is to formulate a strategic security program that holistically covers all corners of your attack surface. In my opinion, the MITRE ATT&amp;CK Framework is the best way to do that, especially because it exists prevalently throughout features/functionality in security products. Therefore, the Framework works well to translate the operational aspect of security operations to the business problems that provide the funding for those operations. Second, I wanted to be a part of the incredible team responsible for planning, executing, and improving ATT&amp;CK Evaluations. The complexity of running consistent and objective evaluations across 30 vendors is astounding, and the team responsible for conducting the evaluations should be held in the highest regard. They should also be deferred to when looking at results. To use the oldest marketing cliché in the book, don’t take my word for it, here are what the experts say about leveraging the publicly available results.</p><p>Some of our external stakeholders — both on the end-user and vendor sides — believe that the ATT&amp;CK Evaluations team should be selecting the “best product” from the evaluation. In no uncertain terms, that is an impossible endeavor. Any attempt to “pick a winner” takes away from the core intent of why we conduct these evaluations in the first place. As Jamie Williams, Principal Adversary Emulation Engineer, describes,</p><blockquote>“If we could responsibly tell you what product(s) were best for you, we would. The answer to that question starts with understanding your needs, and Evals provides one valuable input towards finding that answer.”</blockquote><p>The end goal of our evaluation is to be one (of hopefully many) data points to building a threat-informed defense that works for your organization specifically.</p><p>As an end-user, what do I need to add to my security program? The existential answer to this question starts with understanding that different organizations require different strategies to protect against adversaries. For instance, a smaller regional bank in the southwest region of the United States may face a completely different set of threats than a large tech company in the Bay Area, even if they are targeted by the same adversary. Even if they do face the same threats, their teams may be comprised differently, not to mention one is more likely than the other to have a build vs. buy strategy in their security program. All of these traits, and many more to even attempt to list out, are the reasons that it is important to dive into the data to understand how the results are relevant for your specific strategy. These factors are also true in the vendor participants’ team makeup. We do not restrict the headcount or expertise of the folks that the team members bring to the evaluation. Brent Harrell, Lead Offensive Security Engineer, explains,</p><blockquote>“Organizations leveraging ATT&amp;CK Evaluations data should understand that, in some cases, the high detection rates were achieved by a team of product experts and developers hunting for specific events across multiple days. It’s my opinion that the real value of the data lies in the types of data sources leveraged by the products rather than the raw detection rates in the test environment. Comparing how the various products do their job lets organizations find the one that best fills a gap in their defenses, be it an intuitive user interface for a small team to triage events logs or API hooking for deeper protection.”</blockquote><p>So to explicitly answer the question above, you should determine what to add to your security program by diving deeper into the data to see what fits your specific needs, especially focusing on the data sources listed in the results.</p><p>As a former product manager of security products, I can attest to how a great user experience for any security solution leads to more efficient and effective outcomes. As a community, we have the benefit of some of the best UI/UX in the enterprise solution space. One of the most unique aspects of our ATT&amp;CK E valuations is that we have a deep collection of all the screens captured during our evaluation. If you care about the adversaries we emulated, you should care equally about the way you and your team would experience defense against the adversary leveraging the vendor’s product suite. Jake Spinney, Senior Offensive Security Engineer, explains,</p><blockquote>“MITRE Engenuity doesn’t assign or endorse detection or prevention scores for ATT&amp;CK Evaluations because this data alone is insufficient for making a determination on which product is best for your team. In my opinion, the best data from Evals are the screenshots. These paint a much bigger picture for security teams on how these products look and function during an attack by an emulated, real-world threat: What does an event for this substep look like? Does it make sense? Does it provide enough or too much information to act on? Can your team effectively use this product to stop threats in your network? No percentage can answer those questions in meaningful ways to any individual organization. Coming from a consulting background prior to joining the Evals team, scoring these metrics would be like grading art at a museum.”</blockquote><p>To extend the metaphor slightly, as curators at our museum, our goal is to showcase the art, not editorialize by giving our opinion on the screenshots we’ve captured.</p><p>To view both the data sources and screenshots for any specific vendor, follow these steps:</p><ol><li>Navigate to the <a href="https://attackevals.mitre-engenuity.org/enterprise/wizard-spider-sandworm/">Evaluation Overview Page</a>.</li><li>Navigate to a vendor participant’s Overview Page by clicking a logo. You will see a header that says [Vendor] Overview at the top right of the page. This page is a consolidated view where you can find results for all Evaluation Rounds for which the vendor has participated.</li><li>Select the “Wizard Spider+ Sandworm (2022)” link. If you’d like to drill down into results from previous rounds, please click that round in this step.</li><li>The page you are taken to will allow you to toggle between the results for different Evaluation Rounds and each Scenario within that round. If an option is grayed out, the vendor did not participate in that round or portion of the Evaluation.</li><li>The following is just some of the content you should focus on:<br>a) Overarching screenshots that pertain to the Scenario at large. These serve as a fantastic way to get a big picture view of how a tool looks and feels, as applicable to the specific Scenario.<br>b) Deeper information that pertains to specific sub-steps within the Evaluation. To view this information:<br> i. Scroll down to the table under the overarching screenshots.<br> ii. Hover over the “Detection Type” or “Detection Note” areas for the sub-step (1.A.1, 1.A.2, etc) you’d like to drill down into.<br> iii. When you click the drilldown, a modal will pop-up to the right of the table that will show the Criteria, Data Sources, and individual screenshots relevant to that sub-step.</li></ol><p>We also view this as a fantastic opportunity to pull the community forward as a whole. Reductions of our results to mere percentages take away from our opportunity to learn from these Evaluations and understand behavioral activity of the adversaries who always seem to be a step ahead of us. To that end, we really hope that the community looks into our emulation plan, step by step, to better inform their defensive postures. Lex Crumpton, Lead Cybersecurity Engineer — SOC &amp; Blue Team, explains,</p><blockquote>“ATT&amp;CK Evaluations is an excellent way to test a vendor’s defensive posture and our team’s understanding of adversarial activity. Just like in a real cyber-attack scenario, no one solution is perfect. We as a cybersecurity community learn as a collective on how to improve. Not only should you look at a holistic picture, but you should also pay attention to the details each step in this process provides. There is always something new to learn on how to better improve ourselves for the betterment of the community.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*f56PsNlwfbpqgLErgBfmMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*xs_qGnPjkCCGYqRqDysZlA.png" /></figure><p>In conclusion, it’s important to remember that our industry would disappear altogether if there was a “best product” out there. If there was a one-size fits all solution, breaches would disappear, and we’d all happily be out of a job. As far as security operations is concerned, we like to emphasize that we want better alerts, not necessarily more alerts. “Better” is subjective, which is why we believe that these evaluations are so valuable. Brendan Manning, Senior Cybersecurity Engineer, explains,</p><blockquote>“We encourage you to dig deeper than the upfront numbers and also utilize resources such as data sources and screenshots to help determine what product(s) are best suited for your organization. All organizations have varying capabilities and needs. No product is perfect or ‘the best’ for every organization.”</blockquote><p>Overall, we are in this fight together and it is our responsibility to go the extra mile to use every avenue we can to fight against the bad guys. We hope that the ATT&amp;CK Evaluations data is one of those avenues in your path to build an effective defense. Again, we encourage everybody to use the <a href="https://attackevals.mitre-engenuity.org/enterprise/participant-comparison/?adversary=wizard-spider-sandworm&amp;participants=&amp;cards_open=false">Vendor Comparison Tool</a> to do side-by-side comparison between vendors to use raw results for their own analysis.</p><h3><strong>What’s Next?</strong></h3><p>As I described in my last post, we have a number of exciting and new offerings being released in the coming quarters. You’ll see the Trials Evaluations: Deception results at the end of the month and an announcement about our Managed Services Evaluations vendor participants sooner than that. Both are net new offerings which we are excited to bring to the market, and we are eager to continue to innovate and provide value to the information security community.</p><p>We are also announcing our first ever Community Advisory Board. Our commitment to innovation and consistency requires deeper interaction in a more structured format with the community we are looking to benefit. Our goal is to learn how we can make the data we present more accessible to the community at large. Though there are guardrails we need to operate within to maintain objectivity, consistency, and substantive value, there are likely some improvements we can implement to give the community better access to our data. In true product management fashion, we’d like to gather those requirements from the folks who benefit from our offering: the infosec community at large. The only stipulations to joining this Advisory Board is that you have a role in information security and are not currently employed by one of our vendor participants. If you are interested in joining our Community Advisory Board, please fill out this <a href="https://share.hsforms.com/1Ugtpx-MrRDKOY1uDM67K9Q4m7ji">form</a>. Based on interest, we will be exploring different strategies to gather feedback and deliver solutions that fulfill the requirements we gather. Remember, we are ALL in this together and our team at MITRE Engenuity is committed to continue to deliver on cybersecurity solutions in the interest of public good.</p><p>© 2022 MITRE Engenuity LLC. Approved for public release. Document number AT0031</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab195a349c1b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/how-to-leverage-att-ck-evaluations-results-announcing-our-first-ever-community-advisory-board-ab195a349c1b">How to Leverage ATT&amp;CKⓇ Evaluations Results + Announcing Our First Ever Community Advisory Board</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wizard Spider and Sandworm ATT&CK Evaluation Results: Data Encrypted For Impact (T1486)]]></title>
            <link>https://medium.com/mitre-engenuity/wizard-spider-and-sandworm-att-ck-evaluation-results-data-encrypted-for-impact-t1486-b4cf47a2d5ca?source=rss-c34c9e212bb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4cf47a2d5ca</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[product-evaluations]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <dc:creator><![CDATA[Ashwin Radhakrishnan]]></dc:creator>
            <pubDate>Thu, 31 Mar 2022 20:27:07 GMT</pubDate>
            <atom:updated>2022-04-14T06:16:32.021Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*5l0SPb2wyejQMj03-uGcXA.png" /></figure><p>The 2022 ATT&amp;CK® Evaluations for Enterprise <a href="https://attackevals.mitre-engenuity.org/enterprise/participants/?adversaries=wizard-spider-sandworm">results</a> are now available on the ATT&amp;CK Evaluations <a href="https://attackevals.mitre-engenuity.org/enterprise/wizard-spider-and-sandworm/">website</a>. This evaluation emulated behaviors inspired by Wizard Spider and Sandworm. A total of 30 vendors participated in this round, which represents an increase from 29 participants from last year’s Carbanak and FIN7 Evaluation. Notably, we experienced a significant uptick in vendors who participated in Protections. While there were 17 Protections participants for the Carbanak and FIN7 round, there were 23 Protections participants in this round. As always, we’ve been extremely fortunate to have collaborated with such a wide range of vendors for this research project. It is clear to us from the results that our participants have grown in the past year and are as excited as us to prioritize threat-informed defense capabilities within their product suite.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWj1MCZmnUiGUYP8TxcxBQ.png" /></figure><p>With the participants in mind, let’s explore the adversaries we emulated in this round. <a href="https://attack.mitre.org/groups/G0102/">Wizard Spider</a> is a financially motivated criminal group that has been conducting ransomware campaigns since August 2018 against a variety of organizations, ranging from major corporations to hospitals. <a href="https://attack.mitre.org/groups/G0034/">Sandworm</a> is a destructive Russian threat group that is known for carrying out notable attacks such as the 2015 and 2016 targeting of Ukrainian electrical companies and 2017’s NotPetya attacks. These two threat actors were chosen based on their complexity, relevancy to the market, and how well MITRE Engenuity’s staff can fittingly emulate the adversary.</p><p>For any first-time viewers of our Evaluations, the easiest way to start getting value from our data is to peruse the results of each vendor participant. With any complicated emulation plan comes an equally complicated set of collected data. The results can be intimidating at first glance, but the best resource to get started can be found <a href="https://attackevals.mitre-engenuity.org/using-attack-evaluations">here</a>. Additionally, you can view some <a href="https://medium.com/mitre-engenuity/making-sense-of-att-ck-evaluations-data-42ca844940b9">tips and tricks</a> to start leveraging the data more effectively.</p><p>For those already eager to sign up for Round 5, we will be opening the Call for Participation in the coming months. For those keeping track, this year also marks the first year we are undertaking 2 new Evaluation types, both of which are already in flight. Our first ever Trials Evaluations will yield similar data for vendors who feature <a href="https://medium.com/mitre-engenuity/introducing-att-ck-evaluations-trials-7f482cf4fa74">deception</a> capabilities. The final results will be published on April 30 in a format that is relevant to the purple team exercise we conducted for that round. Furthermore, we’ve officially closed the cohort for our first ever <a href="https://medium.com/mitre-engenuity/att-ck-evaluations-for-managed-services-d8c60fda118b">Managed Services Evaluations</a>, which we are excited to release more about in the coming weeks.</p><p>I also want to highlight the tireless efforts of the team at MITRE Engenuity, who came together to deliver an innovative iteration of an already fantastic Evaluation offering. Again, we would like to thank the 30 vendors who participated in our research project. It is the joint collaboration that makes our Evaluation truly unique and substantive to the information security community at large. We’ve received overwhelmingly positive support from our vendors, and we’d like to highlight some key points as shared by our participants:</p><p>“We are delighted with the outcome as it clearly reflects our ceaseless effort to develop better technologies for our customers. By continuously leveraging the ATT&amp;CK Evaluation as an opportunity to test our security capability against advanced threats, we will keep enhancing our security platform and threat response system in accordance with the global standard.” — <strong>Eric Jun, CTO, AhnLab</strong></p><p>“In a market filled with over-hyped claims, Bitdefender is deeply committed to validating our capabilities through independent third-party testing. We consider the MITRE Engenuity ATT&amp;CK Evaluations as a “gold standard” benchmark for real-world cybersecurity assessments. Our ongoing participation in MITRE Engenuity’s rigorous tests using tactics, techniques and procedures of sophisticated groups like Sandworm and Wizard Spider, demonstrates our continued innovation and excellence in threat prevention, detection and response.” — <strong>Dragos Gavrilut, Director, Cyber Threat Intelligence Lab, Bitdefender</strong></p><p>“With the attack surface rapidly changing and becoming increasingly complex, organizations must prepare to mitigate various threats from sophisticated threat actors. BlackBerry is committed to mapping to the MITRE ATT&amp;CK framework, providing customers with increased confidence in discovering gaps in processes and defense tools and enhancing overall network defense and protection. We believe that third-party testing is essential to help vendors and end-users better understand their product’s capabilities, and BlackBerry’s vision of stopping tomorrow’s threats, today.” — <strong>Billy Ho, Executive Vice President, BlackBerry Spark Group Products</strong></p><p>“Sophistication and frequency of attacks increased dramatically over the past year, reaching new peaks. In this reality, organizations should adopt a threat-informed security strategy. MITRE Engenuity ATT&amp;CK® Evaluations help them achieve that by evaluating cybersecurity solutions’ ability to defend against real-world cyberattacks and threat groups. Endpoint security plays a crucial role in protecting the hyper distributed workspace. The latest ATT&amp;CK® Evaluations results highlight Harmony Endpoint leadership once again, for the 2nd consecutive year, as industry-leading threat detection and full attack visibility capabilities. Harmony Endpoint Customers get all the endpoint protection they need against all imminent threats like ransomware, malware, phishing while enjoying robust detection and response capabilities at the best TCO.” — <strong>Ofir Israel, Vice President of Threat Prevention, Check Point Software Technologies</strong></p><p>“Achieving 100% prevention in the fourth round of the MITRE Engenuity ATT&amp;CK Evaluation shows the power of the Falcon platform, which was designed to enable organizations to take a unified approach in detecting and preventing attacks across the endpoint, cloud, identity and data. CrowdStrike is setting the industry standard with a cloud-native security platform that is designed to deliver the most robust protections and stop the most sophisticated threats.” — <strong>Michael Sentonas, Chief Technology Officer, CrowdStrike</strong></p><p>“These results validate the superior detection capabilities that Cybereason delivers against the most complex attack sequences. The ATT&amp;CK framework is the go-to standard for assessing solution efficacy today, and we are proud of both our outstanding performance in all four years of the evaluations, and of our ongoing collaboration with MITRE CTID to further improve detection based on the most subtle of attacker behaviors. This is how we begin to defend forward as a community, stop relying so much on reactive approaches and take the fight to the adversary through behavior-oriented predictive response.” — <strong>Lior Div, CEO and Co-Founder, Cybereason</strong></p><p>“CyCraft MDR automates investigations and alert verifications, providing SOC teams with the ground truth to an attack faster and more hands-off than existing solutions. Our fast, accurate, simple, thorough AI-driven approach to cybersecurity has led CyCraft to manage millions of endpoints from both government and enterprise customers in over a dozen countries worldwide.” — <strong>Benson Wu, Founder and CEO, CyCraft</strong></p><p>“Cynet was pleased to participate in the MITRE ATT&amp;CK Evaluation for the second time. This testing process helps us ensure that we continue to provide maximum protection to our customers. Threat actors are rapidly evolving their tactics, and defenders must make continuous improvement in cybersecurity protections to remain a step ahead. Throughout the process, we had the opportunity to evaluate our XDR platform in combination with our detections. The feedback we receive from the MITRE ATT&amp;CK evaluation has been invaluable to help guide the evolution of our platform.” — <strong>Aviad Hasnis, CTO, Cynet</strong></p><p>“Deep Instinct is thrilled to have participated, for the first time, in this year’s round of the MITRE Engenuity’s ATT&amp;CK Evaluations emulating advanced threat groups. We are pleased to have demonstrated our deep learning powered, prevention-first approach by achieving excellent results in all tested prevention scenarios and providing analytic-level insight across almost all of our detection coverage. These evaluations provide Deep Instinct and others in the industry with valuable insight needed to continuously improve our offerings and prevent organizations from succumbing to prolific and destructive threats.” — <strong>Shimon N. Oren, VP Research &amp; Deep Learning, Deep Instinct.</strong></p><p>“The MITRE Engenuity ATT&amp;CK evaluation is an objective and definitive test to assess vendor detections and protections against adversary tradecraft. These tests are invaluable to the security community for their transparency in confirming vendors’ security practices. We are thrilled to have participated in MITRE ATT&amp;CK evaluations every year since their inception.” — <strong>Paul Ewing, Threat Researcher and Principal Product Manager, Elastic</strong></p><p>“ESET believes in taking a multi-layered, high performance approach to developing our detection technologies. ESET Inspect is the foundation of our extended detection and response (XDR) capabilities and works together with ESET PROTECT to offer a complete security solution that is optimized for ease of use. We have been tracking Sandworm since its inception, being the first to identify the work of its subgroups BlackEnergy and TeleBots and to discover the origin of the NotPetya outbreak. For us, it’s critical to keep ahead of the curve with our telemetry and put our solutions to the test through the expert lens of the MITRE Engenuity team.” — <strong>Roman Kováč, Chief Research Officer, ESET</strong></p><p>“Cyber security buyers struggle to understand the quality of the products they buy. Security testing is hard, so MITRE Engenuity must be applauded for the objective, realistic tests they perform. We’ve taken part in every ATT&amp;CK Enterprise Evaluation to date, which reflects our commitment to being transparent to our customers. Our results confirm why none of our customers has suffered a serious loss from a cyber-attack.”<strong> — Leszek Tasiemski, Head of Products for Business Security, WithSecure</strong></p><p>“As cybersecurity threats increase in both number and sophistication, it is essential for organizations to strengthen their prevention and response capabilities. Malwarebytes’ Endpoint Detection and Response platform includes the powerful tools that individuals and enterprises alike need in order to ensure an effective proactive security posture. The results of our ATT&amp;CK evaluations are testament to our leadership and continued growth in this space.” — <strong>Barry Mainz, Chief Operating Officer, Malwarebytes</strong></p><p>“Achieving 100% prevention and protection in the fourth round of the MITRE Engenuity ATT&amp;CK Evaluation shows the power and maturity of Microsoft 365 Defender, which delivers a unified XDR approach to detecting and preventing ransomware attacks spanning multi-platform endpoints, identities, applications, and multi-cloud environments” — <strong>Rob Lefferts, Corporate Vice President, M365 Security, Microsoft</strong></p><p>“MITRE Engenuity results are the best measure of security product effectiveness for today’s threats and an important vendor evaluation criteria for customers. Our performance is a testament to the continuing innovation we bring to Cortex XDR and proof of our ability to provide customers with outstanding protection. We value the threat-informed approach MITRE takes that helps drive the industry forward, making it a safer, more secure world.” — <strong>Gonen Fink, Senior Vice President, Cortex Products, Palo Alto Networks</strong></p><p>“The MITRE ATT&amp;CK framework has become the industry standard that provides common language around adversarial behaviors — ultimately allowing security teams to create and implement a top-notch threat informed defense. As a contributor to ATT&amp;CK, Qualys is thrilled to have participated in the 2022 MITRE ATT&amp;CK Evaluations. Just one year following the launch of Qualys’ Multi-Vector EDR, the company is proud to stand shoulder to shoulder with some of the industry’s best. Our solution demonstrates the dedication and collaboration of our engineering and research teams, and their overall mission to deliver visibility, protection against threats, and provide detection of malicious behaviors to our customers.” — <strong>Travis</strong> <strong>Smith</strong>, <strong>Vice</strong> <strong>President</strong>, <strong>Malware</strong> <strong>Threat</strong> <strong>Research</strong>, <strong>Qualys</strong></p><p>“MITRE’s ATT&amp;CK framework and evaluations continue to elevate threat-informed defense planning, discussions, and response across our industry. Rapid7 appreciates the opportunity to participate in this evaluation; it’s one very important way we learn and get better. We are excited to give our customers and the security community greater visibility into our endpoint capabilities and strengthened signal-to-noise that is possible with InsightIDR.” — <strong>Sam Adams, Vice President of Detection and Response, Rapid7</strong></p><p>“ReaQta applauds and appreciates the work of MITRE Engenuity ATT&amp;CK evaluations in helping organizations to make transparent and informed decisions that can help them become more cyber resilient. This is the 3rd time ReaQta has participated in the evaluations and we are always impressed by the professionalism and dedication of the MITRE Engenuity team.” — <strong>Alberto Pelliccione, CEO, ReaQta, an IBM company</strong></p><p>“One of the biggest cybersecurity challenges for organizations is buying and implementing the right tool to protect the organization while empowering its security teams. MITRE Engenuity ATT&amp;CK is one of the most objective and definitive tests to measure EDR and XDR capability, bringing much-needed visibility in helping practitioners sort through a complex vendor landscape. SentinelOne is proud to partner with the MITRE community, actively contributing to adversary behavior research so that blue teams can better detect, protect against, and ultimately prevent these kinds of sophisticated attacks.” — <strong>Jared Phipps, Senior Vice President, Worldwide Sales Engineering, SentinelOne</strong></p><p>“SOMMA considers secure visibility, threat behavior detection with data analysis and its root cause tracking as the most valuable in cyber breach circumstances. The MITRE ATT&amp;CK evaluation is quite effective to evaluate and prove our value and aim. We are convinced that MITRE Engenuity is doing great things to counteract effectively in today’s evolved cyber threat environment.” — <strong>Mr. Yong-Hwan Roh, CEO and Founder, SOMMA, Inc.</strong></p><p>“The challenge of protecting your organization against real-world cyberthreats is a demanding effort in precision and scale. Providing the highest level of context to a defender is key to improving the speed at which you can identify and respond to attacker tactics, techniques, and procedures (TTPs). Adversaries continuously adapt and evolve their toolsets and activity to seize new opportunities, evade detection and try to stay one step ahead of security teams. For that reason, we’re proud to be recognised in the MITRE Engenuity evaluations, which focus on the actual TTPs of two assertive and modern-day attackers.” — <strong>Joe Levy, Chief Technology and Product Officer, Sophos</strong></p><p>“We are constantly innovating to stay ahead of attackers, and the results in this latest evaluation demonstrate our success in doing so. We believe participation in these evaluations, as well as contributions to the MITRE ATT&amp;CK frameworks, help us and our customers in stopping complex attacks. We are committed to mapping to the ATT&amp;CK framework throughout our solutions so SOC analysts can better understand the tactics and techniques being targeted in their environments.” — <strong>Wendy Moore, Vice President of Product Marketing, Trend Micro</strong></p><p>“MITRE’s rigorous testing reflects the type of threats that our customers are seeing in the real world, where there are no redos. With highly effective detection and prevention, our results demonstrate the power of VMware’s comprehensive endpoint and network visibility, which not only works out of the box, but also works the first time.” — <strong>Scott Lundgren, Chief Technology Officer, VMware Security Business Unit</strong></p><p>“Ransomware is a growing scourge for all types of organizations and the focus of these MITRE Engenuity ATT&amp;CK evaluations could not come at a more appropriate time. Security teams can use these evaluation results to identify gaps in their detection coverage. Our strong performance in both the Windows and Linux portions of the evaluation demonstrate how Uptycs helps these Security teams to detect even advanced ransomware actors, in addition to the hardening needed to minimize the risk of ransomware in the first place.” — <strong>Ganesh Pai, Co-Founder and CEO, Uptycs</strong></p><p>© 2022 MITRE Engenuity LLC. Approved for Public Release. Document number AT0029</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4cf47a2d5ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitre-engenuity/wizard-spider-and-sandworm-att-ck-evaluation-results-data-encrypted-for-impact-t1486-b4cf47a2d5ca">Wizard Spider and Sandworm ATT&amp;CK Evaluation Results: Data Encrypted For Impact (T1486)</a> was originally published in <a href="https://medium.com/mitre-engenuity">Center for Threat-Informed Defense</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>