This blog written by Peter Havens
The results of the fifth round of the MITRE Engenuity ATT&CK Evaluations have been released, and while every vendor is claiming 100% of everything, the truth is in the details, and the data does not lie. This year, Cortex is the only vendor with 100% prevention and analytic detection.
In the ever-evolving landscape of cybersecurity, staying one step ahead has never been more critical. Adversaries continuously seek new and sophisticated ways to breach defenses. It's a high-stakes game where the rules constantly change, and the consequences of falling behind can be catastrophic.
This is where MITRE Engenuity enters the scene with their Enterprise ATT&CK Evaluations, offering a beacon of clarity amidst the chaos. These evaluations have become an invaluable resource for CISOs, security professionals and anyone tasked with safeguarding an organization's digital assets. They provide a litmus test or a performance review of how the industry’s endpoint security solutions hold up in the face of the most resourceful cyber saboteurs.
But, what exactly are the MITRE Engenuity ATT&CK evaluations, and why should you care about the results? These annual evaluations emulate the tactics, techniques and procedures (TTPs) employed by some of our time's most active and notorious threat groups. The MITRE Engenuity red team invites cybersecurity solution providers to defend against carefully choreographed attacks (in both detect-only and prevent modes) to provide insight into three capabilities:
A staggering number of vendors participate in these evaluations – 29 this year – which is a tribute to their value and reflects the painstaking efforts that MITRE Engenuity pours into the assessments to ensure they provide a challenging and provocative engagement.
This year marks the fifth annual evaluation, and the MITRE Engenuity red team focused on emulating the methods of Turla, a threat group our Unit 42 threat researchers have studied extensively. Turla is an extraordinarily well-funded and sophisticated Russian-based threat group that has infected victims in over 45 countries. They have targeted government agencies, military groups, diplomatic missions, as well as research and media organizations. Turla’s infamy stems from its covert exfiltration tactics, including water holing of government websites, custom rootkits, elaborate command-and-control network infrastructure and deception tactics. In speaking with defenders who participated, it is clear that MITRE Engenuity took a great leap forward in the sophistication of their attack methods this year.
The blue team deployed the Cortex XDR Pro for Endpoint agent on both Windows and Linux endpoints. No additional solutions were deployed, and Cortex XDR was configured with default settings as it would be out of the box, with the only changes enabling the quarantining of malicious files and, for Linux, enabling the option to treat grayware as malware.
This year's evaluation was broken down into two detection-only scenarios, named Carbon and Snake, corresponding to notable tools created and used by Turla. A protection phase followed this, mirroring the techniques in the detection tests, with enough entropy injected not to look identical to the detection test. The two detection scenarios each had 10 steps, consisting of multiple substeps that map to actual techniques in the MITRE ATT&CK framework. Combined, there were 143 substeps that each vendor had the opportunity to see. For each of these substeps, the MITRE Engenuity team recorded whether each solution had a detection for the action taken.
Each of these detections was categorized based on the detection quality observed.
MITRE Engenuity Detection Categories:
The MITRE Engenuity effectively identifies two types of coverage: Telemetry Coverage and “Analytic Coverage.” Telemetry Coverage is defined as the number of substeps where a solution produces a Telemetry detection as its highest value detection. Analytic Coverage is defined as the number of substeps that contain either a General, Tactic or Technique detection.
It is also worth noting that all detections might have one of two detection modifiers:
For the Protections scenario, there were 129 substeps organized into 13 major steps. In the case of these substeps. Each Protection substep was either blocked or not blocked and recorded as one of the following:
The purpose of these evaluations is to provide insight into three capabilities:
Cortex XDR stands alone in providing 100% Protection while delivering 100% Visibility and 100% Analytic Coverage (detections) with zero configuration changes or delayed detections.
In addition, the quality of our detections is unparalleled, with 142 of 143 detections as Technique level detections – the highest-quality detection possible. The one other detection was recognized as a Tactic level detection. Every one of the 129 substeps in the Protection evaluation was blocked. All of this was accomplished with zero “configuration changes” and zero “delayed detections.” In fact, if we exclude detections resulting from a configuration change, Cortex XDR was the only vendor with no missed detections (detection type, None). In other words, Cortex XDR was the only one with 100% visibility.
The results from this year’s evaluation serve to reflect the tremendous amount of effort that Palo Alto Networks continues to pour into both adversarial research and engineering in endpoint security, putting that knowledge to work to help our customers remain safe in the face of an increasingly hostile cyber world.
If you’ve been following the ATT&CK evaluations for some time, 100% might seem commonplace to you. But don't get confused; when other solutions claim to have achieved 100% in this year's evaluation, they have at least one detection and/or prevention in all the major steps. Only Cortex provided these detections for every one of the individual malicious actions that were taken – the substeps.
100% detection and prevention is the bar we should all strive for. As Forrester’s Allie Mellen pointed out after last year’s evaluation, “detecting an attack inspired by a known threat actor should be the floor for what security products ...” We completely agree, and Cortex XDR is setting the bar.
That said, Allie also rightly pointed out that perhaps 100% detection isn’t necessarily a good thing as solutions that “are detecting on every technique, they have the potential to be noisy, have a high rate of false pos...” We also share this perspective, and this is why we heavily leverage machine learning to understand which activities represent normal operations and which are potentially malicious actions. In the case of Cortex XDR, this is only strengthened by the fact that we integrate additional telemetry from many other key data sources that critical context clues to help determine where you should focus your investigations.
Hats off to the MITRE Engenuity, as this year was significantly more challenging than years past. In addition, we are thrilled with the improvements they have made to help defenders make informed decisions. The ATT&CK evaluation results, hosted on MITRE Engenuity’s site, now allow you to compare the results for three vendors side by side. You can choose each of the three evaluation scenarios and up to three vendors. By default, the results are shown with delayed detections and those resulting from configuration changes included. Still, you can also view the results with detections with those modifiers removed.
We’ve heard for many years that it is difficult to understand the ATT&CK evaluation results as no data visualization tool allows you to view the results across all vendors. To aid in that effort, we’ve created a tool to help you explore this year’s results, as well as all previous years. With this tool, you can select the vendors you are interested in and the particular adversaries that have been emulated. All of the resulting charts are built directly from the JSON files provided by MITRE Engenuity for each participating vendor. Explore the ATT&CK Evaluation results for yourself and let the data reveal the best choice for defending your business.
