Cutting to the chase, this blog discusses a new Capability Maturity Model to help guide and report on your Red Team’s maturity. As leadership, you can enable the Red Team’s maturation, or reap larger benefits, in a few key areas. We believe those to be the following: Processes
Red teams provide pivotal data for organizations to understand how their security posture would withstand an attack from a motivated adversary. But how does the organization know the data generated by their Red Team is as effective as it could be? How does the Red Team itself understand where it is relative to other threat actors and where it needs to go, and how does it communicate that status to leadership to build further trust and confidence? For many fields, a Capability Maturity Model (CMM) has been one answer to address these questions. Starting in the software engineering field, a CMM describes behaviors and processes across five levels of maturity, moving from unplanned/ad hoc activities through defined and documented practices and ultimately practices that are consistently being tracked and updated with lessons learned. That sounds good, so how can a Red Team get involved, you ask? Great question! The problem, as we’ll discuss in a moment, is that the Red Team field lacked a standardized CMM that teams could use for these purposes. The next few sections provide context on the why and how this CMM came to be. If you want to skip straight to information on the model and considerations for using it, click here. If you want to go to the model itself, click here.
As we set out on our own internal measurement and planning, we came across a published CMM from some folks from Norton Life Lock and Bishop Fox (you can find it here). The CMM consists of four categories (Processes, Technology, People, and Program) in keeping with other CMMs, but diverges by only addressing three levels of maturity. Through later discussions with one of the authors, they shared that they decided to focus on simplicity with their CMM to enable more use and avoid some of the inherent ambiguity of our field. Their offering filled a critical need where no other option existed. The authors captured many key areas that demonstrate Red Team maturity and presented it in a compact and easy-to-digest format that was able to get us started. We definitely want to thank Jordan and Noah Potti and Trevin Edgeworth for putting a stake in the ground when none existed, despite some of the challenges we’re about to discuss! As we began to the use the simplified CMM, we ran into two primary issues due to its structure:
As we headed into our internal summit, we decided to take on creating a standard-format CMM as one of our summit tasks with two purposes in mind: give ourselves more of what we needed to plan and report on our team’s health and create a resource that other members of the community who may run into the same issues could use.
After drafting the CMM during our summit, we sought feedback from the community to ensure we didn’t fall prey to biases of prior experience. After four, 2-hour sessions of excellent conversation with three industry partners (see the contributors section below) who brought a wealth of different experiences to the table, we ended with a CMM that contains four categories (Processes, Technology, People, and Program) and 27 subjects spread across the categories. If you want to skip straight to information on the model and considerations for using it, click here. And as a reminder, if you want to go to the model itself, click here. Otherwise, let’s talk nerdy for a moment on what we did, how we did it, and what challenges we faced.
Our first considerations were to address the challenges we faced with the simplified CMM. To provide ourselves with a “common language” to use with leadership and other teams, we expanded the CMM to the traditional five levels: 1 – Initial 2 – Repeatable 3 – Defined 4 – Managed (Capable) 5 – Optimizing This change allows users of the model to accurately report on current status or future plans without needing a translation layer. To address the issue of simplified categories, our draft started by inventorying the key pillars of Red Team work that we believed needed to have the full spectrum of maturity levels. To do this, we promoted some of the individual items from the simplified CMM to full subjects and filled remaining gaps with new offerings. As an example, the simplified CMM has a subject for “Team” that includes things like operational capability and work management; both of those became subjects of their own in the standardized CMM. Last, to keep consistency through drafting, we aligned descriptors for the different levels to other known CMM models. These descriptors are high-level phrases that generally describe behavior at a given level; things like “documented” or “automated”.
Obviously that figure of speech isn’t true, but we did face our own, new challenges during drafting. As it pertains to the level descriptors, while we wanted to keep standardization with other known and accepted models, we quickly found that the list did not account for everything we needed to describe Red Team maturity, such as operational security considerations. So, we added additional descriptors that kept with the spirit of the levels but provided us more wiggle room. These descriptors are critical to ensuring consistency across the 27 subjects; we’ll cover them in the section on the CMM itself a little later. Another typical feature of CMMs is that they are “additive” in nature, meaning that as you progress from one level to the next, you’re adding capability to your existing toolset and generally keep doing the lower level in addition to the new stuff. That doesn’t apply to everything we wanted to cover, particularly in the Technology category where some “level ups” replace the lower level. In these cases, what we really needed was a sliding scale of maturity rather than an additive model, so some of the subjects follow this flow rather than the traditional additive approach. Our biggest challenge, though, and one that still warrants review by more eyes over time, was keeping the number of subjects reasonable while not falling into the same pit of consolidation we were trying to avoid. There is undoubtedly overlap that could potentially lead to consolidation in the subjects that made the cut, but our core number of subjects did not fluctuate much beyond the 25-28 range. One of our guiding principles for this endeavor was if a subject did not have more than three relatively obvious levels of maturity, it might be a candidate for consolidation if there was something close to it.
Before digging in, there are a few assumptions to be aware of to best understand and utilize the material. Assumption: This CMM is currently written for internal Red Teams. Consultancies would need to modify some of the subjects and maturity levels to match their situation. For example, a consultancy doesn’t need to foster long-term relationships with various defensive or legal teams, but they would need to understand and plan for their maturity of maintaining client relations overall. Assumption: This CMM presumes you have staffed your Red Team with dedicated operators. If an organization only has a general Offensive Security team or perhaps a Red Team manager who coordinates third-party Red Team assessments, this CMM doesn’t apply. Assumption: Generally speaking, except for levels that describe a negative or some of the technology subjects noted below, the Red Team must exhibit the prior level’s behavior before it can progress to the next level. This is to keep with the additive nature of other CMMs as much as possible. In other words, you can’t skip over defining a process simply because you have metrics for it. The model also includes some definitions, such as potentially foreign terms and the descriptions of each level. Those can be found on the model page.
The model is broken into 4 categories and 27 subjects spanning elements that affect Red Team maturity. The model itself can be found here, this blog will only discuss it from a mid-level view and provide context that may help shape its usage, such as the intent of categories or particular things to be aware of when using a particular category or level within it. Processes Processes is one of the largest categories in the CMM, containing 9 of the subjects. The subjects within this category focus on a broad range of procedural elements that improve the team’s maturity, from maintaining a knowledge base to selecting and planning for operational targets. The subjects:
Technology While we believe red teaming is a mindset, in practice it does require tools. This category discusses how those tools are built and maintained, and how effective they are at meeting the team’s needs. The subjects:
People The People category includes relationships with key stakeholders outside the Red Team as well as the team’s internal capacity to accomplish its tasks. For consultancies, the People category is likely the area to focus most of your modification efforts. P.S. – if someone does take that on, we’d love to hear about it and perhaps offer an A/B version. The subjects:
Program Program focuses on the elements that make the Red Team part of the business rather than a mere collection of folks doing hackery things. The subjects:
Planning If this is your first time using a CMM, there are two things you should know about planning:
Reporting Teams can report on status and progress in three ways: a specific subject’s maturity level (goal or current); a category’s aggregate maturity level (goal or current); overall aggregate maturity level (goal or current) For category and overall scores, the score is an average of the contained subjects. If a team does not meet level one in any subject, it is still counted towards the average (i.e., in the total count of applicable subjects with a score of zero for that subject). Some teams may wish to weight certain categories or subjects heavier in their scoring. We provide no such recommendations or capability in the CMM spreadsheet as that will be highly specific to each team, but we’d love to hear how different teams have elected to do that. Modifications We tried to make this generalized, but we know there’s a broad range of needs that the Red Teams will need to fill for their respective organizations. First off, we welcome feedback. Please use our GitHub page to submit modifications that you think would be broadly useful to other teams, because this is only V1 we know things aren’t perfect. Secondly, if you plan to modify this CMM for your internal use, please read the challenges section above to avoid pitfalls that are waiting for you. The biggest item here is to ensure you stick to a common set of level descriptors so you keep consistency across levels, otherwise it is very easy (trust us) to accidentally create hard-mode and easy-mode subjects.
We want to offer a big thanks to the internal and external reviewers and contributors to version one: Andy Grant (Head of Offensive Security, Zoom), Johann Rehberger (Red Team Director, Electronic Arts), Matthew Bjornstad (Principal Security Engineer, Red Team, Charter Communications), Josh Huff (Red Team, Humana) Willa Riggins (Associate Director, Pentesting, Humana), Tim Straub (Threat and Vulnerability Management, Humana), Dion Baccus (Red Team Product Owner, Humana) Note: Individuals participated in their own right and none of the material should be considered indicative of their companies’ stance. If you would like to contribute: Please visit our GitHub page to review our contribution guidance submit suggestions and comments. We plan to update this model periodically and want it to be broadly applicable to the community.
The Red Team Capability Maturity Model is licensed under the Creative Commons Attribution-Non-Commercial-ShareAlike 4.0 International License. © Brent Harrell and Garet Stroup, 2022