Allie Mellen and Jeff Pollard navigate the nuances of meaningful metrics, highlighting their pivotal role in decision-making, cybersecurity growth, and career progression.
Guests:
Allie Mellen, Senior Analyst at Forrester [@forrester]
On Linkedin | https://www.linkedin.com/in/hackerxbella/
On Twitter | https://twitter.com/hackerxbella
Jeff Pollard, VP & Principal Analyst at Forrester [@forrester]
On LinkedIn | https://www.linkedin.com/in/jpollard96/
On Twitter | https://twitter.com/jeff_pollard2
____________________________
Host: Sean Martin, Co-Founder at ITSPmagazine [@ITSPmagazine] and Host of Redefining CyberSecurity Podcast [@RedefiningCyber]
On ITSPmagazine | https://www.itspmagazine.com/itspmagazine-podcast-radio-hosts/sean-martin
____________________________
This Episode’s Sponsors
Imperva | https://itspm.ag/imperva277117988
Pentera | https://itspm.ag/penteri67a
___________________________
Episode Notes
In this new episode of Redefining CyberSecurity with Sean Martin, Allie Mellen, and Jeff Pollard engage in an in-depth conversation exploring security metrics' critical role and power in the infoSec decision-making processes. Throughout the dialogue, listeners can gain an understanding of the importance of implementing relevant metrics, such as Mean Time To Detect (MTTD) and Mean Time To Respond (MTTR), for tracking growth within cybersecurity contexts. However, there’s much more to metrics than just these two figures.
Both Allie and Jeff emphasize that metrics should be perceived not merely as numerical values but as valuable guideposts aiding decision-making. This perspective, attributed to the Lean Startup philosophy by Eric Ries, encourages using metrics to guide future actions, understand current decisions, or evaluate past outcomes. They stress that metrics should have a genuine purpose and contribute meaningfully rather than just providing quantitative data.
Furthermore, the conversation underscores the relevance of metrics to the decision-making audience. Allie and Jeff agree that metrics should differentiate between what matters only to your team and what's necessary for strategic decisions in the broader organization. They become truly impactful by ensuring metrics support decision-making and reach the right audience, whether it's senior leadership, the security program, or the tactical metric practitioners.
Storytelling's role is highlighted as vital in presenting these metrics to various stakeholders, making the data more meaningful, understandable, and actionable. The conversation extends the notion of metrics, applying concepts like readmission rates, commonly used in healthcare, to measure incident recurrence in cybersecurity.
The trio also spotlights the need for a synergistic relationship between the Security Operations Center (SOC) and Vulnerability Risk Management (VRM). Such a relationship fosters improved security posture through effective incident management and prevention, with Allie reasoning that translating data into something meaningful for other business units is crucial.
Touching upon individual metrics in the context of career progression, both Allie and Jeff emphasize the necessity for individuals to define their career-oriented metrics based on their personal goals and organizational expectations. This understanding can help leaders prove their program's success and influence others.
The conversation ultimately underscores the importance of the right data sources for calculating meaningful metrics. Without the correct data, generating truly impactful and actionable metrics becomes impossible. Jeff cites an example of a financial organization that used a unique metric to measure insider risk, emphasizing the complexities and challenges of deriving meaningful and actionable cybersecurity metrics.
There’s a lot to unpack in this conversation. Listen to the entire episode so you don’t miss a beat.
____________________________
Watch this and other videos on ITSPmagazine's YouTube Channel
Redefining CyberSecurity Podcast with Sean Martin, CISSP playlist:
📺 https://www.youtube.com/playlist?list=PLnYu0psdcllS9aVGdiakVss9u7xgYDKYq
ITSPmagazine YouTube Channel:
📺 https://www.youtube.com/@itspmagazine
Be sure to share and subscribe!
____________________________
Resources
The Lean Startup: https://theleanstartup.com/
____________________________
To see and hear more Redefining CyberSecurity content on ITSPmagazine, visit:
https://www.itspmagazine.com/redefining-cybersecurity-podcast
Are you interested in sponsoring an ITSPmagazine Channel?
👉 https://www.itspmagazine.com/sponsor-the-itspmagazine-podcast-network
Please note that this transcript was created using AI technology and may contain inaccuracies or deviations from the original audio file. The transcript is provided for informational purposes only and should not be relied upon as a substitute for the original recording as errors may exist. At this time we provide it “as it is” and we hope it can be useful for our audience.
_________________________________________
voiceover00:15
Welcome to the intersection of technology, cybersecurity and society. Welcome to ITSPmagazine You're listening to a new redefining Security Podcast? Have you ever thought that we are selling cybersecurity insincerely buying it indiscriminately and deploying it ineffectively? Perhaps we are. So let's look at how we can organize a successful InfoSec program that integrates people process technology and culture to drive growth and protect business value. Knowledge is power. Now, more than ever.
sponsor message00:53
Imperva is the cybersecurity leader whose mission is to protect data and all paths to it with a suite of integrated application and data security solutions. Learn more@imperva.com.
voiceover01:10
And Tara, the leader in automation, security validation allows organizations to continuously test the integrity of all cybersecurity layers by emulating real world attacks at scale to pinpoint the exploitable vulnerabilities and prioritize remediation towards business impact. Learn more at pin tara.io.
Sean Martin 01:38
All right, here we go. This is Sean Martin, you're very welcome to new episode of redefining cybersecurity here on ITSPmagazine. Where I have the distinct pleasure trying to figure out how to operationalize security in the business not just to protect what it has, but to protect and enable growth as well. And even just protecting what you have can be difficult when you start thinking a little more broadly. And progressively, it gets even more challenging. The conversations at the below this level to make that happen. Become a little more complex. And so thankfully, I don't have to do that job. I just get to talk about it with people who have done it. Don't want to do it, talk, analyze it, help people in that role. So I get to pick their brain. And I'm thrilled to have ally and Jeff on today to help me have a conversation around security metrics. How do we measure? How do we measure? How do we know we're doing it? Right? What really matters. So not not a small topic. There's a lot to lots of cover there. We'll see where we go with this one. I'm excited to have this conversation. Before we get there, though, out of you've been on the show. But for those who haven't listened to those episodes yet, first go listen to them. And then to do a quick, quick intro to who ally is and what you're up to at the moment.
Allie Mellen03:07
Yeah, first off, thanks so much for having us super thrilled to be back. I'm Ally melon, I'm a senior analyst on the security and risk team at Forrester, I cover security operations. So all things people process technology in the sock, from a technology perspective, that's EDR X, Dr. Sims or security analytics, and then other is ransomware. mitre attack nation state threats. I've been working with Jeff quite a bit on security metrics specifically for the sock, but he is our resident metrics experts so glad that he was also able to join Jeff, do you want to do a quick intro?
Jeff Pollard03:42
Yeah, absolutely. So Jeff Pollard here, Vice President, Principal Analyst on the security risk team here at Forrester. One of the topics I cover and spend, actually most of my time on involves security metrics. And I love it. In fact, when I joined Forrester about seven and a half years ago, I said I wanted to cover security metrics, and everyone's slowly backed away. So maybe not the best way to introduce myself, I guess. But this is certainly a topic I'm passionate about Ali. And I get to spend a lot of a lot of time together talking about this in context of security operations and detection and response. But in terms of what I cover, I get to kind of bring it out beyond that, as well as talking about it through the context of conversations with the board, which also matters.
Sean Martin 04:20
And then the first thing I think of when I think of measuring a security program and more probably more specifically a sock, some some four letter acronyms come to mind straightaway. And I don't know if those are meaningful or novel. I'm sure we'll touch on those at some point. So let's first talk about I'm not going to spoil that. But what let's first talk about what are security metrics? Let's let's define what they are. I mean, who wants to wants to go first because it sounds like there's stuff for different departments obviously. Right then yeah, in So who wants to kick it off?
Jeff Pollard05:02
I'll go first. And I'll talk about how I talk about metrics in this context. And for me, it's sort of about not so much what metrics are, but what they're designed to do. And that's where I think you have to start with this topic. And so my number one rule when it comes to security metrics, and I take this from the lean startup and Eric Ries in particular, which is that a metric is only useful insofar as it leads to your next decision. That to me is what metrics are, what they should accomplish, what they do. And when they're right. That's what they enable you to do. Right metrics are data points, quantitative, and maybe even qualitative measures that allow you to understand either what decision you should make next, what decision you should make now, or give you the results of a decision you've made in the past and give you some insight into whether or not what you expected to happen, actually did happen. So metrics are really used to help you understand a decision or make a future decision, or a decision moment to moment. That's generally how I summarize it.
Allie Mellen06:01
And I really love this because it's particularly I find it very challenging when it comes to SOC metrics, because we often think about, okay, how is this translating up? But in reality, there's, there's a lot of different places where metrics can be useful with the way that Jeff has, has defined them. And that's, of course, at the level of what's happening in the sock as well. But what do you translate up what's actually being translated up, that's going to actually be useful for decision making, versus just reporting to report on something and reporting something that matters to you, but doesn't matter to anyone else in the org. So creating that differentiation, I think, is very important. And it comes back to what Jeff said about always leading with what's going to be helped you make a decision.
Jeff Pollard06:45
And that's the way and that's the secret to making metrics relevant is exactly that part. It's about sharing that metric, sharing that information with the person that is empowered to make that decision. If you are targeting your audience correctly, from a metrics perspective, then the people that you need the support have to make the decision you need to make or the audience for the metrics. So that could be a strategic metric, in the sense of the senior leadership team, executive leadership team, maybe even the board, it could be an operational metric one within the security program itself, you could be making decisions, or even a tactical metric, one that has more to do with kind of the day to day, you know, operational or practitioner oriented workflows, that that run the security program. But that's really kind of that other part of this, which is that it's not just the ability to make a decision. It's also that the audience that you're sharing metric with is the audience that makes that decision. And when you when you create that, when you have a metric that accomplishes both of those objectives, you ensure and kind of guarantee that they're relevant to the audience that you're speaking with. So there's not a lot of secret sauce to it, it's pretty straightforward. But what I find most of the time with metrics is that we settle for counts and quantities of things that our technology tells us about, and don't so much focus on the actual sort of outcomes or decisions that we want to make about those counts and quantities.
Sean Martin 08:06
It's interesting in that I love what we're saying already. And the thing that comes to mind for me is, well, why, who is who's producing the metrics? And for whom? And I can, I can see the easy path, like security teams, producing metrics up to their manager, I'm doing a good job. That goes up to the the line of business for security. Yes, our investments are producing good results that goes up to the executive staff. Yes, we're mitigating risk risk at the right level. And up to the board, we can we can talk about our ESG and how we fit into ESG and all the other stuff that they're currently caring about right? And then potentially back down, right? Well, here's how our department's doing doing we need to adjust your role in these ways. And perhaps even cross team, right? So element two SEC ops two, sock two in kind of those interactions, perhaps, is there a role for self metrics? And which raises the question for me of, well, who provides that person with the metrics so that they can know what this is? Here's what I'm doing. I need to make that decision based on what I'm doing in the moment. And how does that get once I get put together?
Jeff Pollard09:34
Yeah. So I think part of that I'll ally I'll, I want you to answer this in context of the sock and security operations, especially for practitioners. But often when I see that from a leadership perspective, whether it's a leader evaluating themselves, or maybe even their direct reports, that's where you do need strong coaching, strong talent development. You actually need something in place where you have more of a curriculum and succession planning so you understand the kind of career journey that personnel are on I'm because those are meaningful metrics. One of the one of the best things that I've ever heard from, from one security leader I talked to is that they kind of made liberal use of internal HR platforms, because they had to use those not because they wanted to, to be clear. But they made liberal use of these platforms to conduct surveys after a security practitioner was staffed on a project across teams, for example. So a security architect security practitioner, was assigned to work with the dev team, or the you know, the data analytics team, something like that, and provide some, you know, security expertise within that team. And when those projects would wrap up, that leader would send out a survey and effectively say, Hey, did did the security architect help you? What was this engagement? Like? Did they solve problems for you? Were they you know, did they act as a policy gate or as an enabler of success? Like, what was that indirect interaction like, and this leader did this, not just because they wanted to understand the sort of sentiment of what happens when security works with other parts of the organization, that was definitely part of it. But the other thing is leader wanted to understand is the recognition that often one of the ways when we're developing security talent that we struggle, is that security practitioners have boatloads of technical expertise, right? We are an expert oriented practice and profession. And often the thing that you have to develop with future leaders with more as you want to gain influence in the org is to develop those soft skills, what we often call soft skills, right? You know, kind of working with other groups, having some empathy, understanding shared objectives, and helping people accomplish their goals. So this was a mechanism for that leader, to then go and understand whether or not the people reporting to them needed some help with things like that, right? Was this person enabling you? Or were they an obstacle, so they would do a survey on that. And that was one way that they collected some of that not self feedback, but it was feedback about practitioners across teams. I think that when you get into to earlier stages, in careers, it's a little bit different. And I think when you get into some of the functional roles, like security operations, for example, that's where some of you know a lot of those metrics about the individual are super important, not just about the program, because you want to cultivate and enable your talent to progress the way they want to progress. And Ally. I know you've, you've talked about this a lot in your research. So I'll kind of hand things to you on that.
Allie Mellen12:14
Yeah, thank you. And I think it's an interesting question, because to me, it's very, there's a lot of different ways you could take that it depends a lot on what you want to get out of it. Like, one of the challenges that I see and that I've seen in a lot of the research that I've done on the security analyst role is that they're kind of on their own island, you know, they're kind of left to their own devices and a lot of ways. And that means that you have a lot of control over your own destiny, it also means that you are scrambling from the start to make sure that you're headed in the right direction. And you don't actually have the experience to know what the right direction is. So it's very challenging in that way. And so when I think about this question, I think about well, what do you actually want to do from from day one, like, what's your goal, if your goal is the leadership route, then the metrics that you choose to operate under are going to be leadership focused, and they're going to be leadership focus through the view of that organization. If it's technical, and it's much more individual contributor style, then they're going to be very different ones. But at the end of the day, especially at that individual level, when you're thinking about what you are looking for in the next step in your career, you need to be the one defining those metrics. Now, there's going to be hopefully, in an ideal scenario, someone who is your manager who is setting some type of metrics for you. I think that we've all been in situations where we haven't had somebody who has an idea of what success looks like in a particular role. Unfortunately, it's an unfortunate reality of business. But even in those circumstances, it's there's two sides of this coin, right? There's the side of what is your career? What do you want your career to look like in five years? And what are the metrics that you need to align around that? And then on the other side of the coin, what do you want your time at in this role in this opportunity look like. And that's where it has to be super aligned to the business objectives and to the the team objectives in particular, and to excelling in those. And there's a lot of synergies that typically come between those two, especially if you map it out early and continue to reevaluate over time. If you kind of go in and just go where the wind takes you, it's going to be much more difficult to align those and that's where you'll end up butting heads with either the system or what you want to do next in your career.
Sean Martin 14:30
So we're kind of very, very broad straw. I mean, I took it down to the individual person I know but it's still kind of broad stroke metrics here. And I'm wondering how is there a way to break it down into what matters for security programs? To Well, I guess what I'm trying to think of what to ask here, what what's important for them to make decisions on That would then define what metrics they should be looking to create.
Jeff Pollard15:06
Yeah, sure. Yeah. Yeah, no, I think you're exactly right. So one of the I think one of the reasons why metrics are so important for security programs, I'll give some examples of interesting ways that I've seen metrics used and some examples, but one of the reasons I think they're so important, is because often, within security, we find ourselves being forced to prove a negative, right, we weren't breached. And in some cases, security leaders are frankly, that superstitious, hesitant, painfully concerned about celebrating a victory, because we sort of live in this world where we assume breach, right? It's not when or if they only have to be successful once a lot of those things are just kind of, we're kind of conditioned to that, right. So there's almost this degree of superstition of, you know, don't talk about the fact that you're not breached, because that means you're definitely breach and you just don't know it. And I think metrics are incredibly helpful here to help security leaders, you know, a kind of brag about the program, right? Have a little bit of swagger that yeah, we've accomplished things. Yes, we've, we've been successful. And so what I often find in scenarios like that, is looking at things like security maturity, as a very basic starting point, I don't really consider maturity or maturity measure to be a metric per se. But it can give you a foundational sort of quantitative ish number on where the security program is. So that's kind of a starting point, hey, we're at this maturity here, we want to stay at that or get better. So this is going to be our journey over time. But I've also seen scenarios where metrics are used for security leaders to influence the rest of your I'll give you another example. Everyone knows about what is my least favorite metric? And I'll explain why the metric of are our systems patched or not. So you know, number of systems patched with an SLA often broken up by severity level, or sometimes broken up by severity level, often it just kind of systems patched or not, maybe not even that granular? Well, that, to me, is an excellent metric, if what you're interested in exploring is how much it and security hate each other. Because it deploys the patch in security tells them to deploy all the patches. So it's a great metric and figuring out how tired of security it is, right? Because they're not patching because they're the you know, they're they're not choosing not to patch. Because they just want to spite you probably, they're busy, and they can't patch everything. And it takes time. So this is one of those, to me, that's a metric that reflects disconnectedness between two two entities better than it shows anything about what you're doing. But there are some scenarios where you can make that a really powerful metric. And one example is to take that and look at it by business unit, by infrastructure, team, by region, by asset criticality, or by, you know, vulnerability severity, even where you can start benchmarking things and saying, hey, you know, it turns out that when I work with this department within it, we actually get critical volumes patched really, really quickly. But when I work with this other group within it, I don't and people aren't competitive. No one wants to be worse at this. So a lot of times what happens is, again, we go back to these counts and quantities, which which actually cause a problem for security leaders, especially when you look at things like incidents, for example, if your reporting counts and quantities of incidents, well, I have really bad news. When you get more budget and you add more controls, your incidents went up. So now you go in front of the board, and you say, Well, last quarter, we saw a million bad things. This quarter, we saw one and a half million, and that's better. The board's like, Wait, hold on CFO is like, wait a second, we gave you more money. You saw 500,000 more incidents, I thought you were going to see less things. It's like, oh, no, no, I finally saw them. Now, when I told you that we only had a million last time. It's because I did. Well, wait a second. Yeah, I don't know if I want to say that. Because suddenly you're saying you're admitting the last number I gave you. Well, that number wasn't real, it was just all we could see now that you gave me more money, I can give you the real number. So they wind up kind of trapping security. So often, what you need to do is take those those existing metrics you have add a numerator and a denominator. And that's what can make them compelling for the rest of the organization. It's when you don't do that, and you just report them kind of directly, that you lose out on a lot of the advantages. You could build things like comparing it by group where people are competitive. And suddenly, if you're benchmarking internally, everyone wants to patch a little bit faster. That to me is what starts to make a more compelling metric. Because the decision then is, well, do you want to patch slowly? Or do you want to patch quickly? You're deciding, like, you want to be as cool as the other group here that's got a bright, shiny green color, or do you want to be bad like the group that's got the, the bad red color, like you pick?
Allie Mellen19:36
And it also gives context, right? Because like, the original, like, if you just have the incident metric, then the board whoever's looking at that is like, Okay, but what does that mean? Like it's meaningless whereas once you incorporate these other aspects, you actually it starts to mean something, it starts to sound and look like for progress or backward progress or something tangible also
Jeff Pollard20:00
an ally. And I use this a lot in context of detection and response, one of the things that I find is take mean time to detect mean time to respond.
Sean Martin 20:08
I was gonna see if we get through with them. And oh, we can I mean,
Jeff Pollard20:11
just like the obligatory MTTB MTTR mentioned. Right? So
Sean Martin 20:14
those are there for four letter I didn't There you
Jeff Pollard20:17
go. Right. Well, so are these metrics good or bad? Should you have them? Yeah, probably. But there are ways to make that more interesting because these are good metrics in context of forecasting, headcount and capacity and throughput. For example, if your company is growing at a certain rate, if you have a certain number of employees, and you plan on hiring more, and you have a certain meantime to detect, respond, you know, to detect or respond today. Well, you know, if you add 10%, to the employee count of the enterprise, and that leads to 10% more incidents, if you start to understand that fraction of how these things relate to each other, well, you can suddenly take him TTD and MTTR, and say, Well, if we see a 10% increase in incident, then it's likely that we're going to see a 15% slowdown in meantime to respond because of throughput or bandwidth. That's a great headcount metric, it's a great operational metric, but you can also use it with, you know, that, that would CFO with HR with it, if you report into it, or even a CEO, if you're fortunate enough to report there to say, look, if we're going to grow the company, I have to grow our sock team. Otherwise, we are going to take things, it's going to take us longer to respond to things. And we've agreed as a group that we want to have an MTTR of 15 minutes, or one hour or two hours, et cetera. So there are ways to take some of those conventional ones that we use, like those four letter acronyms, you mentioned the MTT X's of the world and make them meaningful, but not by saying it takes us 15 minutes to detect like that. Okay, is that good or bad? I don't know that maybe it's great. But if you start to say, well, it takes us this long, but if we grow this much, we need to make some changes, we need to grow with a team, we need to work with an external partner, suddenly, that's more meaningful.
Sean Martin 22:04
And I often look to other other industries for inspiration for some of these things. And I know in the healthcare world, they put a big emphasis on readmission, if you can actually treat the problem properly, up as early as possible, and when you finally catch it, to get it to where you're not bringing in patient back, or keeping them in for extended period, it's kind of maybe the MTT X example as well. So one is easy, right? They don't come back, or they come back very few times or whatever. But how do we? How do we as a, as a group, have these conversations and with with the leadership, because a lot of what you just described Jeff, and Ali's kind of presenting the negative presenting that we gave us more and when we're doing or which increased numbers, which don't make sense. There's a lot of weirdness in this for people that don't understand. And then perhaps maybe not meaningful enough information for the for the teams that are doing it as well. So I do we have those conversations that both ends of the or both sides of the metrics?
Jeff Pollard23:22
Yeah, so. So I think on the readmission side alley, I want you to jump in with that one in a second. But I think that what you what you hit on, it's funny, the journey of this conversation is kind of how most of my inquiries go about this topic, which is, we start with a discussion on metrics. And I've kind of walked security leaders and their teams through this methodology of, you need to make a decision and kind of strategic operational tactical, leading lagging coincident, and sort of walk them through it and make we make up a few metrics. And then they say, okay, that's awesome. It's also super hard to do, right? And I'm like, Yeah, that's why I do this, I get to talk about it, you'd have to do it, but we can help. So So like you said, in the intro, Rayshawn. But then what happens is, then it turns into storytelling. Because when you start sharing this metric, whether it's inside the team, or outside the team, look, no one gets excited about data unless they already understand the problem, right? That like the only time you get excited about data is when you actually understand something, people get excited about stories. And that's when data can be evidence. But it matters to tell the story, just like that headcount, when of look, if we want to maximize, you know, our response time, if we want to keep that as low as possible. Well, the sock team has to grow at the rate of the enterprise maybe a little faster, maybe a little slower. But we've got to add that that's when you're starting to add context to that data point. And I think that, to me is what's powerful. When you when you add that storytelling element, that's where talking to the board, talking to senior leaders, even talking across the team, you know, those data points become sort of the color to your story, right? Those aren't the plot points, but they're the interesting things around the plot points as you start to tell that story. But I love your example of readmission because Ellie, I think there's like a huge tie into the sock there and what you look at into detection and response around topics like readmission rate.
Allie Mellen25:03
Yeah, no 100%, is it. And I mean, it comes back to operationalizing those metrics and not having them just be stagnant, right? Like, a good example of this is a just released a report with Eric nos, who covers vulnerability management, at Forrester. And it was all about the crossover between what the SOC does and where there's opportunities, synergies between the SOC and VRM. And in particular, a lot of it comes down to the metrics that you're passing between the two and the metrics that you're providing, particularly from the SOC to VRM, to say, hey, this vulnerability got exploited like 16 times in the past month, maybe we want to patch it might be useful for us might be helpful. Things like that are where you can start to reduce the readmission. And it's also an element where you can stop having security analysts feel like all they're doing is just closing tickets, and actually have them feel like they're doing something really valuable, because they're improving the security posture of the org, it's getting to that point of actually making things better instead of just being on the hamster wheel. And so that's a very important piece of this, but readmission, it can look like something like that, where you're actually stopping the incident, it can also look like something like during detection engineering and the work that you're doing there. And what are you doing to make sure that one of the detection fires, it's accurate? And what metrics do you have around making sure that that detection is becoming more accurate over time, or whether it's time to deprecate that detection, whether it's time to take a different route and look for ways other mitigating controls, other compensating controls, so that you don't have that detection anymore, whether that's something like increasing the priority with DRM or whatever, you know, there's a lot of different routes there. But having that understanding of the data, and then translating that into something that the other business units that you have to work with care about, that's what's really important here. And I think, coming back to Jeff's point with the story, it's not just about coming and saying, Hey, this is causing us to have a lot of trouble on this side, we're like facing this many incidents, because at the end of the day, that's not going to mean much to another team. But it's about saying, this should raise the likelihood of prioritization, because if this vulnerability gets exploited again, and we get popped because of it, you're gonna be a lot more trouble than I am. Because I've been closing this over and over and over again, I'm telling you about this over and over and over again. So there's a big opportunity to kind of move things off onto other teams, or at least make them aware that the work that they do contributes, and is part of this like mesh of effort to stop particular breaches from happening, it puts the sock in a very weird position, because ultimately, like we're just trying to stop anything that's gotten through everything else. But it's very pivotal because of that, that we feed back into the cycle of improvement for prevention, for protection for identification of vulnerabilities and help with the prioritization there. Because otherwise, it's missing a huge piece of what comes down to actually making sure that we're stopping attacks earlier.
Sean Martin 28:22
You're very close to something that I that I often often talk about in most of my episodes, because we ended up there somehow, some way. It's a very pollyannish view of, of this world where cyber says, you know, that machine, we keep patching and patching? I mean, that keeps getting popped and popped, are we detecting stuff on? Well, you know, what, it's critical to the business. And if we just change that operating system or that application, so we don't have to patch it. We can save the Aryan VRM teams time, we can save the sock team's time and perhaps even make the experience at that particular point in the workflow even better with an updated system. So I have this view that the security can actually change the way the business is architected. And the workflows are deployed on the systems that were responsible for protecting. Let's let's talk a bit about I don't know good percentages, you mentioned denominator or putting things in comparison to each other. How do we how do we get to the point where we can figure out what what is it we're trying to measure? I'm sure that's driven, obviously, as we've talked about, driven by the decisions we want to make the outcomes we're trying to achieve. But how do we how do we do that math?
Jeff Pollard29:54
Yeah, so it is math. And I think that I think first and foremost, one of the most important things to prove Did this is actually a pragmatic approach is to recognize that the data that you have dictates some of this, or all of it, frankly. So if you have data, then you can create metrics. If you don't have data, if you don't know where the data is, then you can't. And one of the things I often talk through is I have kind of this, it's a blurred spreadsheet, because it's from a real organization. And it says, you know, to prove that this is all real, this will exist somewhere. And it's a spreadsheet that takes the metric, the category, how it maps to NIST, or whatever framework you align with, although mostly NIST CSF, you know, kind of kind of rules the world, saying that, as you know, from from the USA, so high, but it was sort of sort of, we lean towards NIST, for the most part, it conquered most of the globe from a framework perspective, you know, aligning it to that, and then you've got your data sources, you've got your math and your equation, and then you've got notes on what this is measuring who you're using it with, and why you're doing it. And that does have to exist, there are great metrics out there that I tell organizations, look, I'll give you an example of an awesome metric. And I'm gonna tell you how you can't use it, and you can, and this isn't gonna work for you, because you don't have the things in place that would allow you to measure this, I'll give you an example of one of my favorite ones. One of my favorite metrics, and I got this from a financial organization about six and a half years ago, is that they, they effectively created a metric that gave them a leading indicator for insider risk. And what they did is they looked at their organization, they looked at the churn rate, so the retention rate of employees that had access to sensitive intellectual property now, right there, you've got to define what what does that mean, what a sensitive IP, what departments are those. And so what they effectively figured out is, look, we know an ally. And I have to know this, because our boss loves insider risk. It's what he covers. So he makes us care about it, too. So, so shout out to JB hopefully he listens. But, you know, it's sort of like if you have a lower retention rate of employee, so a higher churn rate, that means that it's likely that things are going to leave the building. We know that data leaves when employees are upset, disgruntled, changing jobs, etc. So if you have more of those things happening, there's more likelihood. So what they did is they created a metric that was churn rate based on employees with access to sensitive IP. Now, that metric is very difficult to create, first and foremost, because in almost every circumstance where I've had a client implemented, the HR team comm HR team comes back and says, No, you can't have that data. It's not your department, you don't get retention rates, then the security leader has to say, right, but I'm using it for this. And then HR says, Okay, you can't tell anyone on your team. And of course, CISOs like, of course, because they're often not the one putting together the metrics spreadsheet, right, they have a direct report. But it's compartmentalized information, right? This is considered to be dicey info to sort of broadcast across the enterprise. So the security leader does that. But let's talk about the decisions that you can make as a result of that. And these are the real decisions, the company that shared this with me wanted to make. So what they really wanted was to be able to go back to these departments and say we are overly permissive, we are giving way too many people way too much access. And while security owned a provisioning, meaning the actual allocation in entitlements, they didn't own the decisions for it right that the group decided who got access to what. So the first thing they wanted was to basically be able to say we should refactor how we get permissions, because we're just overly permissive. They were unsuccessful persuading business units to do that. Well, their fallback position was, well, we have you EBA, or security analytics, probably the better term today, capabilities. And we also have EDR, that could give us insight into information leaving, and the deployment is slowed down in various departments. So that was kind of the fallback position, which was, okay, you have a high churn rate, you have employees with significant access to sensitive intellectual property. And we have endpoints that are not covered enough. Well, that's what we're going to use to accelerate it. So then they add another component to this, which is they have coverage on the endpoint or not to detect insider threat, or they have an insider risk tool. So when they go in front of the steering committee and say this metric and talk about the fact that this is a problem, high churn rate, low coverage for security controls that could detect this and sensitive access to IP. Well, look, this is a little a little bit of a Mary Poppins view, to some extent, to your point about sort of Pollyanna, but what did they get from the from from the steering committee is okay, we need EDR to be accelerated in terms of deployment on these systems. There are no more excuses from end user compute that are acceptable period, we've already bought the tool. We're paying for the tool deployed right now, about a year and a half later, they actually did get the full refactoring of permissions and entitlements, which they wanted. But there's an example of where you can take and kind of a long one, sorry, but there's an example of where you can take some things you might have things like EDR coverage, things like access to IP and I am entitlements, and then leverage those things together in concert with other metrics may be outside of security, to then go to the steering committee and say, this is an unacceptable risk. So that's kind of how this process works. Now that's a that's a unique example. And again, that to me is like a level 400 metric, right? Like that's, you know, you're getting close to graduating college when you have that metric. You're not starting there when you're in 101. And tool one, you've got some other things to get there. But that's an example of where you can get when you start building it out this way.
Sean Martin 35:11
I love it not not too long at all, because it's real. Yeah,
Jeff Pollard35:16
yeah. It's not a simple one, right? It's a difficult one. But it's a powerful one. And, again, really what happened is, you know, keep in mind, in this scenario, security already bought the EDR tool, they already had the budget for that they weren't convincing anyone for that. They were literally just saying, hey, we need to deploy this and injuries or compute was coming back and saying, okay, but you have 35 agents on the endpoint, which, which also was a valid concern. That's Ally's problem to solve, because she also covers Dr. Flores, and I know she's ranting and fenders about that, about too many agents and things like that. But that was an example of where they were saying, Hey, I understand like, We're sorry, but at the same time, this is a real problem.
Sean Martin 35:51
Sally, I can take you two different directions. You're for my next thought, because one is who who defines what the metrics should be? Because in your scenario we've had you have I don't know who led that. But it sounds like a risk thing sounds like? Well, certainly insider risk was the driver, but security is involved there. HR is involved. I don't know what other datasets. So who drives that? That's one path we could go. The other is, are we too close to our own data? And should we have some, like a data engineering role that owns those relationships, and helps us security leaders kind of figure out how to get what we want out of it. Maybe they're connected? I don't know. Take take a path.
Allie Mellen36:42
Interesting. Well, Jeff, I want you to start by talking a little bit about like, because it sounds like security was what led that. And so for that specific example. I'd like to hear your point of view. And then I can talk a little bit about like, from the SOC metrics perspective, the way that I think about the different levels of abstraction and where where it's being led. So Jeff, do you want to start by tying a tying a bow on the example?
Jeff Pollard37:05
Yeah, so that was led by security? It was it was 100%. Within security, basically saying, Look, we have this problem. I know stuff can get out of the building. And it wasn't this, this wasn't their first start at this. Right. They had tried other things that weren't successful. But eventually they sort of stumbled into Okay, well, if I look at what causes insider threat, insider risk when stuff leaves, I know that turns a factor. I don't have visibility into that. So what are the ways I could figure it out? I don't even know if we have churned in those departments. Let me go ask I know, I have turned on the security team. But let me go check. And so that's kind of how they drove it. So it was security leading it and then ultimately, kind of finding a way to represent this data that that wound up being compelling, but took a while to get there.
Allie Mellen37:47
And then on the the SOC side, so it, this is actually very challenging. Like, this is a challenging operational problem. This is not like a simple thing, right? Because there's a lot of different levels here. So when I think about SOC metrics, more completely like one of the things that I talk a lot about is the importance of having a program manager who's managing some of these metrics and who, like they have other responsibilities as well. But one of the big things that they're doing is they actually understand operations, and they actually understand metrics. And they don't necessarily need to be someone with a security background. It's pretty cool if they are, because then they really start to get it. But ultimately, what they need is a mix of experience where they can they can work with, in this case, the security discipline, to understand what metrics are going to be important. And then to identify how to measure those metrics. Now, that's at that operational level of what's going well in the sock and what isn't. The the handoffs are really what's what's most important here, because they're the drivers for collection and presentation of these metrics, and operationalization, these metrics are going to be within the individual teams that are driving them. So you're gonna have the seaso coming to the scenes and saying, Hey, I need to have metrics around success. The SOC needs to be in the program manager or the SOC manager needs to be prepared with the right metrics there. They cannot be just taking whatever it is, whoever wants some information and saying, Okay, these are the metrics they asked for, these are the metrics they getting. They need to give information that's actually going to be useful for them. And I think that's really important is to not just accept what you're asked for. I mean, obviously, give them what they asked for, but give them things that are really useful if they're missing them too, and kind of take some take some creative freedom and creative license there because that's a representation of what your department is doing well or poorly And it's much more important for you to take ownership of that and to showcase that in the way that makes sense that you know is going to make sense for them. And it's gonna give them what they need. When you take this a level, a level below, like, I think the most important thing, and the thing that I see missing the most often is just actually having someone who is keeping track of these metrics, like that's what's really missing. Like when I think about a detection engineering function. As an example, one of the issues that I often run into is you get a bunch of people who are really excited to start building detections, but they're not actually testing them. They're not actually seeing if they're working, or how often they're firing, or whether they ever fire or ever fire again after the first time. Those are the types of things that you need to actually measure. And so at that point, it comes down to more of recognizing this as like an actual enterprise function as opposed to just something that people are doing because they know they need to do it. Jeff, I kind of definitely butchered that compared to
Jeff Pollard41:04
I think you're totally right. It's like, look, there's nothing like the best thing in the world is building a yaar rule. The worst thing in the world is figuring out if it matters, like sort of, it's just not exciting. It's like, I'd rather build another Yarra rule, because now I learned how to do it. So Ali, I think, I think you're exactly right. And I think your point about having someone attached to this, you know, metrics are a luxury problem. And I don't mean that they're a luxury problem in terms of money. That's kind of true, too. But they're actually a luxury problem in terms of time, everyone is busy. And there are a few people in the world. Again, don't judge me that want to sit around and think about this. Now I do. I talked to one security leader a few years ago, and they actually hired someone that had a marketing analytics background that could come in and work with Tableau work with Power BI and other things because they wanted like really, really snazzy metrics. And they just knew they were going to build themselves. And they basically said, I needed someone that worked in an industry to your point earlier, Sean about about that, that worked in an industry where they make decisions that are impactful in terms of money based on data. So they brought in someone for marketing that didn't know security, but had done marketing analytics, in various companies for looking at things like A B testing, which version of the website works better, et cetera, you know, money is made based on these things. And they said, that transform part of the metrics program, because now I've got this person that doesn't just do the metrics, they also do the visualizations. And they also help with the storytelling when I spread this internally. And I would say, Look, if you wind up in the fortunate position of having a boss that doesn't understand what you do every day, never get upset about that, right? There's nothing better than a boss that has no clue what you do, because then you can tell them what you do. And you can decide what they know if they're a decent person. Well, that doesn't always happen. But you know, vacuums get filled, right? So if you're the if you're given the chance to to establish the metrics that dictate if you're doing well or not, man, are you kidding me? That's fan that's the best place to be in the world. Like I worked two hours today. That's 50% More than I worked yesterday, or you know, things like that. Maybe you want to work on that one a little bit. But that's the point.
Sean Martin 43:04
Yeah, no, I love I love that you went to the visualization and the storytelling. I teach a security and analytics class at Pepperdine. And I really instill the idea of storytelling. And it's not just listing a percentage or a graph of percentages, it's trying to figure out what are you trying to show? What does the data tell you that will tell that story for you, in a way that gets somebody to your point, make that decision? That you Yeah, you think they should be making, right? So I love that. And I probably talk for another hour, just on on that whole thing. But
Jeff Pollard43:46
let's do another one, we'll do another one database and storytelling, how's that?
Sean Martin 43:49
This is fun. I became for that. So in, in closing, then anything you want to say, to maybe just kind of leave somebody with a thoughts of where to go, what to do next.
Jeff Pollard44:02
So for me, and I'll turn it over to Ali. Yeah, I'll turn it over to Ali after this one. But for me, it's it's very much a situation of, you know, basically, you know, metrics help make decisions. So figure out the decision that you want, and then start plugging away at how you can divide this up to make that decision. That the numbers you need.
Allie Mellen44:22
Yeah, and for me, and this sounds weird, but for me, it's an exercise in empathy, because the handoffs here are what's really, really important. You need to understand how other people are going to be using these metrics. When you're in a business context. Like you do your own metrics, great. But at the end of the day, you have to interface with other teams, and you need to give them the metrics that matter to them, not the ones that matter to you. And so that's a very, very important piece that I that I often see people struggle with is like one of the main issues with sock metrics is we're passing off metrics that like meantime to remediate Eat, and you get to a different department and they're like, I don't know, like radiation, so handing the metrics that they can actually use, and making sure that that handoff is is clear and concise is going to get you a lot farther and also get you in a very positive position with that team.
Sean Martin 45:18
Yeah, cuz your your reality is not necessarily what they're perceiving, either. Yeah. I love it. Well, this has been fantastic. I'm super grateful for the two of you to come on and join and and have this conversation together. I hope it's hope it's meaningful for the audience. It certainly was for me, and you're welcome back anytime happy to have the date of his one as well. So maybe we lined that up in a few weeks or something. Of course, for those listening, they know there'll be links in the show notes for for profiles for ally and Jeff and any resources they think might be useful. On this regard, research, you've done reports, articles, whatever. I know we miss mentioned NIST a few times if that's a relevant resource or not, but anything I think would be useful for folks will include in the notes for this. And of course, thanks, everybody for listening. Please share, subscribe, connect with Ali and Jeff. Find what you want to achieve and build your metrics or not. Thanks, everybody. Thanks so much.
voiceover46:32
Penn Terra, the leader in automation security validation allows organizations to continuously test the integrity of all cybersecurity layers by emulating real world attacks at scale to pinpoint the exploitable vulnerabilities and prioritize remediation towards business impact. Learn more at Penn terra.io.
sponsor message46:58
Imperva is the cybersecurity leader whose mission is to protect data and all paths to it with a suite of integrated application and data security solutions. Learn more@imperva.com
voiceover47:16
We hope you enjoyed this episode of redefining security podcast if you learn something new and this podcast made you think then share itspmagazine.com with your friends, family and colleagues. If you represent a company and wish to associate your brand with our conversations sponsor, one or more of our podcast channels, we hope you will come back for more stories and follow us on our journey. You can always find us at the intersection of technology, cybersecurity, and society