Join host Sean Martin and guest Gene Kim on the Redefining CyberSecurity Podcast as they explore the transformative concept of 'Shifting Left' in DevOps and its implications for information security.
Guest: Gene Kim, Author
On Linkedin | https://www.linkedin.com/in/realgenekim/
On Twitter | https://twitter.com/RealGeneKim
____________________________
Host: Sean Martin, Co-Founder at ITSPmagazine [@ITSPmagazine] and Host of Redefining CyberSecurity Podcast [@RedefiningCyber]
On ITSPmagazine | https://www.itspmagazine.com/itspmagazine-podcast-radio-hosts/sean-martin
____________________________
This Episode’s Sponsors
Imperva | https://itspm.ag/imperva277117988
Pentera | https://itspm.ag/penteri67a
___________________________
Episode Notes
In this episode of Redefining CyberSecurity on the ITSPmagazine Podcast Network, host Sean Martin engages in an insightful conversation with Gene Kim, co-author of "Wiring the Winning Organization". The discussion revolves around the transformative concept of 'Shifting Left' in DevOps, a strategy that has allowed tech giants like Amazon to achieve a staggering 136,000 deployments per day.
Kim likens this breakthrough to a collaborative effort between developers and operators, comparing it to the teamwork required to move a couch. He also explores the crucial role of information security in this process, underlining the necessity for security to equip developers with the tools to work independently, thereby serving as the first line of defense. Don't let security sit on the couch while you're trying to move it!
The conversation transitions into an exploration of the three mechanisms of performance: slowification, simplification, and amplification. Kim uses relatable real-life examples to elucidate these concepts, emphasizing the importance of timely and accurate information for effective decision-making and problem-solving. The more you know up front, the better off you'll be.
Drawing on his extensive work on the state of DevOps research, Kim discusses the predictors of high performance and how these principles apply to DevOps. He also points to the growing trend of specialization within DevOps and the emerging need for 'platform engineering,' a system that enables developers to focus on solving business problems while specialists handle the complex technical aspects.
This episode provides listeners with a deeper understanding of the evolution and future of DevOps, the importance of information security, and how these principles can be applied to enhance overall security programs. It also serves as an introduction to the Gene co-authored with Steven J. Spear. Be sure to listen to the podcast that Marco Ciappelli had with Spear on his Redefining Society Podcast.
About the book
Forget vision, grit, or culture. Wiring the Winning Organization reveals the hidden circuitry that drives organizational excellence.
Drawing on decades of meticulous research of high-performing organizations and cross-population surveys of tens of thousands of employees, award-winning authors Gene Kim and Dr. Steven J. Spear introduce a groundbreaking new theory of organizational management. Organizations win by using three mechanisms to slowify, simplify, and amplify, which systematically moves problem-solving from high-risk danger zones to low-risk winning zones.
Wiring the Winning Organization shines an investigative light on some of the most famous organizations, including Toyota, Amazon, Apple, and NASA, revealing how leaders create the social wiring that enables exceptional results.
This is not feel-good inspiration or armchair philosophy but a data-driven prescriptive playbook for creating excellence grounded in real-world results and proven theory. This is the rare business book that delivers concrete tools―not platitudes―to convert mediocrity into mastery.
____________________________
Watch this and other videos on ITSPmagazine's YouTube Channel
Redefining CyberSecurity Podcast with Sean Martin, CISSP playlist:
📺 https://www.youtube.com/playlist?list=PLnYu0psdcllS9aVGdiakVss9u7xgYDKYq
ITSPmagazine YouTube Channel:
📺 https://www.youtube.com/@itspmagazine
Be sure to share and subscribe!
____________________________
Resources
Wiring the Winning Organization: Liberating Our Collective Greatness through Slowification, Simplification, and Amplification (book): https://www.amazon.ca/Wiring-Winning-Organization-Simplification-Slowification/dp/1950508420
Google Leaked Memo "We Have No Moat (and Neither Does OpenAI)" through the Lens of Slowify, Simplify, Amplify: https://www.linkedin.com/pulse/google-leaked-memo-we-have-moat-neither-does-openai-through-gene-kim-0oghc/?trackingId=hPCsZXK8T8OhZVEe2Bz8Pg%3D%3D
Google "We Have No Moat, And Neither Does OpenAI": https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
Book | Wiring the Winning Organization: Liberating Our Collective Greatness through Slowification, Simplification, and Amplification | A Conversation with Author Steven J. Spear | Redefining Society with Marco Ciappelli: https://redefining-society-podcast.simplecast.com/episodes/book-wiring-the-winning-organization-liberating-our-collective-greatness-through-slowification-simplification-and-amplification-a-conversation-with-author-steven-j-spear-redefining-society-with-marco-ciappelli
____________________________
To see and hear more Redefining CyberSecurity content on ITSPmagazine, visit:
https://www.itspmagazine.com/redefining-cybersecurity-podcast
Are you interested in sponsoring an ITSPmagazine Channel?
👉 https://www.itspmagazine.com/sponsor-the-itspmagazine-podcast-network
Please note that this transcript was created using AI technology and may contain inaccuracies or deviations from the original audio file. The transcript is provided for informational purposes only and should not be relied upon as a substitute for the original recording, as errors may exist. At this time, we provide it “as it is,” and we hope it can be helpful for our audience.
_________________________________________
Sean Martin: [00:00:00] And hello, everybody. You are very welcome to a new episode of Redefining Cybersecurity here on ITSP Magazine Podcast Network. I am Sean Martin, your host of the show, where I get to talk about, I don't actually get to put fingers on keyboards too much, uh, in a practitioner, uh, sense. But, uh, I get to talk to a lot of people who do get to talk to and do this work out in the field.
Uh, with the ultimate goal of operationalizing technology and security in a way that. It not only protects the business value, but helps hopefully generate some value as well. I think that I do truly believe there's an opportunity for that. And, um, so today's conversation, uh, we're going to be, it's driven by a book that, uh, actually my co founder.
Already had a chat about it's called wiring the winning organization and there are two writers, two authors for this Steven J Spear who Marco had the chance [00:01:00] and pleasure of speaking with and Longtime friend and and counter for not counterpart for a peer I would say in the industry Gene Kim. Gene, it's a pleasure to have you on the show
Gene Kim: Sean, it's so good to be talking to you.
And, uh, we were talking before the show and I think it's been 20 plus years since we first met. Uh, it explains the sand
circles,
RSA circles. Yeah, exactly.
Sean Martin: Yeah, definitely. Uh, definitely conference circles. Uh, yeah, it goes back to, uh, a lot of mutual friends that, uh, That ran in circles with me at Symantec and CrossPaths, obviously, with the work you did at Tripwire, and, and, uh, it's great to, great to be connected with you still after all these years.
Gene Kim: So, a few days have passed, a few days have passed. Years and decades. That's amazing. Years and decades.
Sean Martin: Um, and, uh, Yeah, congrats on all the all the amazing things that you've done. And [00:02:00] I know one of your earlier books was definitely a huge hit and continues to be a resource for many folks. I continue to hear that.
So congratulations on that. We're going to be talking about your your current book.
That's
that's available to folks if I'm not mistaken. But before we get there, I'd like maybe for you to paint a picture of Your journey into infrasecurity, maybe touch on a bit, uh, about what Tripwire did and some of the things that were revolutionary in those days, um, as you, as you brought that organization and that team to life.
And, uh, and then from there, some of the other things you've been involved with, um, to, to bring us up to date, at least. And then, and then we'll, I suspect we'll use it to kind of figure out what has changed in our lives, uh, from a technology and threat perspective.
Gene Kim: For sure. [00:03:00] Yeah. In fact, maybe, uh, I can rewind, uh, uh, if let's start from the beginning, like how did I get into information security?
And, uh, uh, the answer. Starts when I was in high school. It was, uh, 1987. I was, uh, actually working part time as a system administrator at, uh, Prisma Supercomputers, which eventually got bought by, uh, Sun Microsystems. And, uh, this was, uh, around the time of the internet morse worm, uh, that hit on November 2nd, 1988.
And, uh, I think anyone who's doing sysadmin work, uh, or information security will remember that because 10 percent of all the servers on the internet were taken down by the system. Uh, remote execution vulnerability and it couldn't, uh, so I was applying for colleges and. Uh, the person who was doing the most amount of writing on this was, uh, Dr.
Gene Spafford at Purdue University, and so I ended up going there for my undergraduate, uh, did an independent study with him, and, uh, this is actually what, uh, eventually ran into, uh, a project that, uh, our goal was, alright, how do you sort of detect and ideally prevent these things from happening and enable quicker detection and recovery, [00:04:00] and, uh, that was really the, uh, the birth of the, uh, tool that eventually became Tripwire, and, uh, After I left Purdue, I went to University of Arizona where I got my master's degree in compilers and networking, and we created a startup to help commercialize Tripwire in 1997.
So, I was there for 13 years. I was the technical co founder, I was the CTO, and I left in 2010, right before, um, it was right after we finished our filing to go public. And even though Tripwire didn't make it to the public markets, I mean, it was just, I'm so grateful for that experience. And so after I, um, left Tripwire, I spent three years, uh, finishing a book called the Phoenix Project, which was a novel about a, uh, reluctant VP of operations who, uh, had to deal with, uh, terrible code that was never really tested before it got pushed into production, was never secured.
It was just a, um, it was modeled after one of my favorite books called The Goal. That was about [00:05:00] a, that was written in 1986. It was a novel about a manufacturing plant manager who had to, uh, uh, fix his cost and due date issues in 90 days. Otherwise it would shut the plant down . And so, um, uh, my co-author and I, for almost uh, 15 years, we wanted to write essentially the goal, but for the IT context.
And one of my favorite characters in that, uh, book is, uh, not the protagonist who's the kind of the, uh, VP of IT operations, but, uh, John, the Chief Information Security officer, uh, who was really kinda a caricature of myself. He was the shrill. Uh, hysterical person who is focused on technical malnutrition that everyone despised, ignored.
And, uh, um, you know, it has this kind of transformation that, uh, as he realizes that, you know, he needs to shift left, integrate into, uh, uh, the daily work of development QA operations, uh, and really help kind of advance the goals of the, uh, the organization, the business. Um, and so that came out in 2013, and that sold on almost a million copies since then.
And so I'm just so delighted that it's really been [00:06:00] Uh, adopted as a kind of like a banner of like, uh, around the DevOps community. And so that's been my field of study for, uh, at least 15 years. Uh, this really kind of a subset of the 23 year journey studying high performing technology organizations, you know, that had the best project due date performance development, that best operational reliability, stability, and ops.
But they also had the best posture of security and compliance. And so, That's a journey that took me into the DevOps movement. Uh, since 2014 I've been running a conference called the DevOps Enterprise Summit, um, studying not so much the tech giants, um, you know, the Facebook, Amazon, Netflix, Google, Microsoft, but instead large complex organizations that have been around for decades or centuries and showing that how they've been using those same principles and patterns and, uh, you know, using it to help their organizations win.
And we, we recently renamed it the Enterprise Technology Leadership. Uh, Summit, uh, because it's not just dev and ops, it's dev and ops and security and, you know, and the business and the product owners and, you know, things like generative AI, platform engineering, uh, you know, so it's [00:07:00] just same programming, but, uh, many people thought that we had sort of outgrown that name so that we've had, uh, 19 conferences, uh, over 1100 technology leaders across some of the best known industry verticals.
Um, and so just to wrap up, I mean, uh, what I've been working on for the last three years, is, uh, working on this book called Wiring the Winning Organization. My co author is Dr. Stephen Spear, uh, who, uh, pioneered the study of the Toyota production system. So he wrote the famous Decoding the DNA of the Toyota Production System.
Uh, that's the, uh, 1999 Harvard Business Review article, probably the most widely downloaded Harvard Business Review article of all time. And, uh, so he extended that work. Beyond just manufacturing, but to engine design and Pratt Whitney, to helping build a safety culture at Alcoa. And, uh, the quest we've been on for the last three and a half years is, you know, what is in common between agile and DevOps and safety culture and Toyota production system and lean.
And, uh, our conclusion is that there are all incomplete expressions of a [00:08:00] far greater whole. But it's actually one that's very simple to explain, and so I just, uh, it's been the most intellectually challenging thing I've ever worked on, but also one of the most rewarding because it can, I think, help us explain why does DevOps work?
Why do organizations work the way they do? Uh, how is it that, you know, what must information security do to truly integrate into the work of everyone? in the technology value stream. Anyway, Sean, how am I doing here so far? This is fantastic.
Sean Martin: I mean, my, it's fantastic. My, my brain's going a mile a minute here.
Um, yeah, because I, I think if we look back, it was networks and applications, right? And a few, few different operating systems and. And things have just blown up. I mean, you have multiple operating systems, some of them on devices local, a lot of them in what's called the cloud now. And there's abstraction layers and containers and virtualization and all kinds of, all kinds of stuff in there and APIs to connect them all together.
And all that [00:09:00] I just described is kind of the tech piece. And then you, you pointed out. There's also the culture, right? Yeah. And, and oh yeah. There's also the business. What are we doing this for? Not just because we can or because it's cool, right? Yeah. We're doing this for a reason and oh yeah. By the way, we need to have high quality and good performance and and and security, which connects to the other two.
To the other two as well. And even as you're describing your journey and as I'm. Kind of putting out a few words here, even in not in connected form, it's become really complex. Um, so I'm intrigued by how you approach this in a way, because you said it's simple. To me, it's complex having, having not, not recently been back in the day.
I used to build stuff, used to QA stuff, used to run teams, used to run products. And. Those [00:10:00] were challenging, right? Bringing a product to market, it's challenging. Small or large, challenging. Enterprise, super, super complex. Um, so how do you simplify what we're trying to talk about here?
Gene Kim: I love the, I love, there's a saying that says, uh, anyone can make something.
Simple appear complex. It takes a different problem entirely to take something complex and make it look simple. And I'll just make sure kind of three kind of huge aha moments I've had over the last two decades that really kind of in the last three years just came to searing aha moments. Uh, so one of them is, uh, the notion that, you know, in almost every domain, you know, whether it's software, whether it's manufacturing, whether it's engine design, whether it's, uh, uh, in military, uh, operations, uh, one of the key problems is managing the, is managing the socio part of the socio technical system.
And so, uh, um, you know, and there's [00:11:00] actually so much in common. In fact, I think what's exciting about this work with, uh, Dr. Stephen Spear is we're saying the same sort of engineering sensibilities that we use designed and manage the technical parts of the system, we can also use to manage and understand the socio parts of the socio technical system.
They call that sort of layer three. Layer one is You know, the object being worked on, whether it's the code, uh, the application, uh, the, uh, part being assembled, whether it's the, uh, engine, uh, being designed. Uh, layer two is the tools that we use, but layer three is the social circuitry. And, uh, we use the word circuit very deliberately because a circuit in an electrical system, in hydraulics, is always taking something in abundance in one place and delivering it to where it's needed.
Um, and so, you know, in the ideal, everyone has what they need, uh, when they need it, they're talking to who they need to, they get at the right time, the right place in the right format. In the not ideal, and this is so much of what went wrong in the Phoenix project, is no one has what they need, right? Uh, they have to, uh, scavenge [00:12:00] and forage for what they need to get information rights, approvals, decision rights, uh, requirements, code, license keys.
You know, um, you know, no one can get anything done. And even when they do, it's not at the right time, in the right place, in the right format. And so. Uh, and that is all a function of, uh, the social circuitry. Uh, the, the second, I think really big insight for me was that I learned that managing hospitals. Um, was actually much, much simpler 70 years ago, because there were essentially two functional specialties.
You had the doctors and the nurses, and there was really no technology. Um, and compare that to now, you have scores of functional specialties just in doctors, right? Let alone nurses, supply chains, uh, you know, uh, you know, pharmacy, et cetera, right? In fact, the radiology department is actually now four separate technologies, which all have the layer two and layer one work.
I mean, so the amount of communication coordination Uh, is, is so sadly, right, often outstrips the ability for the layer three social circuitry to actually work, which is why [00:13:00] it's so awful often to be in a, uh, healthcare setting. Um, and so you take a look at technology, like, uh, you know, before, maybe 70 years ago, right, uh, we didn't have dev and ops.
The devs who are responsible for operations, now you have dev, ops, security, you have containers, you have platforms, you have all these things, right? And so, uh, this explains, I think, why it's so increasingly challenging, you know, to manage, you know, these very complex socio technical systems. Um, I think that the third thing, um, is that.
Uh, you know, shifting left is right. In fact, one of the metaphors we use, um, in the book that I'm just so proud of is like, uh, is we're saying so much of knowledge work is like moving a couch. So you have two people moving a couch. Let's call them Steve and Jean. And you would think that this is all. No, you know, thinking needed.
And yet, you know, for them to move a couch, there's actually all these problems they need to solve. Like, where's the center of gravity around? Which access do you rotate to get through a narrow doorway to get through a narrow [00:14:00] winding staircase? Who goes first? Should they face forwards or backwards? You don't need consultants, you don't need focus groups, right?
Just by picking up the couch and trial and error, fast feedback, you know, Steve and Gene will be able to figure out how to, you know, achieve the goal. But there's all these things that we as leaders can do to make it far more difficult. Like we can turn off all the lights, right? And it gets more dangerous, takes longer, we can damage the furniture and the room.
Um, we can introduce a lot of background noise, uh, like a siren. And, uh, or we can introduce an intermediary preventing Steve and Gene from directly talking to each other so they can't communicate and coordinate and so they can no longer operate as a team. So instead, we make them go through, uh, JIRA tickets, or we have them go through complex approval processes through the GRC systems.
And so, um, it means that they can no longer communicate and coordinate and solve a problem together. And so, the metaphor, um, of the couch is really around joint cognition, joint problem solving. I think the reason why, uh, Shifting Left has been such a breakthrough. Well, first, allowing dev and [00:15:00] ops to work together and move a couch together.
is what created these amazing outcomes where you can, uh, organizations like Amazon, they do 136, 000 deployments per day safely, securely, you know, with reliability. Um, because we have them actually talking to each other, right? Then moving a couch together. Where does information security fit in? It's that before the information security person had to help move the couch.
So they had to be there for every couch moving operations and there's just not enough of us. And so what does security really need to do? They create tools so that Steve and Gene can do the work independently, right? So that, uh, you know, uh, security can create these, you know, paved roads that, you know, if you, uh, I'm sort of mixing metaphors here, but as long as, um, You know, developers use that paved road to get in production.
You know, they can be the first line of defense, right? Because they will own 80, 90 percent of the control requirements. Um, you know, liberating the developers to worry about the things that only they can do. So, uh, anyway, uh, how am I doing [00:16:00] here?
Sean Martin: Yeah, I love those. I love those three points. And, um, it reminds me of, yeah, just, I mean, so many things we can talk about, but just yesterday I, um, I had to, uh, I had to go to the dentist.
Or B2 or two days ago or something that was going on with, with the tooth. And, um, I arrive and they said that we didn't allocate enough time for your appointment. Because we need to do all this work. And I said, uh, no, you've already done that work and it didn't work, which is why I'm back. And it was this weird, this goes to your point of giving the right information at the right time for who needs it.
And there was this multiple exchange between the front desk and the doctor. And then myself sorting out that, no, the work didn't need to be done because it had been done. Um, Which [00:17:00] very inefficient, right? Um, if the doctor had actually set up for that procedure and pulled all the materials, it could be a lot of waste.
And of course, my experience as the customer, I don't, I don't really care. I mean, it all got sorted out in the end, not a great experience for the customer either, so not, not having. The information ready at the right time can have a huge impact on, on the outcome of what you're trying to achieve. And the other thing that I want to point to is the, um, just the moving the couch and.
I think you described it as a bit of trial and error, right? We can, we'll, we'll go up these, the stairs and through the landings and through the doorway, um, you can try, or you could bring somebody like security and or quality assurance in and say, here's what you can expect. Let's do that math. Let's run through those scenarios [00:18:00] before we even try.
So that we don't either before we release or in front of the, in front of the customer, run into this, run into this problem. And, uh, yeah, so I'll, I'll leave it there for you.
Gene Kim: So, no, I love that. In fact, I think. You've really picked up on like what we're saying that there's really three mechanisms of performance.
Um, and maybe just set this up a little bit just to maybe describe adequately just how much of an aha moment. This is, uh, many of you might know that I spent, uh. One of the things I'm most professionally proud of is working on something called the state of DevOps research. Uh, this was Dr. Nicole Forsgren and Jez Humble.
It was this cross population study that spanned over 36, 000 respondents, uh, over six years. And the goal was really understand, what does high performance look like? And so we, it was just amazing. These high performance during multiple deployments per day. Uh, there, uh, you know, the amount of time required to go from code committed to running in production was one hour or less.
Uh, you know, when things went wrong. Uh, [00:19:00] you know, they could fix it in, uh, again, one hour or less. And this is like two or three orders of magnitude better than their peers. And so the top two predictors of performance were, one was architecture, uh, which was, you know, to what degree can teams work independently of each other without a lot of fine grained communication and coordination.
Um, and you know, when something goes wrong, you know, does it cause a small failure or a big failure? Kind of, uh, we often call that the blast radius. And the second, uh, thing that was, uh, the top predictor of performance was, um, was the presence of like psychological safety. Um, and, and so what we've So whenever you see, you know, these amazing transformations from high, from not high performing to high performing in technology, if you look at something similar in manufacturing, in engineering, you actually see something very, very similar at work.
And what we're saying is there's really only three mechanisms of performance to go from, you know, not. Great to great. And so it's basically three things. One is slowification. Uh, in other words, there are certain things you should do, [00:20:00] uh, when you are working in highly consequential environments where you can't undo, uh, that have kind of enormous outage cost.
Uh, you can't look and, you know, and under those conditions, you can't learn, right? Uh, you can't do a lot of experiments. So therefore you have to do that in planning and preparation. So whenever you see something, uh, someone doing something, some splendid, exquisite work in highly consequential environments, you just know they invested in planning and preparation.
And so we think, see that things like that and chaos, like chaos monkey, right? We, we test and inject faults into the production environment to see if we are as resilient as we think we are. You can see that in like, uh, uh, you know, consequential, you know, military operations, um, and so forth, right? Planning and preparation, right?
Is where you not. Production is where you learn, but it's better to learn it in these kind of safer environments where it's slower, we can have more control, you know, we can iterate and so forth. The second, uh, um, mechanism of performance is simplification. Make the problems easier to solve. So [00:21:00] we do that by solving problems either incrementally, so it's like agile, uh, like lean startup and so forth, right?
We don't solve the whole problem at once, we solve them in slices. The second is that we split up. problems up so that instead of one large, interconnected, highly intertwined, highly coupled system, right, we divide up into smaller pieces. So that's like the Amazon, uh, 2001 transformation where they went from one code base, uh, that all had to deploy at the same time to hundreds of them.
So that's what allows them to do 136, 000 deployments per day. And then the orthogonal one is linearization. You know, we do the same thing, but for linear sequential processes, and it creates this independence of action. allows teams work independently of each other to, you know, uh, improve independently of each other.
And then the last mechanism, uh, we had slowification, simplification, and amplification. We have to amplify weak signals of failure so we can act upon them decisively, you know, before they become huge failures, right? Uh, and so we can better prevent, detect, and correct. And I think for information security, right, it's so important that we, uh, You know, we detect [00:22:00] these weak signals of failure, uh, so that we can fix them before it gets into production.
Right? We can, uh, so we can, you know, train engineers. We can, uh, you know, improve and harden the environments or the, uh, create better CICD pipelines to deploy code into promotion. We can increase the telemetry. Uh, because again, you know, we want to be able to take every one of these weak signals, right? And act on them so that we can, you know, uh, better achieve our most important goals.
Um, and so your solification, one example I love, right? Uh, before we actually do the staircase operation, maybe we should walk it, walk the path with someone who has a lot of experience doing it, right? Like our information security, uh, peers. Uh, so we have a better shot of, uh, not having to see it for the first time in production.
Sean Martin: Yeah. So how, I love it, Gene. Um, I mean, I geek out on this stuff anyway, so, um, it's fantastic. How let's stick with the area that you've been most involved [00:23:00] with over the past few years, uh, in, in the DevOps. Um, so we can use, let's pull on some examples there. What I'm, what I'm trying to understand is, I mean, let's start with.
Team structure and then maybe we can move into some of the processes because what, what you're describing, um, he touched on it earlier and in the context of specialization, right? Looking at the, the, the doctors or the hospital, there's a ton of specialization, which leads me to believe perhaps we need specialization and.
DevOps as well. So how, how does, how does this fit, or how do you apply what you're learning directly to DevOps? And maybe we can, we can blow it out to the bigger security program to, to help.
Gene Kim: Yeah, for sure. That's a great, um, it's a great place to start. And I think, uh, Yeah, I think one of the, there's a lot of indications that just from looking at the space [00:24:00] that says there's a lot more functional specialties than there used to be, right?
In fact, you know, you have just, you have containers, Kubernetes, right? I mean, uh, in fact, I'll tell you about all the work I don't want to do as a developer. I don't want to understand logging and how to connect my, uh, you know, Java logging framework to where it needs to go. I don't want to, uh, uh, every time I connect to a database, it takes me a week, right?
Uh, authentication, authorization. I'm hopeless. Uh, data masking, uh, haven't, maybe I'll do it someday. Secrets management. I finally figured out after like my, uh, CICD tool leaked my cloud credential. So I had to figure out how to do that. That's a great idea. Um, but, uh, that's not, I managed to kick that can down the road for a couple of years.
Um, uh, uh, oh my gosh. Uh, the, I actually saw a properly written Kubernetes deployment file and I almost fell on my chair because it looked nothing like my Kubernetes deployment file. And so each one of these things, I mean, in any significant organization, uh, you know, let's say thousands of [00:25:00] developers. You see these functional groups that say, hey, look, you don't have to learn all these things.
You know, you can use my library, my shared service, my platform, right? And I will take care of all those things for you, right? So that, uh, even at Google, um, it turns out, you know, like you're, uh, there's a group of five people at Google that owns the Java platform and your average, so, To liberate developers from having to learn about how to do their Java 8 to Java, uh, 11 migration, right?
It turns out that maybe not that easy and it changes garbage collection and memory models and all that. Uh, this is actually, um, not saying that most developers can't understand it. It's just that they don't want to. If they're like me, they have other things to do than, uh, understand all the vagaries of, you know, these, uh, different, Uh, JVMs.
Sean Martin: So anyway, uh, so I mean, let me pause you there because what you're describing sounds an awful lot, even if it's not formal. I've been imagining that Google does have a formal platform engineering program. Yeah. [00:26:00] Is that what you're talking about there? Is that what you see us moving towards more?
Gene Kim: Oh, for sure.
And I'm saying, you know, you have Uh, platforms, you know, it was like deployment platforms or, you know, the environment container platforms. Uh, and I'm saying, you know, if you take a, we can even generalize it further and say, there's just a lot of specialization of skills happening, like, you know, the small, you know, Java group that supports thousands of Java developers at Google.
That's an example of a, uh, a platform capability or a specialization that says, Hey, Developers, you've solved the business problem that you're, uh, you know, chartered to do, and we will worry about the intricacies of, you know, the Java platform, including the JVM. Uh, there's similar capabilities for the build systems, the test systems, the approval, uh, the code review systems, right?
All of those are specialties. And, uh, I think if we open up the aperture and look over the decades, there are many, [00:27:00] many, many, many more specialties now. Uh, than there were a decade ago, just like there are many more specialties now in the hospital than there were 70 years ago. Does that resonate with you?
Sean Martin: It certainly does. So I want to know now, curious minds want to know, Gene, uh, how the three principles, the slowification, uh, how do you, how do teams adopt that? Yeah. Is it at an individual level? Is it a team level? Is it a functional level? Is it an ops level? Where does that stuff come in? I
Gene Kim: think the answer is at every level, but, you know, let's be clear.
The leader is ultimately responsible for allowing and enabling everyone to do the work easily and well, right? That's a function of the Layer 3 social circuitry. And when everyone's stuck, right, when everyone let me just tell a quick story of like how bad it can get. Um. You know, at Amazon in the early 2000s, you know, they were doing, say, hundreds of deployments a year, you know, [00:28:00] back in the late 90s, and because everything was so tightly coupled together, uh, as they added more product categories, you know, they went from books and music to 33 categories at the time.
Where, uh, you know, to get anything done, you have to communicate and coordinate across multiple groups. There was this absurd situation that Werner, uh, Vogels, then and current CTO of Amazon, said, uh, you know, the digital team, so it was, uh, Kindle, Music, uh, they, uh, had this absurd situation where in order for a customer to order a product, they had to provide a shipping address.
For a digital product, it makes no sense, right? But so they went to the 60 different ordering teams to try to lobby to make a change, and they said, we didn't budget for it. And so they were stuck, right? Because everyone had lost that independence of action. And so, you know, another side effect was that you couldn't scale any piece independently.
So, um, This is what led to that famous memo that said, we're going to move to two pizza team, right? So that every team can work and deliver [00:29:00] value independently to the customer, right? Without this massive level of communication coordination. Um, and that's what led them to create these microservices, um, service oriented architectures.
And so they, you know, went from hundreds of thousands of services, uh, and that's what allowed these teams to work independently of each other. They, we liberated, uh, thus liberating the teams to work independently of each other. Another key piece was that, uh, instead of the dev versus ops silo, where ops did the deployment, ops had to immobilize when things went wrong.
They moved to the, you build it, you run it. So the feedback went to the right place. And it turns out security, this is a huge boon for security because it means that it's not only you build it, you run it, but you secure it. It's your responsibility. You are the first line of defense. We can act as coach and consultant to help you secure it.
But, and we can even give you tools to do it so you don't have to make your own seatbelts, right? We can provide you a trusted seatbelt that everyone, uh, everyone can use. Um, so this is, this is a great example of [00:30:00] simplification. We took a big, you know, uh, Intertangled system and split up to multi pieces to liberate independence of action.
That was enabled by slowification. Uh, you know, Jeff Bezos, by putting out this memo, said, hey, look, we will do whatever it takes to move to two pizza teams. This was a multi year, allegedly 1 billion plus re architecture of, uh, Amazon. But again, you know, the rewards were huge, right? Uh, they were able to Go from 10 deployments that mostly didn't finish to 136, 000 a day, delivering capabilities at a wild, dizzying rate.
And actually part of that AWS was born. Um, then amplification, the need, you know, uh, to put concretely, you know, they found that, uh, a whole bunch of troubleshooting. Uh, was very, very difficult. Like when one service went down, uh, there was often not enough instrumentation to actually say what went wrong.
Was it me, my service? Was it a service I depended upon? Like who gets paged, right? I mean, so, um, they [00:31:00] actually had to create much harder boundaries around services. And, uh, you know, if you have a 30 minute SLA time and it takes you seven attempts to page the right group, I mean, you've already blown through your SLA window.
Anyway, so. Uh, how am I doing here?
Sean Martin: I think you, you totally blew it, Gene.
No,
I love that story and, uh, I appreciate you sharing that and what I'd like to do. I think we've touched on quite a bit and spoken fairly well to security in DevOps, right? So organizations building stuff that need to have security built in from the get go.
Um, I want to talk about security teams that don't just look after the bespoke stuff, but. All the commercial junk that gets tossed on them for communications and supply chain management and I don't know, [00:32:00] uh, customer service and all the things that, that they bring into the bigger picture. Yeah. What can they learn from these same three?
Gene Kim: Oh, that's a great question. I'm going to try to think my way through the answer. Uh, so let me start with, like, what's often most familiar is, uh, uh, so for custom software that you build internally, uh, one of the most powerful things that security can do is embed a security person. Uh, you know, into those groups and it was actually Justin Arbuckle, uh, mutual friend acquaintance of ours.
He was CISO at, um, GE Capital and he said like one of the best things, uh, and breakthroughs that he experienced was having the security people show up to the, uh, the development standups and the weekly demos. I was initially mystified by this because my feeling was like, it's so informal. It's very different than the kind of heavyweight approvals and, um, kind of the [00:33:00] gotcha exercises where you kind of scan a security application, you know, approve it to go into production.
And, uh, he gave me two examples I thought were just. Yeah, he said, um, this is your opportunity as a security professional to see, oh, hey, look, um, here's the business goal they're trying to achieve. Oh, hey, they're, they're actually taking, you know, um, personal identifiable information. Oh, this is a good way to be able to tell them.
Do you know that. Once you store that information, you have to protect it. And you know, there's things that we can do to help you and tools that we can bring to bear, but maybe the best thing to do is actually not store it. Like, thus obviating that whole surface area of vulnerability and exploit. And I think the aha moment for me was like, oh, instead of this high ceremony handoff process, uh, here's a, here's a, a, a dialogue we can have that's low stakes and it really mirrors kind of what friends do when they help each other as opposed to like, uh, [00:34:00] high stakes approvals.
And maybe that's the kind of level, you know, that's how security can help move the couch, right? Versus, you know, be the inspector at the end saying, you moved all wrong, start all over. So I thought that was just a,
Sean Martin: uh, We want security sitting on the couch where you're trying to move. Oh,
Gene Kim: that's right. Not like that.
Like this.
Sean Martin: Exactly. No, turn left, turn left. Yeah.
Gene Kim: Um, so, yeah, that's a great example of like, uh, Amplification, uh, is a great example, just like, uh, you know, a more integral part of the process is where you're valued. I think for, like, things that are commercial things, you know, I think this is where, again, security can help because, uh, you know, whether it's, uh, leveraging Uh, processes like, oh, hey, here's a standard process that you can use to sort of give to the vendor and, uh, have them make sure that they have their SOC 2 review that, uh, we can, uh, you know, that we can rely on them for security.
So, therefore, we're not reliant on the security of that, whether it's for the applications or data. [00:35:00] Um. You know, if it's not that, then at least giving them, uh, giving whoever wants this tool in a process that they can go through that, uh, we can set expectations that if they do X, Y, and Z, then we can respond with A, B, and C.
Um, and, you know, I think this. Just, uh, shows that security isn't free, right? Uh, I love that. Things in technology and IT is, uh, like a puppy. You know, puppies aren't free. It's not the capital cost that gets you. It's the ongoing, uh, maintenance of it that's really expensive. So, you know, these are things that I think, one, show security is a value add, but it also allows people who need to be responsible.
We enable them to be better responsible for that.
Sean Martin: So I, we're coming close to the end here, but I want to touch on the social circuitry. Uh, point. Um, I think it's related to what you just described, but I have a feeling you have a much more succinct point to make [00:36:00] specifically around social circuitry, bringing broad security mechanisms throughout the organization.
How does that look? Stand up meetings? Yes. But beyond, beyond that, do you have any more? Yeah. Oh. Are examples to share
Gene Kim: where that That's a great question. And I would say instead of, uh, a concrete example, lemme give you kind of the spectrum of examples. And for me, um, you know, there's a spectrum of what interactions look like.
And you know, in software architect we call that, you know, coupling and cohesion. And so what we're saying is like when Steve and Gene move a couch, right, they are inherently coupled together and it means that, uh, you know, if Gene walks off , right? Steve can't do his work, right? And, uh, but it's because their inherent need to.
Communicate and coordinate and co create. Um, but there's also examples of things when we don't want things coupled together. And so, like, one example is like air traffic control and pilots in a plane. Right? And so, in an air traffic controller, right, the controllers can change [00:37:00] shifts without permission from the pilots.
And pilots can change who's on the radio without permission from the air traffic controllers. Because all of the, what's important is in the message, not the messenger. They've agreed on protocols, they've agreed on, you know, processes and norms, uh, vernacular, um, and that actually creates independence of action.
And so, as a security person, I would say, what things do I need, kind of, highly coupled, uh, behaviors, right? Where, you know, you just want the security person to sit side by side with the developer and solve a problem together, right? And you don't want to go through work tickets, you don't want to go through forms or eight levels of management, just go up eight levels and down eight levels, you know, just Have them sit next to each other and solve a problem together, right?
Kind of the ultimate DevSecOps co creation activity. But there's other things that you don't want security coupled to, right? We don't want every security change and every security test to go through a security engineer, right? Those are the things that you want built into the tools, so that that can happen independent of the security professional.
Right. Or there's approval forms, right. Where [00:38:00] you don't want, uh, uh, you know, you just, I fill out a form and I know that within a week or two, right. I will get a response back. Right. So we decoupled, uh, security from those functions. And so I think for me, those are just two great examples of like, uh, for a security leader.
As we design kind of the interactions that we want, where do I want, you know, face to face contact? You know, we're working on a problem for a half a day, a day, maybe a week, right? Where you want that security person involved in every step, like in a generative AI project, right? Where, hey, look, uh, we, we have a common goal.
We're going to create our own processes, but security has got to be in the room versus, uh, you know, activities where, hey, look, uh, I understand what the inputs are. Uh, I understand what the outputs need to be. You do not need to be coupled to me, right? Tell us what you need and we'll prioritize it and, you know, get it back to you, right?
And we have a negotiation about what the SLA windows should be. Um, but this is something that, you know, we can do by ourselves. [00:39:00] I think this is a really good lens. And I think what went wrong in GRC systems is that, uh, we mistook a lot of activities that we thought had to be highly coupled that should not have been coupled at all.
Sean Martin: Interesting point. Interesting point. And I, I was certain we were going to take a dive into Gen AI. We almost made it through without touching on it at all. You mentioned it once there. I think maybe we'll come back and we can have a dedicated conversation to that. Because what I want to do is talk about people sitting on the couch.
Reading the book that you and Steve wrote for them. Um, quickly tell me, well not quickly, tell me uh, tell us who you wrote the book for and what you hope they will get from it as they read it. Is it a resource? Is it a, is it a thrilling novel? Is
it Tell us who it's
for and what you [00:40:00] hope to
Gene Kim: Yeah. Oh, that's a great question.
In fact, you know, um, you know, look at the back cover because often, you know, just a little secret between, um, authors. You know, we often use the back cover as a kind of a proxy for who the audience is. So, um, one person is Paul Gaffney. So he used to run technology for Home Depot, Dick's Sporting Goods. Uh, uh, he ran, he was a CTO and supply chain head at Kohl's.
Anyway, so yeah, a technology leader. One of them is Phil Venables. He was a He was the CISO Goldman Sachs for, uh, I think the longest tenure of any CISO I've ever known. And he's now CISO at Google cloud, but he was also a former board director for Goldman Sachs bank. Um, and, uh, Dave Silverman, he was, uh, one of the coauthors of team of teams, Jeffrey Leiker, Dr.
Jeffrey Leiker wrote the Toyota way. So I think, okay, so I'm just sort of reloading in my head. So it's for the technology leader, uh, but also their boss, right? I think over my journey, I've seen so many things where the success of the technology leader, and that includes security. [00:41:00] is so dependent on who their boss is.
You change out their boss and you, and it was this great organization, uh, falls apart because they no longer had the person who's setting the tone at the top that was creating the air cover for the technology organization to succeed. And to me, that breaks my heart. So the goal, one of the goals of this book is to show the, uh, that person.
Who the technology function reports to that, you know, these things that leaders need to do is not just, uh, just for technology. It should be familiar to any person who succeeded in supply chains, manufacturing, engineering, um, and incidentally, one thing I'm super proud of is that the forward was written by Admiral John Richardson.
So, uh, he used to, uh, be in charge of the U. S. naval reactors. So they are comprehensively responsible for all the design operations of nuclear reactors for every, uh, reactor in the seagoing fleet for the U. S. Navy. And he became the chief of naval operations, um, [00:42:00] For the entire US Navy, so 600, 000 sailors and civilians who support them.
Uh, and so after having left the Navy, he's now on the board of Boeing, uh, at Exelon, uh, the largest operator of nuclear reactors in the US and now Green Energy. So, uh, I'm hoping that, uh, this shows that the book is really aimed to show that whether you're a technology leader. I think a lot of these themes will be familiar, but this, there are universal truths and principles at work here that isn't just technology, it's manufacturing, it's whatever domain that successful leaders come from is applicable to them too, and it should be familiar to them.
Sean, how am I doing?
Sean Martin: Yeah, and I, so I was trying to do the math and I stopped at two, but I have a feeling others will, will keep doing the math. Two being one book you buy for yourself.
Two.
The book you buy for your boss, but then I'm just thinking, well, there are other leaders, uh, that, [00:43:00] that need to understand these things as well to some degree, right?
Especially when we start talking about the social circuitry. Absolutely. That, that common understanding and appreciation for what's going on is needed. So perhaps the boss pays it forward and buys it for the whole team, all your, all your peers. Um, but I think the point is for me, uh, these are concepts that you've.
Studied. You and Steve have studied, have practiced, have guided, uh, and, and seen in action, right? Good and bad. And, uh, Yeah. Why not learn from, from somebody who has a lot of information and knowledge like you and you and Steve. So
Gene Kim: I love that. I can't leave with maybe two quotes that remind me of, uh, Winston Churchill said, uh, we shape our buildings and thereafter they shape us.
I mean, it's the same thing with the architecture and the social circuitry of an organization. We [00:44:00] create the social circuitry and architecture and forever after they shape us, right? And so it's one thing for us to do that within the technology organization. Maybe we can create a little island of happiness and productivity and greatness, but ideally that fits, should fit in, you know, in Overall architecture of greatness, right?
Uh, so that maybe the organization can learn from technology and get the air cover, right? And there we get the air cover that we need. And the last thing that I think has made this project so satisfying to me is, they say the goal of science is to explain the most amount of observable phenomena with the fewest number of principles.
Confirm deeply held intuitions and reveal surprising insights. And that's exactly what this, uh, whole creation process has been for me, which is why I find it so intellectually challenging, but so gratifying because, uh, you know, the notion that you can explain the most with the least. And, uh, I'm hoping that people will walk away from the book saying, Oh, this gives me a language to talk about these things, either why things went right or, uh, why things went wrong.[00:45:00]
Sean Martin: Yeah. And it reminds me of, which I'm, I'm often told, uh, write less, be more, be more succinct and to the point. Um, And I think, yeah, I think, I think if you can, if you can be simple and share that simplicity in a way that others can, can absorb it and then share it with others too. I think you've got a winner there.
Um, Gene. I could talk to you for hours. Um, I'd definitely love to have you on again. And, uh, so we'll, we'll make that happen. Uh, we got to touch on Gen AI at some point. Um, but until then I want to thank, uh, thank you for one, putting that book together with your coauthor, uh, congratulations on that. And I want to thank everybody for, uh, joining us for this conversation.
I'll put a link Show notes, uh, so people can find your book so people can find you and, [00:46:00] uh, and I'll also include the link hesitantly to my co founders, Marco's conversation with Steve. Uh, I think, yeah, Steve definitely brings a different perspective, uh, than you and I, Marco and Steve bring different perspectives than you and I did today, but none, certainly not any less important.
Gene Kim: Oh, and highly recommended. I mean, just, uh, that was a phenomenal interview. And, uh, just to have, hear Steve talk about from a different perspective, uh, you will hear just, uh, you'll see glimpses of why it was so rewarding of a journey for me. So, uh, highly recommended, Sean.
Sean Martin: Uh, good, good job sticking with it to do the
hard part.
I just realized we only win if they lose. Nevermind, don't listen to it.
Nice one. Well, Gene, it's a pleasure to see you. I'm grateful to, uh, have you in, in my circle after all these years. And, uh, Hope to, hope to have you on again soon and, uh, keep well.
Gene Kim: Uh, Sean, [00:47:00] thank you so much for having me on and looking forward to seeing you again soon.
It has been way too long. It has been. All right.
Sean Martin: Thank you.