Ep. #30, Improving Security Culture with Justin Somaini

Guy Podjarny: Welcome back to The Secure Developer, happy to have you back with us. Today we have Justin Somaini.

Justin has a very long and extensive history in security, ranging from being the Chief Security Officer in a variety off interesting companies.

Coming from Symantec and Yahoo! and Box and SAP. I'll let him tell his story in a sec, but welcome to the show, Justin.

Justin Somaini: Thank you for having me. It's great to be here..

Guy: So, Justin, before we dig into the many interesting questions that we can debate here. Tell us a little bit about how did you get here? Who you are a little bit, some short version of that history.

Justin: Yes. I've been in security for 25 years, and started out many moons ago at Pricewaterhouse as a penetration tester. During that process I realized that if I was ever going to be a great consultant, I might actually want to do the job at least once in my life.

I got the amazing opportunity to come over to Charles Schwab to run security operations, and then realized that this whole fixing security was a lot more difficult than just consulting on it.

So, I stuck with it and from there I went to VeriSign where I ran all security for five years, went over too Symantec after that as their CSO.. Then over to Yahoo! as their CISO, to Box as a chief trust officer. Then just recently left SAP as our global CSO.

Then of course, as we do in the valley, we have a tendency to participate with VCs and security companies, so I've been doing advisory roles and advisory functions for VCs and security companies as well.

I try to be very active in the industry that I love.

Guy: That's excellent. All those things also keep you fresh a little bit. I like to say they help you learn in parallel, and not just sequentially.

So, it's quite a journey. Maybe we start a little bit with that historical perspective, you've gone from these different companies that are all so different and different times.

How would you say the security landscape has changed in the ways you've been tackling, maybe the same title, across a decade or two?

Justin: Yeah, we have gone through a lot of maturity and a lot more is still yet to come. But over the past 25 years, when I started security was very much what I would call a very simple audit-and -compliance.

"We have a firewall. You know what? Nobody can ever really execute anything on a web page." Literally that was a statement that was made to me way back in the day. It's very sophomoric thoughts about security, even though we had luminaries in security theory back in those days.

But as we've matured and grown, I think that we've really grappled a deeper technical base of what's going on, hence application security and driving it deeper into the nuances of coding and development, and how coding is really done in spite of the acceleration that software development has gone through over the past 15 to 20 years.

But on par and parcel from a network standpoint, an education standpoint, from an organizational standpoint. We've done those as well. Some of the things that I think that will mature, but maybe a little bit later, if we look at global law enforcement.

I remember probably up until maybe 15 years ago law enforcement was not deeply focused on the cyber problem. It was a bit of a joke in a lot ways.

I hate to say this, I have a lot of deep love an affinity for law enforcement globally that deals with this problem, but there were other challenges that they were looking at.

That's a very different story today. When you look at the FBI or Interpol or Europol or some of the other agencies around the globe, they have dramatically matured to deal with the problem as that threat has increased.

So we've seen a lot of changes, but quite honestly, I think we're maybe in the first quarter of maturity in application security, or in security as a whole.

Not unlike they're still robbing banks, and that's been around for a while, and we look at other industries such as legal or finance that have been around for a very long time.

In spite of that, they're still maturing themselves, at a slower pace of course. I think we have a long way to go.

Guy: We're still at the beginning. Some changes are obvious in the landscape as some tech has changed, like mobile devices or others.

But what would you say are the beacons of change, or what are the key drivers of difference in today's world in security versus maybe a decade ago?

Justin: When I look at security I have a tendency at a meta-level that you're referring to it, I take a step back and I think about--

There's a big difference between security theory and applied security.

So if we take security theory about how we attach confidentiality and integrity and availability to data and its transactions, in a mainframe world that's fairly simple.

You apply it in the mainframe, it's a centralized model, you've got green screens that are basically just windows into that data and transactions. It's fairly easy to implement a CIA model to that.

What we saw with a transformation from mainframe to client server, the world dramatically changed, and theory didn't change but how it was applied did.

So, where that data was and where the transactions were, now shifted to workstation servers and multiple servers running those services, workstations' data being handed back and forth.

We had an explosion of complexity simply because the data and transactions moved, and we did not have the mechanisms to apply CIA to it.

If we move forward from client server into client cloud, now those services are on the Internet with a more insecure and hostile direct access environment, shall we say.

But really underneath the hoods what you're seeing is an interconnection of those services as we look at Web 2.0 and some of these other API integrations that you have. That transaction and data service model becomes even more complex.

Now as we move even beyond that, and what I would call multi-cloud and containerization, while the concept of the service doesn't necessarily change in a great degree in a container model, but what we do have is an exact aspiration on the operational side that we saw with virtualization.

We have many containers with many data and transactions, and how we manage and govern them becomes even more complex, and it's being done in a multi-cloud space around the globe with a whole bunch of different things.

So, how we leverage technology to do things has a significant effect on how we apply CIA to govern those things, and so that's the basic concept of how I look at technical security to start with.

What you see is you're able to predict a lot of the challenges that we face if you keep an eye on what is the new technology that is going to be adopted eventually by businesses and organizations as they try to accelerate their growth and revenue?

But containers came onto the market, and now it's flushing, and now everybody's trying to run and keep up. We can also do the prediction with serverless, a whole bunch of things are just starting to take hold.

Guy: I fully relate to that, I feel like for starters there's many moving parts and a lot of the safety of controlling the environment has gone away, because now everything is going to be all interconnected and mobile and on the internet. Before we go to the next challenge, what have you seen as strategies?

Like, successful strategies to help tackle that? It feels like definitely a big battle. What do people do, or what have you done maybe, that you've seen work to conceptually try and adjust to this fragmentation?

Justin: Yeah. The security management model that I try to approach, I'm a big believer in a general statement, the centralization of a security function so you can get some leverage. But getting to your point--

You're never going to be as agile as the lines of businesses or the development teams or the functions in a centralized model. So, how do you have a distributed or integrated security model to be able to deal with a day to day handling of issues as they come up?

As a result of that, generally what you have-- This is more pronounced in application security than it is with security operations, as we have shifted to an Agile model, the velocity of change, or releases, has dramatically shifted.

Which is for security to be able to handle this velocity, which is where they struggle to a great degree in AppSec. But one of the models is having embedded individuals in those lines of businesses.

But secondary to that is really finding ways to not only educate and empower the development teams, but also make it clear that because of the models and the decisions that they've taken there is a significant accountability that they have in the security model as well.

This is where the first problem really comes in with basic security teams and developers, because they are two very different camps or mindsets and beliefs.

We've grown up in security in this operational model, where I create a policy and a standard, and somebody is just going to build it to that standard.

That's what DBAs and system administrators do when they build service, that is not how development works.

Guy: Definitely not today.

Justin: You have a problem and the developers are part of the process to engineer a solution. Most security people have never checked in code. They don't understand what a CICD pipeline is.

If you talk about new technologies or new capabilities like mesh networks, or even containers to a great degree, it's just not really part of their normal ecosystem.

So the first step in any security build out for AppSec is to really establish leadership or a partnership with the development leads.

One, a general understanding an agreed upon problem set, security coming to the table with proposed solutions but not a dictation on what the answer is, but a series of answers so that the development leads or development teams can be part of that conversation.

But being part of that conversation also means that they're now enrolled, that they're adopting, and that means you're going to have greater efficacy in integration and implementation of those. Like static analysis tools, and things like that.

Guy: Yeah. I love the-- Going even back to an early start of that answer and talking about embedding security knowledge within development. But it sounds like it's the security champions even, or the likes.

But you're talking about also just as equally important as almost the embedding of developer knowledge within security, also ensuring that people are abreast on it.

Having those shared conversations and when, like in Agile there is no like "There's some glorified design meeting that happened that security needs to be a part of."

How in practice does that work? Do you take developers and make them a part of the security team? Do you send your security people to learn how to code? How have you seen that work?

Justin: There's two, what I would say, parallel paths. One is "How do you get just as much automation of security into that CICD process as possible?"--with the developers, o f course because they're the ones that are actually going to have to deal with the output.

So if they're not part of this, it's not going to go well. In my experience you have really an opportunity to teach developers how to fish, and if you teach them well what comes back to you, security, is exponentially more than what you put into it.

Let me explain. If I can teach a developer the basic concepts of security that we look for, and how we look for them in general, they're able to shard that into the myriad of different ways that things are actually developed and implemented and coded, etc.

To be able to come back into that process and actually give advice on better solutions. So education and training is really big, but it's very difficult to scale.

You have labs. Hands on experience is probably the best one to do, and so you have those mechanisms and it's a train the trainer model. I'm a big believer in that.

Train one developer on the security and have them host a lab for 10 or 15, and eventually you've got to go through that process.

But on a day to day basis I believe that having an individual that is born from the development community be identified in the security team as the security lead, for lack of a better word, or the security accountable individual for that LOB, that effectively is part of those scrum meetings.

That's part of that leads. That's doing the hand-to-hand discussions, and walking through, and working with the developers .

Because I've found that developers in the community that they have on a working relationship day to day is very powerful on being able to get their buy in, get their proactive approach, and get their focus. Versus a gate check that they have to go through.

So I want people embedded as much as possible. Usually identifying those individuals and having them be a part of it is the best way to go.

Guy: Yeah. What I love about that perspective as well is that it's very analogous or very parallel to how DevOps operate. You've got those sysadmins, ITs that were outside the team.

You look at what happened to those sysadmins, they became DevOps automation and they became SREs. They're probably paid double.

They're embedded into the application while most of the activity, most of the ops activity is handled by the dev teams, but the proficiency is built within-- It depends on the org. But in many times within the org, basically let's apply this for security and create security's equivalent of an SRE.

Justin: It does have some problems though. So taking a step back and maybe not droning on this too much, but Agile has come in years ago but has had a lot of changes, not just on how software is developed. That's very easy to see.

But the impact on the business, what we call digital transformation on how we digitize the entire supply chain, and pull customers in and put suppliers in, and actually have those analytics dramatically change the products that are actually being delivered.

Netflix is probably a great example of this. On the security side it's had a significant impact as well, that I think we're just now starting to realize, but can take some insights from newer companies.

As Waterfall to Agile started to approach, we saw higher velocities. So what does security do? We need to get tools in, we need to do training.

But at the output we still have the same vulnerabilities, or the classes of vulnerabilities.

OWASP Top 10 haven't really changed that much, right? So, what do we do?

We're gonna put it in an SDK and give it to developers, and give them standards, we're gonna have third parties come in. And yet what we see is the same classes of vulnerability are still there.

Cross-site scripting is still probably the biggest vulnerability, and everybody knows how to solve cross-site scripting.

Guy: Yeah.

Justin: The question is, why is it still there? What you have in businesses, you have this problem where developers are caught between a rock and a hard place.

They are on one side being paid and accountable for feature functionality of a service, and on the other side you got security saying "You need to be very clean in your code." So you need to dedicate time to both, and it just doesn't work.

Which is why we have cross-site scripting issues, etc. Because the developers are going to go where their paycheck goes. Feature functionality.

What I see security doing is moving to solve this problem, is moving what I would call from a governance and advisory type function, to one of governance and product delivery.

The reason for that is, let's take something like cryptography or data at REST encryption. Historically you have two organizations in a security team, an enterprise security and an application security team.

They would go out saying "You need to have a peak AI environment, here's your standard end build to solve that problem." What we see now to be agile, to be the product security, the security teams are saying "Stop coding data at REST solutions.

We will do it and provide you a service, an API service, to be able to leverage it." That means that enterprise and AppSec teams are combining into one team, to be able to do the PKI and the APIs all the way up the stack to be a production service to the various applications.

That's a very different organizational and skill set model than we've ever seen before, and I think that's a significant change that we're going through right now.

Guy: That makes a lot of sense. You're tapping into some of the advantages, there's a lot of conversations about the advantages of DevOps from a security perspective, and the advantages of agility. Probably the most well touted one, or most touted one, is speed.

The fact that you can patch faster, but you're highlighting a different one which is, "If the system is more library-oriented and micro service oriented and the likes, then I can interject, or the security team can interject more elegantly by being a component of the system that is security conscious."

Justin: Yeah. When you look at born-in-the-cloud companies, security started with one developer solving a problem. Being in the valley, of course, you talk to tons of them.

It's one developer saying "I need to create an identity and authentication mechanism for our service" "OK, then how did it grow?" "I needed to create a logging API." Then, "I needed to create a crypto."

And then eventually we had customers saying "We need to be certified, so we got some of those governance people." But it was born a very different model than what other companies that have been around a long time are going in reverse.

So, I think there's a lot of lessons in regards to why they have done it, and the scalability that they have as a result. I think it's an incredible opportunity.

Guy: It makes a lot of sense, and for that you need to transform not just how you work, but actually the makeup and the skill set inside of your security teams to be able to provide these types of solutions.

Justin: Yeah, I would say a security team's staff will rotate about a third, maybe a half, from process management that we do today generally into developers. As a result of that they're developing internal tools as well to be used. But, what does that do to the security workforce?

Guy: Yeah.

Justin: I think that's a big impact that will be going on for 10-15 years.

Guy: On the flip side, it might be an answer to the infamous security talent shortage that really does get resolved. Not that workers are that plentiful, but a slightly more varied talent pool there.

Justin: Yeah.

Guy: We talked a lot about application security, but when we've spoken before you were mentioning how application security extends more, or will extend more when you compare it to endpoint security, that's an area that is not as mature as it should be compared to the risk.

What's your view on the AppSec industry, if you will?

Justin:

Yeah, I think the AppSec industry is absolutely horrible. The reason why I say that is, not that good people haven't tried, but it's a very difficult problem to create a business.

For example, how many firewall vendors, end point solutions, etc. do we have out there? A lot. Go to RSA, how many of them are dealing with application security versus anything else?

Yet, when you look at the problems that we face as an industry. 90%, 95%, 98% perhaps are at the code level. It makes no sense whatsoever.

The only reason why it makes sense is that it's so much easier to create a network control access point than it is to do a mature static analysis model.

It's very difficult to do AppSec from a vendor standpoint.

Because of that, what you see in organizations is that the solutions that a vendor can put in has no context to the application development service or business in which a company has. It makes it very difficult to create a one size fits all solution when our environments are pretty different, especially for cloud services.

Let me put it that way. So what happens is that the internal teams in that company need to take that on, since the product delivery statement that I made about security teams. I wouldn't say all of them are doing that necessarily, but that's what should happen.

This is why I say application security is pretty bad. On one side, we have a complexity and a problem, and a difficult business model to create a vendor in this space to solve the problems.

On the other side, we've got developers that are paid for feature functionality and are not able to dedicate-- Or even security teams to get the resources to do all the things that need to be done inside the application to solve the problem.

The third problem I would say is, if we look at any of the other solutions that are out there to do WAF application firewalls and things along those lines, while it's an effort, it's what we will call a secondary control versus a primary control.

The primary control must be in the application where it has context, it understands what that is versus being outside of it.

I think that it's great that we create it initially into the market to get something there as a whole, while we have a long term solution on driving it inside the application.

Generally companies haven't done this, so we are only left with a WAF that ultimately causes problems sometimes, right?

Guy: Yeah, it's the same notion that you started with talking about how it was easier to protect the mainframe than all these moving parts. The application has a lot of moving parts. The network there's one of it, or there's five. It's just easier to tackle.

Justin: I mean, nothing for nothing, but how are you able to protect a modern day application that is containerized, is multi-cloud, it's got connections up and down your supply chain and developers are pushing releases at least once a day?

Guy: Maybe the good news is that there is an opportunity to try and crack that in terms of the need of the industry, but the complexity is very much there.

Justin: Yes.

Guy: Is it safe to add to that, there was a shift in people? I definitely talk a lot about that, which is one aspect of it, is the complexity.

But I guess I wonder whether we are talking to the wrong people, whether the solutions that are built-- I personally built a product called AppScan Dev Edition that had developer in the name, but that was pretty much it.

I mean, it was a really good product and it succeeded financially, but it really was an auditor's product built into a developer environment.

Do you see that happening? Do you think that's indeed an important part of the solution or is that secondary?

Justin: If I understand your question correctly, it's how do we make sure security is acted upon by the various LOBs, in this case, the product team? Versus driving security to the networking or ops team, which can't really solve the problem?

At least that problem, on the application side. I think that that's absolutely the case, but that gets into a very different problem which is, how do you drive a culture of security into the executive management team?

Second to that, how do you ensure that security is a business driver, so that it is important to the executive management team? Which also requires a security person to be business aware.

We generally have security people, not to make it overly complex, but we need to be more participatory in the executive management team. To do that, we need to at least have a modicum of understanding of what a business is.

"What is a funnel? What is marketing? What are conversion rates? What are sales cycles?" That's generally unknown. What that allows you to do is to use all the tools in your belt.

Customer requirements, demands, competitive analysis of other competitors in the market from feature functionality standpoint, solicitation on deal flows and the sales cycle and how you can shorten that, and bring that to the product managers and say:

"With better security, we can increase net promoter score. We can shorten the sales cycle. We can actually possibly drive top line revenue, if we have really lock solid security implemented and have a lean-forward approach on how we open the APIs to other partners and networks, and all these other things."

I believe in that model very significantly. It's what I did at Box, as a Chief Trust Officer.

Driving that competitive differentiation on security, it has a massive impact on the culture and what the developers really see as important, because it's coming from the business.

I mean, I consult with my clients a lot on that one topic and it is very rarely-- I've never heard anybody really talk about it, and I feel very passionately about needing to drive that.

Guy: Yeah, I agree. It's hard, anything that requires an org-wide change is not something to be taken lightly. You also hear a lot about that even in the worlds of marketing needing to adapt to Agile.

It's almost like everything is adapting to this DevOps, Agile change. Because the whole business is becoming one big feedback cycle, and security needs to be part of that group as they adapt the business.

Also, at a certain cadence that is now not just based on feedback of users.

One of the key challenges is that security's feedback cycle is bad. It doesn't hurt until you're breached, and then and it hurts really bad. It's hard sometimes to know if you're doing it right.

Justin: I really like the way you just put that feedback cycle. Honestly, I've never heard that. One, it makes a lot of sense and it relates obviously to a whole bunch of things.

Yeah, I think if I was gonna phrase it a different way-- What I was saying a different way, using your words, "How do we get security part of the feedback cycle of the business?"

They are still focused on hearing about what Gartner is saying, what customers are saying, and what the competitors aren't doing.

How do we pull, tease out the security-isms of those three and have it be part of lifecycle? So that now they pay attention and drive it inside their words?

Guy: Hallelujah. Now we need to actually get that done. Actually, with that there's one other bit that I want to make sure that we cover before we do it, and this is actually a decent segue.

It's to talk a bit about corporate cultures and security within those environments. You've seen big and small and also over time, what would you say working with Symantec, Yahoo!, Box, SAP.

How does the corporate culture play into it? Maybe to try and give it a practical sense, how did you need to adjust your approach? To change what would work, given that surrounding?

Justin: Culture is an interesting thing, and there's no one-- Take any company, there's no one culture.

But all of them have it, so when you look at security and you approach about implementing it, what you're really talking about is "How can I move the needle for somebody else? How do I understand what their challenges are, what they're trying to do and get them onboard with the problem that we're trying to solve and know their role into that?"

Culture is a massively important thing to not only break into that behavior change, but also solidify security as a core value of that organization.

When you look at a Yahoo!, being in Sunnyvale and one of the bastions of the internet back in the day, and coming around. At that particular time going in it was you had Facebook, you had Google especially coming in, and a lot of competition that they once dominated.

There was a bit of a turnaround within the company culture that needed to be aware. I had five CEOs within 12 months, not a fun experience.

To get to the developers, you had to understand the challenges that they were facing when they were going through their own transformation in the business, and how can you help solve some of their problems, if you're asking them to help you solve some of theirs?

It's the "We're in this together, how can we move forward?" I think that statement holds true in any other company, at Box for a great example.

You know what? We're starting out, we're a startup. We're breaking through, we're going to go IPO. We've got massive competition from Microsoft and Dropbox and the like.

We have limited, massively limited resources and money, and all those other wonderful things. We're in this together, and we need to fight.

You can't have that "We're in this together,"when you're doing a keynote and then walking away, issuing a standard. That doesn't happen.

You have to be in the thick of it, day to day. Listening, hearing and working, and rolling up your sleeves and working with them. That's the first thing about culture.

The second thing is, the development culture vs. any other team's culture is very different. Most security people don't understand that as well. Where the developers, they're the creatives of technology, for lack of a better word.

They're very accustomed of being presented a problem, and they're tasked with figuring out how to solve it. Versus what we've historically done in security, which is security comes up with a policy and standard, i.e. what the answer is, and we give it to somebody else to just implement.

That doesn't work very well in development communities. So, what you need to do is you need to go in with not only a clear definition of what the problems are, but probably a couple of options on how to solve it and solicit the input advice and guidance of the development community on that ultimate answer.

If you don't do that they're going to reject it. It doesn't work. You don't understand the development process, the codes, the languages, all the other complexities that they deal with.

Honestly, at the end of the day, a business is either a product or a sales company. It's not a security company. You don't have the political power. So, you got to be in there, you've got to win the hearts and minds, and to do that you've got to check your ego at the door. Because everybody else knows it.

Everybody else knows you're not as smart as you think you are. Generally, they're smarter than you, in the development space.

Guy: Well, lots to unpack there, I think we might need to do another episode. Because there's so much more to even chat about there. A lot of great insights here, going from of starting comments about security needing to move from this governance and advisory, to governance via product delivery.

The need to build security expertise within dev, whether it's the security champions that make that a part component. We've talked a lot about how you have to figure out the security business value.

And from there to understand "What's the value? How do you turn security into a business differentiator, and get that into the feedback loop of the business, and fundamentally do all of that within the context of the culture? To understand what makes the surrounding tick."

Which it sounds like the end goal is the same. It's still those same elements of embedding into the fabric, but maybe the business drivers and the what's doable and what's not changes per the organization.

Not easy tasks, but I think really useful perspectives to run forward with.

Justin: I really enjoyed this. I ramble on for 10 minutes, and you succinctly and accurately say "Yeah, OK. One sentence, this is what you're saying." That's fantastic. Thank you.

Guy: Communication matters. So, before I let you go here, I like to ask every guest on the show if you had one tip or one pet peeve you want to talk about to try and give to a team that's looking to level up their security calibre, to do security better. What would that be?

Justin: I love my industry. I love security. I feel that I'm possessive, "It's my industry," so to speak. "We're all in this together." But within that environment there's a lot of collaboration.

We share a lot within the security industry. We're open, we're accepting. So the one thing that I would say to a developer, is just reach out and say "I want to learn. I'd like to be a little bit better." You will be amazed.

To anybody, myself, anybody on LinkedIn, anybody that you see. You will be amazed at the response of welcome and participation that you see to level up education or anything that you might need.

Guy: That's a great tip. Seek out to learn, and you shall find. That's also-- You just tee'd this up, but I was just going to ask if somebody had further comments, feedback, questions for you to ask on the Twitters or others, how can they find you?

Justin: Justin@Somaini.net is my email address, I'm on Linkedin and my website's down but I'll be building that up again. I don't really do Twitter, but I'm on LinkedIn a bit.

Guy: That's good to know, you're a-social network. Justin, this was a pleasure. Thanks for coming on the show.

Justin: Thank you very much.

Guy: Thanks to everybody for tuning in, and I hope you join us for the next one.