Ep. #39, Build, Break, and Defend with Mohan Yelnadu of Prudential

Guy Podjarny: Hello, everybody. Thanks for tuning back in to The Secure Developer.

Today we have a great guest from Singapore, back in the Asian side of the world, Mohan Yelnadu.

He runs the application security part of Prudential. Mohan, thanks for coming on the show.

Mohan Yelnadu: Pleasure's mine.

Guy: So Mohan, before we dig into a whole bunch of topics of conversation, tell us a little bit about yourself.

What do you do, and how did you get into security in the first place?

Mohan: It started with building. I was primarily working at a university teaching application development, primarily for what are postgraduate engineers, and then I joined a startup company that I actually started.

From bottom layer and reached the application layer, where I'd defined kernels, I developed drivers, and then I went the application layer.

That's how I started working in mobile applications in the beginning and on the web application. I've been a builder first, then I moved to testing.

I became a "breaker," I would say, and then eventually I moved into the pen testing world where I was "true breaker."

Eventually I realized being in pen testing, the sphere of influence is very limited to that application that I was working with.

But if I have to make an actual impact, I have to move slightly towards the DevOps side, and that's how I moved into DevSecOps.

Now I'm a defender, so builder, breaker and defender now.

Guy: Yeah, no. That's a great journey actually, because you come from--

Oftentimes we try to draw analogy in security to the world of quality, and you've literally done that journey.

Going from assessing quality to assessing security, to now building security.

What is it that you do today? Tell us a little bit about AppSec and what's your role today?

Mohan: Okay. At Prudential at group level, we have different local business units and each business unit because they're based in different countries, there are local security compliances that we need to follow apart from the regional, at group level.

So we basically have recently kickstarted the DevSecOps program and we have deployed the fourth primary area from the software security through SAST, static application security testing.

Then we have open source software security, and third is the container security, and the final is they do DAS, dynamic application security.

We have chosen different tools and working with development teams to onboard their applications.

We have just kickstarted the onboarding process, so eventually we plan to go to different local business units and give them workshops on how to use these tools.

Then at the same time we are slowly building something called an "Ecosystem."

Where because we are very few in number in OpSec team, we basically plan to create this ecosystem which basically fills the gap between what developers want and what we can actually support, being small in number.

We are planning to put "How to use different tools," one of the frequently asked questions in terms of vulnerabilities and how to fix those vulnerabilities.

Then some small 5-7 minute videos about how to use the tools, how to fix those tools, and primarily one or two major technologies and then eventually the questions can come from different areas.

We will basically try to help them with some with some tests to track and close and guide them.

Guy: Got it. That sounds like you're thinking "Scale," so you're building the AppSec team to build supporting --

Like, do you call the team the "AppSec Group" or is it the DevSecOps? What's the identity here? How do you call yourself?

Mohan: Yeah. We have family. Within AppSec, two teams. One is engineering team, another is the operations team.

Engineering team basically does all of the R&D part, and the reviews, and they test tools and scripts and any process implements.

They do it in a limited environment, and then the operations team basically does the rollouts and education and tracking different issues around day to day life. Yeah, the daily activities.

Guy: But both of these groups support the engineering teams in doing it, or do you mean--?

Mohan: Yes.

Guy: So the engineering side is the one writing code or reviewing code, but really a lot of it is about working with --

It's not about the code s they build, it's about the codes the engineers build and keeping it secure? Is that a safe assessment?

Mohan: Yeah.

Guy: Or are they actually building solutions and maintaining services?

Mohan: Yeah. We in the application security team, we don't develop any applications, but we do have integration code.

Like writing playbooks for installing automated installation of various scripts, two sets of terms of, you know, maybe develop that part of integration code, but otherwise there are no applications that business into this.

That's a complete development activity. We basically support that.

Guy: Got it. No, that makes a lot of sense. You talked about the local businesses, but then it sounds like the initiative this DevSecOps initiative is not local.

It's a global initiative led by a local group, and then expanding into the other different Prudential subsidiaries or local businesses around the world?

Mohan: Yeah, that's right. At the regional level, we basically have developed the whole concept and set up different tools at a central level in the cloud.

And then we have different local business units, which use this cloud through their own 10m subscriptions.

These development teams actually, primarily the earlier focus was on the infrastructure security, and the very less focus was given on the application.

Although the real pen testing as part of compliance and other important tests exist, but AppSec as such was not a priority, basically.

So we started with this DevSecOps programs, more focused attention towards the application security.

Guy: And then you drive, you build that expertise locally and then you expanded and permeated out to others.

Mohan: Yes, exactly. We have our extended hands in the local business units through IT execs, IT security managers.

Some of the IT execs are really very good. Some are good in other security areas, so we basically tried to balance this out and we helped them build additional skills and skillset, and we work with local government teams.

Guy: Got it, cool. It's always probably both a blessing and a curse to be the trendsetter, to be the one driving and mobilizing the organization.

You have to shape a new future, but you probably have to battle some blockers. Do you find --?

As you engage, it sounds like you're in this interesting position where you are thinking ahead and you're thinking about how security needs to look over time, today and over time.

You have these two groups you need to convince or bring along for the journey.

On one side you have the development teams that need to be signed up to this, but on the other side you also have your peer or other security teams in different regions that need to do it.

Do you find people are resistant to this change, or they embrace it? What's your view of the overall--?

Mohan: Interesting question. What I've read and what I've come across is slightly different.

Normally when I read articles and blogs I get a feeling that developers are trying to register security part of it, but in here I found the developers do appreciate the importance of security.

Only thing is if it is transparent to them, they love it, because that's not slowing down the process to production. But if it is slowing down, they want to find better ways to manage this.

I've read another story that people tried to bypass it, but somehow I don't see that being here.

So in a way, I would say I'm lucky, but developers do understand the importance of supplying secure software.

But sometimes a business's timeline to take their products to market forces them to do it as quickly as possible, but that time you have to balance out.

How much you can put in production, and how much security in depth that you can reach.

So sometimes it can go in a tricky way, I would say, but again it's not the developers fault. It's basically the whole ecosystem around that, it's pushing them not to deliver faster.

But as an AppSec consultant my job is basically to simplify this friction of taking quicker and doing it securely.

My job is basically to simplify this and DevSecOps program basically, tries to do this.

So if I'm able to give the developers feedback early in the lifecycle and a constant and transparent one--

These three things if I am able to give to the DevSecOps program, I think developers would love to invent security and I don't have a different feeling that we are not enablers, but blockers.

Guy: Yeah, I love that. Spoken like a true partner.

Mohan: Yeah, exactly.

Guy: At the end of the day I think it all starts with people, and I think probably the worst way to approach it is with an adversarial "It's us versus them."

Really, you have to understand the context of a developer. They're being demanded to deliver function and speed and the likes, and you need to roll with that and you need to adapt to their surroundings.

So maybe, let's get a little bit deeper into detail. How is it that you engage with the developers?

I know there is security champion programs, you mentioned a little bit about automations. How do you work your team with the dev side?

Mohan: There are multiple areas in which we need to focus. It comes to onboarding developers to do secure development.

One is definitely the tools, an early feedback. If you have automated pipelines to whatever orchestration tool or platform that you use, early work gives them a sense of where they stand in terms of the application security posture.

But at the same time, that is not enough. You need to have a process around supporting them, as you rightly said, not being partners.

So that's where we need to build a good relationship across different development teams from different regions, and that's what I'm trying to target.

My main challenge in this process is basically the people, if we had enough number of AppSec folks on the team, it actually helps building that relationship faster and we can deliver the requests earlier.

But having said that, we know what our challenge is in terms of getting a good number of people in AppSec domain.

We need to go for the third option, basically the ecosystem. So first, tools. Second, the process, and then third part is the ecosystem.

That's how I prefer to build that whole relationship. The ecosystem basically helps the progress without the AppSec consultants on security.

Because that's the CISOs these days. Since we have hundreds of developers spread across different local business units, building an ecosystem basically helps them reach their destination faster.

Say for example, look at the static application security testing tool. Take any tool in fact, in fact design and develop part of the one of the leading secure code analysis to compiler.

I know that domain quite closely, so I know that this domain, although it has been there since three years or more.

But the inherently, the core problem of this type of code analysis, either for quality or for security, is the false positives.

Because you can not develop models to understand the technology, because there's ways in which developers can use the technology to deliver business value.

But in such cases, the consequence too is an obvious byproduct, and more than a number of false positives, less the faith the developers have in their tools.

It's very important to iron out this part very quickly and in a smooth manner.

As part of the ecosystem, we plan to build this stuff 12,000% false positive list coming from a particular application for a particular tool.

And then I want to say, "Look at your results and look at this table. If you see these issues in your application straight away, go ahead and monitor the false positives.

You don't have to worry about fixing it, we will not break your build in any case."

So that's one level up, resolving or reducing the number of issues to be resolved. Secondary is basically sometimes there are possibilities with a peculiar situation.

For example, there's a CV reported on a library, for example, an open-source library.

Then there's a caveat saying "If you use this API in this particular library, then only the CV up the cable. Otherwise, you're not."

So if I have a dependency and then there is a transitive dependency on it, and if I'm not using this API from that transitive dependency, I am not required to fix this issue. Basically, I will not ever need it.

But in such a case, we need to build that confidence of the development teams that, "We have to analyze this CV. If this doesn't apply to you, don't worry on this."

But at the same time, because it's a robust organization, there will be another thing which would be affected by the same CV, but they are well, their applications would never--

So we need to balance this out, and this has to be done on a quick basis because the more longer you take time to resolve this clarification, you are actually going away from the development teams.

This is very crucial. This analysis for any new VC, because imagine the severity of these vulnerabilities on the higher side.

Drop and leave aside everything and then focus on this analysis, and clarify to the most types of development teams.

"What are you? This doesn't apply, or for you it does apply. Okay, you better fix it. Or you have some other problems in placement, or some kind of monitoring.

Guy: Right, another mitigation technique.

Mohan: Exactly, yeah. So that's what we mean by "Build the ecosystem around them," so that there's less and less friction.

Guy: Yeah, no. It makes a lot of sense.

I think there is a lot of conversations in the security world today that goes from maybe a perfectionist perspective, that says "I want to report and I want to tell new developers about all the problems, and you should fix all the problems," to being much more diligent and understanding that there is only so much time and so much attention that you can demand of developers for security issues.

You have to use that time wisely and ensure that you really invest in using their time well in avoiding issues. Sounds like a really good thinking.

Mohan: Absolutely, yeah. The extension of DevSecOps program through some mechanism where they get feedback, and at the same time there is something which controls the whole delivery to ensure there is security in production, which is to build better programs.

So build better programs, I think you test up on that part. It has to be implemented in a manner in which it is pragmatic, so every tool should give different types of vulnerabilities, like some may give critical, high, medium, low.

Some will give threat level ten, nine, eight, seven and so on.

When we target these breaking the build program, once they're comfortable with the whole ecosystem we need to be considerate, as you rightly said, of their available time to fix these issues.

Developers are not there just to fix issues, they are there to deliver business features.

So we have to balance this act very carefully .

So the way we plan to test and deliver this is maybe in 0-6 and allow them to scan, get the fish back, and have a clear understanding.

But at the end of the third month you basically would like to tell them "Your build is going to be spoken for. If not all the criticals, maybe some core criticals."

For example, a simple injections. Maybe process the thing or any other injection issues.

That gives them enough time, basically, to look at subset and then focus on fixing it actually.

Then eventually after maybe nine months from the start, you basically come down and say "No criticals allowed now. You had this much time to fix it, now no criticals."

And then eventually in the next step you would say, "Now that you're focused on critical issues, now that you're done. This was down, and say no high issues." Enough, and so on.

We give them time enough to get comfortable with the whole ecosystem, with DevSecOps tools and scanning, and then we give them "You start from the top code," and then slowly increase it, so they have enough time onboard to delivering business features as well as fixing the issues.

Guy: Yeah, sounds like a good way to roll out. You have to stagger it, you can't just come in and be a bull in a china shop and just break everything, you have to be wise about it.

I do feel in the world of open source defenses and VCVs, another practice that I'm fond of is this separation between a new vulnerability that comes along and a change.

This notion of establishing a gaze that is more forward looking, so you have all these vulnerabilities right now, but can a break the build in whatever fashion that --?

Whether it's in a pull request or in a build, or whatever it is. Only on a new vulnerability that you've added, because you've modified the co-changes, so you don't necessarily just go backwards.

But it's also a stop the bleeding and draw a line over here, and then onwards. I think, Mohan, you've already touched on a bunch of those.

But I want to make sure we don't miss them because I like that framework.

You've written about this notion of the six principles of continuous security, and I liked the way it was organized.

Maybe what we can do is we can go through them and see-- I think some of these we already elaborated and talked through them. Does that sound good?

Mohan: Yeah. I would love to do that, in fact.

Guy: Sure.

Mohan: So this was done for one of the conferences in Europe that I visited. V6 multiple family focused tours, one clear message that irrespective of the passion or the budget that you have, you can have continuous security.

Basically it's exceedable for all the organization , the district of the site, etc. You can still achieve continuous security.

The first principle basically talks about "I code, I predict." Which basically means, "I'm developing something as the developer, and it's my duty to have some security protection and develop that as part of the whole module development."

When I say that at the same time I'm saying "As a developer I should not expect another layer, or a higher layer to protect my application."

For example, DAS. For example, real time application self-protection or DAS.

"I shouldn't expect these controls to protect me. As a developer, I should have any code that I put in production should have both the business value as well as the security built around that."

So that's the first principle, the second one is basically that "Security is everybody's responsibility."

So be it a developer, it's not just the developer's responsibility.

It is the responsibility of the SCRUM, it's the responsibility of the product owner, it's the responsibility of the people in the leadership as well as people in the marketing area as well.

Unless everyone is aware of this these are very challenging otherwise. Leaving security only to development teams, it's a walking disaster.

Because people -- The hackers have found different ways to get into the network, through maybe deceptions of e-mail in order to enter by e-mail force.

Or through some other ways within the ecosystem, either through the application layer or perimeters.

So there are multiple ways, unless everyone takes the responsibility of not having enough awareness about security, things will get really difficult.

Guy: I like how that one very much complements the first one. Our VP sales has a phrase that says "Accidents happen at intersections," which is--

I like that phrase for the context of security, just as we discussed. Like, "Everybody has to take this."

Just like developers shouldn't expect somebody else to do it, that someone else also shouldn't expect them to just be perfect at what they do. And also they are--

Mohan: Exactly.

Guy: Everyone's responsibility. Love it. OK, so we've got one and two, let's go on to number three.

Mohan: Sure, OK. Security through automation, simplified security through automation.

I love this principle because these days everyone expects to deliver whatever in the shortest time.

We came from waterfall background to DevOps, CICD, etc.

The prime factor here is everyone is trying to reduce what is going to production.

Instead of 10 features, I'm actually developing the MVP and then adding one or two features within the lead, or enhancing those features in the subsequent releases.

When they are simplifying from the business feature side, I think it is our duty on the security side to simplify whatever security process or tooling or any other procedure through automation.

Automation can be brought out at different levels, so I see primarily three areas.

One is automation through onboarding your application in your DevSecOps program. If it is smoother, people start adopting it quickly.

The second thing, the false analysis of the ties process, if you are able to automate this through either configuring tool itself or having the mechanism where they can address these issues.

And the third is the metric submission, so metric submission if you are able to automate, that gives them a very good feedback in terms of where they stand from different dimensions of security.

The static analysis of the code that you develop, the code that you use, then the containers that you use, the ecosystem that you use.

Like Kubernetes or so as a number of the inputs coming from manual pen testing.

So if you are able to put these in together in one single pane of glass, the point is if you have any security dashboard the developers in one shot get very quick feedback and then they can analyze.

"Last series we had this much vulnerabilities in these areas, and now we are progressing way ahead because it shows the graph is not going downwards."

By chance, it is going up. There could be multiple reasons. Increased code, new features, multiple new features. In that case we can get a sense of how they have to act upon if the trend is going in an upward direction.

They can think of all this only when they are aware of this, and to be aware you need to have a single dashboard in a manner in which they can consume it.

The automation in this area is something that I really see boost the overall security posture of the applications across the group.

Then the fourth one, "Build ecosystem and build confidence." This is already impressed upon.

Guy: Right.

Mohan: You can build an ecosystem around many different areas, because the number of AppSec consultants in their field is not proportionate to what you'd expect, basically.

When you are not there, the offline developers should be able to get their issues resolved and their issues addressed.

If you are building an ecosystem of documentation to smaller videos about how to use the tool, or how to address a particular vulnerability, or frequently asked questions about the possible list of issues coming from different tools.

This is what I mean by "The ecosystem." The moment I have found personally that say, for example, in the previous organization I started, I started with an ecosystem and all the people I observed, for almost two years.

Developers started using the whole ecosystem and then the feedback coming from tools, and eventually they feel more comfortable with delivering security inside the business features, and they feel more confident about the deliveries.

Guy: They grew. You taught them, so they're better at their craft right now. They're able to do it, so they're happy. You're giving them something back, right?

Mohan: Absolutely. In fact that's quite gratifying, by the way. When you see that confidence in the developers' eyes, more than anything, I would love to see that.

Moving on to the next one, get feedback as soon as possible from all sources.

We step on it as well. Coming from those 4 areas which I talked about, the static, dynamic area, container security and the OS scanning.

In fact I like to try to integrate other areas, and I'm planning to do that.

One is the back rooms, if you are able to get walk through set for applications, you get more visibility in terms of feedback about that and what attacks are happening on your application.

The vast compliance with the SOC, not the security office center.

SOC people would be able to give us some insight in terms of "In this period of time the attacks were primarily focused on a small injection."

Or, "In this period of time the attack was primarily to the network and not on the application layer."

These insights basically ought be there to grow the understanding on how the hackers work and how your application should be secure.

The feedback coming from different dimensions actually helps developers mature for the future.

Guy: Yeah, no. If what you're doing is any good, you have to be able to measure.

If you can't measure it you can't optimize it. You have to improve it.

Mohan: Absolutely. And the last one is the focus on process improvement.

That is a continuous process, it's not like you have a DevSecOps program and then you're done. It's never like that.

The process is very crucial because I have seen, if you focus towards the end product you would be able to achieve some significant value, but you will not be able to reproduce successfully in a longer beta and the start of this manner.

But if you focus on improving the process itself, that is making it light and making it adaptable, making room to improve by itself through some other feedback or other mechanisms.

If you focus on the process, the quality of that end product becomes a secondary factor. You can predictably make it successful with every delivery if you focus on the process.

I lack one foot from this lady, Dr. Carol Dweck, maybe you should have a session with her on growth mindsets.

Really, although her domain was completely different when she talked about the growth mindset, but I love that idea and I tend to apply that in not only problems but process improvement.

The process can live in our domain, it can be applied to data science, it can be applied to security, it can be applied to business processes.

So what she says is "You should embrace failure in the journey. That means don't point fingers when something bad happens, instead you say 'This is an opportunity to learn.'

So, embrace failures and take it as a stepping stone and improvise the whole thing."

Then inculcate the desire to be challenged. Now, this is very important.

There is a limit to a culture involved in this, but this culture can be worked upon or repeated over time.

In certain cultures hierarchy's very important, where you cannot challenge your superior.

Guy: Yeah, your boss or your managers.

Mohan: Exactly, yeah. But I've found-- Because I was lucky to work in different cultures so far in the US, Australia, Europe and now in Asia.

I found some of the European cultures are really nice in this respect.

Where they say, "I may be the CEO, but if you are a developer and you have some suggestions, come and share it with me."

And then that feedback goes into improving the process. Culture should enable being challenged as a feature in its DNA.

Irrespective of where you stand, if you're able to give good feedback by challenging whatever existing, I think that's the best thing that can happen to improve the process.

She says, "Believe in process." That was what I was talking about.

So one should not focus on the end result, but if you are able to focus on the process you are definitely going to end up with a better result.

Guy: Yeah, no. Really good principles. I love how, again, they touch one another but a lot of them are about ownership.

A lot of it is around simplification and making it easy, like if the security is too hard people won't do it. You have to make it easy.

Then I think with the whole last few principles in general, it's business in particular, but also the broader set.

A lot of conversation around continuous improvement, "You're never done," and that's a good thing.

You embrace that you're never done versus being-- But you also need to have the mindset and the perspective for it, but you also need to staff accordingly.

It's not a temporary project that you're going to hire some people for.

You have to understand that the people here that need to be constant service providers and enablers, and be on top of the next technologies, threats and the next evolutions, and just changes in your own work.

What doesn't work for it, and adapting that.

Mohan: That's true.

Guy: I love the principles, Mohan. Before I let you go here I like to ask every guest coming on the show, if you have--

This is a wealth of information, but if you have one bit of advice, one either pet peeve about things you're seeing people do that you think they should just stop doing.

Or, one thing that if you embrace this you're leveling up your security, whether it's on the dev side or on the security side or organizational.

What would that bit of advice be?

Mohan: I have two figures, not one but two.

Guy: That's okay.

Mohan: One thing, the top leaders maybe can be the CSO or CISO or CTO, from them if the security message goes down it is easy to percolate.

Because you get funding, you get processes resolved, you get mandates.

So all these things are very conditional, if the message comes from the top.

The second aspect is basically irrespective of which nation that you come from, what domain, you're not the banking or financial institution to work on security.

That is serious and you can be coming from any different domain of industry.

You can actually achieve continuous security, only thing is you need to take baby steps, but you need to take it continuously and you will definitely achieve continuous security, for sure.

Guy: That's great advice. "Get going, get started. It matters to everybody. Everybody can do it and you don't have to start from switching from nothing to perfect, just make a step in the right direction."

Mohan, this was great. Thanks a lot for coming on the show.

Mohan: My pleasure.

Guy: Thanks everybody, for tuning in. I hope you join us for the next one.