Ep. #13, How New Relic Does Security

Guy Podjarny: Hello everyone, welcome back to The Secure Developer. Today we have with us Shaun Gordon from New Relic, he's the Chief Security Officer at New Relic. Thanks for coming on the show Shaun.

Shaun Gordon: Thank you.

Guy: So before we get going, do you mind giving the audience a bit of an introduction about yourself, what do you do, what's your background.

Shaun: Sure well I actually started my career way back when as a software developer actually doing IBM networking to HP mini computers. So many years ago. But moved through a whole bunch of roles and sort of ended up in security almost to accident about 12, 13 years ago when I was sort of helping deal with a security incident at Intuit where I was working back then.

And it turned out to be something I was really interested in. Sort of moved into security, worked at Intuit for almost 10 years in security I think there. And then I joined New Relic five years ago, as the first security hire when the company was pretty small, about 140 people, and built up the entire security team to the point we are now with about 15 people focused on security.

Guy: Cool, interesting, the switch from dev to security was that well embraced in Intuit at the time?

Shaun: Yeah, so I actually made the shift, it was almost to an operations role, where I was sort of looking at the health of the overall website for Intuit.

We had a security incident and there weren't a lot of security people and no one knew how to handle this and so I sort of took it on and it just became a passion for me.

Guy: Cool, I think companies tend to embrace anybody today moving into security and it's always this, first of all, most organizations would have a whole bunch of open reqs in security because talent shortage is a problem, but no, I think maybe 10, 15 years ago that would've been a little bit more novel.

Shaun: Yeah it's actually interesting when I talk to a lot of my peers, they come from different directions.

I was never that sort of traditional hacker who liked attacking things. I never got those skills formally anywhere. I just sort of backed into it and really have approached security as a developer, as a defender from day one.

Guy: Yeah, makes perfect sense in fact and how spot on for this show as well. So I guess while we're at it, can you tell us a little bit about how does security work at New Relic, just sort of the headlines and clearly, a lot of the focus is on dev and how is the dev process involved with security.

Shaun: So we have a central security team as I mentioned, it started with me as a single security person five years ago. We added an app sec person which was something I needed to do, but it was actually a sad day for me because that's always my passion so I was sort of handing that off to somebody else. Since then have built the team up into really three areas. I've got my app sec team, which I'm sure we'll get into a lot more and sort of what they do.

I've got compliance team that does sort of traditional compliance activities, you know, our SOC 2 certification, FedRAMP, SOX, all those sort of things. And then I've got sort of that traditional infrastructure IT security team, that focuses on both the security of our product itself, the infrastructure, data center security, as well as our corporate IT security.

Guy: Okay and it's a 15 person team right now, and you're split, right, you're distributed?

Shaun: Yeah, so it turned out when I joined the company, the company is fairly evenly split between San Francisco where we're sort of headquartered and Portland where we had most of our development and support. I had built the team here, and I assumed most of my hires would be here. It turns out I only have myself and two other people in San Francisco and really have built up the team all in Portland primarily.

A large part of that is because our development organization is there and it really makes sense to have the developers working very closely with my application security people. As well as, we've had really good luck finding really good talent there versus in San Francisco area, there's a lot of competition.

Guy: Yeah, definitely. I think in all fronts in security probably is no different in that front.

Shaun: So really, right now the majority of the team, the app sec team, the infrastructure team are pretty much primarily located up in Portland working with the developers. You know, that's been a great thing for us because they act as developers. They attend the same meetings, they tend to follow the same processes. And although we're a completely separate organization, from all outward appearances, they are developers.

Guy: Hmm, yeah, interesting, so I think that probably brings us into sort of an opportunity to dig into the app sec process but I like the starting point of basically prioritizing security's physical proximity to dev as part of it. Okay cool, so that's sort of the overlay team, you know, how does application security work?

Shaun: So we try and be as lightweight as possible with developers.

I have a philosophy for my team, which is I want to change the way we do security to fit in with the way the developers perform their job.

Instead of trying to get them adapt the way they work to what we're doing. And so that means a lot of what we do, is as transparent as possible, we're trying to do it in the background. So we do a lot of lightweight processes. So some of the examples are when they create a new ticket basically, they're going to create a new product a new feature, they generally create a JIRA ticket for that.

We have tools that sort of automatically monitor that, alert us, and then we can now then go and ping those developers and ask them a few basic questions about what they're doing and decide how much we need to get involved in their process. And in some cases, we just have a short conversation with these developers and we say we're done, you're good, here's a few things to worry about, but don't come and bother us.

Other cases we say we have to dig deeper. We have to go and do some sort of threat modeling and really understand what they're doing, provide a lot of guidance during development process. We also do a lot of monitoring of their commits and things, we have things that we do so automated tools that look at the commits, and try and find things that might be a risk, methods that we know have caused problems before, comments that mention we're doing encryption or passwords, that sort of thing. And monitor that sort of thing, and when we need that, we dig in further with them as well.

Guy: Yeah, that's pretty cool, so basically you're monitoring developer activity automatically so that they don't need to worry about that and they might proactively initiate something but you can track those and then that alerts you to decide whether you need to intervene?

Shaun: We do want to train our developers to reach out when they need to and we've done a lot of things to make it easy for them to do that. But

We want to create the processes so we know when to get involved so developers don't have to spend a lot of time thinking about when they should get security involved. We want triggers that let us say "hey, you're doing this thing, we should talk a little bit."

Guy: Yeah, cool, I think it's really cool, it reminds me a little bit at Snyk we try to surface vulnerabilities from the world of open source GitHub and we track all sorts of activities probably slightly given the volume of traffic. You know, slightly more specific mentions around somebody opening a GitHub issue or the likes, that mentions a potential vulnerability. But never really occurred to me to apply that in the context of a specific organization to try and be an alerting mechanism of sorts, I guess, for the security team to intervene. How often does this catch something?

Shaun: So I mean, I think we have a pretty high level of engagement with the developers, I think with most of our products, we are getting engaged. So we are seeing early on what they're doing and getting engaged, so I think from that standpoint, we're catching most of the products that we think have a security risk. There's very few times when we're actually surprised by something going out the door, I won't say we're never surprised, but it's very few times generally.

Guy: Cool, so this is like security monitoring and maybe proactively engaging and there's a reason you're sort of assessing this risk assessment. What about the dev side, what do you do to sort of empower or educate the devs themselves on security?

Shaun: So we do have a certain amount of training, we try to push a certain number of resources out to them so we have our website, our Jive site, where we basically put guidance for developers in general. We're not doing a huge amount of formal developer training and I actually have mixed opinions on developer training.

I've seen some developer security training done well, but for the most part I haven't seen anybody being really successful with it.

Put them in front of this classroom, have an instructor teach them about all this secure coding practices, and within two weeks, it's basically all gone. So my real focus is how can we catch those things without having to train the developers. I don't want the developers to have to think about security that much, you know, static analysis tools, that sort of thing.

Static analysis tools are actually a real challenge for me, so we do a certain amount of that but one of my concerns is if I want to be transparent, I have to make sure I'm not scaring developers by giving them huge lists of unactionable results. So figuring out how to implement this sort of tool, this sort of monitoring, produce actual results I can get to developers quickly, without having them having to become security experts.

And so that's always been a challenge for me, and that's why we do some of the very lightweight things like the automating monitoring of the GitHub, commits, that sort of things that I mentioned before.

Guy: Yeah, no I think that makes sense, I think that's been the bane of existence for static application security tools.

Shaun: Yeah I'm not going to mention any names.

Guy: I built one in AppScan, that was an acquisition but then before that, developer edition, and it's been a challenge doing this static analysis it's just fundamentally false positive prone and finding the ways to not make it false positive prone. There's gems in there, but and yet there's a lot of noise around.

Shaun: And I'm using that sort of as the poster child for

A problem I see in general is tools that can't really decide if their target audience is a security professional or a developer.

And I think, looking at the industry and the tools around, I see that challenge all over the place. Are you trying to produce very detailed results that can be used by the professionals but then interpreted for developers or are you trying to produce something that's aiming at just the developers. And that latter half is where I think we really need to move as an industry, and where I haven't seen a lot of people do it really well yet.

It's very easy to turn a developer off of a tool very quickly by giving them unactionable information, by calling them out on something that they don't understand what it is, and more importantly, how to fix it.

So that's been a challenge I've seen and it's something I see we're getting better at and I think people are starting to decide, but every time I see a project pitch for seomthing, I can tell they're to be two things and they need to just choose one.

Guy: Yeah like I am 100% on board. That's kind of my philosophy all around.

Shaun: Yep.

Guy: I think it even goes beyond probably that. It's not just about actionable and false positive, it's also about is it onboarding and how well does it integrate with your workflow.And just the cost of ownership the effort level of using it has to be on par competitive with other developer tools. Which is a pretty high bar, I mean, those are good tools and you have to invest in developer UX not just in an auditing tool that happens to run in the context of a developer tool.

Shaun: Yeah I agree, and that integration is key. Adding another step, two steps, three steps, whatever it is to my development process, I'm going to fail because people are either going to get really upset. The fact that I've had, you know, making them do all these extra things, or they're not going to do it. And either way, I'm not getting any value out of this, I'm either ruining my reputation within the company, you know, this is a team that just keeps imposing things on us, or like I said, I bought this tool and nobody's going to use it.

Guy: Right because everybody knows. So yeah, fully in line there. So moving kinda further up that stack, so this is developers building hopefully secure applications, what happens, who owns the alerts if there's something, you're at New Relic you're sort of well familiar with alerting mechanisms and dashboards, who looks at those dashboards?

Shaun: So that is actually my team, the app sec team primarily, as well as the infrastructure team. But we are the ones who are sort of looking at those alerts, triaging when we're seeing things, and then basically translating for the developers. And once again, because

I don't think the level of alerts we're getting through any tool, no matter how good the tool is, are clean enough to send directly to developers.

And so, we have a lot of alerting, we actually use a lot of our own software for doing that, some of our own Insites product, a lot of the alerts go in there, flag us for various things. And a lot of them are feeds from sort of traditional tools and scanners that we're either using as a SaaS service or brought in house, or in some cases they're very custom built monitoring tools that we've created ourselves.

Guy: Cool, yeah and I think that very much aligns with the previous point right, if it's fully actionable and it doesn't require security knowledge, you can put it in the hands of the regular ops team, as long as you need security expertise and/or there's too much noise, you have to sort of bear that burden in the context of app sec.

Shaun: I mean, that's always sort of one of my mantras, it's something I always push my team towards, is make sure that if we're telling anybody anything about the security of their product, that's actionable. And that applies to those sort of alerts that we're sending around security vulnerabilities, and it even applies to things like executive level dashboards. So

If I'm going to create an executive level dashboard that says that this business unit or this group is red because they've done something bad, there better be a very clear path for them to go from red to green otherwise I'm just pissing them off.

Guy: Yep, so this is all very practical in, sort of, the logical world of software development and security. Compliance and regulation don't necessarily always follow common logic, how do you find these practices work with needing to be compliant you know, you're a big public company now.

Shaun: Yeah so, I mean, we've been SOC 2 compliant for four years or so, now we're a public company we have to be SOX compliant, we're going through FedRAMP compliance. So I'mpretty familiar with the compliance product. And it's always a struggle in my mind, and probably one of my biggest struggles is exactly what you asked, how do we be compliant without interfering with all these processes.

And I actually did an RSA talk a few years back about how to do this in the SOC 2 world. I think my major points are really that our job is to educate and manage the auditors and help them understand how the industry is changing and how some of the things we are doing now, not only are equivalent to what we used to look for in the old world, but are in some cases a lot better.

But it's not simple to do. Basically the typical audit process in my experience is they come in with a big long list of, "these are the requests we have, this is the way companies generally deliver this, this is the sort of evidence you provide, provide us this evidence."

And a lot of times what we need to do then, is understand what is the real question, what is the real control they're trying to solve, not what is the evidence they're asking for. And

Where I've seen audits fail a lot is when the security team takes that list that the auditor is asking for and just then hands it to the developers, hands it to IT, hands it to HR, rather than taking a look at that and saying "wait, this is what they're really asking for, this is what we're doing in our environment."

And one of the big challenges I've seen in that, is that in order to do that successfully, you sort of have to have a foot in both worlds, you have to have a foot in the compliance world and you have to have a foot in the development technology world. So in order for me to understand what the auditor's really asking for, I need to understand the compliance world. But then to understand what we can actually deliver that will meet that, I need to understand our development processes.

I need to understand our technology stack. And that's been one of the biggest challenges that I've seen. And I don't know that there are great answers for that, but it's definitely I think what we need to do. We've been, honestly, pretty successful with this. There are very few controls that I can think of right now, and I'll give you one example in a minute, but that we have implemented purely because of audit.

Purely because of compliance, where I don't think it's actually adding any real value. The big one, I actually mention this in my security awareness training, is password resets and password requirements. You know, passwords must be reset every 90 days, passwords must consist of eight characters and mixed type of characters. That's basically not complaint.

Even if you look at the latest NIST guidance, they're saying this doesn't make sense anymore, we're using multifactor, there's no real reason. But this is one we just haven't been able to push back on yet. Not that I won't keep trying, pushing on the auditors saying hey look, this is better than what you're asking for. But we're not there yet.

Guy: Yeah, well I hope everybody after that talk went up and bought you a beer because the notion of educating an auditor, people get a shiver.

Shaun: Yeah and to be honest, it really depends on the audit. I mean, the SOC 2, we've been very successful on that because a lot of companies are sort of, dev ops, very agile, fast moving are doing SOC 2, the auditors get that now.

Other audits, it's a little more challenging, I mean SOX, they've got you a little more over a barrel there than you do for the SOC 2. SOC 2 we're doing because it's voluntary and we want to do it for our customers. SOX we're doing it because we have to do it. FedRAMPS even a different story, in that case, we're doing it because we want to do it but you're also dealing with the government so there are challenges there, there are rules that have to be followed.

Guy: Yeah and I think hopefully the process here is indeed an education exercise that eventually auditors would have a challenge with the first time they encounter, the second time they encounter, when it's the 15th company that deals with it in a certain capacity. And I think we've seen this with the cloud security, where a cloud, as a whole, like running something in the cloud, used to be a no-no. And you would have to educate the auditors to deal with them.

And now it's fairly well understood to the extent that, you know that's also maybe a frustration they might even have the specific AWS IAM thing that you need to set up in their list of prescribed steps.

Shaun: Yeah and one of the other challenges we're facing or have faced with audits too, is just like development world, dev ops world, it's all about continuous improvement.

Guy: Yep.

Shaun: We're doing the same thing with the security team, we're changing in some cases, the way we do things. And in theory we're doing them because it's going to improve our processes.

Every time we change things, we're now going to have to re-educate the auditors and perhaps even take a risk that they're not going to buy into this new process.

But I honestly believe that I'm doing this, I'm not going to drive what I'm doing for security, for compliance, I'm going to do what I need to do to make us secure, to make our customers' data secure, and we'll figure out how to make the compliance follow.

Guy: So first of all, I love the notion, even the terminology you used, it all comes back indeed to your background around being a developer and coming into security from the development approach and how you built there. One thing that stayed with me though from your intro was that New Relic was a small 140 person company when you were the first security hire.

Shaun: Yep.

Guy: Can you tell us a bit about the evolution there, because on the one hand, you know, that's entirely typical and normal and we've had Kyle at Optimizely here and we had the PagerDuty team and it takes awhile until a full time security person gets hired. On the other hand, 140 people is also, to many people, is not that small, and you're already processing a pretty substantial amount of money and data at that time. How was security handled before and can you tell us a little bit about how was the evolution of security at New Relic?

Shaun: So I was actually surprised when I came in that, well definitely nobody had security in their job title. I don't think anybody was really even consciously thinking about security sort of top of mind. But I was surprised at how security conscious people were in general and how little, I mean to use lack of a better term, sort of clean up I had to do when I got there.

It's actually an interesting story, so when I was hired, one of the impetuses for hiring me was the company was growing and realized we need to start thinking more about security. But I think probably the biggest impetus is, we had promised as a company, to get SOC 2 certification. And I joined the company I think about a month before the auditors came, basically to get us through the audit.

Guy: That's a good trigger, yeah.

Shaun: And it turned out that

I was able to actually take a lot of what we were already doing which were good security practices, frame it in the terms that the auditors needed, and we were actually able to get through that audit with no findings.

Basically an unqualified report. It's always a terrible term these people get really unqualified sounds bad but an unqualified report, a SOC 2, that's a good thing. And we were able to do that because we were basically doing the right things before. People were focusing on good practices, good development practices which in a lot of cases is good security practices.

So a lot of what we ended up having to build was just sort of more formality around a lot what we're doing. So being able to actually formally check that we're doing these things, continuing to do them, actually starting to look for vulnerabilities instead of just assuming we weren't creating the vulnerabilities, that sort of thing. So once I joined, I think a big part was just sort of now being able to actually sit down with the developers and really help them as they architect the project and get in early, versus before it was just sort of hoping they'd done the right thing through the process.

Guy: Yeah, I wonder how much is there, even, like there's a lack of expertize but there might even be some sort of reflecting away the responsibility when there is a security person, when there's no security person, then there's nobody else to sort of, I don't know if blame, but sort of expects to have assumed this responsibility, so you have to take it on.

Shaun: That's a really interesting point. That's actually interesting because that gets back to, you know, there's two perpetual arguments I always have with other people and in my own mind in security. And one is sort of, where does the security team sit, the application security people, do they sit on a security team, or do they sit with developers? And two, should the application security team be developing security code, should they be coding, should they be helping fix the vulnerabilities?

And one of my arguments against having them help fix the vulnerabilities, do that coding, is always, it starts saying that security is the security team's responsibility and not the developer's responsibility, so it's a fine line.

Guy: Yeah, no I agree, I fully agree with that. I think vulnerability is a bug and it should be fixed, it should be prevented, it should be regressed, it should be treated just like any other bug. The difference between it and a functional bug is that it's an implication of a control that you did not have more often than not versus maybe the functional bug which is, you know, like it's intentional behavior that you are now not supporting and with security, it's the unintended behavior that you are looking for.

Shaun: Exactly.

Guy: Yeah and I think the size of it, I must say when I started having some of these conversations, if you asked me before what size company do I think for startups for instance, they should hire a security person. I would've assumed or would have recommended, smaller size, but it seems like that rough size company that is in the sort of 100 to 200, clearly depends on the nature of the business as well, seems to be where a security person comes in.

And maybe it comes back to that tooling, the better the tools get, the more they allow dev teams to sort of build security practices without necessarily that focal point.

Shaun: Yeah I would absolutely echo that. And

I get asked this question a lot is, "when should we hire a security person." And a lot of it really does depend on the nature of what you're doing. If you're a company that's collecting some really sensitive data handling credit cards, Bitcoin, whatever it might be, you should be hiring somebody as your 10th, 12th hire or something like that.

But for a company larger that's handling less sensitive data you want to focus on a lot of the other things, the stability, the feature set, and just provide the developers the tools and as we talked about earlier, there's a lot of these tools that are becoming more and more developer friendly, so they can handle a lot of this themselves.

Guy: Yeah, I think this is, to an extent, an evolution of the whole dev ops and dev ops sec revolution. So maybe on that topic, New Relic is oftentimes at least in my mind, I think in many, one of the pioneers of the dev ops notion, right, it's definitely one of the enablers of it. And in fact, many many teams today don't have a dedicated, or you know, fairly good sized companies don't have a dedicated ops team.

You know, they might have dev ops as a concept and they clearly do ops work, but that ops work is done embedded in the application. How do you see, maybe in practice, but also where should we strive to the evolution of securities, is it along those lines, is it different?

Shaun: No I do believe, the two things are getting security close to the developer both in terms of the tools and process they're using and in terms of time, making security information available to real time, so as soon as I do my commit, I get some sort of information about what I might have done incorrectly, where I might be introducing security vulnerabilities.

As soon as my code goes to staging, it's scanned immediately and I get real time notification that there's something there. And that's the way we're going to need to scale our team. I mean

Automation is how I see my team scaling and I see it's the only way we can continue to keep up with this very rapid development cycles. The number of developers is growing and we're never going to be able to scale with them.

Guy: Yep, without the, yeah. So pushing back on that a little bit, I mean, when people talk about dev ops, automation and maybe controls and visibility, we're sort of key on it for sure, but many people would come back and say well dev ops first and foremost is a cultural shift. Yes, automation was sort of a key enabler for it, but it's a culture shift about understanding developers, understanding that they need to build operable software and ops people understanding that they can't point the finger at developers.

But that they need to be yeasayers you know, not naysayers in the company. You know, what comes first, the chicken or the egg here in security, I mean, is our primary barrier tooling and automation, or is it more our culture.

Shaun: I guess I haven't thought a lot about this one. But I mean, I think the tooling has to be there in order to be successful here. And I'm torn, the reason this is hard for me to answer is I'm torn between that whole wanting security to be as transparent as possible and wanting developers to have to think about this. So I do want developers to understand that they are responsible for the security of their code. You can't throw it over the wall to the ops team and say support this, you can't throw this over the wall to the security team and say secure this.

So they have to understand they are responsible for that security. But at the same time, I don't want to have to make them security experts because that's taking away from their day jobs and the security skills, it takes a long time for a lot of us to learn.

Guy: Right.

Shaun: And it's not the best use of their time.

Guy: Yep, and I guess that does come back to both what you mentioned previously about having a path to green. But also the dev ops side is you can demand and try to set up developers to own security but it's only if you have a path to green there.

Shaun: Exactly. We've been discussing a lot within my team recently about things like score cards, dashboards, that sort of thing. So maybe

We're not consciously telling developers to do something, but we're going to show developers their security scorecard, "this is how well you are doing with your product" whether that's a combination of the number of vulnerabilities they're introducing, or what it might be, I don't know yet.

But I think that's a path to helping them understand that they are responsible for security, that what they do impacts the security of their product and the company in general.

Guy: Yeah, I think these types of mechanisms are critical in my mind, the security is naturally invisible, I mean you use the word transparent in the sense that it's not behind a curtain or something, you want it to be a natural part of their flow, that they do it by default. But at the same time, security's too invisible, right, there's no natural feedback loop for security except at best actually, failing an audit and at worst getting hacked.

So there's no like, medium pain that you feel before the big pain that would kind of move you or push you to the right behavior so we have to give some mechanisms that give you that visibility over time.

Shaun: Yep.

Guy: Cool, so this is fascinating actually I have like a whole other slew of topics to dig into but I think we're getting out of time and so before I let you go, I always ask my guests here for one tip or if you were thinking about a dev team that is trying to kind of upgrade their handling of security, what's your sort of one tip or maybe like pet peeve or something they should do better?

Shaun: So if there is one thing I could teach them to do, and this is going to go back to some things I talked about before, particularly to the transparent point. I don't want them to be security experts, well sure I want them to be security experts, but I don't expect them to be security experts, I'm not going to try and make them security experts. I want them, developers, to understand risk.

And understand that every decision they make is a risk decision and that they need to understand what is the appropriate level of risk that they can take on as they develop their code. And what that means is that in some cases, I'm going to take a shortcut or I'm going to do something that might introduce a little bit of risk for my one feature. And in some cases, I might be doing something, creating a new feature, creating a new product, that's going to expose the entire company to an enormous amount of risk by, let's say, collecting sensitive information.

Developers need to understand what their risk is, why it matters, and when they're empowered to make those risk decisions, and when they need to bring in experts from the security team or somebody more senior in the company to help them make those risk decisions.

Guy: I like that. Understand the fork in the road, you know, just being able to assess how big a deal is this decision you're making right now

Shaun: Exactly.

Guy: Cool. Well hopefully people take that to heart. Well thanks a lot Shaun for coming on the show.

Shaun: Thank you.

Guy: And thanks for everybody who tuned in.