Ep. #32, Security and Compliance with Duncan Godfrey of Auth0

Guy Podjarny: Hello, everyone. Welcome back to The Secure Developer, thanks for tuning back in. Today we have a great guest, Duncan Godfrey from Auth0. Thanks for coming on the show, Duncan.

Duncan Godfrey: Hi, Guy. Thank you for having me.

Guy: We talked about a whole bunch of things to review, but I think maybe before we dig into the actual topics maybe you could walk us through a little bit about what you do, and maybe the journey of how you got there?

Duncan: Sure. I wasn't into security as a kid, as probably a lot of people who come into your show did.

I was always obsessed with networks and the internet, and that's very much what I wanted to do. I did do computer science and I've been into software, but I got my dream job working for a telecoms company, so that was my first job in the UK.

Then day one I arrived and I had actually been tapped to work in the government department of a telecoms, which I think everyone knows that now is a standard part of business in the world that we live in.

So, day one of my career I was in the most top secret environments possible and working my way through, and I was introduced to the security community that way.

Guy: It really wasn't a security role, it was just the security surrounding?

Duncan: Secure surrounding but a security role, building secure networks for government clients and doing security reviews and secure systems for data storage, and things like that.

It was eye opening and it really framed the way that I viewed systems from then on, because you have to meet certain standards and you have to design them in a certain way, and you have to meet all these really interesting and specific requirements.

Actually, that is really what pulled me into the world of security and thinking about security and thinking about risk, and it was a great way to start out. But as you say, I have done quite a few things since then on my journey.

Guy: You got your-- You started with a dose of paranoia, a little bit?

Duncan: Definite paranoia, yeah. That's certainly what happens in those environments. The worst case scenarios, you're always catering for. So that was an interesting way, but I did that for a couple of years and then I pushed back about being in that world a little bit.

I had a really fun adventure living in Mozambique for a while and actually working for a telecoms company there, and it was at the beginning of when Fiber was landing in southern Africa down the coast.

I actually helped build out some of the networks there. They only had satellite communication at a lot of the countries along the East Coast, and suddenly the Fiber bonanza arrived and there was tons of bandwidth and it was-- That was a lot of fun.

Then I came back to the UK after a couple of years and I worked for a very established enterprise company, which was the complete opposite of that.

I actually worked for the post office in the United Kingdom. But that in itself has a lot of interesting security challenges because they have a lot of retail outlets, they have a lot of very old systems that are running all of the tills.

It's actually, in terms of end points, one of the biggest number of end points in the UK.

Guy: Yeah. You have to be close to customers, so you have a lot of those different points.

Duncan: The customers are interesting and demanding. So, that's fun. But I think really what put me on the tech adventure that I've been on for the last couple of years was actually moving up into Amazon.

I joined the security team there and that was really working at scale, working at Amazon scale, which is just something that I'd never seen before and may never see again.

Because it's probably the biggest network and the biggest system in the world.

Guy: This was post-AWS era already, right? This is when AWS was already alive and well?

Duncan: Actually it was a fun time to join, because it was during the transition. During my time there a lot of what was Amazon.com moved over to AWS.

I was working on the dot com side, but we ran a lot of the network security. That's always-- You'll see a theme in what I'm talking about. Network security, security monitoring, that's what I've been interested in and that's what I did for them.

During that period we were transitioning some things over to AWS-- Amazon itself was transitioning a lot of its own services over to pure AWS, which was interesting.

We as a team were actually using AWS services internally and dog fooding them for the first time, and we were customers too. We were actually helping to design secure ways to use those services and advise our internal customers.

It was a fun time to be there, and it was great to see security and systems at that scale. Then I resisted a little bit working for a little bit of a big enterprise, I think a couple of years was there I'd been on that adventure.

Then I founded my own thing with a friend of mine, w hich looking back was a very fun year. Unfortunately the company didn't work out, but it led me to where I am now so I'm grateful for that part of it.

I was really trying to take the learnings of what I had done as a security professional for bigger companies and try and do that for smaller companies.

So that's what we were trying to do with this little company called Cadency, working in the Netherlands.

Guy: This was more of a consulting, advisory-type role or as a product capacity?

Duncan: It was building out security monitoring. So, what you would classically call a seam, or just your data lake of logs, but doing that at a much smaller scale.

My grand idea was to do that for lots of little companies and that would turn into a big business, but the funny thing about that was actually it made me realize that a lot of small companies have a lot of very terrible security problems.

What they actually need is someone to come in and help them fix those security problems, they can't make use of the tools that bigger enterprises have because they just have to fix all of the basics.

I found myself getting caught up in that, so it's funny you say "consulting" because I actually ended up doing a lot of consulting gigs to try and fix the security of these companies so they could use the system that I was trying to build.

I never quite saw us getting there, and I actually see some interesting companies now coming up.

I think the way it's working out that we might be headed in that direction, I might have just been a little bit ahead of it, but it's definitely a space in terms of logging and enabling companies to have access to those tools that the bigger companies have was my main aim.

Then I ended up at Auth0, which started up where I am now.

Guy: Very cool. OK, so that's quite a journey. You've seen some things, even Mozambique and some secret government chambers, the big Amazon surrounding and even the startup surrounding.

So you definitely have this rich perspective coming into Auth0. Let's dig into the Auth0 work. Walk us through a little bit into roles and responsibilities that you took on as the company grew.

You joined when they were, you said, about 70 people?

Duncan: Yeah, I think I was the 70th. I wish I'd written that number down, but it was around about 60 or 70. We're of multiples of that now, so it's been a really fun journey.

I split it into two halves, so I'm now a security leader. I'm leading a security team, but when you join a startup-- My first year there I was a principal engineer and I was building out security systems, helping developers deliver secure software.

Primarily I was focused on "We're a very heavy AWS shop." So, I was just trying to make sure that our Cloud infrastructure was secure and that we could scale and keep up with the rapid growth, not just in terms of people, but in terms of services that we were deploying and things like that.

Then as the team and the company grew, and that's the great opportunity of working at Auth0, is I've grown with it. The team never stops growing, which is fun.

But now I'm the director of security and compliance, so it's a combined role.

The second half is primarily focused on leading and building and structuring that team, and hiring and making sure we have the right people in the right places, and we're doing the right things to build that trust with our customers that we need to build for them to use our services.

Guy: Before we dig into the org and this group you're doing, when you were joining-- Just going back a little bit, because I find it very interesting to talk about "When do you do the first security hires, or subsequent?"

When you joined at employee 70ish with security in your title, were you the first security hire? Or were there other previous people that had that mantle?

Duncan: No, there actually was an existing director. Auth0 were forward-looking, and it was a good friend of mine so that is the reason I joined the company.

He said, "You should come and work for me. It's fast moving but you'll be able to have an impact, so come and work."

It was actually when he moved on that I took over his role and took off from there. But there was one other application security engineer who I am proud to say is still with the company too.

So we're the two oldest serving security people. It's erratic, which I think you know too.

Guy: Cool. It's great, and I'm happy for it. But definitely forward-looking to have three security staff, if you will, when the company is at circa 70 people.

Duncan: Yeah, I think it's key. I'm not going to pretend that we've got everything, we are trying to catch up in some areas.

When you're bringing people in at the beginning and you're not having to bolt it onto the end, you're just saving time and you're saving money, and you're saving frustration.

It's definitely something I'd recommend to other founders.

Guy: OK cool. This is a fascinating journey as we dig through it. You're coming along, you come and you build the AWS infrastructure. Let's maybe fast forward to today, what is the security team in Auth0? What does it look like?

Duncan: It's something I love talking about because it's been quite a journey and it's been a lot of fun putting the pieces into place, but really the way that we divide it out now is we have what we call our product security team, which is a cross-functional application and product security team.

They're our builders and our breakers, so they're people who are dealing with vulnerability management.

They're helping developers through a secure development lifecycle, they are dealing with incoming vulnerabilities and making sure we're dealing with them as quickly and as efficiently as possible--

Just really helping engineering understand security, secure development, and working with them on best practices.

Then we have two other teams, so one of them is a detection and response team. That's something that is actually feeling like a little bit of a luxury this year, because really we've done detection and response as a whole team up until we've been able to get to this size.

Now we have a dedicated team of experts who are looking for issues, responding to issues, automating security response, which is something I'm excited about.

Then we have our Cloud security team, so you can see that was where my strong background was and that's obviously where I have put a lot of effort, then I handed that off to a strong engineer who's driven a couple of engineers now and he's driven Cloud security forward.

That's what would traditionally be called a SecOps team, and that's what it would have been called at Amazon for example, but really we're trying to keep ahead of ending up in a SecOps state of mind by automating as much as possible and not having people sitting working tickets.

It's building software, building automation to keep up with our growth.

Guy: When you say Cloud security, how does that differ from product security? How is the Cloud bit not product, and how is it?

Duncan: It's literally securing the Cloud. It sounds simplistic, but actually AWS is so complex and we move so quickly, and you really need specialists to understand building secure infrastructure.

The way that I've built this program out is I put a lot of emphasis and focus on monitoring, security monitoring, gathering logs, making sure we have a good audit trail for everything that we do.

The charter of that team is to ensure that we're collecting data from every possible facet and nook and cranny of AWS and pulling that in for analysis.

That's the reason they exist, and then on top of that it's building tools for developers to be able to deploy safely to the Cloud too, making sure that they're not having to go it alone if they want to deploy some infrastructure.

We're reasonably flexible in Auth0 around what the development teams want to do, so you do have some DevOps teams who are running the full stack, and that's actually where our team has touch points with those teams a lot more.

When they're running their infrastructure we have a couple of straight up development teams who just deploy services to a platform, and things like that. That team is really just securing and monitoring in the Cloud.

Guy: Is it a fair analogy to say that this team is the SRE equivalent for security? You're engineering the surrounding to be available and to empower the dev team, but it's still on its own turf of expertise?

Duncan: Yeah, I think that's where we're headed more and more. I find myself hiring more actual-- I don't like calling people DevOps engineers, but that's what the title is.

Guy: Yeah.

Duncan: That's what the industry has decided, because we are running services.

We're running the security services for this monitoring and collection, and for the tooling and other services, the development teams can plug into when they're deploying their infrastructure to get the data they need and to build the services they need.

So I think that's a good way of thinking about it, like security SREs. That's a little bit of a work in progress, so it's a mixed disciplinary team of security engineers, but more to the software development side.

I think where people typically think of security engineers, they think of people who run firewalls or they do the configuration on routers and things like that.

But really, that's not how you can operate in a Cloud environment anymore, and that's not how you can operate. In theory everyone's writing software, so it's security software engineers building security services with some more infrastructure ops people now mixed into the team to help them scale those services too.

We're also giving good quality of service.

Guy: Got it.

Duncan: Yeah, it's been an interesting development.

Guy: Very cool. I think we've dug in a little bit into the Cloud security team, or I'd love for someone to figure out a term for a security SRE.

You need to do it, like make up make up a term and write a book about it. So this is the Cloud SRE, can you unravel a little bit what the other teams are doing as well? Just for ownership turfs?

Duncan: Sure. The detection and response team is pretty much as straightforward as it says, as what it says on the tin.

It's barely dealing with the amount of data that the other team-- Cloud security being one of the biggest providers of that, the security monitoring data.

If you imagine in AWS with-- You switch on Cloud trail and you suddenly get audit actions of everything that's happening in your environment.

We have upwards of 150 AWS accounts now, so that becomes a big data problem. We're collecting Cloud trail from multiple accounts and you're correlating that data, you're trying to figure out what's good and what's bad, you're trying to get a baseline of what is normal and what isn't normal.

At a basic level that team is really working with that data to look for anomalies. That is their primary focus, and what we're trying to do is avoid that team getting too large.

Because it then becomes a little bit of a drain on the business. What you'll see is a SOC growing out, you have an analyst role, people will be working tickets on those events.

But what we're trying to do is actually automate response to anything that can be automated, we're automating that way so that team can stay focused on the next level up which is doing the detection work.

Then ultimately that team is our insurance policy, so that team is the team that deals with a security incident if it ever were to happen. They're trained incident responders who can deal with tough issues and they can problem solve.

They can work on the pressure if something is happening and deal with it. So, we have a good outcome.

That's how I view that team, as my insurance policy.

Guy: How do they differ from the ops team? Or, how do they collaborate with the ops team?

Duncan: They collaborate with all the ops teams, because we need to gather data from all of our systems, so it's a pretty similar relationship across.

It's building a relationship to either have to deploy an agent to collect the data you need, or working with them if it already exists to just send it to the place you need to go.

Then it's a two way street, because a lot of this isn't necessarily just security information. Like Cloud trails actually pretty useful for troubleshooting issues, so we're pouring that into a data lake for all teams to use and have access to if they want to troubleshoot things.

But really, that team are also-- They're the center of excellence around what the right things to be doing in secure systems are too.

They're also doing a little bit of consulting work, they're working on hardening operating systems, they're working on making sure that the Ops people are doing the right things and those teams.

But they're just trying to get all the data they can so we have all the visibility we can into our networks and into our systems.

Guy: Got it. OK, but unlike the Cloud security people they both have this guidance element. But they also carry a pager.

Duncan: Yeah, ultimately. Cloud security are on call for-- We run it like a DevOps team, so they're on call for the services they build. It is an Ops team in that way, but when you're on call you're focused on dealing with anything that's happening.

We have an interesting environment because we have so many customers, and they can use our product in so many different ways, that we're actually often offering a lot of advice to customers and that often comes back to the security team in questions.

So you're on call, you can be helping a few funny customer issues that have come up, but that's where your focus is.

Your focus is dealing with issues, and then when you're not on call then you're trying to automate away anything that frustrated you during your on call period, like any Ops team should be doing.

Guy: Yeah, indeed. One of the beauties of this new DevOps mindset, and the reason you own it. You get to experience the pain and then hopefully you get to resolve it.

OK, cool. Can you give us a sense of the sizes a little bit before we move over as well to the product team? I think you mentioned that the Cloud security is a couple of folks.

Duncan: Yeah. That's been a project for this year, so we're around about four people, so that's a functioning rota. We're going to grow that out to five or six people this year, so that's the main aim.

It's a slightly longer rotation, and I think that's a good size. That's a good-- I think as a rule of thumb when I've been talking to friends and colleagues in the industry, you tie it to your head count the amount of incident responders you have.

One per hundred, or one and a half a hundred of the different things that we've discussed so we're keeping pace there. But I don't really want that team to get too much bigger than that, as I said I really want to be based on automation.

The Cloud security team is similar in size and Auth0 is-- We're fully in on AWS. We are an AWS partner and we are building all of our systems there, so that is a significant real estate to protect.

That's how I've been able to get the investment in that team to scale it up.

Guy: Got it. OK, cool. So we've covered two teams, what is team number three? Actually, maybe the most Dev-ish product security team.

Duncan: Product security team is an interesting name, it's one that I came down on and I think at some companies that's doing product development work, but for us it's really being a part of the product development lifecycle.

We have a secure development lifecycle and that team is supporting developers through that, and then that team is also triaging incoming vulnerabilities and assessing them.

We have a White Hat program and we use Bugcrowd to run our program through there, so you get interesting issues cropping up from time to time with people playing and testing with your product.

So, they're dealing with security researchers and they're dealing with people using that platform. Then we've always had a strong internal testing ethos, which is something that I've always been a big fan of.

We have a couple of people on that team who are very offensive-minded, and they will go out. It's not quite a red team. Our Red Team is a is an organized-- Semi organized function, and he'll go and try and break your environment in a non structured way.

But really this team is trying to do internal testing engagements on where they feel is the most important to test and play around with and try and break.

A good way of thinking of that team, it's breakers and builders. They're the people who are helping build things, but part of it is also trying to break the things too.

Guy: The product team is breakers and builders combined? I've heard-- I was new to the term of the purple team.

We're talking about the blue team and the red team implying that they do attacks, but they do so in collaboration with the defenders, so that everything can be loaded and logged and automated and repeated.

Is that an interesting term for those two breakers in this team?

Duncan: That's a really good way to think about the structure of our teams in general. The first two teams are my blue team, so the detect and respond in the Cloud security-- they really are the defenders.

And then the product security has some aspects and I think purple team does capture that well, but I'd say it's more a combined purple team across all of the security teams, because we're not really enough that people are in silos.

Not that I'd want people to get in silos, and we do run some quite fine internal testing exercises where we're trying to defend against--

Our breakers are trying to attack and our blue team's are trying to look for that, so that is interesting, but I like that way of framing it.

Guy: OK, cool. This team has those two people and then also has the people helping build secure products, as you said, working with the bug bounties and the likes. This is what you'd call application security, in more classic terms?

Duncan: Yeah. We're trying to have as frictionless as possible experience for developers that run through our secure development lifecycle, because the development teams at Auth0 can choose any way they want to develop software.

We're just trying to plug in wherever we can, but keep it as safe as possible so our SDL is really at the beginning. You fill out a very short form. I hope the developers would say it's short too.

That will steer you in the direction of how much help you're going to need from security. You'll fill it out and say, "You're going to be writing a service that's going to be handling customer data, or PII."

Or, "You're going to expose a public end point." Things like that. So it's a different paths down our secure development lifecycle, and then it's really working with our engineers and the security team working with the engineer developing to then do some general threat modeling.

Just exposing them to that concept of how you can break an application and how typically people attack applications, and how data is moving in ways that you might not expect.

Working them through so hopefully upfront we have the requirements in place and the good controls in place that we end up with good secure software, and then at the end we're running some more tests.

For our core services we're getting third parties to come in and test them too, because you need to have the external validation.

After we've done some external testing then we're pulling in our pen testing company, we used different vendors, and they'll come in and do a final formal check.

But the way that things move right now that is definitely the older model, is having that formal check gate which is getting harder and harder to enforce in a DevOps world where software is moving.

I find us moving more to just try and insert security in the deployment pipeline as much as we can. So it's just transparent to develop, plugging in different tools into the way that they deploy code and trying to catch everything on the left side of the development cycle as early as we can.

Guy: Yeah, and when would they fill this questionnaire or this short form?

Duncan: Right at the beginning, so ideally--

Guy: The beginning of setting up a new service, or--? Like, at the beginning of what?

Duncan: The beginning of their development, so whether it be a feature or a service, or anything of significance. Anything a new team has-- Either a new team or a new service or a significant change to what's happening. Then you engage the security team and engage the security process, and then be guided.

Guy: Got it. Maybe this is a good segue way for us to talk a little bit about, indeed we talked a lot about "How do we operate?"

It's Agile, you move around and you build and you run your Cloud security capabilities in DevOps. But this shifts us a little bit more into the governance mindset as well in this world of compliance.

So, let's dig into compliance a little bit. Compliance, understand also it's part of your mandate and title as well, "Keep Auth0 compliant around the security mantle."

So, what's your approach to it and how do you tackle compliance of this fast-based surroundings?

Duncan: That's really been our journey of the last year, and was my primary focus during last year. So it's been really interesting.

Previous companies, I think Amazon in particular, compliance was a very scary department that would come in and they would audit you and you were having a compliance audit. It was something that you were worried about happening.

It was something that you needed to make sure you had a weird your ducks in a row. I think I can see that playing out all throughout my career where the compliance department is a little bit in conflict with the rest of the business.

Or at least, you as an engineer, they feel like they're getting in your way or you're checking a box. But I think the nature of Auth0's business, the way that we need customers to trust us--

And trust is this foundational pillar for everything that we do, so whenever I'm talking to customers I'm having to explain to them why they should trust their most sensitive transactions with us and really compliance is how we build that trust.

So you internalize that and then you move forward and you treat it as something that's enabling the business it's enabling sales, it's enabling us to grow and to get the bigger customers that we want.

So that makes it easier, and then I've set myself off as a main contributor to getting those compliance.

So I come from a very technical background and I've worked hand in hand with who is now the director of compliance and he reports in to me. Last year in particular we worked in hand-in-hand to get that done.

But I think in our environment in a fast moving environment, you have to similar to what I was saying about pen test reports happening at the end or having a checkbox at the end. That just doesn't really work anymore.

You need to think about how you can be compliant but in a fast moving environment, so you need to actually be building tooling internally to track all of your assets because you don't have a list of firewall configurations to give to your auditor anymore.

You have a huge amount of data in AWS that you need to process and you need to present to them in a way that they can understand.

You need to come up with frictionless processes in the way that we've done with our secure development lifecycle, to allow developers to keep moving forward and to keep developing in a secure way.

But also fundamentally you need to demonstrate that you have shown maturity along the way that you're doing things like change control, y ou're doing things like code reviews, you've got security tooling built in.

So it's coming up with flexible interesting ways to get the job done to be compliant but not have that as a burden on the business. That's been my approach to security at Auth0 since I've come here.

The model of information security that we had at Amazon was you had a very centralized security department, which had a lot of power and a lot of sway to block you or to slow you down or to report you.

But when I entered Auth0 I was-- That was not the power I had or the role that I had. It's not what I wanted to do, and I got on in what is now called SecDevOps.

I worked within the development teams and I was building infrastructure, building services for them. That's the same way that I want compliance to be.

I don't want them to feel like we're doing this to them I want it to feel like it's important and that it makes sense and that it's enabling the business.

Guy: How do auditors feel about that? I like it and I believe in what you're saying in terms of the way to operate the company when you're coming to an auditor with a digest from your data lake, instead of your firewall rules.

How has the reception to that been on the auditor side?

Duncan: It's a conversation, so it was definitely more interesting in the initial conversations as we're trying to explain how we do things, but it's really about having your evidence and having the confidence to explain why you're doing what you're doing.

But I think the industry is changing, so we're not the only people who are Cloud native and we're not the only people doing DevOps, so they've seen companies like this before.

I think five years ago you might have been laughed out of the audit room, and I hope that we're making it easier for other companies to go down this path too, because I think it's the way to do it.

Guy: Yeah. It's definitely a much needed transition. One of the early episodes on this podcast was with Shaun Gordon from New Relic, who I think to an extent maybe we owe a bit of a debt for being one of those early educators in this world of Cloud for the compliance side of the fence.

OK, cool. So we're doing those and you're investing in compliance and you're building all those components, and I like what you said before about the trust element and fundamentally using that to help foster trust once you build those security controls and you get those compliance certificates.

Maybe get a vote of confidence from the customers for it. We definitely feel like since we got-- It's been a while now, but at Snyk when we got the first SOC 2 compliance it just streamlines so many conversations.

Duncan: Yeah.

Guy: It was absolutely worth the effort. So, we're getting lots of great insights and I think it's super useful as people look at "How do they build their org teams, or their security orgs, and facilitate them?"

Before I let you go here I like to ask every guest that comes on the show as a parting question, if you have one tip or one piece of advice or one pet peeve maybe that you want to get off your chest that you want to share with a team looking to level up their security stance, what would that be?

Duncan: I worry sometimes as an industry we keep making some of the same mistakes again. I really want to dig into that as a concept.

We're moving some of our infrastructure to Kubernetes, for example. So that's a whole new paradigm of how we operate infrastructure, and I can just see us making the exact same mistakes again when we're exposing the infrastructure to the internet.

They're doing a really good job of catching up with role based access control now, but that wasn't built in from the beginning. It's really hard as a security team to audit that and really understand what the infrastructure is, unlike--

What we're building on top of it, I absolutely love. I love containers, I love immutable infrastructure. I'm loving where we're headed that way, and I'm just worried again that we're not thinking about the basics.

That would be my pet peeve to any security team, is before you get onto the more advanced stuff, really just doing the basics. Locking down your access, vulnerability management, making sure you have good hygiene.

I'm sure I've heard it before on your podcast, people have said that, but also it's just vital. So just getting that solid foundation to build on.

That's my current pet peeve, when I see an exposed case management infrastructure on the internet I'm like "This feels like mid-2000s again."

Guy: Yeah, cool. That's a great tip.

Make sure the basics are there before you get into the shiny new toy area.

Duncan: Yeah.

Guy: Duncan, this has been a pleasure. Thanks a lot for coming on the show, and thanks everybody for tuning in. I hope you join us for the next one.