Ep. #1, Prioritizing Secure Development

Guy Podjarny: Hi, everybody. I'm here with Kyle Randolph from Optimizely, who handles AppSec there. We've had several conversations in the past, and I'm really happy to have him here on the show to share some of his experience at Optimizely.

Kyle, do you want to maybe introduce yourself a little bit, talk about what you do, your history?

Kyle Randolph: I'm Kyle. I've been doing security for about 15 years: Citrix, cleaning up the mess at Adobe, building security at Twitter, protecting free speech there. And then now over to Optimizely where we're building security from scratch at a startup.

Guy: Very nice. Cleaning up the mess that Adobe, I guess this was at the time of the repeated Adobe PDF Reader vulnerabilities and the Flash former abilities.

Kyle: If you want to get malware on somebody's computer, it was headlines in the news, send them a PDF or a Flash file and take control of their computer.

Guy: I guess that there's the security aspect and there's the victim of their own success in the sense of having Flash at what, some 90-high percent of machines everywhere.

Kyle: There was more computers with Flash at one point than there was Windows machines.

Guy: You say you're at Optimizely, you're building security. What's your role these days?

Kyle: My role is, throughout engineering, all of our products that we build, ensuring that we have security built into them. Making sure both that all the different features that we're building are secure, as well as having the right security features in place, in a price customers would expect in a product that they would use across their organization.

Guy: I understand. So, you started working closely with the product itself. You don't just deal with the security of it after the fact or the audits. It's more about building it in?

Kyle: We built our program to be tied very closely to engineering. We sit with engineers, interact with them every day, from design reviews to code reviews to advising on libraries, design patterns, tools in automation that they can use, working very closely with Dev Ops.

From the beginning of an idea to when we ship that out the door, all the security that we need to have built in, making sure that it gets built in.

Guy: Is it the same team? This is like an AppSec team as opposed to an InfoSec team?

Kyle: We call this an AppSec team because it's within engineering, so we work with the product management and engineering teams to build security in there. We partner with the InfoSec team, which is more focused on protecting the company from traditional IT security risks. So making sure our Wi-Fi security are actually directory, domain controllers are isolated and have good access control.

Guy: You and your group are in the development organization, right? You work with Dev?

Kyle: I've been in organizations both where the security team's inside and outside of engineering. And trying to work with engineers when you're not part of the engineering organization is much more difficult. It's harder to get traction with them or you may not seem as approachable, but

being tied in to engineering, you know what's being built, you know what's on engineers' minds. It's much easier to gain influence over engineers and also have them come to you.

Guy: Makes perfect sense. How does, just for context, the Ops side of the fence works in Optimizely? Is that also kind of a part of that same entity?

Kyle: There's a DevOps team, we call it E-Squared, or Engineers for Engineers, building tools in automation to help engineers get their job done easier. And so we partner with this team very closely.

For example, with Amazon Web Services, a lot of our infrastructure is deployed there, and at E-Squared, they want to automate a lot of that. So developers don't have to think about, "How do I get my code into production?" And then we want to bake a lot of the security in during that deploy process.

You have security groups and BPCS and cloud trail logs and so on. There's like dozens of things that we don't want developers to have to think about in order to make their future secure. So we build that into the automation. When you're pushing this thing to production, you're also baking all the security into what they're shipping.

Guy: It's all about, one, lowering the bar, just in making it easy for them to consume it. And the other is sort of avoiding all the specific manual, laborious tasks and make it just happen, right?

Kyle: Yeah.

Guy: It's all about scale, I guess. Which is amazing, how that type of concept has really become natural in the world of quality testing, where it's not that QA is decreasing really, but in many more startups, for instance in Snyk, we don't really have a QA team, and there's really no plan for it.

The understanding is that you own it and you own the quality. You build tests to build components and they grow there. And yet somehow it's a little different novel to sort of add to that in security. That's cool.

So, you're doing Optimizely. When we first chatted, you still had a lot to build out, right? You joined in to create this ecosystem. Can you kind of share a little bit about how that transpired? What did you see when you came in, and what did you do to improve it?

Kyle: I joined Optimizely in October 2014. It was a startup. They've been around for five years, built primarily in Python on Google App engine. So, large monolithic web app in Python, so you didn't have to worry about buffer overflows, because it's an interpretive language.

And then, because it was built on Google App engine, you got a lot of your security for free because Google has taken care of all the security lower in the stack, and they know what they're doing.

That helped Optimizely get along for a very long time without a formal security program in place. And then the developers were security savvy, they had the right idea for how to build some security based on reading OWASP, being familiar with OWASP top 10 and common vulnerabilities.

As this monolithic application grew and they got more customers and larger enterprise customers, the security requests from the customers began to grow: asking for two-factor authentication, single sign-on and audit trail, and so on.

Guy: Security features, not just guarantees but straight up security features.

Kyle: Yeah, and then the security risks started to become more sophisticated as the product gained traction. Now a lot of the largest websites and one of the most popular ones, they have Optimizely on them.

So as an attacker, before our little startup, you may not even noticed it, but now you're like, "Hey, I could compromise all these sites at once, by compromising Optimizely." That raises the stakes, and that's where more advanced security was needed to be baked in.

Guy: That makes sense. I mean, today, Optimizely has its grips on so many pages, right? So you can get a lot of access to data, and you can actually manipulate the content. Optimizely modifies that content eventually, at its core, being to sort of optimize the site's performance, feels like a rich target. I'm sure you have your hands full.

Kyle: Yeah.

Guy: There was no formal program. There are a bunch of lead developers doing the OWASP thing. How did you approach it? What did you do first?

Kyle: This was interesting. I would say the engineering team, there was no security engineer there ever, but the engineers want to do the right thing. They just wanted to know what is the right thing to do, and there's thousands of security things they could do. But what are the most important ones they should be doing first?

As the security engineer coming in, I wanted to make sure I built up a good relationship with engineering and then just bog them down with tons of security requirements and keep them from building new innovative features. So some things I wanted to do from the start are like, find what are like all the easy wins to begin with. Let's find out all the low-hanging fruit that'll give us high-security impact with very little effort and very little disruption in engineers.

Building out a back log of those, just going around talking to engineers, buying them beer, buying them coffee, whatever's needed to talk about what's on their mind. Not even saying anything that they should be doing for like the first month or six weeks, but just listening, and everybody brings security issues and concerns to you.

You discuss them, but it'll make them defensive and tell them what they did right or wrong or whatever. But you collect a lot of information that way. And then from there you can start building out your plan, and you know where the hotspots are that you need to focus on or put some process around or build some defenses in.

Guy: I guess it goes back to the fact that everybody wants to be secure, and it's more about dealing with either the lack of expertise or just the time crunch or the fact that

if you invest that whole chunk of time in making something more secure, oftentimes that's invisible.

That's just not seen by anybody, and yet the fact that you have not built this feature or this functionality, that's painfully visible. So it's kind of hard. I guess then, to an extent, you can be there for the things they want to do. You can be their excuse, their support person, and also their expert way to sort of say, "Is that really worth doing or not worth doing?"

Kyle: That's where, what should you do first? like security vulnerability, severity ratings, those help somewhat. But what helps more is telling compelling stories of like, "Here's a real world example of when this vulnerability lead to the company actually being compromised," something very similar to Optimizely.

A company called Gigya, they also load JavaScript on a lot of different companies' websites. And so what happened to Gigya is the Syrian Electronic Army said, "Hey, let's compromise Gigya," compromise their JavaScript on all these different sites like Forbes. All these different sites, all at the same time, said, "You've been hacked by Syrian Electronic Army."

That was a great way to drive home what Optimizely's security vulnerability would look like in the real world, and nobody would want that to happen. So we had things we needed to do to prevent that.

Guy: Yeah, that definitely sounds like a good example of it and I guess as close an example as you can get to the actual business. Kind of makes people relate to this can happen to us.

So, those were some of the tools that you used to be able to educate or sort of promote, like you give them these real world examples. I guess you also used these conversations both to build relations which, at the end of the day, it's all about people, right?

But also to understand and kind of get the lay of the land. At some point you do need to start tackling those, right? You've identified your priorities. You understand them. You're kind of buddies with the different people, so they would just kind of trust your judgment and listen to you. What are the first steps that you introduced that are still kind of process oriented or tooling oriented?

Kyle: A guiding principle was, if I'm doing something myself, I'm probably doing it wrong. Because there's only one of me and there were dozens of engineers. So it's all about enabling engineers to be able to do these things, setting up many different things at once.

Showing off tools at Lunch & Learn, it's like, "Hey, look at this really cool security tool that can help us find cross-site scripting." And just show it off, show how cool it is, show a success story, and then cross your fingers that at least one engineer in there will think it's so cool, that they'll go run it themselves.

That's kind of like fishing. Not "phishing," but just fishing for people who are interested. And sometimes that works, sometimes it doesn't. But it's a good way to get interest. Getting a security group put together, we call it the security cabal at Optimizely.

This is any engineer interested in security. We meet each week, just talk about security stuff or cool security tools. We also have a Slack room where we just talk about these things daily. And, just by having a constant conversation, there are a lot of engineers who will pick up something, play around with it for the afternoon, and start trying these things. That helps a lot.

Guy: These are just the people that are interested in security naturally. It's not their job, even though they're engineers. Alongside their day job, that just is one of their passions.

Kyle: In the early days, there were so many things that needed to be done for security. It was like, "Okay, we have this." Let's say we have a hundred things to do. We have 20 things we know we need to do now, let's put those 20 out there and see how many people are interested in it.

Chances are, if they're interested in and passionate about it, they're going to be much more productive working on it than if it's assigned to them.

So, let's get those out of the way first.

Other ones we kind of kept on the shelf and waited. Let's say there was a quality improvement that needed to be done, where you're going to split this feature out into a microservice. That's a pretty good time to add in some security that we had sitting on the sidelines, that we knew need to be done.

Or another quarter we have initiative to speed up the delivery of our CDNs, and that's a great time to say, "Hey, now it's time to look at TLS optimizations," for example, and "Let's clean up the insecure cipher suites that we have in TLS." So, satisfy the security guy, making it secure, but then also we get some speed out of it when we start turning on TLS optimization.

Guy: Basically sort of bundle it in, I guess maybe as a natural aspect of quality for those components, and just make it be around. If you're going to build it, build it right, and that includes building it secure.

How did you find out, as you say, you're sort of the one-to-many in terms of number of people. So, absolutely awesome, I think that you're not a gate, I'm a firm believer that. It's just that it would never scale, it has to be built in, and it has to be kind of spread out between the people in kind of this continuous fashion.

You also can stop any process and have them be done manually, but you still need to know that a micro service is being built. You need to maybe advise a little bit about the opportunities. Did those end up just permeating to you because relationships, I mean, how does that work?

Kyle: That one is a challenge, how you scale up with the organization. And in classic waterfall times, you go through these big requirements phase. You see all the requirements in one POD document, and you can just mark the ones that you have concerns about.

Now, in more agile development, everybody's building on their own schedule. They may or may not use Harman or Agile or whatever. There's no one place where you can find everything, so with that I found you just have to spread yourself everywhere, that the engineers are to have a lot of ears.

Hanging out in Slack rooms where the action is, watching email lists, joining teams' email lists, going to their meetings, going to their special interest groups, speaking every week about security in different meetings where you know engineers are going to be, just make yourself as available as possible. And then you have a lot of engineers coming in to you, which is great. Especially after they see you're going to help them and not slow them down.

And then also having your ears open in all the places that you need so you get wind of things. Sooner or later, you'll find out about them, one way or another. It may not be in the design phase, but you want to find out about them as soon as possible.

A great place to kind of find out about things is source code. And so our code review tools, setting up rules like, if anybody touches these files, like opening up a new route in our monolithic web app, then we know that's something. That's like a new feature, and so that's something that is a good place to flag for security.

Guy: And so you've got little spies there, right? Not people spies, but just sort of sensors, I guess, kind of spread out to tell you that something interesting might be happening.

You do this, and I think it's awesome, and again, I kind of believe in the approach of permeating and making it a natural part of the culture. I've seen a lot of that success also in web performance, because I've been kind of oscillating from web security to web performance, you see a lot of that.

You don't want to be a performance janitor. You want to permeate that and make it a natural part of your company, and security needs to be the same.

A lot of the conversations we had there were about celebrating successes, which is always tricky in security because not getting hacked is not sufficient, because hopefully that's the normal course of operation. Otherwise, you've got other problems. Did you consider that? I mean, are there ways in which you somehow sort of celebrate either an investment or a success in security?

Kyle: The company has different ways to recognize success, and we kind of tap all those for security and make up some of our own as well. Company-wide, there's a way for one employee to thank another employee for something, and that's very public and recognized each week at the company all-hands meeting.

Using those thanks to thank people, even just fix one security bug, anything with security, or they had a good security question, to give them links there. Stepping that up a bit more, in quarterly retrospectives: what went well, what didn't go well.

Security guy, you can always just say, "Well, security wasn't as good as it could've been." But more pragmatically, talking about who did do awesome security, like, "This team is great! They fixed all the security bugs they said they would." Or, "Everything they made this quarter they had a design review before they built it," and so we knew security was baked in at that very early phase.

Going further than that, giving out T-shirts. We actually give out T-shirts that say, "Security Hero" on them. This is more exclusive, so it makes people want to step it up and really go above and beyond to make a security contribution. Maybe you eliminated cross-site scripting, you built it into the framework your team uses. Then that would warrant a Security Hero T-shirt. And then you get recognized in front of the whole company.

Guy: Nice, that one you need a slightly higher bar because if you start giving them out too much, then you actually lose the prestige of having that shirt.

Kyle: Yeah, so many different tiers of reward there, in recognition.

Guy: I guess, though, there are specific examples, you gave a little bit with the cross-site scripting framework or with sort of the team that is fixing this, but you know, do you have a specific story that you feel kind of manifests, really, really demonstrates how they succeeded? You know, converts or the like?

Kyle: Yeah, cross-site scripting and cross-site request forgery. In both of those cases, we had an engineer who may have had a bug or two assigned, and they're like, "I've seen this before. I don't want to see this again." And so they just went out or their way to really say, "Here's how we're going to fix this comprehensively."

In one case, this one team with cross-site scripting, they didn't even talk to me. Within their own team, a guy made a presentation like, "Here's why cross-site scripting's bad. Here's how we're going to make sure our team never has it any more." And then they're like, "By the way, we did this, we banned cross-site scripting."

That was awesome, and I don't even have to get involved. They just took it upon themselves to do it. That's even better than roping the security guy in to tell you what to do. But to decide on your own to do it?

Guy: That's great. That's definitely reaping the fruits of your labor, right? When you sort of planted those seeds and they grow inside. This is all process and people and how do you celebrate them. Are there specific tools that, I know you had sort of throw out that you find a special use for?

Kyle: We've been focused in AWS. How do we get our heads around that? And it's not so much security tools that help there, but just latching on to the automation that's going in there.

DevOps teams and engineering teams, they want to manage just their networks and VPCs through cloud formation, but hey, you could also manage security groups in there by bundling, we have a nice tool called Demeter which manages their security groups, nice and simple.

We're using a lot of Spinnaker for our deploy automation, which is not a security tool, but that's just the place that you can bundle in all the other security configuration that you want to have happen. Those have done very well for us.

The third party libraries, we know we need to address that, and Snyk looks very promising. So we're looking forward to trying that one out. That's crazy to think how much third-party code we have and we don't know what's going on if we don't have a tool that can actually do all that analysis.

I remember being in Twitter and wondering, "How's this monolithic Ruby on Rails app pulling in code from all over the web? The Twitter scale, not have some kind of compromise, and it's really scary. So that's looking very interesting, and then we have a web app security scanning tool. We've never found one that works.

Guy: I've built two, and I can point out many, many flaws in either one.

Kyle: Those are the tools now. Definitely looking more into automation and open sourcing things as we go.

Guy: Cool. I don't know if it's the dark side or the necessary evil, you know, in security it's good and well, kind of what you need to do is to sort of permeate and build it, let it come organically. Sometimes you do have kind of hard lines, right?

We talked a little bit about PCI just before they started, and you do have some other regulations. There are security questionnaires, sort of joyful, hundreds and hundreds of questions and diagrams that you need to provide. How do those get handled?

Kyle: With those, it's kind of like customer stories, just kind of putting on your product manager hat and telling these stories very frequently. You have to. Unfortunately, you have to repeat yourself very, very often.

You're being this security feature evangelist and reminding people and constantly, like, "Our customers want PCI, and then PCI wants to preserve the integrity of our JavaScript, and then we need X, Y and Z to preserve the integrity of our JavaScript." That's our contract to the customer.

So, tie it all back to that. Or it could be like ISO or something else, data privacy, and you're tying it back to a story, and then just telling that customer story over and over. But it keeps it in people's heads on why they need to do that.

Guy: I guess when you come to think about it, that's indeed oftentimes how you promote, how you manage to get a customer story around a feature or a need, or a capability comes in, "This is a core necessity." I imagine those do eventually get captured by more the product managers, but then into a prioritized backlog?

Kyle: Typically it's security feeding them into the product management backlog, and then once they're accepted in there, then it's much better if you can to get a security feature on by a product manager.

That's their specialty, and they're very good at seeing those things actually get built and built properly with the right documentation, marketing whatever you need, so that you can focus on doing the engineering side of things.

Guy: Stepping out of Optimizely for a moment, you sort of have this experience across different companies, right? You've been sort of big and small and I guess in different phases.

Do you feel, like these techniques that you have in Optimizely, is this just the way you would've approached it almost regardless of company? Versus things that you think you know, you need something there to be able to apply this. How broadly applicable are they?

Kyle: Yeah, with this one,

I like to look at the BSIMM, the Building Security In Maturity Model.

This is where Cigital did interviews with over a hundred software companies to ask them just, "What are all the things you do to build secure software?" And it was across different industries, types of companies and so on.

The interesting thing there is a lot of the activities are common across different types of software companies. Could be an insurance company, could be a SaaS company. They may do them in different ways or for different reasons.

It's not one size fits all, but across different domains of security, I believe they have like 12, there are very common activities, about 120 of them. Some of them make sense for Optimizely, others don't. Some of them makes sense for Microsoft, others don't. But that's a good place to start and use it kind of as measuring stick like, "How does my security stack up against "everybody else in the industry?"

Guy: It's always an endless sort of task that you could invest in. How do you know that you're doing sufficiently well? I guess you know that's a million dollar question.

I guess sort of within this, do you have a favorite? This is all kind of broad, and I've kind of asked for the anecdote as well, but what's your pet peeve? What's the thing that you really, really either like the people do, or you really get annoyed when people do not do, and kind of surfaces. Is there one?

Kyle: I guess the struggle with security is keeping it on and keeping it prioritized and a lot of times,

people want to do the right thing. But then when you get to resource crunch, features often win out.

The thing to be careful with here, that has been a struggle throughout my career, is just getting teams to commit to building things. Like, we agreed this is a security issue that needs to be addressed, but when the horizon goes out more than like a sprint, then it's like, "It'll be done next sprint, next sprint."

Next thing you know, you're a quarter out and something has fallen off the radar, even a year out. And with this one, I found that it's unfortunately very proportionally how much energy the security team puts into evangelizing those things and pushing for them. So you have to be very selective and identify those at-risk things that may not get done, and save your energy and your political capital for those to really drive them through.

Guy: It's focus, focus, focus, right? It's all about sort of choosing the things that you want to handle well versus handling a whole bunch of other things.

So, this was super fascinating, and I've got a whole million other questions to ask. But I think we're going to run out of time in a little bit. Before we part, just as a word of advice, I try to sort of ask this with all the guests. What do you, if you talk to a development team right now looking to up their game in terms of security, what's your one area that you would advise them to do? Sort of one thing that you would advise them to start with.

Kyle: I think that getting all of your security issues out in the open, discussing them. A lot of teams will say, "Oh yeah, we've known about that, or heard there's a security bug over there." But you don't even see them in their bug tracking system.

Can I even go see all the security concerns that we have? And not even having that transparency about what your security exposure is or what you would like it to be. Without that information, you don't even know how to do the right thing.

So, I think getting all the concerns on the table like, "What do we expect our security to be, and what are the known problems that keep us from being there?" That's a good way to bootstrap on the security that you need to do and at what priority.

Guy: Makes perfect sense, get your head out of the sand. It's much more convenient to just not think about these things and hope for the best and just forget about it. I know we see this a lot with third-party security, right? You use them, it works, you'd rather just not really open up that Pandora's box.

But otherwise, once it starts surfacing, then I guess you'd be more motivated to actually start making a dent in that list and improve them. This was fascinating. Thanks a lot, Kyle, for coming over. Looking forward to chatting some more.

Kyle: Great, thanks a lot.