Ep. #41, Optimizing Team Communication with Sara Dunnack of InVision

Guy Podjarny: Hello, everybody. Welcome back to The Secure Developer, thanks for tuning back in. Today we have Sara Dunnack from InVision. Welcome to the show, Sara.

Sara Dunnack: Thanks for having me.

Guy: Before we dig in, can you tell us a little bit about yourself? What is it that you do, and how you got into the world of security?

Sara: Sure. I do application security at InVision, and it's definitely evolved throughout my career.

I started off as a classic ASP developer, and that's definitely dating myself.

From there I went into web engineering, and I think when you get to the infrastructure world you get to a point where you're like, "I just want to start breaking things."

So that's how I came into AppSec.

Guy: Interesting impulse there. That's all within InVision? Or is that in different companies, big/small?

Sara: No. Right out of college I went to a small design firm, which might use InVision now as a tool, and from there I went to a large insurance company and got into web engineering there, and into security.

When I came to InVision I've been on the AppSec team the whole time, and I was actually the first security hire for the company outside of the VP.

Guy: How many people were roughly in the company at the time?

Sara: I was employee number 150.

Guy: 150? OK.

Sara: We're over 800 now.

Guy: Yeah. That's a good sized company.

Sara: So, when I got to InVision I did pretty much everything. Customer questionnaires, talking to the customers about our security.

I didn't do so much on the DevSecOps side because we have our own DevOps team there.

We do have a security DevOps team now, but now I'm focused more on AppSec again.

Guy: Got it. OK, so let's indeed dig into those.

This sounds like a journey, and it's interesting.

It's always interesting to grow with a company as you had one person show earlier on.

How does indeed the team look like right now? Roughly how many people do security on the team, and how is it divvied up?

Sara: I think we're about 15 people now, and it's split up amongst a compliance team, the DevSecOps team and AppSec.

Guy: I think "Compliance" I understand, it's its own run. What's the difference between the DevSecOps team and an AppSec?

Sara: DevSecOps definitely focuses more on the infrastructure side, and not so much the application side.

They handle our web app firewall, hardening any images that we might need, and basically just doing anything that you'd think a regular DevOps team would do, but very security-focused.

Guy: OK, interesting. But more infrastructure-oriented?

Sara: Yeah.

Guy: Then, the AppSec side?

Sara: The AppSec side does any testing within our actual application, mainly our customer application.

We do security reviews, we manage tooling such as dynamic scanning, source code analysis.

We build our own tools to do some linting and other things like that, and we are the main team that interacts with the engineers in order to make our code more secure.

Guy: How often or how easy is it to maintain that split?

As a lot of software becomes software-defined, and maybe there's some TerraForm script or some helm chart, or whatever it is on some repository.

Does it come up at all, this division of responsibility? Like, engineers dealing with something that is in front and that's suddenly the DevSecOps team?

Or does the split work fairly smoothly?

Sara: For us, it works very smoothly. All the roles are defined, and engineering doesn't really come to AppSec for any of that stuff.

Even the WAF, they all know to go to the DevSecOps team. We call it "Security SRE," so for us it hasn't been an issue.

Guy: Interesting. I think that's a really -- I love the model, because it reflects and I generally really like DevOps analogies as we dig in.

So let's dig deeper into your neck of the woods here with AppSec, we talked before about how you as an application security team work with engineering.

Can you can describe that a little bit? How do you engage with the engineering teams?

Sara: One thing that we implemented recently was we have an AppSec person aligned with all of our engineering teams, especially the ones that do development.

We have roughly 40 different teams that do development because we do micro services, so teams are split up specifically for a specific function or to provide code on one part of the app. What that's really helped us with is communicating openly with the engineering organization as a whole.

For instance, I attend stand ups almost every day. I try to hit two a week, because some of them do overlap on time.

But I attend stand ups for the team, so I know what's going on and I actually participate in the stand up with them.

And also I'll take time if there's any projects that we're working on, any functionality we might roll out in terms that might affect them.

I make sure to give them a heads up as soon as possible so they're never going to be surprised by it, and it really helps us not be a hated team. Because security--

Guy: Is sometimes seen as the nay-sayer, not necessarily the one you want to call to the party.

Sara: We don't want to be a blocker, so it's something that's really helped us in terms of opening the line of communication and getting a better relationship.

InVision is a very strange company in that security isn't thought of as an after the fact, if we tell a team that there's a vulnerability we don't get pushback.

We don't get told, "That's going to take three months to fix."

Depending on what it is, especially if it's a critical or high, it's fixed the next day if not that same day.

Guy: That's amazing.

Sara: Even on a low, we don't get any pushback.

Guy: No pushback at all?

Sara: Right.

Guy: First of all, amazing. Great job. I feel I'm definitely probably one that you've had as an early security person in the in the company, you had good success building that up.

How do you --? I guess I'm still on the alignment bit, although now I'm super intrigued by this achievement of getting people involved.

So if we go back to this alignment, how do you divvy it up? You're aligned with a bunch of these groups, and you work with them, you attend their stand ups.

You're pretty embedded inside the relevant engineering teams that you align to, do you aim for a certain ratio?

Or how do you decide how to maintain that over time?

Sara: What we do is there's four of us, not including our manager, and we've split it for the most part evenly.

But it was also based on position, so if you're a higher title you'll be responsible for just a few more teams.

Guy: You might have more capacity.

Sara: Right, so we do it that way. But on average everyone has somewhere around 10 teams, which is working out well for us.

It's divvied up by our engineering org, which has zones for different functionality types of the application.

So each AppSec person might have a large part of a particular zone and all the teams under that zone, so that's one easy way to split it up.

Guy: To map it, OK. I get it. That makes a lot of sense.

So you're basically aligned with the way that the teams are split in the first place, in the development organization, and you map with them.

It's not-- I guess, InVision is a very distributed company. So this isn't about co-locating with the teams.

The relationship is still virtual, as is the relationship between all employees and InVision.

Sara: Yeah. We are a 100 % remote organization. We have no offices. If we do have an office space, we work.

So I think with us using Slack and every meeting is a video call or a Zoom, you don't really feel disconnected from the rest of your organization because you still see people on Zoom.

For instance, if we have an offsite and I see someone on my team that I've never met in person but I'm good friends with them at work, I'll run up and give them a bear hug when I first see them.

It's just a very different company than most in that we have no ego, we don't hire people who have egos, and we're a big family.

Guy: Yeah, I'm a big fan of InVision as a company, as a culture.

Definitely one that I look up to, but it's just giving me another reason to like InVision in this security mindfulness, if you will, or awareness.

Let's dig into this wonder, so how do you maintain that? How do you celebrate successes? What do you find helps you keep that approach up?

Do you encounter at all pushbacks from developers, or is it literally just "We've built the culture and now it just runs itself around people responding and thinking about security problems as we go?"

Sara: The only time we've really had pushback is if there's a very tight deadline and they'll say, "Can we put it off to a couple of sprints from now?"

That's pretty much the only "Pushback" that we'll get, but even so, no one will fight us on making any changes, really.

I think we've been able to maintain that as we've grown through good hiring, and we make sure that all the engineering teams know that we don't want security to be a blocker. We're not putting ourselves on the critical path, we want to work with them and not against them, and that has definitely helped us maintain that good security culture.

Where they know that we're not out to get them, so it's a "Help me help you" situation, and it's so far worked really well and I don't see it changing anytime soon.

Guy: So, how--? Let's understand a little bit the relationship of who does what, as you talk about either the DevSecOps team or the AppSec team and the engineering teams, who implements security solutions or runs them, or deals with the vulnerability outputs?

Are these things operated by the security teams, are they operated by dev? How do you decide those?

Sara: For the most part, anything security-related will be done by the security team.

There are some instances where teams have said that they want to have GitHub integrations that have a security impact, so they'll do that on their own. Which is great that they want to--

Guy: Take initiative.

Sara: Take that initiative, and we do have engineering teams come up to us and say "I found this in our code, we're going to fix this. We're just letting you know as an FYI."

Guy: Do you have a standard way of celebrating that, or acknowledging?

Sara: We want to make T-shirts and give out T-shirts. For instance, the T-shirt I have today on the back of it just says "Rockstar."

We would love to be able to do that and give that to them, but another thing that we do as a company is something called "Bonusly" where you give peer dollars to each other for a job well done.

The main way, and certainly the easiest way for us to do that would be to give a person Bonusly if they tell us something.

We do that even if someone gives a really good talk during architecture office hours, or something like that.

It's celebrated in that the organizer will be like "So-and-so did a great job."

You shoot them some Bonusly and it really encourages people to get keep staying involved, and strive to help the organization get better.

Guy: It's interesting that in most companies generally you try to get developers to care about security, and when you're in a really good place like you are, you almost need to be careful not to overly rotate on the cognitive dissonance side.

If you have to start compensating or overly rewarding for it, people might start thinking about this as an exterior motivation for doing the good job.

Where right now, it sounds like they're addressing it as something more internal, more intrinsic to building good quality software.

Sara:

They've asked us if we would have an internal bug bounty program for finding and fixing issues, and we keep pushing back against that because we don't want them to purposely write bad code to help their friends out to get money.

So we're holding off on that, I think that'd be a little overkill.

Guy: I think that's wise. I've actually heard-- It's not the first time I hear that, but people consider.

It's like, "OK. You can you can't report vulnerabilities from inside when you're the one that might be writing it."

Sara: Right, there would definitely have to be some rules around it, some guidelines around it. But I don't think that's anything that we'll implement anytime soon.

Guy: Maybe we'll talk a little bit about a few practices that you have, given this reality or tight relationship with dev.

How do you handle the security training in an organization?

Sara: We do have specific modules that developers have to take for compliance purposes.

Actually, our OKR-- It's funny that you mention this, because we didn't talk about this beforehand.

But our OKR this quarter is to essentially start from scratch and the AppSec team is going to write the training documents, so specifically tailor it towards our environment.

It's aligned with ASVS points, so we're going to start from scratch and really tailoring it.

Because one thing is the technologies that we use don't have a lot of documentation, especially around security.

So it's been a pain point of ours which has led us to focus on creating all this documentation this quarter.

Guy: It makes sense, and it aligns with how you described it before as well, on the security team as an enabler and helping developers do their job.

This is one that you'll do, I guess if you write it from scratch or not, it's a modern tech and it's hard sometimes.

Sometimes you have to do more if the ecosystem hasn't caught up.

Sara: Yeah, it's definitely nice because we've been asking the engineering org for feedback and what they want to see documented, so we can really focus on what they think that they need help on and where they might be having more issues.

It's definitely going to be good for us overall.

Guy: Have you encountered any cases where there was a bit of a conflict between the need of security and the need of dev, maybe even because of this relationship?

Cases where you wanted to maybe introduce a specific solution, and there was too much dev friction so you had to give up on the security side?

Or has it ever come up as a challenge, matching up the priorities on the security side and on the dev side?

Sara: Not really. I think the most pushback we've gotten is if we're trying to roll out a new tool, and it would possibly impact the engineering teams.

What we've been pretty good at doing is getting them involved in the POC so they can give us feedback on that, and there have been a couple of tools that we were very close to signing with and then we got the engineers involved and they were like, "No. Please don't."

Guy: "I'm not going to do this," yeah.

Sara: So that's the most we've gotten real pushback on. Like I said, InVision is just strange in that we don't get the pushback typically.

Guy: That's wonderful.

Sara: Like AppSec-wise there hasn't really been pushback, but I think there's been pushback on the DevSecOps side, the security SRE side, with turning off or enabling certain features and not being able to do that quite yet.

So that's probably going to be the biggest pushback, because it would be mainly configuration changes to perhaps specific tools or the environment itself.

I think our SRE side has probably had more pushback than AppSec.

Guy: Interesting. So it's more, but because they apply constraints and those constraints might be a little frowned upon, but they might be necessary.

Sara: Yeah, I wouldn't even say "Frowned upon." It's more that it's understood that these things need to change, but it just might take some time to get there.

Guy: Got it. To build the right secure configurations or frameworks to allow developers to tweak and modify and use those technologies, or use those configurations without compromising security.

Interesting. Is that DevSecOps group also aligned like you are on the AppSec side?

Sara: They're not, they'd be more aligned with the platform engineering teams.

But I don't think they have a one to one like we do, because we're even aligned to a degree with the platform engineering teams as well, and performing reviews for any code that might touch production or handles the build platform.

So, we still need to look at all of that as well.

Guy: Got it. First of all Sara, this is excellent. All the power to you for being in this state and getting to this great relationship.

I think for all of us commoners, if you have some good tips or if you were talking to a team that is looking to level up their security-fu, whether it's on the dev side or the security side, what would be one tip you have or maybe a pet peeve of something you see people doing wrong that you would share to help just move up the journey of doing security better?

Sara: One tip I would have is something I'm working on myself, where I've been out of development for so long that if you want to level up security, especially on the AppSec side, stay in development as much as possible.

Because if you need to do code reviews, it certainly helps if you are still in the mindset of regularly reading code.

Everyone on our team is really developer-first and security-second, obviously strong security, but you need to be in the code.

That's one thing that we do for our security reviews, is we actually do full code reviews.

I think that's the biggest thing that many security folks may not think about, especially if you come from the sysadmin side. You need to be able to get into the code and truly help the developers, and show them the specific line.

Like, "On this line you need to do this."

Guy: That's really good advice. It's just a very relevant skill. You have to understand the material, like the thing that you're reviewing.

I think that's sound advice, and I think one that hopefully more and more security teams are getting, but still need to keep improving on.

Sara, this has been a pleasure. Thanks for coming on the show.

Sara: Thanks for having me.

Guy: Thanks everybody, for tuning in. I hope you join us for the next one.