Ep. #5, Continuous Security at Chef

Guy Podjarny: Hello, everybody, and welcome back to The Secure Developer. Today we have an awesome guest on the show, Adam Jacob from Chef. Adam, thanks for joining us.

Adam Jacob: Hi.

Guy: We'll talk about various cool topics we teed up, that I think are interesting with the world of security and the world of DevOps and CI/CD, and some very interesting new package management, build system capabilities coming out of Chef that I think are very relevant to the security play.

Before we dig in, for the few of you that might not know Adam Jacob or Chef, Adam, do you want to give a quick intro of your background?

Adam: Sure, I think there's probably more than a few who have no idea who I am. I'm Adam. I wrote Chef, originally. I'm the CTO at Chef, and I wrote a thing called Habitat not that long ago that does application automation. And that's sort of new stuff.

Then, mostly what all of that really boils down to is, I spent the last 10 years going around talking to big web companies like Facebook and Google and Yahoo, and I've also spent a bunch of time with startups. And then I've also gotten to go see really large enterprises ranging from giant banks to insurance companies, retail companies like Nordstrom, Wal-Mart.

So I get to just travel around and see what everybody is doing and see what they're worried about and try to help them get better in terms of the time it takes or the speed to deliver, or their organization or their culture. So a bunch of it is software, but a lot of it is just helping people sort of understand how better to build their organization.

Guy: Yeah, I guess there's always this conversation about whether DevOps and, well, continuous deployment maybe is a little bit more specific, but DevOps is really more about the tools and more about the people and it tends to be a consensus that it's more people.

Adam: It's both, right? Like, if you have a great culture, it's really easy to say that you want to have a good culture. It's really easy to be like, "Oh, I want to empower people," or, "We want to streamline a process," or whatever. It takes nothing. It's just words.

Usually, it's the technology that reinforces those cultural behaviors that hold you back.

So, a good example is, "Oh, I want to do continuous delivery, but we use a terrible source control system that makes it almost impossible to do effective continuous delivery. But because that's the source control system we use, we'll never change."

Therefore, which one's true? Do you want to do continuous delivery, or do you love your bad source control system? So there's this reinforcing circle that hides inside there, and I think you see that all the time in security, too, that same circle and that same behavior.

Everybody says, "Oh, we want to be secure." I was working with a bank who I won't name, and I was on an engagement for a couple weeks. One of my first questions that's going to help them do continuous delivery, and they wanted to, their first target was, to harden the operating system.

I asked them, "Okay, do you have the hardening spec? Do you know what you want to harden?" And they were like, "Oh yeah, yeah, of course we do. We're a global bank." And I'm like, "That's amazing, great! I would love to see that." And they're like, "Oh, we'll have it for you on the day you arrive."

I left three weeks later, and they had never found it. And what they had realized was it didn't exist. Everybody thought someone else had built that, and that it was someone else's job, and no one actually did. It was just this loose conglomeration of stuff that theoretically they were supposed to do, but no one actually could track or knew. And that's a global bank. It probably should matter.

Guy: I think sometimes it's about the mess or the fact that tools can help surface information and hold information in a way that is accessible. And sometimes it's just the sheer obstacle for many of these more complex topics when you talk about deep ops topics around how containers operate or how some machines are orchestrated.

Or in the world of security, when you talk about deeper security, understanding about what's an attack, what's not, and that sort of a constantly moving landscape. Just fundamentally, you have to have the tools. You cannot overcome those problems by sheer education. And at the same time,

if you have tools and people don't know how to use them, or they have no understanding of what they're for, then you would eventually not achieve what you're aiming for.

Adam: Context is everything, right? Tooling is great, but it's not enough. One of the things that we do at Chef and in the security world, is we have a thing called InSpec, which is a language for letting you describe security posture and code.

So you can say, "This particular machine should have this particular policy. That policy means that port 25 should be open, you should be able to auth this way, you shouldn't be able to auth this way." You should be able to talk about packages being installed, you should be able to talk about all of those things and then relate them back up to the actual security line items and talk about their severity.

We care about this thing because of this piece of HIPAA, or we care about it because of this piece of CIS, or whatever. And those tools are great because they sort of combine the documentation of, this is what the standard is, with the check that actually is executable that says, "And are we meeting the standard, yes or no?"

I think whether it's InSpec or tools like InSpec, when you think about that operational part of security and how it applies to those large enterprises, especially large enterprises, but the big web, too, more and more it's becoming that policy is executable.

So that conversation between security and developers and operators becomes a conversation around code as opposed to a conversation around documents, and controls, which I think is really the conversation most people have now.

Some security guy wrote a control and then here's the list of people that say that they can validate the control. But is the control any good?

Guy: Is it actually happening?

Adam: No. I mean, it's not, right?

Guy: Yeah.

Adam: And that's true everywhere. We wrote a control. Here's the list of 10 people that you could go interview that know how to do that control. Okay, what about the other 10 people who could also do that, that aren't on the interview list that wasn't updated on the last year? Do they always do the procedure the right way?

Do they always make that process right? And of course, the answer is no. But we just sort of let it ride because the auditor passed.

Guy: Yeah, it's the notion of protecting yourself from audits, not from attacks.

Adam: Exactly!

Guy: It becomes sort of increasingly there. I love that notion. So with InSpec or, in general, with security, one of the key challenges is that it's invisible. So if you're not doing it, you really have no immediate indicator to the fact that it's not happening.

The user experience, if you will, of not monitoring, not watching for a certain vulnerability, and the user experience of watching for it and having nothing happen, is the same thing, which is, nothing happens, right?

Adam: Yeah, which is nothing.

Guy: Which is good news. So I think anything that helps define and articulate the controls that are supposed to be happening, giving you some mechanisms to understand that the action has been taken so you know that InSpec, the locking or whatever, say, if we use a simple example, the limiting of ports being open, that has been explicitly articulated and explicitly tracked by the tools here, and you can get informed that it has happened.

We see this in Snyk a lot. We watch projects for vulnerabilities, for vulnerable dependencies, and then you have really, again, it comes back to the fact that you have the same user experience. If you're not watching a project, or if you're watching a project for vulnerabilities and none happened there, and we have a lot of these conversations with users today about, "How do you want to hear about it? How do you want to know that you're watching it?"

And today, we remain fairly simple. We just remain in this ongoing report that just sort of shows you, "Hey, you're monitoring five projects" or something, and they have those things.

Adam: Right, it shows that it reacts often, or whatever, and we can measure change.

Guy: But it's great to sort of increasingly have those and have some requirement, whatever, in your Chef and InSpec, and that defines it as well, project X is being monitored for X, Y, Z, so you know it's being enforced.

Adam: Right, and then you stick that in the pipelines.

When you think about continuous delivery and security, security and the validation of that compliance is part of the process by which software goes into production.

And it's part of the way that software gets maintained once it's there, and it's part of that build process. You bake it in sort of throughout the SDLC, instead of it being a thing that happens at the end.

And that's one of those things that's obviously a good idea. So, as soon as you hear it, you're like, "Oh, of course, security should be baked into the process, all through the whole thing."

Guy: The whole built-in versus bolt-on.

Adam: Yeah, right? Sort of, duh. But actually, doing it is actually completely--

Guy: A whole different ball game.

Adam: A whole other ball game. And the thing that we really came to realize, especially with InSpec, but just sort of in general, is that if you can't figure out how to manage that security posture the same way you manage the rest of what you do, it's really difficult to then tell a software developer that it's their responsibility to ensure that that posture is good or bad. Because, sure, they can make that they write "good code." You can't hear the air quotes, but m I was making little air quotes in my brain, probably.

Guy: I can attest to that.

Adam: It's good, but you can't really ask them to understand the posture of what it's going to be like when it's deployed.

Because the distance from a software developer making a decision to a software developer talking about how that software should be in production, and what it's posture ought to be, is so vast. And their ability to influence it is so low that it's really difficult to come back to them and be like, "Oh, this was your responsibility." Like, clearly it was on you.

And when those tools give you the ability to talk about it as code. They allow people to participate, you can code-review them. You can have security people audit the code as opposed to audit your documentation, and then have those things as a living piece of that deployment model, I think everything gets better and it gets a lot more secure.

One hard part of doing things that way is that it's very not the way that most security regimes are set up right now. If you go talk to a security officer and you ask them, "Hey, could we remain compliant to whatever the standard is, HIPAA?" If everything we did was continuously delivered, nine times out of 10, the answer is just a flat no.

And what's interesting is that there is, one out of 10, where that security officer is like, "I don't know, maybe. Tell me more about how that would work." And I think if I roll my clock back three years, it was 10 out of 10 that we're telling me no and that was crazy. And now it's like nine of 10. And I predict in six months, it'll be six of 10. And then in two years, it'll be like talking about, "Should you be doing continuous delivery or should you be doing Agile?"

Guy: It comes back actually to the commentations sometimes or even that document or something that big bank did not have. Like, these things eventually come back to the documents and to the guidelines. So at the end of the day, you go down the compliance route and you actually have that in the compliance document that it says, well, if you are using continuous delivery, then to be HIPAA-compliant, you need to ensure that you're doing these things as opposed to just sort of having the goal in it.

Many of these regulations have the same flaw, the same notion that says, "Well, you have to do these actions," as opposed to, "You have to achieve these goals." So they give the vague goals, but then everybody prescribes to whatever recommended actions that have been prescribed, because that makes passing the audit the easiest.

Adam: Yeah, and I mean,

the relationship between the auditor and your security posture is pretty tight.

Like, the real test of your security posture, in most cases, is the auditor. Not like a pen test, or people actually trying to break you. I mean, they are, but you're not doing it proactively.

I think when you think about those CD pipelines, the idea that they're applied continuously, that as applications change, that you're reapplying the security posture to see if that application has done something that violates that posture, and that when you change the security posture, you're re-validating the applications and you're doing it sort of throughout the whole cycle, it's super powerful. It's what people are starting to be able to do. I think you don't see it a lot yet, but it's sort of the future.

Guy: I think that's interesting to understand, the potential versus what's happening, right? High-level continuous deployment and your whole infrastructure's code, the fact that you've prescribed it, built it in, allows you to get predictability. You know what's where, and you know that a certain test or a certain enforcement has been done to the extent of bugs. But at least, not as human error.

On the flip side, it requires security auditors to change how they behave. It's not just about that compliance, it's also the fact that today, many of these security audits, you gave the example of a security auditor reviewing your code, are done as gates.

They're done as a way that says 'stop here,' which is the antithesis to continuous.

The whole notion of continuous is just, you roll out. It's okay to pause for a moment, but you can't stop. You can't accumulate a backlog, because otherwise it'll deteriorate, unless you automate that.

Adam: Yes and no, right? There's continuous deployment, and then there's continuous delivery, and they're not quite the same. So yes, in continuous deployment, there's nowhere to pause. I think in continuous delivery, there is.

In continuous delivery, the idea is you're shipping when you should be able to ship, any time that the business required shipping or it made sense to ship. And that's different than saying, "We just ship every time you commit," right? And so in continuous delivery, I think there's plenty of space to say that, "Hey, this project, in order to ship, requires a security review." And that process can still be continuous.

The question is, that should be the only gate between you and getting to production. So if they say yes, could you ship? And then the question of course is, is that true for every commit? Like, could you ship today, if today was the time? The answer to that is very rarely yes, and so that's where the difficulty comes in.

Guy: Yeah, and I think it's an interesting comment. I accept the delta between continuous delivery and continuous deployment, but it seems to me that from security route, well, first of all, even from a quality perspective, one of the values of continuous, both delivery and deployment, is the fact that you spot errors when they're small, at any given time.

So definitely, for continuous deployment, when you ship every small code change, whatever, at some low resolution, when a problem occurs, it's much easier to pinpoint to what was the issue, the source of the problem.

That value proposition is pretty compelling in security as well, right? If you are looking for a security flaw, being able to, only looking at it within the range of whatever the 100 lines of code to change versus the 100,000 lines of code or 100 million that existed, is very valuable. If you accumulate those changes and you need to now do the security audit, I would argue you're not actually ready to deliver in that spot.

Adam: I would, too. So, by accumulation, the question is where have you been applying those things in? This is where a pipeline shape becomes important. If you're doing continuous delivery well, or correctly, I would go so far as to say, then almost certainly, what you have is an acceptance environment.

The idea that you haven't deployed to production is separate from saying that you haven't deployed the code at all. You have deployed the code. You should've been deploying it. It should be in a running environment. You should be able to see it. It should be close to identical to production, and you should be able to run the exact same set of checks and gates that you would run when you deployed it to production. And that acceptance environment can live anywhere.

In some organizations, like parts of Facebook, for example, that acceptance environment is a slice of production. Some percentage of production users see the acceptance version of Facebook. And that happens to you every once in a while, you'll be like, "That was weird. I saw this feature and then it was gone." And then like, what happened is for that minute or whatever, you were on the acceptance slice and you were seeing if everything was cool.

Then, in other organizations, that's a real environment that's completely walled-off and users can never see it and whatever. But in that pipeline, in that flow, there's a moment where you're applying those changes, you're doing the same check you'll be doing in production, and then the question is, "Do I or do I not want to ship it all the way out to all of my users?"

I think that can still give you the same benefits of a continuous deployment model, even if it has a gate. The tricky part is, that what you can't have, is arbitrary numbers of gates, and you can't have gates between environments.

How many gates you have matters a lot.

So, automated deployment into five environments, where each one has an arbitrary gate; it might take you eight months for that code to work its way through production, and each environment is different. You don't do the same thing in each one, that's accumulation that's useless. That's danger. Because the delta between the time it takes for your change to go to production is so long and then bad things happen to you, for sure.

Guy: Yeah, it becomes relevant. So, I guess maybe if I translate that to practical steps, I would say there's a certain amount of effort that you would be willing to make in just that sort of promotion to production, if you will, that is okay. But that amount still needs to be contained in terms of duration.

Adam: Yes.

Guy: Probably in terms of manual effort.

Adam: Absolutely.

Guy: It cannot scale linearly with the amount of code that's just been shipped.

Adam: Absolutely not.

Guy: So I think in the world of quality, you'll talk about how you have a whole bunch of unit tests and all that, and you're deploying to some staging environment or acceptance environment, and maybe over there you're going to run a more extensive battery of tests for its duration or cost or whatever, or stress test or something that, you cannot afford to do on every commit.

Adam: Right.

Guy: But you wouldn't sacrifice. You would still aspire to find as many of those flaws earlier on.

For security, you would want security controls as part of the continuous delivery aspect of it, so that they're always there.

It's okay if, in the promotion to production, there is another security audit as long as it's contained to that.

Adam: Right, I would argue there should be another security audit. There should be, you should repeat it, right?

Guy: Right, like some bigger pen test. You probably should aspire to automate that, as well. It just might be more expensive time-wise, cost-wise, complexity-wise.

Adam: Right, there's trade-offs. That's exactly right. And the thing people don't tend to do is bake it into their pipeline. I personally believe that there's not 100 useful pipeline shapes. I actually think there's maybe one.

And we got off track because we talk about, we use different vocabulary words, to talk about the different pieces. But in the end, the number of people who go through this journey toward continuous delivery and think about having security as a step, is quite small, right?

People don't think about it as a stage, or like as a phase that happens as you move through deployment. And they have to start thinking about it that way if they're going to be able to push it through the entirety of their organization, or if it's really going to transform the way they work.

Until they start thinking about security that way, in the same way that we think about operability, in the same way we think about reliability, like it's a thing that's your responsibility that's baked in from the beginning and it lives in the same pipeline and the same flow as the rest of your work, well, we'll continue to see failure there.

Guy: Yeah. And did you see, so I guess from an ecosystem perspective, you mentioned that not that many people think about security, do you even feel that this is sufficiently encapsulated, I guess, for you to indicate whether there's a trend of a growing amount of security activity in the pipeline?

Do you see, like in the terms of the conversations you have about embedding security, leveraging Chef, like, are there Chef Recipes or Cookbooks that are specifically focused on sort of some security control or action, or are they increasingly used? What do you see?

Adam: Yeah. So the conversation we see, especially in large enterprises, is super high. It's happening all the time. They don't have good answers, and they want them. And that's because there's this macro-economic trend that's pushing every large enterprise to become a software company who is good at delivering software faster than their competitors. Like smartphones.

How many? I took an Uber and I didn't take a cab. Cab's cheaper, but it's a pain in the ass to get the cab, so I didn't do it. And that same thing is happening in retail, it's happening in all these different markets. And so that, over the last couple of years, has floated to an executive, C-level conversation. It wasn't before.

Three or four years ago, we were having that conversation with an operations director or maybe a VP of engineering or something. But you weren't having it at the CIO level, the CSO level, the CEO level of Citibank or someone like that. You are now, right? And it's not one of them. It's all of them.

And so the question for them really is, "What are we going to do and how will it work?" And so that trend, I think, is pushing for answers. I think when it comes to specifics, Chef, the software, has, you can think about it as remediation in the security context, right? So when you run Chef, it checks to see if the system is configured the way you said it should be. If it's not, it fixes it.

There's lots of Chef Recipes that do security posture, where it's both. Each resource is a check to see if it's right. If it's not, then the remediation is sort of baked in. And that is pretty effective. And what's interesting is how much of that you can or can't share.

There's some things you can share, like CIS benchmarks or whatever. They are what they are and they tell you precisely what to do. And then you look at like Sarbanes-Oxley, which basically, there are no rules, except for the rules that the auditor tells you, right? However, the auditor interpreted the very small piece of legislation, that's your business now.

And so there can't be a generic Sarbanes-Oxley checker, or like a generic Sarbanes-Oxley recipe that sort of fixes you because every organization's requirements are different.

I think one of the things that you see in that space is there is more sharing than there used to be. It's mostly around best practices, or it's around the obvious benchmarks that you can hit really clearly.

And then for any given security posture for any given organization, maybe they can use something to start out with. Like, there's a template or there's an example, and you can sort of start with that, and then you can add in the things that are specific to your organization. But there's always enough specific to your organization that there's a bit of a heavy burden.

I'm not sure how that gets resolved, because it feels a little inherent, the nature of security within organizations that are still being built or already existed. You have an investment that exists in a particular posture. And so from a developer point of view, like, I don't know how much impact you can have there other than trying to sort of understand the specifics of what your life looks like already.

Guy: I think a developer can still quantify the requirements, not quantify maybe, but write down the requirements, just like a security person would into those Recipes. So it's all about that predictability. I mean, when you talk about one requirement coming in, about which ports are open, and then that requirement could come from a security person. But really, if you're a developer and you've received it four times, you could also take initiative and put that in yourself.

Adam: You should, right?

Guy: The other aspect is areas of application security, which are controls that the security team cannot keep up with. So you know that you've just installed an express web server on this thing, or that you're opening this web socket port and you can try to put or add some initial controls to talk about it. You're the ones pulling in open-source packages and you want to put some controls around their viability.

Adam: And revision, their origin.

Guy: Exactly, their version, known vulnerabilities. And then you would want to also control the actions, maybe, that the system as a whole is willing to do, so you can also define what actions come out. And especially, I think also this notion of, there's the dev, the ops, I'm a little bit hesitant to separate those two, but they're in the era of DevOps.

There's those dev actions, there's the ops actions, and there's the security actions, which are just way, way too separated today than they should be.

So as containers, for instance, they come in and you're building a container and you're shipping it somewhere, or as you're building your software using some IT automation system like Chef. Then really, the separation between dev and ops is very, very blurry.

But at the same time, somehow, security hasn't gone to that mix. Like, still, for security, we expect, or not all of us but all too often, expect some other security team to come in and tell us what is it that we're supposed to do here.

Adam: We definitely have a security posture. Yeah, we just don't know who.

Guy: Yeah, precisely.

Adam: Yeah, I think that's still true. The thing about DevOps that's easy to misunderstand in my experience, is that it's not that people are generalists. I'm a systems administrator, and I'm really good at it, and mostly that's because I've been doing it for a really long time. I was 16 when I got my first job, and I'm 40. So I've just done it enough that I'm pretty good at it. But does that make me immediately a great security engineer or a great software developer? The answers are no.

I'm not bad at either one, because my discipline is close enough to those disciplines that I sort of understand what's going on. But DevOps is actually about having high-functioning teams of specialists who, because they're high-functioning and because they're working together, come to a better understanding of the holistic system. And so when you think about security and security engineers, then you think about software developers.

The goal for a software developer, I think, is to get closer to those security engineers and to get closer to those operations people so that they better understand how the software they build can support that piece of their software's life cycle.

Because it's not like you write the software, and then it goes off and gets deployed and then some security stuff happens. In fact, all of that stuff, applications without infrastructure, don't exist. Infrastructure without applications are wasted heat, right?

If there's no security on either, then your customers will eventually stop trusting you, as they should. Because you'll be awful. So, understanding that, that ecosystem is holistic and that you need all of those components to make it work, means that you have to build a team that's capable of doing it. And then those teams need a way to work together, and I think that's what's pushing that trend toward continuously delivered security.

Where do we work together? The answer is on the code. And then the question is, "Well, I'm a security guy, where do I put my code?" If I did that, which I don't now, but if I did, where would I go? Providing those answers, that's the next frontier.

I think sometimes it's stuff like InSpec, I think sometimes it's things like better active scanning, it's better penetration testing. There's a bunch of those sorts of things that are hiding in there, and it's because not one-size-fits-all, because it depends on the environment and it depends on your problem.

Guy: Yeah, I think I really like the idea of allowing a security person to write their code into that pipeline. Because again, it doesn't sort of shake off the responsibility. You might put some ops requirements; it doesn't shake off their responsibility of building operable code. But it does allow the expertise to come into the code, into the continuous pipeline.

Adam: Where other people can see it!

Guy: Yeah, can see it, can learn from it, maybe can put it after the fact.

Adam: You can ask questions! It's right there and you review it. Who reviews a security person's code? And the answer should be, the developers and the operations people who have to deploy it, and vice-versa, right? And that's just not how we work in a lot of cases. And once we start working that way, everything gets easier.

Guy: Yep, although, I think this is very much an environment in which we need better tools. Because most security tools today are not really built for that mindset. They're built for audit purposes, for manual operations. So it's really functionality like InSpec that would allow you to say, "Hey, you are not allowed to open a port," or whatever.

Again, I'm sort of thinking in the simplest environment here, and it's maybe sort of some vulnerability feeds for the Ubuntu or Red Hat, sort of patches, fixes, that allows you to say, "Well, the system needs to be unpatched." But for many of these other things around vulnerability scanning, around fixing those things, around pen testing...

Adam: They weren't built that way at all.

Guy: Around authentication tests, there are amazing technologies out there absolutely and entirely not built for this use case. So probably useless in this context. And we need a new breed of tools, right?

Adam: It's the next frontier in terms of that tooling. Because, you think about where that stuff fits in and how it's going to work. And it's so important and it will absolutely get there in the same way that you look at, you can look at the popularity of languages,

the difficulty of getting a language into CI is a huge burden in the adoption of a new language. Until that tooling supports gets to a certain level, people just won't do it.

The number of people who will invest is--

Guy: There's only so much adoption you're going to get.

Adam: There's only so much you're going to get.

Guy: The die-hards.

Adam: Yeah. And so in security, because it's showing up a little late in the conversation, but not too late, time is funny, like, if you've been thinking about it for a long time, it always feels late to you. But for most people, it's super new. I think that you're going to see those ideas refashioned in a way that makes sense in a world of collaboration and continuous delivery, and it's super interesting and exciting.

In terms of being an entrepreneur or thinking about security as a place that you could invest as a software developer, it couldn't be a better moment.

Because sort of the game is completely open on a bunch of, probably, very traditional industries and really entrenched software that you could probably displace it simply by fixing the user experience around collaboration and continuous delivery.

Guy: Yeah, absolutely.

A- And like steal a significant chunk of market share.

Guy: And fix it also in the right spot, right?

Adam: Yeah!

Guy: And then again, kind of fix it for the future a little bit, right? Like, fix it for the systems that are being built right now, and then retrofit them to the back, versus the other way around.

Adam: Yeah, that's right.

Guy: So on that topic, a little bit on tool rethinking, existing models. I know you've been working a lot on Habitat over the last year or two. Do you want to talk a little bit, we had some conversations about the security angles within that. Do you want to give probably a quick review of it and then we'll talk about those?

Adam: Yeah, a super high-level review of Habitat is that we call it application automation. And what we realized was that if what we were doing in all these organizations was really getting to a place where we were trying to ship applications faster and better. It's weird that all of our automation starts from the infrastructure and works up. It's just a weird starting place.

Habitat starts from the opposite direction. It says, "Okay, well, what does an application need in order to make that stuff work?" And then it kind of works its way down. So an example of that is, you should be able to build the application, have an artifact. That artifact should have all the things that it needs to run in every environment that it's going to run in, on any runtime it has to run on. That should include all of its runtime configuration, it should include all of its dependencies, it should include everything that it needs to do.

And then it also needs to include the infrastructure that it might need in order to do things like deploy into a complicated topology, or to update itself in a smart strategy like one at a time or in pieces. And then you can start to make the conversation around how an application behaves, be separate from the conversation about what infrastructure it runs on. What we have tended to see historically is that the infrastructure it runs on dictates the application in terms of its shape.

You're like, "Well, if you want your application to be good to manage, then you have to run it on Kubernetes, or you have to run it on this infrastructure style." And Habitat sort of flips that around and says, "Well actually, the application should just be easy to manage." And then there's a conversation about what's the right place to run that application based on its own needs and posture and all the rest of it.

One of the things we had to do when we built Habitat was think about things like, "Okay, how did those applications deploy and how did they bring along their dependencies?" One real problem in continuous delivery is you have your application, let's say you use OpenSSL, which is not crazy, like a lot of applications in the world.

Guy: Yeah, most likely are, yeah.

Adam: Right. And if you're not, you're using it through some other thing you built that does need it. So the odds that you rely on that somewhere in your stack are pretty high. And so you have this moment where there's a security vulnerability in OpenSSL, and so if you cart along all of your dependencies and you're responsible for your whole environment, how do I know that, that vulnerability exists?

Then, how do I know that the software I'm running is running the right version of OpenSSL? And the historical answer to that is, "Well, I patched the machines." I go to Red Hat or whatever, or Debian or Ubuntu or Windows or whatever, and I run the updates. And now I'm secure, right...?

The problem with that is that it's secure on disc. So you've definitely updated the library. Is the application that's using it actually been restarted and read that library into memory? And how do you know? And if you're using something like Chef, if you wrote the Recipe right, then the answer is you could be pretty sure because you can put a trigger that says if this package is updated, restart this service.

Guy: Predictability at its core.

Adam: Yeah, but now you're relying not. But that relies still fundamentally on the idea that someone knew to write that, that way; and that someone who reviewed it, reviewed it and got that done. And Habitat flips that over.

Basically, we had to go all the way down to the bottom of the build system and build a packaging system that allows you to be really explicit about your dependencies, and then have those dependencies be the only possible things you load into memory.

So, if there's a new version of your application, it might be because you have application code that changed. It might also be simply because a dependency moved. And now you need to rebuild and make sure that your application now works on top of that dependency.

Habitat, the experience of doing that when we started, I didn't think I was going to have to go that deep in order for that property of being like, "Okay, how can I tell you for sure what's running when you say 'I'm running this version of this piece of software'?" And the truth was, the only way to get it was to go all the way down to the dynamic linker and make sure that the only possible thing you could link to is the right version. That was fascinating.

Guy: Yeah, it makes perfect sense. It feels sometimes a little harder to digest when you come at it with the mindset of these containers and VMs and kind of that evolution. But at Snyk, actually this was at DevOpsDays Amsterdam, I had this conversation and talked about how in Snyk, in the code itself, we would unravel the dependencies trees.

Which, in code, is actually very much declarative, because it's just in Node. It's a packed JSON file, and it says which dependencies are there, and all your software bill of materials, if you will, is very implied.

I had this conversation with somebody after the talk, "Hey, can you do the same four steps we talked about there? But addressing those, can you do those for Chef Cookbooks?" And we had the whole conversation, and I was spending a lot of time thinking about this since.

It seems like, in the world of ops today, you have the explicit dependencies and the implicit dependencies, because you're bringing along, you're carting along, these VMs at the end of the day, no matter what you call them, like packaged operating systems that have all these implicit dependencies.

Adam: You're bringing the userland along.

Guy: Yeah, you're just sort of dragging them along, and it's really hard to track those, know how they're processed. It's much easier with the explicit dependencies, not the implicit ones. And it sounds like in Habitat, you're basically killing this notion.

Adam: There are no "implicit," that's right. And that was really hard. As developers, it was really hard to decide that, that we had to go that deep. But an example there is, you can take a Habitat artifact and then you can export it to various formats.

So you could export it to a container or to a VM. And what's in that artifact is nothing except your application, its explicit set of dependencies and the explicit userland you decided to bring along. So if you can get by with just a BusyBox userland, that's an explicit runtime dependency of your application and, therefore, it will appear.

If you don't make it an explicit runtime dependency, there might not be a user land at all, right? Other than a kernel, if you needed it, if it was a VM. And that as an approach is so powerful. It takes a minute because you have to build an ecosystem of software so that people don't have to spend their time.

Guy: Fortunately, you've done that once.

Adam: Yeah, ideally we're going to do it one time. I think we're at, like, 250 packages or something like that in the core right now, which is a lot of the most common dependencies are already there. But every dependency, everybody else's dependencies, are common, and then your dependencies are always the weird ones, is sort of the rule.

I think when you think about how you get those guarantees, especially in security, I think the future is really re-thinking the fundamentals in the frame of saying, "Okay, when we built these systems 20 years ago, our goal was to be able to use them at all."

We needed to build them, human beings were going to deploy them, the pace of innovation was slower, the pace of change was slower. We weren't delivering as much stuff to the internet. It was still weird to have a good website if you were a retailer. Amazon.com was losing money every thousandth joke, right?

Guy: That's true up until two days ago or so.

Adam: Yeah, and so when you think about how it's going to work in terms of the future and security, and also with Habitat, that willingness to drive deep and to say, "Well, actually, right, there's this principle here that's really valuable: active scanning."

It's really valuable to be able to take an arbitrary artifact and scan that artifact and tell you what's inside and what its dependencies are.

That's incredibly valuable. It gets a lot easier when you can look at that artifact, and 90% of the data is standardized. But in order to get to that place, you have to be willing to break a lot of glass in between here and there. And those investments, they're just expensive. I think as an industry, we have to make them. And as software developers, they're really fun to make.

It's weird to get permission, for example, to go re-compile Linux from scratch so that we can give you a tool chain from libc up, that has explicit dependencies and no implicit dependencies. That was a super fun project because who tells you that, that's what you should do today? You know what I mean? Like, it was awesome! And the people who built it had a really good time because they came to work one day and we were like, "All right, yeah, F it, let's do it. Let's get it done." And we did.

It's going to take a minute, and that's fine. We're willing to make that investment. I think you'll see more and more, both security companies and individual software developers, the InSpec guys, that wasn't a corporate project for us. They were two guys in Germany who worked as consultants and they built InSpec to solve that problem, when they see that niche in the market. They did that as software developers. And they didn't have security backgrounds. I mean, they did, but not--

Guy: They weren't security people.

Adam: They weren't security guys, necessarily. I mean, they were, but they were software developers first.

Guy: Not their primary job, yeah.

Adam: Yeah. And so when you think about developers and security, for me, the other piece of that,

that's really important, is how much developers are going to drive that revolution in the way we approach security.

There's so much interesting software development to be done, and you just need to find it. And then when you get there, you get to re-think all of these really fundamental assumptions about how the system operates. And you get to gain all of this knowledge about weird, esoteric internals, which, if you're that kind of nerd, it's the best, right?

Guy: Yep. Like there's a quote from, I forget if it's Gene Kim or Josh Corman, talks about how, "To fix security, you need to leave security." I kind of like that sentiment, in the sense that you have to sort of break out of the confines of your perspective of how you look at these things.

Same concept, maybe here for packaging for Habitat, and then when you sort of combine, maybe even never having been necessarily first primarily identifying yourself as a security person, still have this opportunity to now, with this case, specifically. For instance, do a substantial improvement for the security posture.

On one hand, it has better predictability. It's reduced the attack surface, it's the opportunity to insert security controls as a component of the continuous delivery of the software, that sets up your system. So yeah, sounds like a lot of potential here and most definitely sounds like a fun project to work on.

Adam: Yeah, I think it's super fun. But I think, when you think even outside of Habitat, just the more broad question of software developers, what can they do tactically to make impacts in security?

There's always that tactical list of things that's like, "We'll think about that posture, run scans, there's tooling you can use that is helpful, there's books you can read." I think, more than anything, it's being willing to understand, pick a thing that's interesting, and be willing to dive deep into understanding how it works and why. You hear a lot about "don't write your own cryptography," right?

Guy: Yeah.

Adam: This is good advice, because it's hard. Also, what that tends to mean is that people don't investigate how cryptography works because they've been told how hard it is. Do you know what I mean?

Guy: Yeah.

Adam: And the truth is, the only thing that's hard about it is finding out where to start reading. And then how to go from one paper to the next. But once you do that, the mystery goes away. Now, it's still mysterious.

Guy: It's still complicated.

Adam: Yeah, it doesn't make you Dan Bernstein because you read the papers. But it's not like incomprehensible gobbledygook. You can get it.

As a software developer, the more you invest in understanding those fundamentals from other disciplines, the better you get as a software developer.

And when you think about that arc of your career, it's those abilities to play in other fields that are what make people more senior and make you more valuable and make you better on teams.

Guy: Yep, especially in the security space, which is one of the best assets to accumulate.

Adam: It's huge. If you think about income, you think about job security, that's an incredible space to be in, right?

Guy: Yep, that sounds like a really, really good advice.

Adam: Yeah.

Guy: So, this was a great conversation. Thanks a lot, Adam, for coming onto the show.

Adam: Thanks for having me.

Guy: And I think that's it for us today. Thank you.

Adam: Right on, bye.