Ep. #12, Keeping Cloud Foundry Secure

Guy Podjarny: Hello, everyone. Welcome back to The Secure Developer. Today, we have with us Molly Crowther from Pivotal and Cloud Foundry. Welcome to the show, Molly.

Molly Crowther: Thanks. You actually pronounced that right. I'm feeling impressed.

Guy: I get a lot of practice with the Guy Podjarny, not the most catchy tune that you'd heard. Thanks for joining us today. I think we have some really interesting conversations to talk about today about the Cloud Foundry security and open source as a whole. Before I give away too much, do you mind introducing yourself a little bit?

Molly: Yeah so I work at Pivotal. I've been there for about a year. I'm actually a technical program manager, which is called a TPM. What we do is we bridge the gaps between all of the small, highly focused, microservice-focused engineering teams at Pivotal. I help them talk to each other, breaking dependencies between them, and we also tend to focus on horizontal aspects of the whole Cloud Foundry platform including security or docs or just other things that every single team needs to get right in order to create a coherent platform.

Guy: Okay, interesting. So the security ownership itself still lies with the different teams, and you're more the overseer, or is it the other way around?

Molly: It's very distributed. So we have a few product teams that work on security. So we have a security enablement team that builds tooling that helps the teams get better at security. I also co-PM a security triage team that ingests vulnerabilities from customers in the outside world, and tries to understand how bad they are, and the priority of getting them fixed.

Guy: Okay. Sounds like a pretty broad role because I'm tempted to ask a bunch of questions there but maybe before we dig too much into that, you work on the Cloud Foundry for Pivotal. Can you give the audience who doesn't know just a couple of moments on what that is?

Molly: Yeah so Cloud Foundry is a application developer-focused platform. Usually when I'm explaining to people who know nothing about the cloud, I just say that developers write applications, and it helps them easily get their applications up on the internet.

I think the best thing about Cloud Foundry is once you have your application, you can just run one command, and it takes care of all of the routes and getting your code to the right places and putting them in the containers and making sure the containers are healthy.

Guy: Yeah, the wondrous cf push commands to just move it, and it's a platform as a service, right?

Molly: It depends. Depending on the company running it, because it's an open source product, it can be hosted, it can be on-prem customers. It is distributed in all sorts of ways.

Guy: Interesting. And yeah, it's definitely a very broad platform. I must say that I was aware of Cloud Foundry and Pivotal for quite a while but really only dug into this ecosystem in the last year. I've been quite overwhelmed by both how powerful it is and how widely adopted it is. For instance, the fact it's the backbone for IBM Bluemix and the SAP Cloud and just the wide use that it has in enterprises that are looking to modernize their processes and control it so definitely an interesting environment. Sorry to maybe belittle it with a platform as a service but I do see that sort of concept, I associate Cloud Foundry amongst other things as the manifestation of platform as a service as well.

Molly: Yeah. What's really great about it, working on it in general is just it is open source. It's managed like all of the intellectual property is managed by the Cloud Foundry Foundation. While Pivotal is a really big contributor to it, we're among all these other huge companies in this ecosystem, and we're working together to improve it.

Guy: Cool. What was the history of that? I mean how did that come to be?

Molly: Yeah, it's kind of complicated. I think Cloud Foundry started as a project of VMware back in the early 2010s, and then at some point, EMC bought what was Pivotal Labs at the time, which has been around since the '80s, and is still like software consulting arm of Pivotal. And then at some point, they took Pivotal Labs, spun it out with Cloud Foundry and a bunch of data products as just Pivotal software.

And that was in 2013, and then I think the first Pivotal Cloud Foundry distribution came out at the end of 2013. And then at some point, all of the IP was turned over to the foundation so that we could build a larger ecosystem with a lot of other players, and it's not just Pivotal.

Guy: Okay. Definitely an elaborate history but also I think an interesting model today for running this between the foundation and Pivotal but also IBM and a bunch of others that are participating. Okay. We'll get back to that a little bit. Thanks for the background. Let's dig into security. So you run at the end of the day, or at least help manage a lot of the security practices of the Cloud Foundry platform, right? Out of Pivotal. What are the highlights? I mean what are the key tenants in managing Cloud Foundry's security?

Molly: It's interesting because

As of 1.5 years ago, a lot of Cloud Foundry security was somewhat ad hoc and very crisis-oriented. People would find things, and everyone on as many teams as possible would rally around whatever the issue was and fix it as soon as possible and do whatever public disclosures were necessary.

I think it was also a scary process. So around the time when I was hired, we started focusing more on trying to refine the security process a little bit more. So we spun up a triage team that takes care of incoming reports and understanding the CVSS scores of those, and being able to handle communication back to a reporter because obviously, when somebody reports something, it's important, and we want to fix it as soon as possible, and we want to make sure that that person is happy, and wants to keep reporting things to us.

Guy: How do you receive such reports? 'Cause it's an open source project. When somebody to just open a GitHub issue? Does that happen?

Molly: That does happen sometimes. We try to balance somewhat discouraging that with also understanding that we are open source, and trying to be as transparent as possible that there are issues. We accept reports at various email addresses. Security@cloudfoundry.org goes to me and a couple other members of the foundation who work to triage issues, and then we also get reports specifically from Pivotal customers to security@pivotal.io, which also goes through the same process.

Guy: Okay, cool. So now there is an email.

Molly: Yeah.

Guy: Somebody reported a vulnerability in your system that goes to you and a few others. The triage team kicks in and others on what's to do. What's next?

Molly: So what we usually do is we take our first pass. Is this issue reproducible? Do we think that this is a real issue? And then we've done a really good job at personally talking to most of the different product teams so we have connections with all those teams, and when we find an issue, we'll jump into their Slack channel, or if they're in the right city, we'll go tap them on the shoulder.

And we've learned a lot from that just because I think when we first started doing it, and people weren't really sure what the triage team was doing, any time I would walk around the office, people would be afraid that I was coming to talk to them. It was like you're the principal of the school, and nobody wants to get in trouble.

But I think people have learned that when we're triaging an issue, we're really just trying to understand the severity of it, and not making them drop everything immediately to fix it because that can be really disruptive and somewhat terrifying.

Guy: Yeah, yeah. Not a very pleasant experience. Okay, understood. So you're doing this, and you're running around. This communication and these product teams span companies, right? Some of them are Pivotal, some the foundation, some maybe IBM or SAP or others?

Molly: Yeah, definitely. It's just depending on what comes in because there are various things that either aren't maintained by Pivotal teams or open source teams that happen to work in the Pivotal office. We have people from SAP, IBM, VMware, Spring in general, which is a Pivotal product but is also open source. So we have like tendrils into lots of different organizations, yeah.

Guy: Cool. And then let's say it was resolved or not I guess, you know? You take care of getting a CVE or process like that?

Molly: Yeah, I'm pretty proud of where we've come with our CVE process. I think last year in 2016, we ranked well in cvedetails.com. There is like a website that aggregates a lot of CVE information. I think we were in their top 50 companies of number of vulnerabilities reported, and that was really cool because that was mostly me like manually writing out CVE notices and having them posted, and we have also split up a little bit.

All of the notices used to go directly on the Pivotal website, and that was really confusing for people who are like, "This is an open source component. Why is this on the Pivotal website and nowhere else?" So we split all those out onto cloudfoundry.org. There is good RSS feeds for them now so you can subscribe to them, and we're, in general, just really proud of how open we are when there are vulnerabilities, making sure that they're publicly disclosed and fixed as quick as possible.

Guy: So Cloud Foundry, maybe by its very nature, it's a platform, and this is how I see it. It's a platform that is a good mechanism for enterprises to modernize their practices, and as a result, it's adopted by many security-conscious enterprises, right? If you look around and you see who's using Cloud Foundry, many of those are banks or insurance companies or these telcos, large companies that are very familiar with security and have produced stringent security requirements.

How do you communicate with them? In my experience is that organizations like that oftentimes for instance like to get early notifications, they like to have somebody to blame, they like to have some clear ownership. I'm trying to avoid saying somebody to sue. How does that impact your work, if at all?

Molly: It definitely impacts in terms of how we explain potential vulnerabilities to the development teams and the product teams. That was one thing that we had to learn pretty early on that when a customer comes to us and says, "Oh I think there is something here where like an operator could exploit the system," so someone actually running the platform at the company, one of our employee is like, "This is a way they would exploit the system," and for a lot of teams, that's like, "Well, they're an administrator."

Guy: Yeah, they're an admin. They can do everything anyway.

Molly: Right, they can do everything anyway so why do we have to prevent them from doing this one particular thing?

So we've had to get the engineers to understand a persona of the malicious operator who either has socially engineered their way into a job or is disgruntled and wants to set booby traps for later.

So that's like a big education thing that we've had to do just to understand how to securely develop your code while planning for people within the system trying to break into areas that they shouldn't be. We've also had to learn a lot about interacting with large enterprise customers when they come to you, a penetration test or results of security scanning. Sometimes based on the results that we get, it's not that we have to turn them away but we do sometimes have to compromise with them and what ways we're gonna address some of these issues.

Because a lot of security teams at some enterprises don't necessarily know this. They have the security experience rather than the specific Cloud Foundry experience so if they run scans on every single VM or every single container within a Cloud Foundry installation, they're just going to come back with 100 pages of, "I got to an IP that I shouldn't have been able to get to," or something like that so there is a lot of work that we do with our customers to understand how to address issues in an agile way rather than a waterfall way.

Guy: Yeah, I love the personas aspect of it. Definitely a difference in there, maybe the sensitivity indeed to which malicious actors to be aware depending on the audience you're in. So before we move off this, maybe dig into the agile dev and such, a couple of other topics I'd love to still explore on the vulnerability reporting side of it. Do you give any early notifications?

I mean you're an open source project so to an extent, as soon as it's out there, as soon as a vulnerability is fixed, that fix is in the open, is there still some mechanism to have the enterprise customers find out a vulnerability ahead of time?

Molly: So there have been a lot of philosophical discussions, I could say, about this within the foundation and a lot of the companies that belong to the foundation. Currently as a closed source distributor, we don't offer really access to information to customers.

Guy: Sorry, is it closed source?

Molly: Yeah so in this process, we have to wear a lot of different hats like me in particular. So I'm a representative of the foundation in the process but also a representative of Pivotal. So the information that gets shared with customers who purchase closed source distributions is restricted until the vulnerabilities are public if that makes sense.

Guy: Yeah. Let me echo this back and see if I got this correctly. Technically you distribute closed source Pivotal Cloud Foundry so you could introduce a fix there before you made it publicly available but that would make it somewhat unfair to those who use the open source version for it, and in the dynamic between the closed source vendors and the open source foundation especially given you managed security issues for the open source foundation as well, the lack of neutrality here is too big of an issue to bear.

Molly: Well, it's not that we can't release software that is patched, it's that we have to be very careful about what we can actually say about why we're releasing that software because the member companies themselves do receive some early warning about things that are coming out.

And we try to reduce the amount of time between when a fix is committed and when that information is actually public so once a fix is committed, people are working quietly to patch everything that they can, and all of the member companies are doing that but we're not allowed to say anything about it until the vulnerability itself is made public.

Guy: So the fixes themselves would make it into the open source code without much detail, and maybe after that, somebody would come back and edit some commit messages.

Molly: Exactly.

Guy: Afterwards to clarify. Okay. So I find that relationship between Pivotal and the foundation to be a fascinating topic especially when it comes to security. As you said, you have this at least dual hat if not more, and beyond that, there is also other member companies that are, to an extent, competitors. They're not really because I think every one of these platforms has its own unique differentiators.

But other commercial Cloud Foundry-based solutions that are benefiting from Pivotal's work, and specifically from your work on the space. How does it work? So the split in responsibility, security responsibility between the open source Cloud Foundry Foundation and Pivotal?

Molly: It's interesting.

Most of the member companies do their best to not differentiate themselves based on security because we think that is detrimental to the platform as a whole.

So generally everything that we do in security is either, if it's a separate service, it ends up being open sourced. CredHub is a really good example. It's a credential management solution that's tailored to Cloud Foundry that has been recently open sourced. We definitely work as closely as we can with the other companies. In particular, timezones are a huge issue for me just because most of the other people that I work with at the foundation on security tend to be in Europe.

So getting timezone overlaps is really hard. As far as I've seen, the relationship between the companies and their relationships to the foundation are all really productive. It seems like they would be competitive, and I think probably the sales departments are very competitive but in product development, we all understand that we're working on this together and for the ecosystem to survive as a whole, our main goal is to collaborate well with each other.

Guy: Yeah. To me, it seems great. I love to see both or multiple companies collaborating on security and specifically the relationship with the open source element in it. I think it's also again fascinating because for the most part, when you look at the open source business models out there, differentiation on security actually is the differentiation oftentimes, not just security but rather on enterprise greater liability from assurances and support and all of that jazz but also security and security of the platforms and handling vulnerability management. I'm sure this is a frequently discussed topic.

Molly: Definitely.

Guy: But it's great to hear that, that the companies are collaborating well on it. Maybe switching a little bit to a slightly different topic, and I think it's interesting to look at how security is managed on the platform itself, which is what we have discussed so far, but I think another area of interest to our audience is to talk about the platform maybe as a means for security, right?

To talk about securing in an agile development environment. So maybe if we can discuss that a little bit. Pivotal is maybe one of the most committed development organization in terms of modern methodologies, pair programming, all the extreme programming techniques around, and it's interesting. We heard a little bit. You told us a little bit about security playing a role there in the security enablement and the likes and the tools running there.

What aspects do you see, what interesting development practices you think people can adopt from how Pivotal does security, and which interesting security practices do you see running on the platform itself if that makes sense?

Molly: Yeah so one of the things that is really cool about Pivotal that you mentioned is the focus on extreme programming and agile development in general, and what's interesting about Cloud Foundry is that we're kind of an embodiment of those philosophies.

We're experimenting on actually taking that Pivotal model and scaling it so it's not just one development team that's doing pair programming, it's hundreds of engineers within like a 2,000-person company all over the world.

So there are some interesting things that we've been doing with security if you consider each of the little product teams that are building the platforms is their own application or service teams. One of the things that we've been doing a lot more is running what we call security workshops, which are both to level up the experience of particular engineers with respect to understanding threat modeling and just basic security concepts while also trying to learn how to attack their specific part of the system.

So we're both improving the security of the system and hopefully teaching these engineers more about security so that when they inevitably rotate to other teams, they can bring that expertise and help scale the security education as a whole for the company.

Guy: Is this a educational workshop? Is this like a mock application that you run through with some frontal, like some presentations in it, or is this literally pen testing their applications and running tools?

Molly: It's taken a couple of different forms. We actually have some cognitive science pivots who are working with us on it to design better.

Guy: Sorry, pivots is Pivotal employees?

Molly: Oh yeah, Pivotal employees. Yeah so we have some cognitive scientists who are helping us with understanding how people make a switch in mindset so one of the things that's really interesting is the

Product teams are often really focused obviously on their own code and pushing features and things but they also work from a mindset where they're empathetic to their users and empathetic to each other and assuming the best of everyone, which doesn't necessarily help you get better at security.

Because you need to understand who the people are that are attacking you and the fact that they don't have any respect for your stories or your code or the law so those security workshops, we usually have a couple of different activities that are related to understanding threat modeling and understanding that mindset of an advanced persistent threat against your part of the system but also drawing architectural diagram of what your component looks like, where does the value move through that system and how can we protect it.

Guy: First of all, cognitive scientists? Wow. I guess I never really thought of the complexity or of how building empathy and thinking about your users and putting them first could actually get in the way a little bit of security mindset because you trust or you love your users too much to believe they would do something so evil. So you do the workshops and you teach those components. It sounds like the dev teams themselves own many of the security aspects or at least should care about security. Have you seen this as results? Like do you see initiatives, security-related initiatives coming from the team? -

Molly: Definitely.

Guy: Central tools that were actually initiated by the teams?

Molly: Yeah so one of the parts of the agile process in Cloud Foundry is running what are called inceptions. You basically take a large feature and understand the scope and the inception is a meeting where you meet with the whole engineering team and other stakeholders to understand the goals and anti-goals of that particular feature initiative.

So I've been to a number of these where they were related to specific security improvements for the open source that weren't initiated necessarily by an engineering director telling them they had to do it or somebody from a security team telling them that they had to do it. But it's like, "Oh here is something "that we have to accomplish, and it turns out if we do it this way, it'll be more secure."

So teams are definitely starting to understand that security is debt that they have to pay down, and that when they do new things, if they do it in a more secure way, they don't have to spend as much time paying down that debt.

Guy: Yeah. Interesting. I think Pivotal has always been at the forefront, as we just discussed, around development practices so we'd love to see the Pivotal dev process or in fact, maybe we are seeing it being an example of embedding security into that process as well, right? Of having that security ownership there. I think everybody believes that security should be an aspect of quality, should be something that's built-in, and it's just really hard to do it.

I also really like, even just that name, I mean I've mentioned this a few times, the name of the security enablement as a name of a team because I think it very much demonstrates the fact that this team is there to enable security that is performed by the dev teams because that's the only way to scale, that's the only way that it would get embedded in.

Molly: Exactly.

Guy: So I still want to dig into. I think I have time for one more short topic, and then maybe a bit of a closing question. This was all about Cloud Foundry and your processes, when you talk about your users a little bit, you say Cloud Foundry organizations and they're using a PaaS or maybe they're running containers on it. Let's focus a little bit on the platform as a service angle of it. How do you see that impacting security? Does that make it more secure or less secure to use a PaaS versus, say, running containers or running your own systems?

Molly: I guess I think back to my experience previously before working at Cloud Foundry. I was working at a small energy efficiency consulting firm, and we had a particular person who was more excited about DevOps than most of the other developers, and we basically figured out Docker. Before Docker was a thing, he was trying to run Docker and trying to run our applications, and I think of people basically building their own platform and not realizing it versus "okay, we have a company with 500 people, dozens of teams working on this for you."

There are always gonna be things that you, as someone trying to roll your own platform, don't think about, and that I think for our customers and Cloud Foundry customers in general, the actual line of where people are providing value in their organization is at the application level. They're not winning awards for running Linux servers securely. So they should focus on finding a platform that they can live with in terms of its security because the alternative is to do it yourself and probably mess up.

Guy: Yeah. I would say even more strongly, I'm gonna think maybe it's the same concepts that you're saying right now, which is the more you do, the more opportunity you have to mess up security in it, and when you run with a PaaS, then you have the opportunity to put your security constraints and practices into the platform itself, and have a very small number of people do that, if at all.

Maybe somebody else is operating the cloud for you and that reduces the opportunity for a developer to mess it up and fundamentally it's the pros hopefully that are running the platform itself. I definitely believe that that type of notion extends to servlets and function as a service, which is at the end of the day just a variant of platform as a service with some specific attributes to it. I'm a fan from a security perspective of centralizing security when it doesn't get in the way of agile.

Molly: The point is to make it as secure by default as possible, and also build in as many layers as possible to prevent your valuable data from getting taken.

Guy: Cool. So this was fascinating. Before I let you off the hook, one question to ask all of my guests on the podcast before you disappear. If you had one tip to give a dev team looking to level up their security practice or maybe just a security pet peeve that you're trying to get people to finally get right, what would that one thing be?

Molly: People have probably said this before but credential leaks are a huge deal. I think they're in the news practically every week. Somebody made an S3 bucket public or somebody's credential was in GitHub and bitcoin miners spun up a bunch of VMs that cost them a lot of money. So yeah, they're very preventable. We've built a lot of tooling around securing credentials and trying to scan for when they get leaked. So I think developers in general should just be really careful about their credentials.

Guy: Their credentials. Cool. I think that's a good one. I'm not sure if others have mentioned it on the show, and I think it's this single point of failure that you have, right? You built all these controls about who can run what and who can do it, and then you end up leaking that person's credentials and it's all free and open for the attackers. Cool. That's a good tip. Molly, thanks a lot for being on the show today.

Molly: Yeah, thanks for having me.