Ep. #8, What’s In A Security Policy?

Guy Podjarny: Hello everyone and welcome back to The Secure Developer. Today with us, we have Geva Solomonovich. Hi Geva.

Geva Solomonovich: Hi Guy.

Guy: We're going to be talking about security policies which sounds all sort of formal and bureaucracy like but in fact there's a lot of good substance in them. They're not just a necessary evil but something that could be quite useful. Geva has a fair bit of experience there and we'll explore that world in a bit. Just before we get started, Geva, do you want to tell us a little bit about your background, how you got into security?

Geva: Sure, absolutely. I've been, for the last 10 years or so, involved in risk management. I've been in an anti-fraud company. I worked at Paypal, then I worked at another payments company. In all of those, a bigger component was risk management which is very similar to security in the mindset that you're always thinking "what can happen and what can I do to protect myself and how can I better manage it"?

Towards this year, I started doing security consulting. Helping companies with their cloud infrastructure, with their policies, with their development policies and practices. It's been great so far.

Guy: Yeah, it's a good point about risk reduction and sort of all these practices around fraud and security because sometimes we get stuck when we talk about security. About all the technical implementations of them when you talk about running some tests and your continuous integration process or protecting from SQL injection.

When you take it up a level, security and all the security controls are about risk reduction.

They're about reducing the probability of something bad happening. Whether you've done it via reducing fraud and catching a bad payment or whether you've done it via some input validation in your code that prevents an attack. Kind of interesting path, but yeah, totally logical. Let's focus a little bit on that latter section you talked about. So, you do consulting today about security policies. How does that come about? Who do you interact with when you are building a security policy? What's the motivation for starting that conversation?

Geva: Most of the clients or customers I've worked with so far

There comes a point in their lifecycle as a startup where they get their first big customer who says "before we integrate here and I give you some APIs or I show you some data, show me your architecture documents. Show me your security policies. Show me that you're serious". And most of the companies just don't have it.

Then they're at a point where they're scrambling to have something that they can show to this customer so they get the business. At that point, they raise their hands and they look into somebody to help them solve that immediate pain point. That oftentimes is a point where I get introduced into the company and that's a segway for us to work together and then explore other things. But oftentimes it does start with the need for a security policy to satisfy some big customer who wants to see seriousness from the company.

Guy: Yeah, nothing like a big customer requirement to sort of motivate you into action. We're in this situation, I have a startup. I have this first big customer. They ask for a security policy, I come talk to you. What happens? What's the first conversation then?

Geva: The first conversation is, you know, get rid of this pain point that they have. I mean, most of the startups are generally in existential mode so they're working mostly on their product and getting more customers. And security, everybody knows they need to do it, but it's kind of an aftermath. In a sense, it doesn't increase the value of your product when it's secure. But if you have a bad security experience, then it can devastate your company.

Most of the companies aren't as proactive as they could be in thinking about security, and again, it comes to a point where somebody's demanding it from them and that's when they're scrambling and they're ready to open up and invest and put those resources in to do what needs to be done.

Guy: Yeah, I guess security is invisible, right? You sort of you don't see the problem or the fact that you have high risk unless it was exploited in which point, you're feeling the incredible pain but there's no obvious feedback loop. We're there, they understood, they want to take the pain away, what happens?

Geva: Then when we're talking about policy,

I usually discuss three categories of security policies with clients: 1) A presentable policy 2) An internal policy and 3) An elaborate policy

One which I call "a presentable policy" which is a policy that kind of details in very nice English that you're a serious company. That you've taught about security and all these different aspects. I call it presentable because the goal of the product is to give it to a vendor, a customer, a partner, or somebody you're trying to show the level of seriousness.

This is a great policy to have. It doesn't serve with other needs they might have in their organization. A big one is for example educating your employees so it's not a good tool to tell the employees what they need to do and to teach them. That's a different type of a security policy. I call it an "internal security policy" which is more like maybe a handbook, a set of best practices, a set of, you know, this is the stuff we do, these are the kind of stuff we don't do, and this is the rationale.

The internal policy is geared more towards being pragmatic advice that employees will read and understand and do. Then there's a third one. Let's call it the "elaborate security policy" which would be tens of pages, maybe approaching a hundred pages. It covers every different aspect of security in the company including all the procedures, all the processes, all the tickets, all the event handling.

That, of course, is not a good tool for educating employees because nobody wants to read a hundred pages security document. It's not a good tool to send to customers. But it is something you'd need if you're kind of engaging in audit level relationship with somebody serious like you're working with a bank or you're working with the government or something of that nature.

And they don't want to see something that's at a presentable level but something that's really, really deep and that they can verify and then go in the organization. You say you have so and so many hours SLA to handle vulnerabilities. Well, they want to see that you actually can do it.

Guy: What's the relationship between the security policy that you wrote down and the security policy that you apply? I mean, let's take the first example you talked about, the presentable security policy. You go through it, you create a security policy, we'll talk a little bit about contents in a bit. You say you're doing stuff there, right? You say you are using whatever, two-factor authentication on the thing or you're encrypting some private data.

How does that interact with actually doing it? I'm trying to understand the delta between the presentable and the employee-oriented. Is it often that you build the two together with the second one being, the employee one being meant to actually do it, actually sort of apply the security practices.

Geva: I see it more as just the level of the details.

The employee one is like a real practical one, maybe even down to the code level. "We don't use this library, we use that library", which of course is not interesting to your customer. The presentable one is more between a high level and mid level description of your thought process.

Try not to put there too many very detailed things so not to get anybody in trouble but again the focus is to show that we are thinking about these different assets. We have thought about how we access our environments, how we treat our physical devices, the laptops, the locks in the office, the employees, that we do background checks on the new people that we hire. We have an onboarding and offboarding policy.

Although we might not detail it in a whole lot of words in the policy but we do say we have one. How we manage data, we have retention, we have a backup, we have encryption. We have separation between different customers. Your data is not going to get mixed up with somebody else's data. How we look at the network security. We have a firewall. We have routing rules. We test them.

With monitoring, we don't go into detail. How many employees are sitting and monitoring and who's listening on the queue and is anybody getting woken up in the middle of the night. We just say we're monitoring all these stuff, using all these tools, et cetera.

Guy: Right. So this is both indication of the areas of concern that you've identified and all the areas where you understand this is a security consideration. But the presentable security policy or the one that you want to disply, should give you some wiggle room. To say first of all, I want to be able to apply it in a reasonable sense, not sort of commence to some specific components that you wouldn't need to change it all that frequently because you could just evolve it.

But also probably a big component of it would be evolution, right? So I can start off by saying I'm restricting access to my servers and do it by tracking these specific individuals or for the small company, you know exactly who has it. Then, as your company becomes bigger, there might be some other form of management that you're doing that has like more audit levels or has more key management because it was necessary.

But the same policy or the same statements in the policy kind of apply to it

Geva: It's a beginning for a conversation. If one of your customers wants to drill down, then you know he has a starting point and then you can drill down. So far, the feedback's been pretty good. Everybody's been happy. Helping people get more business is always a pleasure.

Guy: Yeah. I find, whenever I interact and even in our own security policy when we kind of built one that we worked with Geva on doing that. Even the areas that you know, even the actions that you know you want to take when you sit down and you need to write them down in a policy. It forces more structured thinking about it. And it forces you to understand, I knew that this is a risk but am I actually taking action here?

Am I modeling it or addressing it in a way that I find satisfactory? It felt very much like a useful exercise to do regardless of your level of expertise, right? Given if it's a type of activity that you already know what to do but then of course for people that are not in the security landscape, many of these might be just eye opening, right?

Geva: Definitely, that's the case. There's definitely different levels of education and different levels of attention to their security. Companies of different levels have invested in it or have different types of knowledge. So when I come in and help, starting points could be vastly different.

But with everybody, the first thing I say "I can't come in and spend 10 hours with you and then say you're secure". Because security by itself is a mindset. It's something you need to integrate into everything you do, how you think, how you're building stuff.

We start a conversation, and every change we need to make, we also talk about why is it good or what could be wrong and how you need to think about it going forward so you can apply the same kind of logic for other stuff you're working on. In general, you're always going to have risks. There's always going to be security vulnerabilities but you need to think about it as a risk surface that you have for your company, for your portfolio, for your cloud and for your infrastructure.

How do I reduce the risk surface all the time? Keep it as minimum as you can. If you don't want any problems, you can just be out of business and you have no problems but you want to be in business so you want to have an infrastructure but just trying to keep the risk surface of your infrastructure as low as possible.

Guy: Yeah, I like that concept of security as a mindset and your task, I guess, the topic that you get kinda pulled in to do but also that maybe is the pain point that people are trying to do is pass an audit in a sense, right? Or be able to provide the policy as opposed to it's really kind of apply the spirit of the law here. But in the process, you're sort of forced to consider it and to apply it.

Of course, you need to be kinda straightforward in those policies. Lo and behold, that security policy might just make you more secure. I like the means, the use of the security policy to help trigger trying to establish a mindset of security and not just the policy.

Geva: An additional part of that mindset was which I try to tell the companies I work with is "you need to protect yourself from what you can see or what you know but also from stuff that you can't see. It doesn't have to be a concrete vulnerability for you to put protection there".

I've been managing companies for many years, and some of the conversation I would have with my engineers is they would ask, "why do we need to put this protection here? How would a hacker get into our database"? And kind of that's a wrong mindset to have.

The right mindset to have is let's assume a hacker is in the database, what can we do to minimize his/her access to our data? What can we do to get notified that we're being hacked more quickly? You don't have to be able to explain how a hacker can get into your database to put more protections on your database or your network.

One more piece of advice I give companies, security is all about layers. Every layer you have, you need to put in some security. I call it don't be an M&M. Don't have a hard shell in the outside but then have a soft stomach in the inside because just because you think your network security might be good it's not an excuse not to protect and patch your servers, not to protect your database. You never know how hackers are going to get inside.

Guy: Yeah. You should use the UK version of that which is don't be a Smartie.

I agree. It's all about defense in depth, it's about the layers and protecting there. A lot of this has been on the concept of the policy and you gave a bunch of examples of it but maybe let's touch on the top highlights. Like if you're a startup, you're a B2B startup. Let's even say the big customer has not approached you yet and you haven't created a policy quite yet, but what are the kind of high level bullets of areas that you should worry about when thinking about your security practices that would then be translated into a policy?

Geva: That's a good question. I tell the companies I work with, you need to think about security, about the design of your security, kind of like you tell your engineers. They need to write good code that has a good design so it can A) scale up, B) it will have less bugs over time, C) it will be easier to detect their problems. The same thing you need to do with your infrastructure. As you're building your cloud, they'll just go and hack servers and throw them in there and let them live by each other and give all your employees access to everybody and everybody has super root access and everybody's SSHing to the machines.

Give it a little bit of a thought. Generally, when you work clean, when you work neat, when you separate your environments. You have your virtual cloud for your production, a virtual cloud for your development, for your staging environment, you give the right access to the different roles in the groups. You give good naming conventions. You start working organized then you can immediately see if you have any problems. It's very easy to scale. You can add in more employees later. It will be easier for them to contribute.

First and foremost, just work organized.

It's not even anything related to security, but working organized gives you the infrastructure to be secure.

Then, kind of additional things to think. There's a big trade-off between the ease of administrating your system and having more security. It's very easy to give all the employees SSH access to the machines. It's harder to maybe integrate the solution that will pull your logs out of the machine and present them for everybody so they can see the logs when they need them. There's a big trade-off between system administration and security, right?

It's very easy to have everything open, give all the employees super root access, everybody's SSHing into the machines, pulling off the logs, changing the code with VI on the machine. But of course it's easy to administer that way but it's not very secure and kind of having a multifactor authentication. It's annoying to get SMS every time you want to log in here or you want log in there.

You need be somewhere on that curve. But definitely, pick only the solutions that get you a lot of security value and not a lot of administration overhead.

Guy: I think that one actually aligns well with thoughts about reliability. That's because reliability at the end of the day is also about risk reduction. It's also about reducing the risk of something accidentally falling down and crashing. I think access control and kind of a good visibility and just sort of knowing that there's a smaller set of potential paths to a destination is also good for just again to help the system.

I can see your first point was as well. If you're not messy, if you're organized, if you know what's where and who can access what, then the likelihood of something blowing up whether it's because of an attacker or because of mistake are on our door. Sorry, but I cut you off. We talked about those two things.

Geva: That's one. Another one to consider is

Work on a minimum permissible policy instead of a maximum permissible policy.

Whitelist the access that you need instead of give access to everybody and try to close it. At the end of the day, you'll find your servers don't need that many IPs open and they don't need that many ports open, so why give access to all the internet. Not 100% of your employees need access to your server so don't give everybody access. Give just a minimum amount of people access.

If you think that way over time, again going back to the analogy of the risk surface, then your risk surface remains as small as it can. You have a lot of employees and they're of course your biggest asset in the company but from one perspective, they're also kind of a liability because they all have these laptops that they carry in their bags and they put in their computer when they're driving to hang out at night.

Guess what, their password is the same one they've been using for the last 10 years and it's their girl friend's name and maybe the date they got engaged or they first kissed. At the end of the day, the laptops hold your code most likely, hold passwords to important infrastructure, maybe to your cloud, maybe to the company email. The least amount you give people ability to accidentally cause damage over time, and you'll get the rewards from it.

Guy: Yeah. It is I think kind of a good emphasis to go back to that concept of a mindset, right? You sort of wish for the best but plan for the worst or try to not be quite as optimistic when you're just sort of granting permission and access.

How much does physical access in today's modern world when you see people discussing security policies whether as the creator of it or as those demanding it. How much does physical access come into play? What type of recommendations come in there?

Geva: This age where everything is in the cloud, we tend to ignore the physical aspect and we feel our office is safe.

The truth is, a lot of the hacks these days are happening from inside. I'm not talking about the malicious employees but stuff that happens by accident to employees.

I heard this great story about someone paying 100 bucks for a cleaning lady to drop a few USB devices on the floor and they have a little sticker that says "Salaries 2015". Guess what, it's almost a guarantee somebody's going to pick that stuff up and stick it into their laptop. And you know that USB is going to install a trojan on the computer, and by chance that's your sys admin and he's typing in all his passwords and there you go, it starts from there.

That's kind of a little social engineering trick and you think to yourself it's not scalable but actually all the bad guys work at this stuff at scale, so there's always this big smart guy at the top of the pyramid who comes up with all the schemes and it trickles down all the way to mules who actually do the physical work on the physical level at the office, at somebody else's office. Even stealing money from ATMs, unrelated to security. You think it's unscalable but it does scale. It does scale from the bad guy's side.

Guy: Yeah. I guess physical access doesn't necessary need to be somebody kind of breaking into your office at night or stealing your laptop. I guess the solution to that is pretty easy. Just to get everybody the new Apple laptops. They don't have any USBs, and you're sold.

Cool. I think there's probably a lot of those components we're not going to be able to go through every single one of the different policies. What do you in practice see when you go into companies that people are doing poorly? Like you know, odds are a listener to this podcast who has sort of involved in some form of B2B startup. What are the most common mistakes that you see in existing actual setup that the policy flushes out?

Geva: Some of the big mistakes are, I don't want to call them mistakes, they might have a vision of how their network is structured but then when you come and look, you see the database is sitting on the internet. It's not protected on a secondary tier, and guess what, it's listening on the default Postgres from my SQL port. That's a recipe for disaster. Kind of there's a difference between what they're thinking they have or what they want to have or their architecture in mind and what's in place, that's definitely one thing.

That's just a matter of attention. Most of the companies don't have a dedicated system administrator or don't have a dedicated network engineer or something like that. So that's kind of one category of things you would see. The other one, I call it separation of concerns. When you start and you have a few employees and most of them are your buddies, kind of everybody gets access to everything and a single person generally has permissions to build and ruin your company at the same time.

It's much better if you can divvy up the responsibilities so no single person has the ability to single handedly cause a lot of damage.

Let's give a concrete example. Your engineers are writing code. Of course you don't fear anybody, any of your employees writing malicious code but let's say they can write code with vulnerabilities. But on the other hand, do they have the ability to also put that code to production? If a single person has all that access from the beginning to the end, he has more potential to cause even accidental damage.

If you separate that concerns, there's one engineer and one release manager, and the release manager is another step that your code or that your product needs to go through, then you have one more gate, and that prevents a lot of those mistaken vulnerabilities and definitely it prevents a lot of malicious attacks but let's assume nobody has malicious employees.

Guy: It's an interesting kind of comment that indeed adds a layer of protection but it does so by basically adding a gate which are often times things that we try to avoid, as part of like continuous development.

I'm curious what your mindset is on this around the use of some tools like you might be able to tell some Slackbot or something to deploy, so it technically have the permission to do it because you go to Slack and you'll inform it or you sort of you provide a command. But one, it's kind of well logged and documented and second is, you kind of had to compromise another system to get that done.

Geva: Those are definitely the way to go because you spend one time, you put in effort, you're very conscious, you make sure it has all the controls whatever you need, and it does only one activity, and it does it well. It gets your code to the right place on the system and doesn't deviate left or right and has a very little chances of screwing something up. That's definitely the way to go.

From my perspective that ties to one more thing that you see more often than not. That's kind of the separation between your different environments. You have the production environment, developing environment, the staging environment.

You really don't want to make the production keys, readily available to everybody. If you're not thinking about it from day one, then what you see more often than not, then the production keys are checked in to the same GitHub repository as all the other keys, and any employee including the 100th employee that comes to the company is going to download that repo is going to have all those keys there, which is not the way to go. I recommend to everybody

Put the production keys somewhere else, and more importantly, tie it to an automated system that would pull them from wherever they're sitting and get them on the production server without anybody having to touch them on the day to day.

Guy: I think all that are very sound advice. In general, I think the notion of thinking about the developer as an attack factor is something that's probably overlooked, right? They consider the risks that happen when an attacker attacks their systems but not when they attack their developers. There was an interesting play with XcodeGhost with the malware in iOS world where a malicious Xcode was distributed and developers of mobile apps were somehow compromised.

They basically used the XcodeGhost, but really they weren't the target. They were just the distribution vehicle. The real target was attackers adding and injecting malware to the applications that those developers submitted to the App Store and therefore, compromising that many more users that installed those on their phones. In this case, it's literally a distribution vehicle to many systems but to the kind of new many devices and users but in the case of a B2B startup or even like a B2C startup, you get compromised, the developer gets compromised, the software gets compromised.

That might be your path to many user's data, that might be your path to many users they're visiting. A lot of these components I kind of enjoy the conversation or the highlight of the practices in them. I think that might even be like a good finishing note is just to say that a security policy gets triggered from a big customer asking for it, right? Or sort of from an audit requirement, but it really is a good opportunity to just have all of these conversations and to think about them if you haven't done them. It's not all just like necessary evil and a piece of paperwork that you need to create.

Geva: No, definitely not. Whatever gets the conversation going inside the organization, whatever external driver, external push you need, just as long as you have it, that's what counts.

Guy: Cool. I guess before I let you go Geva, just one last question that I ask many of my guest is what's your one pet peeve or recommendation to a developer or a dev shop, an organization trying to up level their level of security. What's the kind of one thing that you would recommend that they do?

Geva: Let me give one concrete piece of advice that everybody can go out and do today or check if they're doing today. One, you take into consideration you have a company, you have your products, you're specializing in it, but there are all these great infrastructure companies that do have security specialists, network architects, database architects and they're masters of building all the stuff. They're definitely way ahead of your organization.

If there's one recommendation, actionable recommendation I can go give everybody is

Make sure all of your servers are behind a load balancer, a CDN or some reverse proxy that's managed by your cloud infrastructure.

If you go to your DNS table and you see an IP of one of your servers there, that's just not the way to go. Shut a few box, put it behind a load balancer, and let Amazon, Google, and Microsoft whoever your cloud is be the first layer of defense between your company and the internet.

Guy: Yeah, that sounds like really good advice. Introduce a little bit of the pros, kind of you and world to not be the low-hanging fruit attack target.

Geva: Yeah and in that sense, there's always the story about the two guys who were walking in the forest and they see a bear. And the bear starts running after them. Well, you don't need to be one that's running faster than the bear, you just need to be the one that's running faster than your friend. Just be a little bit more secure than your competitors. That can be a milestone for you that'll definitely put you in a better place.

Guy: Just be more secure than the next guy. This was a super enlightening conversation Geva. Thanks for coming over. If people have further questions for you, they want to connect with you, contact you, how can they reach you?

Geva: Best just to email me. My email is geva@snowypeaksecurity.com. Or use the contact form on my website, www.snowypeaksecurity.com.

Guy: Cool. Thanks a lot. I hope you enjoyed this episode. Tune in for the next one.

Geva: Thank you.