Ep. #2, Making Security A Requirement

Guy Podjarny: Today I'm proud to have here Gergely Nemeth from RisingStack to talk about how they work in security, both in the world of Trace. Before I kind of give too much detail about you, Gergely, do you want to introduce yourself?

Gergely Nemeth: Hi, everyone. Thanks for the intro. My name is Gergely. I work at the company called RisingStack where we originally worked on customer projects. We were a professional services company mostly working with Node.js.

We introduced Node.js to quite a few big and small companies as well. In the past half a year we started to build one of our products called Trace, which we will mention later. But that's me in a nutshell.

Guy: Before we dig into Trace, which is super interesting by itself, maybe kind of pause a little bit. You've done a lot of work, still in the customer side and in online education with various-sized customers, working on Node.js projects.

When you talk about security, how did that manifest in those engagements? Was it often an explicit requirement or a high emphasis? Did you get pulled in to respond to a security problem? How did that manifest in your work?

Gergely: Sadly, it usually wasn't a thing that was required by our customers. It was mostly a thing that we paid a lot of attention to.

Also, to answer your other question, maybe it was only once when we were pulled into some security-related issue that we had to fix as soon as possible. But it wasn't really the generic kind of contract that we had, so most of our users weren't really into security.

It wasn't really a huge priority for them. Maybe they thought that it's something that comes naturally, but they didn't say that this is something we have to address.

Guy: Did you find you had to educate them about it, to basically tell them, "Listen, this is going to take some additional amount of work," or, "A part of delivering this functionality has to include these security controls"?

Gergely: Definitely.

We had to tell them a lot of times that it's better to fix that before actually you heard about it on Twitter, for example.

So it's really something we had to tell them a lot, and I think we successfully managed to do so.

After we started to work with clients, a lot of security issues were fixed in those applications, which I'm really happy about.

Guy: After you did some work with some customer, you educate them, frankly, they bring you in because of your expertise, right? So hopefully they're in an open mindset to take your advice. And then you went on and you built some security components and you explained those to them. You tell them what is it that you built.

Did you find that was well responded to? Were people happy after the fact? Or was it just pretty much disregarded as a mandatory kind of debt, but not really something they were excited to have?

Gergely: Mostly they were very happy about it. They didn't even know that they have to address these things, so they took it as a educational approach where they can get better. So it was a really good experience for us.

Guy: I often feel that everybody wants to be secure. It's a good thing, right? People want to do it. It's just about balancing it out with the rest of the activities that you have, as well as just sort of sheer lack of awareness, right? What is it that you need to do to be secure?

Gergely: I think it's something even in our customer work. What we did a lot is, not just actively paying attention to security when developing new things, but also be proactively seeking or adding existing vulnerabilities that these applications had. So they really appreciated when we found issues that they didn't think of.

Guy: Actually, can I take you even a step further back? How did your security knowledge start coming into being yourself and the team? How did you build that security understanding?

Gergely: It was something I was really curious about for a long time. It was, I think, three years ago when I started to read a lot about it, started to experiment with it. I just really fell in love with security and I've enjoyed doing it ever since.

Guy: Which reading sources did you find most useful when you started building up your understanding of it?

Gergely: Mostly I started to do it by reading articles on the website of the Open Web Application Security Project.

Guy: The OWASP project.

Gergely: Exactly. They have a lot of really good readings for both beginners and advanced readers as well. I think that's something that everyone can start learning from. Also I read a couple of books, I follow security-related persons on Twitter. I think that's it. And after that, a lot of research, whitepapers.

Guy: I guess for those that don't know OWASP, the Open Web Application Security Project is this pretty long-standing initiative, I think it's maybe about 15 years old, and it has a lot of amazing information.

The website is kind of messy. It's not necessarily the easiest to just navigate, but it has a lot of great output. I guess the OWASP Top 10 was one of those I've sort of seen two or three incarnations of it. Right now I think it's on its third? I'm not sure if it actually updated the second.

Gergely: It's quite outdated now. I think the latest one is from 2013 so actually, if I'm correct, they release it ever three years so one should come out pretty soon.

Guy: Yeah, so hopefully it does. I remember the conversations. I was working on web apps back there, sort of AppShield, AppScan, firewall, first ones. And a lot of the conversation with the OWASP Top 10 was that it was nowhere near sufficient, but that's okay. Let's just start with 10 basic things and have those handled.

People generally understand those and do those well, and then we'll move on to the next 10 sophisticated things. And that didn't quite happen. Really kind of those 10 core things just multiplied despite increased awareness to a decent amount thanks to that OWASP project.

Gergely: Even those 10 really basic things are not addressed in most applications, so I think really a lot of companies should start with addressing at least those Top 10 security issues, and after that we can move ahead.

Guy: It can advance. On that note, back maybe to your work on custom work. Which types of problems did you encounter most often? Or did you choose to focus on?

Gergely: Most often what we see is people still have a lot of cross-site scripting vulnerability attacks in their applications. Actually,

there are statistics that around 50% of the web applications out there have cross-site scripting attacks, whenever it is.

So actually it's quite central in what we sell, most of the customer work that we have done, and that weren't greenfield projects, had some kind of cross-site scripting problems.

And the other thing we saw, if we were talking about Node.js, were DDoS attacks, mostly denial of service attacks using RegEx, so that's something that we saw as well. These two were the two main issues that we saw.

Guy: Cross-site scripting: you're probably familiar with it. If not, that's a different problem, but fundamentally it implies an attacker can trick a user into running a malicious script within the context of a certain domain.

And then that violates the same origin policy, it violates the protect the browsers give us to protect session cookies, and other components from the evil attacker, right? From the evil player. But once you have one flaw, one of the problems with many of these security vulnerabilities and cross-site scripting specifically, is that

once you have a mistake or a flaw in one page on the site, you've compromised the entire domain.

And if an attacker can get a victim to click that link, they can expose session cookies and that private information, break that model.

You mentioned regular expression denial of service. That one is one that I find especially interesting, just because it's a little bit less well known. Cross-site scripting, SQL injection, even the general concept of denial of service, of taking your web server down, is relatively known.

At least high level, many people know those words and the very simple meaning of what they mean. But regular expressions for denial of service, those are not always well known. You want to explain a little bit what those are?

Gergely: Regular expression denial of service attacks in Node.js can happen because the V8 engine that interprets your code represents these regular expressions in trees, and the traversal of these trees can reach really extreme situations in some cases.

As Node.js is single threaded, it can happen that if the Node.js process itself takes a lot of time matching what you have, then user requests like HTTPS can not be served in that time.

So this is why it's an attack that cause service downtimes, because actually you don't need a lot of resource to do it. You just have to have a crafted input for an API, for example, and if this request takes a couple of seconds to fulfill, then in the meantime other requests cannot be served.

Guy: It's interesting how, in the world of Node.js, the single, not single-threaded, but rather the event-based scaling really is the core of its strength. It's what makes it so scalable and so lightweight, and the fact that you don't need, even in a massive production system, you really only need a small double digits in a sort of a big system number of threads.

That makes it a little of a single point of failure, if you get those threads churning for a long time on the process of anything, of any sort of algorithm, then you can tackle those. In the end, ReDoS are just regular expressions. You don't think of them as algorithms, but they're algorithms, and because of that, as you say, they can take this non-linear amount of time compared to the size.

My co-founder at Snyk, Danny, he did this capture the flag competition in the Chaos Computer Club. Capture the flag competitions are really interesting, might even do an entire episode on those. But this CTF had one specific Node.js challenge that involved a timing attack against Node.js, which was really interesting.

It was, again, using the fact that it's a single thread and the fact that there are unsafe comparisons when you compare two strings. The natural comparison in most platforms and specifically in Node.js, you do "equal equal" and it would compare them, character by character.

Say, if the first characters differ, it would fail faster. And it seems minuscule, it seems tiny, tiny. But when you run them enough times, you can actually deduce those. And then you can effectively guess a secret one character at a time and just sort of grow those.

Again, it was easier to do in Node because timing attacks, any platform is susceptible to them in one form or another. But in the case of Node, you could actually do them out of ban. You can send a request in one spot and check how long it took in another spot, because it was a single thread and you can kind of rely on that.

Gergely: Performance matters.

Guy: It's interesting. ReDoS or regular expression denial of service are interesting ones. This was super interesting.

Let's maybe kind of move off the customer work a little bit. You mentioned that you've sort of switched to building Trace. You want to tell us a little bit about Trace and maybe how you handle security there?

Gergely: Sure. Trace is a Node.js and microservices solution.

What makes it special is that it was built with microservices in mind in the first place. So it really treats them as first-class citizens.

What makes Microsoft's infrastructures a lot harder and more complex is the additional complexity interused by the network itself. In more applications you really don't have to think about that, because there is little or no communication across boundaries, application boundaries.

While in microservices, it can happen that you have tons of services communicating with each other, and the network matters a lot. Trace really helps debugging and monitoring these distributive systems. For now, Trace supports Node.js only. We are working on adding additional languages, but for now it's Node.js.

Guy: Micro services are definitely an amazing evolution, but you can't just think that you can monitor them the exact same way that you monitor servers, and have 100 times the number of logical servers without having a substantially higher amount of effort to monitor them. If you really want to flatten or control that, then you probably want to rethink a little bit how to you monitor those services.

I guess, when you do a monitoring system, clearly there's a lot of sensitive systems and sensitive data that might be flowing your way. How do you think of security in Trace? Are there some specific highlights you can share?

Gergely: First of all, we have to make a difference between our side of the things and the customer's side. On the customer's side we have our agent, the Node.js agent, which is basically just a simple require from actually starting to use it. And we have our own hosting service, so Trace is actually a software service product that you can start using, and we have to secure both sides.

On the customer's side, it's really important to make Trace as opt-in as possible. So by default, Trace won't send a lot of information about the traffic you have. For example, we don't send issue headers by default. We don't query strings by default.

If you're using signatures or just access keys, then these things won't be collected, and the same applies for customer IDs if you have them. It's really important to know what kind of user information you can handle and what kind of user information you shouldn't.

Guy: This is where privacy overlaps with security as well, right?

Gergely: Exactly.

Guy: In part you might expose secrets, and in part you might just expose private data, which for that specific individual whose data just leaked, really not that much different than a security flaw.

Gergely: Exactly. That's on the collector side. On our infrastructure side, we try to do basically everything to protect our users' data, which means the following: We are running in Google Cloud on Kubernetes, and all of our servers are on a private cloud, so our servers are not exposed to the public network.

Inside the public network we still use request signing, so even if it's in a private network, we still sign every request with private keys. We try to share only between the instances that have to communicate with each other.

Guy: This is using client-side certificates in HTTPS? How do enforce the signing?

Gergely: For now we are using Joyent's signature library. It's very similar to the SSL one. All our databases are in the VPC as well, and what we are doing is that the single entry point to this VPC is through a Nginx proxy, and only the application itself and our collector APIs are exposed. So that's only the two interfaces that anyone from the public network can communicate with.

Guy: The same challenges you're trying to address with Trace, in terms of microservices communication, also manifests in terms of security flaws, right? It's the same notion that says you now have to care about the security of the communication between the different systems, not just monitor their performance.

Gergely: Absolutely.

The funny thing is that Trace is monitored by Trace, so we're proud of that.

Guy: It's always nice to do some dogfooding. That's always fun. So, building a product of your own, you're building those components and you're doing security. Do you find yourself struggling sometimes with deciding whether to add a security component versus wanting to just get something out?

Gergely: Not really. In these cases we always pick security. We really don't want to play with the users data, so it's really important to us that we keep the information private and don't share it with anyone.

Guy: Fundamentally, if you ask yourself what might ruin your company tomorrow, generally not investing in a security component fits the bill.

Gergely: Exactly.

Guy: If we just sort of step out a little bit, this is super interesting around the activity you've had with the customers. You've also done a decent amount in educating others around security and the Node ecosystems specifically.

I really like one of the pieces that you wrote at RisingStack about a Node.js security checklist. I liked it because it was just sort of a nice list of "These are the things you need to do." To be prescribed and not overthink it. What made you write that in the first place? Are there some highlights of that that you would want to repeat here or that you think can be shared?

Gergely: The trigger point for me for this article was one of our customers that, back in the time that we did professional services, that given customer, we had a lot of security related problems, and actually this article started as a checklist for them.

What I did is I added some extra things that I thought were really necessary for any developers that wanted to put a Node.js service. Or not just Node.js, because some items in this list are quite generic, so it can be applied for other languages as well. Just a standard original list made available on our blog so anyone who's interesting can learn a lot about how to secure your project services.

Guy: I think it's a really good list. It's always the problem, that security is a bottom-list bid. I guess that's true for many aspects, for quality, for performance. Many of these things you can put a ton of effort into them, but oftentimes it's just hard to see how you make some improvement.

How do you take one thing and improve that? I find the checklist just to be a good way to do that.

Here's a list of items, are you doing all of them? No? Maybe add one of them to your next sprint.

Maybe add a couple of others to the sprint after that. Just sort of make it practical, embedded into your process, one step at a time.

Gergely: Also what I think is quite important here is that this list can be applied for a lot of applications. Maybe, at the same time, for a lot of applications, it doesn't make sense. You should always think about your use case.

Do I even have to care about this aspect? Because it can happen that it's simply not worth the effort. Because our protecting something for $10, it doesn't make any sense to put hundreds of engineering hours into protecting something that's worth $10 only. So, as always, you have to think about trade-offs here.

Guy: It's all about risk reduction, so the question is a little bit what happens if the risk materializes? I used a bit of an extreme example before, talking about something that would ruin your company tomorrow. But you have to balance this out. You can't just sort of live in fear. You need to protect what's worth protecting.

I guess of those, you deal a lot, you've got the checklist. I know you've given a bunch of talks about security controls and things you could do. Do you have a favorite tool or technique that you find you often tell people about, that might be a little surprising?

Gergely: I can mention a couple. One of the concepts that I really like is called attack trees. This is a methodology which helps you to describe the security of the system. Meaning that you take an attackers goal, for example, if you are thinking about a safe, then the goal is open it up.

And in this case, this will be the root node of an attack tree and you create sub goals, and the sub goals will be the sub trees of this tree, which basically describes what an attacker has to do to achieve its goal.

So in this case, for example, we're talking about a safe. There can be a couple of paths how an attacker can achieve this, so it's a really good technique and it's also it's not new. It's around 15 years old or something, so you can really find a lot of material on it if you search for it. I highly recommend to check it out.

Guy: Attack trees are sort of like thinking like an attacker. Figuring out how you, if you were in the bad guy's shoes, how would you approach this? How would you tackle it? And then try to figure out how you would know that you know that, or at least you theorize how to block those paths.

Gergely: If you are thinking about the attack trees, you really should think about, as you said, how an attacker could think. Or also, you have to think about who they are.

An attack tree won't be the same for a student who just hacks in his free time or someone who is sponsored by some rich guys.

Guy: Or you're in the government.

Gergely: I didn't want to say that.

Guy: When you talk about attack trees, clearly an attack tree is a logical technique, right? You can use paper and pencil to draw them, but still, within that, are there some specific tools or methodologies that you use? Or do you straight up just write them down?

Gergely: Mostly what we do is that there is an extended list at the Open Web securities website which has a couple hundred items. So if you want to do a complete penetration testing, that's something that you can check.

Also, there are a couple of automated tools that help you a lot if you're speaking about Node.js and you want to check, for example, if you have modules with known issues, then there are already a couple of tools that you can used for example, the lost security project or Snyk, which are both really amazing tool.

Guy: We've got the checklist that sort of has a whole bunch of other tools in there, right? People should just check it out. I think if you just look for, like if you just go to the RisingStack blog then you can easily find it. Or you can even look up Node.js security checklist then.

Gergely: Yeah, or just Node.js security. That would be the first example.

Guy: It would pop up pretty quickly. Just sort of shows how interested people are in actually finding that.

Maybe before we part, I'll ask you one more thing, which is, if you had to choose, you're talking to a development job that is looking to just improve one thing about their security poster, what's your recommendation? What's the one thing you would tell them to make sure to do first.

Gergely: It's a tricky question. I know my answer will be kind of a cliche, but actually an attacker has to be right only once. And you have to be right all the time. So I think there is not one thing that you can do to secure your application.

It's really about thinking how an attacker would think, what's really valuable for your business, and try to defend against those attacks and try to protect that information, those things that are the most important for you.

Guy: I think that makes perfect sense. It's actually something that we do very naturally when it comes to product features, when it comes to quality decisions, right? We don't just sort of spray and pray. We try to be a little smarter about it, to talk about which features would impact more users, which aspects of quality are most likely to affect bugs.

It's just about being able to step back and do the same for security. And you don't have to be some top-tier, red team penetration testing. Clearly you can be that and you would probably do it better.

So there's definitely opportunity to bring in those types of teams to think about it, but I find if you just sort of stop, if you're just a technical person, you think you know your system, you just stop and you try to think how would you attack this, you don't have to have deep security expertise to get some good conclusions.

Gergely: No, you don't have to. But also, generally speaking, it's not a good practice to do penetration testing when products are deployed into production. It's really something that should be part of the development flow. If we are talking about agile or whatever, it doesn't really matter, but it's still should be something that all developers care and think about it on a daily basis.

There's a really great article on VoteWorks about security and how it should be part of the agile workflow, and I fully agree with that. If you are talking about Scrum, for example, then most stories should have a security acceptance criteria as well. Not just from a user's point of view but what aspects you have to take into account from a security point of view.

Guy: Makes sense. Again, kind of bringing it into your own processes. Well, this has been great. Thanks a lot for coming, Gergley, and good luck with Trace. Keep educating the world on security.

Gergely: Thank you. Thanks for having me. Bye.