Ep. #6, Developer War Games: Capture The Flag!

Guy Podjarny: Hello everyone and welcome back to The Secure Developer. Today we actually have a slightly unusual episode. We're not going to talk necessarily about best practices but rather about Capture the Flags, CTF, these types of security competitions.

And to do that I actually have my co-founder here from Snyk, Danny Grander. Danny, thanks for coming to the show.

Danny Grander: Thank you and hello.

Guy: Before we dig into CTFs, Danny, can you maybe say a little bit of, about your history, about how you got into this whole world of security?

Danny: Nowadays, in Snyk, I'm responsible for the security side of things, mainly dealing with vulnerability, database and collection of data, and in my past I've been doing development for about 10 years, later I switched to research where I've mostly been doing reverse-engineering and hunting for vulnerabilities in different systems, but mostly embedded ones.

Guy: So we started this chapter about CTFs because you've participated in a bunch of them, but before we kind of go deep in, can I ask you, sort of, to say a few words about, what is CTF, you know, what is this "Capture the Flag"?

Danny: Yeah, so,

CTF is essentially a hacking competition between teams or individuals. They compete against one another and are measured by their skill in different areas.

All are related to security, so this can be cryptography, stenography, finding vulnerabilities, reverse engineering, web challenges, and, like--

Guy: So this is all like different aspects of, I guess, information security, so still, mostly digital but all sorts of digital security?

Danny: Yeah. I haven't seen CTFs that do physical security things. But I could guess that there are some.

Guy: Yeah, that they exist. I guess, you know, if they're going to do lock-picking sessions at DEF CON, then, you know, there's no reason you wouldn't include those in the CTF competition as well.

Danny: Yeah, actually, now that you mention that, in the last DEF CON conference, in the car-hacking village there was a car-hacking CTF. I haven't participated in that one, but I know there was one.

Guy: It's funny, the "car-hacking village", you know, like every conference should have a car-hacking village just for kicks. So CTF is this sort of hacker competition and people come in, I guess, you know, what's the typical format? Is it like time-boxed? You know, how does it work?

Danny: Yeah, so basically, there are two types of capture the flag competitions. One is called jeopardy-style, where there are usually dozen of stand-alone challenges that each has a score, usually related to the difficulty of the challenge.

And the team that wins is the team that solves the biggest number of challenges by the time the competition ends, which usually, it depends, but it can run for a day or for two.

The other type of CTF is the attack-defense one, where, unlike the jeopardy-style, each team has to protect their systems, their services but also attack others. It's a much more dynamic and, in a way, represents the reality better in that sense.

And the skill set is also a little bit different that is needed for that CTF. You both need to find vulnerabilities and attack the other teams with these vulnerabilities and with the exploits you create from finding these vulnerabilities. But also make sure that you are defended against those.

Guy: How does, I guess what I mean is, split the two apart, just sort of dig a bit into them. So like, for the jeopardy-style CTF, what's an example of a question, you know, what's an example of a challenge?

Danny: The one I personally participated in several times was CCC, the Chaos Communication Congress capture the flag. It's a conference and, during the conference, there is a 48-hour capture the flag competition.

So there are about five different categories for challenges. So these include crypto, forensics, web, binary reverse engineering, protocol analysis. So challenges in any of these are pretty much different, right. Classic, I would say, reverse engineering challenge would include binary that, well, you have to reverse engineer, find a vulnerability and possibly also get an IP address and port where that service that you're looking at is listening on.

So then you actually need to develop an exploit and attack that system, getting the flag, which is most often is a string or a hash, something like that, where you submit the solution to the competition website and get the points.

So crypto challenge could be breaking some, well, crypto algorithm that, again, might be in a form of binary that you are given, or just some ciphertext. And you need to figure out what's the, what are you looking at and what's, how to tackle that.

Guy: Cool, so I guess that's the reason for the name, is capture the flag, right.

In all these cases you're trying to somehow find a secret, some digital treasure or flag that's either hidden behind a crypto algorithm or hidden behind a server.

Danny: Right, and the attack-defense type of capture the flag competition, it's similar to the outdoors game, the Capture the Flag game, where every team has a flag on their system, and each team should attack the others and capture the flag by compromising others' systems.

And like, in the jeopardy style where you sort of solve each challenge at a time, and the flag is somewhere there, either encrypted inside the data, if it's a crypto challenge, or a file or some stream that is, you can get to only after you attack and exploit the system.

Guy: So for these different types of challenges, do you need to be like a super security expert that controls all these different attributes to tackle them? Like, how many people, I guess, on a typical CTF team?

Danny: Yeah, so usually each person on a team has some kind of specialty. There are some people that do all the challenges but yeah, typically, like in our team we have about 20 members, but not in every capture the flag competition, all of us actually compete.

So just an example. For the last DEF CON capture the flag, it was an attack-defense style competition. We were nine people on-site and three others were helping from back home.

Guy: Oh cool, so actually not everybody had to necessarily be there in person to participate in the CTF.

Danny: And that's for the attack-defense. Usually the attack-defense is limited by the number of teams and the number of members that can take part in the competition. And in this kind of capture the flag, there are qualifying rounds, because, in the end, in the finals, there is only a limited number of teams that can take part and have their system being protected.

So unlike the jeopardy-style, so usually it's open to unlimited number of players. They don't have to be on-site, on the conference or whatever's taking place. They can be, yeah, anyone who chooses to sign up and join and play the game.

Guy: Yeah, cool, so I guess the jeopardy-style ones sound pretty straight-forward, right? It's basically a quiz that happens to be sort of a challenge, you know, you go in, you do it. Yeah, and it's kind of cool, I guess, in today's digital world, you can just set up sandbox environments with as many of those as you want, or you know, some environment where people can go and just sort of get that flag.

There's no limit at how many people can attempt a riddle at the same time. But the attack-defense, you go against one another. So if there's a thousand teams, it becomes pretty impractical to work.

Danny: Right, and something to add to your previous question is that, in the attack-defense one, beside the technical skill of finding the vulnerability and writing an exploit, there is also the operational part of timing the exploits and trying to figure out, like, game theory things, like figure out what the other team might be doing and there's a whole set of things that opens up for you because you can steal other exploits.

You can just wait for your service to be attacked, sniff the traffic, you could just try to, I don't know, send random traffic to the other team.

One good example of these kinds of things that I've seen in the last DEF CON CTF where a team would back-door their binary, their service, and other teams, including mine, would assume it's a fixed binary, one that is protected against vulnerability, which was taken as is.

Some things would actually fix the vulnerability but those would also include a back-door that would allow that team that created the binary just to easily exploit it any time, and only by them, because they know about existence of a back door.

So this is the kinds of things that can happen in attack-defense, and it's, it's an open world there. Yeah, in the jeopardy-style one is just stand-alone challenges and they're really more suitable for a person to take on and try to solve and usually, really, can be limited to a person that has a specific skill, that solves the challenge.

Guy: Yeah, so I guess in the attack-defense environments it sounds like the, some things are visible, I guess in any one of these cases you need to set some context. So you can see that, I guess, another company, another team has patched their binaries, so you can see it and you could choose to use it.

What's to keep you from just you know, taking your systems offline? Right, like, say hey, I gained a good number of points, I don't want anybody to hack my server because I'll just unplug.

Danny: Yeah, so it's a good question. And the answer is that for the competition, the organizers, they constantly check for availability of your services.

So if your services are down, if you just went and edited some firewall rules that prevent any communication to your services, you're basically losing points and you're penalized for that. Same goes for, suppose you patched your service and it's now protected against the vulnerability, but its performance went down, also you get penalized for that.

So again, there's a lot of realtime decision-making you need to do about what's good for you, what you can live with, what you should fix, unlike, again, in the jeopardy-style, that's like more simpler and defined.

Guy: I guess you can trust that in a hacking competition everybody's trying to hack the rules and have done so many times in the past, so it's probably pretty bulletproof to sort of find those types of shortcuts and these factor into account.

So in the last DEF CON CTF there was bit of an unusual CTF added into it, right, like it was this DARPA CTF. What was interesting about that one?

Danny: So this year, before the actual CTF there was another capture the flag game played by computers, by machines. This was one that, DARPA, it's called the Cyber Grand Challenge. DARPAs attempt to improve and advance the automated vulnerability discovery and protection field.

Guy: Or at least that's what they say.

Danny: Exactly, so there were, actually, during the last year there were different qualification rounds between different systems, and the final event, in the finals, there were eight different machines, different systems, that were competing one against the other.

And the winner would join the main CTF event.

So there were 15 teams on the CTF, one of them were a completely automated computer system competing against us 14 human teams.

And it actually did pretty well, surprisingly, or not.

It was really interesting to see how a computer could find vulnerabilities, create an exploit for them, and protect against them.

So obviously some of the things, if the machine was doing really well, for example, protection is something that is, it comes with a cost, but it's in a way easier for a machine to do. And the cost is usually performance.

For the vulnerability sides, usually they came when it's better, but again, the machine would try different inputs for discovering vulnerability in a software. But it of course could do it really quickly compared to a human.

So they were, like, seeing all these human teams competing against a machine, and machine is not being the best, it was really nice and interesting.

Guy: It's crazy, it sounds super cool from a technology perspective, seeing the level of technology and seeing this work and the fact that an automated attacker-defender, sort of, security team can actually stand up to a reasonable, and place in a competition amongst already really good security people and hackers.

It's also scary, it's entirely scary when you think about that type of technology just floating around the internet and, you know, people and cybercrime, or, you know, people that are well-funded having access to these types of automated machines.

You know, maybe it means we need some machines on our side as well, on the protecting side, for us to have a bit of a, you know, a shot at defending ourselves.

Danny: This is a really good point, and actually, it's one of the special things about the Cyber Grand Challenge is that they created a special environment for the challenges and the competition.

So in the end, it's a Linux system, but they created a different binary format, different from an ELF, something that has some reduced functionality, a reduced number of system calls, and that it supports, and that again was done for both simplification and gaming determinism in all the, again, attack and defense sites.

But also to prevent the easy use or abuse of these achievements. So the different system in, like, the real world.

Guy: Yeah, interesting. I guess that comes back again to that sort of breakers' mindset, you know, which is probably, whoever conceived of this idea, was already trying to find flaws in it as they were conceiving it.

So thought about, how can they reduce the risk. So I think, I mean, CTF is cool, and I think there's a lot of good stuff to read about it. And we talked about conferences, I think DEF CON is like the most popular, CCC being another good one.

I guess, are those the only places where CTF competitions run? I mean,

If I want to participate in a CTF do I need to find one of these conferences and go there?

Danny: Right, so almost in every security conference nowadays there is a CTF. So the big one in Europe is the CCC conference, the Chaos Communication Congress conference. It has the CCC CTF, it's a jeopardy style.

The big one in the U.S. would be DEF CON, it's an attack-defense one. But there are dozens of other conferences, and most of them have CTFs. Also, companies nowadays run their own CTF games.

So Google was, this year, actually, was having a CTF game where we, our team passed and won first place. Thanks. And they, and it's interesting. They had, all their challenges were, it was a jeopardy-style, and the challenges were around Google Services.

So it's also, it's interesting to see, and for us, first of all, to learn about the different kind of services and things in Google's product, but for them I'm sure it was useful to see how a bunch of teams get to play and attack systems that are based on Google's infrastructure or languages or services.

So there's quite a lot of CTFs going around, and for somebody who wants to give it a try, I would suggest, just signing up for a jeopardy-style CTF. These are open for everybody, you don't have to be in the conference.

So the next capture the flag competition, just can play it. And another thing is to go back and look at previous competitions. Usually, definitely for the major CTFs, there are quite a lot of write-ups. And all the challenges are open, so the information is there, you just could try and take one challenge and try to solve it and, of course, see the different solutions and learn from that. So yeah, there's quite a lot of opportunities there.

Guy: Yeah, that makes sense, I guess the jeopardy-style ones, you know, the bar is actually substantially lower than I originally thought. It sounds like the jeopardy styles have many teams, you don't need to, you know, pay or even to travel or do anything like that to join them, and they're available.

And you can even, kind of, practice a little bit ahead of time with some of these older ones. So yeah, sounds, sounds very useful. Also, it's an interesting play around the companies doing the CTFs. I guess, on one hand, it's a recruiting one, but yeah--

Danny: Yeah, there is as well, obviously, a recruiting angle there. And also, one thing I want to add is that, the different challenges in the jeopardy-style CTFs, they have different points. Well, the difficulty is different. So you could also pick on the simpler ones, small ones, and see how it goes, and then progress from there. So that's also, like, a good place to start.

Guy: Yeah, and I guess the one thing to note that happened sometime earlier this year, was this, that Facebook open-sourced their CTF platform. And I think they were using it in universities and the likes, probably at the end of the day, for a similar purpose.

So, recruiting awareness, you know, making people aware of the services. But the fact that they open-sourced it implies that maybe at a certain size of either an event or a company, that you can just choose to do it even to raise education and awareness amongst your employees or amongst a certain community, and just choose a bunch of challenges and you can probably tailor the level of complexity of the challenges to the audience that you have.

So not everything has to, you know, match the top security conference tiers. That same format could happen for things that are more specific. You know, just attuned to, maybe more of a developer audience or more a, you know, an audience that's maybe a little newer to security.

Danny: Right, good example is also in the CCC CTF, they focus mainly on reverse engineering and exploitation which other CTFs can be more focused on web.

Guy: So, Danny, thanks for the review of CTFs. I think they're a really fun thing. I've not yet participated in one myself, and I think I should make the time, you know, and sort of go and join one of those.

I guess maybe before we take off, you know, can you share maybe one example of, you know, like the most interesting or sort of fun moment or learning that you've had from your CTFs. Do you have an example like that in mind?

Danny: I have. So one fun story I remember from two years back in a CCC CTF. There was a web challenge which required exploiting across scripting vulnerabilities. So there was, sort of, a blog platform where what we were supposed to do is leave a comment on the blog post, and the admin to later visit and click on a link and, well, that's where we were supposed to steal the session and get the admin credential and all that.

But because we were trying to solve this challenge on the second day while all the other teams did that in the first day, during the first day, the automation around clicking on the link, on the organizer's part, wasn't working. So we would understand the vulnerability, we would create the special comment and all that, but nobody clicked.

So we didn't realize that there is, that that's the problem on the organizer's side. So after a lot of different attempts, of finding different vulnerabilities, eventually we found a vulnerability in the whole system and we, well, popped a shell on the box that is running the challenge.

It wasn't the intention, it wasn't, and it actually didn't help us too much because well, we ended on a box. We got to the root and we started looking for what's going on, which is much harder when you have all the different components that you need need to figure out what was the challenge and design for.

So eventually we contacted the organizers and they fixed the problem, but the funny part is that, one year later, they actually liked the challenge, the vulnerability, and both the vulnerability and our exploit, that they created another challenge that was exactly like that and, and it was fun. Because for us, that one was really easy to solve because we'd already been there, so yeah.

Guy: Even an organizer of a CTF competition is not bullet-proof to have any vulnerabilities that they did not plan in the system itself. Cool, well, this has been super fun. Thanks, Danny, for coming and joining us, and I guess, good luck in whatever future CTFs you get, you know, keep on winning them.

Danny: Thank you, thank you for having me.