Ep. #42, Zarf with Wayne Starr of Defense Unicorns

Marc Campbell: Hey, welcome back to The Kubelist Podcast. Today Benjie and I are joined by Wayne Starr, a Unicorn Engineer at Defense Unicorns to talk about Zarf, their air-gapped Kubernetes solution. Welcome, Wayne.

Wayne Starr: Thanks for having me.

Marc: To get started, you know, I'd love to hear-- Your job title is Unicorn Engineer. You've worked for a company called Defense Unicorns. Tell us a little bit about that, the job title and what the company does.

Wayne: Yeah, so Defense Unicorns was started by some former Air Force folks that I had worked with actually in the past when I was in the Air Force, and it was really a way to try to address problems that we saw when we were in the Air Force.

There was this thing that started back in 2016 that was kind of called like the software factory movement that started with a software factory called Kessel Run that basically kicked off a lot of modern and agile software development methodologies for the Air Force. That progressed into other organizations. I worked with a few of them.

We kind of identified that there's problems around a lot of the tooling that exists out in space to kind of answer the needs of the Department of Defense, and a lot of us really believed strongly in open source and trying to give back as much as we could. And so we wanted to sort of start a company that was able to do that for the community.

The reason why it's called Defense Unicorns is really to set it apart from other defense contractors, be cause we don't see ourselves necessarily with all of them, and we want to make sure that we're distinct from them, and it's a good sort of talking point, because while there are movements within some of them to try to improve themselves and to engage with open source more, we want to try to actually start that and bring that to fruition.

Marc: So you guys are building a lot of open source with the Department of Defense in mind?

Wayne: Yes, our customer base is entirely Department of Defense, but we are providing open source stuff out there, and plenty of people are using it all around the world.

Marc: And I'm guessing there's a commercial-

Benjie De Groot: Wait, hold on. We're not going to skip over the Kessel Run thing, Marc. Nice try. So there are "Star Wars" nerds at the Air Force? I just have to represent certain people.

Wayne: Yeah, the reason why it got its name Kessel Run is because at the time what was happening was there was basically a serious delay for the Air Operations Center to get a software update. 2006 was when the software update process started. By 2016, they hadn't delivered a single line of code to production.

And so the users in that case actually contracted with Pivotal Labs with the help of Defense Innovation Unit, and then we were able to deliver a piece of it, which was the tanker planner tool, which basically is how do you map KC-130s and things to the sorties and how you plan that out.

We were able to deliver that app in just a few months, and because we were kind of a ragtag team of developers, we called it Kessel Run because we were able to get through the Kessel Run faster than anyone had thought possible.

Benjie: So you you did it in 12 parsecs, is what you're saying?

Wayne: Yes, exactly.

Benjie: Which, controversially, is a distance, not a time.

Wayne: Yep, exactly.

Benjie: We don't have to talk... Marc is really not going to be happy if I dive into this too deep, but just for those people that care, that is a great reference. It makes me feel even better about my tax dollars going to that.

But it's interesting that you're telling us about the modernization, essentially, of software inside Department of Defense, and I'm guessing it's probably not just DoD but it's also across the entire government sector, at least in the United States, is what I'm assuming Defense Unicorns works with.

So your first touch in this was really just being a part of like trying to modernize software practices, and you're an open source believer, and getting that adopted inside the government has been one of your prerogatives, and so that's kind of the impetus of all of this. Is that kind of a decent summation?

Wayne: Yep, that is. That is, yep.

Benjie: All right, super cool. So you were at the Air Force, and you were part of this, and then you joined Defense Unicorns when? Tell us a little bit about how you actually ended up at Defense Unicorns.

Wayne: Yeah, so I joined July 1, 2022, which was about a year and a half or so after the company actually got founded. So it was founded by three folks, two of them that I had worked with in the past actually as part of Platform One, which has done a little bit of stuff within the Kubernetes space. There was actually a talk at this past KubeCon about Big Bang.

But basically that sort of thing was to try to take some of the lessons learned from Kessel Run and pull it into how can we build a Kubernetes platform for everybody. And so they had founded Zarf, and I had known them from all that process, and so I decided that it's something that I believed in as well, and my time was kind of up in the Air Force.

I was kind of fed up with the bureaucracy of being a captain at that point, and wanted to actually get back into software development myself and write code.

Marc: So outside of the shared background, is Defense Unicorns related to Platform One, or it's just the same group of people were working on the same space, then you split off and became Defense Unicorns?

Wayne: Yeah, so for the most part it is just a lot of the same people. We still help Platform One here and there and contribute to some of like the Helm charts that Big Bang is based upon and things like that, but we're a kind of a distinct entity for the most part.

Benjie: Okay, wait. Backing up a second here, you're mainly here to talk about Zarf. We're really interested to talk about Zarf the project. I don't think we even introduced that all the way.

So tell us quickly what Zarf is. Just give us the elevator pitch on that. And then also I do think it's pretty interesting to talk a little bit about Platform One and what Big Bang is in general for those people not familiar with how the DoD works and procurement works inside the government.

Wayne: Yeah, so Zarf, starting with that, it's basically what we call a Swiss Army Knife for air gap and semi-disconnected environments. So what that means is it basically is a single tool that you can bring in and you can bring a couple packages with you and you'll have everything you need to stand up an entire environment.

So in a base case, all you need is a Linux system, the Zarf CLI, a Zarf init package, and then a Zarf package for whatever apps you want to deploy on top. And then you just start up your Linux system, you run Zarf init, you run Zarf package deploy, and then you have everything up and running in the air gap.

And then Zarf also vendors a lot of tools like crane, K9, kubectl, so that you have everything in a single binary that you can pull over to that system and debug whatever might be going wrong within that environment.

So it provides a very kind of self-contained and a very easy way to deploy things into the air gap with minimal effort on the deploy side.

Benjie: So does Zarf bootstrap Docker and Kubernetes for me as well?

Wayne: It can, yeah. So it's not required. Zarf can deploy to existing Kubernetes clusters. It can, you know, push to existing registries, it can push to existing Git servers, but the base Zarf init package, what it does is it will first start a k3s server on the Linux system, then it will do an injection process for a registry.

So it takes just the Docker distribution registry image, it splits that up into a bunch of config maps, it pushes all those to the cluster. There's a Rust binary that recombines those into an image and serves it as an initial registry, and then it becomes a real registry in the cluster.

And then after that it can deploy other applications, and the default init package has a Git server, Gitty, but we can work with GitLab and other things as well that basically can allow you to do Flux deployments or Argo CD deployments as well.

Marc: What were you carving up into the config map in writing it, and then why are you doing it that way?

Wayne: Yeah, so what we're doing is basically taking the distribution library image, we're basically pulling that out of an OCI directory within the archive, the tarball that makes up the package. We're splitting that up into about 13 config maps because you have about a megabyte size of a config map that can be, and that's about an 8 or 9 meg image or so.

And then those get added into the cluster and then get reassembled into an image that can then be pulled by the cluster to start the registry image. Reason we do that is because, at least in the initial init package case, we know what the cluster is, it's k3s, we could inject it that way, but to make it distro agnostic, where we don't necessarily know what distro you're on, what control plane you might be using or whatever, we can actually do it with config maps and that will work everywhere.

Marc: It's pretty clever.

Wayne: Yeah, that was our co-founder that came up with that, Jeff McCoy. So kudos to him on that.

Benjie: The real McCoy. Sorry.

Wayne: Yes, exactly.

Benjie: Okay, well, that is... I need a whiteboard to make sure I understand exactly all of that, but that sounds super duper clever.

So the bootstrap mechanism itself is actually leveraging k3s, which is super cool, big fan of k3s over here. And so you said I only need a Linux kernel.

Wayne: Yeah.

Benjie: So obviously there's probably some requirements to what that kernel needs to be, but then you can actually bootstrap Docker or whatever build kit. I don't know what you need to do there, but you can actually bootstrap all that.

But the k3s is just a binary. And k3s, does that include Docker, or do you need that or?

Wayne: Yeah, so k3s is pretty self-contained. You need cgroups v1 at least on your kernel, and our init package requires systemd, because that's how... Like we've got a k3s unit file that gets in inserted there. We could make it work on other inits things, but we did systemd just because it's most common.

But yeah, that's pretty much all you need. K3s in and of itself is pretty self-contained. It actually also can symlink in kubectl, and some other commands in and of itself. And so Zarf supports that out of the init package as well.

Benjie: Side note here. I don't know if you know, I'm on a mission to spread "Kube cuttle" is the right way to say that, so I just want to tell you. That I ask everybody towards the end of the episodes what their answer is, but you already have said it twice now.

So Wayne, big fan of yours already. It is "Kube cuttle." That is the correct way to say that. And continue to spread the gospel on that one.

So, okay. How and when did you get involved in Zarf? Sounds like you obviously have a really good understanding of it. When did you get started? Was that 2022 when you joined Defense Unicorns, or were you working on it prior, or how did you get involved?

Wayne: Yeah, so when I started with Defense Unicorns I was 50/50 Zarf and working with one of our customers, and so I was kind of bouncing between that, and then I really like product and I enjoy software engineering, so I kind of slowly migrated my way more and more onto Zarf projects.

At that time I was just an initial, like kind of developer individual contributor, and then became maintainer of it in February of this year, because Jeff McCoy moved on to other projects.

Benjie: Congratulations. That's a pretty cool project to be a maintainer on.

Wayne: Yeah, it's fun.

Benjie: And so your day-to-day now is 100% Zarf for the most part?

Wayne: Yeah, it's 100% Zarf. So it's managing the team that Defense Unicorns has assigned to Zarf, and kind of the tickets that come up through GitHub. We do everything in GitHub and use the issues PRs projects there, as well as any open source folks that come in to either a k8 Slack or PRs they may open, and trying to kind of manage between all of those different things.

Benjie: Typically we get to this a little later, but you said you're using GitHub. Where do we chat with you? Discord, Kubernetes Slack? Where do we chat with you?

Wayne: The Kubernetes Slack. So there's a Zarf channel there for users to come in, and there's also a Zarf dev channel if folks want to contribute or have PRs.

Benjie: Awesome.

Marc: And Zarf is not a CNCF project right now, but it is fully open source?

Wayne: It is not, but it is Apache 2.0. Yeah. And we are looking to donate it to Open SSF.

Marc: Okay.

Wayne: We chose that because Zarf can do more than just Kubernetes. It's built around Kubernetes and Helm, but it does have generic files and actions, and so you can do more than just k8s with it.

Benjie: What is the main thing that Zarf really solves? What is the main thing that it's solving at the core?

Wayne: I would say the main thing is that it brings everything that you need together. And so one of the things that I was kind of passionate about and one of the things that actually affects the Department of Defense is that a lot of the folks that are doing these jobs are not necessarily trained for it.

So if you think about where Zarf came from, it came out of some research at the Naval Postgraduate School for the Navy.

And so their particular problem was, say you have a submarine, it's underneath the water, you can't even get radio signals to it, and you have sailors on board that might be trained in Linux, but they probably don't know anything about Kubernetes. Maybe if they do, it was all learned by themselves. They weren't taught by the Navy how to do that. And so how do you have a sort of a self-contained solution that can deploy the applications and clusters that you need and maintain them, but is also easy enough to use and simple enough that a sailor can quickly, you know, get that system up and running if something goes wrong?

'Cause say, you know, an SSD dies or an array dies while you're underwater. You know, you need to be able to replace those disks rebootstrap the cluster potentially, rebootstrap your application, and do all that quickly with ideally as minimal mental workload as possible so that you can get things back up and running.

Benjie: So Wayne, again, we're getting way ahead of ourselves, but this is great. I love to ask folks where they've seen the craziest installs of Kubernetes and some of the CNCF tech that we have.

So it sounds like if you're, I mean, whatever it's appropriate and allowed to talk about, I'd love to hear. It sounds like a submarine is one of them. Do you have any other cool ones or anything else like kind of cool anecdotes around like real...? I would say a submarine is water gapped. They should be calling that water gapped.

Wayne: It is, yeah.

Benjie: It's not air gapped, it's water gapped. I've never thought of that one before.

Wayne: Yeah, a submarine is probably one of our cooler ones. You know, we've done others Zarf deployments, though, where we've had to take entire like dev environments and pull packages together, and basically replicate almost entire corporate domains almost, which that was a fun one, to try to make build tooling not have to be modified in the air gapped environment, because they were actually doing development in the air gap instead of doing like, just bringing applications up.

So they needed to bring over like their Python and node packages, things like that. And they didn't want to edit their build scripts, so we had to replicate a lot of their corporate domain stuff.

Benjie: I'm just going to say this, because I don't work at Replicated, but Marc works at Replicated. Obviously Zarf and a bunch of the stuff that the folks over at Replicated are doing, there's obviously some really interesting overlaps here.

I might even ask Marc some questions about usage of Zarf in the future at Replicated, but you're using the word "replicate," so I just feel like there's an elephant in the room here. And so I, as the representative of fairness, will bring up that Replicated has a lot of stuff here.

And I might, Marc, can I ask you some questions about your usage of Zarf? Are you guys going to use Zarf anytime soon?

Marc: Well, let's take that at a different time. Let's talk about Zarf. We have Wayne here.

Benjie: All right, all right. I got him on the hot seat for you. But yeah, please continue.

Wayne: So yeah, there is some overlap there, but there's no reason that you couldn't deploy your Kubernetes cluster with kURL and use COTS under the hood and various things.

A lot of what Zarf is focused on is not necessarily vendors providing and building Zarf packages. It's more about building, like you as a third-party trying to build a third party Zarf package around something that you might not fully control.

And so there's a lot of things that we've built into like what we call the prepare process right now, although we're going to be changing that probably to Zarf package development, but there's commands within Zarf, like Zarf prepare find-images, where it'll actually template out your Helm chart and your manifests, and it'll find all of the images that you need to include in your Zarf package.

There's also some things that help you deal with Flux and Git repositories within an air gap on that side. We're working on more of the DevX, so being able to test these packages and figure them for the air gap quickly. And so a lot of it is around like how do you take all of this software where it might not be designed for the air gap or built in a ways that support it and bring it over to the air gap.

Benjie: I have a quick question about that. And this just popped in my brain, and maybe you can't speak to this, but say I am a tech on a submarine and I'm water gapped. How do I get to docs? Like literally there's no internet.

Wayne: So yeah, you don't. What you have to do is you have to bring the docs. And one of the examples we actually have in our project, I don't know if they use this in any production sense, any of our customers, but we've shown it as an example because I think it's useful. It's called Kiwix, and it basically is a way to bring Wikipedia offline with .zim files.

But basically you take these Wiki archives, and you can pull them in, and with the Zarf package you can bring Kiwix and those ZIM files over and have, you know, DevOps dot Stack Exchange in the air gap. That's K-I-W-I-X.

Benjie: K-I-W-I-X.

Wayne: Yeah.

Benjie: Interesting. Next time I'm on a submarine, I'm bringing Wikipedia with me for sure.

Can you get me onto a submarine, actually? Wait a second. That's the real question.

Wayne: I don't work directly with them, but I'm sure there's people in our company that could.

Benjie: Okay, well, we'll follow up on that one. I want to go on a submarine.

Marc: So I want to go back and talk about like the use case here, like right? So air gap, disconnected environments, hard to deploy environments totally makes sense. But I think, Wayne, like in the way you're explaining it, you know, like, yeah, as Benjie mentioned, you know, at Replicated we work pretty closely in this space too, and I think there's some nuances where like it's not just, "Oh, deploy Kubernetes, and now deploy applications."

There's first-party apps, third-party apps, but what you were describing was even, it's like deploying a third-party app where you don't control it, you're not the software vendor who's packaging an application for somebody else to consume.

It might be like open source. Is it generally open source, or like what are the common use cases that the types of applications that you actually package with Zarf? Like who's the packager? Is it the end customer? Is like the one who's going to be running it ultimately is the packager?

Wayne: Yeah, so it really depends on the particular use case. So there's kind of two main ways that Zarf gets used. One where the packager and deployer are usually kind of the same person or same set of persons, and that's usually like more of your kind of online style environments, or where they might be connected but not entirely connected.

And so you'll be deploying things in Zarf into an egress-limited environment or something like that. And then you have environments where your creator and your deployer are separate people, and maybe don't even necessarily talk to each other a whole lot.

And usually the creator, it's up to them to create the package and then be able to test it in a very standardized way, and then deliver that to the deployer. And usually in both of these cases, people are bringing over usually open source applications. Sometimes it's bespoke applications.

There's nothing really stopping you from doing that with Zarf. But one of the bigger uses for it is stuff like Big Bang, where you have a lot of stuff to orchestrate together, it's all open source software, and you have to find all of the images for all of these things, and they increase versions relatively often, so you have to go through that process continuously.

And so one of the things we have within Zarf is kind of a concept of extensions. So with Big Bang, we have a Big Bang extension that will actually go look at the Big Bang repos, and then it will, based upon the Big Bang version or the tag that they have in Git, it will do everything for you.

It'll pull the right version of Flux, it'll pull all of the right images, it'll pull all of the manifests you need and everything to have Big Bang up and running and ready for the air gap.

Benjie: Let me back up a second here. I actually have some familiarity with this, and I know it's an important platform, Platform One. Can you tell us a little bit about what Big Bang is?

Just, we haven't had someone related to the defense industry, so I think it would be great to just explain to us what Big Bang is and even what Platform One, I believe the people that kind of support that is.

Wayne: Yeah, so Big Bang really is just an umbrella Helm chart, but it has a bunch of configurations that meet certain security controls. So there's a process within the Department of Defense and government at large that's called the Authority to Operate, or the risk management framework that basically gets you an authority to operate.

And so RMF will allow you to sort of answer security controls to get an application approved to run in a secure environment. And Big Bang provides kind of a pre-configured layer of configuration that will answer some of those security controls.

And it came out of some of the original lessons learned from Kessel Run, where we kind of wanted an open and standard sort of platform that people could run applications on and have some of these security considerations thought of already for them that was a little bit more standard without everybody doing their own thing.

Because before Big Bang came along, it was up to everybody to create their own security layers, their own answers to these controls, their own platform effectively, to run everything on top of. But at its core, it's just an umbrella Helm chart that has a whole bunch of other Helm charts inside of it.

Marc: Yeah, to your point though, it's more than just like that packaging. It's like the entire delivery pipeline around like security, around like, you know, Zarf has co-sign built into it, right? Like so you're signing the artifacts and making sure they're verifying with the other end.

Wayne: Yeah.

Marc: Okay. That's cool.

Wayne: And there's more than just Big Bang that Platform One does. There's also Iron Bank and Party Bus. So Iron Bank is a complement to Big Bang in that it provides hardened images.

So Big Bang, the Helm chart uses Iron Bank images to run the software. And then Party Bus is their, I guess, software as a service version of Big Bang and Iron Bank, where you can use it and just log into it.

Marc: We need to stop and kind of talk about naming things for a little bit, like, you know.

Wayne: Yeah.

Marc: Platform One, you know, I think that makes sense, but Party Bus... Like what's Zarf? Where did that name come from?

Wayne: So Zarf actually came from, it's a name for a coffee cup. So it's used in the Navy as like a place on the ship to hold your cup of coffee. The original name for Zarf, if it's an Arabic word though, and it's just because it's really a very ornamental, traditionally a coffee cup.

Benjie: I like your mascot too. What do we call that thing?

Wayne: An axolotl.

Benjie: Of course. Can you spell that, please?

Wayne: Yeah, A-X-O-L-O-T-O-L, I think. Something like that.

Benjie: That is a made up name. Or is that something I should know what that is?

Wayne: No, it's native to Mexico actually.

Benjie: There's an animal with three ears?

Wayne: Yep.

Benjie: It's a real animal?

Marc: Well, we'll put a link to the Wikipedia page for this animal in the show notes here.

Wayne: Yeah, it's A-X-O-L-O-T-L.

Marc: It does. See, the little mascot looks the same.

Benjie: Okay, wow. I really thought you guys were super creative with your mascot, but it's actually... Okay, is there a reason for that being your mascot?

Wayne: It's just an underwater creature that's relatively unique, and it's cool.

Benjie: All right, I don't know who's naming stuff there, but it's good. You guys are good. Keep it up, keep it up.

Wayne: Yeah.

Benjie: Replicated is pretty obvious, and Shipyard is pretty obvious, but you guys are going a little bit of a different direction, but I enjoy yours more.

Okay, so where in the Zarf project are you? You're not CNCF, so it's not like incubating or graduated or anything like that, but give me like the equivalent. Like how far along is this project now?

Wayne: Yeah, we're gearing up to go open SSF, and we're probably going to enter Sandbox there at first. So we're still pretty 1.0, and we're still experimenting a lot with kind of the base schema and the way that things interact with Zarf.

And so every couple versions will have some breaking changes here and there as we kind of learn more from our community and improve things, because we really want to get the UX right, and we want to make sure that things are usable.

I mean, one of the examples of something that came up that seemed innocuous when it was first implemented was the required state for components within the Zarf package. So each package is made up of components, but we had required was an optional key, which people wound up asking the question, "Well, where are all my components? They're not deploying," and stuff like that.

So there's things that were still changing to make things like that. Like okay, it shouldn't be required. It should be an optional key, and then we should change these things around as we get feedback. Because we really want to make that process as smooth as possible for that deploy persona.

Benjie: Right, and also, especially if you're on a submarine and you can't ask someone the question, this UX is actually extremely impactful.

Or maybe a better way to say that is that in the air gap world, UX really, really makes a huge difference when you have limited ability to contact and to to get on the Discord or Slack and say, "Hey, what's going on here?"

Wayne: Right.

Benjie: So okay, so UX is something you guys are super focused on. How many committers, how many maintainers are there? How many folks are helping right now?

Wayne: So right now the team is four folks that are kind of maintaining it day to day. I'd have to look at the latest commit status and GitHub as to how many we have.

Benjie: Not to put you on the spot, but do you have a timeline for when you think you're going to get to like 1.0? Or maybe what are the goals to get to 1.0?

Wayne: So the main goals for 1.0 for us are kind of reliability and stability of our package definitions and our API, and our ability to kind of handle those over time.

We're kind of already in the process of handling lots of breaking changes with, you know, backwards compatibility shims and kind of like making sure that we can continually move forward and become more mature.

So our main goal is that we feel we're in a place where we don't have to make a ton of breaking changes going forward, and that it's going to be a more usable thing for these bigger use cases.

Benjie: But you already have a lot of folks using this in air gap production, whatever you call that.

Wayne: Yeah, the main difference is that a lot of those people are either our customers, so we are interacting with them there, or they're very active in the open source community, and so we talk to them a lot.

If somebody were to not talk to us and try to adopt Zarf, there might be... Like they might be confused as to why things break from version to version, at least in terms of some of those keys.

Benjie: Right. Until you get to 1.0.

Marc: How often are you, or Defense Unicorns, not, you know, you personally necessarily, but like how often are you all involved in the actual installation of the packages or supporting it or versus just completely, "Nope, it's open source. We're hands off. You can use the product"?

Wayne: Yeah, so we do actually deploy to different DoD customers, and so about maybe half the company is focused kind of on what we call our delivery side. And so I'm on the product side working on our projects, but there's also a delivery side of the company that actually delivers the products into production.

So they use Zarf, they use Pepr and Lula and our other products as well, and then they actually deliver to our customers. And so we're listed in things like GSA. People can contract with us in the Department of Defense.

GSA, General Services Administration, they have a list of government contracts that the government can go out and purchase things off of, and so people can buy support subscriptions for Zarf in that way. And then usually that's us building a package or supporting a package for them.

Benjie: So Zarf is used internally as a packaging tool, and this is kind of the symbiosis between Zarf and Defense Unicorns?

Wayne: Correct.

Marc: And we've spent the whole time talking about these, I think you described it before as air gap and semi-disconnected environments both, so some sort of limited egress. Is that really where that's the use case? If you had a system that's totally connected to the internet, would you recommend Zarf for that use case or something else?

Wayne: It can be useful in some cases. It depends on your environment and kind of what you need. The use case in a fully online environment that Zarf would provide is a little bit of sort of segmentation away from those upstream mirrors.

So if Docker hub or Quay goes down and you're maybe doing a node roll or something and you lose image cache on a node, and now your pods aren't coming up because they can't pull those upstream images, Zarf is a way to package some of those images for your deployments and have them in a registry that's local to you. So that can either be Zarf's internal cluster registry, or maybe ECR or ACR, whatever cloud registry you want to use.

And then Zarf does have some nice things that sort of are built around it. One of the things for ECR specifically is that it doesn't support push-to-create, which basically means that if you do a Docker push to ECR, if the repo doesn't exist, it won't make it for you.

So in a normal sense, like it'll just fail, but with Zarf you can actually use that and Pepr in combination with one another, which is another product which we could get into later, but basically Pepr will go out, and Zarf will tell Pepr, "Hey, I'm about to deploy this Zarf package." The Pepr capability that you write can know that and can know about ECR and pre-create those images for you, and then say to Zarf, "Hey, I did my thing," and then Zarf can push the images to ECR.

Benjie: Interesting.Wait, hold on. You mentioned two things just now you. You mentioned Lula, I think, and Pepr.

Wayne: Yeah.

Benjie: Tell us about both of those.

Wayne: Yeah, Pepr is basically a really clean way to do effectively a mutating and validating web configuration. And so it's written in TypeScript, it has a fluent API for actually doing things. So it's like when a secret is created then do this, is kind of how you write those out.

We feel it's an easier way to kind of write these things for people who aren't super familiar maybe with those kind of concepts in Kubernetes, and we feel that combining with things like Zarf allows us to do these more advanced capabilities.

One of the things Zarf does as part of that init package in addition to injecting the registry is it has an agent inside of itself that will see any images and pods really coming into the cluster and will mutate the image references.

Pepr is kind of the next generation of that, where it can do what the agent does and do that simple case, but you can also write your own capabilities and do these more advanced use cases like the ECR hook or maybe some Istio configuration or key code configuration, or these other things where you need something else to do something in the cluster while it watches cluster resources.

Marc: That's an interesting problem. Like that makes sense, but I think it's probably worth talking about a little bit more.

So I have a Helm chart, or whatever the packaging format is, and I want to deploy it. Helm chart generally... Helm charts are customizable, right? Like most Helm charts, a good Helm chart will have overrides in the values to specify the registry and be able to inject imagePullSecrets.

But what you just described is that it doesn't really matter. Like I don't have to know that at deploy time, because Pepr will inject a mutating webhook controller, so when it sees something like a pod... Is it a pod that it does and then it like rewrites it? It injects that stuff into it for me?

Wayne: Yeah, yeah. So yeah, it could be either the Zarf agent as it exists today, which is just kind of a component of Zarf, and then we're going to be migrating that to Pepr longer term, but Pepr will do the same thing. But yes, basically what it sees a pod come into the cluster, it will mutate the image reference to the upstream reference and then it'll move that along.

The main benefit to that is that a lot of, especially in the Department of Defense, but there's kind of this like "dev low, deploy high" sort of methodology. And so you might want a package that works in a couple environments, maybe it's two, maybe it's three if you have different classification levels.

And so if you build a package with Zarf, because the mutating webhook is configurable as with part of the state, and on that init process, you can configure what registries and Git servers you use. You can basically have the same package deployed to all three environments, and it'll mutate the image references correctly based upon the domains of each given environment.

Marc: So that basically moves it into config, and so it allows you to take that same artifact and deploy it across, like promote it across those different environments? That's the benefit there.

Wayne: Yes. Yep.

Benjie: So wait, I think Pepr is super duper duper cool, and so I don't want to... I want to make sure I understand this. So I can have this Pepr agent basically, and it's P-E-P-R, correct?

Wayne: P-E-P-R. Correct, yep.

Benjie: And I can have this agent that basically can mutate anything in the Kubernetes API, anything at all? Or is there limitations there?

Wayne: Yeah, it's up to your configuration of it, but yeah, you effectively write your capabilities in TypeScript, and then you build it into a Pepr module that you deploy into your cluster, and that becomes a mutating and validating webhook.

Benjie: So I can do crazy stuff with that then, where I can be like, "Hey." Can I do like auto scaling with it? Like where it's like, "Hey, this pod keeps crashlooping." I mean, I could do anything I want with this?

Wayne: Yeah, basically. Yeah.

Benjie: So that's really cool. And it's TypeScript, so it's just like living in the cluster and I have these like little modules that are... So it's kind like Zapier for Kubernetes, if that makes any sense?

Wayne: Kind of, I guess. Yeah.

Benjie: Like in living in your own cluster and has full access? I mean, I'm a little worried about the RBAC.

Wayne: There is an RBAC mode where you can turn it on and it'll try to detect like what APIs you're using within your capabilities, and it will try to set that up for you.

That's a relatively new feature, so that might require a little bit of manual configuration to make it work perfectly, but there is that option as well.

Benjie: I feel like Pepr is like a really cool missing piece. It's dangerous.

Wayne: Yes, it can be. Can be.

Benjie: It's very dangerous, but I think that's really cool. I'm going to check out Pepr some more. Marc, did you know about...? I didn't know about Pepr. Did you know about Pepr?

Marc: Nope, that's actually super cool.

Benjie: Wait, Lula, Lula. Hold on. Lula. What's Lula?

Wayne: Yeah, so Lula is effectively an OSCAL generator, validator for OSCAL, which is the Open Security Control Assessment Language.

And so that basically is a way to define security controls. It's managed by NIST, and it's basically a way to manage your security controls as sort of declared files. And so you can ingest those and run them against policies and make sure that things are still valid within your cluster, you're still meeting controls.

And the hope, or at least our hope with it is that it gets adopted and you can potentially build entire authorization packages using it, and potentially could integrate a lot with the existing authorization programs that the government uses, like eMASS is one of them, as a name.

Marc: Does this watch stuff that's being deployed at deploy time using like in the operator pattern, or is it getting in and actually watching the runtime of the pods too?

Wayne: Right now it just kind of watches what's in the cluster. It still is relatively new for a project, so it may change over time, but it's just mostly collecting data about what's in the cluster, and then, it's not really like a running policy engine per se, like a normal policy engine.

Marc: Got it. There's no eBPF going on here.

Wayne: No.

Benjie: Marc, though, I love eBPFs. I'm like, I'm the Wasm, eBPF guy.

Marc: I know. I had to say it before you this time.

Benjie: Yeah, you just, Marc got me in the episode. He said eBPF before I did.

Okay, so Lula is another one. All right, is there any other really cool projects to check out? I mean, I'm looking at these right now.

They're obviously pretty young projects. Is there any other young projects or not young projects that we should be checking out at Defense Unicorns' GitHub?

Wayne: Yeah, the other one to mention would be Leapfrog AI, which is our sort of pulling open source AI technologies really into Zarf packages, and then being able to deploy them into the air gap to have these capabilities in the air gap as well.

Benjie: Leapfrog, you said?

Wayne: Leapfrog AI.

Marc: So that's actually delivering like the models and then like an inference engine into the environment.

Wayne: Right, right. Yep.

Marc: So that probably interestingly shows that like Zarf can handle some pretty large packages. There's no size limit on the packages in order to be able to make that work.

Wayne: Yeah, yeah. Some of our Zarf packages can get 20 gigs or larger. They can be very large.

Benjie: So wait, so Leapfrog is my way to deploy my LLM, the model, the inferencing engine, all that other stuff.

Wayne: Yeah. And the UI on top.

Marc: Onto the submarine.

Benjie: Onto a sub- Oh God. That's literally the plot of "Terminator 3," I think. Guys, let's not do that, please. Wait, so you guys are a big contributor. How big is Defense Unicorns? How many developers are you guys?

Wayne: It's a little over 100 folks now.

Benjie: I mean, I think maybe this is a topic for another conversation, but it's really interesting that you guys, like the backbone of this company is clearly open source.

Wayne: Yeah.

Benjie: Everything is is open source. So obviously that helps you from an audit government contract perspective, I would... I mean, I assume that that's an accurate...

Wayne: Yeah, the folks that understand it within the government definitely appreciate that we're open source. Not everybody does, but yeah, those that do definitely appreciate it. And we believe strongly that it's important to put it out there in the open because that makes it better for everybody, and it's just a much cleaner way to do things.

You know, one of the things that I personally kind of felt when I was going through the whole Kessel Run process was that like there's a lot of things that have issues in the large bureaucracy that is the Department of Defense, and by putting stuff like Zarf out there as open source software, it means that people don't have to go through the acquisitions arm of the Air Force, the Navy, to be able to use it. They can just pull it off GitHub, run it by their information assurance shop, and we have like, we're in Iron Bank and some of these places so they can get some of those approvals for it and then start to experiment and use it in those things, even if they're just like a single lieutenant or a tech sergeant or something.

They can go in, use Zarf, and it can answer their mission needs as long as they're capable of using it. And then, you know, the way that we make money is just with those larger programs that also want to use Zarf for their own needs. We contract with them, but we're kind of giving to both sides of the house by being open source.

Benjie: I love to see the open source world actually being used the right way, and I feel like this is kind of where it comes from, especially at the government. We all know there's a lot of waste at the government level, and this seems like a really, really good way to bring some transparency, and efficiency obviously, underneath it all.

I'm proud to hear that we're doing stuff, using technologies like this, let alone open source. You know, I've heard the horror stories of what the IRS runs on.

Wayne: Yeah.

Benjie: Anyway. Okay, that's super cool. Is there anything else going on at Defense Unicorns that you want to highlight, or anything else that you want to bring up?

Wayne: So yeah, one of the main things we're working on, kind of behind the scenes, behind all these projects is something that we call the Unicorn Delivery Service or UDS.

You'll find a lot of repos in our GitHub that are labeled with UDS, but that's kind of our way of building these packages and trying to put them out there as open source things that people can use as well to deploy Big Bang components like GitLab or Istio, or, you know, kind of underlying things.

That's our sort of service that we're going to be building off of, but we're also building these packages that belie that as open source packages underneath.

Benjie: Yeah, absolutely. So I'm interested. I don't know. I want to protect the government, so I won't contribute any code, but for others that aren't me, where would I go to contribute?

You mentioned that you're on the Kubernetes Slack, and I know you guys are active on GitHub. Is there anywhere else? Is there any community meetings? Anything like that?

Wayne: Yeah, so we're probably going to start those up in the new year with Zarf for community meetings. We haven't yet, but the Slack and GitHub are the best places to look, and just read through the contributing guide.

And anybody can contribute to this stuff. We have enough controls around the way that that works that keep our customers happy. So we've had contributions from Europe, from India, from all kinds of places, so like don't be shy to contribute.

Benjie: Awesome. Well, I've learned a whole lot. I love it when I get homework from these episodes, but I feel like you gave me a lot of homework.

Marc: And just learn about a new project. We came here to talk about Zarf, and like, you know, Pepr and Lula, and it's cool.

Benjie: Yeah, Pepr is like really sticking in my brain. Leapfrog I might contribute to, because I'm scared about the submarines, but other than that. Oh, I had a question. On your website, you say, "On another planet." Can you tell us something there, or is that just hyperbole for now?

Wayne: We do have space people that we work with. I'll say that.

Benjie: All right. Well, it's super cool what you're doing. Thank you so much for coming on, Wayne.

Wayne: Yeah, thanks for having me.