Ep. #14, Kyverno with Jim Bugwadia and Ritesh Patel of Nirmata

Marc Campbell: Hi, today I'm joined by a couple of maintainers of one of the newer CNCF Sandbox projects, Kyverno.

My guests today are Ritesh Patel and Jim Bugwadia from Nirmata.

Ritesh and Jim are founders of Nirmata and have created Kyverno.

Welcome, Ritesh. Welcome, Jim.

Ritesh Patel: Thanks, Marc. This is Ritesh Patel.

I'm co-founder at Nirmata, also lead products and marketing.

And nice to be on this podcast.

Marc: Great. And, Jim?

Jim Bugwadia: Hey, Marc.

This is Jim Bugwadia and thanks for having us.

Marc: Great, so before we dig into the Kyverno project I'd love to hear a little bit more about in Nirmata.

What does the company do?

Jim: Yes, Nirmata is a management plane for Kubernetes.

And, of course, as enterprises everywhere are trying to run Kubernetes clusters and workloads, our approach is to, kind of, unify both of those concerns and provide a complete management plane that not only can bring up or connect to existing clusters but also does workload management and integrated policy management.

Marc: Great, yeah, that makes sense given the little bit that I know so far about Kyverno.

So great. How long has Nirmata been around?

Jim: So we started Nirmata in 2013.

And that was, of course, pre-Kubernetes, so we were in the Docker ecosystem where we did quite a bit of consulting as early projects for migrating to containerized workloads then.

We also had a solution in the Docker ecosystem.

But, really, what was exciting to us was seeing the growth of Kubernetes from the beginning, how that space evolved.

Kubernetes, of course, as everyone probably remembers, started around 2015.

But, really, it was about 2018, 2019 when we saw the maturity of the product for enterprise use cases.

So our customers tend to be larger or midsize enterprises.

They are a little bit more traditional, slower moving in terms of migrating some of the core applications.

So, certainly, we were looking for those key inflection points and the maturity of the project.

And around 2018 is when we launched the Nirmata/Kubernetes platform.

Marc: Yeah, that tracks pretty well.

I think around, Kubernetes has been around for, what, about five years or so at the time right now?

Jim: Right.

Marc: And we're rebuilding everything into the Kubernetes ecosystem.

And there was definitely, like, some inflection point around 2018, 2019 where it just seemed like, okay, Kompose is great, like, all these other schedulers are great.

But Kubernetes won at this point

Jim: Right. Yeah, there was a lot of, I remember in the first couple of years it's like Swarm versus Mesos and Kubernetes, are all three going to be viable options?

How do you choose? Of course, all of that's gone now and we can, sort of, focus on building on Kubernetes, which is awesome.

And that clarifies things quite a bit for customers too.

Marc: Yeah. Twitter wrote Mesos, and Twitter now adopts Kubernetes and Mesosphere is now a Kubernetes company.

Yep, it's awesome. So we're going to dive into quite a bit here today about the transition, I think, and how you're actually implementing everything on Kubernetes.

Jim: Right.

Marc: So let's do that. Let's dig into Kyverno.

Can you help me understand, give us a quick overview of what the project does?

Jim: Sure, yeah. Let me give a quick overview and I'll let Ritesh, sort of, jump in with some context too.

So Kyverno is a policy management solution for Kubernetes, right?

And it started off as a module in Nirmata itself.

So as we looked at, so both Ritesh and I come from a operations and management background.

We've built centralized management planes in various domains like telephony, networking, so Mission Critical, 24/7 type of domains.

And what we learnt over the years is that, for operational concerns, policies are really essential to become like that glue, not just for enforcing and validating security, and best practices and things like that but also for automation and advanced, sort of, intelligent management, right? And policies, sort of, become that glue function between different concerns.

So what we, as we were approaching this space and looking at some of the challenges that development teams and operation teams face, one thing that became obvious is, of course, you want Kubernetes itself, containers, all of this drives agility.

But then what's also required is each one of these functions requires autonomy and wants to do self-service, wants to perform their jobs the best they can without having to create tickets, wait for somebody else to look at that, eliminating all of those manual procedures and functions.

So, to achieve that, policies become that handoff, really, between dev and ops.

And as we realized that we had built a policy engine within the Nirmata platform and as Kubernetes matured, and there were features like the whole CRD feature became a fairly robust and mature, I think, around one 116, 117. And then the whole also admission controllers, with webhooks both validating and mutating, that became available.

It just then at that time, it made complete sense to move what we had built within our management plane down into the control plane, make that an admission controller in Kubernetes.

And approach this problem space in a manner that was very native to Kubernetes itself, right?

So that was the early motivation and the beginnings of Kyverno.

And then it was just recently, like you mentioned, in November that we donated, we made Kyverno a CNCF project.

So for the first few months, about eight months, eight to 10 months, we had our own open-source project.

We saw some momentum on that.

And, as the community grew, it just made sense to, kind of, introduce it as a CNCF Sandbox project.

Marc: Yeah, and I think we're going to dive in a little bit later into the benefits that you've seen since donating it to the CNCF, because that's always super interesting.

So you mentioned a lot of different stuff there about how Kyverno works.

It is policies written as custom resources and then applied through a CRD that's running as an admission controller in Kubernetes. Is that an accurate description of it?

Jim: That's correct, right.

So there is an admission controller which will receive, so Kyverno registers for all API requests coming in but then has a cache built in where it loads all of the policies that you would install, either as part of your default deployment or, over time, you can add/remove policies.

So as each request comes in from the API server, we have a very strict deadline of, I think the default is something as low as three seconds.

Just as a comparison, I think by default with webhook, I think it allows up to 30 seconds, but we have been very aggressive in keeping that low, because we want the cache lookups and the policy execution to be extremely efficient and fast.

And then, once a policy rule is triggered, it can apply various types of logic.

So there's different types of rules, and we can dive into that, but there's the admission controller.

There's also a background scan component, right?

So that is just in case something's missed during admission controls or, as other checks need to be done in the background, those would be applied, just on a periodic basis, based on your resources and other configurations in the cluster.

And, finally, there's some rules which kind of operate in a hybrid mode.

The trigger happens at admission control but then the actual action is performed later.

And a good example of that is, one of the very powerful features in Kyverno is the ability to generate resources based on various triggers.

So let's say you set a label on a service and, based on that, you want to generate some custom resources, like perhaps a certificate.

So Kyverno will detect that the label got set during admission control but the actual action of generating that resource, we don't want to block the API request so that is deferred to after admission control and happens in the background.

Marc: That's cool. It reminds me a little bit of what operators do often today, right?

Like, I'll deploy a custom resource and that's going to go do some different things and create other objects.

So Kyverno does that in the admission control workflow?

Jim: That's correct.

So and then one of the things we have heard is a lot of folks who have either built or are using various operators, they end up consolidating and perhaps even replacing a lot of those using Kyverno policies.

So now if you were doing, using one operator, perhaps inject a sidecar, perhaps other operators to generate other resources, you can consolidate those, write some policy rules, which helps you automate all of those type of behaviors.

So it's a combination of doing some validation checks, of perhaps mutating resources, mutating data as needed and then even triggering other changes that happen.

Ritesh: Just wanted to jump in here, Marc.

And this operator analogy is awesome, because in the early days of Kubernetes you saw a lot of buzz around writing operators for every little use case, right?

And that was promoted, obviously, in the community by lot of large vendors.

And now with Kyverno, you really don't need to be a super hotshot programmer and learn Go or whatever language to build a simple operator, right?

An operator, which like Jim explained, that does some validation, that's inject some configuration.

All of that can be done very easily with a few lines of YAML.

And, in that sense, Kyverno pretty much democratizes that aspect of customizing your Kubernetes cluster to meet your requirements as a platform engineer or cluster operator.

So it's a very powerful use case that we're seeing emerge in the community.

Marc: Yeah, that's cool.

I mean, I can write Go code, and fight go.mod and set up a CI/CD pipeline and everything to do it.

And then deploy that and then manage it or write YAML.

And it sounds like if I just take Kyverno, then I write YAML and I get GitOps to do these transformations, these mutating operator style logic with just a few lines of YAML.

Ritesh: Correct. That's absolutely correct.

Marc: So Kyverno's not the only policy management application or project in the CNCF ecosystem.

The big one, obviously, is Open Policy Agent, they just recently graduated.

How do you compare Kyverno to OPA and where are the strengths and the nuances where each of those projects shine?

Jim: Good question, right?

And something that, of course, as you can imagine comes up often, as folks are looking at the projects, trying to decide what to use.

So OPA's a great tool for writing policies.

And, as you probably are aware, it uses a language called Rego in which your policies are written, right?

If you're using OPA the value is that you can use the same way of defining policies, and these policies now are externalized from where you're applying policies, and you can do that across different systems.

However, what we saw, before we sort of built Kyverno, we had several discussions with several of our customers at Nirmata, other folks in the community.

I think everybody could agree that OPA was solving an important problem, but there still just seemed to be something missing in that experience for Kubernetes administrators, which is our target audience, right, both from our business point of view and also from a Kyverno perspective.

So there, as we dug in a little bit deeper and tried to understand what some of the challenges were, the main challenge was context switching between the mental model that you have when you're working with Kubernetes is, of course, driven through the declarative configuration management approach.

And that, very simply, means as a Kubernetes administrator you're used to defining the desired state and then the controllers work hard internally to match the current state to the desired state. And that's what we do with every resource as we specify that, and we put that in our GitOps pipelines, et cetera, for any Kubernetes resource. So the question we started asking is why can't we do the same with policies?

Why does it have to be this complex external language, which is, sort of, a different paradigm, requires a different way of thinking of things, perhaps a different team to build these policies?

Where, as an admin, I just want to do certain things, I want to automate tasks, I want to validate configurations, maybe perhaps do, insert some best practices, things like that.

Why not take that same approach and make it extremely natural to anybody who understands what a deployment, or a pod or a StatefulSet looks like, should be able to whip up a Kyverno policy was the goal we, sort of, set.

So that's where we started evaluating.

And as we, kind of, dug in there and said what is the minimal syntax and, which was still allowed for these powerful operations but do this in a very natural manner to Kubernetes.

The other thing we decided up front is we wanted to focus on Kubernetes.

And that allowed us to take advantage of every pattern, every idiom, every kind of other nuance of how Kubernetes operates.

Like, for example, being able to recognize that pods are created by pod controllers and designing for that, being able to understand how finalizers work, and garbage collection works and being able to write rules where the policies just take advantage of, again, all of those things.

So those were some things that we started out with, which I think helps differentiate the whole Kyverno user experience.

And, really, that goal is to make it extremely natural and easy for Kubernetes administrators, as well as provide a very good developer experience, right?

So provide the right feedback.

So Kyverno, from the beginning, again, we have had a CLI, a command-line tool.

And a requirement has always been that the ability to use that tool in a CI/CD pipeline to do offline validations.

The other thing we, in one of our releases, I think about like six months ago or so, we switched completely to structural schema, which allows you to use things like kubectl explain, you can use Visual Studio code to get help, to get syntax checking, things like that.

So just, again, making it super easy, seamless, and using existing tools like kubectl, Kustomize, et cetera to write and build your policies versus, with OPA, you would need to go to Rego, use other tools.

Of course, there's plugins and things available to make that easier as well but it still requires learning a new language and a new approach.

Marc: Right, so Rego is kind of a general purpose, like, it can do everything, AuthN, AuthZ in my application, it just also happens to apply into things like pod security policy or admission controllers through Gatekeeper.

But Kyverno is, like, purpose built specifically to be as simple as possible, to give me everything I need in that context.

Jim: That's correct.

And then the other thing, so mutate as well as generate.

So Kyverno does three types of policy rules, right?

So validate, where you can check for configurations using patterns, using a Kustomize-style approach to declare bits of YAML that you want to enforce.

So you can either, as part of validate you can either block configurations at admission control or you can passively scan and report configuration errors.

And Kyverno creates a custom resource for a policy report as well.

In fact, the report that Kyverno uses is something drafted by the policy working group in the Kubernetes community.

So it's a broader purpose than just Kyverno but we have adopted that within the Kyverno project.

And then there's also mutate rules, which can allow you to change configurations on an existing resource and then finally generate, right?

So mutate has been added, I believe, by Gatekeeper also recently, there's some support for that.

But generate is a feature unique to Kyverno.

And, of course, a large part of that was this whole automating that handoff between dev and ops, that I mentioned upfront, as one of the goals for Kyverno.

Marc: You mentioned a couple of times there, and earlier, that you're using Kustomize.

And I'm assuming it's Kustomize with a K, that project that allows you to write parts of YAML that target and replace.

Can you explain a little bit more how you're using Kustomize in Kyverno to let me create these policies?

Jim: So we were, in Kyverno the policies were inspired by Kustomize tier, right?

'Cause as we are looking at some of the challenges with things like Helm templating and other ways of doing configuration.

And I think there was a paper published, I believe it was by Brian Grant, on the whole approach and the mindset behind Kustomize.

And they talked about some of the different ways of doing configuration management for Kubernetes.

So, certainly, that overlay style did inspire Kyverno policies and the validate rules, like I mentioned.

The other areas where Kyverno also uses things like strategic merge patch, which is also supported by Kustomize, right?

And that's what we wanted to allow.

And wherever possible, again, like when it comes to especially applying mutate patches Kyverno does, in fact, the implementation also reuses what Kustomize does for strategic merge patches.

There's other syntax above that we support.

But the implementation, the guts of it are the same.

And then, finally, Kyverno policies being resources itself, it's extremely straightforward to use Kustomize to customize them.

So you can, in your CI/CD pipeline, let's say you want to, for certain namespaces maybe or certain clusters you want to validate and report errors through policies.

But, for others, you might want to block and enforce.

Kustomize is perfect for doing that through GitOps.

And that's how we see Kyverno getting used across these pipelines and deployments as well.

Marc: Okay. You mentioned this white paper by Brian Grant and it is a great white paper.

Like, we'll make sure to link to it in the show notes here. It really, we use Kustomize a lot in my day job and what we're building. And helped me just realize, like, wow, configuration management's a hard, unsolved problem. There is, like

Jim: Right. (both laughing)

Marc: so many approaches to it. Like, Google hasn't figured this out yet.

Like, who are we thinking, we're going to figure this out quickly, so--

Jim: Exactly.

Yeah, and that's where Kyverno plays an interesting role.

And one of the things we debated and still, as we are seeing different usage patterns, there's community discussions ongoing on this as well, is what exactly should happen in the Git Repo, what should happen at admission controls.

And then, of course, there's operators and controllers running inside the cluster, which are also making changes and creating either resources or updating resources, right?

So a good example is if you have a fleet of clusters and if you want to even do further customizations to your policies, should that be, one of the things we're actively discussing is should we allow config maps, which are stored in Git and those get translated to further drive, like data-driven behaviors, or should those be customized actions?

And what are some of the trade-offs?

Now, of course, if I were doing this in an imperative manner, I could set labels and things like that.

But you can't really do label management well with GitOps always, right, 'cause you have to kind of maybe go and, if they're existing namespaces, things like that, then those labels need to be managed in a different manner.

So, yeah, there's still questions like that, and we're exploring those patterns and trying to come up with some set of best practices for how to use each of these components properly across the pipeline.

Marc: That makes sense.

I think GitOps is a great pattern because it allows us to, like, as developers, we know how to use Git, and how to make a commit and how to roll back a commit or revert it.

And it allows us to take all of that and all the power of Git and apply it to what's running in the cluster.

But we have this conversation frequently, does that mean that every YAML that we deploy has to be completely fleshed out and written to the Git Repo or is, like, a mutating or a generate policy still valid?

And I'm sure conversations like that are probably super fun and lots of opinions from folks in the ecosystem about that.

Jim: Yeah, absolutely, right?

And I think what we believe is important is that the state needs to be reproducible, right?

So as long as the operations are idempotent, and you can reapply them and you can, kind of, get back to the right state from a Git Repo, that's what's important.

So for example, to answer that specific question, let's say you're using something like flux-to, you're using that for GitOps, if the sequence of installations is, when that runs it installs Kyverno, it installs a set of policies and then it installs the right resources, then you are going to be able to reproduce that exact state 'cause the policies will execute and make changes or generate resources, things like that as required, right?

So it's all still being driven through Git and your policies are managed in the same repo.

Like we mentioned, you could still apply the necessary customizations, things like that.

So it actually works out quite well. And you get reproducible state, which is managed through version control.

Marc: Yeah.

Ritesh: And what we are also seeing, a pattern we are seeing is that there are certain resources which are probably not going to be in Git, right?

Whether it's like anything related to secrets or security certificates, things of that sort. So those have to be produced somehow for your application.

And there's several different ways to do that, because one way to do that is through Kyverno's generate option, that way you can generate resources, you can trigger certificate generation, if you're using cert-manager and things of that nature.

Marc: Yeah. I mean, that makes sense.

I can put a Helm chart into my repo and flux has a Helm operator, will deploy it.

But that Helm chart is not deployable Kubernetes manifest.

To your point, I can reproduce the exact state through an idempotent GitOps deployment method, which is really, like, the heart of what GitOps is, implementation aside.

And there's many different implementations of GitOps.

So as long as you have that end state being reproduced, that's really what we set off to do. The generate function seems super useful and cool. Was that in from the beginning, was that always planned to be in Kyverno?

Jim: That's right. That was one of the key reasons, again, when we had looked at OPA and some of the things that were possible.

And, again, like, from our customer sort of conversations, we always felt that policies drive that automation, drive that handoff between different functions, allowing each role to, kind of, still be autonomous and sort of independent.

So that's where we wanted to make sure that the ability to automate these fine-grain configurations is always possible.

Another great example of this is just even with multitenancy and trying to apply that, use namespace-based multitenancy.

So permissions, role bindings, roles, they're extremely powerful constructs in Kubernetes but extremely cumbersome to work with if you're doing this manually or even through, like, GitOps or something like that, right? It just doesn't fit into that model. What you really need is a way to do this dynamically and in a very fine-grain manner.

So with Kyverno you can precisely generate, even new roles based on what somebody is requesting of specifically for that entity, whether it's a user or some operator.

And you can then pare that down once that resource or that user removes what they're requesting.

A simple example is when you're creating a namespace, you can then create a role just to delete that one particular namespace and also to maybe get access to that single namespace for that user, for the owner of the namespace.

And then using other projects, like the HNC, the Hierarchical network Namespace Controller, you can then allow more flexibility within that namespace.

So it leads to all sorts of interesting patterns and it's amazing to see, even in the community, what folks are coming up with.

There was one user who shared what they have done.

And they're using a labeling on namespaces to drive things like compliance to PCI requirements.

Because now they can, based on the namespace labels, they can make sure that workloads are placed correctly, they're pinned to certain notes for compliance, and regulatory requirements and things like that.

So it almost becomes like a if this then that for Kubernetes, right?

You can kind of program certain rules and logic, and then allow that to all happen in an automated fashion as things are changing in your cluster.

Marc: That's a great analogy, if this then that, for Kubernetes.

To be able to build those pipelines and actually still have a GitOps pipeline is crazy powerful.

Jim: Right.

Marc: That's cool.

I want to dive in to a thing that's happening right now in the Kubernetes ecosystem.

Kubernetes pod security policies, they've been around for a while.

But I think a lot of people know, hopefully everybody knows, that they're using them, that they're deprecated and being removed from Kubernetes.

And there's a KEP, an enhancement proposal out to replace them.

If I have pod security policies could I just replace that functionality with Kyverno today?

Is there anything missing?

Jim: No, there isn't. It is possible to fully replace them today with Kyverno.

In fact, there's a library, if you go to the kyverno.io site, and click on the "policies" tab on the top bar.

You will see there's pod securities and there's just a single one-line Kustomize and kubectl command you can use to apply all necessary pod security policies.

So where the community, like you mentioned, PSPs are being deprecated.

There is a great document out there called "Pod Security Standards", which is where things are moving towards.

The KEP also that you mentioned builds on top of the "Pod Security Standards".

And the intent is whether it's in-cluster implementations, which will be very heavily focused on namespace-level privileges or applying these "Pod Security Standards" whether they're for privilege workloads, for baseline workloads with some security and then restricted workloads with highly, sort of, enforced security best practices.

So those are the three levels described in that document, in the "Pod Security Standards".

And then through what's being proposed, you will be able to apply default and appropriate standards to namespaces.

So with Kyverno you get a lot more flexibility, right?

And you can achieve everything you can with PSPs today.

And then also go further in terms of if you want to mutate things.

And also, for example, perhaps even do other updates, whether it's for adding in the security context variables or just changing them on the fly, of course, depending on how your containers are set up, if that's supported then you can also edit and update things on the fly.

But we do have a set of policies we have tested, and we will continue to curate and test and support that set of policies for pod security.

Marc: Okay, so yeah, if you're running PSPs today and they're deprecated, just switch.

I'm browsing through your website right now and looking at the policies that you have on the site.

And, yeah, it seems pretty complete.

Jim: Yeah, and we will, there is ongoing efforts as the KEP evolves and too also based on the "Pod Security Standards", to be able to come up with some test utilities, whether it's a set of YAMLs or perhaps even like some sort of conformance tool to make sure that any implementation, whether it's Kyverno, or OPA Gatekeeper or others in the future, and the in-cluster implementation are all compliant to the "Pod Security Standards".

Marc: Great. So let's dive into how you actually created the project.

You mentioned at the beginning that there was this policy engine that's part of Nirmata and you pulled it out and created Kyverno.

Was that like a rewrite or were you able to extract the code, what languages are in, can you help me understand that a little bit better?

Jim: Sure. Yeah, it was a rewrite.

So we, Nirmata uses a mix of Java, Golang and there's some JavaScript also.

So the initial engine was written in Java but there was a rewrite, of course, as we kind of did this for Go.

But it used a lot of the same concepts, and especially around this mutate and generate which we had built in into Nirmata from the beginning.

Marc: Got it, yeah. Seems like a lot of projects are being rewritten in Go or Rust now.

And just it becomes a little bit easier to work in the Kubernetes ecosystem in those languages, in those platforms.

Jim: Absolutely.

And just taking advantages of all of the controller run time and the other tooling available in client-go, that made complete sense.

Marc: When you extracted it out and you were building it as a standalone project, were there technical challenges that you ran into that made it difficult to make it run completely independent from the rest of the Nirmata platform?

Jim: Not particularly, right?

So Nirmata is a set of microservices, so there's fairly loose coupling between different modules, different applications.

So Nirmata as a platform, we do other things like cluster operations management, also workload management.

So extracting out the policy engine from that, of course, there was effort involved but because we had already refined some of those patterns, et cetera, it was more of a translation.

And then adapting it to fit in with the Kubernetes API machinery and things like that for admission controls.

So the challenges are more in that area.

And, of course, a lot of things have matured just in the last couple of years, also in that whole spectrum with writing controllers, writing operators.

Like, one of the things that we're doing with Kyverno right now is we're looking at what components, as Kyverno is being used in larger clusters, how do we scale it up?

How do we dynamically tune that for different workload sizes, things to that nature?

So we see more challenges in that area but the guts of the policy engine itself was a fairly straightforward translation.

Ritesh: And I think, just to add to that, in the last, I guess, six to nine months, right, Jim?

I mean, Kyverno has matured quite a bit, obviously, given all of the community usage and the feedback we have received.

It's important for Kyverno to be rock solid because it's in the admission control path, right?

So it needs to be very highly available, resilient and process requests at a very, as quickly as possible.

We set a very aggressive threshold for processing these requests. And all of that has evolved over time, working with the community.

And those are the kind of challenges, I should say we didn't really have, policy engine was part of Nirmata per se but, obviously, kind of the scale we are seeing as Kyverno gets more-widely used, it's kind of helped shape the project quite a bit.

Marc: That makes sense.

So when you extracted it and you made it open source, was the intent always to try to have it join the CNCF and become a Sandbox project and mature or was that a decision you made after it was an open-source project?

Jim: So our philosophy with any Nirmata component is if it runs in the cluster, if it's something that applications depend on, we open source those components, right?

So because we want to, sort of, minimize any lock-in or anything like that of which users would have as they're using our solutions.

So from that point of view, of course, making it open source was always one of the requirements.

In terms of CNCF, that was something, we had always thought we would do that at some point, we just didn't know what was the right timing.

And then, hindsight, I think we should have done it sooner rather than later 'cause there's just the benefits we get being in the community, being able to collaborate is great, right?

So, but the Sandbox process has also evolved, right?

So as we had looked at some of the initial Sandbox projects and the process to, kind of, get a Sandbox project approved that has changed quite a bit.

And, based on some of the evolution, it just made sense to apply, get the feedback.

Of course, one of the main questions was, what we also talked about, how does this compare with other products, what are the benefits to the community, things to that nature?

But it was significant.

I think there was, sort of, unanimous consensus that there is a need for a solution like this.

And that, of course, is what we're also seeing with the rapid adoption.

Marc: That's great. The Sandbox process, as you mentioned, has evolved and made it easier to submit and get into the Sandbox.

But, I mean, from what I've been seeing, and I think what I've been hearing too, the quality isn't really been going down or anything like that. It's just a lot of really great projects out there right now.

Jim: Absolutely.

Certainly, one of the benefits we're seeing immediately is also the collaboration now that's enabled, 'cause other CNCF projects are much more comfortable working with Kyverno than if it were a independent open-source project, right?

And, like I mentioned, we had some collaboration.

I mentioned flux-to previously and there is some work we're doing together with them.

There's also some ongoing work with cert-manager.

We're looking at ways Kyverno can solve certain things.

And, also, we have had discussions with the OpenESB folks, with the Litmus-Chaos folks.

So there's a lot of interesting things that can happen across these projects.

And, of course, we use other tools, like Prometheus, et cetera as part of our standard offering with Kyverno.

Marc: That's great.

And I know you just became a Sandbox project in November of 2020, so it's only been a few months.

And so this may be way too early to even think about but has the team started to think through what's going to be required to move to the next stage, which would be incubation?

Jim: We have. And we are certainly pursuing that path.

So Sandbox was made easier, incubation was made in a more stringent, that's where the bulk of the work needs to happen and then graduation is, sort of, just growing adoption and working through other aspects of the community management, et cetera.

So, for incubation, two main things we are looking at.

And we will start engaging on very, in fact, this quarter, is to go through some initial security reviews for Kyverno.

One of the things on that note I should also mention is, so Kyverno is sort of a closed system, right, it works within the cluster.

And we don't allow making external calls to any other system outside of the cluster.

The pattern we adopted is something, again, if there are controllers that need to make external calls they can make those but then they would populate config maps or other resources, which would drive Kyverno behaviors, right?

So we wanted to design Kyverno in that manner where, from a security point of view, because it's sitting in admission control path, you can't just haul out to some third-party system or do something else in there.

So security review, going back to the Sandbox to incubation, that's a big part of what needs to happen.

And then, of course, is growing the community in terms of committers, maintainers, adopters, there are milestones and certain criteria for that as well.

Marc: How do you manage internally at Nirmata having an open-source CNCF project, plus, you're a commercial company, and you should be making money and you have this platform, right?

So how do you think about features, where they're going to land?

Do they all go into the open-source project or how do you think about hiring engineers?

Do they split time between commercial and open source or do you have dedicated engineers working just on Kyverno?

Jim: We do have a dedicated team working on Kyverno.

So there's about four engineers working on Kyverno.

We will continue growing that as well.

And in terms of the product sort of strategy, I'll let Ritesh chime in and, kind of, give our thoughts and our view on that.

And, of course, that will evolve also as we see Kyverno evolve. But, Ritesh?

Ritesh: Yeah, so in terms of features we kind of review them individually and decide if there is an opportunity to leverage Kyverno to automate those capabilities.

So one of the things that Nirmata provides is a self-service for cluster provisioning or the self-service for namespace provisioning, which we call namespace as a service.

And a lot of those are high-level features that require several different systems to work together.

And for those kinds of features, there are definitely opportunities where we can leverage Kyverno.

And especially the mutate and generate capabilities of Kyverno, to automate some of the workflow, right?

So those then translate down.

And then there are other set of features where in Nirmata, obviously, policy management is an area that Nirmata provides as well.

And that's where those set of features are directly related to managing and operating Kyverno at scale, Kyverno and Kyverno policies at scale.

Depending on the type of features and capabilities, we kind of decide whether or not to integrate with Kyverno.

But any features or capabilities around policy management directly relate to Kyverno, whether it's deployment of policies or reporting policy violations.

And then providing that feedback loop, alerting all of those other aspects that an enterprise solution needs.

Marc: But it's like integrations in management functionality, you don't see that you're going to add specific policy capabilities only to the commercial platform.

Like, all policy capabilities, the ability to generate a certain type of resource or something like that, that's all to be in the open source project?

Ritesh: That is correct. That is correct.

Kyverno is that engine and all of the policy capabilities, enforcement of the policies and the types of policies that are supported, all of that is exactly the same.

It happens in open source.

The cloud, the platform, our SaaS is really to enable the enterprise usage of the policy management capabilities.

Marc: It's a great model. It works.

You get, like, easy early adoption from the project.

And then when the enterprise needs to wrap their, be able to think about it more from a compliance and an organization perspective, they have that functionality.

And that tool's available to them, so they don't have to go reinvent all that or go build something on top of it.

Jim: Exactly. Yeah, and one of the things we decided early on is we did not want to, in any way, sort of limit or cripple Kyverno for any monetization strategy, right?

So we want to make it easy to use, simple to use and full featured by default.

But there's enough challenges to solve if you're managing a fleet of clusters or even like, let's say, a dozen clusters, Nirmata solves those problems, with integration, with collaboration, with reporting, with all of that built in.

But Kyverno is completely full featured, can be used with all of the other open-source tools that we mentioned if folks are operating or building their own platforms.

Marc: Got it. So we talked a lot about the validating, mutating and generate type policies and actions.

And we talked a lot about the process that it took to build it. Let's talk about the roadmap.

What is the team working on right now? What's the next thing that we should see?

Jim: Yeah, so some, several interesting features.

And there are, of course, just kind of continuing down the path of more complex policies, more complex operations.

Like, there's several constructs in Kubernetes where Kyverno can call out into the API server and retrieve data.

And then as you're iterating and processing those, so to do more advanced operations, right?

Kyverno supports JMESPath or JMESPath, however folks say it.

And that is extremely powerful in, kind of, extrapolating and extracting JSON data and then applying that to policies.

But we're continuing to, sort of, evolve to more additional use cases there.

But, beyond that, there's also very interesting conversations and discussions going on for things like Kyverno also handling, for example, there's a subject self-review type of request that happens during your authorization.

So there's discussions in the community about that and what Kyverno can do to, sort of, help elaborate on that and maybe allow policy-based decisions on it.

There's also one of the things we've been looking at, of course, as things are happening with some of the recent incidents, et cetera, and exploits around the whole supply chain security and image security, right?

So that's another area we're exploring.

There was also some questions and some things coming up in terms of if there are workloads running, which are operating in a Kubernetes native fashion can Kyverno be used for some of the decisions there, right?

And that's also another area we're exploring and trying to figure out what comes next.

But there are, the published roadmap also there's features, like I mentioned, with distributing some of the workloads across, especially for generate, if there's hundreds of namespaces without a batch created, making sure we can scale out and things like that.

So, yeah, several categories of features, our roadmaps are, the active features are all published on the Git Repo.

And we're seeing also an increasing, the community growth has been great.

So a lot of folks jumping in, contributing, both in leading some of the design discussions as well as even in implementations.

Marc: And do you have regular community meetings that folks interested could join?

Jim: We do.

Yeah, we have started with a monthly community meeting but it's becoming apparent and we were just discussing maybe even going to weekly either office hours or additional meetings for maintainers, contributors, because sometimes you need to get on a conference call rather than just Slack conversations, getopt discussions, things like that.

So, yeah, we'll most likely go to a weekly model fairly soon.

Marc: That's great, good problems to have. I think, being in the CNCF, you get a lot of benefits.

But, like, only one of them, but a big one of them is just the marketing, the outreach, the visibility that you get being in there.

Jim: Absolutely. Yeah, no, that does help.

And I think it's just, for all of us who have participated and worked in the community, just the mindset, the approach collaborating with any other project is fantastic, right?

And just working together, whether it's on issues around pod security or any of this has been an awesome experience.

Marc: So when you see people adopting it, Kyverno, do you see a pattern of adoption where maybe they say we're going to start with generate, obviously that's one of these features that you have that nobody else in the ecosystem is doing really well, so that's what we want to do?

Or do you see just I want to implement and enforce policies at the namespace level?

What's a common pattern that you see for, kind of, dipping your toe in and getting started with it?

Jim: So based on, yeah, recent, or, again, like the discussions around pod security, that tends to be one entry point, right?

So say, okay, well, how could we make it easy to either replace PSPs or adopt if PSPs haven't been adopted, what's the easiest way to enforce and start validating for pod security.

So that tends to be a fairly common entry point.

And then the other one, like you mentioned, is just to generate rules to be able to based on.

And namespaces is a very fundamental building block, of course, in Kubernetes.

So starting to build these rules around namespaces and different usage models for workloads or for teams within an enterprise.

Marc: Do you have an adopters file or anything that folks can add their name to if they're just using just the open source, no commercial relationship at all with Nirmata?

Jim: Yes, we have added that and we haven't been very, sort of, active in promoting that so far.

So we do have a few adopters who have added themselves in there but there will be, and as we sort of continue to grow Kyverno.

Yeah, one of the interesting milestones that we've been looking at, somewhere around that November, December timeframe, we migrated from Docker Hub to the GitHub container registry.

And, since then, we've been tracking downloads.

So it took us about three months to get to a million downloads.

But then, in the last three weeks, we have now surpassed 2 million downloads, right?

So just the rate of adoption, the rate of usage has grown quite a bit.

And we're certainly very excited as well as humbled to see that.

But, yeah, in terms of having the adopters file, that's something that will continue to, sort of, build with the community too and have more public adopters who are presenting their use cases and discussing how they're using Kyverno.

Marc: Yeah, that's fun.

It's always challenging with open-source projects to understand, not everybody wants to be forthcoming and transparent about how they're using the projects.

And that's great. That's one of the benefits of open source,

Jim: Right.

Marc: they don't have to.

But, yeah, seeing those numbers is that's a lot of accelerated growth. That's cool.

Jim: Absolutely.

Marc: So I want to, as a newer Sandbox project, I'm curious, is there like a specific use case or if I want to get started with it, what type of feedback, or what use cases or what are you looking for from the community right now, what would be the best way to get involved and help you?

Jim: The first option is of course to, just as you're trying out the policy library, seeing the use cases that are there and then any suggestions for new policies, missing policies, other examples that need to be solved.

So that we're constantly looking for those and fairly quick to either help out, help write policies or be able to, if new features or extensions are required, we can turn those around too.

So that's what we're looking at, kind of broadening the set of use cases and types of things that Kyverno can do.

And then, beyond that, of course, there's also folks in the community who have built things like taking the policy reports from Kyverno and building tools to be able to visualize those.

So there's some really interesting things going on.

And then looking for any feature ideas, extension ideas, like we talked about, a few different areas, of triggers other than just admission controls that Kyverno can use to make policy decisions.

Ritesh: Yeah, I'd like to add to that, right? And I think we touched on this a little bit earlier.

And if there's somebody in the community trying to maybe write an operator for a certain use case, maybe you want them to look at Kyverno and see if that addresses, various constructs within Kyverno can address that use case, right, before they jump into an operator.

And, if not, what are the gaps and how can we address those?

If we get that kind of feedback, we can definitely make Kyverno, kind of, more capable of addressing vast majority of the use cases that people end up writing custom operators for.

And I think that's definitely would be a win across the board because now you have a common way of addressing so many different use cases.

Marc: Yeah. That's awesome.

Yeah, operators are fun and they're amazing.

And I think they make operating and running software a lot easier but that doesn't mean we all need to write them in, like, we all have to do the same thing. Cool.

So I learned a lot about the Kyverno project today.

Thanks, Jim. And thanks, Ritesh, for joining and digging into the full details and helping me understand it.

Jim: Thank you, Marc. A pleasure and, yeah, really excited to see the momentum and growth.

And looking forward to seeing everybody.

We're active on the Kubernetes Slack and the Kyverno channel. So come see us there.

Ritesh: Yeah, thanks, Marc, for having us. This was great chatting with you.