Ep. #24, SpiceDB with Jake Moshenko and Jimmy Zelinskie of Authzed

Marc Campbell: Hi, and welcome to another episode of the Kubelist Podcast.

Today's episode is going to be fun.

We're going to get pretty technical and dive into SpiceDB, an open source project from Authzed.

As always, Benjie's here with me. Hey Benjie.

Benjie De Groot: Hello. Hello. Hello.

Marc: So let's just jump in. Let's get started with an intro to our guests.

Today we have Jake Moshenko and Jimmy Zelinskie from Authzed joining us.

Jake and Jimmy have an interesting background and they have worked on two you've probably used in the Kubernetes ecosystem.

Welcome Jake, Jimmy.

Jake Moshenko: Hey, good to be here.

Jimmy Zelinskie: Thanks for having us.

Marc: Alright.

So we're going to spend time today talking about SpiceDB, a project that you've open sourced as part of Authzed, but before we jump into that, I would love to hear a little bit about each of your backgrounds.

Jake, will you start and just share a little bit about your background, how you got into the cloud native ecosystem?

Jake: Sure. Coming out of college, I worked at some of the traditional tech companies you may have heard about.

The most recent big company I've worked at, or before starting our first company was Google.

That's where I met Joey. Joey's our CTO and third co-founder. We left Google to start a company.

That company was called DevTable, and we were doing an online IDE.

The IDE never really took off, but we did build a thing out of that called Quay.

Quay was the first private Docker registry not if you're unfamiliar.

This was around even before like Docker Hub or Docker Registry or any of those things.

They had the public registry, but nothing that you could put private, sensitive images in. So we went and we built that, and that's the thing that dragged us into what I would call the infrastructure space.

So through that, that's when we started getting into things like container images, containerization, distributed systems, the whole, how do you run these container images at scale? That area.

From there, we got acquired by a startup you've probably heard of called CoreOS. At CoreOS we continued to build Quay and it became an integral part of CoreOS's offerings.

Then CoreOS got bought by Red Hat. CoreOS's distribution of Kubernetes became part of Red Hat's offerings.

Then we just kept growing our presence in this cloud native or containerization first infrastructure system.

Then, as time went on, Red Hat ended up getting bought by IBM.

Then we decided it was time to go out and do something new.

One of the problems that we kept hitting as we were building these companies and launching these products was the one-off permissions, like how do I add permissions to an app in a sane way and in a flexible way and in a way that, as time goes on and new customer requirements come in, they can be updated?

That's when we found out about the Zanzibar paper, decided that this was something that really hit the mark on solving those problems that we've been running into, and decided to go off and build a company based around it.

Marc: Yeah. That's great. I think a lot of us have used Quay.

Being the first external Docker registry outside of Docker Hub was super appealing to be able to see some of the enterprise functionality that y'all were building in there.

I think we're going to dive into that a lot.

Before we do, Jimmy, same question for you, your background.

How long have you been involved with the... I think you were with the Quay project too?

I'd just love to hear your background story.

Jimmy: Sure. So effectively my background is, in college I got super frustrated in my first course where we learned how to program with Concurrency.

I thought the Java APIs for thread pooling was really bad so I found this very young pre 1.0 language at the time called Go and started writing some open source software in Go.

Basically I wrote a piece of BitTor infrastructure that became part of Tupperware, which is the container orchestration system at Facebook.

Then when I went to graduate, I was looking for basically interesting jobs in the space and new CoreOS from their project at CD and effectively saw, literally just a hiring thread on Hacker News where Alex Pauley was asking if people wanted to work in New York at CoreOS.

So the rest is history basically interviewed with Jake and Joey, my co-founders and then joined the Quay team helped Quay eventually transitioned into product management roles and was managing Quay some operator stuff.

And then also some Ansible stuff when we got acquired by Red Hat, the rest is history, right?

Marc: Yeah. That's actually super interesting.

I met you way back in the day when you were sitting at the CoreOS office in New York.

The transition from deeply technical writing part of container orchestrations in Go is a pre-early language into product management.

While you did that there, is there anything else that you can share there as though, what drove you to make that decision and move over into the product role?

Jimmy: Yeah. Honestly, I think Jake was a huge help in pushing me to reach out of my comfort zone for that.

But I think he had correctly identified there're a lot of what drives me as an engineer is actually the outputs in producing really useful software for the user.

So where my motivation was coming from made sense for the product role.

I still deeply technical in the product role, you can be a technical product manager.

So I don't think that was actually limiting and in fact, it probably helped with having empathy for the different parties involved, right?

You can empathize with the customer, who's just trying to solve their problem.

We can empathize with the engineer for us to build and maintain the stuff, right?

Product managerial role is like to be a liaison between these two people or these two types of people.

And so I think having as much empathy as possible is what really helps with that.

Marc: Yeah. I think as we're building these like infrastructure product selling to engineers or DevOps related and things like this, the traditional role of a product manager who doesn't understand the implementation, it's really, really hard to get into that.

And so coming into that role, excited about it with a strong technical background where you understand what it's going to take to build it, you understand what it's going to take to adopt it and how to meet customers, where they are.

It's almost a requirement in order to shift these types of technical products.

And I'm sure the same is true at Authzed.

You're building it into that same super technical ecosystem and super technical end user.

Jake: Yeah. When your customer is a developer, right?

Your developer facing, it's hard to be able to empathize with the customer without having locked in those shoes.

So I think in our space, it's a hard requirement.

Benjie: So I want to hear a little bit how you guys started Authzed and what was the thing that got you guys?

"Hey, let's go do a new project." Is there anything in particular, is the problem space that was exciting.

What was the real impetus of you guys branching out and doing a second startup?

Jake: Well, second startup was always in the cards.

It's something that I knew I wanted to do from the minute we landed at a big co again.

Why Authzed, so, like I mentioned back in the summer of 2019, when Google published the Zanzibar paper, I read it and immediately mapped the solution that they were putting forward to the problems that we had experienced in the past.

This was back in 2019. And I was like, Jimmy, Joey, let's go start a company.

For various reasons, we didn't end up doing that until 2020 when we set out to go build the thing, I was like, "Well, surely in the last year there must be some competitors in this space. Someone else must be trying to do Zanzibar."

And in that whole year, nobody else really seemed to be pursuing it or taking it seriously.

And I'm like, this space is still wide open.

As it usually goes of course there are other people trying to do Zanzibars now.

Some people have done it internally. There's a couple other projects open source, and otherwise that are doing it.

But I still think that the space is largely open and largely unsolved.

So, it is just going and creating that solution to solve the problem that we had experienced in the past.

And then having the framework of the paper to help us have confidence that the method or the route that we were pursuing was going to be scalable and bare fruit and be as flexible as it looks on the surface.

Benjie: Great. So you guys found the problem space, "Hey, this is something I've dealt with and he a really good implementation."

All right. So for those of us that don't know what Zanzibar is, myself included, what is Zanzibar?

I know it's complicated. I would also love to hear any insights you might have of, did you use some version of Zanzibar internally when you were at Google?

Because you said you were at Google earlier in your career.

So just give us a little more understanding of what Zanzibar is and how you've used it pragmatically or not.

Jake: Yeah. So potentially this is a lucky coincidence or lucky happenstance, but I was at Google before Zanzibar launched.

So I have no insider knowledge and therefore I'm not worried about accidentally violating something or doing something that I shouldn't know.

So getting back to what is Zanzibar, Zanzibar is a way of representing all of the relationships between people and data, data and other data, people and other people as a directed graph, and then setting up permissions queries like trying to solve permissions problems as doing a graph traversal.

So basically you can think of one end of the graph, you have all of your resources.

So in Quay, this would be things like repositories, in GitHub it would also be repositories. In Google Docs, it would be the documents themselves.

And then at the other end of the graph, you have your subjects.

So those are usually things like users or access tokens. And what Zanzibar does is it turns this problem into saying, "Can I find a path through this directed graph from the resource to the subject?

And if you find the path and the path has an edge with the proper permission, then that user is considered as having that permission. Does that make sense?

Benjie: Yeah, totally.

So it's like you're traversing the graph to see, "Hey, does user X have permission or some type of permission for repository Y?"

We'll use the GitHub analogy that you used there.

And so that gets interesting and I'm inferring here a little bit, but that seems like it gets interesting when you have groups and subgroups.

I know with my GitHub repository, we've got a product team and then we also have an engineering team and these are just the teams within the GitHub org itself.

So is that a correct assumption, that the interesting thing is when you start having nested and relational types of things within these groups, is that right? Like a hierarchy?

Jake: Yeah. That's exactly it.

Whenever you have a complicated hierarchy and moreover, when that hierarchy implies things about the way permissions should be computed, that's when you really benefit from modeling it as a graph and being able to solve it as a graph traversal problem.

Take, for example, like a bank account, this is something that everyone's familiar with.

If you want to be able to read the transactions in a bank account, what are some of the ways that you get that permission to read those transactions?

Well, number one is you own the account, right? That's the really obvious one.

Number two could be that you're an authorized party on the account, potentially like a spouse, a financial advisor or something like that.

Another way is, this could be like a corporate account and you could be the CFO for that corporation and you get to read access on all accounts.

Another way could be that you are the counterparty to a transaction, maybe that gives you the right to read that transaction, right?

So you can see that the way people structure things is like a social problem in terms of the way people relate to one another.

And then we structure our data in very similar ways where we like to put things in like nice clean hierarchies.

And we like to have things inferred based on our role within those hierarchies.

Benjie: That totally makes sense. Yeah.

Marc: Yeah. So then Zanzibar is a white paper that Google release that defined how to build that graph model, how to define this and to say like, "This is an approach."

And then Authzed your startup right now, is it fair that Authzed is essentially an easy to use reusable implementation of the Zanzibar paper that I can implement into my stack and then have a reasonably well thought out and easy to use permissions model.

Jake: Yeah. I think that's fair. Spice DB is our implementation.

If you want to think of it as the clean room implementation of the Zanzibar paper, and then there are some things that we added or inferred or innovated around the Zanzibar paper, because we were finding that...

Sure when you're at Google, you can just tell your engineers, you must go interact with these APIs, however good or bad they may be.

But when you're a startup, you have to entice users to come and to use the platform.

So we've invested a lot into making the APIs easier to understand, making sure that there're something that solves more use cases than just the ones that you might have at Google.

And then we have our schema layer built on top of the very rudimentary name space, definitions and text that's in the paper as something to just help people write better schemas and reason about their permission systems better.

So, take the Zanzibar paper, do a clean room implementation and then start innovating on top of that to make sure that this thing is actually usable by the general public or the people who would be implementing one of these things.

Marc: So SpiceDB is completely open source and it's a clean room implementation of the paper.

It might take a little bit of work for me to integrate it, but like I could literally just grab SpiceDB, drop it into my app, write integration code and then have an implementation of Zanzibar in my SAS application today.

Jake: That's exactly right.

Jimmy: Yeah. Not only that though, you can additionally use, Authzed.com and have to download and run SpiceDB as well.

So you can log in and have an effectively serverless version, like hosted SpiceDB for you?

So if you want to service dependency, you can do that.

Benjie: So, I would imagine that one of the challenges with Zanzibar in general is availability.

And I'm assuming that SpiceDB does a lot of work there to make sure that that's a thing that's available.

Can you talk a little bit about the challenges and what SpiceDB does in regards to availability and how do I...

If I'm using Authzed, for example, for my cloud version of Zanzibar or of SpicyDB, how do I know that you're not going to go down and cause my application to stop functioning?

What's the model there? How does that work?

Jake: Well, first of all, all services go down. The idea is to just minimize that as much as possible.

So Zanzibar as running in Google has a demonstrated performance of five nines of uptime.

Maybe if you're unfamiliar that's 99.9% uptime and the way Google measures uptime, that means that 99.999% of requests to the service are successful requests.

The way that this is accomplished is making sure that there are no single points of failure.

So Zanzibar at Google is built on top of Spanner.

So Spanner is Google's globally distributed asset database that you can use to program on top of, to not worry about a lot of the consistency concerns that usually creep up in some of these types of distributed systems.

For SpiceDB and Authzed, we've closely followed the implementation details or the implementations and performance recommendations and strategies in the Zanzibar paper to be able to replicate that type of uptime and eliminate the single points of failures whenever possible.

So one of the ways we do that is we use CockroachDB or Cloud Spanner instead of having access to Spanner directly on the network like Google does.

Benjie: Okay, great.

Because one of the things that I find really interesting about SpiceDB, the project is getting Google scale infrastructure and Google scale papers type thing, obviously there's all kinds of implementation details there that can be challenging, but how you deal with uptime is really interesting as it's a great way to look at it.

I think we all are aware of services going down to Jake's point.

We all had a fun December with AWS and Docker hub and ECR and all kinds of things.

So that's a very fair answer there. So I think backing up a little bit.

So you started Authzed, SpiceDB, was that something that you planned on from the beginning or is that something...

How did this open source project come to be? And it seems like it's starting to get a lot of traction here.

So just talk to us about where you started with Authzed and how SpiceDB became this project and the evolution there.

Jimmy: Sure. So with our background, just coming into this space, Quay was entirely proprietary for a super long time and then eventually it was made into an open source project.

But outside of that, we had worked on other open source projects in the cloud data space for a while, like the Operator Framework, for example, and lots of different CoreOS components.

So having worked at CoreOS and then Red Hat open source was in our DNA.

We understand how to do it, but at the same time, it means that when we do it, we want to do it right.

So at first we were a little bit hesitant as to like, well, what are the parts that we're going to actually make open source, what are not? But ultimately when we started to really frame and understand that our product was more of a database, that was when we had the realization that the status quo for databases is they have to be open source these days.

Like excluding Oracle, which I feel like just has prior traction from like another era.

I can't really think of many proprietary database that have really had success in more recent years.

So I think there's a level of transparency that you need when you want to run any database, considering how critical it is to your application and viewed through that lens it was obvious that we need to take that component and run that as an open source project and reap all the benefits and all the negative aspects of that style of development.

Marc: That's actually super interesting and I'd love to spend a minute or two digging into more the open source side of things like you probably could have and likely had conversations where it's like, "The implementation of Authzed or SpiceDB is actually a database, but we could just wrap this thing as a service."

And that's an implementation detail, it doesn't really matter.

This is a hosted thing, but making the decision to say it's a database, what led to you actually deciding that that was an important thing that you need to share with end users and folks who want to implement the project and position it, that transparently.

Jake: Well, I think fundamentally, it came down to a lot of our early conversations were highlighting the fact that users were actually somewhat surprised that they were storing data in SpiceDB.

So part of the Zanzibar model is to centralize all of that relationship data so that when you need to make a permissions query, the data is already there.

It can be requested uniformly from various different microservices or applications and an application suite.

And therefore, because it's centralized, it's also highly cashable it's available basically to their permissions enforcement points, pretty uniformly.

So in order to do that, you have to send the data over to SpicyDB and without that, like DB moniker on there, people were like, "Well, how do I connect this up with my relational database to make sure that it's reading the right relationships."

It was a conversation that we kept having over and over, which is like, "No, you have to send data to it so that it knows ahead of time how to make those permissions decisions."

And so now that we call it a DB, we don't really get those kinds of questions anymore.

People understand that they're going to be storing and retrieving data from this service.

Marc: So by positioning it that way, it actually made it easier for folks to adopt, because you could fit it into the mental model of what successful implementations looked like instead of thinking, "It's an API."

When you are able to say, "It's a database," the average developer who's integrating it says, "Now that makes a lot more sense."

Lot less questions. I know how to integrate this thing.

Jake: Yeah, exactly. Whenever you're trying to do something brand new.

And I think having permissions be a service is brand new for most users.

You want to give them as many affordances and hold their hand as often as possible in order to give them a mental model where they can reason about what your service is going to do.

Because otherwise it's just going to seem like a complete mystery, complete magic.

So this is just one of the ways that we went and helped the users have a model for how to interact with this thing.

Benjie: Yeah. I think it's a really interesting point, Jake.

I think that being opinionated in our space is something that is underserved often and it really makes it more challenging for newbies or for later adopters to understand where it is.

So I think that's super interesting. So, that's the DB part.

What's the spice part. Tell us about where the name spice came from.

Jake: As far as we know, or according to the Twitter Rumor Mill?

The original name of the Zanzibar service at Google was Dune.

And as you know, the big thing out of Dune is the spice must flow.

So we just wanted to pay homage to that original naming.

When we picked the name, well, at least I didn't realize there was a Dune movie coming out that year.

So it also just was an interesting timing wise, because people were like, "Yeah, the Dune movie, right? Yeah. I get it."

It was like, "Well, yeah but not really more of an OG reference."

Benjie: Yeah. No, and we will go on record as saying first off I've known Jake and Joey and Jimmy for that matter for a very long time.

Now, back in the CoreOS days, actually pre-CoreOS days.

And I recall discussing ideas of this project and the name Arrakis came up a few times.

So I can attest that before the movie was on the radar, that there was a Dune theme here.

I want to understand a little bit more about when you guys decided to flip the switch to open source because I know you guys were working on this stuff and then what were the big challenges there around both switching to open source, but just also just building up a business around this model.

Jimmy: Sure. A lot of it has to do with planning.

In terms of like, you're going to change entirely what your workflow looks like.

So prior to that, we had very few individual like open source projects.

We did have a couple minor things, but only little libraries or our client libraries that we wanted our users to actually be able to use to integrate.

So internally we had monorepo and that's where all the code lives.

Well, that means now you have to tease apart the dependencies within your monorepo.

You have to change how all things interacting get tested.

And then there's basically a whole workflow change on how you're actually going to do planning for the open source part versus the private part, right?

There are just like so many different aspects.

I still think we're like dealing with some of these organizational aspects, but trying to identify the minimum viable, teasing a part of the proprietary code with what code you want to be open is basically the first step there.

And then depending on what you're trying to do with your open source project if feel like you have to plan a marketing side for it.

So we wanted to make a big announcement and have a big splash.

And for us, we decided we wanted to do that as a one year anniversary to the company being founded.

So there's a lot of them prep work around that and prepping under embargo, the announcement and then hoping you get traction on social media, it's a whole workflow and whole initiative, right?

A spring to multiple weeks, right?

Marc: Yeah.

So, what you're saying is it's not like go find the latest Google white paper, put some headphones on, write some code in a weekend and then you have a profitable business, like with that's open source yet still be working.

There's a lot of work involved in it.

Jimmy: Yeah, absolutely. I think it's more about facilitating open source in the ecosystem as well.

There's a whole lot of meta work there, like maintaining a open environment and trying to foster a community is a completely separate task from just like pumping out some code internally.

And for example, one of the things that we've been investing a lot in and seen a lot come out of is our community discord.

So if you go to like authzed.com/discord, you'll join a couple other 100 users that are all developers modeling their own permission systems, asking questions, answering questions for each other.

And we've been pushing all of our customers into that community as well, so that they can give back with their knowledge.

I feel like there's lots of other companies out there trying to do open source.

They'll use whatever different forum tooling, or they'll maybe use GitHub discussions and things like that.

And I think it's all about finding the right tools that not only does your company want to use, but also the community that's forming around the project, what they want to use.

Marc: So, Jimmy, that's interesting.

I think we've had a few other projects on here often we're talking with folks who have CNCF sandbox or incubating or graduated projects and the CNCF is really the Kubernetes ecosystem and the broader ecosystem around using Slack channels to communicate.

But we have had a few other projects really recently that have started moving off to Discord and going off in this direction for just a second.

You chose Discord and was there a thought process and how intentional was it to say, "We don't want to use Slack and we want to use Discord."

You mentioned GitHub discussions is another option too.

Curious what drove you to land on Discord?

Jimmy: I'm not really sure, we had a very long discussion about this.

I think generally we wanted something that was similar to the Slack experience.

But basically when we realized that for Kubernetes Slack or the CNCF Slack or any of these Slacks, you have to go through an approval process to be able to be given a channel and a place in that community.

And I think between all of us, we wanted to foster our own community.

So starting from scratch, you're allowed to break free of these things.

And we think that a lot of the functionality in discord was appealing and is effectively free and it gets to retain all of your chat history and things so you can actually search and see older conversations and track some metrics slightly better community wise than I've seen at least with our corporate Slack.

So there are a couple tools there that are more useful, but also I feel like it's a platform that's just more inviting in general.

It's easier to join a new Discord server than it is to add yet another Slack server.

And then also Slacks aren't open. So then you have to run some automated invite system.

Slack just isn't made for this, Discord is, and/or is pivoting into that space more and more every day.

So every day we see more things added to the moderator channel that are actually functionality we can take advantage of.

Benjie: Yeah. It's an interesting trend that I'm seeing as well.

Maybe I shouldn't say this, but I prefer Discord over Slack for community stuff 100%, some of the bots and some of the interactions, it just seems pretty slick, but that does bring up a really good question.

Are you guys going to look at getting SpiceDB into the CNCF or some other foundation? Apache.

Is that something that's on the horizon? Is it something you haven't thought about yet?

And how do you guys look in general at governance, in the governance model that you guys want for SpiceDB?

Jake: So like, why would you contribute a project to one of these foundations?

There are basically rights and responsibilities that are associated with belonging to one of these foundations and obviously the benefits from the creator's perspective, right?

Because at the end of the day we're building something and we're putting our effort, our blood, sweat, and tears into building something.

And from our perspective, as a creator, the benefits would need to outweigh the costs.

Is there a community that is a perfect fit, like would be a perfect home for SpiceDB, I'm not sure, right?

It's not a thing that you want to just donate.

You can't really fire from the hip when you donate your project to one of these foundations, because you can only do that once, right?

It's like a latch that once you cross that Rubicon, it belongs to that foundation.

So I wouldn't roll it out, but I also don't see that there's a perfect fit.

We're not limited to the Kubernetes or the cloud native ecosystem.

So is CNCF a perfect fit for us? Probably not.

Would Apache be a perfect fit for us?

I don't know. It's not something that we've gone through the evaluation on.

Jimmy, do you have anything to add there?

Jimmy: Yeah. I think I can follow that exact same train of thought.

I think generally databases can be applied in many different environments and not just necessarily the cloud native one.

And while you absolutely do want a community driven project where there is some influence over the community to own parts of that project and help control that project.

I don't think you necessarily always need a foundation to be able to do that.

You can accomplish that without one.

And if we had to make our own mini foundation for just our project, I think that might actually make the most sense.

I would rather like to see where our project goes and who contributes to it rather than just assume that we are a part of the CNCF space.

Because even if you look at just Lennox Foundations, there's also the open SFF or SSF their new security foundation.

Doesn't make more sense for us to be in that foundation.

Doesn't make more sense to be in the cloud data foundation.

It's actually more of a product of your community to make that decision.

So I feel like we can't be really the judges to say, "Hey, I'm going to donate this to this specific foundation."

Rather, I would like our community to reach up to assume responsibilities in the project and maybe give us the hint that like, "Hey, this would actually be a really good fit. Or maybe we should do our own thing or just where are things that we can hand the reins to our actual community directly, rather than putting that on a foundation where we don't necessarily know exactly if that's what our customers are going to value, right?"

It's the foundation's values versus your users.

Marc: Yeah. I think that's totally fair, right?

Jake, to your point the benefits have to outweigh the cost.

And when it's not a project that you built to support your company, but it's literally the internal, the core project that you're building your entire business around, it's a thoughtful decision that you have to put into it.

There's obviously lots of value, when large organizations look at a startup and they say, "I want to think about continuity and longevity. I want to put a lot of work into adopting this project. How do I know it's going to last?"

But there's lots of ways to solve it like donating the trademarks to a foundation are one way, but also it's an open source project it's going to live having permissive licenses that allow people to use it the way they want to get what possibly what you need also.

I actually want to shift for a second and really dive back into the product.

I don't have a Greenfield project that I'm going to write.

I have an existing product that's out there in the market. I have tens of thousands of users using it but like now I have the need-

Jake: Congratulations.

Marc: Yeah. Well, this is a fictitious example here.

So I want to make a more robust permission model on how to actually do it.

So it looks the like SpiceDB or Authzed is a possible solution, but I already have some R back in my app and I already have some homegrown implementation.

Can you tell me what it would be like to adopt SpiceDB or Authzed, bring it in and replace all the homegrown stuff that I have that are really high level?

What would I be looking at to do?

Jake: Yeah. And I'll also say that this is the majority of the people that are adopting it right now, because they found themselves in a place where continuing to maintain their existing code or their existing homegrown solution just is not tenable anymore either because they're extending the number of languages that they're using internally, or they're extending the number of services or they're adding applications to the application suite.

So this is not an uncommon pattern at all. So essentially what the adoption process looks like is that you first have to go and develop a model.

We advise our adopters to start with model their existing model exactly as it exists today, right? Don't try to go straight to your ideal model.

Once you've got that model in place, then you can start backporting your data into it and whenever you need to write a relationship, you can continue to write it in your database.

And then you can also write it into SpiceDB while you move all your existing relationships over and into SpiceDB.

Benjie: So, how do I model is the real question?

Do you guys have a DSL? Do you guys have a schema? How do I model?

Jake: We have a DSL that helps you write your schema.

And then we have a playground where you can go and you can develop these schema, like a synchronously.

You don't have to actually call the live service to test it.

You can create a set of test relationships and test assertions to make sure that the model is performing the way you expect.

And you can do all of that just ephemerally using our playground.

Once you have the schema ready, then you can go and you can provision a permission system, or you can launch your SpicedB cluster and you can write that schema into SpiceDB cluster.

Once the schemes there, that's when you start bringing your relationship data over.

Marc: So because of that and some of the stuff that you mentioned there, Jake, like it's very testable.

I hook this into my CICD pipeline I can ensure I'm not getting regressions.

I don't have permissions like that.

I accidentally wrote some code and broke something I can continue to test this same way I test any other part of my application.

Jake: 100%. And we even have a special version of SpiceDB that runs in test server mode that's designed for the unit test and the integration test case.

So basically the way it works is every time you give it a new API access token, it gives you a brand new clean version of SpicyDB for you to load in your example, data, load in your schema, run your assertions against it.

So you can of course do that in parallel, just using different keys.

So this is something that we've thought about, and we have the story around how to do this in your CICD how to roll it out to production, because of course we've done it ourselves, right?

Marc: Yeah. That's awesome.

That's something that's historically difficult to test custom permission models inside an application.

And so you end up doing some crazy and hard to reproduce things.

So like having that built into the platform is amazing. That's great.

Jake: The other thing that happens is people just write V1 of their permissions code and then they avoid changing it at all costs.

Because everybody's afraid to touch it because they remember that one time that they opened a security hole.

Everybody's been burned in this space.

And so innovation here is I think far past the point where we should have something to help us.

Jimmy: It gets in the weeds a little bit too.

But the reason why Jake earlier was saying you can just immediately model what you have and not necessarily be concerned.

Isn't purely based on the fact that we have tooling to test it.

But the fact that you can define new permissions in terms of old permissions.

So you can basically make forward compatible permission changes and then slowly migrate to those in the code over time.

So all of a sudden, it's not only that making a change to your permission system is a matter of how much can I test to it.

But actually it becomes less friction full in terms of how much code you have to actually even make the change.

And sometimes it can be literally no code changes at all. You change your schema and that just does it.

Benjie: I want to ask a quick question here. First off.

I love hearing about people thinking about ephemerality and how to test things so kudos for that.

But how would this work taking it back to the CNCF Kubernetes landscape a little bit?

Is there a concept here of how RAC fits into this.

Walk me through how I could use this with a Kubernetes setup and in conjunction with my RBAC set up or just give me like a practical implementation there that might, that might have some weight.

Jake: Now, are you talking about your applications RBAC back or are you talking about your Kubernetes clusters RBAC?

Benjie: Well, that's probably a good question for both, but I'm particularly talking about my Kubernetes RBAC can I integrate that in?

So say I have some operator that has a permission that I only want certain things to do. Does that make any sense?

Jake: Yeah. We haven't really explored integrating or replacing the built-in Kubernetes RBAC with something based on SpiceDB, although it is something really interesting and something that we would be very, very keen to explore in the future because if you've worked with Kubernetes RBAC, it's like an additive only model and sometimes that's insufficient, right?

There's no way in Kubernetes RBAC to say a person is allowed to delete the resources that they created or that they've owned, right?

That's just not something that you can express.

So we would love to be able to make Kubernetes RBAC better, but we're primarily focused at this point on providing permissions for your application.

And one thing to keep in mind is RBAC is just one of the models that you can model in SpiceDB.

Another really common, popular model in SpiceDB is point to point sharing.

If you want to collaborate on a document, usually I don't add a user to a role.

Usually I just share the document with them or I send them a URL.

Those kinds of like sharing things are much easier to model than they would be in like a traditional RBAC situation.

You can also model a lot of like social graph and like aback style things in SpiceDB.

So RBAC is just not the end all be all of permissions models.

Benjie: Well, that's what you say, but some of us know. That's a really valid point there.

And where my brain was going was there are applications that probably shouldn't, but definitely do intermingle Kubernetes operators or CRDs with application logic itself.

And I was just trying to think if like, if a spicy be had some type of a bridging ability there, and it sounds like it very well might in future pull requests welcome, it sounds like.

Jimmy: Well, we actually have some experiments in this space.

We have open source, all of it's purely experimental and none of this stuff we've really shipped in any form, but we have experimented with the idea of connections.

And for example, a connection would basically be using something else as a data source for those relationships so that you don't necessarily have to copy data into SpiceDB, but SpiceDB would datively understand how to read data out of another data source.

So for example, you could bridge that with, one of our open source connectors is Postgres and how that works is it just looks at foreign keys across different tables in a Postgres database.

And it uses that as a primary key foreign key, primary key to the other table.

That is a relationship that is similar to subject relationship resource.

So you can theoretically see a world where we're hot plugging these different things into data sources rather than just using a single data source.

Sorry, it's a Kukuklok, if you can hear that, but you can see a future where you can read from external data sources and not just from the solo SpiceDB database.

Marc: So I love the Kukuklok, that's amazing.

So you read the Zanzibar white paper, and then you realize, okay, great.

This is something that we can build an implementation foreign ship and it's around Authzed.

And I can actually implement this really easily in my application.

I'm curious if there are any interesting use cases that you never thought about that you've seen obviously keep as anonymous and everything like this, but like more than just, I want to protect the objects and control who gets into them.

You alluded to some other use cases, but were there any surprising ones where you're like, "Wow, that's actually really clever. That's a use that we wouldn't have thought of for SpicyDB or Authzed."

Jake: Well, I think one of the areas is entitlements.

If you think of software entitlements, those are just a type of permission.

Do I have permission to run this software?

Do I have permission to enable this feature?

Similarly feature flags can and also be considered as a type of permission.

So we've seen people add those kinds of things to their SpicyDB models, which is a surprise.

Another one are opportunities to use things that aren't natural persons as the subject in a permission check.

So rather than saying like can Mark or can Benjie do this thing to this document, it's like, can this machine do this thing to this document?

Or can this access token access this API? Or is this organization itself a member of some hierarchy or whatever?

So, this is actually an extension that we made to Zanzibar.

In Zanzibar everything has to be backed by this unique user ID that Google assigns, but in our system, you can actually use any object type in the system as a subject in a permissions check.

And we're finding some really, really neat use cases where people are taking advantage of that.

Marc: Yeah. That's super cool.

Like thinking about replacing an RBAC model but adding entitlements and potentially feature toggles and stuff like this.

Yeah, totally makes sense. I'm curious, like when you were explaining that, I started thinking about the fact that like I also now need to know in audit log, did somebody attempt to access something?

They didn't have permission? Like who accessed it, when?

Is that part of either the Zanzibar white paper or your implementation through an extension or something on your roadmap?

Or do you just say, "No, we're going to keep that over here and figure out audit logs separately.

Jake: Yeah. This is one of the roadmap things.

It's a feature that we've been waiting for people to say, I have this concrete need for an audit log and then we'll go build it.

But so far it hasn't actually turned up, right?

If you're looking at this as a replacement for your homegrown authorization code that you have in your application, nobody does that right now, right?

That's actually something that we built into Quay really early on, where you could see who was doing what to repositories.

A lot of people just don't have an answer for, right?

If someone attempts to access a Google doc that they don't have access to, they get like a request share screen, but you as the doc owner are not notified and that doesn't show up in an audit log anywhere.

So, it is just something that we've been patiently anticipating.

We don't like building things that nobody's asked for, because it just you can often build the wrong thing, but it is something that we think will come up in the short to medium term.

Jimmy: Yeah. There's also just two different pieces of functionality that we already have that I think like maybe satisfies people already, we have structured logging that is actually really robust throughout the whole application.

So you can see the different request level logging, if you actually just wanted to write an app produced job over your log output from just serving traffic.

But additionally, we also have a street GRPC API that is a watch API where you can actually watch for changes in data.

And while that's not going to catch accesses, it will catch modifications, like rights going through.

And I think the combination of these two pieces of functionality, like people can build their own software that listens to the watch API.

And then creates whatever audit list or auto log that they want.

They can capture that data on their own, record it to whatever systems they want.

Marc: Yeah. That's awesome.

So if somebody has adopted, I don't know, Splunk or something like this.

They could throw a little bit of glue code, connect the two up and then have all of that into their SIM or their auto log, whatever that is.

Jake: Yeah. I had a somewhat surprising interaction with the user the other day where they were like, I want to be notified if a permissions check fails a certain number of times.

And it's interesting because in their use case that's a red flag, right?

But in a lot of use cases, if you were iterating through all of the documents on a system and then sending a check for which ones, a user at your webpage has access to, you would expect more to fail than not.

And so that's not necessarily indicative of attack.

And from our perspective, we successfully said that this person cannot see it.

We do not attempt to sign any motivation or emotion or anything like that to our permissions check failing, right?

Like saying that there is no permission because it's very use case dependent whether that's actually something interesting or whether that's something potentially malicious.

Marc: Yeah. That totally makes sense.

There are, like I mention and Sims and other tools out there that are designed to do that.

So as long as you can get the data out of SpiceDB and fed into a different system, it's data that can be used to make informed decisions about whether or not there is an attack in progress.

It's not a complete closed system that is responsible for making that decision.

Talking about some of these though brought up another interesting thing in my mind.

You recommend somebody adopting Authzed by replacing their existing permissions model. Modeling it exactly like their current model and then be able to expand it.

So over time, I imagine that these hierarchies and rules and policies start to get like really, really complex, hopefully as like that's the power of the system is you can create these really complex and intricate rules.

I'm curious how SpicyDB helps me debug a lot.

I thought I had this set been a way that this user shouldn't have permission, but is granting them permission, is there tooling built into the project to help me understand which rules are happening, where in the complex hierarchy?

Jake: Yeah. As we mentioned earlier with the playground and with the test service, we try to prevent these things from happening in the first place.

So you can make a set of test data and a set of test assertions where you can model up a user as like... I think a user who has these relationships should never have access to this thing.

And that will be validated every time you make a mutation to your permissions model.

But in terms of figuring out why that happened at runtime, we do have a few APIs like our expands and our lookup API, which will help you work through the existing relationships and how those go into the graph.

But there is no single API where you can just click a button.

There's no explain plan, essentially right now.

And that's something that we are thinking about and get gathering requirements for, but haven't implemented yet.

Jimmy: So we do have something interesting in the playground in the same vein, but not the exact same thing.

We have an expected relations tab.

So if you feed test data into the system and you have your schema defined, when you go to the expected relation tab in the playground, you can actually ask SpiceDB, "Hey, what do you think everyone should have?

And it will actually generate an exhaustive list of like, these are all the relationships that these people have and why, and that can help you immensely when debugging this stuff.

But you still have to distill it down to a test case that you fit inside the playground and not a live system, but ideally you've basically done a ton of testing before you promote something into the live system.

Benjie: So, I think that brings up a really interesting question and that is what is on the roadmap.

And how do you guys determine the roadmap you alluded to? We don't want to build it until people want it.

Tell us how you measure that and tell us about your upcoming roadmap.

Jimmy: Well, how we think about the roadmap is divided into two different forms.

Obviously there's Authzed.com like the companies roadmap, and then there's also SpiceDB, the open source project roadmap, the opensource project roadmap is effectively.

You can just go to the GitHub issues and see it.

We haven't adopted a concrete release cadence, so we don't necessarily use milestones, but everything is prioritized.

And there are various proposals that collect feedback and like filter to see things that are currently in discussion, currently need discussion.

They need to get fleshed out more. They want more feedback from users.

We also have external contributors that are actually working on realizing some of those proposals that are not necessarily us.

If you've got the resources and this matters to you and the community agrees that this is the proposal to move forward, you're free to move forward.

And there's nothing necessarily stopping you.

We want to facilitate any work that fits that model and community.

Do you want to take how we think about the enterprise roadmap, Jake?

Jake: Yeah. There's no magic for the enterprise roadmap.

It's like enterprises have very enterprisey style requests and then as those requests come in, we try to map them to one or more customers and then just make sure that we're solving the biggest problem first, whenever we can.

So, that's just standard enterprise software development.

Benjie: Is there an interplay between that and the open source roadmap at all?

Jake: Yeah. There's always going to be some overlap and there's always going to be some things that...

These features show up in the open source because they were either requested by an enterprise user or they support an enterprise feature.

This is part of the magic and reality of open source businesses, right?

At the end of the day, it's better for SPiceDB if Authzed, the company stays around, right?

So there's always going to be some tension there and there's always going to be compromises to be made and things like that.

We do take a very strong approach to making sure that SpiceDB can be useful on its own.

And that things that go into SpiceDB when possible can be made more generally useful than maybe the minimal implementation just to support some enterprise feature might have been. Does that make sense?

Benjie: Totally. Okay, great.

So I'm convinced I want to use SpiceDB and maybe even the Authzed cloud product, but I want to contribute also because there's a missing feature.

How do I get involved in contributing? What's the best place to go? Are there meetings?

How can I become a contributor to SpiceDB and just because Jake and Joey and Jimmy as well know how good of an engineer I am.

This is a hypothetical I will not actually contribute I promise.

Jake: Yeah. On our SpiceDB repo, there are some issues that are labeled as good at first issue.

Those would be a good place to dip your toes in the water, get your feet wet a little bit, we have the community Discord, which Jimmy mentioned earlier is @Authzed.com/discord.

That'll auto invite you to the server and in the Discord, that's where you can get to know the team and get to talk to other users that are integrating things like that.

And then beyond that, if you want to do something that's not already marked as an accepted proposal, then the place to start would be to open an issue either as a feature request or as a implementation proposal on the SpiceDB repo.

And then from there we would just handle it the way we would any other proposal or feature your request.

Jimmy: Yeah. I would just add that our discord has a specific development channel as well.

So there's general questions room, but then also if you want to specifically get your handheld, like walk through an issue, there's discussion dedicated to that in one of the Discord channels.

Marc: Great. All right. Thank you to our guest today.

Jake can Jimmy from Authzed, I'm excited on how you're thinking about this problem and making it just so easy to integrate and test and go build authorization the right way.

So I don't have to go figure all of this problem out myself, whenever I'm building a SAS application SpiceDB, go out to Authzed.com, check it out.

Jake: Thanks for having us.

Benjie: Thanks for coming guys. Really appreciate the time. Thank you.