Ep. #25, Tilt with Nick Santos

Marc Campbell: Welcome to today's episode of The Kubelist Podcast.

We have a great guest with us today, Nick Santos, CEO of Tilt.

Nick is an engineer who decided to create Tilt because, well, I think we'll dig into that.

But I think he just felt sorry for us trying to use Kubernetes as a development environment and buried under the weight of the tools.

We'll give him an opportunity to explain the motivation behind Tilt though.

But first, as always, Benjie is here today. Welcome, Benjie.

Benjie De Groot: Hello, Marc.

Marc: So we rerecorded our intro to the podcast for this episode to talk a little bit more about Shipyard and the work you're doing there.

We're going to have to definitely dig in and spend a little bit more time talking about Shipyard at some point, because what? Ephemeral environments, you want to give a quick intro?

Benjie: Ephemeral environments are the future, or actually the past that we're finally catching up to the rest of us. But yeah, we'll dive into that in a future episode, but very, very excited to get into the intro.

Marc: Awesome, great. All right, Nick. Welcome.

Nick Santos: Hey, Marc. Hey, Benjie.

Marc: So really excited to talk about Tilt and your background, so to get us started would you mind just telling us about your background, your career and what led you to creating Tilt?

Nick: Sure. It's been kind of a windy path, my background is actually as a front end engineer. I started out pretty early on doing a lot of UI work. I worked on Google Sheets for a long time.

One of the things we ran into when we were building Sheets was just complicated JavaScript apps were really a pain to develop at the time, and so I also ended up while I was doing the UI stuff, I drifted more towards the tooling side.

How do I get the build system to support JavaScript better? How do we get engineers to work together on a big, shared, JavaScript codebase?

I ended up building a lot of type checking compilers, that sort of thing, to help people be more productive.

After I worked at Google I went to Medium early on, it's a blogging platform, really to work on writing tools.

And I'd worked on writing tools for a while, I was a UI engineer, and after a while as more people joined and the system got more complicated we started seeing a lot of the same problems around developer productivity.

Medium had different build system problems, but it was a bunch of microservice apps.

To run Medium locally you had to run a node.js app, you sometimes ran some ghost services, you sometimes ran a fake database, you sometimes ran some message queues.

I had spent some time when I was there investigating how can we make this experience better?

And looking at what different build systems could do for us, I think I investigated Buck and Bazel and Pantz was big at the time, and I even looked at Gradle.

What I felt at this time is that a lot of these build systems were great pieces of tech, they're very complicated, build in a graph of your build logic and figuring out the right way to build artifacts.

But we weren't really solving the problems that I thought we should be solving for microservices, that it was really more about, like, "Okay, how do we bring the services up? When should we restart services? How do I know which services are misbehaving if my dev environment is going wonky? Obviously something's not working, how do I hot reload things?"

Those are all problems that build systems didn't really see in their domain.

So after I left Medium I said to myself... I knew a bunch of build systems people from my past, including my co founder, and we said that, "We're just going to start this as a kind of research project."

We know that with microservice apps and dev environments becoming composed of many services, could we create a build system that recognized that and acknowledged that?

And we did a bunch of experiments. Marc, I think I met you and Benjie both around this time, when we were still very experimental and we still had a lot of prototypes and Tilt was the last in a long line of build system experiments around microservices.

Benjie: So when you first sat down to get started on this build system, what were the main initial goals for the R&D project that you were trying to hit there?

What was the main pain that you guys were starting to focus on?

Nick: I think we talked to a lot of people and we saw everyone trying to figure out how to bring out multiple services in dev, but we weren't really sure what the shape of a solution would look like.

I think the first thing we really tried was some sort of remote build system where all your services were running in Kubernetes cluster and you would only build the services that you needed to rebuild.

But, if you didn't need to rebuild, those systems would leave the service running. We said, "Maybe the real thing to start is testing. Maybe you would need a remote test environment and a kind of interactive testing system that let you run the tests in the right order for the services you were working on right now."

But we weren't really sure, there was a lot of different experiments.

I think we ended up with Tilt in particular is we felt like we had spent a lot of time with the remote dev environments, and felt that it was going to be too hard to make that the starting point in how you try it and how you get it working, and so went back and said, "We think containers are the future, and we think Kubernetes is in the future."

And Kubernetes is a big, abstract category of how services talk to each other in a production environment.

So we said, "Okay, what if we could take some of these ideas and make them really easy to create a dev environment locally on top of containers in Kubernetes? And could we get that to work?"

That's the experiment that people really just seemed to love, and so we said, "Yeah, this is probably what we should be working on."

Benjie: So what year are we at right now? When was this?

Nick: I think we started, we started doing this as a research project around early 2017 and then I think we launched Tilt, oh boy, end of 2018 or maybe early 2019 so there was a pretty long experimentation process in there.

Marc: Yeah. I remember a quick story, you mentioned doing customer research validation, understanding where to dig in.

Your co-founder, Dan, was out at our office and we were struggling at the time with how to get a decent dev environment.

The complexity of our stack had grown, developer productivity was impacted, and there was a day, I remember this Dan walking around and just looking over the shoulder of an engineer and you could just see his jaw dropping, like, "Wow, that's what you're doing? No, there has to be a better solution here."

I mean, I'm sure you did that to a lot of different folks to get research and understand where are the pain points that we can actually solve?

One of my favorite stories is very early on we had a bunch of random experiments and prototypes, and we wanted to get feedback from people so we just--GopherCon New York, or GothamGo was going on. So we were like, "This is great."

What we do, is we just bought a booth at GothamGo and we went to 23rd Street and got some medical supply equipment so we could dress up like scientists and put a huge science fair poster board thing, with all of what we were doing and just grabbed people at GothamGo and showed them what we were doing and got their feedback on it.

I love that, the mad scientist approach to product development, of just, "Let's build something and put it front of people and see what they do with it."

And then if it doesn't work you can also just be like, "Yeah, we're a mad scientist, it's going to blow up most of the time. That's fine."

Benjie: The mad scientist excuse is one of the best excuses.

So you were doing all this R&D, you were doing all this research, you guys get to a problem that interests you, a problem that is a realm problem for everybody else.

Was that the moment you guys were like, "We're going to this as open source," or was there an earlier inflection point, or is that later on in the story?

How did open source become an avenue for the project that you're working on?

Nick: That's a really good question. So when we started, it was not at all open source.

We were doing more of like the remote build system, interactive CI days.

We were like, "This is probably going to throw away, we don't want anyone to depend on this, let's just do this in our private repo."

Once we started doing Tilt, I think I definitely come from a tradition of when you have stuff running on your own machine, it's much more important that that be open source as an exercise in both trust building and also as an exercise of, if it's misbehaving, you want to be able to dig in and understand how is this interacting badly with my machine?

Just client software in general, I think, is easier open source because you're deploying to an environment you don't really control.

And so the ability of the person who does actually control that environment to be able to dig into it to understand why this is breaking on their system was really important to us.

Also I think this like a cultural tradition from since maybe the late 80s, early 90s of just build tools being more open source and being more free for people to use.

So we were like, "Yeah, most people are going to expect build tools to be open source anyway. Anything that you want on your own machine should probably be open source. Let's just make this open source."

So when we started Tilt, it was open source while a lot of the other experiments were not.

Marc: So obviously it becomes a little bit table stakes, right?

You want to ship something around in a different environment, it needs to be open source just so that somebody can look at it, make sure it's not doing anything they don't want.

If they don't trust your build process, they can grab the code, look at it, they can build their own binary, they can do whatever they want to do.

But you ended up building a company around Tilt and actually working to monetize an open source project, was that part of the initial plan?

Or was there a lot of trying to juggle and figure out what belongs in the open source part, what belongs in the commercial product that we're building?

Nick: My answer to this maybe is an unsatisfying answer, but I think we had talked early on, we were still in the research phase, we had talked to VCs.

We told them, "We have this plan to monetize this." And they just laughed at us.

Most of the ones, like the DevTools VCs were just like, "Yeah, the first thing you need to do is get traction and the second thing you need to do... Don't think about monetization yet, because if you're experimenting a lot and you're also trying to build something valuable and also trying to think about how to monetize it, it's going to be too distracting. It's not going to work."

And I think that was generally true of this sort of research. Maybe if we had started with a very clear product idea.

I think a path that I think you see as increasingly common is you build some product inside a big company that gets a lot of traction, and then you're like, "Oh, let's build an open source version for the Marcet, and we have a pretty good theory of this is going to have traction and we know how to monetize it."

Where in our case we were like, "We don't even know what the product is going to be, we don't even know what parts people are going to want to pay for or not."

So we just didn't even think that hard about that at the early stages, I'm not sure if we've even really found a sustainable monetization strategy for this open source project yet.

Though, we can talk about monetization of open source too, if you really want.

But I'm not going to say I'm an expert on it, let's put it that way.

Benjie: Yeah, it's tricky. You were talking about that traction, separate from monetization and building a commercial business around.

So you started of as like an experiment, the science experiment, watching what people do, throwing ideas out there.

Can you talk about that moment when you realized, "We have a solution to a problem here, there is initial traction, let's stop searching and let's start building more on this one topic that we have"?

Nick: The company is called Windmill Engineering, by the way, even though the product is called Tilt, so if you hear me say those words that's what they mean.

In the early days of Windmill, I think mainly there were two blog posts that went viral.

There was one blog post that I literally wrote in a couple of hours in an afternoon, and it was basically reasons I disliked Bazel, and it was in a good, elbowy, jokey way.

I like Bazel people and I have a lot of respect for them, but it was like, "Yeah, these are all the problems I have with Bazel and why I don't think this is a good entry point for a microservice build system."

And that went viral and people had lots of comments on that, so I got a lot of feedback on that.

I was like, "Okay. That seems like a message that resonates with people, that these tools aren't really fitting what they need."

And I think we had a early blog post that was just... I forget what it was called.

It was local Kubernetes development, we had a blog post that did somewhat well.

We had one that was explaining all the ways that local Kubernetes development really sucked, and I think it might have been that Dan Miller wrote it.

But that just got massive amounts of traffic and massive amounts of comments from Hacker News and the various commenting platforms, and just going through and reading those comments really understand... Okay.

When you read those comments I always take them with a grain of salt. I don't necessarily treat all of them as truth at face value.

But I think when you get a big response to a post you hear what people are saying with it.

You know, "Okay, there's something here. Maybe we don't all agree on what it is but the problem space is interesting and the problem space is something that resonates with people."

Benjie: So really part of it was like, "Hey, we're working on this stuff, put stuff out into the world, and maybe we haven't figured out the answer to the problem."

But you had this feeling that people are thinking about this and maybe they're not telling us exactly what they're thinking, but that's how you got to this product Marcet fit, if you will, for Tilt itself.

One quick question for the rest of us, what is Bazel? I know you come from Google so it's pretty ubiquitous there, but tell us about Bazel and what is that?

Because I think that's interesting because you have some history there.

Nick: Boy, I'm just thinking if I can explain Bazel.

Every once in a while the Bazel people complain to me about this blog post.

But the idea at Google was to have this big... this cross language build system where, for every target, you have to define the inputs and you have to define the outputs.

The idea is that if you express your build process in that way, it no longer matters if the build process runs on your local machine or if it runs in the cloud somewhere, that because you've completely defined the inputs and outputs there's actually a...

It is really good at distributing and parallelizing your build process in a way that you can't necessarily do with Make.

Bazel came out of a system that was originally a bunch of makefile generation scripts at Google very early on, and over time became more and more solidified into a, "We're going to build a directive graph of all your sources and all your outputs, and figure out how to build things in the right order and figure out how to not build things when things aren't out of date."

So Bazel was originally called Blaze inside of Google, it got open sourced as Bazel.

But also in between that time, between when Google built it and when Google open sourced it, there were a lot of clones that other companies had started to build internally, from Buck at Facebook or Pants which was a kind of Twitter and Foursquare collaboration, and some other companies.

Though, I think Bazel seems to be becoming more popular in the last couple of years for companies that need a multi-language build system where you have a lot of dependencies between different languages that can't be expressed in the language's native build system.

I think Kubernetes tried to use Bazel for a while and found that just the Go build system was good enough for what they needed.

I think there's a whole discussion about Bazel's history with Kubernetes, which was interesting back then.

Marc: That story sounds like I'm drawing parallels to Google had Borg, and Facebook and Twitter and Square, they all had their own container orchestration systems, and then Borg is rewritten as Kubernetes as this open source thing.

It sounds like a playbook there that's working.

Nick: Maybe.

Marc: Cool. So that helps understand what Bazel is. We probably should've started with this, what's Tilt?

Nick: What is Tilt? So Tilt, we've been calling it Your Dev Environment As Code.

People use it in a lot of different configurations.

The main way we see teams using it is it's like you have a bunch of services that normally are running Kubernetes, and you want to be able to run them all together in dev.

Well, what you should do is you should set up, say, a Kind cluster or a Docker Desktop cluster locally and use Tilt to orchestrate the build and deploy and the visibility into what you're running.

That is, you tell Tilt, "This is where my Docker files live, this is where my Kubernetes manifest lives."

Tilt will figure out the right dependency order of things, and then every time you make a change Tilt will try to bring the dev environment up to date.

We have a lot of tools that say, "Hey, if a JavaScript file changed you can actually just hot reload that as the container directly. If, say, your dependencies change we're just going to rebuild the container from scratch and redeploy it."

And giving you visibility into how long that deploy is taking and when the service comes up you can interact with it, and giving you visibility into how the service is behaving and how it's talking to other services.

Marc: So is the value of that really when the stack gets to be so complicated that there's so many microservices?

Kubernetes is complex, you have all this and you end up with developers really spending more time fighting the environment or trying to find out like, "Something's not working, which pod do I look at? What's the current name of this?"

That's where Tilt really starts to shine?

Nick: Yeah. Actually the reason we thought this wouldn't work, and when we were in the experiment phase we were like, "Maybe the only people who will want this is people with 100 services."

There is an old thing that Alex Klemmer says, if you've ever met Alex, who's like, "Kubernetes competes with Bash. If you had a Bash script before, Kubernetes is your replacement for that Bash script."

And I actually feel that way about dev environments, that even we have some teams who use Tilt, we're like, "Even if they're only running three or four services locally, but maybe you don't want to write a Bash script and wire that all together and bring all those services up."

You'd have to write a Bash script that monitors, "Hey, did this service go down?"

Will you have restart it? Or if it's down, you have to surface that somewhere.

Those are all the things that Tilt tries to accomplish for you so that even if you only have three or four services, it's still good to know which services are healthy and which services need to be rebuilt.

Marc: You talked earlier about it was one step too far maybe initially to say, like, "Let's take the dev environment off of your laptop and move it into a remote server, a remote cluster."

Is that still really the belief of Tilt? Where it's like, "No, get Docker Desktop, get Kind, get whatever that is, run it all locally"?

Nick: There's always a trick to figure out where the puck is now versus where the puck is going, and I pretty strongly believe that your local laptop will always be some part of your dev environment, that there probably should be some things running locally.

I am not a person who believes that the future is IDEs in the browser for everyone.

I think IDEs in browser will work for some people, but I think there will always be some way you find the local laptop and the local file system in the loop here.

Where I think Tilt fits in is less as an opinion of, "You should run everything locally or you should run everything remotely."

Of like, "Okay, your laptop as a node and a distributed system."

What Tilt is trying to help make possible is that it doesn't actually matter anymore if things are running locally or if things are running remotely, and maybe to start everything's running locally.

Maybe to start you have a Kind cluster running your dozen services locally, but as your services becomes more complicated, rather than having to rewrite your entire app to run remotely you can just say, "Kubernetes actually gives me really good primitives already for moving services from one node to another so now I can just move things from my local laptop node to a node on a remote cluster, to a shared dev space."

Or to either a per user dev space in a Kubernetes cluster to a shared Kubernetes cluster that reads from the host's common services, that sort of thing.

So trying to blur the lines between what is local and what is remote.

Marc: That makes sense, there's common services, stacks outgrow laptops eventually. But you don't want to make the compromise.

For the record, I totally agree with you, there's value in running services for every developer in the cloud but I don't want to open up Chrome and have a web browser based IDE because there's just little trade offs that gnaw at you throughout the day when you're doing it that don't work as well as when you had a full IDE on your laptop.

Benjie: I'm going to jump in here and say two things.

One is I think that Sun Blade Systems or whatever they were called, probably aren't necessarily the future.

I think we tried this multiple times where we have terminals and then we have these big servers that we log into.

I do think a hybrid solution that Nick is talking about makes sense to me.

But the other thing, and an argument that I always make, is if I'm on a plane and I'm going to KubeCon in Valencia, I got eight hours.

That eight hours is the best eight hours I get ever to any type of development, so I like being able to run as much stuff as possible on my computer.

But then at the same time, sometimes that's not feasible. So I just want to go on the record as saying I completely agree with you, Nick, I think that a hybrid solution is there.

IDE in the web or not, I'm agnostic to that. I don't know what the answer is.

I feel like if I say anything other than Dim is the only way, I might get in trouble with my co founders so I'll be careful there.

But yeah, maybe we should get a little technical on how Tilt does some of the stuff you're talking about.

I guess let me give you a scenario and you tell me how Tilt could help me.

I've got a little company, it's called Airbnb. That's my company, I don't know if you guys heard about it, it's called Airbnb.

And I've got 150 microservices that are running at any given time, I want to do local development.

How could Tilt me as a local developer? Give me a pragmatic version of that and maybe we'll dive into technically how it does it as well.

Nick: Let me see if I can repeat that back to you, you have like 100 services that you need to run just to create a dev environment, just to see the app working, so to speak.

Is that the scenario you're describing?

Benjie: Yeah, I think that's a good one.

Nick: Yeah. I mean, whenever I talk to a company... Tilt--

I still believe we're very user research focused, we spend a lot of time talking to platform teams at companies, and from very small companies to much bigger companies who are struggling with these problems and understanding what is the right solution for them.

I usually tell people like, "Hey, if you have 100 services that can all fit on your laptop, that's the place to start."

Very early on we were like, "Everything should be containerized, everything should be running in Kubernetes."

And there were a lot of teams we found just having a lot of trouble with, "Now we have to containerize everything before we use Tilt, what about this one server? What about this webpack server that I really want to run locally?"

So we said, "Okay, maybe the solution here is you have some host processes running locally and some processes running in a cluster, and some process running in a local cluster, and some processes are in a remote cluster."

So I spent a lot of working with teams, trying to figure out what is the right configuration with them based on what their bottlenecks are?

Whether their bottlenecks are like how much CPU these services take versus what kind of data needs these services take, do they need to access a shared repository of test data somewhere?

All of those things and trying to figure out, "Okay, based on the space of constraints you have, these services should be running on the host, alas, these services should be running containers in Docker Desktop or whatever, these services should be running remotely, and here's how Tilt will set up a proxy to those remote services for you."

That's usually the kind of decision space I move through with people, and it tends to get very specific about what their actual services look like.

Marc: So if I'm new to Tilt and I have this problem, developers have various different types of configs, it's kind of like the Wild, Wild West among the different engineers on the team.

But we are deploying to Kubernetes so at least a chunk of our services are containerized, might be relying on infrastructure as a service for databases and queuing and stuff like this.

Do you have a happy path if you were to work with a customer like that fictitious Airbnb example from Benjie?

How do you recommend they actually tackle that problem, if it's intimidating to bring everything in initially?

Nick: What I usually say to people, "We don't want you to change how you run services for Tilt, necessarily. You have some way of running all your services now, we want to try to figure out how you define them in Tilt's... if you want to call it a build system or if you want to call it a dev orchestrator, I think some people call it. Whatever scripts you have, whatever Docker files you have, whatever Kubernetes manifest you have, let's figure out how to use that, let's get everything up and running."

Then, once that's up and running, let's figure out, "Okay, what's slow?"

Maybe building this particular containering image is slow and so we want to add a live update rule so we can hot rebuild that container, instead of rebuilding every time.

Maybe it's something else is slow and downloading something is slow, so let's optimize that part or parallelize that part.

I am a big fan of, in the realm of software development, "Let's get something working as soon as possible."

And seeing if you like it and seeing if this is the right paradigm for managing your services, and then trying to figure out how to optimize from there.

What's that old saying? That all complicated services evolved from very simple services?

You can't have a working complicated service that evolved from a non-working simple service?

Marc: It's true.

To the point you made also, seeing something written on a piece of paper or a Google Doc or something like this, you can understand how it's going to work but, wow, as soon as you get your hands on an early version of it and you're playing with it you start to really realize where the challenges are, where the compelling parts are that you start leaning on and relying on more and more.

So yeah, getting it in hands early is great.

Nick: I think one of the geniuses of Docker, and I remember a lot of people being very dismissive of Docker files very early, a lot of build systems people were very dismissive of Docker files early on because you just take a base image, you just add all your source files and you just run some shell commands and you get a Docker image.

I said, "Oh, that's brilliant. Optimize that later, let's get it working, let's go."

And not thinking too hard about trying to do things the optimal way from the start.

Marc: So how do you manage that in Tilt?

My dev environment does not match my production environment exactly, it's a lofty goal that I want it to match as closely as possible to reduce bugs, right?

But there's config changes, there's service changes, how do I declaratively hopefully, maybe imperatively, describe what those changes are that are different?

Nick: I think the answer to this depends a lot on the team, so I'll tell you what I do.

I'm a huge fan of Helm, I think Helm is brilliant, and usually when I'm moving onto a Kubernetes project and I want to figure out how to run it locally or how to run it at all, the first thing I do is put it on a Helm chart, define variables for the things that I want to change and then go through and make those variables injectable.

I think some people are big fans of customized, it's a little bit easier to get started with even if it's a little bit less structured.

I think some people literally just, with Tilt, they start Tilt, just put some expressions in or some shell scripts to replace the parts of the Kubernetes manifest to make it more dev friendly.

I think those are all fine. I think mainly the thing I try to do is get something deployed to a cluster.

When you have things, can you run things? Maybe I'll push the question back on you, Marc, and this is what I always do.

Can you bring up a new environment? Can you bring up your Kubernetes manifesto at some stage in the environment or some local Docker Desktop?

Or is it like you pray that when you deploy, it probably will work?

Marc: That's one area, it's a challenge, right?

The stack grows and you have it working locally and then I think you hit these thresholds, and I'm sure that this is something that you see, right?

Maybe I start off and I have a really easy POC, I'm deploying a next.js frontend to Vercel and I have like a Heroku backend or something like this.

I don't need the complexity of Kubernetes, but then I start to bring in additional services, different backends, microservices, and then I can, like, "Okay, great. Maybe Docker compose file works or a relatively simple Kubernetes."

Then you end up with going down this fork you described, Helm being one option, Customize being another option, and the complexity becomes more and more and more. Eventually you're ordering the latest M1X laptops with 64 gigs of RAM for every developer just praying that Apple will advance the hardware faster than your stack continues to grow and use that hardware.

Benjie: Well, let's not forget worry about ARM builds versus i86 with that as well.

I also have to state here, just for our audience, Marc is clearly in the Customize camp, I am in the Helm camp.

I like both, I think we both like both, but I just wanted the record to state that because I know how big a fan Marc is of Customize.

I'm going to interject and say that I think Docker Compose is very powerful for local development.

We all know I'm very biased about that. But one thing that I'm pushing personally is that I do think that we have too many YAML files in all of our repositories, and I think that one of the things that's maybe holding some adoption back is that it's a little overwhelming with all the different options that we have.

So over at Shipyard we always try and be a little prescriptive there, but more importantly I think that it's an open question, it's like, "What is the right answer for config management?"

There are some options out there, I think Starlark is something that is on your radar, Nick, or anything to that effect?

Nick: Well, actually I want to push back on you there, Benjie.

This is my big bugaboo, and actually I really do like Docker Compose, I really do like Kubernetes, but one of the brilliant things about Kubernetes is its object model.

The way it manages objects and the way it manages objects as separate things that have a stack which is in the desired state and status, which is like the actual state in that that is consistent throughout the system and that you can add new types of objects, and that all objects are interactable in the same way, and the others are separable.

I used to work with Vee Korbes who used to say, "What's nice about Kubernetes is that the English language, where you can compare... You have a bunch of words and you can combine them in interesting ways and you can make new expressions by combining the words in ways that you couldn't before, but you can separate things out a lot."

I think that way of handling objects is going to be for the future, that is how we're going to handle infrastructure objects.

It's clearly the way things should be done, it's actually very simple.

I think Kubernetes, if there's any complexity in Kubernetes, it's that pods as a base primitive are pretty complicated and some of the actual objects are complicated.

Probably more complicated than they need to be and I think that scares people off.

And I think in the future what I would like to see happen is deployed systems that use the same way that Kubernetes does for creating objects and manipulating in the system, and being able to have them work together but with much simpler objects.

Versus the Docker Compose way of the world, which I actually think Docker Compose in some ways is more complicated because you start with one file that defines the entire dev environment and then you provide overlays on top of it and it starts easier but becomes more complicated as your system becomes more complicated.

Rather than the Kubernetes way of, as it's more complicated, it's a little bit easier to swap things out, it's a little bit easier to treat them as separate objects.

I'm slightly trolling you here, Benjie, but I'm curious of what you think about that argument?

Benjie: No, I think it's a really good point and I think that the real answer to all of this is that we're still kind of early in this world.

So yeah, we're going to see what ends up winning out.

I think I will go on a limb here and say Kubernetes is probably the orchestrator of at least now and probably the indefinite future, and I think we're all fans of it.

So switching gears a little bit, let's talk about your roadmap. What's next for Tilt?

Nick: I want to think about this in two pieces.

One piece is Tilt as a platform and the other piece is Tilt as a set of features, and I think that will bridge what we were just talking about.

What you need from a dev environment is similar to what you need from Kubernetes, you need a consistent API to build on and a consistent way to interact with objects in your dev environment, in a similar way...

To me the genius of Kubernetes is it gives you a consistent way to interact with all the objects in your cloud environment, to define those objects to react to things happening in to them.

One of the things we've really done over the past year in Tilt, maybe about a year and a half, is when we've started building Tilt, we assumed it would look a lot like Bazel and we assumed it would be a graph, "What we're actually going to do is we're going to build a build graph."

And as we started building more of Tilt, we realized actually Kubernetes isn't a build graph, right?

Kubernetes is a reconciler, Kubernetes is a control loop that is constantly trying to bring the desired state into line with the actual state, and that actually Tilt shouldn't be a build graph at all. Tilt should be a reconciler, Tilt should be a control loop.

So we said, "Okay, how do we move this thing from a build graph to a control loop, but without having to build out all our own control loop infrastructure?"

And at this time it turns out that a lot of people in the Kubernetes community were also really interested in this problem and were also being like, "How can we take basically the API server of Kubernetes and make it generally reusable as a framework for managing any kind of infrastructure?"

Particularly Jason DeTiberus and some of the people working on the KCP Project.

So we said, "This is all actually great work, this seems to be where the Kubernetes community in general is moving. Let's actually rearchitect Tilt on top of the Kubernetes API server and rewrite every object, everything that Tilt does as an object, in the same way that everything that Kubernetes does is represented as an object with a control loop around it."

We are still maybe like 75% of the way through that, in that if you want to write servers that query the Tilt API server... If you run Tilt today locally, you're actually running a Kubernetes API server.

When you run the Tilt CLI, you're actually running a forked Kube pedal with some nice things on top of it, and you're interacting with the objects in your dev environment as you would if you were interacting with the objects in your Kubernetes cluster.

We're about 75% through that. I think you can interact with a lot of the port forwards as objects, as the logs as objects, as the Kubernetes deploys as objects themselves.

I think I have to move a couple of other things as objects. I think Docker images are now objects in that system.

They're just continuously rebuilding as they get changes. Live updates are objects in the system.

So we want to have more and more objects and then be able to have people build in the same way that Kubernetes is very pluggable and you can build things that react to those changes and do metrics and report them to other servers and react to them and add new functionality.

We want to have that same ability in Tilt.

So that's Tilt as a platform, maybe before I go into features, maybe I should stop there for a second because that's a little bit galaxy brained but also I think pretty important to how we're going to manage infrastructure going forward.

Marc: That's actually super cool. I had no idea.

I'm going to repeat back to make sure I understand, Tilt used to be like this open source project that you actually wrote and the functionality still exists but now instead of a very proprietary PI and stuff like this back and forth, actually the Tilt process is running the Kubernetes API server and the Tilt binary is running a fork of kube control.

So it's using more standardized protocols and also being able to leverage...

What are the benefits of that? Do you get to leverage others things in that Kubernetes API server that you would have had to build yourself otherwise?

Nick: Yeah. It just turned out that building a lot of the APIs, we scratched our heads and looked at this and said, "Oh yeah, building all these APIs for managing objects, both from an HEP endpoint and from a CLI was just going to be a lot of work for a very small team."

The company is eight people, right? We're not that big.

When we started this in 2019, when we launched Tilt in 2019, the API server wasn't there that you could do this.

But I think I would say maybe the late 2020-ish, early 2021, enough of the API server really got abstracted away and a lot of people put a lot of work into this.

Controller runtime, which is the framework for writing Kubernetes controllers has gotten really, really good in the last year or so.

If you haven't written a Kubernetes controller recently, that library has just gotten much, much better.

And so to say, "Actually these libraries are ready to use."

We can use this for building our dev environment management system, and so solely have been moving each part of the old graph based engine onto this new Kubernetes API server based engine.

Marc: Cool. There's a lot to think about there. So that's the first part, right?

The Tilt platform moving from the graph based to the reconciler pattern. Then what was the other side of the roadmap?

Nick: That's this all internal, that only matters if you're working on Tilt or if you're trying to hack new features into Tilt, which is important.

It is important for how people debug Tilt platforms, and it's usually we do need to debug Tilt in some way.

As far as from a feature roadmap, we really wanted to make it easy to be able to load all the services you need to use in Tilt and switch between them so we've really tried to make Tilt more of a kind of resource catalog.

We have teams who are using Tilt with like 80 services or 100s of services, but don't necessarily need to run them all at once.

And so trying to figure out what are the right UI paradigms?

What are the flows that you need for, like, "I want to run this set of services when I'm around this feature and this set of services when I'm running another feature, and I want to spin up this service when I'm doing integration tests."

Tilt is kind of a management platform for the services you need in dev, is one big piece of what we're working on.

Kind of related to that, there is increasingly interest in how do you make Tilt working really well, when different servers are running in different places?

Whether it's running a local cluster or it's your dead services are running in multiple clusters, whether you have some services in Docker Compose and some services in Kubernetes.

There's been a lot of really good work in the last couple of years into how to connect different Kubernetes clusters together and how to do network tunnels between your local machine and a remote cluster.

There's obviously Telepresence from Datawire who have really pioneered all our really cool approaches to this.

There is a project called Ktunnel, which is about creating reverse proxies between a cluster and your local machine.

There is a bunch of projects around creating tunnels between Kubernetes clusters.

Trying to make all those things work well in dev so you really can just, at a flip of a button, move a service from local to a remote cluster when you're a little bit overloaded, I think we have a bunch of ideas around that and how to make that easier or how to make that more seamless so you don't have to really care as much anymore which control plane you're running on.

Then the last big bucket of features is better support for IDEs. Right now Tilt has its own.

We strongly believe that one of the nice things about Kubernetes, is that it has a very nice, consistent API and so you can expose it to a CLI or expose it in a web dashboard or expose it to an IDE.

And we feel the same way about Tilt, that it's going to be available in different forms from the CLI to our own proprietary web UI, versus in your IDE as like a BS code extension or something like that.

So we're experimenting a lot with how it looks from an IDE standpoint and different ways you interact with that interface.

Marc: So on that, the IDE, my IDE is not going to limit me from using Tilt today, it's just you're working to make it more native into some of the popular IDEs. Is that the direction?

Nick: Exactly.

Marc: So shifting again, you made Tilt open source, you got a pretty good amount of traction.

It seems like it's good stars, good activity in the repo.

The premise of this podcast often is to talk about CNCF projects, sandbox, incubating and graduated projects.

We don't limit it specifically to that. Tilt's not a CNCF project right now.

Have you ever had any conversations about that? I'd just love to hear more about the thought process inside the company.

Nick: I have not, we have not really talked about CNCF or foundations in general.

There's a whole space of how open source is run and governed. We are certainly nowhere near an open governance model right now.

Tilt is very much run by a single company. We have partner teams who use it and who contribute to it and that give us feedback and give us PIs.

But certainly we are not at the point yet where we move to an open governance model. I don't know.

I don't actually know what the trade offs are there, and before I would, I try not to opine too much on things I don't know too much about so I'll leave it there.

Marc: That's fair. Has the governance question come up from anybody who's been evaluating it, where they've pushed back on the single company maintaining it and wished for more of like an open governance?

Nick: Not currently. I don't think anyone's ever brought it up at this point.

Marc: Cool. So the next step then in there is really just the community, it is an open source project.

CNCF projects have a little bit of a... I don't really know, it's not really super prescriptive, but at least a common pattern of how community engagement happens for different projects and what expectations are there.

Eight people is a pretty small organization to be maintaining a popular open source project, also engaging with the community, also building feature requests and bug reports and trying to run a business.

How do you balance the community engagement and what efforts do you put into that as an organization?

Nick: One correction, we are nine people and someone's going to be mad at me.

They'll think I forgot them. Okay. How do we handle engagement?

I'm a pretty big believer the way to build a good dev tool is just to have everyone talking to users.

Some people enjoy that more than others. We currently have a Tilt Slack channel in the Kubernetes Slack community, and that's always been...

Just in general we've always felt very connected to the Kubernetes community because the Kubernetes community is awesome, just incredibly welcoming people and people who are very supportive of solving these sorts of problems.

Aside from just my things, some of the technical things I really admire at Kubernetes.

But I would say that we try to bring that sensibility of being welcoming and supportive to everyone who tries to interact with the Tilt community, making sure that we do have a rotation of users that support people every week, making sure that they're keeping track of the community and making sure that people are getting their questions answered and that sort of thing.

We do have a full time developer advocate who thinks much more the high level about how we build community and the structural things we should be doing to meet people where they are.

Though, I still think that this is all very early days, so to speak.

Benjie: So I'm a developer and I'm using Tilt, and I really want feature X, let's call it Tilt Attach.

I want this Tilt Attach feature that I've talked to Nick about in the past, but I want to build a feature that I can attach my Tilt stuff to a running Kubernetes cluster.

How can I actually contribute? What's the best way for me to go help and contribute? Obviously I can open a PR.

Do you guys have meetings? Is there anything like that? Or what's the best way to contribute?

Nick: So there is a couple of different levels of contribution right now.

One thing that we've tried to do in line with Kubernetes CRDs, we've tried to make it really easy to write Tilt extensions and to either maintain your own repository of Tilt extensions for your own team and then to contribute those to the common extension Tilt repositories upstream.

So we do have a separate repo aside from Tilt that is just snippets of, like, "Hey, this is how I register my server with Tilt. This is how I do something new with Tilt that you couldn't do before."

And those tend to be contributed by a lot of different people from a lot of different teams.

There actually is a Tilt Attach extension which I wrote one afternoon, just as like an experiment, and it's good, it works.

It should work pretty well, I don't think it has all the bells and whistles that you want, Benjie. But it does exist.

When we have a more complicated feature request, usually when something is built into Tilt's core, it's usually because it is complicated enough that either it's a very simple thing and people just submit PRs for like, "Hey, what up? Someone fixed a coloring bug last week where it was showing the wrong colors." That sort of thing.

But for the more complicated features we're like, "Hey, this is actually impossible to implement this extension."

They usually file an issue in the GitHub repo and we actually often have a pretty long discussion about... I think this was always the difficulty with any kind of developer tool or infrastructure project, is what is the right level of the solution?

Is it, A, it can have an API endpoint so that you can do this from your Tilt file, which is how you configure a dev environment?

Is it something built into Tilt? If it's built into Tilt, how opinionated does it need to be? How many primaries does it need to take?

Those sorts of things. Does it need new configuration functions or should we just do something we do by default?

Usually it tends to be a pretty involved discussion, we try to have those sorts of discussions on the public GitHub issue tracker.

And sometimes we'll have like 10 different people file bugs in the issue tracker and we'll all say, "Actually we can solve all these issues with one solution."

Because they're all trying to suggest different ways of solving this problem, and maybe you just need an API rather than a baked in feature.

Those discussions are always very tricky, but we try to have those sorts of discussions in the open.

Benjie: So GitHub is really the place to be?

Nick: Yes.

Benjie: In and around GitHub, PRs, issues. Do you have any meetings, weekly, monthly, quarterly meetings at all? Anything that you--

Nick: We don't have any currently, meetings. We sometimes run them ad hoc, we've run some office hours but nothing regularly scheduled.

Benjie: Okay. So basically go to Tilt.dev, check out the blog, keep an eye on the GitHub repository, and that's the way to get involved in the project itself.

On that note, is there anything that the Tilt OpenSource project is looking for? From new cases, engagement.

Any ask to The Kubelist audience?

Nick: That is a good question. Actually, I think there's a lot of problems that Tilt needs to solve that we actually don't really want to be solving.

I'll give you maybe some examples, like examples around how a controller runtime works or how the Kubernetes part of Client Go-work.

I think we've talked a few times. There's a bunch of other tools in this space.

One of the things that we collaborated a bunch on with a lot of the cluster operators, one of the big problems that people had when they started using Tilt... Understand I'm giving you the illustrated example as it's a little bit hard to get, I can give you a concrete case.

Running a cluster locally with a registry was really a pain for a long time and particularly the way it was a pain was that every cluster, whether it was Kind or it was MicroK8s, or whether it was K3D, or whether it was minikube, all had their own ways of doing it. Then once you've set it up, you had to configure all your tools to know where that registry lived.

On one side you have four different local cluster solutions, in this example, or five if you count the Docker Desktop in there, and maybe six now you have Rancher Desktop, and on the other side you have all these tools that need to know this is where my cluster lives, this is where my registry lives.

Why does every DevTool on the other side have to know about every cluster's way of configuring things on the other side?

So we spent a lot of time working with the Kubernetes community on a standard way to document in the Kubernetes cluster, "This is where the registry lives."

Once we had set that up, it was like, "Okay, you bring up Tilt, Tilt can read where the registry lives off the cluster."

So we coordinated with all those different teams so that they all, K3D and Kind and MicroK8s, all did it the same way and that really helped a lot.

It just made it way easier to setup Tilt, way easier to set up a like local Kubernetes dev environment.

I'm telling that story because I feel like there's a lot of things where there is just like, "Hey, how do you express the live update rules on a Docker file? How do you express ways to connect a Docker file to a Kubernetes manifest?"

Those sorts of things, those kind of interrupt things are things that I wish the Kubernetes community paid more attention to.

Not just like, "Hey, user. Go figure out how to configure these things."

But Kubernetes is a great system for how to manage in for a configuration, can we come up with some standards for standard ways that tools can configure each other and document their configuration for other tools to read.

That's coming from as a Kubernetes community as a whole.

I would love to see more collaboration along those lines, and I know that's very abstract and in some ways it's not technically interesting.

It's more of like a human problem rather than a technical problem, but I do think that work is often the most high leverage work because often it means that people who are new to the community are trying to set up these dev environments for the first time or trying to set up Kubernetes for the first time don't have to think about how these tools fit together.

Marc: Yeah, and you rely less then, Tilt or other tools, rely less on having to have their own implementations of all these different things.

Going back to the example and the story that you talked about where these four or five, six different dev clusters can identify where their registry is, it's a great story.

Honestly, I didn't even know that feature existed in those right now, so that sounds like a great thing. Where does it start?

I actually want to know. It's a config map, it's a config map in the Kubernetes public named space that just says, "This is where the registry lives."

That's all it is. It's not that technically interesting, but getting everyone on the same page was a lot of fun and took a lot of talking to people.

The best features are the ones you don't know about, the ones that they're like, "I actually don't have to figure this out myself." Yeah, that's cool.

Benjie: Look, it sounds like you in particular, but Tilt as a company is very invested in cloud native and, while you're not a CNCF project yet, maybe one day that's on the horizon.

Contributing the greater ecosystem is something that you guys are doing and to be helpful for Tilt itself, and how does Tilt fit into that system.

So that's really cool. I think we're coming up on time. I did have a little question for you.

Throughout the process of building out Tilt, is there anybody you want to call out, not to put you on the spot, but that has been really helpful in the community or internally or anything like that?

Nick: Boy. I just want to maybe thank all the Kubernetes projects that we built.

We try to use so much of the Kubernetes ecosystem, from the Kind Project which has been particularly helpful, to helping us sight the bug things.

I know there's been a ton of work to move Kubernetes from a mono repo to individual repos that you could import individually and use to interactive with Kubernetes, and that has been an enormous help to us.

Yeah, I mean this is just a love letter to the Kubernetes community at large.

Marc: All right, Nick. It's been a really fun conversation, talking about how you actually view the dev environment and the changes from the built environment, and what's created Tilt.

Nick: It was great talking with you all. Maybe I'll see you at the next KubeCon.