Ep. #121, Reliable Serverless Functions with Tony Holdstock-Brown of Inngest

Brian Douglas: Welcome to another installment of Jamstack Radio. On the line we've got Tony Holdstock-Brown. Tony, hello. How are you doing?

Tony Holdstock-Brown: Hey, I'm very good. Thank you. Thanks for having me.

Brian: Yeah, a pleasure. We crossed paths on a Slack and then we met in person which every now and then people catch me from another thing, I guess in San Francisco it's probably way more common. But yeah, you were like, "Hey, we're in a Slack together." And I was like, "Oh yeah." And then we started talking about your product and I was like, "Oh, you should come and talk on Jamstack Radio." So want to talk about Inngest, but also first we'll start with who is Tony? How did you get here?

Tony: Cool. Yeah. It's good to cross paths with you. I feel like as soon as I saw you, I knew you. Which is interesting because of the stuff that you'd done, so super fun. How did I get here? It's sort of by chance, I never anticipated moving to America, kind of fell into it, but was always in tech.

I taught myself how to program when I was a little kid, back in the day when you would download music on Napster and what not? Yeah, downloading music, Boyz II Men albums, and then I saw a book on programming and I was like, "This seems cool," and downloaded it.

Brian: So you went from pirating music to learning programming through pirated books?

Tony: Yeah, yeah. Exactly, exactly. Yeah, I taught myself how to program when I was a little kid, got really interested in tech and computers and found my way through that as a teenager, to getting a job in my teens and being interested in startups ever since. So it makes sense that I'm here, I guess, but yeah, I fell into it.

Brian: Yeah. I didn't know this about you, but I saw you did a tour at Docker as well. How did you get at Docker?

Tony: Yeah. So back when I was living in New York, we were using Vagrant for a bunch of our stuff and Docker had just come about. It was all over Hacker News, I think they'd just raised their Series B or something, it's like 2014. So Docker had just come about and it was all over Hacker News, all over these blogs, and it was talking about you can write once, deploy anywhere, like a better Vagrant.

So I started messing around with it locally so we could improve our dev flow at the company that I was at, and instantly fell in love with the tool. You know how a developer tool comes along one time and you're like, "This makes my life so much easier"? It was exactly that process, and I knew that I wanted to work there. So I was doing something on the side already for a different company, and working at a startup, but figured that Docker was the future, as it were.

I ended up just straight up cold calling somewhere there that worked there, being like, "I really want to work here. I've written a blog post on you, and can I interview?" So yeah, I ended up flying out to San Francisco, interviewed and got there. It was around pre-Qube so, what's that? 2015. Yeah. There were fewer than 100 people, but still relatively big at the time.

Brian: Yeah. That's amazing too, as well, because I crossed paths with Docker as soon as I started writing code professionally, or right before. I had met someone who worked for Docker remotely in my home town and he spoke at a local meetup and was like, "Oh yeah, Docker. It's the greatest thing ever, you can do this, this and this." And at that time I had just learned how to deploy to Heroku with Ruby On Rails.

Tony: Yeah, yeah. Give push.

Brian: Yeah. Prior to that I was basically just doing HTML and CSS and JavaScript, so I didn't really connect. But then around the time you joined Docker, I actually joined Netlify and then I knew exactly what Docker was.

Tony: Amazing. Dude, there is so much to talk about your Netlify journey, because obviously being Jamstack Radio, Netlify had a big deal in making Jamstack, they came out with the name. This is super interesting because I think before then, developer experience hadn't been something that people cared about.

If you look at how clouds work now at AWS, it's infrastructure, you have to figure it all out yourself, you have to do the monitoring yourself, you write Terraform because the UI sucks and the experience is not very good. It's just provisioned in for it. Netlify came along, changed this entire thing, and it's super interesting that you were there in 2015 because that was almost the start of the developer cloud journey.

Brian: Yeah. It's almost like you can go to this podcast and find the first episodes back in... I guess by the time we shipped it was late 2015, and definitely check it out. I was in the inception, the conversations in the backroom around what to name this thing and what to work on, and the deploy previews.

I was actually just talking to somebody about deploy previews and how that came about, on a Twitter space yesterday actually, of all places. But this actually a perfect segue because what you're working on today, which is Inngest, so what is Inngest? What's the problem you're solving?

Tony: Yeah, for sure. So Inngest is a platform that gives your function superpowers. So Jamstack is great, you build APIs and the APIs communicate with your client, in React or View or what have you. Serverless functions are really good for HTTP APIs. They're maybe not so good for business logic. They're maybe not so good for business processes.

There's this big push to the edge, and that's all about reducing latency, making your users happy because your website is faster. But if you're pushing stuff to the edge, you want your latency to go down. You want your APIs to be fast, and that means you need to push work the background. How can you do reliable background work on serverless functions? It's kind of an unsolved problem.

You need to tie in different things like SQS and Lambda, you need to have this tiered architecture where you have database state or queues, and really, really difficult to set up. And so for quite some time, Jamstack has been about HTTP APIs and less building your entire business in a serverless manner. We give you all of that out of the box so with one line of code you can get retries, reliable functions, scheduled functions, event driven functions, any provider.

Brian: Very cool, yeah. I mean, around the same time that we were both working at Docker and Netlify, this serverless infrastructure thing, serverless.com came out and I think it really kind of solidified this. But also Serverless Days. There was so much serverless, it was almost too much serverless. It was almost a meme at a certain point.

But it was also true, building infrastructure around a bunch of Lambda functions or a bunch of serverless infrastructure, it's a painful thing to manage. Especially when you take a step back, I think was it last year? Everyone got the notification, if you had an old running node serverless function, you needed to update it and in my mind I'm like, "Oh man, I've got hundreds of these things.

I'm just building projects left and right. I guess these things are now going to be sunset because I don't want to update all these things." But that's how it was, they were a dime a dozen, it was like, "Oh, I can do compute without paying out of pocket?" My AWS bill has been like 35 cents for the longest time.

Tony: Right, right. It makes a lot of sense. It's a super nice way to do things in theory, but a classic example that I think about is something as basic as signup. Cool, you can use Clerk, you can use Descope, you can use what have you for auth. But you still need to do a bunch of stuff in the background. A user signs up, there are like five things that happen, send them a welcome email, you might want to add them to Zendesk.

Like any provider, Stripe. All of this stuff needs to happen in the background immediately, and do you do that in your APIs? No, you want this to be reliable, you don't want any of them to fail. So serverless is great, managing all of this is a real problem, and there are different frameworks like serverless framework is great. Different stuff to manage this. But there's no answer, there's no answer for each platform.

You can't do this necessarily on Netlify, you can't do this easily on Vercel or CloudFlare Workers. So there needs to be something that fills this gap and gives you retries automatically on your functions. It also ties in nicely with events, because stuff like this is cool, if you have a user signup event, propagate that out to any function that listens, similar to how you do it on SNS-SQS, and then have all your functions automatically be run with retries, but with no setup.

It's nice to get that where your project is currently hosted. I think this is the biggest thing with Netlify, Netlify has a lot in once space. Deploy previews, they've got the edge, they've got different background functions. It's really nice if you're deploying to Netlify, you're deploying to Vercel, to just keep that process the same and have all of this stuff in one place. That's kind of the ultimate dream.

Brian: Yeah, and speaking of that dream, as a Netlify stockholder. I'm curious what your take is. There's AWS, which were also glowing, talking about Netlify's developer experience. Do you think Netlify gets to the point where it becomes as big as AWS, but with just a better experience?

Tony: This is super interesting.

Brian: This is a $10 billion question.

Tony: Right, right. Exactly. It reminds me of the classic business thing, which is there's two ways of making money, bundling and unbundling, and there's AWS which is the bundled everything, there's Netlify which is the unbundled cloud front but done in a really precise manner which is super valuable. Then you start laying on different things like serverless functions. I think it's really interesting, and there's definitely a lot of room to grow.

A question is, what happens with developer clouds overall? In my mind, the rise of the developer cloud is super interesting, with Planet Scale, Netlify, Vercel, Neon if you're looking at PostgreSQL. What happens in the future with all these platforms? How do they interplay? Because if Netlify were to grow, then there are a few things that a full stack app needs, which is... I know you did a show on Redwood and Redwood kind of covered this in some way as well.

You need auth, you need data, you need a place for background jobs/queues, and you need a place for hosting CDNs and functions at the edge. Right now, AWS can do them if you don't mind the experience and you really like Terraform, which I don't know many people that love Terraform, but it's a thing. Or you can go on Netlify and have GitHub PRs which automatically populate into different preview environments.

And so I'm wondering how all of these different things work with each other in the future. Planet Scale branches that work with Netlify PRs that work with Inngest background functions. It's going to be interesting to see how this evolves and whether or not AWS embraces this or whether or not we all overtake. Yeah, interesting future because developer experience really matters.

Brian: Yeah. It is very interesting too as well, because I've chatted with quite a few infrastructure, up and coming infrastructure, projects. Clerk, I definitely want to chat with them. I run into them all the time with the founder in San Francisco at random events, but I've failed to invite him or his team on the podcast. So I'll get to Clerk eventually. But no, it's true, the bundling, unbundling, we also just talked to Pantheon recently.

Pantheon is coming out of the Web 2.0 era, trying to get people up to scale and originally deploying WordPress apps, but now doing so much more. Also Google Cloud, they are also very clear about the stuff that they acquired. They still have their internal brands for that as well. They're all bundling the Google Cloud platform and building that behemoth.

But Netlify is doing the same thing with the Jamstack Innovation Fund and all the new integrations that you can do on Netlify, which I think we're all benefiting from it. Me, personally. We're solving problems, that problem where you talk about the welcome email, we're using Superbase for our auth today.

Superbase is also our data host, and then to be able to send a notification of like, "Hey, something happened." Or even beautiful, realtime updates, like, "Here's a new notification of, hey, something happened in the app or you got a new message which we haven't shipped yet."

But we're working on this where you can send, "Someone requested to work with you on Open Sauced. Here's a realtime notification, here's a realtime response back." That's all stuff that, honestly, not knowing Docker very well until Netlify, I wouldn't even know where to approach that and now I can just be like, "Hey, is this the feature at this product? Or is this a feature that Netlify integrates pretty well?"

And you're right on the money where Vercel, Netlify, I know Railway is also up and coming as well, Render. They're all building these nice... I don't know if they're walled gardens, they're still pretty open. But they're building these nice platforms for developers who maybe can ship a quick React app, can also ship Chron to send notifications and et cetera.

Tony: Yeah, right. I think this is the most important thing in the last 10 years in the developer space. I don't want to be an expert, I don't want to be a PHD in learning how to do Kafka topics, back off idempotency retries. We don't want to do that with databases, we don't want to learn how PostgreSQL Wow works and we don't want to have to setup replication and failover. We just want to know, "I need this feature, what's the best way to implement it? And has a lot of the details been taken care of for me?"

Because ultimately, building a product now is way more complex than building a product 10 years ago. The table stakes have shifted higher and the stuff that we have to do to solve someone's problem is 10 fold. A product released now has to have notifications, like you were saying, it has to have a decent onboarding experience, the UI has to be fancy.

Thank you, Tailwind. And so we have to do a lot more work to get a viable MVP up, and we don't want to sit around dealing with infrastructure for four months just to build something basic, like, when this happens; do that. That's kind of crazy. So I think people learning that and the developer experience is so fundamental to getting things right, that's it's opened the door to a lot more engineers and what we can build. It's going to be an interesting future.

Brian: Yeah, it's true. So I make this correlation a lot, if anybody has heard me pitch OpenSauced, apologies, I pitch it a lot. But I make this correlation to what we're building, which is the money ball for engineers, where if you're looking for the best talent and best skills, so if someone's contributed to one of these open source projects. You might want to hire them to work internally on one of your features.

But what I'm getting at is now you don't have to be... I guess on the other end of this it's like now we have all this, it's way more competitive if you want to go play pro sports because now you have kids who have trainers who are all the former pro athletes, training them. You're seeing this weird shift.

I'm a big NBA fan. Every NBA player that's come into the league in the last 10 years actually has gone through the process of having specialized training, whereas 20 years ago me and you were growing up, it was just like, "Oh, you just have to spend way too much time on the court and get a lot of practice and you became good."

And that was how you made it. You now have to do way more. You have to get your parents to pay for specialized training, you have to go do club sports. With engineering, it's like that but also not that, where you come out the gate of a bootcamp and just know Heroku or just know Netlify really, and still be dangerous and be able to ship stuff within the first couple weeks.

Where before it was like if you just spend enough time reading Hacker News and buying the right books and writing the right code at the right time. Now it's almost impossible to stay up to date on everything, so if you're a senior Tailwind engineer, there's a place for you. There's definitely a company out there that's like, "Yeah, we just need someone to know Tailwind really good."

Tony: Yeah, exactly. That's the thing. It's a really good thing that people can come out of just learning CS, just learning the fundamentals, and be able to ship. I think that's a testament to the things that you did at Netlify seven, eight years ago now that Netlify has done, that other companies like Vercel, Planet Scale and Neon are doing, and that we're doing for events and durable functions.

I think it's a challenge that the table stakes have gotten so high, that realistically people are so used to tech nowadays that they expect world class experiences. When, if you look at Twitter back in 2008, the experience was less than good. Bootstrap was a big deal.

Brian: Well, it was text messages at first.

Tony: Yeah, crazy. Crazy. It's an interesting time, and there's more ahead so I think that if we want to make it as easy as possible for engineers to do what they need to do, then we need to focus on the developer experience. And so that's one of our foundational things at Inngest, how fast can you learn us? How fast can you get setup writing durable functions?

If you've never written a serverless function before and you need to do retries, let's say for example someone was using us for OpenAI, GPT a couple days back. Let's say you need to chunk an OpenAI request into four chunks, you want each four chunks to succeed and then after you're done chunking and summarizing text, you want to convert the summary into one whole. Summarize the summary, as it were.

That's quite a pain for engineers to do because each four jobs needs to be built, needs to be enqueued, needs to be durable and reliable. Then you need to know when all four jobs are done so you can summarize the summary. How fast can you do that? Turns out, if you focus on developer experience and you make this really nice, you can maybe do that in 10-15 minutes.

Whereas realistically, if you were looking at SQS, it might take you a few days to set up the queue, to setup Lambda to do the end to end testing. And so we're going to have to do more as a practice, as a profession, and we need tools that enable us to work faster.

Brian: 100%. I wanted to ask, because we spent some time talking about the higher level and the problem, where the state of the ecosystem is, but what was the story that got you into building this and solving this problem?

Tony: Yeah. So I'm a huge nerd, as previously explained, by downloading books when I was a kid. Infrastructure, architecture is really interesting. I ran engineering at a healthcare company for five years prior to doing Inngest, and healthcare, it's complex. Events are really great in healthcare because they're immutable facts about what happens in your system.

Someone signs up, someone books an appointment, someone speaks to a doctor, the event goes through your system, it's a fact a saying, "This happened." And it serves as a perfect audit trail. Healthcare is also complex, you need to do a lot of tedious user journeys and business practices. It kind of makes sense to make an event driven system because then you can hook into if a patient communicates with a doctor and the doctor doesn't respond within 24 hours, or if a patient books an appointment and they've previously complained they might be at risk.

So a lot of these things, complex state event driven systems, systems that we needed to build but it's just too hard for engineers. Kafka is great, even if you use NATs, NSQ, they're kind of fungible, it doesn't matter, the infrastructure itself is commoditized. Still building out event driven systems is really hard. You need to build the subscribers, the workers, you need to handle dead letter queues, backoffs, idempotency.

It's very, very challenging and your code can often spiral. So you spend three months implementing the platform before you actually build what you needed to build, and that's no good for anyone. That makes CEOs annoyed. And so there needs to be a better way to do this, because fundamentally it's the right way to do things. You read Brian Fowler's Patterns and System Architecture docs. You read data driven applications.

It's the right way to do things, but it's too complex for the average engineer, so instantly in my mind I was like, "There needs to be a better way to do things because every company that you use, already uses events." You might use CustomerIO, everyone has probably used Segment or heard of it. You use things like Amplitude for your product analytics, and it's all events.

When a user does this, I want to track it. As engineers, why is it so hard for us to leverage events in the same way? We have to mess with this infrastructure to get it setup. So the intersection of the things that you did at Netlify, that Netlify and Vercel are doing with developer experience and serverless, the developer cloud is really interesting because this doesn't exist for events.

And so really wanted to make it easy and show people the power of events, that's really the mission. How can you make event driven programming accessible to everybody? How can you make reliable functions accessible to everybody? Yeah, that's where it came from.

Brian: Very cool. So where in the trajectory are you guys? You've got a product that's live and ready to go. How big is the team and what are you guys working on today and next?

Tony: Dude, wild. The team is only four. We've rebuilt a wild queuing system that's based off this paper from Apple that powers every phone's queues. The product works end to end, we've got the SDK out. We hired this guy, Jack, who's fantastic at TypeScript, a designer, Ed, who's designed our website and is redesigning our product. Then our co founder, Dan, who is also incredible.

The product works. People are using it end to end to build out crazy systems on Vercel, crazy systems on Netlify. We kind of work anywhere, actually. We deploy to any cloud, we work via HTTP so you can use our SDK, deploy your functions as you normally do and get queues, background jobs, scheduled functions, reliable functions, step functions on any platform that you're currently using. People are building wild stuff.

Yeah, someone was doing ChatGPT, OpenAI. It's kind of annoying because these functions have got a 10 second timeout, but we do step functions that can be parallel. So this guy chunks his input from one event to like 10 comparable API calls. We invoke that HTTP function 10 times in the background with memoir's data, and so each API call runs in its own 10 second time limit.

Then once those steps are complete, the function progresses, runs another step to summarize everything, which has its own 10 second time limit. The product is just powered end to end with the platforms that you have used for eight years, without changing a thing. Bypassing all sorts of time limits, and if OpenAI goes down, which it goes down a lot, it will just retry automatically. You don't need to build any new queues.

Brian: That's amazing.

Tony: We power a bunch. But we've been relatively quiet because, again, the developer experience is so important that we want it to be really smooth. We've got Inngest Dev, runs locally, uses the same executor that you use in the cloud so you can test everything end to end locally but with a Yarn dev, and the entire process for us is so important that, honestly, you're the first person that we're really telling outside.

Brian: Oh yeah? This is the first podcast with Inngest on it?

Tony: Yeah, yeah. We've spoken to Joel Egghead, we're partnering with Redwood to do a bunch of their stuff like serverless functions, realtime GraphQL subscriptions. They're amazing people, but a lot of that stuff hasn't really made it out to the world yet because we're just working on it. So it's pretty early, despite people using it for real things and businesses using it for real things.

Brian: Yeah. I saw the first release was like summer of 2021, so about a year after you've been working on this?

Tony: Yeah. And it started very different. The workflow engine is the same. But I came from Docker, it was all Docker based, it wasn't SDK based. It was very hard to use. We realized that the intersection of serverless functions and events is really difficult. There's no way to setup a subscriber with a serverless function because it's stateful, so we need to call you and the SDK was the way forward.

So we built that in September and released it at the end of September and it's so much better to use. It's so good. So yeah, it's changed a bit fundamentally, the architecture, the system is the same. Could talk about it for a while. The SDK does some fancy things. Previously we had a dag of steps, now your functions, your serverless functions that you deploy to any platform like CloudFlare, kind of act as a generator.

So we invoke it one time, it tells us the step it wants to run, we'll call it a second time with the cache and memoir's data, and it will say, "Hey, I've ran this step with this data." Then we call it again with the cache and memoir's data, which is how we bypass time limits. You can do crazy things like actual node parallelism that you could never do before. So yeah, it's a lot of changes in the user experience. Very few changes in the product.

Brian: Okay. Excellent. That's a testament to how y'all build and figuring it out, and getting the developer experience, definitely something that takes time to get right, so appreciate you coming out to chat with me about your product on the podcast. I'm actually really intrigued to leverage this. I have a couple other questions, but wanted to check in and see if there's anything you wanted to mention first before I jump into asking more technical questions.

Tony: No, no. I was really interested in just speaking about the developer experience, because I think you were one of the fundamental changes in that, from my understanding early on, at Netlify. So it's always interesting to talk to people. You were ahead of the game with this, like eight years ago. So yeah, super interesting.

Brian: Yeah. At Netlify we worked on... my first thing was converting to React. We just knew there was something there to leverage React to build out the entire admin dashboard. But the second thing I did was we helped build... Well, we made it easier to purchase domains and put domains on Netlify. But the thing that I was a part of that was early was deploy preview.

The deploy preview was the thing, the catalyst of how we got really ingrained in a React ecosystem. We shipped deploy previews the summer I joined, create React Dep also shipped that same summer so I feel like if you look for the deploy a React app at 30 seconds, that title, that was me. It's been manipulated and updated since by other Netlify employees, but yeah, deploy a React app at 30 seconds because create React app was so...

It was just bundled in Web Pack, they bundled all that stuff together which now people are like, "I don't know if I like this anymore." But it served a purpose, and that was a post that we shipped to show that you could also deploy that to Netlify, despite typing Netlify deploy. Our focus wasn't CLI because at that point, Now, that was the premiere CLI tool that did it, and Surge was the other one that hasn't really got a lot of contributions to it.

But it was just really my whole thing was if you could sell a car by getting someone in the driver seat as fast as possible, then they're like, "Oh yeah, I want to buy this car." It's the same thing with Netlify, it's the same thing with any developer tool. If your first introduction, this is my gripe with a lot of Web3 companies, about four or five years ago it was like, "You got to create a wallet."

It was like, "Why would I create a wallet when I just want to deploy a site? I want to build an app." "Okay, create the wallet." Now you have to spend money and put money in that wallet to go deploy. It's like, "Okay. So when do I start building and writing code?"The whole thing, the barrier to entry for Netlify was always connect through GitHub, pick a repo, and deploy, and you're good.

Then there was also the drag and drop deploy as well, which was take your folder, drop it on the page and you're good to go. Even that drag and drop in React, it was a hard problem back then because it was a soft problem in jQuery but we chose React. Anyway, it was just constantly trying to figure out what's the simplest time to get someone to ship a site.

Tony: That's it, when time is a value.

Brian: Exactly. So with Inngest, it's like time to deployed function or whatever it is, it's how do you get a full experience in the shortest amount of time, and a lot of times if you're starting again or if you're building on top of a product that already has been around, you have to rebuild. You're seeing that with Amplify, AWS CLI, they're building in all those services that get people to ship stuff faster.

AWS is okay if you don't log into the console anymore, because now you have a better developer experience. You only get there if you really, really dive in real deep, which has been a huge, fundamental shift of you had to be in the console to do anything in AWS. Even Lambda functions.

Tony: Yeah, totally agree, totally agree. That's key. I think if you're building a product, time to value is one of the single most important metrics that you can't necessarily track but that you need to optimize for. That's key across everything. So yeah, for us, how fast can you write a function? Wrap your existing code, Inngest, create function and you're done.

Give it the event you want to subscribe to and it will run. Give it the schedule that you want to run on and it will run reliably with retries and such. You don't need to learn anything. That process is really key, because you look at some existing stuff that promised to make your things reliable, or event driven software, getting started, yeah, it's a hassle. It's a three month endeavor and so it's super interesting, yeah, a lot of amazing work. The Netlify thing with drag and drop was huge on Hacker News back in the day, 2016, '17, whenever that was.

Brian: Yeah. It was funny because around that time, Netlify was getting on Hacker News every month pretty consistently. We were still less than 10 people, I think we were seven at that point. But yeah, we weren't a lot of people and we were figuring out a lot of stuff and then talking about it and shipping it. Yeah, it just takes a little bit of tenacity and talking to your users and figuring this out, and that was the beauty of Netlify.

Our users, similar to how you got your job at Docker, we had some of our first engineers after me, were just people in support channels. Like, "Hey, thanks for your help. By the way, if you're ever hiring, here's my skill." And we hired some engineers that way, which is pretty amazing. But I did want to ask a question about edge compute and stuff like that, because edge functions, different from serverless for now, but do you see Inngest also supporting the edge as well?

Tony: Yeah. So number one, yeah, we can run on the edge but we can run anywhere because it's HTTP so you can deploy us to the edge by literally just importing the serve handler that runs on CloudFlare workers or Node V8. The way I see us best supporting the edge is actually offloading work in the background, so let's say your signup API is on the edge because signup needs to be fast.

When a user signs up and you want to do the background work like add them to Intercom, add them to Stripe, send an email, manage a drip campaign, you want to send over one single event and then have five functions run at once. You don't want to add a job to five queues because that will slow down your edge function, that'll increase latency, and that's what this is all about, increasing the user experience.

So the best way to do these sorts of things is user signs up, send an event, which is milliseconds, and then in the background have your functions add something to Stripe based off of that event, create a drip campaign that will wait 24 hours maybe on the edge as well, using Inngest, just tools.sleep for 24 hours, and then send email. In that way everything is so much faster and everything is so much better, the experience is just easy.

It's like send an event, add tracking to your app, add metrics to your app that are organized in the background, then whenever you need to extend business logic you can hook into that event, write the logic you need to do, deploy it to any platform, whether it's the edge or just regular serverless functions and you're good to go. In that way, your API is going to be as fast as possible.

That's the best experience. So I think no matter what happens with the edge, if you're really trying to live up to the best experience and the lowest latency, you're going to have to do something like this. It will be Inngest-like because an event and fan out is the only way to make this experience really smooth.

Brian: Yeah. I'm tracking with you, and looking forward. Honestly, looking forward to all the advancement that folks are doing at the edge and making it viable to leverage. Yeah, happy for folks like you to figure this out so I can just pay my monthly fee and be able to have access to it.

Tony: Classic, yeah. We want it to be accessible to everybody, so the free thing is always something that's going to exist forever. Super important. I think that's one thing that Netlify, Vercel has done really well, just allow anyone to use it same as GitHub. The developer experience and empowering developers is important, so that needs to be always persistent.

Brian: Yeah, 100%. So speaking of accessibility, how do folks get started with Inngest today if they're tracking with everything you've said so far? What's the first step?

Tony: First step is going to Inngest.com/docs, read the docs. It's literally a Yarn install, and then you can create a function and then run Inngest Dev locally and everything will automatically tie together. Yarn Dev, Redwood app, Next app, whatever you're using is going to talk to the dev server automatically, start running functions whenever you send events.

You don't need to do anything but literally implement the SDK, and once you've pushed that live to Netlify, there's a Netlify plugin, that will communicate with us, automatically deploy your functions to the account and then we'll start pooling your functions in the background whenever you send the event or whenever the Chron schedule occurs. So it's super easy, just Inngest.com. It takes a minute to add the SDK to your existing project and you don't need to change anything, everything will just work in the background as you would expect.

Brian: Cool. Well, I actually hit the Octarize GitHub and I'll actually be checking it out because we're not actually leveraging a lot of Netlify functions. In NextJS we do have one API request advantage login, and checking to see if people are logged in so that's the only function we have so far. But we are doing a bunch of stuff on the API in the background, so there's a lot of things that we can actually benefit today at OpenSauced.

So I'm going to give it a little check, read some documentation and then send it over to the team to see if they have any interest in it and we can scope that out. But yeah, I'll have my people talk to your people. I did want to transition us to the picks, so these are Jam picks, things we're jamming on. Could be music, food, everything is on the table. If you don't mind, I'll go first. So I've got a first pick which is GoodTimer.

I just recently joined TechStars with OpenSauced, super excited about that. GoodTimer is part of our TechStars cohort, it's only 12 companies of the 1,000 that interview get chosen and this is one of them. I was really excited about this because I've got a four year old and GoodTimer is basically a little robot, it's kind of like an Amazon Echo device similar-esque.

But you don't really talk to it, it talks to you, and it's essentially a clock that every segment of time if the child is doing good and teaching the timer good habits, it gets a token. So when you unbox the boxes, it's an experience where you read a book about a clockmaker and its new clock which is called GoodTimer. The kid is required to teach it good habits.

My youngest kid has been teaching GoodTimer good habits, you need tokens. It's been an amazing experience. They're about to ship a mobile app as well to help track good habits and making sure kids are having a good time. I thought it was really clever. I think hardware startups are pretty challenging as from what I hear. I think they have a lot of really good traction, no pun intended.

Tony: Yeah, that's cool.

Brian: June had mentioned you had a niece or a nephew.

Tony: Yeah. A nephew. Dude, this is so interesting because I think we feel it as adults nowadays, everything is kind of grappling for your attention and when you're a kid and you don't have that discipline or notion of why, that's got to be so hard. So stuff like this is super cool, to actually instill good habits in kids without them freaking out because the dopamine isn't being hit all the time.

Brian: Yeah. Well, that's the thing. So my kid's definitely got the bad habits, we were very much limited screen time folks, but then during the pandemic it's like, "Oh, I got to work. Could you just take the iPad and just sit on the couch while daddy's on a call?" So obviously screen time has expanded, so then it was like, "Eh, no more iPad, let's go look at each other in the face."

Then it's a meltdown. But with the GoodTimer it's like, "Hey, you got to teach the GoodTimer good habits. You've got to be cool with turning this off, we'll get it back. We've got the tokens, you can cash in the tokens for more time." And yeah, it's awesome reverse... I don't know if I'll ruin my kid with this reverse psychology thing, but the sheer fact that you're teaching the GoodTimer good habits, my kid is all bought into that.

They're like, "Oh, okay. I need to teach it better habits." So hopefully it'll last forever, but there was a couple of times where... Actually on Monday they were like, "I don't know about GoodTimer. I don't know if I can teach it good habits. I might mess up." And I was like, "It's okay if you mess up."

Tony: That's amazing, the responsibility they get from it.

Brian: Yeah. Oh, and the best part of it, if you're having a bad time you actually flip it over and then it stops collection, you stop collecting tokens. So the timer pauses, so it's almost like a good time pomodoro that'll last as long as you're doing good habits and making good choices.

Tony: That is cool. It's also super interesting to hear your child be like, "I don't know about today." The concern that they have for someone else, the responsibility that they take on. It's so good. It's so good to hear. Yeah, that's a good pick.

Brian: Yeah. They have a good team and a lot of good advisors, so they've been obviously doing a lot of research and getting a lot of feedback on the approach. I was very impressed on what they had going into TechStars, so it's definitely going to be good. A great experience for them. I wish I was doing way more revenue, or as much revenue as they were doing. But it is what it is.

Tony: TechStars is going to be interesting. I'm super interested to hear how that goes for you.

Brian: Yeah. Happy to share all the secret sauce on that. I did have another pick, which I did learn from TechStars. What's been awesome about this experience is I got to meet the founder of GoFundMe, one of the cofounders. GoFundMe, it's the platform for giving healthcare to the rest of the United States, which is mostly a joke, but most of the GoFundMe's are basically, "Hey, I need a kidney replacement or my car broke down."

They way they got to that point was this concept called viral loops, it sounds like for what their altruistic things are doing, it sounds almost too grow tacky. But what they realized is when you do these campaigns, the campaigns need to be relevant and the relevancy is really what turns on the viral loop. And so we've been actually looking at OpenSauced and figure out what are our viral loops in our platform that will take...

Okay. Let's say 100 people saw or got an impression of this tweet or an impression of this experience, 10 of those clicked it, of those 10 they did the activity and then shared it, but then the way you share it, you share it in a place that's relevant. So if I want to get a job doing open source, and I get a job doing contributions to Inngest, and that contribution to Inngest got me noticed by Netlify which now I have a full time job at.

Now that story gets shared to another 100 people, those 100 people click the button, do the contribution, and share it. So we're still figuring what the loops are, but once I talk about the actual real variable, which is relevant, it's good to share it but it also matters who's sharing it and where. Which I think you can attest to as a founder, getting intro'd to people like VCs, it is about who introduces you, it is about the timing as well. So as far as our product goes, we're really trying to pull those levers and start getting more people on the platform.

Tony: Yeah. That's interesting, that's interesting. Is it like the K Factor in social platforms? I think that's what it's called, and the K Factor is similar to the R Factor in disease about how fast things spread. And if the K Factor is above one, then more than one person is sharing it when they receive it, so naturally it will grow, and the viral loop is about how fast you can build out that K Factor and the levers you can use.

Brian: Yeah. That's also what's so important about having a good developer experience, because if you get a bunch of people looking at your project, clicking it and being like, "Hey, the promise was there. But when I got there, the promise was not there," you do want to make sure you build a good product too as well.

Tony: Yeah, 100%. Yeah, it stands up. One thing that you don't really appreciate is just distribution. It turns out that distribution is so important. You take it for granted in the things that you do day to day, like you want a new TV, LG, Sony, whatever. They've got distribution figured out, classic example.

But even then, you look at new things that are coming out like how did Docker get its distribution back in 2014? Word of mouth, new tech, but distribution is everything. The viral loops, it makes sense. It's nice and sad that GoFundMe is used for such good and that people have to do that.

It's kind of why I wanted to work in healthcare. For better or worse, it's good that the viral loops worked to help them, in a way. So yeah, that's cool, that's cool. So much. Dude, I'm going to message you after this, just sharing ideas, sharing notes. Yeah, this is cool.

Brian: Yeah, 100%. And anyone listening, hit me up. I love talking shop like this. It's one of those things that I don't do enough of. The whole developer experience, I just do it and I talk with my team about it. But I don't do a lot of podcasting and blogging about that aspect of my life.

But I love just talking shop about it, I just don't think, "Oh yeah, I should talk about my strategy for content creation and how we were able to scale content at Netlify or content at GitHub." Yeah. Anyway, I digress. I will give you space, do you have any picks for us?

Tony: Yeah. Technologically and then non technologically. The technological pick would be OPA, Open Policy Agent, which is just a genius bit of software written by people that I'm ashamed to say I don't know the names of. Essentially, these amazing developers have contributed OPA, Open Policy Agent, to CNCF, I believe it is. Essentially, it was an ABA/AC, Attribute Based Authorization/Access Control.

Why is that good? Well, everyone knows RBAC. I'm in this role, I can do this. Yay, I'm an admin. Oh, I'm not an admin. You can also do that with ABA/AC, you can give them the role attribute, but you can do crazy things with it like, for example, at the healthcare place. Doctors at this clinic should be able to see these patients, attribute based because patients have a clinic ID, doctors have a clinic ID and so on.

So you can do authorization that way. You can do crazy things like if a user requests access to your customer support team, and they've confirmed this customer support member can add access to my account, then give them access. So all attribute based, absolutely genius way of doing things, comes with its own little language and tests that you can write.

Super easy to integrate. Amazing technologically. I think it's the underpinning of a bunch of a new wave of auth startups that are coming out. Basically everything that I know that I've seen recently uses OPA under the hood and, yeah, so good, so good.

Brian: Yeah, interesting. This is the fascinating thing, we started this conversation with talking about how Netlify changed the developer experience and now people approach how you interact with your previewed staging deployment. It's a given. Staging deployment is a thing you just get because you connected to a GitHub repo, and now with authentication, now I haven't actually seen this trend because obviously I haven't had the newer authentication tools come onboard and talk through this.

But I know I know Octa/Auth0 are making a lot of different changes in their product and doing a lot of really cool things around auth, but also Clerk. Yeah, this is why I do this podcast, I love talking to folks about what they're excited about because then I get excited about it without doing all the research and they just give me the links to read.

Tony: Yeah, OPA is great. It's very, very, very cool. The second one would be a book actually. I don't read too much, except for nonfiction now. I have a selection of books next to me about product traction, innovation, that sort of stuff. But there's one really good fiction book, actually I think most people know what Aldous Huxley from A Brave New World. His last book was this book called Island and it's sort of like Brave New World but less dystopian, still kind of crazy.

It's about this guy that lands on a forbidden island and the island have eschewed the rest of the world to focus on mindfulness. So there's elements of buddhism versus capitalism, intention versus non intention, presence, mindfulness versus getting carried away with life. It's honestly the same themes as A Brave New World, blends in so much good stuff and it really makes you think about life. It really makes you think about how you can act and how you go about life in general.

Amazing book. Complete work of fiction, but incredible, and would definitely recommend it just for the themes that it strikes on. Yeah, so many powerful things in it, so really, really good.

Brian: Cool. I'm definitely going to check it out. I've got a huge list of books I've been reading, but I do a bit of fiction as well. I've been reading with the kids at night, which has been a very rewarding experience of like, "Okay. From five o'clock to nine o'clock, stop working, it's focus time for the family." I'm allowed to go back and work after nine, ten o'clock.

But five to nine I'm a part of the process, part of the system, and so we've been reading a lot of old books like Charlie And The Chocolate Factory. We're actually reading it right now. I read it at my son's age, his age now is nine and I read it I think at eight. So it's not in the curriculum today for his school, but I'm like, "Hey, I've got a book for you. Let's read it." Hopefully it's appropriate nowadays because culture changes, but so far, so good.

Tony: Yeah, crazy. I think it's super interesting that you touched on that because it turns out that discipline is really important as well, and honestly, you do your own company. It's so important to have a life outside of that, and family are the most important thing, so it's really good to hear. Discipline is underrated.

A lot of people think motivation is the thing that you need in life to do things, and it turns out that discipline is more important. I think other people have talked about this way better than I can. If you ever look at people's talks on YouTube, motivational talks, I think so many people have got impressive talks about discipline. But the ability to shut off between 5:00 and 9:00 and give your family attention is huge.

Brian: Yeah. It's tempting to keep going, but I know that work's not going anywhere. It'll still be there when everyone goes to sleep.

Tony: Yeah, exactly. What's the incremental improvement that you're going to get after doing a 10 hour day, after 5:00? Sure, there's times in which you might need to do that, but honestly, you look at the negative impact that it has on family, especially your kids at that age. The present is time you never get back.

Brian: 100%. I've loved this conversation. Definitely looking forward to staying connected. Folks who are listening, definitely check out Inngest. This is definitely a tool that's needed in this ecosystem, so try it out, give feedback, give it a start on GitHub. We didn't even go over the open source angle on this, but there's a star there if you want to take a look at the repo. I'm sure your developer experience is going to be top notch because you're paying attention to it.

Tony: Thanks, man. And thank you for having us on the podcast. Absolute pleasure to talk to you.

Brian: For sure.

Tony: All right, take care.

Brian: Yeah. Listeners, keep spreading the jam.