Ep. #147, Secure Local Dev Environments with Chris Stolt and Ben Burkert of Anchor

Brian Douglas: Welcome to the installment of Jamstack Radio. On the line we've got Chris and Ben from Anchor. Chris, why don't you start us off with what is Anchor and what problem are you all solving?

Chris Stolt: Yeah, cool. So Anchor is a company that Ben and I started a few years back. We both found ourselves not working and Ben kind of came up to me and was like, "Hey, let's solve this problem of helping developers run internal TLS for their backend infrastructure." It's kind of a problem that's plagued both of us throughout our careers. So we set off to solve that problem and really wanted to do it differently than it's been done before. So really take a true developer-first approach and build the tools that resonate with developers.

Brian: Cool. Yeah, so I'm actually really interested in this sort of tools for developers piece, 'cause I am a developer. Man, I've been writing code professionally for 10 years, unprofessionally and like garbage code for quite a few years before that. But yeah, the whole ethos of us being on the Heavybit Network but also my sort of come up in starting this podcast is I love talking to folks and founders and open source maintainers about stuff that they're building.

So, I would love to like dig in and find out more about, okay, what are the things that you all are building? And tell me some of the pain points you've been solving with Anchor.

Ben Burkert: Sure, yeah. We got started because this is a problem that both of us experienced previously in our careers. I've been on the IC side my whole career. And, everywhere I went, different startups, I kept running into the same sort of problems with internal TLS. You know, maybe someone had set up a CA and then kind of forgotten about it, and then a certificate expires at midnight and then everybody gets paged, and it's this whole fiasco.

You know, it just kind of kept coming up, kept coming up, kept being a problem where we needed to use internal CAs to deploy encryption TLS to internal services. But, the tooling and just the way we were doing it was so broken. And then around, I think it was around 2015, ACME came along and really changed how people provision certificates on public web. It became way easier for you to get a TLS certificate for your website.

I started playing around with that a lot, used it a bit for internal TLS, and thought, you know, I can replace internal TLS with public certificates, but there's some downsides there. You've got low API limits. Any certificate that you provision from a public CA, it's going to go into what's called a certificate transparency log, which is kind of like a public ledger of all certificates. It's a great technology. It's there to make sure that no one's like accidentally getting a google.com certificate that they shouldn't. But, it does mean that if I've got like betaservice.mycorp, and I get a certificate for it, that sort of information's going to leak. So you could leak customer data not even knowing it.

So ACME was great for public CAs, but I wanted to kind of use ACME for internal CAs. And so that was kind of the genesis behind starting on Anchor. Chris had similar experiences. He kind of came from more of a support angle. I'll let him talk about that.

Chris: Yeah, so previously I was running support orgs, doing some ops, you know, system engineering type stuff before that. And just the toil of provisioning certs and running them, having them expire. Either SxSs are getting paged in, or from a support angle you're having an outage, so the support team's coming in to like help manage and run the outage and communicate with customers. And that stuff was always a pain.

Even prior to Let's Encrypt and ACME, I was at Heroku during those days, and what was going on is there was a big shift to really encrypt the entire application, not just the login page and a couple pages with sensitive information. It's like hey, just encrypt the whole thing. So customers were showing up and trying to upload their certs, and they'd get their cert from their cert provider and they'd upload it and it wasn't working in the browser. Browser was rejecting it, because the cert chain wasn't right.

And from the support side, that's kind of a little bit outside of like what we're focused on for really supporting them. It was like kind of on the customer's end, so we'd be like, "Hey, go get the node chain from the cert provider." And they're like, "This is the one that the cert provider gave us." And we're like, "Oh no. If this chain that the cert provider is given you is wrong, like you're not going to fix this on your own. We're going to have to go in and do this. We're just kind of, or hand was forced."

So we wrote some automation tools that we baked into the CLI, dev worked through CLI at some point that automatically cert chains when you would upload a new cert. And that worked well, and then ACME came along and kind of solved that problem. But we're still plagued at Roku with all the toil of managing this stuff on the backend with all the things that Ben was saying. So, you know, we'd have the occasional outage, because the cert expired, something like that. So when Ben, you know, brought it up, I was like, "Yeah, let's go like solve this problem. Like, let's make this go away."

Ben: Yeah.

Our experience gave us this perspective that a lot of the problems with internal TLS is tooling problems. Certificates are really hard to use. They're hard to understand too. And I think it's a situation where they've got like 30 years of croft and standards that are not great, but then the tooling that we have for them, the open source tooling is also difficult to use.

And I think seeing that, you know, this provisioning problem, the problem of getting certificates onto servers, ACME could really address that. And it allowed you to kind of solve this provisioning problem as a developer. But, for internal TLS, you've got this other big problem which is getting all of your clients to trust those certificates. So that's a unique problem to internal TLS.

For public CAs, we've got the root trust programs that kind of update our browsers with the trusted CAs, and make sure our operating system is installing the certificates it needs to trust public certificates. And this is really a tooling problem that's handled for us by the browser and the operating system makers. But, there's no equivalent to that on the internal TLS side.

So that got us thinking like this is really a tooling problem. We can build a product that gives you that tooling to solve this problem for your internal TLS so that you don't have to think about is this client going to trust this certificate presented by this other service? We'll build language and system packages that help you automate and streamline the whole process. So yeah, we want to take a developer-first approach. We want to use better tooling to kind of address the pains of internal TLS. And we want to take the experience that Chris and I had working at dev tool companies for 10 years and really bring that to a space, this internal security area, a space that's really lacking in good developer tools.

Chris: Yeah, so basically on the provisioning side, we have ACME, right? So ACME is an open API that has a standard for it. There's an RC out there for it that came out of Let's Encrypt. So we can use ACME to automate the provisioning side of things, and that gets the cert to the server and it is really smooth. And you use API keys to do that, so, check, developer friendly there. Dealing with API keys is something developers do on a regular basis.

And then, as Ben was mentioning on the distribution side, getting the clients to trust it. The old way of doing it was really, hope the ops team comes in and drops magic files on the file system. They're not really magic, just like that they're in the right spot. As a developer, you're like, I don't care about these. I'm just going to like point things at them and hope that they're there, and they're up to date.

And what we realized was that we can manipulate the trust stores from inside the language, so we can ship packages for the languages we support. Right now, that's Go, Python, Ruby, and JavaScript. And so, we build those packages for you, and you can add them to your application using Gem or whatever package manager you're using for the language. And updating and make sure that they're up to date is simple as baking them into the CICD pipeline, using Dependabot, something like that to say, "Hey, there's a new version for this." Just bump the version, and you're good to go.

Brian: Yeah, I'm sitting here listening to you guys explain this pain point that you're solving. 'Cause I spent a lot more time in the front end side where like I worked at that whole side, Let's Encrypt, strong partnership there. You don't have to think about it, but whenever I had like a server issue of TLS, it's like, oh I guess I'll just probably just like push this production and like test in production, instead.

Ben: Totally.

Brian: Because, like my limitation of not knowing the tooling is I could either go figure this out or I could just go test live, like do it live. So like, this is now, my brain's kind of melting a little bit of like, oh, of course there's a solution for this. I just didn't have the bandwidth to even think about or go discover this, or didn't want to go bother someone working in infrastructure or dev tooling to to unblock me.

Ben: Absolutely.

Local development with HTTPS and TLS is a huge problem. It's one that both backend developers and front end developers have to like contend with. I think you're not alone. I think a lot of people just kind of punt on adding encryption locally to their development environment, because these tools are difficult to use.

And that's a big reason why we built and just released Lcl.host. Lcl.host is a free developer tool from us, and it really helps you get HTTPS in your development environment. The way we do that is kind of a few different parts. We build a CA that's a dedicated for your development environment hosted on Anchor, so it's managed by us. You don't have to run any server software or anything. And we configure your local system to trust the certificates that that CA issues.

And then we provide this Lcl.host domain where it's kind of configured in a special way. Any subdomain of Lcl.host is going to route to 127.0.0.1 local host. So it lets you provision certificates for Lcl.host subdomains and not have to deal with kind of the DNS set up required. So you can kind of, on the fly, just set up your application or get a a certificate for one of these Lcl.host subdomains, and you know that it's going to route to your development box.

It's going to be trusted by your browsers. It's going to work from the Command line. It's going to work in a lot of the ways that local host works in your browser context. But it's going to give you like full TLS HTTPS. Your browser is not going to know the difference between a certificate issued by a public CA versus this Lcl.host subdomain certificate.

Brian: Yeah, something I did a a ton when I worked at GitHub was we built a lot of GitHub integrations and tested those locally. And this is where I would punt to work in production. Basically, like you'd have to have two different certs and have to basically check if you're in dev mode versus production, because it was like a huge limitation. And the goal for most folks building integrations, like let's test to see if it works, and then do it live. So like you spend very little time spitting in your dev mode, and you just spend a lot of time pushing to then trigger CI to then deploy, and then test live, go back to the well.

Ben: And those are slow feedback loops, too.

Brian: Oh yeah. It's like an all day effort of like, oh I got to fix that one thing again. And you end up building these little integrations and tools that connect to your CI that are set it and forget it. It's like you write, I actually just took a PR on one of my GitHub actions. And I haven't touched that thing in like a couple years, but I get PRS every now and then, 'cause I got a lot of folks using that and their CI. And they were all in Bash, 'cause I wanted it like to be maintainable and like quote unquote "secure." But also, I don't have to like worry about dependencies. And I'm like looking at Bash scripts this morning, we're reviewing PRs, because again, this is a thing I test in production.

Ben: Yeah, absolutely. It's interesting.

I think over like the last 10 years or so, this push for dev/prod parity has really caught on with developers. You know, you say dev/prod parity, developers' development environments should look like their production environments. The more we can make those two things similar, the less likely it is that we'll push out a release with bugs that only appear in production environments.

I think this idea is like largely caught on in development communities. But, in terms of like encryption and HBS and TLS, there's really like still a gap in terms of like dev/prod parity when it comes to encryption. And I think as you're talking about your experience, I think a lot of people end up having to push their workflows into kind of this hybrid kind of deployment stuff, because there's just no good way to get parity in their development environment when it comes to encryption. So yeah, I think if you're not alone, it's a big problem for a lot of developers.

Brian: Cool, yeah. So Chris, I was going to ask what's next? And also I wanted to point out this Lcl.hosting is like a perfect, like I did DevRel at GitHub. It's like a perfect like engagement loop, like to be able to get folks to think about your product, but also like tap it and solve a pain point and then probably expand.

Chris: Yeah, that's right. If developers are coming in, they need local HTTPS for whatever reason. There's a myriad reasons that you might need that. Use Lcl.host super fast, super easy. It's basically one command and everything's set up. We do everything for you. Use it, enjoy it. It's free. I have no plans on ever making it not free.

But, if you get to the point where you're ready to, you know, say, "Hey I like this tool, and I like what's going on here, and I want to push this up into production or staging. I'll use this for my backend infrastructure, maybe do some custom domain names," something like that. That's where like the Anchor side of things comes in. And, you know, we're excited to be working with folks and looking for bringing on new users to help us test. We're still in a private beta for those features, and that's really what we're focused on right now. So being able to bring those from a private beta out to a GA and fully release the product.

Brian: Ooh, exciting. And yeah, excited to have you on the podcast at this stage of the journey. I would love if I can stay in touch and continue paying attention to this, 'cause I think you're solving a very clear pain point that exists that folks have been punting for a very long time.

So, with that said, I wanted to ask, like you said you're going like beta going to do GA, hopefully soon. Like, how do folks reach out to you? How do they connect to find out more and hopefully maybe even test this out in their systems?

Ben: Sure, you can definitely use Lcl.host today. Just go to Lcl.host. Dot host is a valid top level domain. You can sign up there, and you're using Anchor. From there, you can use Anchor as an organization. We're currently open for organizations to use Anchor in development mode for free.

So, just like Lcl.host sets up your local development environment, you can set up a organization's development environment shared between different developers just by signing up on Anchor.dev today. Definitely reach out to us at our email address, contact@Anchor.dev if you want to take this into staging and other environments.

Brian: So cool. Yeah, so I want to transition us to picks. So these are things that we're jamming on, things that get us going, moving, shipping code called Jam Picks. And if you don't mind, actually I'm going to go first. I've actually got a pick, which is a pick, I didn't even plan. But, I recently moved house, so moved from like East Oakland to North Oakland. So folks in the Bay, you're very familiar, North Oakland. It's very nice. Shout out to East Oakland. It's very nice as well. It's changing very fast. North Oakland's like, it's definitely my spot for sure, but I have no WiFi.

So I've been actually working out of this co-working space called Oakstop. So it's like right in the middle of like downtown and uptown. It's great, because you can do like 50 bucks a month to get two days a week. So, if you got to get out of the house, or you need to like grab more stable WiFi, like that I don't have right now, check out Oakstop. It's got a lot partnerships with like the CORE Center and other underrepresented startup foundations and stuff like that. So folks who want to be in the scene and see like a sort of a different wave of founders, check out Oakstop.

And then I have a second pick. I'm just going to jump right in. Starter Bakery, this also opened up in the neighborhood that I just, I technically moved back to. So I've been in North Oakland for years. Pandemic, did a whole pandemic move to get like a backyard and a house out there. Decided we don't need that anymore, so we're back in North Oakland. But Starter Bakery has been a bakery in the Bay Area, San Francisco for years, like 10 years. And they just opened up their first bakery. So, they were like, the bakery who provided all the baked goods to all the coffee shops that are local.

Ben: Cool.

Brian: And now they have like their main spot, and it's actually amazing. Like I think about like the croissants and like, everything's like great there. And I love taking the kids there for a walk, and they get to pick out whatever they want. And they have like pretty on point coffee, because of their close relationship to every coffee shop in the area. So if you're in the Bay Area, go to Oakland, check out Starter Bakery and then walk over to Oakstop.

Chris: That's awesome. I used to live in Berkeley right at the corner, the intersection of like Berkeley, Oakland and Emeryville.

Brian: Okay. Yeah.

Chris: So yeah, I'm familiar with that area really well. That sounds awesome, and sounds like something we need here. I'm in Boulder these days. Need something like that here in Boulder, 'cause I need to get out of the house sometimes. That would be nice.

The things I've been jamming on, sometimes when I need a break, it's doing some woodworking. You know, surprise, surprise, tech worker goes and does some woodworking. I've been getting into that. And we're also undergoing a renovation on the house, so I'm looking forward to building some like shelves, and things like that nature to like add to the renovation that's almost complete.

Brian: Ben, do you want to jump in?

Ben: Oh yeah. I was going to say, I moved out of San Francisco just before the pandemic started, and the bakery scene out west is like definitely one of the things I miss about living in San Francisco.

Chris: It's like MacGrubers in Colorado.

Ben: Yeah. Yeah. What I've been jamming out on, I've been using Ghostty a new terminal by Mitchell Hashimoto of HashiCorp. I believe it's still in beta. It's written in Zig, which is a cool, interesting new programming language. It's really cool. Super fast, very stable. I switched to it and kind of took it as an opportunity to redo some of my development environment, switched to Neovim, redid my Tmux layout. And it feels like I'm like jumped five years ahead in terms of like my development setup as part of that. It's great. It's cool. It's fun to use.

Brian: It's cool. Yeah, I knew he was building something. I haven't paid attention to closely, but yeah, that sounds amazing. And just like three episodes ago we had TigerBeetle, who's also a company built on top of Zig. So it's nice to see Zig getting more love and the community expanding for sure.

Ben: Yeah, it's definitely a cool language. It's one to keep an eye on, I think.

Brian: Yeah, for sure. Well, thanks for sharing about Anchor. Looking forward to actually kicking the tires on this thing as well, when it gets out of beta.

Ben: Awesome, thanks Brian.

Brian: And listeners, keep spreading the jam.