Ep. #33, Tailscale with Avery Pennarun

Marc Campbell: Hi, and welcome to another episode of the podcast. Today we have Avery Pennarun, founder of Tailscale here. Hey, welcome, Avery.

Avery Pennarun: Hi there, thanks for having me.

Marc: And of course Benjie is here. Benjie, how are you?

Benjie De Groot: I'm good. How are you, Marc? We've been off for a little while, but I'm really excited to have Avery and Tailscale on.

Marc: Yeah, me too. Let's just dive right in here. Avery, we'd love just to start by giving you an opportunity to share your story and understand your background a little bit with us. Do you mind doing that?

Avery: Sure. I'll try to keep it super short, I guess. I started my first startup when I was in university, it was a company called Net Integration Technologies. We made Linux based appliances for small business, and that went reasonably well. We ended up selling out to IBM in 2008. I did some banking software for a little while after that, then I went to Google where I worked on Google Wallet and then Google Fiber, and ended up in charge of the Google Fiber Wi-Fi routers.

Some of the technology for which is still in Google, not Google Fiber routers, today. When I left Google I decided, "Okay, that was enough big company for me. I think it's time to do some kind of startup," and I wasn't totally sure what I wanted to do. But that was 2019 or so, I thought I would take two months off, ended up taking six months off to recover from the big company world, and then started Tailscale.

Marc: Great. Let's just dive into Tailscale for a minute here. What is Tailscale?

Avery: That is one of the hardest questions for people to answer. We still actually don't have the world's greatest answer. There's a few different versions. One of them is that it's a VPN, some of our customers have described it back to us as like, "Well, look, I know it's not good to be a VPN, everyone has negative feelings about VPNs, but imagine the best VPN you've ever used, the one that totally changes the way you think about the internet."

That's how I like to think of Tailscale, but the technical definition is that it is a mesh overlay network. So instead of connecting you to a single point as your VPN, that's a single point of failure, it actually allows you to connect to every computer that you might want to connect to directly, regardless of firewalls that might be in between.

So it sort of turns a bunch of devices that are scattered around the internet into a virtual LAN that you don't have to think about encryption and routing and firewalls, and that kind of thing.

Benjie: One thing that I hear a lot is the comparisons to WireGuard. Can you give us just a quick overview on how, if I'm coming at this with a WireGuard understanding, how is Tailscale similar? How is Tailscale different? What do you think about that?

Avery: Yeah. So Tailscale actually relies on WireGuard and includes a copy of WireGuard inside of it. WireGuard is what we call the data plane. So Tailscale, like I said there's many different ways to describe Tailscale, one way of describing it is a software defined network, and if you have heard about software defined networking or SDNs, they have a control plane and a data plane.

The data plane is high throughput but dumb, and the control plane configures the data plane so it's low throughput but smart. So WireGuard, no offense to the intelligence of the WireGuard code, is the high throughput, dumb code, right? Its job is to be a very small number of lines of code that does encryption really, really well and really, really quickly, and moves packets fast and securely, and it does that really well.

But if you download just WireGuard, you quickly find out it's kind of batteries not included. You have to move the keys around yourself, you have to put the public keys in the right place, you have to tell it which IP addresses all of your nodes are connected to, you have to set up your own subnets, you have to load things into the kernel and so on.

Tailscale does that part for you, but at the most basic level, Tailscale handles your key management for WireGuard and it also handles so called nat traversal, which is getting connections established in the first place from between two nodes that might both be behind different firewalls that don't have ports open. So the combination of WireGuard with our little control plane is what makes Tailscale into the thing that it is.

Marc: So is Tailscale something I run completely on my laptop, in my environment, in my home or my office? Or is there a SaaS service?

Avery: So Tailscale, again, it's a split between the data plane and the control plane. So generally you run the data plane part of Tailscale on whichever devices you want to connect to the Tailscale network. Now, you can also run so called subnet relays, so like a more traditional VPN, you have it installed on a router or on a virtual machine that can then relay traffic for the rest of your subnets so you don't have to install Tailscale on all of the devices on your subnet.

Then those devices all communicate data with each other directly over WireGuard, but in order to find out about each other in the first place and exchange their keys around, they go through this really tiny SaaS service that we call the coordination server or the control server that acts as the control plane that tells each of the nodes what they should do and who they should connect to, what kinds of packet authorization to use and so on.

So there is a SaaS service, but it's very important that it's not fundamental to the data transfers. So even, for example, if the SaaS service goes down for several hours, your network keeps working because the data plane keeps on running. The only thing you can't do is reconfigure the data plane until the control server comes back up.

Benjie: So it's almost like a DNS for WireGuard, but it's not because you don't need it up to actually still use it?

Avery: Yeah. That's a pretty accurate way of thinking about it, right. Even DNS, if the root DNS servers went down, well, that wouldn't be great. But any names that you've looked up previously could be cached for a certain amount of time before you absolutely rely on those root DNS servers again. So Tailscale is more aggressive about caching, it makes sure that you have everything you could possibly need to know about the network upfront so that in case the control server had a problem, you can last for a really long time.

Benjie: That's super interesting. One thing that I want to talk about, maybe we'll get back to it, but the whole NAT traversal thing is something that's always really interested me. I think the web socket term was Turn, I wanna say. I don't know if that's right.

Avery: Oh yes, STUN and TURN.

Benjie: STUN, STUN. The STUN was what I was trying to think of. Yeah, that's super interesting. Maybe I'm skipping ahead a little bit, but can you just tell us a little bit how the NAT traversal works? At least at a high level.

Avery: Sure. Well, how many hours do you have?

Benjie: We could do like a seven, ten hour podcast. Whatever, it's fine.

Avery: Seven to 10 hour podcast, all right, that's about right. So just stepping back to the absolute most basic level, I think most of us have played, say, video games that if you want to do multiplayer they tell you to go your router and configure some kind of either UPNP or open a port or something like that. That works kind of okay for video games where you have a residential router that let's you do that, but even then it doesn't really work that well, right?

Quite often you have to go find the technical person just to reconfigure a router so you can play your favorite Xbox game or whatever in multiplayer mode and without sucking. This is fundamentally a problem that has come up in the last 25 years or so of the internet, so called Network Address Translation, or NAT, plus firewalls.

You have a address that's assigned to your on LAN, like 192.168.something that is a private subnet that is reused, repeatedly by other people's homes all over the world. If you want to make a connection to somebody else who's computer is also inside a home, you have to get through your firewall and their firewall, and also deal with the fact that your address might even be the same IP address as the person on the other network, because you both have these private addresses. It converts into something else as it goes over the internet and then converts it back.

So NAT traversal is a series of really obscure tricks you can play to basically make the NAT firewalls that are in between you and the other person believe that you have established a connection, basically established two connections that are both unidirectional outgoing. This is getting more complicated than I hoped for. One thing that your NAT firewall let's you do when you're behind it is make outgoing connections to the internet.

Obviously the internet would not work if you were not allowed to make outgoing connections, right? But the result is that most of the things that you connect to are on cloud services that don't have incoming firewalls, they just let you connect in. What they don't let you do is have the cloud call back and make connections back to you. However, the catch to that is every connection once its established is actually bidirectional.

You make an outgoing connection, of course the other end has to be able to send packets back for the answer for that connection, right? So what NAT traversal is at its most fundamental level is you trick the two firewalls into thinking you've both made outgoing connections at the same time so that the packets from each other's connections look like responses to the connection that you made outgoing.

If you do that you can actually establish a true peer-to-peer connection between two devices that are behind completely different firewalls with no ports open. This is better than what you get with video games, but it's also better than you get with typical VPNs that almost always go through a central relay to solve this problem. The net result is you have the minimum possible latency between any two points.

You actually go through the internet with no intermediate relays to make this connection. To tie this into STUN and TURN, one of the things you need to do to establish this connection is find out, oddly enough, what your own address is. So you make an outgoing connection, you know you are 192.168.something, that doesn't help anybody. You can ask a STUN server, "Hey, what was my address when I came out?"

And the STUN server, all it does is just respond and say, "Well, to me, you look like 1.2.3.4 or whatever." Then you take that 1.2.3.4, tell the other person you want to talk to, "I need you to connect to 1.2.3.4 on such and such port." They connect outwards and they can get in through this little pinhole that you've made in the firewall that STUN told you was 1.2.3.4. Does that make any sense at all?

Marc: That makes a ton of sense, I think that explains it, and I think this all comes down to the fact that I'm sitting here right now recording at home but I have one IP address that's going into my entire house. Everybody else is using it and they all have their own incoming connections and stuff like this, and like you mentioned, I'm on a private IP so this allows me to use the outgoing connection to look up who I am, my advertise address and it'll allow connections back through that.

Avery: Right. That's the whole reason NAT exists, is because there's not enough IP addresses in the world so you have to map all of your home devices on to a single IP address that you've been given by your ISP because otherwise there wouldn't be enough IP addresses. So if we didn't have that problem, and a lot of people say, "Well, look, if we just had IPv6 everywhere we wouldn't need any of this stuff because every device would just have a unique IP address and all the problems would go away."

That's partly true, because firewalls would still exist, I think we'd still want to block traffic from coming into our home network, but it would be a lot easier. The difficulty of everything getting mapped onto that one IP address is then how to make connections back the other way. When someone connects back to your public IP address, 1.2.3.4, how do we know which device that's even supposed to go to when it goes back through, unless that device in behind has initiated the connection in the first place?

Marc: Got it. You mentioned in the description one of the ways that WireGuard is faster or different, anyway, than the traditional VPN is that once it establishes that connection it's a direct connection going through the internet. Not going through a central server like most VPNs do. Is that the primary difference here between a traditional VPN and what you're doing? Or are there additional differences that we should talk about?

Avery: I'd split it into two categories. First of all, WireGuard will do whatever you ask it to do, right? So in fact most people using WireGuard use it in the same kind of hub and spoke style where there's a center point and everything goes through that center point. That was the way that I think most people just expected it to be used when it was invented, but it's fundamentally a layer that if you configure it you can make it do whatever you want.

The problem is if you've got, say, 10 devices and you want each of those 10 devices to be able to talk to the other nine devices, now you're creating like 10x9/2 different tunnels that are bidirectional, which means 10x9 different configuration lines in different configuration files. Which is a lot of configurations and no human wants to do that by hand, right? That's what Tailscale automates for you, is filling out all this configuration for WireGuard and all the different nodes. Regular VPNs are more like WireGuard when used in a hub and spoke mode, and then have a much simpler configuration method.

One of the nicest things about WireGuard is they actually removed a lot of the crud that had accumulated over decades of inventing cryptography. It's hard to criticize, say, IP Sec for being really complicated because as a tech world we had not figured out the best ways to encrypt traffic yet.

So what IPsec did is they just put in a bunch of options in the protocol so you could try a combination of hundreds of different things, and then we'd find out through experience which ones of those things actually turn out to work and then you can settle on the best ones.

What happened as an industry is every vendor picked a different combination, so you can run into the annoying situation where IPsec on one brand of router just absolutely positively cannot talk to IPsec on a different brand of router because they use a different set of standards. Which is kind of a weird way for a standard to turn out. So WireGuard took the advantage of all of that experience and just said, "Okay. Forget all that, we now know the right answer, more or less. The right answer is this combination of cryptographic standards in this style packet, and we're just not going to implement any of the other things. We're going to start from scratch."

It turned out I think it's something like 100 times less code than IPsec, just to make this WireGuard layer work. Then because people spent so much time inventing all the complexity of IPsec, they only had time to implement a relatively simple control plane on top of IPsec, so you almost always ended up with most devices can only make one or two or five tunnels at most, so you'd use it for inter office links.

You'd have like one router in each office, each of those routers would talk to each of the other office routers. Or, even worse, you might have one router in each office talking to the central office, and then going back out from there just because it's such a hassle to configure. The nice thing about having WireGuard pre made for us is we could plug that into the more fancy control layer that we wanted to put in and make it do more advanced stuff.

We could spend all of our time thinking about this control plane and almost none of our time thinking about this data plane that most VPN products have to spend almost all their time thinking about. Does that make sense?

Marc: Yeah.

Benjie: Yeah, it does, vaguely for me. But I know Marc gets it. We don't like when you do math on the show, so be careful of that 10x9 stuff you were doing there, but it's okay. So the big thing here is WireGuard is absolutely not a competitor to Tailscale by any means, it is the backbone of Tailscale, if you will. And you guys are doing some really cool stuff to make it a lot more user friendly, ultimately?

Avery: Yeah. We're contributors to the WireGuard Go repository, for example. You'll see a bunch of Tailscale names in there. We've done a bunch of performance improvements. We're on very good terms with the WireGuard team.

Benjie: Okay, great. So then the real kind of what would I use if I wasn't using Tailscale thing is VPN, there's a lot of options there for VPN. To summarize this a little bit, and obviously correct me if I'm wrong here, basically if I want to use a VPN I'm calling a centralized server to say, "Hey, how do I get to this other connection?" And you guys actually plow a whole through the internet, straight to the other side and that enables this direct connection and disintermediates the man in the middle, if you will?

Avery: Yeah. So there's two different sides of what people might do if they weren't using Tailscale, one of them is a traditional VPN where you'd have a VPN sitting on an outward facing router with a port opened on the firewall that allows incoming connections. The other thing people might do is a lot of people use Tailscale for, say, SSH, or accessing a remote desktop or something like that.

What happens in a lot of cases is people will actually just do a port forward on the router back to one of their internal devices, and leave the SSH port just basically sitting there open on the internet, or leave RDP, Windows Remote Desktop sitting out on the internet and just hope nobody will find it. That is a pretty huge security problem, but it's the normal thing to do. If you spin up a device in AWS or Google Cloud with a virtual machine, the easiest way to get into it is just opening that SSH port to the world. A lot of people do that and it's a really dangerous thing to do.

Benjie: I don't do that, what are you talking about?

Avery: Do you use it with a three letter password too? That's the best--

Benjie: No, I don't use a password. I like to just leave Telnet open with no password, and that's kind of my-

Avery: Okay. Did you know Telnet doesn't even come with Mac OS anymore? It's really disappointing.

Benjie: Really? I actually did not know that. Today I learned.

Marc: It's disappointing.

Avery: It is, it's one of my favorite programs.

Benjie: I love Telnet too, that's always the first thing I do whenever a get a new type of device to play around with, I'm always trying to Telnet in and see where I can get. Okay, no, that's super cool and that's showing it off a bit more. Maybe talk a little bit more around use cases. So RDP is a great one, SSH is another. Well, not SSH, but... Well, actually I think there's a new SSH product, do you want to tell us about that? And just some other use cases around Tailscale?

Avery: Right. There's many use cases for Tailscale because it really is...

When I talk about a network overlay or a network mesh overlay, it sounds complicated but the main thing it does is it makes the internet work the way it's supposed to work, right? It makes it so that your devices can actually just talk directly to your devices bidirectionally, so you can build all kinds of stuff on top of that that we've been forced to not build because of the world where you can only make outgoing connections to the cloud.

One of the things that has annoyed me is if you want to transfer a file between two different devices, nowadays one of the easiest ways to do that is to just upload the file to some cloud server and then download it back down again. Which is okay if you've got a really fast connection and it's symmetrical and you've got like gigabit up and gigabit down or something, but a lot of people don't have that. So you find yourself like, "I want to send someone a photo that's sitting right next to me, and I will send it over iMessage or Google Hangouts or Signal or whatever their favorite chat system is."

It actually has to go out to the internet and then back again. It's really silly. So Tailscale let's you build really simple services on top of that, that can just use the peer-to-peer connection when it's available and if you're not side by side then it goes out to the internet and does that connection instead, but without an intermediary.

One of the simplest use cases of Tailscale is a thing that we have called Taildrop. So if you have Tailscale installed on, say, your phone and your laptop you can just use Taildrop to say, "I want to send this set of photos." And it will send it directly over a peer-to-peer connection, no matter where those two devices are in the world, without going through an intermediary. So it doesn't cost us anything to let you do that because Taildrop is free, but it's also as fast as it can be because if they're sitting side by side it'll actually let you send it over the local link, not over the internet.

But you can use your imagination for all of the other stuff on top of that. Taildrop itself is open source, I wrote actually a blog post about how simple Taildrop is. I think it's called Why Taildrop? Or Taildrop Was Kind Of Easy Actually. The reason for that is when you've got the connectivity layer in place, to build something that allows file transfers is as simple as running a web server that let's you accept put requests, and then the person sending just makes a put request to the server that is sitting there.

It doesn't have to do any authentication between Tailscale has already done the authentication. It doesn't have to do any NAT traversal or firewall stuff because Tailscale already dealt with all that stuff. So you're back to a world like the olden days of the FDP program where you can just FDP to any computer you want and get input files because you're confident that that connection has already been validated right. When you've got that ability you can start applying that to more and more things.

The thing that we've launched recently is called Tailscale SSH, Tailscale already worked with SSH because it's a network layer so you can maintain your own SSH servers. The first advantage is you don't have to expose those SSH servers to the world, you just install Tailscale on your computer, you install Tailscale on the server, and now you can SSH to the server without worrying about firewalls.

That was pretty neat the way it was, but we found that a lot of people still were having trouble setting up their SSH keys to make that process as seamless as possible. Especially when you're dealing with big companies where employees come and go and you've got servers that are shared, in production. It was actually kind of a hassle removing all of people's SSH keys when they left the company.

So what Tailscale SSH does is it actually includes an SSH server of its own that you can install as part of Tailscale on one of these servers, and it'll actually let you into the SSH server based on your Tailscale identity so you don't have to deal with any SSH keys whatsoever. Then if you drop somebody out of your Tailscale network in the Tailscale admin panel, which is the little SaaS service that I was talking about that's part of the coordination server, then instantly any device that's authenticated over Tailscale SSH will stop allowing that person in.

Or if you add a new person to your engineering team, you just put them in the engineering team and it automatically reflects all of that information out so any device that's supposed to let people in from the engineering team can suddenly SSH into that device. So it's pretty slick, it completely eliminates the most annoying part of SSH which is key management, and the second most annoying part of SSH which is net ports and finding them on the internet and naming them and stuff.

Benjie: Wow, that's super cool. Little pedantic question here, but what if I'm currently connected and you revoke my Tailscale privileges? What happens?

Avery: It will immediately disconnect you.

Benjie: Really?

Avery: Yeah. Because it's the network, right? So the network knows this is the control plane informing the data plane, "Okay, you no longer have a route between these two things, so cut it off."

Benjie: So it'll break it immediately because it's all gone? That's super cool.

Avery: Yeah. And every Tailscale reconfiguration is reflected out to everybody in network in about 100 milliseconds or less, so it's really, really fast. You're not sitting around, you can reconfigure your packet filtering rules in the admin panel and you might have 10,000 nodes and you just save the configuration and those 10,000 nodes reflect the new packet filter instantly.

Benjie: So even a fleshing credential type mechanism, if you have some Sec Op stuff you need do here, this is a really valuable tool for that as well?

Avery: Yeah, you can suspend users, for example, if somebody is going on leave and you don't want them to have access or they're going to a country where they might have their computer confiscated or something. You just suspend their user account and instantly that user account is shut off and then you click unsuspend and it's instantly back.

Benjie: Wow. That's super powerful.

Marc: Without having to provision new keys or anything like that, it's just-

Avery: Yeah, there's no messing around with keys. You never have to think about keys with Tailscale because you're talking about users, and it has a list of the user's devices and you can tag devices, and you can talk about users in these groups are allowed to talk to the SSH port on computers that are tagged in this group.

Then people in this group are allowed to apply these tags to these servers when they bring them up, so you can have an operations team who's allowed to use certain tags when they're spinning up servers that will then be accessed by people in other groups. But it's all these high level concepts, you're talking about people and you're talking about tags, as opposed to talking about IP addresses and port numbers and setting up firewalls because it would be really hard to configure a network of firewalls for a mesh network, right?

Where do you even put the firewall in order to block the traffic in the right place? The firewall has to exist on every single Tailscale node, and you obviously don't want to set that up by hand, but it's really neat that 10,000 nodes can have their firewalls reconfigured instantly by the central control service.

Marc: Yeah, that's cool. We'll make sure to include links. I collected a couple, I've been reading through it as you've been describing it too, Tailscale SSH and Taildrop are both cool. Talking about use cases though, you said earlier you can use your imagination and think of anything you can build on top of this meshed network now. As you've been building this and sharing it as open source and hearing folks using it, has there been anything that you couldn't have even imagined when you started building it? You were surprised and were like, "Well, that's clever"?

Avery: Let's see. People use them for all sorts of stuff. One of the earliest things we saw was people just plugging a camera into a Raspberry Pi and building their own monitoring system, right? So you could actually watch the video from the camera from your Raspberry Pi, on your phone without it having to be uploaded to the cloud and back.

I don't know how much experience you guys have with buying crappy internet cameras at the store, and then you try to watch them from their dedicated camera app, and there's always this weird, 10 second delay. Why is it delayed by 10 seconds? The answer is it's going into some stupid cloud provider and getting stored on their disks, and then you're downloading it again from some web server, and there's all these different layers and nobody figured out how to do push notifications and all of this other stuff, so to fix it they just added a 10 second delay.

If you have a camera plugged into your Raspberry Pi that you can connect directly to, you can be streaming stuff straight from the camera, over that HTTP link, straight to your phone and there's no storage cost, there's no cloud cost and you get real time results from this camera. Now, that's the simplest one. Some people use that, they plug it in too.

I'm not a huge 3D printer person myself, but I know Tailscale is really popular with 3D printer people because you can load a copy of Tailscale onto your 3D printer firmware and actually monitor the status of your print from anywhere. Then people go and add that to plug a camera into the same thing that's controlling the 3D printer and point it at the 3D printer and you can see the exact status of your print.

So a lot of people doing stuff like that. There's a tool called Pi-Hole that a lot of Raspberry Pi people use that is basically a DNS filter that can help with malware and ads and stuff, so a lot of people set up Pi-Hole on their Raspberry Pi and they're very excited that this works when they're at home, but then when they leave home it doesn't work.

With Tailscale, you put Tailscale on the Raspberry Pi also and now you can use your Pi-Hole adblocker from everywhere. And that's individual use cases, right? There's companies doing everything you can imagine. You can load Tailscale into containers now, so a neat thing you can do is migrate a container between regions or between IP addresses or between your home device and something in the cloud, and it doesn't matter because it'll keep the same Tailscale IP address and name so anybody who's trying to access that container, they might see some downtime when you turn it off and then bring it up somewhere else, but other than that it's just migrated completely transparently.

Or you can use authorization keys if you need to spin up 1,000 copies of a container, each of those containers has its own little Tailscale connection and you can try to load balance between them, for example. So it goes all the way from pretty simple use cases to quite complicated ones. Or really a strangely popular one is Minecraft.

I don't know why Minecraft specifically, but people like to run their own Minecraft server on a Tailscale connected device and then share that device with their friends over Tailscale, because apparently there's a bit of an epidemic of if you run your own Minecraft server and put it on a wide open port on the internet, there are people out there scannign for open Minecraft servers and they come in and knock down your sandcastle. Which is really annoying, and so with Tailscale they can't do that.

Marc: Yeah, protect those sandcastles. That's cool.

Benjie: So my big takeaway that I definitely did not understand is that there's also a huge speed increase that we're talking about because of this direct connection, which is obvious now that we're talking about this. That's really cool. I didn't realize that one of the big benefits here is you're going to get faster, lower latency, faster throughput, that type of thing. I want to take it back a little bit, or not take it back but take it forward, actually.

This is a really cool project, obviously you've been working on it for some time, I think you said since 2019. Can you tell us how it started and how you started working on it as an open source project? When you decided that this is open source, the way to go about this is open source? Then also a little bit of the company building that you did and how you got here?

Avery: Well, that's a whole collection of things to talk about, let's see. I've been into open source for a really long time. My first experience with Linux was in 1994, I think, and I painstakingly downloaded the Linux floppy disks over my, I don't know, 19200 baud modem and it took hours and hours and hours, and I finally installed it, like, "This is so exciting. I didn't have to pay anything for this, other than huge amounts of my personal time."

But I was a high school student at the time, so my time was cheap. I've created a bunch of open source projects myself, the first one that had any sort of fame was called WV Dial, which I pronounced Weave Dial. It's basically a modem dialer because the modem dialer I was using, once I downloaded these Linux floppy disks was some terrible PPP+Chat nonsense that barely worked, ever.

And so we made this thing called WV Dial that ended up becoming the core of the product of my first startup when I was in university. I've also done VPNs before, the one that got the most traction is one called Sshuttle ("S-S-H- uttle"). It uses SSH to carry VPN connections over it, which is kind of interesting. You can think of it as the predecessor for Tailscale. So anyway, I was really big into open source for a long, long time.

The other two co-founders of Tailscale are also big in open source, David Crasha and David Carney, so we knew we were going to do open source pretty much from the very beginning. I also spent a lot of time watching which things work in open source and which things don't work in open source, and one of the things that used to be the default open source business model, I'm not sure if it is now or not, but it's such a bad business model.

It's like, "We will give the product away for free and sell support." This is the RedHat model of open source, one of the earliest open source business models.

To me it's such a bad idea because it creates these perverse incentives. If I make the product better and easier to use, then I'm going to sell less support which means I'm going to make less money. So why would I spend time and money improving the product when I can make a really big, complicated product and then sell support for it?

I didn't want to do that, I wanted to have a model where we could actually... the product was something that people would be willing to pay for. But I also wanted to make it so that we could have a lot of people using the product for free because I didn't want to try to collect a bunch of money.

If we want to revolutionize the internet, everybody should have Tailscale, so I didn't want to do something where everybody who ever used Tailscale would have to pay us money or it would never spread as fast as the internet itself. So we had to have a very, very large collection of free users. So the trick then was to make it so that giving away the service to a whole lot of people would basically not cost us very much.

So the architecture of Tailscale, this architecture where all the data doesn't go through us and only a little bit of coordination goes through our service which we can afford to provide for basically everybody on Earth without spending very much money, fundamentally comes down to the idea that we wanted to have an open source client, we knew that things would be too complicated and we wouldn't have any business model at all if we didn't have a central control service, but also we wanted to give away most of the access to this control service that most of the people in the world would ever use, right? So the architecture comes down, it has to be super, super incredibly cheap to run.

That's our open source story, I guess. The story behind why is the company even building VPNs and stuff is a different story, it comes down to the name Tailscale. Tailscale is the opposite of internet scale. Internet scale is what a lot of people find really exciting about software development nowadays, like, "Well, if I build this thing, what happens if it's going to scale to a billion users?" So they make a bunch of architectural decisions just in case their toy project expands to a billion users, and a lot of the complexity of rolling things out comes down to this initial decision that they wanted it to scale.

So Tailscale starts from the idea of like, "Look, almost all the projects executed by almost all the people in the world, even the people who are at these gigantic internet scale software companies, most of the projects they do don't ever need to scale to be that big. Why don't we help people build things that are small, and then in case they're lucky enough that they need to make it bigger, they can go back to these other, internet scale type needs?"

But it's a really underserviced niche, helping people do things the easy way. So Tailscale is all about reasonably sized stuff done the easy way. Actually we knew that before we even knew what product we were going to build. When we made a list of things that get in the way of doing things the easy way, the top two things in the list were connectivity and security.

Getting people connected to the thing you're trying to run and stopping the bad people from connecting to the thing you're trying to run. We built that first, and then we never had time to do any other parts of it because it turned out the network problem was such a huge problem that everybody was so interested in that basically it's all about refining the networking now. Not getting to the other 198 things that get in the way of starting a small project. That was a long winded answer.

Benjie: No, that was great. That was really cool. One thing that I'm just going to point out is that the way that you came to this is a primer that I hope a bunch of listeners are using if they're thinking about starting their own project. Solving problems that are real oftentimes are overlooked in our world, versus solving these hypothetical ones. Obviously everyone knows I love Kubernetes, but it can be overkill sometimes for certain projects.

Avery: If you want to see the most extreme example of that that we have, there is a blog post that we have on Tailscale.com called An Unexpected Database Migration, and that's where we talk about how we scaled from the dumbest database we could possibly get away with, which was a JSON file we rewrote every time anything changed, to a slightly more complicated database. This is still how more or less the control server at Tailscale works. We really, really did not over complicate it. We're like, "We need to be our own example of how this can be done."

Benjie: KISS, right? We all got that in computer science school.

Avery: Exactly, that's also why I talk about what happens when the control server goes down. It seems like a weird thing to bring up, why would you talk about the possibility of your server going down? This was architecturally a major decision for us, a lot of people get super obsessed with, "My service has to be up 99.999% of the time." That actually makes all of the stuff you do really hard.

Five nines of reliability is super, insanely difficult. But if you just say, "Look, I architected the whole thing so that it can go down for hours and who cares?" Not that I want it to be down for hours, it's never been down for hours, but if I just say, "Look, it could be," then I can just not do any of that stuff that I need to do to make it five nines of reliability. It makes my job so vastly much easier, and this is something that I think people don't realize anymore.

They're so used to just thinking, "Well, it has to be up all the time, right?" Well, why not design something that doesn't have to be up all the time, because your life will be much simpler?

Marc: Yeah. I think the story here, I agree, it's a really, really great story. I think even the idea of before you know what you were going to build you wanted to make sure that everybody could use it, so you wanted to architect and design the system in a way that your cost isn't going to scale with additional users. It has side effects too where it's faster, it's more secure, data is not flowing through your system. Not just for cost, but that's one of the risks of a VPN, right? Is that that data could be spied on and looked at by anybody in the middle.

Avery: Yeah. That's one of the reasons why we often get the advice to, "Please, would you stop calling your product a VPN because it has such a terrible reputation for privacy and security." Part of that is because there's two kinds of VPN, there's corporate VPNs and privacy VPNs. Tailscale is more like a corporate VPN but it can actually be used like one of these consumer privacy VPNs, because you could if you want set up an endpoint with a Tailscale instance on it and then route all of your traffic through that endpoint over Tailscale. It's called an exit node.

But when you do that, even if you're routing all of your internet browsing traffic over Tailscale, we still don't see it. The private keys for all of your devices, stay on your devices. All we do is exchange some public keys around, which means we can't decrypt your traffic at all. We have absolutely no idea what is going over your Tailscale network, and we never even see the packets.

Marc: That's super interesting. So taking a step back to the architecture and how Tailscale works, where it's built on top of WireGuard, I'm going to ask maybe the dumbest question. So if WireGuard is doing all the hard work of encrypting and handling the actual data transfer, and you have a really simple control plane, like a LookUp connection, what was hard? Tell us what was so hard about building Tailscale.

Avery: Yeah. That's the obvious question to ask, like, "Well, what are you doing all this work for? Couldn't I just throw this together in a weekend?" As people often say on Hacker News.

First of all, making things simple is actually not as easy as it sounds. The default state of software is kind of complicated and not very reliable, and so we spent a lot of time just taking this relatively simple concept and refining and refining and refining and refining.

So the downtime is less and the latency is less, just getting these updates delivered to, say, 10,000 nodes or maybe two nodes depending on how big your network is, in less than 100 milliseconds is actually kind of challenging to arrange for that to happen. If you just look at the basic idea of Tailscale, it's like, "Okay, I'm going to take WireGuard, I'm going to have a little web service that distributes WireGuard configuration files to my nodes, and I'm going to do something or other with NAT traversal with STUN and TURN. How hard can it be?"

There's all these components, I can plug them together, but if you did that you would probably not have the updates going out in 100 milliseconds after you make a change, you would probably not have a really nice set of semantics for the access controls. Integrating it with your identity provider, actually putting a packet filter on each of the devices that is going to run at reasonable speeds, supporting all the different platforms that we support which includes iOS and Android and Windows and all the different kinds of Linux and Raspberry Pis, embedded devices and so on. All these.

It does come down to fine tuning and just perfecting the same thing. So we run into pretty often when we're having sales and support conversations, there will be somebody in the IT department who's like, "Oh, I made a WireGuard configuration tool for our company. It's working pretty well." Then it's easy to list like, "Okay. Well, does your thing do this and this and this and this, and do you trust it? How many hours a week are you spending maintaining this thing?"

The nice thing about starting a software company is you can amortize the cost of making this one thing better across all of the paying customers. So it can be equivalent to them spending a few hours a year maintaining the product, but actually if you add up all of those few hours a year, times thousands of customers, now we've got time to actually make something really beautiful.

So it's hard to really point at one part and say, "This was the super hard part." It's all about the refinement, and we find whenever we make the product a little bit more refined, you can actually watch the growth rate curve of Tailscale go up. People really love this refinement, they want their network...

When I talked about five nines of reliability, right? They need a network that's going to have five nines of reliability because the company grinds to a halt when the network stops working. But it's the architecture that makes it five nines of reliability, it's not the fact that our server is up five nines of the time.

Marc: And to be clear, obviously Tailscale is solving a hard problem and that hard problem isn't encryption here or anything like this. It's literally taking something that was really hard for folks who were willing to put a lot of time into it, and maybe out of reach for folks who didn't have that time and effort to put into solving this problem, and making it not just easy, but pleasant and reliable. Just something that's a good part of their day, that they're connecting to these. There's massive value in that, and making it so everybody has the best way to connect to those remote machines out there.

Avery: Right. And that's kind of our secret sauce. We're doing something a lot of network engineers can imagine them putting together, but we're doing it in this really beautiful, elegant way that just surprises people every time with how nice and simple and easy it is. People don't believe that you can install Tailscale in five minutes and have it finished.

That's the level of refinement it's reached. Nobody has ever deployed a VPN in five minutes, but that's what Tailscale is about because we put all of this work in behind the scenes. "What could possibly be so hard about this? I just click, click, click and I've got a VPN." Well, a lot of stuff happens inside that box.

Benjie: Yeah. I think that the lesson that you're trying to explain here to us is something that we all should apply. It's definitely something that we think about at Shipyard and I know I'll speak to Replicated, trying to simplify really complicated things and abstract it away is really the goal of a successful company and a successful product.

Also that multiplier effect I think is a really cool point you made as well, that when you're subsidizing making a product really rock solid, really well thought out and every time you improve it goes to the masses. So what's Tailscale written in and what open source license is it?

Avery: Ah, important questions. So most of Tailscale is written in Go, there's a few frontends like the iOS and Mac OS ones that use Swift that then call into the Go backend. Which is a little bit unusual on those platforms, but we made it work so that basically the core of Tailscale is always the same code running in Go. It's of course been a little bit tricky getting Go to do the stuff we want as fast as we wanted, but actually it's getting faster and faster all the time and I think we're really happy with that choice.

Open source license, we used... Actually, let me try and remember specifically. What we did, I think, was copy whatever the Go project used because it turned out some of our earliest developers are actually on the Go team or were on the Go team. They really liked the license and they wanted it to be compatible, so it's some kind of MIT license with maybe an extra exception or something like that.

I forgot exactly what we did, but it's very open. The only exception is what we call the coordination server or the control server, which is not open source. However, there is an open source project called Headscale that we didn't start but we've been assisting them and throwing them information every now and then on how to do it.

They've replicated the control server so you can take Headscale, run it completely on your own and then combine that with the open source Tailscale clients and have a completely open source Tailscale deployment that doesn't depend on any proprietary stuff. Incidentally, we just hired one of the developers of Headscale recently, Christopher, who is going to be working with us on both the Tailscale side and the Headscale side.

Basically, we were always thinking of making some kind of open source control server, we never got around to it and we're really happy that Headscale exists so now we can just contribute to it instead of having to start our own.

Benjie: So why not make the control plain open source then, if you guys are open to that anyway?

Avery: We actually just made a blog post a couple weeks ago about exactly why that is, but the short version is the code for the control server was... I don't know, you know how proprietary software gets where there's a lot of hacks and there ends up being a bunch of network specific hacks as we adopt a particular customer with a particular need, then we try to remove those hacks over time and so on. The code just got more and more messy and less and less the kind of thing we'd want to present to the world.

Benjie: I don't know what you're referring to, in all the code that I've ever been a part of-

Avery: I mean, maybe we're special. Maybe only we have ugly code at Tailscale.

Benjie: Those Go developers, those guys are pretty suspect so that's probably the problem.

Avery: Yeah, we were going to get to it but we would've had to strip a bunch of stuff out, it would've been a big clean up job. There would've been a question about are we actually forking this thing or are we maintaining two different projects, or are we going to somehow keep our commercial, weird hacks in there at the same time as having the open source version?

The Headscale project basically solved it for us, they wrote it from scratch, they have some things simple. It won't scale to billions of users running on a single control plane, which is what Tailscale needs to scale to. But it will scale to your organization running on a single control plane, and so I think it's a good balance.

Benjie: That brings up a good question, and this is a little bit about the business, but how do you guys make money? How do you monetize this? Because you had mentioned the professional services not being something you were interested in, but obviously you're running a business here so how do you monetize?

Avery: Right. First thing to know is that the reason that we don't really recommend for most companies to use the open source control plane is it's the only thing in all of Tailscale that needs to have both access to all your public keys and be public facing on the internet. Which is a really bad combination, this is a security nightmare.

The control plane is essentially a certificate authority that has all of the keys to your whole network, so if you're running one of these things you have to spend a lot of time securing that thing because it is where the attackers should focus all of their attacking. So the value Tailscale provides is we run the control plane and we actually have a whole bunch of security policies and access controls, and only a few people can do this and only a few people can do that, there's firewall layers and there's logging and auditing and SOC2.

All of that stuff around this control plane to make sure that we are actually running it correctly, so that then the stuff you need to deploy, you never have to open a firewall port. If you never open a firewall port, like 97% of all the possible attacks are just gone. So that's pretty beautiful, the beauty of Tailscale is not having to run this one little bit that ties everything together.

So the value that people pay us for is not having to run that thing, right? That's the best part. That said, we can segment Tailscale users into roughly three categories. One of them is individuals using the free product that we intend to always keep free because that's not who we want to collect the money from.

Those people tell their friends, there's a huge word of mouth spread of Tailscale, almost all people who find out about Tailscale find out about it from their friends who are already using it, which is great. Some of those people bring it to work, usually on the development team. They'll try it out inside the company, whether under the table or officially, they'll spread it around to a few more of the developers that get excited.

As things grow, then it goes to the IT department, people are like, "Well, why are we using this crappy legacy VPN and Tailscale? Why don't we just drop the old VPN and use Tailscale for everything?" Next thing you know it becomes a top down sale through executives. Then there's a few much larger enterprise customers that we work with, and I think over time there will be even more of those enterprise customers.

But it hasn't been our primary focus of the business up until now. The things that these companies pay for, first of all, the ease of use and the security guarantees that Tailscale can provide. Your security problems go away if we're handling all of the difficult stuff in the control plane, and the stuff that you're doing doesn't have any open firewall ports.

But the next part on top of that is access controls between groups, being able to have fine grained role based access control and that kind of stuff, that as companies get bigger they really want to be able to say, "This group of servers is only for this group of users. I really need to be able to offboard someone extremely efficiently and make sure they're cut off from everything instantly." These kinds of things are super valuable, where you wouldn't have been able to buy them for any price a few years ago.

Benjie: Right. And also that onboarding use case you explained also is super valuable.

Avery: Yeah. The onboarding is really nice, you download Tailscale from the App Store on your Mac, you log in with your corporate SSO identity, and now we already know what group you're in because of your SSO. It drops you into having access to all the right things, it puts your SSH keys in the right places so you can just SSH to all the right devices. There's like nothing to do.

Benjie: That's pretty cool, that's pretty cool. I feel like there's some version of using Tailscale for Git, somehow, that would help me. But I'm not sure if that actually makes sense.

Avery: Yeah. Well, I mean you can host your own Git servers, of course.

Benjie: Right. No, I'm thinking about the GitHub integration in particular. How does that work?

Avery: Oh, there's lots of GitHub integration. So for example, you can put Tailscale in your GitHub actions, we have a GitHub actions plugin, and so you can do continuous deployment, for example. So something builds in a GitHub action and then it pushes something over Tailscale to either your code signing server or to something on your internal network that then allows it to actually continuously deploy the software. Or it can push it to, say if you have a mobile app, you can have a network of little mobile devices that actually can run tests that are connected over the Tailscale network. So people use it for all kinds of stuff in there.

Benjie: Super cool. Wait, sorry, you didn't answer one question though. How do you make money? What do you charge for exactly?

Avery: When you get into the multi user corporate plans... We have, incidentally for people doing free software projects or family stuff, we have a community plan that's free. But if you're a corporation we basically charge a per user, per month fee for either the team plan or the business plan. The main difference between the two is the advancedness level of the role based access control.

Benjie: So just basically enterprise controls, which makes perfect sense. So if I'm trying to protect my sandcastles I can go use the free version?

Avery: Exactly. And you can share with your friends on the free version, because each of you has a free copy of Tailscale and you can share nodes out to other people. But there's no central control then of who gets to share what. If you're on the corporate plan, then the administrator of your company has a global view of everything going on in your network.

Benjie: I feel like Marc would try and knock down my sandcastle, so I'm not sure I'll share that with him. But maybe.

Marc: You can choose, that's the magic. Right? But it's interesting to note that it's per user that you're actually... the commercial side is per user, not per device. The size of my network, the number of pods, whatever, it doesn't matter?

Avery: Yeah. We thought about per device, and it didn't seem like... We know developers are cheapskates, right? That's the one thing we know for sure about developers. We didn't want a developer to have to think every time they're spinning up a device, "Do I want to spend another however much per month on this device? Because it's just a toy." When you do that you're really going to slow down the adoption.

We were just like, "Look, when people are hiring a developer, they can just include that in the budget. Okay, here's an extra $10 a month on top of the 10,000 or 20,000 a month that I'm paying this developer. I think I can afford that, the budgeting is done." And now the developer can just do whatever they want, that was the magic we wanted to go for.

Benjie: Yeah. That dovetails into the strategy you were talking about earlier on about you're not trying to squeeze everything, you're just trying to be a productive tool. By the way, we prefer the term fiscally conservative to cheapskate. But I think everyone-

Avery: Might be an Iowa thing. I think in New York it's the same word.

Benjie: Yeah, I'm in New York. You know that Avery, so I'm just trying to represent some of our other brethren in the middle, maybe. Okay. So how about roadmap and the community? How has building this as an open source project... Obviously you've hired some people from the community, but tell us a little bit more about the Tailscale community and what it's like interacting with them and how it's been running an open source project with obviously a bunch of contributors?

Avery: That's a good question. It's interesting. I think the Tailscale client project, I would not say has had a ton of external contributors. There's been some. But for better or worse, the networking stuff that we're doing is really advanced and there's not that many people who have a lot of spare time who wanted to work on that kind of thing. Headscale of course has a bunch of community developers, because that's more of a client-server application. The control plane is much more standard web stuff, so lots and lots of people are able to contribute and they do.

But I think the place where the community really shines with Tailscale is building stuff on top of the core of Tailscale, and we're seeing not just in open source but there's a whole bunch of partners we recently announced collaborations with, people integrating Tailscale into their products. All kinds of people will post, "Here's how you can use Tailscale with this kind of container system. Here's the Docker file you need to do this and that. Here's something you can build on top of Tailscale's inventory system so if you want to keep track of all the devices in your company you can use Tailscale's node lists and track exactly what's going on, when they come and go, what geographic region they go through and so on."

The community is more of a community of developers who use Tailscale and then help other developers use Tailscale for things, as opposed to necessarily extending Tailscale themselves. Does that make sense? It's a little unusual, but I think it's-

Benjie: No, it makes perfect sense. Literally at Shipyard, we're evaluating some CLI stuff that we're working on and it's like, "Hey, do we embed KubeCuddle? Do we kind of use Tailscale?" And I'm more convinced than ever that Tailscale seems like a really forward thinking solution, so I very much understand that. What about roadmap? Anything interesting on the roadmap that you want to talk about?

Avery: We try not to talk too much about future stuff because we like to keep it a surprise as much as possible. There's some performance improvements on the way, I think we're going to be making an announcement about something like that in the next few weeks and I think people will be extremely surprised at how fast you can make WireGuard go. There's other features about the control plane.

I think most of the stuff on the roadmap in the near future is going to come down to partnerships, where we make it easier than ever for other companies to work with Tailscale. There's one partnership we announced already for Materialize, where they have a data flow product and their data flow product, you can paste a Tailscale access key into it and then all of a sudden their SaaS product shows up on your private network and can access your private databases to help with the data flow stuff.

Which is something that they never could've done otherwise, and this is because they embedded Tailscale into their product, right? I think there's going to be lot more companies doing stuff like that in the next little while. So we've been scrambling to make sure all of the APIs and stuff are in place so everybody can build this really neat integration on top.

Most of Tailscale itself, yes, we're going to refine some of the UIs and stuff, we're going to make Taildrop even better, we're going to keep improving Tailscale SSH. We didn't talk about Magic DNS which is the thing that makes it so that you can just ping host name, and that host name actually works. All of that stuff is going to continue to get improved, but you shouldn't see too many giant, thundering changes. It's going to be mainly more stuff built on top.

Benjie: That's cool. That integration stuff sounds super cool. It's a very different use case than me wanting to connect to something, but it's like two SaaS services connecting to each other. I don't know, the way you described it was like it's a reverse connection where I'm actually integrating the SaaS service into my environment, to get access to sensitive connections that maybe my VPC setup or my routing tables otherwise wouldn't have allowed incoming connections to.

Avery: Yeah. There's so many people right now, you've probably run into it, but they use these so called IP Allow lists on their server, so if you pay the extra expensive plan for such-and-such SaaS service you can then make a list of IP addresses that are allowed to connect in and login as you. Then in order to make that work, you have to funnel everybody through a particular IP address, often using a more traditional style VPN.

Then that IP address changes because IP addresses change sometimes and, "Oh, I got cut off from my service so now I have to go to all the other services that I configured these IP Allow lists with and fix the list of IP addresses so that people can actually get in again." It's really tedious and it's also bad for security because it's actually not real security.

Somebody can be a man in the middle of the network and just pretend to be one of these IP addresses and walk right into this allow listed service. So Tailscale being used for that kind of stuff, connecting one service to another service is one of the neatest things it can do. A really simple use case is, even if you're not using Tailscale at your company at all, you install Tailscale on one server node and you ask your partner to install Tailscale on one client device.

Now that client device can access your internal API server while you're working on this partnership and it wouldn't even cost you anything because you only need one user account for that. But it's the beginning of integrating Tailscale into a company, but it's this kind of cross integration stuff. It's so much nicer than trying to create a VPC to VPC connection on AWS, between companies where you have to agree on IP addresses and not using the same subnets on your two VPCs and you have to be in the same region, and you have to setup all these keys.

Forget about key rotation, because now you have to convince both companies to rotate their keys at the same time, all that other stuff, and it just goes away. I install Tailscale, you install Tailscale, and now it's gone.

Benjie: Cool. So for us, we have a SaaS service and we definitely get some requests from some of our larger customers that they want to make sure from a security perspective that it's a SAML login, what can they do to make sure that nobody that's not authorized to access their team account inside our SaaS service is able to. But they're all working from home, developers today, right? And so is there a way that we'd be able to use Tailscale to solve that problem, where we can say, "Yeah, here's a Tailscale team and if you're able to go through that you can actually access this team in our SaaS service"? Through some obviously deeper integration?

Avery: Sure. So the simple quick answer, or the answer a lot of people are doing right now which requires no special software integration is you just set up a Tailscale subnet router, you configure that subnet router for a couple of IP addresses that are the IP address of the public facing server for this customer.

Then that customer has that subnet router in network, so everybody at that company using Tailscale will funnel the traffic just for your service through that one subnet router and it'll always come from that one address, and then you can create the IP block list for just that one address. Even though they're not all in the office. So you're not running all of their traffic through a single point, you're just routing only the traffic that's intended for you through that single point.

Marc: Yeah. Similar to the exit node that you had the link to earlier, but it's targeted?

Avery: Yeah. But not all of your traffic so it doesn't slow things down or add latency for the average thing you're doing. Then the more fancy version is if you can get them to use Tailscale, you could actually share your Tailscale node with their network and then you'd have an actual encrypted connection. But then they have to be on Tailscale and you have to have Tailscale integrated, so a little more work to do. But that's what Materialize did.

Marc: Yeah. That's super interesting. Thanks.

Avery: No problem.

Benjie: Well, I am going to have to go listen to this episode three times to make sure I understand everything that you said. But this has been super informative, really interesting. I wish we had those extra five hours because I think I have 400 more questions for you, but I think we're up on time here. Marc, did you have anything else to ask before we wrap up or anything else?

Marc: No, I think we went through it. Avery, did we miss anything that you want to share?

Avery: I don't think we've missed anything. If I was to very quickly summarize the thing I try to tell everybody about Tailscale is it really does only take five minutes. I know you don't believe me, but please just try it. It's in the App Store. You don't have to do anything, click, click, click, you log in with your Google account or whatever, and now your connection is made.

When people actually try it, that's when they realize how amazing it is. You can hear people tell you 100 times and not believe them, but when you try it you can get it. We hear that over and over again, it's kind of funny how many people are like, "Okay, I set aside my Saturday to install Tailscale, and I'm like 15 minutes in and now I have to find something else to do with my Saturday." So that's the most important thing about Tailscale, is I know you don't believe me, but it is that easy.

Benjie: I for one, I've played with Tailscale in the past. But I'm going to take some time this Saturday and install it somewhere as well. I'm convinced. Well, look, Avery, we really appreciate having you on. And you know what? Maybe we'll have you guys back and we'll talk about some of the applications for some of this stuff in the CNCF ecosystem. But I think we can all extrapolate how cool this is and where all of it's going. So thanks so much for being on and really appreciate the time.

Avery: Awesome. Well, it's been great to talk to you as well.