Ep. #5, This Machine Kills Vim with Paul Biggar of Dark

Heidi Waterhouse: If you had a law named after you like Conway's law, what would it be? Biggar's Law.

What do you wish people knew in a way that they bought copies of The Mythical Man-Month? I know that's a hard question.

Paul Biggar: My mind is going blank because I have a lot of these things where it's, "When I'm King, this is going to be different," and for some reason none of them are popping into my head at the moment.

Might want to come back to me on this one.

Heidi: Yeah. If you think of it later, let me know.

Paul: Yeah.

Heidi: So what are the tools that you use when you're thinking about trying to build?

I said to somebody that Dark was like the early car dealerships, where it was all in one, you didn't just buy a car, you bought a place to buy gas and all of your service.

How do you think about that? How do you understand what you're doing?

Paul: So I think the most important thing is we recognize that it has downsides and that what we're building is not for everyone.

I mentioned earlier, people can't use Vim.

You can't write Dark using Vim. You can't run it on your own servers.

You can't run it not in the Cloud and you can't use Python.

So these are the major problems that people identify, sometimes they call it vendor lock-in, but there's a lot of reasons why people are afraid of vendor lock-in and we embraced that.

We embraced all of these things are downsides and if they're deal breakers for you, then that's fine.

Just keep using whatever you're using, keep using the Cloud, keep using Python, keep using Vim.

Don't use Dark. And once we do accept that, then it's very freeing.

We're recognizing the constraints of our system, and then we're building great stuff within that constraints only for the people who are able to accept those constraints.

And I think that if we instead had spent a lot of time saying, "Well, we need to support Python and therefore we also need support Node and we need to support Rails and we're building all this."

Or, "We need to allow people to run this on their own services or their own servers or using Vim."

I'm not sure we would have made anything of interest at all.

I don't even know what it would've been.

It would've been boring, probably wouldn't have done anything.

Heidi: That's actually a super interesting point to say, "These are not the things I'm doing."

And I think it's a common problem, especially as people try and scale, to say, "We could do all the things."

Paul: And you hear the feedback from people, "I would use Dark, except I need to run it on my own servers."

Heidi: And you're like, "Okay, then you can't use Dark."

Paul: Right.

Heidi: But it's hard to say that without being a jerk.

Paul: Well, yeah.

I've practiced a couple of ways of saying it, "Well, that's unfortunate. Maybe you'll change your mind in the future."

No wait, that sounds conceited.

The other way to do it is to say, "Our customer feedback is telling us that we need to be able to run this on our own servers or that we need to use VIM or support Python."

And the important thing is what I said earlier the vision, but maybe it's just the conviction that there's value in what we're building and that the knowledge that those things are the wrong direction, and those are the things that we aren't going to build.

Heidi: I think that's actually a thing that a lot of founders struggle with is saying no to cool ideas or possibly market pleasing ideas in order to pursue this pure vision.

Paul: At the start of Circle, the one that we always heard was, "I'm never going to put my source code in the Cloud."

And they were lying to themselves as well, probably, but they already had their source code in the Cloud, it was on GitHub.

So the thing that was important for us was to try to understand the message behind what they were saying.

So they were saying, "I'll never put more source code in Cloud."

What they were actually saying is, "You're a tiny startup. We don't trust you yet."

Heidi: I think this is an interesting thing about talking to people who don't wear masks.

Paul: Politically controversial talk. I love it.

Heidi: I know. But when you say, "Why don't you make them wear a mask?"

They say, "It's hard for me to breathe."

Well, we have science that says it isn't actually reducing your oxygen flow any.

We've done it with a bunch of people, including asthmatic people.

What's happening is you're having a panic attack, but you can't say to people, "You're irrational and having a panic attack," it's not convincing.

And we know from all sorts of public health things, that the way you actually deal with this is to say, "I know it's uncomfortable," to start with sympathizing and say, "How can we work through this together? How can we make this more tolerable for you?"

Rather than denying that experience.

So rather than denying the experience of, "I don't want to put my source code in the Cloud," what you're saying is that you had to dig in and say, "You're uncomfortable because you're scared that it will go down."

"Okay. How do we mitigate that in a way that's not dismissive?"

Paul: And I think the other advantage that we have when you're building a startup is that there's always later.

You have to wear a mask now if you're going to prevent coronavirus now, but, if you don't want to put your source code in the Cloud, maybe you will next year, maybe you will the year after.

We have this concept of the crossing the chasm, and there's the innovators, and the early adopters, and the late adopters, and the late majority and all that sort of thing.

And when you were looking for product market fit, being able to take the people who are not in the category that matters right now and say actually their feedback isn't useful.

And this is something that venture capitalists do very badly.

Heidi: You mean "it's the Uber for-- "

Paul: Well, I specifically mean around developer tools because most venture capitalists do not actually understand how software is built and definitely don't understand the developer mindset.

And so they rely on proxies, that they have a friend that they reach out to, let's say that they're the head of infrastructure at Pinterest or Facebook.

And they say, "This Dark thing, it looks interesting. Will anyone ever use it?"

And the most likely response, if you're the head of infrastructure at Pinterest to say, "Fuck no, there's no way I would let that anywhere near my systems. That's ridiculous."

You're asking someone who's in the late majority on software systems, who has a very low risk tolerance and very little interest in innovation.

And you're asking them, "Do you want to use this shiny new tool that will change how you make software?"

Of course the answer is going to be no.

Heidi: I have a friend who's a sys admin and she's like, "I would love to use Terraform. I would love to learn Kubernetes."

She works in IT for a university.

"I'm working with something called CS Gold. It's 40 year old software and the licensing model is literally impossible to put in the Cloud. I can't use Terraform to do that. And so why do you keep asking me about going to the Cloud? It's not possible. We have to have a licensed server physically located in a data center."

Paul: Wow.

Heidi: And there are a lot of people all over the world who are dealing with this kind of problem.

And we can't serve them all with our shiny newness.

So one of the interesting things about product market fit is, who could even make a change?

Paul: If you're talking to, or getting feedback from a group that has to deploy a licensed server on the local thing, they do not want Dark, Dark has no value to them at all.

Heidi: And I think it's really easy for VCs, everything that's older than four years drops off the radar.

I'm like, you understand that all of our insurance companies are running off an OS390 system in a basement, right? And it's just layers of gel on top of it.

Paul: There must be someone who's cloudifying mainframes.

Heidi: I've actually been to some really interesting DevOps on the mainframe talks at SREcon, but we'll never get rid of COBOL until we write a programming language that actually does math properly.

Paul: That's right.

Heidi: There are these constraints that are not visible in the developer tools world, because-

Paul: So COBOL, doesn't use troubles?

Heidi: It's a floating point thing.

I honestly only sort of understand it, but if you're dealing with financial data, it matters quite a lot how many decimal points of accuracy you get out.

And most of the modern languages are like, "You know, ish, more or less."

Which is fine for everything but financial systems and space math.

Paul: I'm thinking somewhere there's a business where you take an OS390 emulator and you put on Kubernetes and suddenly you can be in the Cloud, but also mainframes never fail.

Heidi: As long as you can get the parts for them.

Paul: Yeah. There's a massive redundancy and I remember reading about a mainframe that was transported across town, one piece at a time.

And it works the entire time despite being split across two locations, as it was being transferred.

You're not getting that in the Cloud, certainly not in Kubernetes.

Heidi: And I think it's interesting because there are different risk factors in different tolerances, but I think it's very easy for, internally at LaunchDarkly we call it the cool kids the VC backed developer tools, we're all selling to each other and then we're trying to get out into enterprise and have this different experience and it's a shock to the system when we start doing it.

Paul: It's funny. I'm not a buyer of any of those cool kids software. I actually don't understand dev tools these days.

I feel like there was just an explosion of dev tools, who was the first wave, technical CDI part of the second wave.

And we're on the fifth wave and there's thousands of developer tools software.

And they're like, "I'm AI for Kubernetes." I was like, "No you're not." I'm trying not to name a specific company here because I think you're all doing great. I'm just not a consumer for a shiny new tool. I'm a consumer for, "This tool works pretty well."

And my hope with Dark, at least, is that it's so much better than the way that you do it, that people will switch over.

But I don't have a need for something that's slightly better at one of the things I'm already doing and slots in nicely.

I think it's actually really hard to build dev tools because you have to slot in nicely.

But if you slot in nicely, it's also hard to have that 10 X improvement you need for people to switch to your software to might even be better off not slotting a nicely and just having a whole sale replacement, which is what we're doing.

Heidi: I was going to say, "Sounds familiar."

And I think it will be interesting to see if it is easier to get people to start using Dark as they start projects.

Rather than trying to replace what they already have.

Paul: What we're telling people is, "Build your thing and Dark. And if you find that it's not able to accomplish what you need to, it will be fairly trivial to rewrite it in something else."

And what you got out of it is the prototyping.

The figuring out what it was that you're building and all of that was super fast and then it doesn't scale or Dark doesn't scale yet to the size that you needed to.

It's, okay, well, now that what it is, it'll take you a weekend to rewrite that in note or whatever.

Heidi: That's actually a super interesting pitch that I hadn't heard from Dark before that you're a rapid prototyping tool.

Paul: Well, we are not a rapid prototyping tool, but worst case you could use us as a rapid prototyping tool if we did not satisfy your needs.

Heidi: It's an interesting. There's not a lot of cost.

One of the things we spend a lot of time talking to people around feature flags is, where do I start?

I have years and years of features, how would I even get going on that?

And we're like, start with the new stuff and then start with the stuff that hurts.

Almost no one can afford to do a complete shift, a complete changeover.

And I'm sure that's part of what you're struggling with too, at what point in the company do you change your development method?

Paul: The new stuff. With testing at Circle people would say, "Oh, we don't do testing. Where should we start?"

And I'm just like, "Write one test." There's an overhead of adding the test harness and adding CI and that sort of thing.

So literally just test one thing and once you've got that in, it'll be so much cheaper to add the other tests, that you'll get real value from them.

Heidi: That's a great point that the longest journey starts with a single step or a single test.

Paul: I'm sure it's the same with future flags.

You want to add feature flags. Whatever feature is coming out next, add it to that, and then you'll pay for it and then it'll be cheap for the later ones.

But you don't want part of your journey to adopting feature flags to be like, "Oh, we also need to train these teams on doing it. Nothing has shifted. We get no value until every team uses it."

Well, that's not true.

You get value if one team uses it and accomplishes something with it and you can grow from there.

Heidi: Incremental and feedback appear to be the watch words.

Paul: And low risk, I think.

Heidi: And low risk. So as we wrap up, who should we be talking to?

Paul: I already mentioned Charity Majors.

I think charity is one of the most interesting people in technology.

Heidi: Yes, I got to be on a panel with her earlier this week and I love it because she's the wild-eyed radical and I'm the smooth, persuasive radical and together.

Paul: There's nothing she won't say. And there's no one that she's afraid of. So it's fabulous.

Heidi: It's amazing. And if we were going to go feed something that was part of your influences for creating Dark, what do you think it should?

Paul: So you mentioned The Mythical Man-Month, that was where I got the phrase accidental complexity and essential complexity.

I think it is a book that is no longer completely universally true, but is so close to universally true that it's a really excellent read.

The rest of my influences are books about type systems and functional programming, they're probably not the best place to start, but mythical Man-Month is wonderful.

Heidi: It is a great book. And would you like to promote anything?

Paul: What I like to promote anything? I've been promoting Dark this entire episode.

Heidi: Well, we knew that when we asked you.

Paul: So I do a podcast with Edith, it's called To Be Continuous.

And it's funny, every episode we plug feature flags, all the time.

It just comes up all the time. So finally I get to my own plug in.

Heidi: I think one of the interesting things is people are like, "So our feature flags a pattern?"

And I'm like, no, they're like an Uber pattern. They're a tool that has many patterns.

There's testing and experimentation and kill switches and circuit breakers, these are all patterns.

But sometimes I worry that I found a little unhinged when I talk about it.

Paul: So this is also one of the problems with Dark. It's like, "What can you do with Dark?"

Well, it's a programming language.

What can you do with a programming language? You can do anything.

And people really want Shopify, I can build a shop online.

And it's like, "Can you build a shop online with Dark?" I suppose, it would take you a while.

But you have program lines, you can do literally anything.

Feature flags, you can do this huge set of things with them and it's telling people why this thing needs to exist when it's so general and so like widely applicable.

A feature flag is an if statement, trying to sell people on an if statement, it's an if statement you can control from your dashboard, it's like, what can you do with that?

Well, quite a lot of things,

Heidi: What do you need to do? It's an interesting problem.

And it's been fascinating for me to realize that a lot of what people are asking me as a developer advocate for is opinions.

They really want an opinionated rule set for how to use feature flags in their situation. I'm like, "Oh, it totally makes sense that you want that."

You're looking at butter and flour and eggs. It could be scones, it could be pancakes?

And they're like, "Please teach me how to make pancakes."

Okay well, if that's what you want, here's how you do it.

Paul: It's interesting because it implies that the people who are going to use this in the absence of this are wild-eyed technologists who can fill in the blanks themselves, but you can't get across the chasm without saying, very specifically, "This is how you make pancakes," because those people are looking for pancake recipes, they're not looking for I've got flour and eggs. What can I do?

Heidi: They're not people in search of a tool.

They're people in search of a solution.

Paul: And you maybe pop up like third blow stack overflow.

And really what they want to do is cut and paste.

And after they get through the first two and realize they can't cut and paste their way to a solution, it's like, "Maybe I should use the service."

Heidi: And I think that probably affects Dark a lot too, where you're like, "Okay, what do you want? Tell me what it is you're trying to do so I can help you."

Paul: So we tried doing a little bit more specific stuff, so we're like, "All right. Dark is the best place to build a Slack Bot," and great.

And what do you do with a Slack bot?

Well anything, Slack Bot is just a way to communicate to a service which does something else.

And so then you'd also need the ability to be able to do everything else.

And we decided we were going to use flour and now what we're still not there.

Heidi: What is the quote, "To make an apple pie you must first create the universe."

It's really interesting to try and be a revolutionary and still cross the chasm enterprise people.

Paul: I never thought of myself as the revolutionary.

Heidi: So I have a talk that I've been giving about what Martin Luther and Che Guevara have in common and they believed in universal literacy.

Paul: Okay.

Heidi: And also the overthrow of the celibate priesthood, but mostly universe literacy, let's go with that one.

But when I think about being able to control features, I don't want it to be just developers who can control features.

I want it to be product management and I want it to be customer support.

And I want this universal literacy and I'm like, "Oh, I'm attempting to distribute power to the masses."

Paul: I'm sure you've thought of it this way but the thing that it occurs to me is that in the past only developers were able to write the if statement and go through the Git commit thing.

Are you guys low code for if statements?

Heidi: Absolutely. We are no code.

You were talking about a Slack bot.

I'm saying I have a Slack bot that will let you, once your developer sets it up, turn on and off whatever.

Paul: Of course, yeah.

Heidi: And it's so cool because I'm like, okay, so the customer support wants to see what flags a person has on or off in Salesforce.

So you go into Salesforce and you check the customer and you're like, "Oh, okay. This is what they have on."

It's so powerful. And I think that a developer tool is just the beginning for how much we can empower everyone who's dealing with software.

Paul: I mean, this is LaunchDarkly really developer tool.

Heidi: So it depends on who you ask. I don't think so.

I think that the closest analogy to what LaunchDarkly is, is actually middleware.

Which is not very exciting, except to me, evidently I'm super excited about it.

Paul: I've just been building rebuilding Darks, middleware stack.

And I was thinking about how could a Dark developer decide to change the middleware stacks?

I've been thinking about like feature flags of middleware stacks, as opposed to like feature flags in the middleware stacks.

Heidi: So for those who are not deep in that particular set of weeds, middleware is the stuff that connects your software to other software at its most basic level.

It's the API level of your software for itself. What do you think?

You all can't see this, but Paul is blinking at me because this is not at all how he would describe it.

Paul: Slow blink. I started my career in middleware, in the games' industry.

And I think middleware in the games industry was just what they call dev tools.

People were building physics engines and audio engines and compilers and that sort of thing.

But it was interesting because it was the only part of the games industry that didn't have these crazy deadlines and death marches and that sort of thing.

So you got to be in the games industry without having a horrific life.

Heidi: Oh, that's not the worst. I spent five years working in middleware that sat on top of IBM, WebSphere, MQ.

Paul: fun.

Heidi: It was so exciting.

Literally their software had originally been born as sewage monitoring software and through a series of transformations, because it turns out that things flowing through a pipe, whether or not the pipe is virtual, need to be monitored the same way.

Is there a backup?

Paul: Really?

Heidi: When you think about it.

Paul: Wow. I mean, it seems like a reach, but I believe it.

Heidi: So MQ series is just making sure that messages get from one place to another.

And this software that we were working with was making sure that there were no backups in the system or sudden outages, like almost as worrisome as an overage in your sewage pipe is an absence because where did the shit go?

Nowhere good.

Paul: Wow.

Heidi: And it was incredibly unglamorous.

And also I got really fast at installing databases because I was a technical writer.

So I had to keep reinstalling the database to check the instructions.

Paul: Wow.

Heidi: I couldn't use them at all, I could just install it. That was irrelevant.

But it was this thing that connected ATM's and airplanes and everything in the world works through message queuing and still kind of does, you have to have something that makes sure things get from one side to the other.

And when I think about what LaunchDarkly may grow up to be, dev tool is like stage one and stage three is a way to make sure that everybody is getting exactly what they want when they need it.

And if they get exactly what they want, when they need it with targeting and rule sets, then we're not over delivering things, then we're making the internet more efficient.

Paul: So you think at some point parents are going to use LaunchDarkly to turn off the internet until kids have finished their home.

Heidi: Yes. And they will never know they're using LaunchDarkly.

It will look like they're using Comcast to do it, or it will look like Net Nanny, and when their kid hits submit on their homework, then we flip the flag that turns their internet back on.

Paul: Automatically?

Heidi: Yeah.

Paul: That's cool.

Heidi: Right.

Paul: It's all the same concepts.

Heidi: It's an if statement, if you turn in your homework.

Paul: Yeah, it's an if statement-

Heidi: So interesting because so many times I've talked to people who are like, "It's a fancy of statement, I can write if statements by self. Thank you."

And I'm like, yeah but it's very fancy and distributed.

So I will say Paul, that this is certainly like the Heidi specific vision of what LaunchDarkly maybe.

And I don't know. I think that we have a lot of space that we're still working through to be the best developer tool that we can be right now and we're still anticipating what the market's going to need.

Paul: I've been recently doing a rewrite of our backend technology stack.

And one of the things that is, it's not a motivating factor exactly but it's very exciting, is that it will finally be in a stack that has a first party LaunchDarkly STK.

Before, it was written in OCaml. And we were not writing our own OCaml client for LaunchDarkly, but now I'm excited to finally use LaunchDarkly after all these years.

Heidi: I read your article about OCaml and I thought it was really an interesting reflection on a bet that you lost.

Paul: Oh, I think we won the bet.

Heidi: Did you? Okay.

Paul: At least we won it for two years. Maybe we started losing it after that.

The decisions that you make at the start, the main thing you're trying to get out of them is the local win.

I think we got the win for a couple of years, and we're now at a point where it's a loss, but I think overall, we won.

Heidi: That's a good reflection.

The things that we need early are not the things that we need later.

Paul: Oh yeah, absolutely. Now we need things like LaunchDarkly.

Heidi: Maybe this is Biggar's law, grow with what you have until you need something else.

Paul: I do believe something like this, which is solve the problem that's directly in front of you.

Don't overthink it. That might be the same as you ain't going to need it.

I think someone else has this law.

Heidi: That's true, maybe, but I don't remember who.

Paul: No one has their name on it, so I'm claiming it.