Ep. #41, Simplifying Developer Workflow with Paul Biggar of Dark

Brian Douglas: Welcome to another installment of JAMstack Radio. In the room we've got Paul Biggar. Hey, Paul.

Paul Biggar: Hey, how are you?

Brian: Good. Paul, you've got a history. Do you want to talk about why you're here today?

Paul: Sure. The history that you're referring to, I'm a host on another one of the Heavybit podcasts, To Be Continuous--

Brian: Which is a great name by the way.

Paul: Thank you.

Brian: I'm pretty jealous about that.

Paul: I don't remember whether it was me or Edith who came up with it. I'm pretty sure it was Edith, she loves a good pun.

Before this I was the CEO of Circle CI, and I'm currently the CTO of Dark, which is at DarkLang.com.

Brian: Cool. Could you briefly explain what Dark is?

Paul: Dark is about making it 100 times easier to code. Specifically, we're targeting back-ends.

If you're something like a React developer or a JAMstack developer who needs a back-end to communicate between the various clients that you're building, Dark is a thing that is supposed to do that super easily.

You can think of it as a server that's taken several degrees further.

Brian: OK, excellent. That's right in the wheelhouse of what JAMstack is all about, the abstraction of all your different layers and services. At Rails Conf last weekend I had this long conversation around how Full Stack doesn't exist.

My day job is GitHub, and we have Full Stack engineers. I was able to tell them that Full Stack doesn't exist even though they think it does, with the abstraction of those different layers.

We would talk-- Before we hit record, we talked about how React first came out the gate doing one thing really well.

Now, React has a lot of different things and I think the nomer that it's not a framework is a lie now at this point. But that's for another discussion with the React team.

Obviously, you've been paying attention to the space and you're now trying to attack one of the acronyms inside the JAM. How did we get here as developers, to now have these tools to solve these problems?

Paul: I think it's actually related to this Full Stack idea that you had. If we look back 10, 12, 15 years then people were Full Stack developers. They wrote the whole thing.

As the industry progressed, things got a lot more specialized and so people started building tools that solved specific problems of their area a lot better. Those tools grew in complexity to the point where people had to specialize.

So now, if you say you're a Full Stack developer-- I probably include myself in that category a little bit, but "How much do you know about the in-depth nature of React?" The answer is "Not that much."

Brian: Yeah.

Paul: But you also know a decent amount, but again, not that much about your back-end framework and about your infrastructure framework, and so on.

So instead in the industry, you have specialization of people who go really deep into the front-end and really deep into the back-end, really deep in the infra.

The result of that is, when people incrementally solve their problems in that area, they expand the amount of stuff that you have to know.

Now you have-- When I was coming up you had embedded systems engineers, and then you had coders. Those were the two categories.

Now as well as the front-end, you have people who specialize on various parts of the front-end. So you have interaction engineers, you have mobile engineers, you have machine learning engineers and data scientists.

Brian: When you say embedded systems, are you talking like people who created stuff on WordPress? Like, WordPress plugins?

Paul: No, like cars. Real-time operating systems, the stuff they would run in your watch or the old, bad phones. Those were the specialists.

Brian: Excellent. The abstraction of the layers are getting really interesting. One of my-- I'm not sure if they are a competitor for you, but Repl.it is a tool that I love using. Because you could essentially--

They're just a bunch of Docker containers, so no matter what language you're trying to attempt to write or learn, you don't have to worry about installing or finding out the Homebrew to get this language to work locally on your machine.

Paul: Right, exactly.

Brian: How is Dark abstracting away that back-end?

Paul: Dark is an editor, a programming language and an infrastructure.

The goal is that we have this editor that is specifically designed for this language, and this infrastructure that's also designed for this language, so it's all combined into one holistic box.

An example of what that gets you, is that in Dark there is no deployment. You are writing code in a safe way that is live, it uses things like feature flags to make that safe.

But the time between writing a piece of code in your editor and it being available, if you choose it for your customers, is like 40 milliseconds.

Brian: OK, wow. I mean, that sounds like my jam. Pun intended.

Paul: Yeah I mean, if you compare it to something like an installation of Gatsby, or something that, Gatsby takes six seconds to compile a production thing-- Maybe it's 10 seconds to compile a production thing.

Then you need to do a Git push, and then your Git push pushes it to Netlify, and it eventually gets there.

The liveness that you experience as you code on your machine, is not the liveness that your customers can experience.

Brian: Yeah. So, I started programming quite a few years after you started programming, it sounds like.

My intro to programming is getting things like Rails and doing these one out of the box solutions, and then you find the Rails way to do whatever you're trying to do.

It sounds like that we're moving more and more away from these giant, monolithic frameworks. Which is like WordPress, or etc. Now you can piecemeal whatever you need to do.

It sounds like where you see the industry going is more towards the sham way. Is that correct?

Paul: I see it going more towards the piecemeal way, whatever definition of piecemeal is. But I think you're totally right. The monolith is the thing that people are doing less and less, and for good reasons.

Mostly due to structural flaws in the monolith, but the monolith was great. The monolith was how we should be coding.

We should have the ability to abstract over these weird distributed systems that we're building, and have like a unified framework of monitoring and understanding and knowing where your stuff is.

That's what we're trying to achieve with Dark. We're trying to make it so that if you're going to write any code that runs on a server, you can write that in Dark in a way that is the same.

You don't have to think about, "This thing is going to be run in Kubernetes and written in Go, this bit is going to be in LAMBDA. This bit is written Postgres, or this bit is written a Hadoop or some data science, big data framework."

You write all this in Dark and Dark will almost compile it to the right back-end for you.

Brian: Yeah. I'm curious of how much that "For you" part is happening. If I'm a front-end developer, I know JavaScript pretty well, if you ask me to do animation I can stack overflow my way into it.

How much of the "For you" is Dark doing in the back-end side if I am pretty shy or timid of really going deep into something like that?

Paul: Part of the problem of the crossing the stack barrier is "What is onboarding like?"

Going the other way, if you're a back-end developer and you want to get into building something in front-end, you're like "OK, it's React."

So you're going to have to read a tutorial, you're going to have to set up a create React, and there's going to be a repo, and you're going to have to get used to all this new stuff that you haven't really seen before.

Things like Babel and WebPack, and all this stuff. It's exactly the same if you're familiar with that, and you're going to the back-end.

"Are you going to be using Kubernetes? Are you going to be using Express, or is it going to be Node?" Are you going to have to pick a different language?

What are you going to do about LAMBDA? Do you have to deal with some warm start, cold start problem? What monitors your LAMBDA? Do you even know that monitoring a LAMBDA is a thing that you need?

If it is, now you have to learn all these potential tools to do it?" So the difficulty with crossing into the stack is the tooling in the stack, not the framework.

Brian: Yeah, that's good.

Paul: Conceptually, what you're doing is you're taking data from a place, you're processing it, maybe you're saving it, then you're sending it somewhere else.

There's JSON coming in, you write a bit of code, the code that you're already familiar with. Write it in JavaScript, or whatever. You might be writing this in Node, so you write in JavaScript.

You're going to take a bunch of objects, you're going to take a bunch of lists, you're going to do some processing and then save it and send it. That's all there is.

That part of programming is the bit that we all know how to do, regardless of whether you're front-end or back-end, or one of the many other things.

We all know how to write a foreloop and take some fields from something and add them, or call functions on them. That's the easy bit. We're stuck in all the other complexity that we've built around that.

Brian: I think of how you come from the tool, and you co-founded the tool Circle CI.

Five years ago when I first came to the Bay Area, CI wasn't a thing I attempted to do. It wasn't a thing that I was really aware of, but I came to the Bay and all this cutting edge technology, we ended up using a CI system to automate the testing.

I remember I made a joke actually at a talk I gave a few weeks ago, where "You run the test and you then go for a coffee break, and then you hope that when you come back, the tests are done."

Collectively, our industry has commoditized the idea of a CI, everybody solved that problem while moving forward. Is the reason for Dark is because now the problem is tooling?

Do you think that eventually we'll get to the point where we're all using the same tools? Whether it's AWS or all these other large infrastructure players?

Paul: I don't know if we'll all be using the same tools, just because there's so much money behind not letting that happen.

The incentive is that you want people using your tool, so if you're AWS and you're making $10 billion dollars a quarter, Google is looking at that and Azure is looking at that and they don't want you to use the AWS products, and AWS absolutely does.

When you look at all the other tools in the industry, everyone else's is trying to do the same thing.

They're trying to bring you into their CI or their monitoring tool, or whatever the thing is. I think that there will always be an incentive, perhaps not intentionally, to that complexity.

Brian: Yeah. The one thing I've noticed too is I think there is a lot of developer pushback on the vendor lock-in thing between some of these large players.

Just listen to How I Built This, which is another podcast, and they were talking about Peloton Bikes.

Their approach was instead of building just an iPad that attaches to a bike, they decided to go with the whole thing, because they don't want to have the whole alarm clock.

The alarm clock in hotels where you have the iPhone 4 dongle, or whatever it was. All those alarm clocks are just sitting there unused because you can't actually connect your phone to it.

I think with the vendor lock-in we have the potential, if that goes down that way, of us being stuck and I have to only use this one monitoring service because it's the only one that locks in with AWS.

Paul: It's funny, because Dark is a thing that is a little susceptible to that vendor lock-in. It's not a thing that we want, because that's a barrier to adoption. More than anything we want people to adopt Dark.

But we think of this exactly like Peloton. There's all these weird integrations that you have to do that form a good deal of the complexity that we're trying to get rid of, and the way to solve it is if you own both sides of that integration then you can eliminate the way it integrates.

Or at least abstract over it in such a way that to the user, it's eliminated. This is very much what Dark is about.

We abstract over infrastructure, you just write code. You don't have to think about what database it's in. You don't have to think about what language it's in. You don't have to think about the integration between the database and the language and how good the ORM is, and how you're going to optimize that, or whether you get to write actual SQL, or that kind of thing.

It purely abstracts over that and the result is you get this shiny Peloton-like thing, like experience rather than a broken iPod Sellotaped to the front of your bike.

Brian: Yeah. Now I'm thinking of comparisons of other things that potentially could be competitors, or maybe even partners with Dark.

I think a company like Glitch. I think most people in the JavaScript space is familiar with Glitch, and I think it's like this euphoric experience when you first tinker with the Glitch or you-- The term is "Remix a glitch repo," or "Project."

I'm conflating all their marketing terms, but I'm not sure which one is right. Once you remix a project, something works and then you move on, and it took away all that JavaScript fatigue.

I don't think they were intentionally-- That's what their marketing was focused on, but they basically did that. I do a lot of these one-off projects that interact with APIs.

I do a lot of training, so I reached for a Glitch remix or repo and then say "Everybody in the room, just remix this repo." All 45 minutes to an hour and a half of just setting up something and making sure dependencies work, is all gone.

I think the whole onboarding thing that you are you were bringing up, every time you hire a new developer there is all these popular scripting tools that we have now.

I remember back in the day, when it was a full four days of "Download this, see if this works. OK, upload this dependency. Download x code for some reason, maybe make this work, run the script."

All that's gone. Sorry-- All that's gone in the jobs and the roles I've had so far. Obviously there are some people who probably should work on scripting.

Is the goal also to one-click, include your back-end through Dark in the future?

Paul: It's very similar, yes. The overlap between Glitch and Dark is very high. Same with Repl.it.

They're both wonderful tools that see the world very much like we see the world. Where the distinction lies is Glitch is targeting this-- You write your front-end on your back-end together and all in JavaScript.

What we're targeting is you write your entire back-end, you can write front-end in whatever you like, and you write your entire back-end in Dark and then it builds the ideal back-end for your thing.

It's a lot more around "How can we safely scale back-end operations?" Glitch has a separate niche than that, and Repl.it has a separate niche than that again.

Brian: Yeah. How would you future-proof things? I think we're at OAuth 2 at this point, what if OAuth 3 comes out or OAuth 5, or JWT-S or something? Some weird acronym comes out?

Paul: This is a really good example of the problem that Dark is trying to solve. We are building the whole thing around this idea of continuous delivery.

In the old days, we wrote these applications-- Today, we write these applications that run on a single machine and that when you make a change to it, you are changing the software that runs on that machine, and then you're distributing it to all your things.

So you're making a change, but in the continuous delivery world what you actually want to be doing is you want to make the new thing, switch over to the new thing, and then step away from the old thing.

Whether or not you're doing that is essentially the factor that correlates most highly to whether your service works all the time, or whether it goes down. Are you able to plan for this nice, easy transition?

Dark is fully built around this idea. You have this code running in production, and you write this new code that you want to be running and production alongside it that has easy flags for switching over for you, for your team, or whatever it is as you want to test it out.

Everything in Dark is versioned to allow this. Every function is versioned, every handler is versioned, every type is versioned.

You can imagine you want to make some change to some little utility function, but the utility function is used to write your code base.

If you make the change, that one change that you're looking for this one API endpoint, you have to make that change to every other place that uses it.

You might not have enough testing, you might not be confident. What Dark lets you do is it allows you to make a new version of that function. You can't change all the functions.

Make a new version of that function and apply it to just one endpoint. Then when you gain confidence in that, we have the list of all the other places that it's used and you can go in and say "All right, I'm going to change this one. All right, I feel confident in that one."

Lets you go through the whole way, and basically tooling around best practices for how people should be writing software that allows it to be this-- Allows you to do it right, much more easily than we do it today, wrong.

Brian: That's a really good point, too. I'm thinking, as you were saying that, when I first got into programming I learned Git. I didn't know what Git was, I didn't know why I needed to know what it was.

It sounds like Dark and all these other tools we mentioned so far, they're abstracting the need to know some of the lower-level things. Is that true?

Paul: We're trying to remove all these integration points. The word we use for it is "Accidental complexity." We're trying to remove all the shit that you shouldn't have to learn, so Git is a really good example of this.

Because what you have when you're writing software is that you have three or four different places where that code is deployed. Something goes into your master branch of your Git repo, you have something hits the server, you have something hits a CDN.

Then once it's hit the CDN then it hits a client at various times for various different clients, depending on when people press refresh, or whatever.

You have all these different places where that transition can happen, and it's like "How many of those can we reduce it to? Can we make it so that a feature flag and deployment and a CDN deployment are the same step, and they use the same flag?"

If you do that then you remove the risk that comes from each individual one. So pushing that thing to Git no longer becomes a thing that you do, because the thing that's important is pushing it into a place that your customers can use it.

Brian: Yeah. There's a talk that Heidi from LaunchDarkly-- She's a developer advocate at LaunchDarkly. I know Edith is your co-host on To Be Continuous, talking about the removal of branch base developing in the future.

I keep running into her and she keeps telling me about this talk, I don't know if she's actually giving it publicly on a stage yet, or to record it, but hopefully it will be in the future.

I don't know, I feel like I'm creating my developer and all these new developers that don't want to have to deal with things like Git. I'm just going to start complaining about, "It was worse two years ago, you don't even know." Almost like this jealousy.

Also the other parallel is I have two kids, and my son, who's five, now is at the point where he can wake up whenever he wants. He knows how to turn on TV, he knows how to get the fire stick out, he knows how to start his day on his own on his own pace.

I just remember back in the day when I had to wait for cartoons to show up. Now it's on demand for him. I'm just waiting for the day where he can just tell Alexa to start the toaster and put the Eggo in and he doesn't have to worry about degrees or settings.

I feel like that same analogy is now with all these new or up and coming developers.

Paul: Everything is becoming on-demand.

Brian: Yeah, which it's insane to think about that. Only in a short amount of time we're seeing this transition happen.

Paul: Yeah. The part of the movement and the thing that ties a lot of what we've been talking together, this phrase the Charity Majors uses which is "Testing in production."

You can think of something like Glitch or Repl.it or Dark or LaunchDarkly. These are all the same ideas, that you're going to be putting something in production anyway, you may as well not pretend that this is going to be some golden handoff that is going to work perfectly.

And instead build your tools around the idea that "Yes, we are going to be doing it in production. We're going to be writing in production."

That transition has to happen at some point, we may as well build our tools around that concept.

Brian: Yeah, it's true. The whole idea of continuous integration that we mentioned before, having that automated all that process, you're going to have monitoring out there. If something breaks or something goes down, it sounds like there's no reason to have a six month staging cycle, or whatever it is.

Paul: If you take that to its logical conclusion, there was no point having a six month staging cycle because the risk of that one big "Everything that we made in six months is going to be delivered to the customer at the same time." That made no sense.

It also makes no sense that we wait an hour for the thing that we want to grab, or we wait five minutes, or whatever is the end of our CI.

If we build our tooling not to be "OK, we've got this monolithic thing and we're going to exhaustively test it and then we're going to release it because now we know it's safe."

If you build it so that "We know exactly the change that you're making in production, and we only need to verify that small thing, and that small thing we can verify in less than a second."

Brian: Excellent. This is a very JAM-y conversation. It's almost intense, but I think all the mentions that we've had so far, hopefully the listeners are taking notes and grasping. We'll have show notes at the end.

Anything else you want to mention about this whole approach to continuous delivery and automatic deployments?

Paul: I think the industry is going to keep getting frothy.

We're going to keep adding new tools and new ways of doing it, and I think that the thing that I'm really trying to do is I'm trying to remove things from it.

Trying to take things that we've done before or that we're doing now and say "You know this one actually isn't necessary. We're only doing that because of the way we use GitHub."

Or something along those lines, and to look at what can we remove. Once you remove things only then can we start to really simplify.

Brian: Excellent. With that, I'd like to transition us into picks. These are JAM picks, things that you're jamming on. I know your picks are always challenging, and you mentioned them before we started.

I'll go first, and my first pick is Beyonce's Homecoming. This is a documentary, it literally came out last week chronicling Beyonce's comeback. I didn't know-- Most of the stuff that I learned from this actual documentary I didn't know.

She pieced it together in this, but basically Beyonce performed at Coachella a year ago. Her Beyhive, which is her collective borg of super fans, essentially renamed the entire show as "Beychella" because it was that big.

She just leaned right into that. She was actually supposed to play the year prior, she got pregnant with twins unexpected, and then basically almost died due to preeclampsia.

I only know what this thing is, because I have a really close friend that also went through this as well. It's a serious thing and then she ballooned up into a very large weight, too. You can watch the documentary to find out what exactly the weight.

I want to respect Beyonce's wishes by not sharing her weight publicly on my podcast. But anyway, she came back and within a hundred days before the show she gave birth to twins, she got back into shape.

She started the entire process of a show, and anyway, it's amazing. You should definitely watch it if you have any interests or you remember that time, or if you don't even realize that it even happened.

Definitely check out this documentary, and you can see the whole why Beyonce is the new-- I don't know if it's fair to compare her to Michael Jackson like on that level, but her show was at that level.

I got chills. The same chills that I got when I was a kid watching Michael Jackson opening up for shows and playing the Super Bowl, and stuff like that, I got watching Beyonce.

I'm a Beyonce fan now, but I didn't grow up listening to Beyonce or anything like that. I have so much more respect because of the show.

This is way off technical, but I think there's definitely a lot to learn from her struggle with learning, and coming back from a long time out. That's my pick. That's the only one I have. Paul, do you have a pick?

Paul: I'm going to talk about the thing that is next up on my Netflix that I have not actually watched, I think it's Knocking Down the House, but it's the story of Alexandria Ocasio-Cortez and her election, and that kind of thing.

I am a super fan. I watch her Instagram Stories, I've basically given up Twitter but I'm back on Twitter for that.

The entire thing around-- I probably don't want to keep too much politics in this, but a politician who's authentic and who seems to actually care about the world that we are living in and is not bought, which I fundamentally think is a problem.

So, I'm super excited about watching that.

Brian: Yeah, I saw that come through because Netflix is clever around having the automatic trailer come through.

There's also a couple other stories who also ran around the same time as her that was included in that. I'm intrigued to hear about the other stories and whether or not they're actually politicians as of now.

I was also very intrigued about "AOC," which is what they call her on Twitter.

Paul: Right.

Brian: About her path and where she'll be a few years from now. It's almost like if House of Cards kept going, the Netflix show, this is real life now.

Paul, I appreciate you coming to talk about DarkLang and talking about this continuous delivery and this new tool that you're working on, and also the space in general.

I think you brought a lot to the table from your previous experience, but also your current experience. I appreciate that.

Paul: Thank you so much.

Brian: Yeah. Listeners, keep spreading the jam.