Ep. #36, The Docker Story with Solomon Hykes

Benjie De Groot: Welcome back to The Kubelist Podcast. Today we have Marc on the call with us, but also we're joined by Solomon Hykes. Solomon is the founder of Dagger, and before that he was the founder of a company called dotCloud, you might have heard of it. It became called Docker. Very excited about having Solomon on to talk about all those things. Welcome, Solomon.

Solomon Hykes: Thank you. Thanks for having me.

Benjie: Okay, let's dive in. To get us started, Solomon, will you just tell us about your background and how you got started in software development? And where it all started for you?

Solomon: Sure. I grew up in France, even though I don't sound French. I am very French, and I sound this way because my dad is American and so I spoke English to him as a child. So yeah, I went to high school in France and then just got interested in computers. I was a big Mac fan in my early teens. I actually got in a school fight once over the Mac being dead and I had to defend Apple's honor. It's ridiculous, but it's true.

Benjie: What year was this, Solomon? What year was this?

Solomon: This was like '97 maybe, '96.

Benjie: So to be fair, the Mac was a little bit on life support at that moment.

Solomon: Oh, it totally was. Of course, yeah.

Benjie: It wasn't dead yet. It was not dead yet.

Solomon: Yeah. You need to be a rebel in some way, you need to tie your identity to being different in some way. That was, for me, at the time being the only Mac person. Anyway, so that was one aspect of me getting interested in computers and eventually after high school I joined a relatively new software engineering school in France called Epitech. It was very confusing to the French people in general because it was not the usual way to do engineering.

But it was the right way to do software engineering, which is just a lot of practice, a lot of coding and acquiring theoretical knowledge through practice instead of years of reading books and then in year three you start coding for the first time. Which is how traditionally French engineering schools approached programming. So I just dove right in after high school, and had a great experience with that school.

Then I gradually started getting obsessed with what would now be called the cloud, but at the time was, I think, barely starting to be called that. But before that I remember being excited about the concept of infrastructure, that instead of working on one computer at a time, one server at a time, you could actually deploy software to a bunch of computers as a cluster and start thinking of those computers as cattle versus pets. That expression didn't exist yet, but that kind of thing, that's what I was interested in so that's what got me pulled into a series of projects that eventually, years later, led to Docker.

Marc Campbell: Let's talk about that a little bit, so you created dotCloud. Dot Cloud, it had been around for a while running dotCloud before Docker. I'd love to hear a little bit about the origins of dotCloud. It was a YC company?

Solomon: Yes. It was YC 2010, summer of 2010. Before that it was a sad attempt at bootstrapping a cloud computing software company out of a French suburb with zero experience or connection of any kind. We were really outsiders to an incredible degree, to be honest. People don't realize just how much of an outsider I was. I didn't know anyone. I knew how to program, but besides that, 2006 when I graduated, 2007 when I started working on that, the project that would become dotCloud. So that's three long years in France, just figuring things out but with no connection to Silicon Valley or the tech industry, really.

Benjie: Right. The entire ecosystem was extremely foreign to you?

Solomon: Yeah. Even working on a software product. Working at a company that published a software product, even that was an unusual thing, an unfamiliar thing.

Benjie: Did you have a job at all before you started the project that became dotCloud?

Solomon: Yeah, I had a job. I worked in cybersecurity initially and then did a bunch of consulting. Programmed a lot on my spare time, occasionally got a chance to program at work. But the thing with France is, for a long time if you were a programmer in France, you were most likely to work at a consulting company. Consulting is a really large and influential industry in France, and I think it still is.

Now you have the beginning of a tech product, like a miniature version of Silicon Valley, that kind of ecosystem gradually taking hold and growing quite successfully. But consulting takes a lot of space, so you're coming out of programming school, you're thinking, "Okay. Will I work at a consulting company for a bank, or will I work at a consulting company doing networking for manufacturing company?" So it's a completely different mindset.

Benjie: Like a big cheese, there's a cheese plant that you do IT support? There's a wine place in Bordeaux that needs to network a bunch of grapes. Anyway, so your experience-

Solomon: I was an outsider.

Benjie: You were an outsider. That's really interesting, so you work on this project and 2010, YC, that was very early days. I guess Reddit was like 2007, something like that, but I don't even know if Dropbox was a big deal yet. So how did you find YC? You applied and did you guys just get up and do the move to San Francisco thing? Is that what happened?

Solomon: Yeah. Exactly. We moved to Mountain View, to be precise. But yeah, me and this little founding team, we were all based in France. A few of them were interested in joining but we weren't enough of a real company to take the jump, and so I was running out of options in France. I had borrowed a bunch of money and was living at my mom's, stopped paying myself.

Basically, failed to raise money in France, couldn't find anyone to write a check or understand what we were doing, really. We were doing consulting to pay the bills, but then consulting eats all the time. So YC was sort of last chance, and we got in. There's a whole story there in how lucky we got, but we got in and, yeah, like you said, we packed our bags and showed up in Mountain View and never went back, basically.

Benjie: And dotCloud, at the time, I was a dotCloud user. In fact, I was looking at this and I found some old emails of me emailing Marc about dotCloud a while ago. Actually, shout out to Ethan, one of the people who works with Marc who I also found some emails where I was talking to him about dotCloud. So dotCloud was kind of like an application, right? It was an application where I could select my own application and then somehow that became Docker.

Tell us about YC a little bit, and then when did that click? Or how did dotCloud become Docker? Obviously before you changed the name, there was some stuff going on. Then, by the way, one other question is were you like a hardcore kernel guy? How did you find cgroup? Where did you find cgroup? Why was Google the only one that knew about cgroups up until you used it?

Solomon: Okay. So first of all, to start with the last question, the containers or the path to creating what is now known as containers because now you hear, "Oh, containers existed the whole time and the Docker guys just slapped a nice UI on it and some marketing." Totally not true, first of all. But it was a gradual step by step process of all of us reaching the design that exists today. Definitely, for me, that was the thread for everything I did starting from right after school.

I quickly became obsessed with that tech, cgroups. I think cgroups maybe existed in a very rudimentary form, but when I started playing with containers on Linux you had to patch the kernel and it was either Vserver which was the oldest patch that I know of, a lightweight server, lightweight virtualization is what it was called. Or OpenVZ, which was definitely the most hardcore, bleeding edge patch.

It had way more, I think it had better sandboxing, better security and also way better resource accounting. I remember that, you could just track utilization on these mini servers in a bunch of ways. That's where containers were at on Linux. There was no LXE, that didn't exist, that wasn't a thing. That came later. I think cgroups, I don't remember when those patches started, but they definitely were not finished if they had started at all.

So it was very early, and I just thought it was so exciting because it was a new layer, a new layer to deal with the state of systems and managing the state of systems, looking at it, and looking at it from the outside. So for me, the starting point was, "Okay. You can deploy, configure a bunch of servers as a whole. That's cool. That's a cool concept, pretty mind blowing." A whole cluster of computers, and that's the computer. That's cool.

Then, "Okay, so how do you get those computers to the state that you want?" Well, there's configuration management. That was a pretty new and emerging field too. But all of them had to change the machine from the inside, right? You boot the server, and then you run whatever, at the time it was like ISConf or RAdminD. I don't know if you remember any of those. I think they're long gone.

Benjie: Yeah, we do. And you had tools like Puppet and Chef.

Solomon: Yeah, CFEngine was like the first mainstream one, I think. Then later Puppet came out and that was a brand new thing when I started getting into it. That was very bleeding edge. Chef didn't exist yet, then later Chef came out as a different take on what Puppet was doing to the best of my recollection, at least. Anyway, all those hadn't come.

Tripwire was kind of related. It was all about running the thing inside the machine to figure out what the state of the machine was and how to get it to where you want it to be from the inside.

The thing with containers, is you got something that was much more like an application artifact, more like a Java JAR or an application artifact. Except, it was a whole system so you could look at it from the outside and whatever code is inside it wouldn't even know it's being looked at or managed.

It's still a fuzzy concept to explain now, but that was just such an exciting gut feeling, a nerd's gut feeling that there was something really cool there and I want to do more of that. That was 2007 to, I guess, now. That's the thread.

Benjie: You know what? I'd never heard someone describe it that way, and I think that's really interesting. The whole you're kind of defining it without turning it on, if you will. Obviously that's what Docker is, we all know what Docker is. But I do think that I didn't understand the context there of when you started playing with this stuff. What was dotCloud originally written in? What was the first version written in?

Solomon: We were trained in C and we were pretty low level systems programmers in general. Just from coming out of school, that was how we learned. But then we were quickly gravitated towards Python as the everyday, getting stuff done, prototyping tooling language and so the first prototypes, the first versions of all that container stuff that we built in 2007, '8, '9, that was all Python.

dotCloud when we launched, a Heroku competitor basically, which was an evolution of that process, that was also all Python. We're a Python shop, actually, all the way to the Docker pivot in 2013 where we took the core engine of our platform that was already all containers, and by then we had made the switch to LXE. Which was, again, very new, very unstable, very low quality. I don't mean the kernel features, I mean the user land tool.

And so because we were running at the time, this is later, 2012, '13, many containers in production, actual containers with a layered file system image and not just a bunch of cgroups on the cluster. But the closest thing to running Docker containers in production, that's what we were doing, and it was really painful because LXE was not stable.

So we built this little tool to manage it, like poly fill on top of LXE and that became an internal component and over time that became our container runtime. Then at some point, we made a bunch of decisions that led to spinning that out and that became Docker. Along the way, another set of decisions led to writing it in Go.

Marc: Let's talk more about that pivot. You had this dotCloud platform, you just described it as kind of competing a little bit with Heroku. Then this was an internal tool. When did you realize or what was the motivation to say, "You know what? This business that we've been building, that's not the business that we're going to do. This internal tool, let's focus on this and get adoption"?

Solomon: Yeah. Zooming out for a second, if we look at the arc there were three parts to the Docker story. I guess now there's a fourth being written still. But the first is we're just a bunch of nerds in France, excited about containers and we're just going to build tooling that we don't even know if anyone cares but it's exciting to us. Turns out, it's almost exactly what Docker will end up being five years later. This first incarnation, no one cares.

So actually the first version of dotCloud was an open sourced tool to run and orchestrate containers on a Linux machine. Complete with versions, images, and a way to move them around, et cetera. Except it was completely different tech stack, it was Python, we tried to put images in Mercurial which is, if you remember, a Git competitor. So it tried to put the whole file system in source control.

Anyway, there was this first attempt and no one cared because you had to patch the kernel, really. So just too much friction. Then we joined YC, just as we're about to die. We move to the US and YC does one thing, this is 2010 so second chapter. Basically they beat into us a basic sense of product, "Hey, this tech is great. No clue what it's for. Can you solve a problem for people today?" Huh, what an interesting concept.

So we just went on this journey to learn the basics of what is now the YC Way and it's really just Startup 101, solve a problem for people now, don't over engineer it, don't overthink it. Just get a toehold and then keep iterating from there. So that process led us to basically pivot to the dotCloud that I think you used and that became a real business, which was basically a platform as a service.

It was a direct Heroku competitor, so we handled deployment and hosting of the application all in one, super easy, we'd scale it and monitor it, blah, blah, blah. Kind of hands off. The big differentiator was we did it for many languages, not just Ruby, because at the time Heroku was Ruby only and there were a bunch of Heroku clones but they were always for only one language because there were basically just fancy managed hosting. So the whole stack was completely hardwired for that one language.

Benjie: Right. Nodejitsu, everyone remember Nodejitsu?

Marc: Oh yeah.

Solomon: I remember Nodejitsu. I remember PHPFog, Jangi for Jango.

Benjie: Very specific platforms here.

Solomon: Yeah, there were a lot of them. So we came out and said, "Hey, multiple languages." And we got lucky because Heroku was sold just at the back of our launch and so we just had a lot of buzz in the PaaS space because we were thinking, "Oh. Well, the simplified math is if Heroku was worth 200 million with one language, and dotCloud has 10 languages... Okay, we must be onto something." So that got us all the way to 2013.

And to answer your question, finally, when we decide we're not doing this business, it was death by 1,000 cuts because we slowly discovered that PaaS, Platform as a Service is a bad business to be in. Or at least it was at the time, it looks like some companies are trying again 10 years later and it looks like they might be more successful this time. But at the time, you were basically a very thin layer on top of AWS.

AWS at the time was still pretty young, so they had their own issues and you were stuck between customer expectations that were pretty high because you're selling an all in one solution, right? "We'll take care of everything." But on the other side, there's Amazon that can't really deliver the same thing to you. So what happened was sometimes things broke and when things broke a little bit, then we could work around them.

But a lot of times when things broke a lot on AWS, then we were just stuck sitting on our hands waiting for it to get unblocked like everybody else, and our customers gradually realized, "Okay. This is very valuable to get going and be very productive because we have a great developer experience. We're great at getting your product started. But then once things got serious and you had a real application with real customers."

And you, as the dotCloud customer, were growing and scaling, then your expectations, we just couldn't really keep up with them and that made it harder for us to justify our margins.

So eventually what happened is people would one day just as they became a valuable customer, they would hire a full time DevOps person and that DevOps person would look around and say, "Let's go and build this ourselves on Amazon."

Arguably, they wasted a lot of cycles doing that, but they still did it. So basically the whole PaaS industry was VC subsidized, lead generation for AWS.

Benjie: That's a pretty good quote. I think that Shipyard has been accused of being a PaaS in the past. But yeah, I don't know if it's changing and I definitely think some of my best friends are DevOps people. I consider myself a DevOps person. But we all are going to rebuild everything and make it better, and it's going to be next week. That's our superpower, is to always think that that's easy to do. Even the people that are building it.

So it sounds like some of these patterns, history repeating itself, it's pretty interesting. Before we talk about anything else, quick question, when dotCloud was a PaaS, what was your orchestration and scheduling layer? Obviously it wasn't Kubernetes or Swarm at the time, but just high level, what were you guys doing for all that stuff?

Solomon: Yeah. So it's really interesting because remember we were a bunch of French people, basically. We were pretty bad at marketing and we were better at technology than marketing, let's say. So one thing we were not super good at was just showing off all the cool tech we were building and just making the cool tech itself part of the marketing because when you're selling to engineers, engineers like to hear about the cool stuff that you're building and how to scale it, et cetera.

So the point is, we had a lot going on and I think actually it's kind of a shame that it's not super well known, what that stack looked like because it was basically the ancestor of what you would call today the CNCF stack. We had a miniature version of all of that because we had to invent all of it because none of it existed. I think, honestly, it's a wasted storytelling opportunity and the story we got instead, I'm just going to decide right now and here to be transparent, the story we got instead is, "Google invented everything, then eventually the world caught up."

And it's a fun story, but it's not actually an accurate story so you guys get the other side of the story which is that unless you happen to work at a company with 10s of millions of employees that can mandate one giant, mono repo with everything custom made from scratch, your own build tooling, your own monitoring tool, your own infrastructure management. All of it. Then your options are basically very limited. You live in VM land. Actually, you live in individual server land and if you want to scale your application, you need things like Nagios for monitoring. Remember that?

Marc: Oh yeah.

Solomon: Monitoring the server, or the many servers, maybe you have lots of servers in Nagios because it can scale really well. But what it could not do well was adding a bunch of nodes dynamically. That was not a thing. You had to manually write the integration for each node you wanted to monitor. But we needed to monitor a bunch of nodes, and I don't mean we're autoscaling VMs so sometimes there's a new VM.

I mean every time any application scales up or down or gets one container rescheduled, you've got to monitor that container. So what we had was a Franken-Nagio, I guess. That was the first iteration. We started from basically modified versions of existing production grade tooling, and then we just instrumented it as best as we could. Then as things broke, we instrumented it some more.

And then eventually several of those pieces, we reimplemented. Sometimes we open sourced the result, and so we did that for the RPC layer so we had a GRPC ancestor. We did that at the network routing layer, so we had a, I guess you'd call it, SDO-

Marc: Like a service mesh?

Solomon: Service mesh, yeah. Exactly. Service mesh ancestor. The scheduling, it was an in house thing to answer your original question. We had a build service, logging-

Benjie: And this was all Python before you even came up with the Docker file, basically?

Solomon: Yeah, exactly. It was all Python, except for the Nagios. I think it's Perl or whatever. But all the tooling we wrote was Python, yeah.

Benjie: You guys really are horrific at telling a story about this, because this is the first I've ever heard that you guys had the entire CNCF-

Solomon: We were quite bad. But that's the thing, people assume Docker was a feat of marketing and when Docker launched and took off, we had one marketing person, and he did a great job at adapting. His name is Julian. Because he had been hired to scale the online marketing of an online product called dotCloud, and he was given a headcount of four or five people to ramp that up, and then one day he learned that we were pivoting away from that and his whole team just got laid off because we had to lay off people to execute this pivot.

It was a painful, scary moment. My point is we did not have... Marketing came later, and it was purely reactive. The one thing that the Docker marketing figured out that was brilliant, is basically everything here is community led. There are people here, we're not sure how they got here or how we made this happen, but now there is a bunch of nerds that are really excited about containers and they want to contribute to this thing called Docker and they want to use it and they want to meet in person to tell each other about how they use it and they want a bunch of T-shirts now.

There's people looking for talks to give, there's people looking for speakers to give talks, so let's just unblock them, give them infrastructure and support and get out of their way. They're going to lead. That's what happened. So that's the one thing we got right in marketing.

Benjie: Whoa, whoa, whoa. The Whale? Come on, the Whale is awesome.

Solomon: Yeah, the Whale is awesome.

Benjie: Okay. Because you got at least two things right, Solomon, come on.

Solomon: Yeah, we did. So to tell the story of the Whale. First of all, I get zero credit for that. We had a designer at the time, also was working on dotCloud like everyone else. His name was Thatcher, and we basically said, "Okay. We're pivoting, so everyone who's not working on this Docker thing, find something to do because that's all we're doing now."

So he worked on finding a logo, among other things. He ran a 99 Designs contest, actually. We had done that for the dotCloud logo before, and I loved the dotCloud logo, by the way. It was great. Then we did it again for the Docker logo, and so Thatcher ran that and he came up with the idea that it should be some sort of mascot. There should be a mascot. Then we had this whole process. You can actually go and look at the contest and look at all the possible logos Docker almost had. It's pretty funny.

Benjie: We're going to find a link and we're going to put a link into that as well.

Solomon: Yeah, it's really funny.

Benjie: Okay. So, Solomon, okay, so that was the second part of the journey and then obviously... I mean, just a quick hit here, did you have customers? Were people using dotCloud?

Solomon: Oh yeah, we had customers. I remember we burned 300K a month when we pivoted, we had five million in the bank so we had plenty of runway. It was more of a morale thing, like, "Okay, where is this going. We're not differentiating anymore." But we did have customers and it was a lot of hard work supporting them, and growth was painful and expensive. But yeah, I think we were probably around, I'm going to say, one million ARR around that.

Benjie: So that was the point at which the early version of Docker had the most amount of ARR. Sorry.

Solomon: Not any more, not any more. But yeah, it took a while for Docker to catch up to that in terms of growth. In terms of growth, yeah, from the beginning of course that's what was so different about it.

Benjie: So it's ironic here a little bit. You get to YC, you start off, you're like, "OK, we're going to build the coolest thing possible. I'm a nerd, this is amazing. Look at what we can do. This is the dream for every DevOps person ever." Then you get to YC, they're like, "Hey, build something that actually people will need." And you do that, and you actually get some level of success with that, you raise money, you have customers, all these things.

Then you're like, "Yeah, but I don't feel that good about myself, so I'm going to now go back to what I was doing in the first place."And then it's this massive success, obviously with this taking off. I do want to touch on the community building stuff a little bit. So for my recollection and understanding, obviously you open sourced all this stuff and then you had this amazing viral effect.

I can remember, again, I literally should go back and find this email between Marc and I where we're just like, "Oh my god, this is it. This is what we've been talking about forever."So you get to that point and then there was a little bit of controversy here and there, and there's been some... I want to make a Zinedine Zidane reference, but I'm pretty sure no one knows who that is in our audience. That's a French football player who headbutted somebody in the World Cup, for those that don't know.

Solomon: Only the best footballer of all time.

Benjie: Okay, yes. And according to the French he's the best footballer. Again, this is not for our audience, our audience does not care. By football, we mean soccer by the way.

Solomon: Oh, you want drama? You want drama?

Benjie: No, I'm just saying there was a little bit of drama. Okay, so there's been some critiques of Docker in the past and I do want to talk about that quickly. You touched on it. The whole, "It's just a UX for containers," thing. I think we already talked about that, but did you have anything else that you wanted to say about that whole critique?

Solomon: Well, in the perfect world everyone who misunderstands you would get to see exactly inside your head and get the full story that you want them to see. But that's not how the world works, and so I consider myself if I have my own story straight and I feel like I can make sense of what happened in my own words and I can learn lessons. What would I do differently? What do I think I did well?

So that's a lot of work already, so trying to convince everyone else on top of that is just too much and that's a lesson I should've learned earlier, by the way.

You can't make everyone understand and love you, and if you try you're going to be miserable your whole life. So it was a valuable lesson in telling a story that's compelling before someone else tells your story for you.

So that's the valuable lesson there. Also, just the main lesson is Docker, it grew so fast and it became so important to so many people so quickly and by many people, I mean users, potential customers, and also a lot of potential competitors. It was something that a lot of people were forced to pay attention to very quickly and people are busy, they got agendas, sometimes you don't appreciate someone else setting your agenda for you.

Like, "Oh, now I got to worry about this Docker thing. What is that?" So we became important to two very different groups of people. The people who were there because they wanted to and because they were excited about Docker, and the people who were there because they had no choice. I think everything else you can extrapolate from that.

Usually what happens is you have people who really are here because they want to be here and the rest of them are people who could just not care less and they will happily ignore you forever. But we had this separate group that they had to be here because of the standardization, viral loop where we just pulled everyone in and you had to have a response to Docker of some kind and some people resented it.

Marc: Yeah, on the positive side though, I remember early days of Docker. At Replicated, we've been around for eight years, but even before that... I'm not going to go into the origin stories of Replicated or anything, but it was Docker. We were like, "Oh my, applications are portable. This is a new technology." I think when we launched, Docker was at like 0.5, 0.6, early days and the community that you were talking about, going to the dotCloud office and sitting on picnic tables there and having screens behind you showing-

Solomon: Oh, that was so fun.

Marc: Yeah. That energy. My origins in getting into Docker were more like the excitement of it, not I had to be there at all. I think a lot of people here that will be listening to this podcast, hopefully have that positive memory of Docker enables them to do stuff that they couldn't really do before.

Solomon: Yeah. Well, thank you for saying that. Yeah, that's very real, and I run into people all the time that I'm surprised... It's only after I left Docker and I took some time off and started reconnecting with the product community or just doing things like we're doing now, just catching up and chatting, that I realize what you're saying, "oh wow. This had a really strong impact, a really positive impact on a lot of people's careers. It created a lot of opportunities."

It created lots of opportunities for me, obviously. I get invited to podcasts and all that. But in parallel to that, since you were asking about that critique of Docker, the toy. There were many critiques over the years, the themes, spring, summer 2014 story. It's like a fashion. And thinking about where that came from, and the reason it got so much traction so quickly, there was a negative narrative around Docker that I think we handled poorly because we were not prepared for it because didn't understand.

We had no frame of reference, if you don't like it, why are you here? But the reason is because they were forced to be here, which is why I was mentioning that. So that's what made the Docker experience so crazy and unique, is just how quickly it grew and how quickly it affected pretty much everyone beyond just the people who wanted to be involved. That makes things very polarizing.

Benjie: Yeah, I literally just checked my Gmail and I found something. I sent Marc an email in 2013 or 2014, where I was just giving all the links to the Docker stuff. I will say personally, for me, Docker has always been extremely positive. I think it has for almost everyone I've ever met. There's definitely confusion at times.

Solomon: Oh yeah, lots of confusion.

Benjie: Like the whole jump from Registry 1 to Registry V2. I don't want to talk about that. But now we're just dredging up old stuff, but I'm just kidding. What I would say is that it created this industry. The reason this podcast exists, the reason CNCF is there is because we finally were able to have accessibility to doing these super cool things, and Docker was about opening accessibility. The coolest thing for me is when I meet a product person or a designer, someone that doesn't have any CLI experience, but they have a Mac, and I'm like, "Okay, you should just run this locally."

And they're so intimidated, and that's when I can be all, "Okay. Well, just do this one line install thing and then just type Docker Compose Up." And it just works, and all of a sudden they're like, "Wait, what?" And it just opens the door. I can't tell you how many people I've seen over the years that were not very technical or definitely did not have any DevOps or even CLI chops, that all of a sudden knew how to 'run a server.'

And that has just opened up the world so much because one of the things that I think was missing in the open source, in the GitHub ecosystem and all these different things was like, "Cool, the code is there. But how do I run it?" Now, that's still a huge challenge today. It's not solved. But it is pretty awesome. Solomon, I won't give you my spiel on Docker Compose but I am controversially a huge fan of Docker Compose still.

And so I think that that, again, is about accessibility. I think what Docker did was made it so accessible. I really want to talk about Dagger, your new project that we're about to talk about before we run out of time. But I do want to ask one last question a little bit, just because I'd be remiss not to. But basically the last stage of you at Docker was right around the time that Kubernetes came to be a big player in the field.

What I've always found interesting is that I talked to a lot of folks and maybe people a little less familiar with the ecosystem. Sometimes people say, "Oh, we use Docker," and they're referring to Kubernetes, and sometimes people say, "We use Kubernetes," and they're referring to Docker, and they have become pretty synonymous of what we'll call higher level, Forester-Gartner folks.

So just talk to us about when all of a sudden Docker, the community took off, everyone was saying this was it, and now other people are coming in with Mezos and all these other things and just the scheduling orchestration layer. Basically, the early dotCloud infrastructure, apparently, that became the next tool. Tell us about what happened at Docker? Obviously there was some stressful times there. But just talk to us on a high level how that all went down, and some lessons maybe you learned from all that.

Solomon: Yeah, sure. Yeah, there's a lot of misunderstandings around what actually happened and also why, what everyone's intent was at Docker and elsewhere because there was all the confusion about Docker in the first place. It was very new, Kubernetes appeared pretty quickly as one of dozens of projects that appeared very quickly in the Docker ecosystem. So the first thing is from our perspective at Docker there was only one thing that mattered and that was trying to keep up with the Docker rocket ship that was accelerating more and more.

Again, led by the community. It was just unlimited demand for all things Docker in every direction, all at once. It was like a fire hose that appeared overnight, right? We were shipping this prototype, like, "Lets show it at PyCon to get a few more testers." And then it leaked, and then explosion and continuous explosion for five years straight. And so one year in, looking back you say that's when Kubernetes happened, but at the time there were like three of those launching a week.

There were at least 20 Docker orchestrators and Kubernetes was one of them. Then there was networking to figure out and logging and storage, and in parallel I was trying really hard to move Docker close to developers to developers could actually use it directly because now it's taken for granted that you have things like Docker for Mac and even something like a Docker CLI available for Darwin or Windows, right?

But all those things were not obvious at all in the beginning because it started out as this Linux server thing. So on top of addressing everything I just talked about, and not to mention support for different layered fossil sims and getting Red Hat off our backs for supporting their old, shitty Rail versions and all that stuff. That was all demands.

Like, "Hey, we need this now, now, now." So everyone at Docker is in reactive mode, trying to keep up and prioritize demands and just keep people happy, while driving towards an overarching plan which was to finish the Docker platform, which was supposed to be a new way to architect and ship and run your application on top of this new layer of containers. It was just a very big puzzle with a lot of missing pieces. Orchestrating containers across a bunch of machines, that was one of many, many pieces in that puzzle.

Docker Compose, right? We had this service composition thing in dotCloud, and we lost it when we spun out Docker and so we wanted to bring it bag. The Fig guys were inspired by the original dotCloud at YAML, and did Fig.YAML and then we acquired them and that became Docker Compose, and so that's the main thing. Kubernetes was not the main topic. What Google was going to do and how cooperative they were going to be was a big topic and things were looking good in the early days because they had this LMCTFY project, Let Me Containerize That For You, which was sort of a low level, container app and then they decided to merge that into Docker.

So all the engineers, the awesome engineers who were working on that joined the Docker project and they became Docker maintainers and later container D and Run C Maintainers and as far as I know they're still doing that with zero drama. So we were very eager to take that to the next level and so we thought, "Okay. We've got Google working with us." Anyway, that's the perspective. There was just a million things going on.

The other thing that was going on was this split in the Docker community between two parts. There were infrastructure people and application people, ops and devs just to simplify, and it was ambiguous who Docker was for and that ambiguity was intentional because we wanted everyone to join this big tent that was the Docker community. We wanted infrastructure people to be excited about running dev on their server in production and we wanted developers to be excited about using it every day to develop.

That was necessary because you needed to bridge those two worlds for Docker as a platform to make sense. If I'm packing something as a developer, and it can run anywhere but no one wants to run it in production, then there is no point. And vice versa, if it's only an infrastructure component, hidden all the way below, then you're not revolutionizing anything. Instead of filling a triplicate form to get a VM, you're just going to fill a triplicate form to get a container from the same IT department and nothing is going to change.

So we needed both and so that was double the exposure because those two communities are very different. And so developers didn't care about orchestration, for example. They cared about service composition, so for developers Compose was more important than community. By the way, still true today. Developers could not care less about communities because it's an infrastructure thing. Infrastructure people care about orchestration, they need it, developers don't.

And so I think over time the split between those two communities became more visible to us and we started understanding the differences between those two audiences and the main thing we understood, or at least I understood and I think I failed to communicate that understanding to all of Docker, is that generally developers are loving us more and more. Us, Docker, at the time.

Also the infrastructure community is just accommodating our presence but the Docker brand appeals first and foremost to developers. It does not appeal to infrastructure people, and it's only going to get worse because Red Hat and VMWare are the infrastructure peoples' brands. They are very, very threatened by Docker stealing that. We're building these products to compete with them, and so we're making them even more scared and so they're going to go more and more negative and you're going to have a bunch of sales people in marketing departments manufacturing FUD around the Docker brand, and it's going to work.

Infrastructure people will be less trusting of Docker over time, and first it'll be about security, then it'll be it's not modular enough, then it'll be it's not standardized enough, then it'll be whatever the thing is. But it's always going to be something because they're swimming against the current. My general thrust was let's go towards developers because they love Docker and they can't get enough of it, and let's just over time accommodate the infrastructure ecosystem by basically complying with the very reasonable demands.

Like if you're running stuff in production, you don't want a big monolith. You want specialized components and you want them to be well scoped and you want them to be stable, and you don't want surprises. That's totally fair. By and large, that's what we did so now if you look at the infrastructure stack today, it's pretty good. You've got runC with the OCI spec, it's not perfect but it is what it is. Then you've got containers which is the dominant runtime.

That's Docker, that's Docker's project. Then you've got communities on top of that, and that's a pretty dominant orchestrator obviously. Then you've got an ecosystem of infrastructure facing things on top of that. Then you've got developers, completely over there playing with Docker Compose and Docker Desktop and bridging that gap is still a problem, but I'm sure it'll be figured out.

Anyway, I've been talking forever. But there's a lot of conflicts, basically. At no time did it feel like Docker versus Kubernetes, it felt like we were gradually being pushed away but it made sense why. I think we communicated poorly at times around that, but it all kind of made sense.

The biggest existential threat to Docker which almost killed the company, by the way, was not Kubernetes. It was a failure to ship a commercial product that was really compelling to a buyer.

The reason for that was lack of focus. You had a bunch of teams, a bunch of VPs, each building their own thing and with unlimited budget, by the way, because we were just growing so fast you could raise as much money as you wanted and hire as much as you wanted. So you just had a very typical execution problem, that's what almost killed Docker. Not Kubernetes.

Benjie: It's very interesting to hear it from your perspective, and I think that that's a pretty good insight. That's really helpful to understand, you guys were always just catering to developers and honestly that checks off every box for me because as a developer, I love Docker. As an infrastructure people, there was times where we had a tough relationship.

Solomon: Yeah. But I will say that it didn't have to be that tough, the reason it became... I think there was always the need to have a pragmatic conversation with sometimes, "Okay, let's get real. Here's what we need."But I think what caused drama and sometimes burnout, I had engineers at Docker who burned out and left, and in some cases burned out from the tech industry because of the toxicity that was sent our way.

That was not necessary, and it didn't come from pragmatic ops people that were asking tough questions. Those were always there and there are always reasonable questions. By and large, I think the team did a good job at addressing them and over time things got better. But on top of that you had a lot of unnecessary drama that came from vendors trying to sell stuff to those same operators, and the tactic they chose to sell them stuff was to throw a bunch of vitriol at Docker and that was not warranted.

That's a choice that you make at some point as a leader, to do that or not to do it. Now I know that's a choice that's available to me, and we only did that once and I'll tell that story because it's important to me. One time there was a Red Hat summit and we were there, and we were always there. But this one time we had this illustration which said, "Accept no imitation."

It was basically a drawing making fun of Red Hat's fork of Docker, because they'd forked it and added a bunch of stupid Red Hat specific stuff that crashed and then caused security bugs, and then they blamed it on Docker. It was the most annoying thing ever. Anyway, the way to deal with that is you just take it and occasionally you bitch about it in a podcast or whatever.

But then you just go out to the market and you talk about how your product is great and what problems you solve. That's the way to do it, I think. If you look at how we communicated, that's the way we did it to a tee, except for two things. One, that one drawing one time, which if I had a say we wouldn't have done it. I'll just say that. Then separately, I had a bunch of meltdowns on Twitter. That was totally me.

Benjie: So it's Twitter's fault. We can just wrap this whole thing up as any type of confrontation, it was Twitter's fault because I will say that's a good thing to blame.

Solomon: Yeah. It's just the thing about infrastructure people had good, valid criticism of early Docker and that criticism was addressed in a gradual and pragmatic way, and I think today people happily run containers and runC with communities on top or whatever the setup is. That was done in great part because Docker enabled that. I wish Docker got more credit for that part because if you remove the drama, those sets of issues resolved themselves in a pretty good way in the end.

Benjie: Well, Solomon, as someone that is still heavily in the container world, I can tell you that a lot of this history that we're talking about, I think it's pretty much forgotten. I'm not going to lie to you. Everyone loves Docker. Everyone loves Docker. Half the people listening to this are going to be like, "What is he talking about?" But yes, guys, for those that weren't around, there was a little bit of... we had some nerd drama.

We had a little bit of Real Housewives of Docker or Real Housewives of Kubernetes, or whatever it was. Real Housewives of Container Runtime, I think that would be the right terminology. But there was a lot of lessons learned from that and what I can tell you right now and I think Marc would agree with me on this, pretty much everyone loves Docker, we don't like it when they make us pay for stuff but we're also like, "Okay, you earned it." One question, is it KubeCuddle or is it KubeControl?

Solomon: I don't know.

Benjie: The correct answer is KubeCuddle, Cuddle is the right answer. I ask everybody this. This is the thing that I'm opinionated about, it's KubeCuddle. It is not KubeControl, it's not KubeCTL. It is KubeCuddle. So can we get Solomon Hyke's official endorsement of that, please?

Solomon: Oh, yeah, sure. I endorse that officially.

Benjie: All right, I'm going to need you to say it.

Solomon: I endorse KubeCuddle.

Benjie: Correct. All right, we got the soundbite that we needed.