Ep. #4, Open Source with CoreOS’s Alex Polvi

Grant Miller: All right, Alex. Thanks for joining me.

Alex Polvi: All right, thank you.

Grant: Cool. I would love to hear a little bit about your background and how you got to where you are, what you're up to now, etc.

Alex: Sure. I hate to give my whole life story here, but a lot of it's connected, so I need to dial it pretty far back in order to connect to where we are today. My first technical job was when I was a freshman at Oregon State. I was abuse@oregonstate.edu, which was the email where Sony sends their DMCA complaints about people downloading movies in the dorm. I was on the security team there, and I remember my first task was "Install OpenLDAP," which even now plagues sysadmins. That's how I learned to use this stuff.

But also my freshman year there was this thing called the Open Source lab at Oregon State, and Oregon State randomly was the biggest hosting provider for the large open source projects. The Apache Software Foundation, Mozilla, the Linux kernel, freenode had their big nodes at Oregon State, and all this stuff. So I got a job with that group doing servers, helping be smart hands just running the infrastructure. That's not exactly where I met Brandon, my co-founder in CoreOS, but we got the job together and we were both in the computer science program.

That's what kicked off the whole thing around open source, which led to me doing infrastructure stuff at Mozilla, that brought me to Silicon Valley which is why I started my first company Cloudkick right when Amazon was becoming a thing. Before they had any UIs.

Grant: What was Cloudkick doing?

Alex: Cloudkick was Cloud server monitoring management. As S3 was released then EC2 was released, and then we were like, "Maybe somebody wants a web dashboard for this." Because that didn't exist yet, database management console wasn't a thing. So it was a combination of Datadog meets RightScale, if you know that product. It's like a web based tool for monitoring and managing, particularly for Cloud resources.

Grant: Was that an enterprise product, or was it targeted at startups? Who was the end user?

Alex: That was a SaaS product and it was targeted towards AWS users, which back then was mainly startups because it was really early days of Cloud.

Grant: Cool. What happened with Cloudkick ?

Alex: Cloudkick ended up being acquired by Rackspace as they geared up to compete with Amazon, and we became the basis of a couple of their Cloud products. We became the management console for their Cloud product. We became Rackspace Cloud monitoring, and then we built a number of other Cloud products for the Rackspace Cloud product suite as their big run against Amazon.

Grant: How long were you at Rackspace for?

Alex: I was at Rackspace for about two years after the acquisition.

Grant: You were rolling your product into theirs, and doing all the stuff we talked about?

Alex: Exactly.

Grant: Staying pretty engaged?

Alex: Yeah. We had a number of Cloud products and we were really going after that. But that was when I did Y-Combinator, and when I started that company I was 23 and it was my first time doing it.

Grant: You're 25 now? So Cloudkick, then you're inside of Rackspace and rolling this product out, and then what was next? What was the insight that led to CoreOS?

Alex: I was at Rackspace, got the products integrated, made sure that whole thing was self-sufficient. Then I left and took some time off to figure out what's next. It took about nine months or a year or so, and as we thought about CoreOS it was really about "What are the big problems we want to go solve? Let's have a big lofty mission that we can go swing for the fences on." So we started CoreOS and the mission from when we started it to when the company was acquired was, "What could we do to fundamentally improve the security of the internet?"

Our key insight was this thing called automated operations which is, "Let's write programs to do the basic operational tasks that sysadmins have problems with, because we believe that's why bad basic sysadmin work is why you get 100 million usernames stolen." These are things like not securing some port correctly, or not patching a version of an out of date library. This is how people write programs to automatically hack you.

Before CoreOS existed you couldn't write programs to automatically fix it. And that's what led to everything that we built, was trying to build a system that allows us to automatically write programs to manage infrastructure and really bring automated operations to life. And we did that.

Grant: That's cool. So, you were doing that. You started CoreOS what was it, five or six years ago? When was the--

Alex: It was 2013, early 2013. Five years ago.

Grant: We'll dive into some more of what you learned while you were there, but then tell the rest of the story to set the foundation.

Alex: Sure. With CoreOS, we were in the garage hacking away our grand plan to go secure the internet, and we started building all the different tools in order to make this possible. The big key to that was Docker and containers. Docker didn't exist when we started CoreOS. It emerged around the same time we started going, and so we built a Linux system that automatically updated itself, which is a key piece of this. Our namesake, CoreOS, Linux. In order to do this you have to be able to do it in a distributed system. You have to be able to take down any component 100% and your applications keep running.

The next critical piece to that, any distributed system needs distributed locking which is why we built Etcd. Then the next piece after that was, "How do you build the distributed process management?" We started building our own thing, but quickly around the same time Kubernetes was released and it was using Etcd, our product. CoreOS was the favored way to deploy these things early on, and so on. So we really gravitated towards Kubernetes because we weren't going to just invent new things for the heck of it, we were taking the best pieces out there. We had Docker, we had Kubernetes, we had CoreOS, we had Etcd.

We built some other components like Fleet, which was our initial one. Then we had Flannel which has become the substrate of most of the networking of CNI ,which is now the standard for networking. We built that. We built our own container called Rkt, to many people's delight. We built a bunch of these key technologies that are now being used across this whole new wave around distributed systems and containers, and then we built an enterprise software business around it to help enterprises adopt this technology.

About six months ago we were acquired by Red Hat which was a natural partner in all this, because they're a big enterprise open source company. We're like a Red Hat V2 in a way, so it's a very natural partnership.

Grant: That's cool. Now you're at Red Hat and the whole team CoreOS, and you're taking that same mission and expanding it at Red Hat.

Alex: CoreOS built all the open source building blocks that make this possible. At Red Hat, automated operations is the future of the product line for Red Hat's OpenShift container related products. Every product on the product line has some aspect that aspires to incorporate automatic operations as well. It's pretty cool. The next thing we would do as an independent company was release this operator SDK to help any company build or bring automated operations into the applications.

We did that as part of Red Hat pretty shortly after the acquisition, and it's a key part of Red Hat's strategy and it's being adopted widely. It's been cool. We were able to build the initial foundation, we built enterprise software business, we coupled that with Red Hat's existing software business which allows us to accelerate it into the market. Then we started building the next gen of the open source world to continue the vision even within the big company.

Grant: We love that operator SDK. When I read the release around what you guys were doing there, we agree. We think the future of enterprise software as autonomous applications that you need to be able to automate all the toil away, and we shouldn't be doing manual tasks.

The more that we can build in that human knowledge and codify it, make it something that's programmatic instead of a human sitting at a terminal looking at a dashboard and then clicking a button. It just makes it easier to operate applications and operate them at scale.

At Replicated we love that.

Alex: That's awesome. It always has bothered me when there's a big Linux kernel bug that every operations shop around the world has to stop everything they're doing and go and deal with that. It's this huge waste of human effort. Then there's the whole half of people that don't even know it existed, that didn't do it not because they're bad at their job, they missed the random post on the mailing list or whatever.

Grant: The CD announcement.

Alex: Exactly. They were sick that day or something. It's like, "Why do we have to? We can write a program to do this. It's not like we're going to eliminate these people's jobs, we're going to allow them to do whatever they were supposed to be doing that day." It's been pretty well received.

Grant: Stop the zero-day rush around.

Alex: Right. Or just any. It seems to happen over and over again. If you're on top of your stuff you get one of these like every other day it seems like.

Grant: For sure. You guys had a pretty quick response to some of the things, like the bigger CDs and bigger vulnerabilities that came out in the last few years. You were repairing those on thousands of systems instantly, right?

Alex: It was interesting. The day the acquisition was announced there was a big Linux kernel bug, and at that point we're partners and we're now one company with Red Hat. To see how the two different approaches work is really interesting. They're just different. I'm not trying to say that in a way that's like, "Ours is necessarily always better," but there are different approaches. They will get the patches out really fast to their customers and in a way that enterprises are used to adopting, and ours is the wild new way of doing things but for the customers that took the leap to do it, they didn't have do anything. They just got it.

But it comes with its own faults. You're trusting the automation and all this stuff. It's interesting because I remember the president of the company that day just being like, "It's cool to see these two different approaches that are distinctly differentiated and have different values both ways. Some good, some bad. But now we're bringing that to the whole product line."

Grant: That's cool. Let's go back in and dig into some of the specifics around your time, both at Cloudkick and then at Rackspace, and then CoreOS, and Red Hat now. Talk about how most of us don't grow up and think, "I'm going to be in enterprise software." How did you find yourself in enterprise software? Because you could have gone down different paths. What made it so natural for you?

Alex: I wouldn't necessarily say it was natural. I learned a lot at CoreOS. Cloudkick wasn't enterprise, it was a SaaS company catering mainly to these emerging tech companies that were using Cloud. CoreOS in modern infrastructure software there's only two approaches. There's a SaaS model, which for lower in the stack infrastructure companies is pretty well covered by the Cloud providers, and you're competing with the Cloud providers.

You're almost the only other option if you want to build a software company and not a consulting company or a support company or a training company, but you want to build a software company where you sell software to people for money.

You have to do enterprise software because the Cloud providers are really squeezing out the lower level infrastructure companies. They're taking all the networking and all the raw compute stuff. It's almost a side effect of doing business in the modern day. I wish I had a more glamorous answer to it, but that's the truth. Enterprise software is still one of the few areas that you can go compete in the market and be successful as a relatively small company. But even that requires lots of resources.

Grant: It's interesting. Your point there is that the infrastructure and the service providers as they expand their offerings, they eat up a lot of the lower level opportunities as a service. You're saying, "OK. What can we do that there is still opportunity in?" It turns out enterprises want to consume different software in different ways, and have different requirements that often aren't met by the broad strokes of infrastructure providers.

Alex: Another way to think about it is a lot of what we did was open source. So, how do you monetize open source? You either have to sell hosting or you have to sell proprietary add ons to it, and the people that buy proprietary add ons to open source projects are typically the enterprise ones. The hosting, if you think about it, more like open source hosting, all the Cloud providers are eating all the open source hosting. They try to find more open source to host for people because that's their business model now.

Grant: This is where some of the new licensing that's come out recently has been trying to take aim at that, right?

Alex: Trying.

Grant: Do you have any thoughts on that? In terms of if that's a viable model, or if you're going to be able to keep the inevitability that the infrastructure providers eat up everything even no matter what the licenses are?

Alex: First, those Cloud companies have legal departments that care about the licenses and stuff.

If there is some license that's in the way of them adopting an open source project, they just won't adopt it, or they will build their own thing that does the same value prop of whatever that open source project was.

That's what I mean, you can't really stop it. Sure. I'm like an open source wingnut, open source zealot to the core. In order to have a software business you have to sell software. So as much as you love giving away software, if you're in the business of a software company, part of your job is going to be selling software somehow. To sell software it either has to be proprietary or it has to be hosted. It's as simple as that. If it's not, you can't really sell it to people because it's free. So you're selling something related to it. The way we approached that was by giving away the core plumbing and then when we had an open source project we were giving it all away.

Grant: Great. We should dive into open source. What we see more and more and particularly in infrastructure, pretty much almost everything that exists is open source if it's going to be an enterprise software or if it's going to get adoption. If you look at most of the companies that are doing well in that space, open source is at the core. So talk about how that changes the go-to-market, it changes so much about what you're doing. Just talk about how that came to be. I'm guessing you were going to be open source from day one, but what other implications does that have?

Alex: We learned a lot doing this. Early days of CoreOS. The way we approach open source is that if the component is open source, it's always going to be open source. Our open core model is on the software package. There's a whole class of the market that buys software and just doesn't think about incorporating open source stuff. They go buy software and they have people that install software and run it, and whether or not it's open source doesn't matter. They still need to buy software, because that's just how their business operates. That's distinctly different than going and building an open source project and releasing it to the world to enable a bunch of developers to go and build new things with it. When we build open source it was much more in that latter bucket, and then we were also one of the companies that built a package and sold it to people.

It was a little bit bipolar because the company had to do both things even though they're distinctly different tasks, and it was tricky because our company cares a lot about open source. We also, everybody we hired, we were like "Here. We're building a business, and we're building a software business so we're going to sell software at some point, everyone." We didn't get in our way in that way, but still to be bipolar like that is a little bit difficult, even if your people are OK with-- If some open source developers are OK working on proprietary pieces of software for instance the next day. It's still a weird culture thing.

Open source can be this tricky thing to master because innately the open source component fundamentally won't have anything to do with your business in terms of your product, because the product is going to be something that you sell.

I'm being very specific, and obviously our products are related to the open source. But people didn't buy our thing because they're getting some open source bits, they're buying it because they're buying a piece of software. They're buying that package and they don't care if it's open source or not.

Grant: A lot of my perception was the adoption came from the fact that it was open source, and so I'm guessing your sales channels or marketing channels was really open source focused, and maybe you did some top-down selling where you would go into an org and try to sell them the package. But I'm guessing there was also a lot of bottom-up pull where engineers at a company were adopting some of your technology, and then the VP of eng found out that they were doing it and decided "That's a core part of the infrastructure, let's buy the enterprise version." How did that work? Tell me the--

Alex: Open source is really good for awareness. We would walk into meetings and people would definitely know our company and know who we are. Docker is a great example of this. Anybody that's even close to infrastructure has heard of Docker, but who's bought the products? People are, but it's really helpful because it allows your reps and stuff to go and get a meeting and stuff, because you have a reputation behind it. It's great from that point of view. Obviously it's good to give away components to the world in general too, don't get me wrong. I'm assuming we're all on the same page about that. But in terms of--

Grant: The value of open source in general.

Alex: Exactly. I'm just saying, in terms of enterprise selling, it's helpful to get you in the door and get a meeting and get going. Just keep in mind that users that love your open source software, they innately love your product for it being free. If you are trying to use that as a lead gen thing, you have to be aware that you're seeding your pipeline with people that want the free thing. They're tainted that way.

Now you're looking for customers that are both enterprise buyers, but also love free open source software. It narrows the pipeline of customers a lot. For an enterprise piece of software with a direct enterprise go-to-market, you want your pipeline to be all software buyers, but of course it's helpful if they already know your company and they believe in it and they think you're doing good stuff. It's good from that point of view.

Grant: That's interesting. It sounds like from a go-to-market perspective, the way that you would approach new customers is not necessarily like, "We know that you're using this." It would be like, "We know that you buy software and you're a software buyer, here's a solution to solve your problem as you go into this new world." Is that--?

Alex: Totally. If we were to come out of the blue right now and start CoreOS V2 as a company, where the market's at and everything, we'd be like "You want to adopt a Kubernetes product? Here's one that's easy for you to adopt, and it's meant just for people like you that buy software and have these requirements, enterprise ready requirements. I'm solving a problem for you around adopting this software." And that's what I solve, and I'm betting on that there's a market for that, versus being "We're the guys that made Etcd and we contributed a lot to Kubernetes and building CoreOS and all that. You should come here about our product portfolio."

It's almost like, "Of course you want to go after the customer's problem." One of the challenges of open source is if you invent open source from scratch, you have to wait for the market to get big enough and that to have a slice of it, to be this enterprise buyer. You almost want to adopt an enterprise go-to-market and open source products that have been out on the market for five years.

The usage market is big enough that you have that little pie slice of the enterprise users already ready to go. If you invent something from scratch, you have to build that, and then you have to monetize your slice of the pie once that's big enough. It just takes a long time and startups don't often have a lot of time.

Grant: That's an interesting point. Oftentimes we see new open source companies come from these more humble beginnings, where it was like an engineer or two, or a little community building something. Gitlab follows this, it wasn't a big project and then they started adding more and more to it. Then after it got pretty broad adoption is when they could really create a company around it and take it to market in the enterprise.

Alex: HashiCorp is a great example of this. HashiCorp was around for a while and it wasn't until very recently, in air quotes, the last two years or so that they really applied an enterprise go-to-market on it. But it was a market timing thing. You have to get your stuff and there has to be a market. Any software product needs a market, and if you're doing it on the heels of some open source project, that has to be pretty widely adopted to have any meaningful market size.

Grant: How do you think someone should, if they want to go down this path and they want to create that open source project, they want to see if it gets adoption. What's the key signal that like, "OK. Now we should start a company around this. Now we should build it. It has enough momentum"?

Alex: I don't think there's a specific signal. The way I would benchmark it right now is Kubernetes is Q1 2018. That level is probably about the right time in the scope of an open source project, to start.

Grant: That's a lot of adoption.

Alex: How many downloads is that, or whatever? I don't know off the top of my head. But it's got to feel pretty mature, because remember, you're going after some sliver which is only 5-10% of the usage. That's your possible customers. Then you're only going to get a handful of those people that are some pie slice of that in order to adopt. It has to be quite large in order to work.

But it's possible, we've seen a lot of companies be quite successful with this. HashiCorp's doing well, Elastic is about to go public, Mongo already went public. The Kafka guys are here doing really good. There's the Databricks folks. They're productizing Spark, and this model is definitely possible. But all those projects I just rattled off, they have pretty wide adoption and pretty mature open source projects.

Grant: That's a great point. It's definitely important to wait. Kafka was out and around being used for years before they started that company. They came out of LinkedIn and the team left to go do that.

Alex: Exactly.

Grant: It seems like the other foundation for companies now is the founders worked at some big company where they did it, they had an internal version of the thing that they're doing, and now they're going to come out into the world and do it for everybody else. You guys described your product as Google infrastructure for everyone else for a long time.

Alex: We were trying to articulate what we did. It's so complicated. The automated operations is the word for folks more now.

Grant: But at that point it was Giphy.

Alex: Right. Early on, for sure.

Grant: It's funny. Figuring out how to describe what you're doing to the market in a way that they can relate to it and identify with it. It's hard. It's a lot of work.

Alex: When you're done you're like the kernel and the distributed lock in the distributed system, which is like the underbelly of the distributed system. It gets quite complicated even for technical people to figure out what's going on.

Grant: Your headline on your website, "Distributed lock," it's hard to get anyone to buy for that.

Alex: Right, exactly.

Grant: That's funny. You mentioned this open source core and you said you had these enterprise ready features around it. You categorized those, but how did you decide what features to put in first? And what customer discovery were you doing? I'm sure you didn't just instantly tag on LDAP integration and in RBAC right away. How did you figure out which one to do, and who did you talk to?

Alex: This is another place where being both open source and enterprise, the company has to be a bit bipolar. Because the way you do open source project management is quite a bit different than the way you do product management for enterprise companies, and a good enterprise product manager can do it on an open source or not. Having experience in open source is good, but the principles are the same, and the product needs are similar. So it does come back down to a lot of the things that Enterpriseready.io talks about, those sorts of features, and that they're these basic requirements that enterprise need in order to adopt a piece of software.

It's just the way it is, and the product managers that have dealt in the enterprise software world will know those, and they'll know how to work with customers, and they'll know how to set expectations with customers appropriately about when things are coming. Because these customers are used to it taking a quarter or two for a new feature to come out, and so on. I'd say one of my mistakes was waiting to hire enterprise product management into the company, because I felt like "I'm a founder. I'm a technical founder, I'm supposed to be the person doing this." But my expertise is in the open source side of things.

Once I brought in a partner to the company who's a pro at enterprise product management it made the world of difference.

When we were in the early gritty stages of figuring this stuff out, Like "How do you work with these customers? What do they expect? How do you run a CAB? A customer advisory board?" All these sorts of things that enterprise product management, it comes intuitively to. My background was not in that. I'm an open source infrastructure person, so it was a big learning experience doing that.

Grant: But I'm sure you learned a lot along the way.

Alex: For sure.

Grant: You have to pick it up, because--.

Alex: I think I get it now. No, I definitely get it.

Grant: But it's a skill and it takes a long time to pick it up, and my first enterprise software job was sitting in rooms didn't know what people were talking about. There would be acronyms thrown around, I'd be like--

Alex: "What's going on?"

Grant: "What does that mean?" So, you felt similar maybe? As you were going into enterprise, you just didn't have the same vernacular? What was the--?

Alex: I like traditional, enterprise IT buyer that is not from a company that adopts open source as their way of doing it. It's a different style than what you might be used to with Silicon Valley startups and smaller companies and just how they consume software. Really understanding that process and how it works and the requirements for that, it's an education for sure. I'll be honest with you, I learned it all as we did our enterprise go-to-market with CoreOS through trial and error.

Grant: Sure. You got to learn on the fly. Then did you notice a big difference between companies based in Silicon Valley and enterprise tech companies, versus traditional enterprises scattered throughout the rest of the world or the rest of the country?

Alex: Regionally, a little bit. But the tech companies, the big tech companies, they generally hire the people that just adopt open source themselves.

Grant: Right.

Alex: The companies we were selling into were the enterprise early adopters, they're forward thinking but they're still traditional enterprise and have requirements that enterprises have. These were like Nike and Starbucks and Ticketmaster, and these companies that they're forward thinking and they're progressive in terms of knowing that their digital transformation is coming and they have to do something about it and all that.

But they're still not a software company at the end of the day, in their bones. The big tech companies, the Ubers or LinkedIn, they're software companies at the end of the day they're just at scale software companies. So they have their enterprise-like issues, but it's different when you hire the people to go install Kubernetes as Kubernetes is being developed.

There's a bunch of companies that could do that. They could maybe have a small tiger team doing that sort of thing, but they still want to work with partners to develop and bring these technologies into the environment.

That's really where we got started was with those sorts of accounts.

Grant: It's interesting that you wouldn't find your early customers from those enterprise software, enterprise tech companies. That's not your customer base early on, you're saying that they do it themselves, generally. Right?

Alex: The ones that do it themselves are the big tech companies, for sure.

Grant: The interesting question is, as more and more companies have to become tech companies-- All the banks describe themselves as software companies now and that's what they do. As that transition happens, which is true. Bits meet atoms and that's the way of the future, you have to be able to figure out both. Do you think that companies will develop more big tech companies and do it themselves? Or do you think that they'll continue to find partners to help bring them into this new future?

Alex: They'll need to bring in partners. They'll have to bring in a lot of automation too. That's why our automated operations is part of it. Running infrastructure, and running Cloud infrastructure that's always on and globally distributed and all this stuff is tremendously difficult. Even with all the tools, there's only so many people out there that are good at it. If every company out there has to be really good at it, it doesn't scale. It's like physics, it doesn't quite work. There just aren't enough people to be really good at it. Automation in this case is the only answer. Or, the companies that don't go fast enough, just getting eaten by the ones that go faster.

Which you'll see Amazon doing quite effectively across many different areas of the market. Automation is the answer, at least on the lower level Cloud stuff for allowing the companies to focus on their core competencies. If you're Starbucks or Ticketmaster, why are they pros at managing their Linux kernels? Why do they have to be good at that? But they do have to be really good at that. If they don't they're going to get hacked, and they're going to go out of business because they get hacked too many times. They have hundreds of millions of users data, so they have to get good at this stuff regardless of their ability or their culture internally.

Grant: That's interesting. The progress that the rest of the world is being caught up in, as it's moving so fast, you're right that partners will be an important part of taking everybody else there and every new company that comes up, and allowing the point of focusing on what your core competency is.

Alex: Or what the differentiation is. You're not going to be differentiated on how fast you patch a kernel.

Grant: But it could impact the bottom line if--

Alex: Right, if you don't do it right.

Grant: When you think about that enterprise go-to-market, one of the things I love to understand is "How do you get your first customers?" You're this series A funded or seed funded company and you need to show some customer interest in adoption and interaction. How do you get in the room? How do you get people to adopt it? Who do you look for and what questions do you ask?

Alex: For us there are different phases of how we approached it. In about late 2016 is when we turned on our enterprise go-to-market in the sense that we had a turnkey product, and one of the big "A-has" for me was the power of a SKU. Once you have a SKU on a product, you have this thing that you can sell multiple copies of to people. You're not selling consulting anymore, you're not selling support and services, you're actually selling a thing . If you can SKU your software product it's ready for an enterprise sales force, because you have this thing that the reps can go and sell a product. It could be a copy machine, or it could be whatever, but your product is one of those things now and it can just go and be sold.

To get to a point where your product is packaged up enough and general purpose enough that it could just be sold to a bunch of different users as a thing that's sold is its own milestone. Once you have that you can then go build the lead gen and the whole machine around "How do you bring that to market as quickly and as effectively as possible?" You really need that thing to be crystallized before you do that. Otherwise every deal will be this custom deal. You're not scaling anything, you're just hitting whatever throughput you have on the people that know how to adapt a solution to whatever that current pipeline is, or whatever people you have in their company that are able to rejigger your stuff to work for that customer's needs.

It's like fitting two puzzle pieces that don't fit together, but reshaping both of them such that they come together.

When you have the SKU it's like, "There's the hole in that company. I'll put my puzzle piece there. Here's the hole in that company, I'll put my puzzle piece there, but the rep doesn't get to modify the puzzle pieces. They just have to find the hole that it fits into.

Grant: I'm thinking through that, and it's interesting because it describes the-- I'm guessing you're saying that early on you're doing this customization-y piece where you're figuring out how to solve the problem at this company that wants to pay you some money and you want to get them as a customer. You're like, "OK. What shape does our puzzle piece need to be in order to solve--?" And you're like, "Tell me about your puzzle piece," and then you're building yours to--

Alex: Fit it.

Grant: And then you put it in. The next one, it's a little different, the puzzle piece.

Alex: And then I have to do it again.

Grant: As you're doing that, is it like your puzzle piece gets a thousand notches on it or does your puzzle piece become something? Do you start to feel-- How do you discover what the common piece is?

Alex: That's the tricky part. That's listening to the market and getting the feedback, and I think the art of Product Management is really nailing that and also being at a point where you're like, "No I have something good enough." Almost making it rigid, because if it's not rigid you can't build the go-to-market side around it and you can't scale it. Because not everyone can be a founder and map it to reshape both sides. Go tell everyone to go rebuild the product, and then go tell the customer that you're going to solve their problem. At some point you want to have the thing that you need a flywheel, and you just keep selling them and keep printing them, and software is beautiful when that starts to work because the printing cost is zero.

It's like, that's what you're trying to get to. Even once we got to that stage on the software product we still had to sell some consulting upfront to help customers adopt it and install it and get their first apps running on it and stuff. Because we were so low in the stack, it's pretty heavy to get it running, but once it's up and running then they have the same thing on every single customer and our automated operations worked exactly the same everywhere. We pushed a button and all the clusters go and update and all this stuff. You have to have the rigidity, not just for the product to work in our case of automated operations, but also for sales and go-to-market engine to function.

Grant: It makes a lot of sense to me. I'm running through why that's so important, but you're right.

You have to have this common thing that you can sell because founders are often comfortable with a lot of change and a lot of iteration as you're feeling it out, but if you're going to take this to market and you're going to train a sales team and you're going to create the collateral and you're going to do all the go-to-market effort, you can't do that every week. That's not a super iterative task to re-collateralize and retrain and re-enable an entire team. That's months of effort if you're selling something like that.

Alex: It took me a long time to crack the nut of "What is a good enterprise sales rep? How do they work and what are they good at?" The thing that took me a long time to understand is that a good sales rep, what they're good at is selling, and the art of selling is this magical thing that's very similar to the art of writing some crazy algorithm. Just like you can have these wicked amazing programmers that you don't quite know how they tick or work, but they are really good at what they do, selling is similar to that. There's something special in the people that are able to really sell, but their skills are not technical or whatever.

They have a completely different set of skills than the programmers, but the level of skill is similar to that hand programmer. Understanding and respecting that, and then understanding those people need a thing to go and sell, and that's why it has to be rigid. Because once you match those two things together, it works really well. You bring in your sales enablement programs that allow them to get the latest and greatest pitches, and get the latest and greatest things. It was amazing to watch our best reps work, and if you look through their history they sold all sorts of different stuff, because they can.

It's just like a good programmer can program in all sorts of different languages or whatever, it doesn't matter, because their skills are really specific. But you can't just give them one thing and have it shape shift on them, because they're not going to be able to use their skills on a shape shifting product. It takes work for them to hone their own art on whatever you give them in the first place.

Grant: I love that. That's great insight. We as technical founders often, we don't love sales, but it's funny. As we scale we almost always get more appreciation for the art. There's still some bad salespeople out there that we don't want to talk to.

Alex: For sure. It's a weird thing, especially as technical founders. But we have to be good at it, and when it works it's amazing to see the engine come alive end to end. From a lead that's generated to almost a support ticket is the other end of the funnel, that means it's been sold and installed and used. To have the whole company working together across that whole spectrum is quite amazing.

Grant: You've been building this morphing piece, and then you finally figure out what it is that's actually the rigid piece that you can then resell, and you do the enterprise go-to-market and you enable teams. Generally the next day, and maybe you didn't get there with CoreOS, but it's second product introduction. Is that something you guys did? Did you have an additional thing that you started, like another SKU?

Alex: We did. Our other big one was Quay our container registry product, Tectonic was our Kubernetes, enterprise ready Kubernetes and Quay was our container registry. Anybody doing anything in the category had to have a container registry, and we had the best one. We had a hosted product and then we had the software licensed version of it.

Grant: We used it. We use it still.

Alex: It's a great product. Red Hat, I don't believe they've open sourced it yet, but they will soon. We sold Quay as a licensed piece of enterprise software. No shame about it. Obviously we used some open source stuff to build it, everybody uses open source to build stuff these days, but it itself was a licensed product that we sold. But you arm the sales team with both. That's challenging early on to figure out how to build your sales force, and you're trying to enable multiple products. Again, I would recommend having one product and getting to some really big scale.

I don't know, $100 million in revenue or maybe $50 million in revenue before you start trying to bring out another thing in your product line. It's only the big companies that have this functioning sales muscle and they're able to roll out lots of products, but even the big companies that do that aren't necessarily that good at it. Oracle's really good at it. They have tons of different products and stuff. VMware, still selling the VMware stuff. The main thing is as much as they try to add other products to it their bread and butter is still vSphere Suite. I would definitely caution against trying to have multiple SKUs too early on. Technically you're going to have multiple SKUs even if it's on one product, because you're going to have the 10 node version and the five node version or something, where it might be different SKUs.

Grant: Product assortment, right?

Alex: It's the same core thing. It's just different ways to buy, or at different price points or something.

Grant: Like different entitlements for quantities you can consume, or added features you might add.

Alex: The pro version gets you a couple more features or something. But it's the same product at the end of the day.

Grant: It's interesting, because you also had many different open source projects.

Alex: But remember, the open source project thing is distinctly different than your enterprise go-to-market. They're related, they help each other in terms of awareness and things, but when a customer or an enterprise customer is going to buy something they're, innately not buying our open source stuff. They can't buy that. We give it away for free. They can only buy the stuff that we sell to them, which sometimes is a package of open source stuff that is all open source but it's bundled together in a package, or it's actual proprietary software that they can't get in the open source stuff.

For scaling an enterprise software model that's definitely more valuable to have something that people really want, that's really hard to build, that they have to pay for to get. Sounds obvious when you say it, but then the open source package is more of a convenience thing.

If it's all open source but it's just packaged up for you and versioned for you, that just makes people's lives easier that can't hire the people to go and piece it all together themselves. You're saving them that headcount, from that point of view. But you have to get one of those things, even if you're building all the open source. One thing people don't realize about open source is that you have to 2x everything. Because you have to build all this open source stuff, which requires its own product management, and its own engineering teams, and--.

Grant: It's marketing. It's go-to-market, you still have to take that and get people to use it. You can't just build it and put it out there, and be like "Everyone is going to use it now."

Alex: Over here you're going to be building your enterprise software product, which is its own whole thing, remember. The companies that do both well are the ones where they own the projects through maintainers. Again, pretty much all those companies we mentioned earlier. Hashi, Elastic, Confluent, Databricks. They own their software projects and they're able to build the enterprise features on top of that, because they control the communities that go and do it.

They're essentially an enterprise software company that has all the software, they just happen to have this third party community that runs out there too, alongside it. But they're running it as one whole thing. That's where it works, but even they have to double their efforts, because they have to have all their engineers working on the open source stuff and they have to have people building the enterprise stuff as well.

Grant: It's interesting the way that you describe that, they own it not because the original creators are still the founders of this company but because they employ most of the people that are the contributors to it.

Alex: It's whatever form of control. They control the project, so however they control it. It's often through maintainers.

Grant: That's an interesting perspective that I hadn't thought about much, because generally since I'm a founder I think about founder-centric companies that have the team that built the first version, and they built the whole thing. But realistically, especially for these longer lived more mature products, it's about having control over the community. That's generally by employing enough of the core contributors to be able to influence what happens. That's cool. If your suggestion is to take a mature enterprise software, open source project to market, you need to control the maintainers.

Alex: It sounds evil to say it, but it's just the way that it works.

Grant: I don't think it sounds evil. "Employ open source contributors," doesn't sound like a bad--

Alex: Often the companies that do have the control have the founders of the projects as well. It all goes hand in hand.

Grant: I don't think that CloudBees has the founders of Jenkins, but they control--

Alex: They have the top maintainers.

Grant: They have all the top maintainers. They do 60% of the maintainers or more. The open source angle is interesting, but when you were building the enterprise version and Tectonic is the primary enterprise one. Were there things that enterprises wanted that surprised you? Or features that were really hard to deliver, and you just didn't understand the requirements?

Alex: I'd say the most interesting lesson learned in that is not necessarily some specific features, and most of the features are things on your website that you go and list. It's these nuts and bolts thing, but if they don't exist they just can't adopt it.

Grant: Sure.

Alex: They can't try some preliminary version, or whatever. You have to cross the threshold of, I'd say on your list it's probably the first five or six of them. I don't remember them off the top of my head--

Grant: Single sign on, role based access control, audit logs and reporting.

Alex: But if they just don't exist it's like, "Guys we really like you, but we just can't adopt it. So let us know when it's ready." You have to cross this threshold of enterprise readiness to even be adoptable, and in our space we were very much emerging the whole category with a whole set of companies. It just takes a little time to mature these features. Our approach to it, because we didn't control Kubernetes, which was a big piece to this, we were just always ahead, our company is always ahead on most of the different features and needs. We had the first product with RBAC in it, we have the first one that allowed you to tidy your corporate LDAP. Kubernetes or kubectl, the command--

Grant: kubectl.

Alex: kubectl is tied to your LDAP as well as the web interface tied to your LDAP, and they're all in sync and if you disable the user online it disables the command line. Having all that work, we were the first product that did it. The problem with that model, and again this is a problem from a business point of view, is we're playing a stay ahead game. Which isn't super sustainable, because at some point the open source communities will catch up and be able to have these features too and it erodes your value. You constantly feel like you're just building stuff that the open source community goes and puts in anyway, because these were our proprietary features that we were adding in.

It also feels disingenuous because we love open source and we want the stuff to be generally adopted, and so it was this weird thing. That's why the companies that own their projects have a lot easier time, because the feature is never going to go in the upstream one. They just won't prioritize it, so you don't have to worry about that. Our model was always like, "OK. We have LDAP integration. But next release Kubernetes will have natively. What are we going to put next to keep it interesting for people? That's the struggle with not owning it. This is why open source businesses are so hard.

Grant: You were building your own orchestration and scheduling with Fleet, but you saw Kubernetes and you were like--

Alex:

Our goal at the end of the day was automated operations, and we just wanted the best tools available to bring automated operations to market. We were going to build anything in our way to have it exist.

So whenever we built software we did the whitespace open source projects and then we gave it out to the world. When Kubernetes came along it was just obvious, because we were just copying the Google white papers anyway. I'd have Google, the people that wrote the white paper working on it, and then them using our products in it. "Obviously we'll just adopt that instead," because our goal wasn't that part of the stack.

It was further down, the automated operations piece. Our proprietary IP at the end of the day was automated operations. It was code that knew how to sweep your whole cluster and reboot everything and upgrade all your Kubernetes nodes and all of that. If you remove that you still have a completely vanilla upstream Kubernetes cluster. It's just you have to go manage it now, you don't have software to go and manage it. It's like plugging in autopilot to the car and you sold the autopilot, but all the sensors and everything were already in the car and they were all open source so somebody else could build one too. But nobody has, still even today.

Grant: That's going to become an important feature in OpenShift?

Alex: For sure. The products have merged. Red Hat's business model is slightly different, they have a very unique business model. All Red Hat software is open source and I don't know if Red Hat could be recreated again, but Red Hat has a very interesting business model. In that it's closer to what I would call, probably unflattering, insurance. It's an insurance product, I believe. I'm not speaking on behalf of Red Hat in this, or maybe I am. I do wear a Red Hat. But Red Hat's insurance product is brilliant for open source. It just says that for Red Hat Enterprise Linux, which is the bread and butter of Red Hat's business.

They have many different products though and many of them are growing really well. But Red Hat and Linux, the way it works is they'll guarantee you that your applications will continue to run on Rail for some period of time. The big policies are 10 year policies. What that means is your application, you don't have to modify it at all, but the thing it's running on will get it security patches. It'll get its performance fixes and stuff.

To do that it's actually quite messy work, but that's why they make so much money doing it. Because it's a really hard problem that people really want, and it can all be open source because it's this ever changing thing. There's always another patch that needs to be back ported and whatever into it, but yet they're able to stamp out a consistent product to all their customers at the end of the day. It's a unique model.

Grant: I love it. What it does is it enables the adoption of open source in enterprise, because if that wasn't a service that they provided, then that would be this huge amount of overhead that every single organization had to take on themselves to use upstream Linux and keep this all together. I think someone putting it into a great package and then making it available and being reliable enough to being there in 10 years, that's super valuable.

Alex: As part of the future of Red Hat, and this is maybe where I'm putting on my Red Hat hat a little bit, is we are bringing automated operations to all this stuff and you can't do that if the APIs break. If the APIs break between versions we can't automatically upgrade them, because on the other side of the upgrade your stuff's not going to run anymore. It comes hand-in-hand together, and that you need the stable APIs and the stable interfaces to work into guaranteed work, in order for automated operations to work.

That's why again, the marriage with CoreOS and Red Hat makes so much sense. Because truly, CoreOS worked, our technology worked and it worked really well. But for it to be ubiquitous in the market and for it to work in a way that never takes anybody out and is truly automated across a wide swath of the market, you have to be partnered with a software product that never breaks. That's what Red Hat offers. You have to have both.

Grant: That's why it was a natural fit for you guys together?

Alex: Totally natural fit, and Red Hat sold futures based around their Kubernetes product OpenShift. Now with Tectonic and OpenShift, they've come together nicely. The next major version of OpenShift will have all the different components of the Tectonic stuff built into it, like we took the next version of Tectonic and the next version of OpenShift and brought them together. The automated operations is becoming ubiquitous through the SDK, and Red Hat can use its huge network of ISV partners in order to bring automated operations to all their products really quickly. It's this really nice complementary thing that they're doing.

Grant: That sparked a new question I thought about, which is one thing that is fairly misunderstood in the market. The role that VARs and MSPs play, or channel partners play, in enterprise software. Did you guys get far along with that at CoreOS? Do you have much experience with that, is that something you can shed light on?

Alex: We didn't get to the channels yet. Around when we were being acquired is when we would have started the program, which probably would have yielded around a year later. But you have to carry your own water before you can expect a third party to carry it, which means you have to have that SKU buttoned up. You have to have that thing. That puzzle piece needs to be super sharp. Sort it out and as an early stage company it's messy figuring that out. At maybe $20 million revenue on your product, assuming you got that in three or four years on the enterprise go-to-market tops.

You know that you have it if you can scale it from zero to $20 million in three years, you probably have something that an enterprise go-to-market can chew on.

At that point you could successfully put it in a channel that can go and blow that out with their reach of whoever they're working with, but my theory and we didn't get far enough along in this but my theory and the advice that I was given and made a lot of sense to me is, we've got to be able to carry our own water, we've got to get our own machine functioning before we expect to drop it into someone else's machine and have any chance of success.

Grant: From the stage earlier, it's like you as the founder had to be able to sell it before you could. Then you had to have one or two sales reps that could sell it, and then you can have a sales team that can sell it. Part of it is having that rigid puzzle piece that is easy to repeat. You can sell anything, but that's because you understand the product and the customer.

Alex: Good early stage reps are more adaptable too, and more malleable to it being a mess.

Grant: But you don't have 50 of those, you have 3.

Alex: Exactly. But when you go dropping the channel in one of these big companies, and they have people that are selling tons of different things, it just has to be good. The other component is, "How do you make those partnerships truly mutually beneficial? How does that work?" If you're going through these traditional software resellers your product definitely needs to be mature. If you're going through one of these big software companies through their channel, it's a special startup deal, that's where it becomes truly mutually beneficial.

Because what's going on there typically is the company has some deficiency internally where they're unable to deliver the product themselves, so they're working with a fun little startup to go and do that. But then it starts to be a conflict really quickly, either because they want to buy you or because they're going to just go build their own product and shut you out once they've been able to build it, and so on. What you're talking about more the traditional software reseller, the VARs.

Grant: Sure. But there are other channel partners that are interesting too, because for companies that are series A, B, C even, finding the right channels and you often hear a lot of the marketplaces that exist for software aren't amazing.

Alex: Right. Again, we were just saying "No" to partnerships a lot. Because there's the blog post partnership, what does that get you? Then there's the big company snooping around on your product line to see how well it's actually doing. What's that going to get you either? Even if you're trying to have an exit or something to that company, letting them see inside the house before they bought anything is not a good idea either. There's no circumstance that I think doing channel, unless your model because you're a master of this stuff already is going to be a channel model, which there are some companies that do that really successfully.

Grant: Box is famous for being so channel oriented.

Alex: It's built from the ground up to do that, or at least it's very intentional. There are some products that can go that way for sure. But again, as we're talking, we're talking about more like technical-- Or at least I'm talking more from a base of "I'm an open source sysadmin hacker person." We've got to figure out the nuts and bolts of enterprise go-to-market, we have to carry our own water successfully first. Then naturally channel will just be part of extending that as the go-to-market matures.

Grant: That's great. Finally I'd love to give you a moment if there's anything else you think is interesting about the future of software, autonomous operations, enterprise software. Advice for folks that are trying to figure out how to build and create enterprise software companies, where they should be focusing and what-- Where's the puck going? Think about the future. What's the important things for people to hold in their mind?

Alex: Back to the work that you're doing, every application is becoming a distributed application. This is why things like Kubernetes are so popular now, because essentially it's a framework for "How do you manage a distributed application?" On the Cloud side of things there's nothing that runs on one machine anymore. Even if it does from a resources point of view, it still should be distributed in case that server goes down, because the things always have to stay running.

That's applying all over in terms of needing to run on prem distributed, needing to run on Cloud distributed, and that's why Kubernetes is inevitable across the ecosystem. You'll see it get embedded more and more into things. I want it to become like Linux kernel, just get stable and solid and boring, and nobody thinks about it anymore because it's just something that's out there. We've all moved on and up to something bigger. But I think that it's just the way that everything is moving, and so the work that you're doing around helping companies bring their own applications and build distributed versions of those that can run autonomously on prem. It's just brilliant.

Grant: Thank you.

Alex: It's a great idea.

Because it's coming, and every customer I talk to wants things to be easier to manage on prem.

There's this cycle off of Cloud a little bit too, everything comes and goes, and we're definitely full on move into Cloud mode right now. But I see that cycling if it's easier to run it in any environment and so on. That's the play with all the stuff. I definitely think that's the way things are going.

Grant: That's awesome. We agree. Kubernetes has got to become the foundation. We're betting big on that. You obviously bet big on that. A lot of companies are betting big for that to be the foundation of distributed applications and the future of software. It's still hard, there's still a lot to do there. We have work to do to make that a future that we can get to. But if we get there it'll be wherever you want to run it. On prem, Cloud, anything else. We can run these autonomously. Everybody can have autonomous operations--

Alex: Automatic operations.

Grant: Cool. Alex, thanks. This was a pleasure.

Alex: Thank you.