Ep. #13, Standing On The Shoulders Of A License

Steve Boak: Welcome back to Don't Make Me Code. Back with me is David and with us today is Ilan Rabinovich, the director of technical services and evangelism at Datadog.

And we're going to talk about open source software again, and particularly how people are building companies on open source, and the different ways that they're doing it. And we're calling this episode Standing on the Shoulders of License.

So we always kick things off with a little bit of background, so Ilan, why don't you tell us a bit about your work at Datadog, and how you got into this?

Ilan Rabinovich: Yeah, so, as Steve mentioned I lead the community and evangelism team at Datadog. We work with our open source community as they're trying to build integrations, or help improve our projects and help to grow and foster that community.

And we also get to collect stories from our customers and just tell them, help them meet with each other, help them feel like they're part of a bigger thing than just them and their monitoring tool.

The way I ended up at Datadog was actually, I was a customer for a number of years. One of our larger customers at the time, and, you know, when my time there came to an end I was looking for a new opportunity and this came up at Datadog.

This is my first official job as a community director, manager type. Prior to this, mostly this was a hobby, so I ran a bunch of open source and technical conferences, and participated as a contributor in the open source community but not necessarily, never really had the title of community guy.

So this is a little new to me, but a lot of the experiences from my past as an individual contributor have been interesting to apply here.

Steve: Yeah, I saw that you had been involved with the Linux community for a while.

Ilan: Yeah, so this started out of the sorta user group community in LA, but a bunch of friends and I got together and started a conference in the Southern California area called SCALE.

Stands for So Cal Linux Expo, though we call it SCALE these days because honestly there is a small subset of the content that's about Linux proper, and then a ton of content about everything open source, whether it's hardware, or software, or free culture, or other things like that.

And so it started off there, and other friends in other cities were like, hey, you've done this before, can you help me do this in Austin or some other city. So now there's the non-profit LinuxFest dot org and we're involved in about a half a dozen to a dozen, depending on the year.

Technical conferences that we help organize, or at least do the fiscal sponsor for, so, you know, the contracts and accounting and all the sort of boring work so that the local team can just work on building a great conference.

Steve: One of the reasons I wanted to have this conversation now was because of this really big and interesting launch that just happened with Datadog. So Stripe has a team that's contributed several person months of work to a new open source library called Veneur.

Now, Datadog is not a purely open source product. There are open source pieces of it and it was surprising to me that paying customers would contribute so much time and energy to something like that, on the back of what ostensibly is a SAAS product, and the two of you were having interesting conversation about this before.

So I wonder first if maybe Ilan, you could tell us a little bit about that project and what it means for Datadog and Stripe.

Ilan: Sure. So Datadog, for those that aren't familiar, is a hosted monitoring solution. So we take time series, metrics and events, we help you graft them, alert on them, store them, so you can observe your environment, your applications, know how you're doing as a business.

One of our customers is Stripe, you know, a large credit card processing company here in the Bay area.

You know, their observability team happens to use us, and they wanted to do some of their metric aggregation in a slightly different way than our projects, than our collectors do.

Our process is basically, anything that runs on your computer is open source, anything that runs on our computer may or may not be.

It's a fairly standard SAAS model, and so you run an agent on your systems, it collects metrics and sends it to us, you know, the folks at Stripe wanted to do some of the metric accounting in a slightly different way than we did, and so they built this project called Veneur, you know, their observability team spent.

As you said, a few definitely person weeks, possibly person months, on building this project, and then released it as open source. You know, there's been a lot of interest around it from our community.

So as you mentioned, it's always an interesting conversation. A lot of our open source projects you know, they are open source, they're released generally under either the Apache or the BSD license.

But, you know, they're not necessarily useful without a service back-end to them, and, you know, in our case being Datadog I think that does raise some interesting questions around where you're contributing towards a service that you're already paying for in some way or another.

Turns out we have a very passionate community. You know we get pull requests all the time on new plugins, new large projects like Veneur, even on our documentation.

That's an open source project in and of itself and we get pull requests, fairly regularly of, I learned how to do something new that's not covered here or I found a typo, let me fix it for you.

David Dollar: So we're kinda here to talk about developer experience. I'd be curious to know about sort of like the origin of open source at Datadog?

Like, has the collector or whatever open source pieces you have always been that way, or was there some sort of decision behind that process?

Ilan: So again I'm relatively new at Datadog having been there for about two years. Datadog's been around for six, so. So, you know, I can tell you the stories as I've heard them and know them, but I was, you know, open source pre-dates me at Datadog.

What I'll say is, as far as I know, all of our agents and SDKs have always been open source. Some of them started off as other companies projects that we, you know, contributed to, and eventually forked.

Some of those folks have, you know, in turn re-forked it back. You know, our agents have also been forked by other folks and used elsewhere and I'm sure we'll touch on the benefits and pains of that whole forking situation at some point during the show today.

But yeah, so, I think it's just sort of, open source at this point is, if you want a developer community contributing to your projects, if you want them to be engaged, you have to have that source available otherwise, or some sort of plugin model, some set of APIs.

You know, you have to be extensible, otherwise, folks wont have an opportunity to contribute anything. I think these days it's very rare for me to run into libraries or SDKs that aren't in some way liberally licensed, or, you know, whether it be open source or something close to an open source license.

It's much easier to know how to self-support when you know how something works. If you want to be able to fix a bug when you find it. You want to be able to add features or change something when it's lacking for you in the product.

I think people have changed the way they want to consume software, even if they don't necessarily want to rewrite a very large database project from scratch or some metric store from scratch, they at least want to be able to see how it works and better understand it so they can reason on it.

Steve: Yeah, and that to me is an interesting point, that as developer tools, exposing the product and even making the product available in this way has. Here's this processing and visualization engine that we call Datadog.

But it's available to you to contribute your own projects to, to customize the way you want, and that, you wouldn't see that with consumer products, perhaps.

So like, it's an interesting way of building a product from the ground up to encourage contributions. And it sounds like it's something you're doing with Convox as well.

David: Yeah, I mean definitely. So it's, the question for me is always really interesting. I think if you're building a tool for developers like some part of it is going to have to be open source.

Like you said, it may be as simple as like a plugin architecture, it's at least going to be an API. There's gotta be some way for people to fiddle with it and customize it.

The most interesting question to me right now is sort of, where do you draw that line? And how much of it is open source? And how much of it are you building something that's totally useful on its' own, or is it a satellite thing to your product?

Or, kinda, are you giving too much away for free when you do that or not? And like, whether you charge for it or not? It's a lot of interesting questions.

Ilan: I think that answer's going to be different for every business. For us, again, it's anything sort of behind the APIs that we expose, we consider that our SaaS service.

Anything that runs on your machine is open source, and so, it seems like a reasonable place to have the border or the interface for me.

I mean that, admittedly you want to think one of the benefits people don't talk about a lot with open source is, if you go buy a license to a proprietary product or proprietary license.

Lets say you go to Oracle and you want to include their library for interacting with their databases in your product. It's not released as open source, so you have this commercial agreement with them that you now have to take to your lawyers to figure out what's legal and what you're trying to do.

How much do you have to pay them, how does this all work out? With open source licensing, we've effectively, I know lawyers in some cases are afraid of them, but really we've made their lives easier.

We've now got a standard for how you interact with a company, you know, whose software you're consuming.

It's not these weird bespoke licenses for every new venture and so being able to bring something like a BSD or MIT or Apache License, and say look, legal team, you know, this is a thing you've seen before.

Don't be scared. And they go...I'm not scared, it's great, you can use this in your product. It's much easier than having to go through all of the negotiations and pains of doing this in the commercial world.

Steve: Isn't there even some controversy over the kind of open source license? Is it GPL or some of the ones that have caused problems for certain kinda of developers?

Ilan: The nice thing about open source licensing, yet again there's some standards there and so you know what you're getting into.

It's very easy to take these long contracts, I imagine, and, you know, there's a reason lawyers are paid for this and I'm not a lawyer, but, you know, they spend a bunch of time painstakingly reviewing these things and understanding them.

That's no different in the open source world, it's just you've now got, let's say five or ten of them, and each one has their requirements and you have to abide by those.

You know, in our case, because we're working with a lot of libraries, we're working with a lot of things that people are embedding within their own products, we're not using things like the GPL, but rather using things like BSD and MIT and things that are called more permissive licenses.

They let you package that up and include it in your software. Usually our customers aren't distributing things built on us. They're building things like web services or back-end services within their own organization.

That still makes them feel more comfortable to have the set of permissive licensing. There are companies, you know, both recently and in the past that have got into trouble by not understanding the terms of their license.

Maybe they don't include a copyright notice they're supposed to include, or, you know, in the case of the recent WIX Wordpress pains, they didn't realize what some of the terms around the GPL were.

They didn't realize that some of the software had been re-licensed around the GPL. I don't think this is a problem with the license, it's a problem with people entering agreements without reading them.

I mean, it's like if you were to go to the bank, sign a bunch of paperwork, not realize you'd agreed to this crazy mortgage that doesn't meet your needs. But you show up next week and you go, but I had to pay you money?

Steve: So like what actually happened in 2008, like ARNs.

Ilan: Sure. I wasn't even thinking about that recent history, but yeah. When you sign an agreement, you should actually read it. And I don't think, you know, licensing is any different.

The nice thing again about open source licensing is your legal department can probably give you like a two-sided one pager that says, these are the licenses we've read, that we agree on, that we're not concerned over.

Use those. If you want to use something else, we should read them and talk to each other about them to make sure they make sense.

In my previous role out Ooyala, one of the things that I did was work with our legal team to build that open source policy.

I know that's not totally the topic of this show today, but if you're in an organization and you're consuming open source, and I'm pretty sure that's everybody that's listening because there's nobody that's not consuming some open source somewhere, you should talk to your legal team.

Find out what that two pages of sort of approved procedures is. And they'll generally stay off your back as a result of it, and you'll be a happy developer, they'll be a happier legal team and when your startup gets acquired or your software gets bought by somebody, there's not going to be questions as to like what did you do and why did you do it.

Steve: Yeah, I guess part of what I find so interesting about this topic is that I haven't actually worked for a company that, until Datadog at least, had even had open source components to it.

So I don't know very much about this. Companies like Docker that you know, have started with a purely open source product and have built a massive community around that and only after that have then tried to build an actual revenue stream from it.

It almost seems like an accident. I don't know whether a founder at a company like that sets out with the very clear and pure intention of doing that.

Or whether, with Combox you set out with a very clear and pure understanding of how you were going to approach what was open source and what wasn't. That part of it is really interesting to me.

Ilan: Yeah, definitely, I mean, so, open source is a very broad, expansive label, right, for all sorts of things and I think there's definitely like the giant community plays, you know, Linux, things like that, where they just exploded with you participation of like thousands and thousands of people.

I think, with Combox essentially, open source just basically helps us achieve our business objectives. We think it's sort of the right thing to do for some things.

For one it gives us a certain amount of trust that people can kind of pick it apart and see how it works and you know, customize it to their needs.

But also personally I've long tried to open source as much of my code as I can, wherever I've worked. And I think it's important to make sure that we still remember how to build things as a society.

When you have these mechanical things you can take them apart and see how they work, but if you have just this black box of code somewhere nobody can figure out what that does.

And if the only code on the internet is example code, that doesn't actually do anything, like nobody's going to learn how to do anything.

Steve: In the example that you gave, Ilan, of Datadog anything that runs on your host is going to be open source. That almost seems like an expectation for developers.

Ilan: Yeah. I think it's great for security teams as we put software on your machines and, you know, you want to prove that maybe, it only sends metrics and doesn't, you know, make changes to your machine.

You know, that we're not making changes to your machine, or you want to know what metrics we're sending and why we're sending them. You have all of the code right there for an audit.

I think thats starting to become the expectation. I mean, there's still plenty of proprietary software that you can buy and download or get on DVD somewhere and then consume within your organization.

I don't think that's going away anytime soon. I just think people are starting to expect to have visibility, especially when it comes to developer tools and libraries.

If you're going to require some Ruby or Python library, you want to know like, what's it going to be monkey patching in my code? Because that changes the way my what a method I was expecting to do does, then I want to know what's happening there.

Otherwise as you're trying to debug, you know, some odd behavior, you don't know if there some magic in this black box, as we were talking about before?

Is it, you know, what's going on there? And this way you have that visibility. I mean, for us it's also just the idea that we couldn't instrument everything under the sun.

It's just not possible. Like, there are so many long tail services or projects or products out there, whether it's commercial or open source, you know? And we only have so many human hours within Datadog.

If you have a database that you want to monitor and you're the only person under the sun that has that database, maybe it's something you built in house or maybe it's just not a very popular database.

You want to get metrics, you want visibility and so having a plugin model that was easy to contribute to or easy to write for, makes that possible.

Otherwise you're blocked on us and, you know, we want to meet all of the needs of all of our customers.

But you know, at some point, grabbing something like MYSQL or Postgres that's used in probably every customer.

Every one of our customers likely uses one of those databases and it's going to happen much more quickly than say if you wrote a bespoke database at home and you would like me to monitor it.

Steve: And as both a tool and a company, drawing those lines seems like an interesting and important exercise.

One, maybe by identifying it, you're identifying where your company is going to provide the most value. Like here are the parts that we are core service for, and then here are the other part that we're going to let people extend.

And understanding that distinction seems useful.

Ilan: Yeah for us, the value I think that we provide is all of the smarts that are in our platform.

Whether it be things like anomaly detection, or outlier detection, or connections with all the third party services that we integrate with at a service to service level, the ease of use of our UI.

That's where the value that we provide as an organization is, I believe. I don't want to discount the agent. I work very closely with our agent team, especially since it's one of our more popular open source projects.

But, you know

I think the ability to be up and running in seconds with an agent that's collecting the majority of the metrics you care about is very important. It's an enabled end story that you have to do if you're in our space.

But there's lots of agents out there, right? Like I think ours is great, but there's other tools out there, whether it be CollectE or some other solution for collecting metrics and sending them to us.

We have APIs for that as well, and so the idea, is if somebody were to write a better data log agent and send the metrics our way, as the Stripe folks did, I don't know that that changes the value that we offer them.

We still offer them this amazing platform around alerting them on only the things that matter and helping them identify problems in the metrics that they're sending us.

Like you said, I think identifying where the core value that you offer your customers is is important from the start.

Steve: I mean that seems to be a testament that somebody would rewrite you know, the agent entirely, basically, I think speaks to the fact that the core value wasn't that for them, because they rewrote it themselves.

Ilan: And they and a lot of our customers that do end up writing custom agents or collectors or whatever it might be, do still, you know, they often still do use our agent. They just use it in a place that what they wrote is to scratch a particular itch.

And then there's other places where our agent or our projects meet their needs. You know, it's not just big things like the agent or the plugins. You know, because the APIs are available there.

I've seen people write like, Erlang SDKs for the Datadog APIs, or you know, other languages that we likely were not going to touch right away. Not because they're not important, just because the majority of our customers are elsewhere.

And as much as we love the Erlang community or we love the Haskell community, you know, those are not the number one people sending us metrics at any given point in time.

So you know, we're going to start off with more common languages, maybe like, you know, the JAVA world is huge, the PHP world is huge, the Python world is huge.

These are places that you know, we can get the most bang for our buck early on so that's where we started. I'm sure there's an Erlang or a Haskell user that feels like I've slighted their community.

I do not mean that in any way, it's just sort of the realities of number of users that need monitoring.

David: I'd be interested to know like, how does open source sort of fit in at Datadog? Like, do you consider it, is it sort of like a support service for existing customers?

Is it a way to reach new customers? Do you think about it as part of like the funnel of, you know, when people are going to pay you more money they tend to write more open source code related to Datadog?

Ilan: In our development of open source software we have teams, I mean the teams that work on open source components. They follow an open source first model.

So, our agent team for example, the integrations that they write and plugins that they write, it's all happening in Git. So you can call us up and you can ask us what our roadmap is for something specific and we're happy to have those calls with our customers.

But you're likely to get a much better view into that looking at the branch for that, you know, for features that we're working on in the agent because that work is just happening right there.

It's not like there's a closed source repository behind our firewall and then once in a while we sync it out. It all happens right there in the repo. And that's been great because also our customers, sometimes they'll show up and they'll say, hey, I want this feature.

I know there's a ticket for it, I don't see it, or I noticed that you're working on this, I found a bug in your code long before you've released it. Here, let me send a pull request.

You know So we've had this happen. The folks at Lithium saw we were working on an open stack integration.

They didn't want to wait for it to be done, they grabbed the code straight out of our code source control, tried it out and they're like, hey! You don't have these features we want.

We're like, yeah, the card was in like three sprints from now but thank you for the pull request, you know, we can now work on the other features that you want.

We had some folks at, you know, another customer here in Palo Alto that saw one of our integrations, wanted some things that you know, again, there were issues for in the queue. We were going to get to eventually, probably very soon.

Steve: Not fast enough.

Ilan: Apparently not fast enough. We'll go to a pull request for it. But they can, they have that visibility into our development process on our open source projects because it's there in the open.

Steve: I'm also curious how often roadmap ends up getting pushed into the company that way. Like if enough customers are demonstrating interest and contributing code, does that ever alter the roadmap for the rest of the company?

Ilan: It definitely shows interest from our customers. I mean we're very customer-oriented in how we determine what we're going to work on.

If it's a request we're hearing often, that's what we're going to, you know, maybe the squeaky wheel gets the grease. But really customers are showing demand, we're a service, so we only continue to make money when our customers see that we're providing them with value.

So if they are making a demand because it's something that they need, we're going to spend the time to help make that happen. I think containers are a great example of this.

It's something that we saw, we saw the Docker folks working on this long ago.We're like, yeah, that looks interesting, we should spend some time on it. The first Docker plugin that showed up for Datadog was a pull request from a community member.

You know, at the time, did we plan to have an entire team focused on container related technology and metrics? Probably not. So yeah, it shifted that. Now I don't know if you can say that the pull request shifted our roadmap.

Or did the industry or did the infrastructure space changing entirely based on what Docker and CoreOS and other container folks are doing, did that change our direction or was it the pull request?

I don't know. I mean, I think they probably go hand in hand. So right now one of the things that we're working on is making it easier for you to contribute plugins to Datadog.

That has changed our roadmap for the team. We realized that having contributors wait weeks, months, whatever it might be for feedback on a plugin they sent us is not a great.

It's not a great experience, and so one of the things we've been working on for a bit here is splitting up those release processes, making it easier for individual contributors to release their own plugins for Datadog.

And yeah, that wasn't on the roadmap, but based on the number of pull requests we were seeing, and sort of some of the frustrations in our community, and from our team on how fast they wanted to have these things move, the roadmap changed.

So I think any roadmap changes with input from your customers, otherwise it's probably not a good roadmap.

Steve: That was something we talked about with Michael from the Node Foundation a few shows ago, of how creating the open source process, creating the process for people to contribute.

Making sure that that's streamlined and that people feel that they're being listened to and that they're code's being accepted, that is a really important part of the open source experience also.

Ilan: Yeah. And in the same note, I mean the Docker folks have talked about this a few times in their keynotes, right. I think Solomon mentioned this in his keynote at Oscon.

Yes is forever, no is maybe temporary, and so, as you're accepting pull requests from your community it's also reasonable to say, it's important to know what you're not willing to accept.

And, you know, there are projects where it's basically very permissive, you know, you can basically make a pull request, merge your own pull request, move on, and there are projects where that's not necessarily the case.

Knowing where you fit on that spectrum is important. And communicating that out to your community, I think we can probably do a better job of that, and we're making efforts towards that.

David: It seems like a nice transition to talk about the downsides of open source. I was listening to you talk about the Docker support that was contributed and it sounded like that maybe even came before you had planned to do it.

So, yeah, if you get a feature like that, it's great, because it enables a whole bunch of new things, but now you're also like, you're maintaining that code now that you've accepted it.

Ilan: Yeah. You know, in the case of Docker that's turned out very well for us, I mean it's, you know, folks have probably seen that there's a Docker adoption study that's up on the Datadog website.

The title's "Eight Surprising Facts About Docker Adoption". We've been doing this about every six months or so. You can tell from the numbers there just by the explosive growth of container usage that like, containers have done us well.

So I'm very happy that we took that pull request. But yeah, it changed the direction of things. But if you want to come drop an amazing idea on my lap and say like, here's a great way for you to succeed in your business.

I'm not going to say no to that. I don't see that as a downside, right. It's another way to interface with your customers and learn from them. Otherwise, historically the way you hear from your customers is like you have an account manager and the feedback comes up through them or through support.

Or you have these quarterly business reviews where like, the whole team gets in front of the sales guy and a product guy and they're like, this is, they give you input, their speil and then you come back and say, but that's not any of the things I wanted. I wanted these things.

And then you kind of negotiate back and forth. I think having this other angle to connect with our customers, which is, you know, through this community, offers us a little bit more.

It's a little bit more of a relaxed conversation, it's a little bit more of an open conversation of like what do you need and how can we help you, how can you help us? Let's work together.

Again, I don't see it as a negative, you know? There's other pain points and, you know, in the open source world, but I don't think ideas coming and flipping your business around is necessarily a problem.

David: Yeah, definitely not. I mean more just about like, the long term maintenance of new things that are submitted, and having to say no often to some things because of that reason. It could be the greatest idea in the world, but it doesn't make sense for most of your customers.

Ilan: One thing I've loved about our community is, you know, we don't say no a lot, but when we say no to some things we say, this doesn't look great. Doesn't meet our coding standards.

Or it seems like it's going to have an impact on the user systems or whatever it might be. Oftentimes other members of the community will pipe up and be like, yeah, you know, you're probably right.

They will often reinforce some of the feedback that we're offering in the PRs. And we've started to see situations where, over the years customers have started to help each other out with their pull request.

If somebody has written a plugin and we haven't had a chance to get to it and offer our feedback, you know, there's oftentimes other customers or community members that are going to pop in and say, oh hey, the issue that you're, the tests are failing because of X, Y and Z.

Or, you know, this is likely not going to pass lustre, you know, later on in the review process because you're doing something inappropriate. But again, you're right.

I mean, having more code to manage and support? Yeah, it adds an overhead, but I think, you know, if our goal is to monitor all of the things and make it easy for you to do it, then we're going to need more code for that to happen.

Steve: I remember Mitchell Hashimoto of HashiCorp talking about a similar idea, that they've taken a more heavy handed approach to contributions and that they're more careful about how they review them and that they want people to contribute in a certain way.

So they try to encourage it but they also work with people pretty closely for important PRs because they want them to be done a certain way.

David, like what you were saying, that introduces entropy, like, you have to now manage all this stuff and allow it in, but yet, then you get the right kinds of contributions, I guess.

Ilan: Yeah. It's important to recognize people for the work that they're doing as well. So often, maybe not often, but

Once in a while we'll get a pull request and we're like, you know, this is an amazing idea. I like generally what you've been doing here, but we're going to need to change this.

And if you don't have the time, don't worry. We're happy to do it, but when that PR lands, the person still gets credit in the changelog.

They still get recognized in all the places that they would get recognized, even if that commit had landed unchanged, because the kernel of that, the idea that they brought forth, it was their work.

It was their inspiration that, you know, that brought this along. I don't think credit is everything, but it's important to recognize your contributors and it makes people feel good about the fact that they participated.

Steve: Yeah, I've noticed in the company release notes there's always the shout outs to everyone that contributed that month.

Ilan: It's important to recognize people for their work. I mean, again, yes, they're probably getting paid by their employer to do something here, but often they're doing this in their spare time as well.

And so it's important that your community feel like they're part of something bigger, and that they're connected with each other, and that it's not just a one to one vendor, you know, consumer producer relationship.

Steve: That, too, is a funny twist on all this for me, is like, here's a person at some other company.

Their salary's getting paid not by Datadog or by Combox, but they're spending potentially 100 per cent of their time, for some, you know, period, writing code for this product.

Ilan: Yeah, I mean, but is that different than say let's say Twilio or Amazon. They have APIs that customers consume and they build things on top of.

Is it strange to you that somebody would build an application or run the Twilio API and then release the libraries for it? Or the same with EC2?

Steve: That's a good point. I guess it's kind of a slippery slope. Like anyone that's ever posted anything to Stack Overflow, you know, you could argue like, why are they wasting their time contributing knowledge back to the community?

Ilan: Yeah, but I mean, I think the difference is, again, it's about, what is the value that you're offering your customers and where is it at?

And if you've picked that demarcation right, they're not going to have a problem building around those APIs, or building into the projects that are sorta at the edge or the fringe of that.

Steve: Yeah. I think that brings up important, like, altruism and community. It's great that those are actually values that this community shares.

Ilan: I guess, I mean, I'm not, I don't try to kid myself here, right. I don't think people are showing it to the Datadog community because they want to make the world a better place and that's why they dropped the PR in there.

They care about their own needs, which is, they wanted to monitor a thing, and we either didn't monitor it in the way they wanted, or we didn't monitor it at all. Sure, occasionally there's altruism in it.

But you know, I think this is self-serving and I guess this is also the difference between open source software, you know, and free software, to some extent as well.

You know, the philosophies around it. You know, a lot of folks think of open source software as being more of a, it's more like the means to an end. This is a technically better approach, or this is, you know, I need it.

It's again the visibility that we talked about earlier, into the code. On the free software side, again, it's all about that altruism. I think it's, you know, I want to make the world a better place and that means everything is open and free and accessible by everybody.

I'm sure I'll get corrected by somebody in the comments, but that's the way I read it.

Steve: Yeah, that's true. I mean it's an important distinction between Linux and Datadog.

Ilan: Yeah, again, we are a hosted service but I do think the open source component of it enables us to be successful on that side. We also use a lot of open source internally, right?

I mean, whether it be HAProxy or NGINX or, you know, Cassandra are some of, you know, Kafka, other things that we use internally, Redis. I mean, these are all open source pieces of open source software that we try to be members of those communities as well.

So whether it be writing better documentation for them, or, you know, contributing back when there's a feature that we need in those upstream projects.

Like I said, I don't think there's any company out there these days that doesn't have some open source component in it somewhere.

Steve: You hinted at something before the show about risks and that in at least one case I think, someone has come along and taken open source pieces of Datadog and basically founded another company on the back of that?

Ilan: Maybe not founded, but you know, there's people that've forked our agent and used it in different ways. You know, and basically the agent takes a config option of like what API do I push this to?

You know, they've either changed the type of API that it talks to or they've changed the URL and they built their own service behind it. I think that's, again, I think that's okay.

There are projects that we use that had started off as somebody else's projects and we, you know, eventually we contributed and eventually had the need to fork it.

You know, I think like every project maintainer, we'd love for when, you know, if they make fixes or improvements or what have you, we'd love for them to send PRs back.

But, again, we know what license we chose. It's not by any means required in many cases that when our open source projects are consumed that they're re-released.

The requirement is generally, you know, a copyright notice and some attribution. Again, you read the contracts that you sign and then you abide by them and that's both us understanding that, you know, we've put something out there that the world may consume.

And the people that consuming it understand that they have to you know, abide by that as well. Again, is there a cost to a fork? No, I mean it's code somewhere else that we're not interacting with.

Our customers are usually not running those forks, so it I don't see any negative to it. I mean, we'd love to collaborate and get those features back.

David: I think it's actually kinda nice. It's almost like a good balance, right?

Like, you know, you have Datadog, the service that you're selling, but like the agent is open source and all the plugins that all these other people that aren't you have helped build are all open source.

Maybe not every company in the world has to figure out how to monitor Open Stack, you know, individually, a thousand times.

Ilan: Yeah. Like you said, it's, you know, you can take apart a piece of machinery and you're like oh, this cog hooks up to this cog, that's how these two pieces work together. Doing that with compiled software? Maybe not as easy.

Steve: I'm curious, although I don't know if this is a great way of asking the question about the risk, the decision of deciding like, here's our core value, here are the pieces that aren't open source.

Here's the service that we're going to provide, and then here are the pieces that we're going to open source and let people contribute to or customize.

That if you sort of miscalculate there, and you maybe underestimate your core value, that as people are building and maybe unintentionally or whatever chipping away at that core value, can you lose your business that way?

Ilan: I mean, I think that's the risk of a lot of the open core projects, right? So, in the early 2000s, you know.

So, in the early 2000s, you know, there's this open core business model which is, you know, a bunch of startups figured, you know what, what I'll do is, I'll make an open source project that doesn't have all the features or maybe isn't easy to deploy, and then I'll sell an enterprise version of it that solves all of the challenges the people in the open source community had.

And has another label on it. So maybe you call it a freemium model, maybe you call it, you know, open core. Whatever name you want to give it. I think some companies have helped strike that balance very well in that they like, they add new features.

And at some point those features migrated into the open source project, then they go find another feature to add. That definitely keeps you on your toes, you have to keep finding value to give your customers.

And hopefully you're also differentiating on support and some other services that you can offer and helping, you know, having the relationship be of value to the customers that are licensing your proprietary software.

But there's other companies that have gotten the balance horribly wrong and so you'll get like a pull request for, I don't know, user management in your project and you're like no, that's an enterprise version.

We wouldn't want you integrating this with active directory, that's for enterprise users, you shouldn't be using the open source model for that. And I've seen this in a number of projects. I'm not going to name and shame.

But that's not an ideal situation because now you're in a combative relationship with your community, you know. They've gone through the effort of building a thing and you're like, no, I'm not going to accept that.

Or you need to go do this off to the side, I want to hide your work. I'm not always as excited about that. I think, when people go down the open core model, they maybe have not figured out like where the right balance is for that.

It's not my favorite approach for running an open source project. I think we're lucky in that we're a service. I think the SaaS world has definitely changed the way we consume software in general.

But open source specifically as well, both in terms of the licenses that now exist for working with that, but also in terms of just what users expect, right.

David: I think definitely like software service like sort of, all of the as a service things have definitely changed open source. I mean, even, like, down to their purest intent, right?

Like, the GPL doesn't work when, the GPL's a distribution license, so if I take a piece of GPL code and apply it on my server and sell that, I don't have to release the code.

Ilan: Right.

David: So yeah, it's, in a way it has sort of like gone around the intent of open source in some ways, but it has also, I think, allowed some really interesting open source businesses to flourish.

where they can sort of say we're going to like, build some value around this as a service part and then give away whatever on top of that.

Ilan: In a way, I think that's that's true and there's licenses that approach that situation as well, right? There's the AGPL, for example, which, you know, again, not a lawyer, do not take this as legal advice.

But where, you know, accessing your software over the network is considered distribution. Again this is about picking the right licenses and making the right decisions about where your border's going to be between your commercial offering and your non-commercial offering.

And, you know what, there's always the option of, you know, sell a services and support model, and you can run a business that way.

There are many organizations that have been successful at that. I guess it's just a question of your definition of success and how you want to get there.

Steve: Do you think we're trending away from that? The service and support model?

David: I think so. I think it's just like, you can't really build a long term sustainable business unless you're Red Hat around that.

Ilan: Yeah, I think it depends on how you define long term and sustainable, right? Again, generally if you're a venture-backed startup, they have a very specific definition of success and maybe it's harder to attain that model on a service and support piece.

I mean, again, Im not a lawyer, not an MBA, you know, I'm going to use the wrong words here, somebody will correct me later but, the size of the win your investor's going to get is a little bit different in a pure service and support model than it is over product licensing or what have you.

And so, I'm not discouraging folks from running that way, it's just, again, where do you see your company going? Where are you trying to get to? From point A to point B, and then you pick the right path.

I think there's lots of people that have built small or medium sized sustainable businesses on pure open source, you know, consultancies or what have you. There's of course Red Hat on the other side of that, there's the MYSQLs of the world.

I mean there's success stories in that space. It's just a question of can everybody do it and how big can they get?

David: Alright, I feel like, this is just anecdotal from my own personal experience, but I feel like companies are moving, the customers are leaving the model even more than the companies providing it.

Ilan: If this is a topic that you're interested in and you're listening, there's a couple of great pieces out there on the internet. One's, look up anything written by Stephen Walli or John Mark Walker on Open Source.com

Among other places that they've written, hey've both written some fantastic posts on this about the delineation between your product and your open source communities, and open source versus free software, and where all of this falls.

And they speak to it much more eloquently than I can, and you should have one of them on the show someday.

Steve: How much can a random person learn about Datadogs roadmap by looking at the open source roadmap, and is that a risk to a business that's trying to publish you know, have their open source community flourish, but at the same time build a business?

Ilan: I mean, look, PR people like a secret. They want to have a big pop, they want to go to like the TechCrunches of the world or whatever the publication is, and say, here's this new thing we just released.

And sometimes that's hard with open source projects because everybody knows what's being released. It's been talked about on the mail list for six months, or it showed up, you know, in our case, it showed up, it's been in as a pull request there for a while.

Is that a problem? I don't think so. I don't know. Someone in PR might disagree with me. I think you can still announce a feature as being done once it's actually done, even if people knew it was coming for a while, right?

It's not like you know, you go to the big Apple keynote once a year. It's not like you don't already know there's probably a new laptop coming today. You may not know the specifics of the laptop, but you know it's coming.

It's just, people like secrecy. I guess open source maybe flies in the face of that, but I think you can still be successful in your announcements and your PR. You just have to plan them differently. Because you actually had a, you know, the plan's been written there for a while.

David: I've had this conversation with a lot of folks at a lot of different companies I've worked at. I've always been sort of like, the proponent of open source.

I feel like if you're in the position of like worrying about the media leaking your features as news, like, you've got different sort of problems that are probably okay.

Ilan: Yeah. I mean, it's not like the Mozilla folks don't end up in the news pretty regularly anyways when they make an announcement, right? When they add a new feature, whatever it might be.

But, yeah, you could probably go on Bugzilla, again, six months in advance and see something's coming. It actually means they probably get covered twice. Once when the commit landed, and then once when it actually gets released.

Steve: I think like you were saying, getting in the news at all for most of us is a bigger problem than controlling the coverage, right?

Ilan: So, I don't know that it's a problem. I think it's just something that you need to take into account and, you know, make sure that your PR teams and your, you know, whomever are ready.

And if you're lucky, also, like you said, you know, maybe you don't even have a PR team. But just make sure that as an organization you know what's coming.

Steve: Alright. Thanks again to Ilan for coming on the show with us, and if people are looking for you online, where can they find you?

Ilan: So, irabinovitch on Twitter is probably the easiest place, or pretty much anywhere else. But yeah, Twitter's probably the best way to engage with me.

David: Thanks for coming.

Ilan: Thanks for having me.