Ep. #4, A Billion-Dollar Idea with Tobie Langel

Brian Douglas: Welcome to another installment of Open Source Ready. John is back from KubeCon. John, how you doing?

John McBride: I'm doing well, still here in Salt Lake City, but it's a beautiful day. Happy to be calling in, how are you doing, Brian?

Brian: I'm doing well and I'm hoping that folks will catch your talk at the event tomorrow or specifically on the recording, but I actually want to introduce our guest who is Tobie Langel. Tobie calling in from Switzerland, I believe. How you doing Tobie?

Tobie Langel: From Geneva, Switzerland. Yes, I'm doing great, thank you for having me, I'm very excited about this whole conversation we're going to have.

Brian: Yeah, yeah, so like I've crossed paths to you a few times when I was at GitHub. I think you had spoke to a few open source maintainers and I was in the background listening. We were actually at OpenUK early this year, we both gave a talk there.

Tobie: We did.

Brian: And that's the first time I think I met you face to face, but you've done a lot of work in open source and around open source. Do you want to introduce yourself? Tell us what you're doing, what's your background?

Tobie: Sure, absolutely, first, I do want to state that it is quite surprising to think that OpenUK was earlier this year. It feels like, I don't know, five years ago or something, like there's this time distortion going on.

So I started a career as a software front end web developer, completely by accident.

Now, quite a long time ago, I was actually a professional jazz musician and we were touring and we were trying to like figure out like how each of the band members could help the band. And my brother was really good at computers and so I thought, I'm going to ask him if he can teach me how to build a website.

And so he showed me the LAMP stack and I loved it, like right away, like, I had a scientific background in school and then just clicked, it was like, I really enjoyed myself and so I built this website, was like an online shop where you could order a CD and like we would send it to you and then every musician in town came to me and said, "Hey, Tobie, could you build one for us too?"

And so, as a professional jazz musician, I was entirely broke, obviously, so I thought, oh, this is a great potential side gig. And right about that time, Ruby on Rails got started and it was kind of like, oh, this is the perfect tool to sort of like build a SaaS around this idea of like building websites for musicians.

And as I started playing into it, I discovered the whole JavaScript libraries that did this script calculus, these great effects that you could do and I just like completely fell in love with the whole thing and essentially learned JavaScript by documenting the prototype JavaScript framework, then started co-maintaining it and that's kind of like how it got into tech.

And then from there, worked at Facebook, 'cause they were looking for someone that knew open source well and that knew web standards and I was fairly familiar with both. Then did more standardization work and then sort of came back to open source, but more from sort of like strategic business angle than from a developer angle.

So I still code, but mostly for myself. Now, essentially what I do is strategic consulting and I do that mostly for tech companies, but also increasingly for nonprofits and more sort of like governmental-ish work. So really operating in that space at this point.

Brian: Yeah, yeah, I mean I had a great conversation with you, which I would love to have on this podcast in regards to government, 'cause like we just went through early this year, this vulnerability with the XZ vulnerability.

Do you want to explain for the listeners who maybe not aware of that, like maybe if they were not paying attention to Microsoft and PostgreS at the moment, what happened then? And like your involvement in that? Obviously you didn't disclose the vulnerability or attack anything, but yeah.

Tobie: Right, so I wasn't directly involved with XZ Utils, but the high level story of XZ Utils is someone who got in the position of being a trusted maintainer of a project, turned out to have an agenda that wasn't at all the one you would expect from a trusted maintainer of a project.

And the way that person got into that position is essentially by leveraging the fact that maintainers are overworked and burnt out and aren't properly supported to maintain the projects that we all rely on. And as a result was kind of bullied into, or pushed into accepting someone as a co-maintainer on the project who had bad intentions.

And so, this kind of social engineering attack, is a pattern that the industry has seen for a long time, but it's like, it's the first time that it shows up like so blatantly in an open source project.

So the idea of getting the ability to live attack a project like this rather than have a vulnerability that is not managed, like a bug that is leveraged, like having someone who's in a position to essentially create back doors, that's something that was entirely new.

And so as a board member and as cross project council, vice chair, of the OpenJS Foundation, I was involved closely in what looks like a similar attempt against the foundation that was disclosed by Open JS in a collaboration with Open SSF and a blog post, I would say maybe a couple of months after the XZ Utils made the news, which had a similar pattern.

And without going into the specifics of the details of this, what I found really, really interesting, and so sort of what converges with the talk I gave on funding open source maintenance at Open UK that we were talking about earlier, is the fact that--

When cybersecurity incidents, or social engineering attempts at penetrating like an organization happen, regular businesses know what to do and well, I mean at least the savvy ones and have a security operation center and a bunch of processes and a bunch of practices of how to manage such a threat and then how to investigate it and then how to go to the authorities with that information and seek help and support.

What we discovered in the process of managing a similar situation as an open source, a set of open source project as in Open Source Foundation, is that open source by its very nature isn't structured in a way where there is a central security operations center that controls the network in which all of the software is built and that controls the development machines and that just maintainers can turn to to ask for help or to ask for investigation.

And we ended up in a situation where no one really knew what to do, essentially because expectations from government agencies were that, well someone would do an investigation internally and show up with data and like pointers to what had happened and well, we had no one to do this obviously and plus, it's not like you can go onto the machines of like maintainers across the world and try to figure out what exactly happened because you just don't have access to them.

And so to me that really showed, well, I mean, not only work that needs to be done by everyone to sort of like bridge that gap and figure out solutions, but also, and I think I explained a lot of this in, "Five Questions to Five Experts," article that I was invited to talk about this in.

But yeah, it really shows not only that we have all of these processes and things to figure out, but also that if open source becomes a stepping stone to getting inside of valuable targets where those open source projects are used, then we pretty much are going to have to build security practices that are similar to what those organizations already do to protect themselves.

And I don't see a concrete way for the industry and the open source community to do this without having paid maintainers that are supported by professional security operations centers. And so, these two things that I read at the sort of heart of my work for the last years, open source, sustainability, and then securing the supply chain kind of like collide, where I don't believe you can have security without professionalization of maintenance and sustainable open source communities around projects.

So that's kind of really for me, like the two big learnings are those. Like security and sustainability, like there's no security without sustainability of open source and we collectively have to figure out the governance structures and the structures and the processes to deal with threats of that nature in the future.

Brian: John, maybe you can help me out, 'cause I remember specifically early this year there was like a Go library that was like basically feature complete and everyone was very concerned on like who owes the keys and I dunno if you remember that from, was it late last year or early this year?

John: Was it Gorilla?

Brian: Was that under the Mux Project?

John: Yeah.

Brian: Yeah, that was the one.

John: Yeah, so what happened there was something similar where they were looking for new maintainers, having a hard time finding new maintainers and this was a like HDP routing library and Go that was used all over the place and he essentially was just going to shut it down and kind of sunset the whole thing because who was going to get the actual keys to the kingdom office.

Thankfully Red Hat stepped in and kind of like Tobie was saying, really like professionalized and kind of centralized the maintenance of that library, probably 'cause they use it throughout their products.

Something that was crazy to me about the scale of the XZ Utilities attack was really the amount of time, 'cause this person, Jia Tan, took years to gain that trust and then a lot of the commentary is also that this probably wasn't a solo individual, it was probably a group of nation state actors backed by some government or something to make this possible and then create a ridiculous SSH backdoor.

So Tobie, my question around that is like what kind of scale do you envision needing to be achieved in the open source to combat the kind of scale that is now clear around these kind of social engineering attacks?

Tobie: The angle that I use and that I used in the talk that I gave, me referring, the 1 billion for open source maintainers talk, is on the opposite.

I look at it from the opposite side of the lens and the way I look at this is if you do sort of back of the envelope math on how much worldwide we collectively spend on software engineering and per year, I'm paying essentially software engineers wages, it is roughly a trillion dollar.

And so when you take 0.1% of a trillion dollar, you get a billion dollar and 0.1% is not a lot, but a billion dollar is a lot of money and, so that's one way of looking at it.

And then the second number that I just bring up is, the size of the open source stack in shipped products, which is roughly depending on how you count, 75 to like 90 or something, it doesn't matter, it's like way more than half and now, when you combine this idea that 0.1% is a billion dollar and it sounds like a huge amount of money and it would be crazy for the open source community to say would be useful to pay for maintainers, yet the stack is like 80% open source.

The real way of looking at the problem is saying if things were more reasonable, what would it look like? And so if being reasonable is taking 0.1% of what is spent on software engineering every year to support 80% of what is like being shipped, that would essentially pay for something like 10,000 maintainers, or maybe let's say 8,000 plus like management, support, training, like offsites, whatever.

And can you just imagine what that would do? Like even if you suddenly had 1,000 professional trained maintainers supported by security experts focused on like the, I don't know, 10,000 most used libraries in the world, like it would be huge, it would entirely change everything.

And so that's the angle to look at, because we don't really know, it's likes really hard to say upfront, "Well, here's what's needed."

I'm seeing the Sovereign Tech Fund make such a huge difference with tiny investments, like at the OpenJS Foundation, we were very lucky to get like a really big amount of support from Sovereign Tech Fund.

And I mean, like really big compared to like the fund, but we're talking less than a million dollar, rough, I mean, I don't exactly remember the numbers and I think they're public, but we're talking about like that kind of money and the amount of like good that that did for the security and the practices of the organization, they're huge and when we had this XZ-Utils-like threat, we were able to get direct support through that fund and the security person we had as support and it was huge to have someone that was knowledgeable and had the contacts, knew how the things worked, and was able to just like walk us through it.

So imagine like that on the scale of like 100 or 1,000 for the whole industry, like long term, wow!

John: Yeah, part of me wonders if it's even more than that. I had a very interesting conversation with a maintainer here at KubeCon about some piece of Kubernetes infrastructure they built in the open source.

And this is one maintainer with one project, but by some estimates of things that he's been looking at, there's like some many billions of dollars of business flowing through this one piece of software, on top of all this Kubernetes that he maintains and has had a hard time maintaining this and having a hard time finding funding for his time to actually work on it.

So yeah, the magnitude of scale around this are pretty mind-boggling.

Tobie: Yeah, it's crazy, right?

When you look at numbers, it doesn't make sense that we're struggling to just provide very basic support, when the end game is obvious. Now, I do really believe that the legislative push that we're seeing both in Europe and in the US is going to be a turning moment for this, it's going to make or break it.

Brian: Yeah, can you speak on the pushes on legislation, because I'm not sure if folks have been paying attention that closely, 'cause I know like the DoD and the US have some requirements coming up in the future around SBOMs, et cetera.

Tobie: Yeah, so, I looked at the US push from a bit more of a distance, because it is, in the way it is driven through the industry, different and probably a little more lightweight than of what Europe is doing with the Cyber Resilience Act, the CRA.

And so I'm more familiar with the European side also because I'm directly involved in it. The CRA is essentially a set of obligations and requirements for compliance.

To be able to place something on the European market, you're going to to have to be able to demonstrate that you comply to a large number of security requirements.

And of course the process of that writing that law up kind of bumped into, oh, and there was open source, at some point and that was like kind of like a shock to everyone on all sides of the equation.

And at first how it was addressed was incredibly concerning, but then folks on both sides did some amazing work and some amazing collaboration and frankly came up with legislation that I think is frankly like surprisingly good when you look at like, the complexity of the problem.

And what happens is that there's a clear understanding that the liability and the compliance requirement falls on the people who are actually building software and building products using software and not on the open source community itself.

But it is also quite clear that without the open source community collaborating with the folks who are actually shipping products, it's going to be impossible for the folks shipping products to comply with the legal requirements.

And so, with this, you really have like the incentives for people to work together and essentially sort of like fund the fact that it is necessary to shift security considerations left.

And to some degree, like either that's going to work out and is going to generate support either directly through government funding or through other means.

Like in the European CRA there's a provision for a system of attestations that open source stewards, so essentially code hosting foundations, but not only, could leverage to essentially test the security aspects of the software that they're providing and could somehow monetize those attestations.

And so I think we're going to see some form of brokerage happening through mechanisms of that nature. So I think that there's a forcing function now and think it's going to either like work out or it's going to explode, but I don't see this going like this for a long time without like some pretty drastic changes? Maybe, I'm just optimistic.

Tobie: Yeah, I mean, it seems to be more top of mind, 'cause the CRA stuff, I got wind of that earlier this summer and got to dig in on that stuff. and even at OpenSauced we had like doubled down and tried to support some of that, in our product.

You mentioned NASA in passing and like John you told me a story of like folks reaching out to projects you maintain asking for SBOMs and like all these requirements where you just weren't prepared for.

John, do you want to share your background, your Open Source journey and how you've been exposed to this?

John: Yeah, sure, so I've maintained a library called spf13/cobra, which is a very, very mature Go library for bootstrapping command line interfaces. It's been around for a long time, it's kind of become ubiquitous in the Go community.

But every once in a while I do get a strange email or a strange request that often revolves around like, "Hey, we need you to like fix some vulnerability that's like way deep in the dependency chain of this thing."

Or, "We need you to supply some sort of like certificate of origin around like all these libraries and SMOB," et cetera, et cetera.

And I did get an email from NASA about this and I think a lot of people in the Kubernetes community got the same email about providing some kind of SBOM and it was very bizarre, because like I was not at the time well equipped enough to be like, what do you want?

Like there's the go.mod file and the go.sum, like that tells you what's in this library, go look at that!

But I think what some of these enterprises, some of these government agencies, are really looking for is kind of that almost assurance and kind of like you said Tobie, kind of shifting left and maybe putting some more of that responsibility on the open source developer and maintainer, which I just don't think many people in the open source are well equipped at all for.

Tobie: Well I also think that like, it's not for them to do that. I mean, at the heart of the open source contract there is the, "And the software is provided as is," like that's the contract.

The contract isn't I'm going to do this for free and then I'm going to now be liable for like you being compliant to something that I have no idea what it is for no money. I mean like that just doesn't compute.

So I don't want to fault the organizations who are trying to comply to legislation that is being forced upon them for good reasons, like, to be very clear, I am a 100% supportive of legislation that is making software more secure, improve its privacy, et cetera, because those are things that without government mandate is impossible to have organizations to focus on.

I mean, this is just like we have laws and like governments for a reason and one of the reasons is that there are things that without laws and government just never happen because the market incentives just like don't align, things that we care about that don't happen because the market incentives don't align.

So I'm 100% for this, I think it's great. However, folks who need to comply through this suddenly are faced with these whole areas that are just not structured in the way that their process is designed to handle.

So the way you handle this is, well, you use something from a third party and that third party like sells that thing to you or licenses it to you and it comes with a bunch, meets a bunch of legal requirements, it has an SLA, like there's a bunch of things that comes with it.

And suddenly like, oh, but 3/4 of like what we depend on or like these odd licenses and like there's no one when we pick up the phone? Like where's the service agreement contract? Like, who's the person that we can call when there's a problem?

And so I don't think you can... I mean, sure, it's obviously funny to mock on these situations, 'cause they're cute, right?

But at the same, it is just like that's how the whole economy and the whole system is structured and the open source, the open innovation model is new, it's not like how the rest of the world functions and now we have to account for it. So, it's kind of normal that go-to reaction is, "Well, let's make them do this."

I mean, roughly speaking, there is no way, and it makes no sense to have a stack where everyone is securing things and all of their dependencies and their transitive dependencies downstream. Doesn't make any sense, it's absurd! The whole point of open source, the whole point of collectively scratching our itches together is to avoid this problem. So, 100%, this has to be upstream, but it has to be funded.

And the funding either needs to be sort of like driven, collectively towards maintainers or it needs to be state supported or it's going to get vendored, you're going to have some vendors that are going to say, "Well, we secure these whole pipelines and we do the legwork and we sell you a vendored open source project or project collective with a license that is not an open source license."

Like the license has all of these guarantees that comes with it. And all of these different options are perfectly fine, I suspect we'll see combinations that are driven by what kind of usage of those projects you're looking at, where they sit in the stack, whether they're more into defense contracts or like depending on kind of the industry, the risk profile, et cetera.

But we'll need to see some commission of all of this stuff, because if not, I mean the idea that like a random maintainer should be able to provide all of these different attestations and docs for free and have just the time to pick up the phone and answer these requests or answer emails, that's absurd, doesn't make any sense.

Brian: Yeah and you mentioned like state funded and vendor like funded like solutions, like folks, I've seen a lot of folks working towards this. As we're winding down, 'cause I want to get us to picks and round up the conversation.

We're actually working on a newsletter, so like a lot of the stuff we'll have linked in the newsletter, also the show notes, some of the links that you mentioned earlier with the five by five questions, stuff like that, we'll have that in the show notes for folks, so definitely check that out on Heavybit.

But two things I was going to mention, so came across like my inbox, Department of Homeland Security in the US specifically has a call for proposal for, it's like a $1.7 million contract for companies to solve the SBOM, but also the sort of like the problem where John had, folks trying to request this information.

I don't know where that's going to go, but just wanted to shout out to anybody who's in that space, that's available for folks. And then, I don't know if you saw a GitHub Secure OSS Fund that was announced like a month ago or maybe it was only a few weeks ago?

Tobie: I did see it, yes.

Brian: Yeah, so I think it's still pretty early, it might not be a fully formed thought yet, but I definitely would love to have the GitHub team on the podcast to talk more about this.

But the idea is like pay folks to secure open source and give them the funding to actually set up their projects for success. That's the TLDR, there's probably a lot more context that will be added to that in the future.

Tobie: I mean look like, I just want to interject here, 'cause I think at some point, like we need to try stuff, right?

Brian: Yeah.

Tobie: Like there's no silver bullet, there is going to be a combination of different things that like end up making this sustainable and workable and we just need to try stuff and like somethings are going to work for some communities or some projects and others for others and so, folks should just try stuff.

So I'm all for like any kind of experimentation in that space right now, we should do.

Brian: Excellent, well I would ask you a question, Tobie, which is, are you ready to read?

Tobie: Okay.

Brian: Excellent, Johnny, are you ready to read?

Tobie: As always!

Brian: I've got some reads, I'll have one that I'll share now and then one I'll punt to you, which I mentioned earlier before we hit record, Tobie, but we specifically, OpenSauced, so John and I both worked at OpenSauced, we actually have, as of today we're acquired by the Linux Foundation.

So vision still continues forward, would help the Linux Foundation solve some problems in getting intelligent insights into not only just GitHub repos but also Linux Distros and et cetera, so we're going to continue that vision over there.

So John's going to lead AI and infrastructure, I'm going to manage some developer experience for the LFX platform, but OpenSauced will continue, we have a blog post on our blog and a LinkedIn post as well if folks want to dig in there and find out more details, if you are an Open Source user.

But wanted to share that, 'cause that's a read, definitely read that. I had to read about the Swiss government releasing its software as open source and the requirements and with you being in Geneva, Tobie, I wanted to get your feedback on this, 'cause you might have more context.

Tobie: Yeah, I do have more context, because I actually read the law. So I don't know when this came out, I don't know, it was like a bunch of weeks ago, and everyone was like, "Oh Tobie, you're Swiss, like you must be so excited, right?"

And I went to look at it. And so the backstory is one of the cantons, so Swiss cantons, like Switzerland is like the super tiny country that is structured from a government perspective, well very close to the US, it's like we're a bunch of like tiny states, we're called, "Cantons," and we have a confederal government.

So one of those tiny cantons created open source software as part of like a agreement they had was like some supplier and made that open source and shipped it and they got sued by a company that builds that kind of software in Switzerland, who was like, oh, this is an anti-competitive practice, you're essentially like ruining the economy, blah, blah, blah.

And so that went to court and as a result they formalized in the law, the fact that it is illegal for government to create open source. And so there is no mandate at all, this is not what the law says at all, it says essentially like, "Other parties cannot sue a government, a Swiss government or canton government, if it wants to release open source."

And it even says in the legal text, "Granted that doing so respects the intellectual property rights of the party that has created the software." So this is not at all what people believe it is, unfortunately.

Brian: Wow, that's amazing, 'cause, yeah honestly, I'm glad you you gave us the actual real news on that and read the law, 'cause I was still going by the headline.

Tobie: Oh yeah, everyone is, I know.

Brian: Even reading all the blog posts, yeah, it's like kind of unreal, but I guess you kind of had to read on the other side of the box for stuff like this.

Tobie: I think like there's a small insight in there, which is we all tend to really hear the part that we want to hear when we read stuff.

And it's like, oh, this is exciting, like, a legal mandate for open source and then we all run with it 'cause it's cool and it's sort of like, it's well aligned with our belief system and what we think is right and what we think is good and then we sort of like, stop there, don't dig further and like there wasn't a lot to dig.

I think the main article itself, like because it was in German, I don't really read German that well at all, like, I was taught in school, like I'm really bad at it, so I put it into Google Translate and it was just spelled out in the like last paragraph, quite clearly. So it like played to the music that we wanted to listen to.

Brian: Yeah, it makes sense. There's a lot of that going on right now in the global political sphere.

Tobie: Yes, let's say it's global.

Brian: Yeah, excellent, John, you had a read, do you want to share?

John: Yeah, sure, so it's a short read on Spotify "Creating a SaaS product for Backstage," which if people aren't familiar, Backstage is a sort of developer portal for discovery of services and ownership and has all these plugins and integrations and is a really awesome, awesome project and can really help big orgs work across tons of teams and tons of services.

Red Hat also has something, some kind of service for this, but I think it's more like on-prem focused for developer portal. I just find this fascinating, 'cause for years people have been asking about, "Is Spotify going to become essentially a B2B business?"

Where obviously Spotify is more a consumer product, but it seems that they're finally entering the enterprise business realm with portal for Backstage. It's very, very interesting to see where this goes.

Tobie: I'm speechless about this news. It seems like so different than what their regular business is.

John: Exactly, yeah, that's what I find so fascinating about it is like such a 180. But I mean, it's a great product and they built Backstage internally, so if somebody was going to SaaS-ify it, they got it, right?

Tobie: Who knows? Maybe it's going to be their main source of revenue in a couple of years, right?

Brian: It's not too far off from like Facebook and now being in the VR goggle game, 'cause like, I'm not updating my status feed.

I know you spent some time there early days, but like Meta is like a totally different company than how I engaged with it than when I did in college, for sure. So maybe Spotify has an out in the future.

John: You could say the same thing for Amazon. Like, Amazon was retail and now Amazon is cloud, like AWS is that entire business, basically.

Retail and I'm sure some people are going to fight me about this, but Retail and Audible and the rest of their businesses basically serve to use Amazon web services right under the hood.

Tobie: I mean, it's a really big chunk of the revenue, for sure, but the rest of it is quite profitable too and the whole logistics aspect and the whole marketplace, all that stuff is highly profitable.

John: My takeaway is to go read the latest revenue report for AWS from this.

Tobie: There we go, that's my read! I came without a read, now I have one!

Brian: Quick read, just some light reading over the weekend. Well, with that, Tobie, thanks for talking about open source standards, your exposure into the space.

I feel educated about the Swiss government and their sort of, I guess, validation in open source at this point. And listeners, stay ready!